Category: Tutorials

Rapid, Secure Patching: Tools and Methods

Rapid, Secure Patching: Tools and Methods


Charles Fisher
Mon, 01/29/2018 – 10:45

Generate enterprise-grade SSH keys and load them into an agent for control
of all kinds of Linux hosts. Script the agent with the Parallel Distributed
Shell (pdsh) to effect rapid changes over your server farm.

Servers, HOW-TOs, Security, SysAdmin

It was with some measure of disbelief that the computer science community
greeted the recent EternalBlue-related exploits that have torn through
massive numbers of vulnerable systems.
The SMB exploits have kept coming
(the most recent being SMBLoris presented at the last DEF CON, which impacts
multiple SMB protocol versions, and for which Microsoft will issue no
corrective patch.
Attacks with these tools incapacitated critical
to the point that patients were even turned away from the British
National Health Service.

It is with considerable sadness that, during this SMB catastrophe, we
also have come to understand that the famous Samba server presented an
exploitable attack surface on the public internet in sufficient numbers for
a worm to propagate successfully. I previously have
discussed SMB security

in Linux Journal, and I am no longer of the opinion that SMB server processes should run on

In any case, systems administrators of all architectures must be able to
down vulnerable network servers and patch them quickly. There is often a
need for speed and competence when working with a large collection of Linux
servers. Whether this is due to security situations or other concerns is
immaterial—the hour of greatest need is not the time to begin to build
administration tools. Note that in the event of an active intrusion by
hostile parties, forensic
may be a legal requirement, and no steps
should be taken on the compromised server without a careful plan and
Especially in this new era of the black hats, computer
professionals must step up their game and be able to secure vulnerable
systems quickly.

Secure SSH Keypairs

Tight control of a heterogeneous UNIX environment must begin with
best-practice use of SSH authentication keys. I’m going to open this section with
a simple requirement. SSH private keys must be one of three types: Ed25519,
ECDSA using the E-521 curve or RSA keys of 3072 bits. Any key that does not
meet those requirements should be retired (in particular, DSA keys must be
removed from service immediately).

The Ed25519 key format
is associated with Daniel J. Bernstein, who has such
a preeminent reputation in modern cryptography that the field is becoming a
DJB monoculture.
The Ed25519 format is deigned for speed, security and size
economy. If all of your SSH servers are recent enough to support Ed25519,
then use it, and consider nothing else.

Guidance on
creating Ed25519 keys
suggests 100 rounds for a work factor in
the “-o” secure format.
Raising the number of rounds raises the strength of
the encrypted key against brute-force attacks (should a file copy of the
private key fall into hostile hands), at the cost of more work and time in
decrypting the key when ssh-add is executed. Although there always
is controversy and
with security advances,
I will repeat the
guidance here and suggest that the best format for a newly created SSH key
is this:

ssh-keygen -a 100 -t ed25519

Your systems might be too old to support Ed25519—Oracle/CentOS/Red Hat 7
have this problem (the 7.1 release introduced support). If you cannot
upgrade your old SSH clients and servers, your next best option is
likely E-521, available in the ECDSA key format.

The ECDSA curves came from the US government’s National Institute of
Standards (NIST). The best known and most implemented of all of the NIST
curves are P-256, P-384 and E-521. All three curves are approved for secret
communications by a variety of government entities, but a number of
cryptographers have expressed growing
that the P-256 and P-384
curves are tainted.
Well known
cryptographer Bruce Schneier has remarked:
no longer trust the constants. I believe the NSA has manipulated them
through their relationships with industry.” However, DJB has expressed
limited praise of the E-521 curve:
“To be fair I should mention that there’s
one standard NIST curve using a nice prime, namely
2521 – 1; but the sheer
size of this prime makes it much slower than NIST P-256.” All of the NIST
curves have greater issues with “side channel” attacks than
is certainly a step down, and many assert that none of the NIST curves are
safe. In summary, there is a slight risk that a powerful adversary exists
with an advantage over the P-256 and P-384 curves, so one is
slightly inclined to avoid them. Note that even if your OpenSSH (source)
release is capable of E-521, it may be disabled by your vendor due to patent
so E-521 is not an option in this case. If you cannot use DJB’s
2255 – 19 curve, this command will generate an E-521 key on a capable

ssh-keygen -o -a 100 -b 521 -t ecdsa

And, then there is the unfortunate circumstance with SSH servers that
support neither ECDSA nor Ed25519. In this case, you must fall back to RSA
with much larger key sizes. An absolute minimum is the modern default of
2048 bits, but 3072 is a wiser choice:

ssh-keygen -o -a 100 -b 3072 -t rsa

Then in the most lamentable case of all, when you must use old SSH clients
that are not able to work with private keys created with the
-o option,
you can remove the password on id_rsa and create a naked key, then use
OpenSSL to encrypt it with AES256 in the PKCS#8 format, as first documented
by Martin Kleppmann
Provide a blank new password for the keygen utility
below, then supply a new password when OpenSSL reprocesses the key:

$ cd ~/.ssh

$ cp id_rsa id_rsa-orig

$ ssh-keygen -p -t rsa
Enter file in which the key is (/home/cfisher/.ssh/id_rsa):
Enter old passphrase:
Key has comment 'cfisher@localhost.localdomain'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

$ openssl pkcs8 -topk8 -v2 aes256 -in id_rsa -out id_rsa-strong
Enter Encryption Password:
Verifying - Enter Encryption Password:

mv id_rsa-strong id_rsa
chmod 600 id_rsa

After creating all of these keys on a newer system, you can compare the file

$ ll .ssh
total 32
-rw-------. 1 cfisher cfisher  801 Aug 10 21:30 id_ecdsa
-rw-r--r--. 1 cfisher cfisher  283 Aug 10 21:30 id_ecdsa.pub
-rw-------. 1 cfisher cfisher  464 Aug 10 20:49 id_ed25519
-rw-r--r--. 1 cfisher cfisher  111 Aug 10 20:49 id_ed25519.pub
-rw-------. 1 cfisher cfisher 2638 Aug 10 21:45 id_rsa
-rw-------. 1 cfisher cfisher 2675 Aug 10 21:42 id_rsa-orig
-rw-r--r--. 1 cfisher cfisher  583 Aug 10 21:42 id_rsa.pub

Although they are relatively enormous, all versions of OpenSSH that I have used
have been compatible with the RSA private key in PKCS#8 format. The Ed25519
public key is now small enough to fit in 80 columns without word wrap, and
it is as convenient as it is efficient and secure.

Note that PuTTY may have problems using various versions of these keys, and
you may need to remove passwords for a successful import into the PuTTY

These keys represent the most secure formats available for various OpenSSH
revisions. They really aren’t intended for PuTTY or other general
interactive activity. Although one hopes that all users create strong keys for all
situations, these are enterprise-class keys for major systems activities. It
might be wise, however, to regenerate your system host keys to conform to
these guidelines.

These key formats may soon change. Quantum computers are causing increasing
concern for their ability to run Shor’s Algorithm,
which can be used to find
prime factors to break these keys in reasonable time. The largest
commercially available quantum computer, the D-Wave 2000Q,
under 200 qubits
for this activity,
which is not (yet) powerful
enough for a successful attack. NIST announced
a competition
for a new
quantum-resistant public key system with a deadline of November 2017
response, a team including DJB has released source code for NTRU Prime.
does appear that we will likely see a post-quantum public key format for
OpenSSH (and potentially TLS 1.3) released within the next two years, so
take steps to ease migration now.

Also, it’s important for SSH servers to restrict their allowed ciphers,
MACs and key exchange lest strong keys be wasted on broken crypto (3DES,
MD5 and arcfour should be long-disabled). My previous
on the
involved the following (three) lines in the SSH client and server
configuration (note that formatting in the sshd_config file requires all
parameters on the same line with no spaces in the options; line breaks have
been added here for clarity):

Ciphers chacha20-poly1305@openssh.com,

MACs    hmac-sha2-512-etm@openssh.com,

KexAlgorithms curve25519-sha256@libssh.org,

Since the previous publication, RIPEMD160 is likely no longer safe and
should be removed. Older systems, however, may support only SHA1, MD5 and
RIPEMD160. Certainly remove MD5, but users of PuTTY likely will want to
retain SHA1 when newer MACs are not an option. Older servers can present a
challenge in finding a reasonable Cipher/MAC/KEX when working with modern

At this point, you should have strong keys for secure clients and servers.
Now let’s put them to use.

Scripting the SSH Agent

Modern OpenSSH distributions contain the ssh-copy-id shell script for easy
key distribution. Below is an example of installing a specific, named key in
a remote account:

$ ssh-copy-id -i ~/.ssh/some_key.pub person@yourserver.com
ssh-copy-id: INFO: Source of key(s) to be installed:
ssh-copy-id: INFO: attempting to log in with the new key(s),
   to filter out any that are already installed
ssh-copy-id: INFO: 1 key(s) remain to be installed --
   if you are prompted now it is to install the new keys
person@yourserver.com's password:

Number of key(s) added: 1

Now try logging into the machine, with:
   "ssh 'person@yourserver.com'"
and check to make sure that only the key(s) you wanted were added.

If you don’t have the ssh-copy-id script, you can install a key
manually with the following command:

$ ssh person@yourserver.com 'cat >> ~/.ssh/authorized_keys' 

If you have SELinux enabled, you might have to mark a newly created
authorized_keys file with a security type; otherwise, the sshd server
will be prevented from reading the key (the syslog may report this issue):

$ ssh person@yourserver.com 'chcon -t ssh_home_t

Once your key is installed, test it in a one-time use with the
-i option
(note that you are entering a local key password, not a remote
authentication password):

$ ssh -i ~/.ssh/some_key person@yourserver.com
Enter passphrase for key '/home/v-fishecj/.ssh/some_key':
Last login: Wed Aug 16 12:20:26 2017 from
yourserver $

General, interactive users likely will cache their keys with an agent. In
the example below, the same password is used on all three types of keys that
were created in the previous section:

$ eval $(ssh-agent)
Agent pid 4394

$ ssh-add
Enter passphrase for /home/cfisher/.ssh/id_rsa:
Identity added: ~cfisher/.ssh/id_rsa (~cfisher/.ssh/id_rsa)
Identity added: ~cfisher/.ssh/id_ecdsa (cfisher@init.com)
Identity added: ~cfisher/.ssh/id_ed25519 (cfisher@init.com)

The first command above launches a user agent process, which injects
environment variables (named SSH_AGENT_SOCK and
SSH_AGENT_PID) into the
parent shell (via eval). The shell becomes aware of the agent and passes
these variables to the programs that it runs from that point forward.

When launched, the ssh-agent has no credentials and is unable to facilitate
SSH activity. It must be primed by adding keys, which is done with
When called with no arguments, all of the default keys will be read. It
also can be called to add a custom key:

$ ssh-add  ~/.ssh/some_key
Enter passphrase for /home/cfisher/.ssh/some_key:
Identity added: /home/cfisher/.ssh/some_key

Note that the agent will not retain the password on the key.
ssh-add uses
any and all passwords that you enter while it runs to decrypt keys that it
finds, but the passwords are cleared from memory when
ssh-add terminates
(they are not sent to ssh-agent). This allows you to upgrade to new key
formats with minimal inconvenience, while keeping the keys reasonably safe.

The current cached keys can be listed with ssh-add
(from, which you can deduce
that “some_key” is an Ed25519):

$ ssh-add -l
3072 SHA256:cpVFMZ17oO5n/Jfpv2qDNSNcV6ffOVYPV8vVaSm3DDo
     /home/cfisher/.ssh/id_rsa (RSA)
521 SHA256:1L9/CglR7cstr54a600zDrBbcxMj/a3RtcsdjuU61VU
     cfisher@localhost.localdomain (ECDSA)
256 SHA256:Vd21LEM4lixY4rIg3/Ht/w8aoMT+tRzFUR0R32SZIJc
     cfisher@localhost.localdomain (ED25519)
256 SHA256:YsKtUA9Mglas7kqC4RmzO6jd2jxVNCc1OE+usR4bkcc
     cfisher@localhost.localdomain (ED25519)

While a “primed” agent is running, the SSH clients may use (trusting) remote
servers fluidly, with no further prompts for credentials:

$ sftp person@yourserver.com
Connected to yourserver.com.
sftp> quit

$ scp /etc/passwd person@yourserver.com:/tmp
passwd                              100% 2269    65.8KB/s   00:00

$ ssh person@yourserver.com
   (motd for yourserver.com)
$ ls -l /tmp/passwd
-rw-r--r--  1 root  wheel  2269 Aug 16 09:07 /tmp/passwd
$ rm /tmp/passwd
$ exit
Connection to yourserver.com closed.

The OpenSSH agent can be locked, preventing any further use of the
credentials that it holds (this might be appropriate when suspending a

$ ssh-add -x
Enter lock password:
Agent locked.

$ ssh yourserver.com
Enter passphrase for key '/home/cfisher/.ssh/id_rsa': ^C

It will provide credentials again when it is unlocked:

$ ssh-add -X
Enter lock password:
Agent unlocked.

You also can set ssh-agent to expire keys after a
time limit with the -t
option, which may be useful for long-lived agents that must clear keys after
a set daily shift.

General shell users may cache many types of keys with a number of differing
agent implementations. In addition to the standard OpenSSH agent, users may
rely upon PuTTY’s pageant.exe, GNOME keyring or KDE Kwallet, among others
(the use of the PUTTY agent could likely fill an article on its own).

However, the goal here is to create “enterprise” keys for critical server
controls. You likely do not want long-lived agents in order to limit the risk
of exposure. When scripting with “enterprise” keys, you will run an
agent only for the duration of the activity, then kill it at completion.

There are special options for accessing the root account with
PermitRootLogin parameter can be added to the sshd_config file (usually
found in /etc/ssh). It can be set to a simple yes or
forced-commands-only, which will allow only explicitly-authorized programs to
be executed, or the equivalent options
prohibit-password or
both of which will allow access to the keys generated here.

Many hold that root should not be allowed any access. Michael W. Lucas
addresses the question in SSH Mastery:

Sometimes, it seems that you need to allow users to SSH in to the system as
root. This is a colossally bad idea in almost all environments. When users
must log in as a regular user and then change to root, the system logs
record the user account, providing accountability. Logging in as root
destroys that audit trail….It is possible to override the security
precautions and make sshd permit a login directly as root. It’s such a bad
idea that I’d consider myself guilty of malpractice if I told you how to do
it. Logging in as root via SSH almost always means you’re solving the wrong
problem. Step back and look for other ways to accomplish your goal.

When root action is required quickly on more than a few servers, the above
advice can impose painful delays. Lucas’ direct criticism can be addressed
by allowing only a limited set of “bastion” servers to issue root commands
over SSH. Administrators should be forced to log in to the bastions with
unprivileged accounts to establish accountability.

However, one problem with remotely “changing to root” is the statistical use
of the Viterbi algorithm

Short passwords, the su - command and remote
SSH calls that use passwords to establish a trinary network configuration
are all uniquely vulnerable to timing attacks on a user’s keyboard
Those with the highest security concerns will need to compensate.

For the rest of us, I recommend that PermitRootLogin
be set
for all target machines.

Finally, you can easily terminate ssh-agent
interactively with the -k option:

$ eval $(ssh-agent -k)
Agent pid 4394 killed

With these tools and the intended use of them in mind, here is a
complete script that runs an agent for the duration of a set of commands
over a list of servers for a common named user (which is not necessarily

# cat artano


if [[ $# -lt 1 ]]; then echo "$0 - requires commands"; exit; fi

R="-R5865:" # set to "-2" if you don't want
 ↪port forwarding

eval $(ssh-agent -s)

function cleanup { eval $(ssh-agent -s -k); }

trap cleanup EXIT

function remsh { typeset F="/tmp/${1}" h="$1" p="$2";
 ↪shift 2; echo "#$h"
 if [[ "$ARTANO" == "PARALLEL" ]]
 then ssh "$R" -p "$p" "$h" "$@" >"${F}.out"
  ↪2>>"${F}.err" &
 else ssh "$R" -p "$p" "$h" "$@"
 fi }    # HOST                                          PORT CMD

if ssh-add ~/.ssh/master_key
then remsh yourserver.com                                  22 "$@"
     remsh container.yourserver.com                      2200 "$@"
     remsh anotherserver.com                               22 "$@"
     # Add more hosts here.
else echo Bad password - killing agent. Try again.


# Examples:           # Artano is an epithet of a famous mythical being
# artano 'mount /patchdir'      # you will need an fstab entry for this
# artano 'umount /patchdir'
# artano 'yum update -y 2>&1'
# artano 'rpm -Fvh /patchdir/\*.rpm'

This script runs all commands in sequence on a collection of hosts by
default. If the ARTANO environment variable is set
instead will launch them all as background processes simultaneously and append
their STDOUT and STDERR to files in /tmp (this should be no problem when
dealing with fewer than a hundred hosts on a reasonable server). The
PARALLEL setting is useful not only for pushing changes faster, but also for
collecting audit results.

Below is an example using the yum update agent. The source of this
particular invocation had to traverse a firewall and relied on a proxy
setting in the /etc/yum.conf file, which used the port-forwarding option

# ./artano 'yum update -y 2>&1'
Agent pid 3458
Enter passphrase for /root/.ssh/master_key:
Identity added: /root/.ssh/master_key (/root/.ssh/master_key)
Loaded plugins: langpacks, ulninfo
No packages marked for update
Loaded plugins: langpacks, ulninfo
No packages marked for update
Loaded plugins: langpacks, ulninfo
No packages marked for update
Agent pid 3458 killed

The script can be used for more general maintenance functions. Linux
installations running the XFS filesystem should “defrag”
Although this normally would be done with cron, it can be a centralized
activity, stored in a separate script that includes only on the appropriate

Agent pid 7897
Enter passphrase for /root/.ssh/master_key:
Identity added: /root/.ssh/master_key (/root/.ssh/master_key)
Agent pid 7897 killed

An easy method to collect the contents of all authorized_keys files for all
users is the following artano script (this is useful for system auditing and is coded to remove file duplicates):

artano 'awk -F: {print\$6\"/.ssh/authorized_keys\"} \
     /etc/passwd | sort -u | xargs grep . 2> /dev/null'

It is convenient to configure NFS mounts for file distribution to remote
nodes. Bear in mind that NFS is clear text, and sensitive content
should not traverse untrusted networks while unencrypted. After configuring
an NFS server on host, I add the following line to the /etc/fstab
file on all the clients and create the /patchdir directory. After the
change, the artano script can be used to mass-mount the directory if the
network configuration is correct:

# tail -1 /etc/fstab
 ↪/patchdir nfs4 noauto,proto=tcp,port=2049 0 0

Assuming that the NFS server is mounted, RPMs can be upgraded from images
stored upon it (note that Oracle Spacewalk or Red Hat Satellite might be a
more capable patch method):

# ./artano 'rpm -Fvh /patchdir/\*.rpm'
Agent pid 3203
Enter passphrase for /root/.ssh/master_key:
Identity added: /root/.ssh/master_key (/root/.ssh/master_key)
Preparing...                          ########################
Updating / installing...
xmlsec1-1.2.20-7.el7_4                ########################
xmlsec1-openssl-1.2.20-7.el7_4        ########################
Cleaning up / removing...
xmlsec1-openssl-1.2.20-5.el7          ########################
xmlsec1-1.2.20-5.el7                  ########################
Preparing...                          ########################
Updating / installing...
xmlsec1-1.2.20-7.el7_4                ########################
xmlsec1-openssl-1.2.20-7.el7_4        ########################
Cleaning up / removing...
xmlsec1-openssl-1.2.20-5.el7          ########################
xmlsec1-1.2.20-5.el7                  ########################
Preparing...                          ########################
Updating / installing...
xmlsec1-1.2.20-7.el7_4                ########################
xmlsec1-openssl-1.2.20-7.el7_4        ########################
Cleaning up / removing...
xmlsec1-openssl-1.2.20-5.el7          ########################
xmlsec1-1.2.20-5.el7                  ########################
Agent pid 3203 killed

I am assuming that my audience is already experienced with package tools for
their preferred platforms. However, to avoid criticism that I’ve included
little actual discussion of patch tools, the following is a quick reference of RPM
manipulation commands, which is the most common package format on enterprise

  • rpm -Uvh package.i686.rpm — install or upgrade a package file.

  • rpm -Fvh package.i686.rpm — upgrade a package file, if an older version is installed.

  • rpm -e package — remove an
    installed package.

  • rpm -q package — list installed package name and version.

  • rpm -q --changelog package
    — print full changelog for installed package
    (including CVEs).

  • rpm -qa — list all installed packages on the

  • rpm -ql package — list all
    files in an installed package.

  • rpm -qpl package.i686.rpm — list files
    included in a package file.

  • rpm -qi package — print
    detailed description of installed package.

  • rpm -qpi package — print
    detailed description of package file.

  • rpm -qf /path/to/file — list package
    that installed a particular file.

  • rpm --rebuild package.src.rpm
    — unpack and build a binary RPM under

  • rpm2cpio package.src.rpm | cpio
    — unpack all package files in the
    current directory.

Another important consideration for scripting the SSH agent is limiting the
capability of an authorized key. There is a specific
for such
particular interest is the from="" clause, which will
restrict logins on a key to a limited set of hosts. It is likely wise to
declare a set of “bastion” servers that will record non-root logins that
escalate into controlled users who make use of the enterprise keys.

An example entry might be the following (note that I’ve broken this line, which is not
allowed syntax but done here for clarity):

from="*.c2.security.yourcompany.com," ssh-ed25519

A number of other useful restraints can be placed upon
authorized_keys entries. The
command="" will restrict a key to a single
program or script and will set the
variable to the client’s attempted call—scripts can set alarms if the
variable does not contain approved contents. The
restrict option also
is worth consideration, as it disables a large set of SSH features that can be
both superfluous and dangerous.

Although it is possible to set server identification keys in the known_hosts
file to a @revoked status, this cannot be done with the contents of
authorized_keys. However, a system-wide file for forbidden keys can be set
in the sshd_config with RevokedKeys. This file overrides any user’s
authorized_keys. If set, this file must exist and be readable by the sshd
server process; otherwise, no keys will be accepted at all (so use care if
you configure it on a machine where there are obstacles to physical access).
When this option is set, use the artano script to append forbidden
keys to the file quickly when they should be disallowed from the network. A clear
and convenient file location would be /etc/ssh/revoked_keys.

It is also possible to establish a local Certificate Authority (CA) for OpenSSH
that will allow keys to be registered
with an authority
with expiration
These CAs can become
quite elaborate
in their control over an
Although the maintenance of an SSH CA is beyond the scope of this
article, keys issued by such CAs should be strong by adhering to the
requirements for Ed25519/E-521/RSA-3072.


Many higher-level tools for the control of collections of servers exist
that are much more sophisticated than the script I’ve presented here.
The most famous is likely Puppet,
which is a Ruby-based configuration
management system for enterprise control. Puppet has a somewhat short list
of supported operating systems. If you are looking for low-level control of
Android, Tomato, Linux smart terminals or other “exotic” POSIX, Puppet
is likely not the appropriate tool. Another popular Ruby-based tool is Chef,
which is known for its complexity. Both Puppet and Chef require Ruby
installations on both clients and servers, and they both will catalog any
SSH keys that they find, so this key strength discussion is completely
applicable to them.

There are several similar Python-based tools, including Ansible,
and SaltStack.
Of these, only Ansible can run
“agentless” over a
bare SSH connection; the rest will require agents that run on target nodes
(and this likely includes a Python runtime).

Another popular configuration management tool is CFEngine,
which is coded in
C and claims very high performance. Rudder has evolved from portions of
CFEngine and has a small but growing user community.

Most of the previously mentioned packages are licensed commercially and
some are closed source.

The closest low-level tool to the activities presented here is the Parallel
Distributed Shell (pdsh), which can be found in the EPEL repository.
pdsh utilities grew out of an IBM-developed package named dsh designed for
the control of compute clusters. Install the following packages from the
repository to use pdsh:

# rpm -qa | grep pdsh

An SSH agent must be running while using pdsh with encrypted keys, and there
is no obvious way to control the destination port on a per-host basis as was
done with the artano script. Below is an example using pdsh to run a command
on three remote servers:

# eval $(ssh-agent)
Agent pid 17106

# ssh-add  ~/.ssh/master_key
Enter passphrase for /root/.ssh/master_key:
Identity added: /root/.ssh/master_key (/root/.ssh/master_key)

# pdsh -w hosta.com,hostb.com,hostc.com uptime
hosta: 13:24:49 up 13 days,  2:13, 6 users, load avg: 0.00, 0.01, 0.05
hostb: 13:24:49 up  7 days, 21:15, 5 users, load avg: 0.05, 0.04, 0.05
hostc: 13:24:49 up  9 days,  3:26, 3 users, load avg: 0.00, 0.01, 0.05

# eval $(ssh-agent -k)
Agent pid 17106 killed

The -w option above defines a host list. It allows for limited arithmetic
expansion and can take the list of hosts from standard input if the
argument is a dash (-). The PDSH_SSH_ARGS and
environment variables can be used to pass custom options to the SSH call. By
default, 32 sessions will be launched in parallel, and this “fanout/sliding
window” will be maintained by launching new host invocations as existing
connections complete and close. You can adjust the size of the “fanout”
either with the -f option or the
FANOUT environment variable. It’s interesting to
note that
there are two file copy commands: pdcp and
rpdcp, which are analogous to

Even a low-level utility like pdsh lacks some flexibility that is available
by scripting OpenSSH, so prepare to feel even greater constraints as more
complicated tools are introduced.


Modern Linux touches us in many ways on diverse platforms. When the security
of these systems is not maintained, others also may touch our platforms
and turn them against us. It is important to realize the maintenance
obligations when you add any Linux platform to your environment. This
obligation always exists, and there are consequences when it is not met.

In a security emergency, simple, open and well understood tools are best.
As tool complexity increases, platform portability certainly declines, the
number of competent administrators also falls, and this likely impacts speed
of execution. This may be a reasonable trade in many other aspects, but in a
security context, it demands a much more careful analysis. Emergency measures
must be documented and understood by a wider audience than is required for
normal operations, and using more general tools facilitates that discussion.

I hope the techniques presented here will prompt that discussion for
those who have not yet faced it.


The views and opinions expressed in this article are those of
the author and do not necessarily reflect those of Linux


An exploit compromising
was recently demonstrated that relies upon
custom hardware changes to derive a usable portion of a secret key.
Physical hardware security is a basic requirement for encryption integrity,
and many common algorithms are further vulnerable to cache timing or other
side channel attacks that can be performed by the unprivileged processes of
other users. Use caution when granting access to systems that process
sensitive data.


Creating an Adventure Game in the Terminal with ncurses

Creating an Adventure Game in the Terminal with ncurses


Jim Hall
Fri, 01/26/2018 – 10:26

How to use curses functions to read the keyboard and manipulate the

My previous
introduced the ncurses library and provided a simple
program that demonstrated a few curses functions to put text on the screen. In
this follow-up article, I illustrate how to use a few other

An Adventure

When I was growing up, my family had an Apple II computer. It was on this
machine that my brother and I taught ourselves how to write programs in
AppleSoft BASIC. After writing a few math puzzles, I moved on to creating
games. Having grown up in the 1980s, I already was a fan of the Dungeons and
tabletop games, where you role-played as a fighter or wizard on some
quest to defeat monsters and plunder loot in strange lands. So it
shouldn’t be surprising that I also created a rudimentary adventure game.

The AppleSoft BASIC programming environment supported a neat feature: in
standard resolution graphics mode (GR mode), you could probe the color of a
particular pixel on the screen. This allowed a shortcut to create an adventure
game. Rather than create and update an in-memory map that was
transferred to the screen periodically, I could rely on GR mode to maintain the map for me,
and my program could query the screen as the player’s character moved
around the screen. Using this method, I let the computer do most of the hard
work. Thus, my top-down adventure game used blocky GR mode graphics to
represent my game map.

My adventure game used a simple map that represented a large field with a
mountain range running down the middle and a large lake on the upper-left side. I
might crudely draw this map for a tabletop gaming campaign to include a narrow
path through the mountains, allowing the player to pass to the far side.

Figure 1. A simple Tabletop Game Map with a Lake and Mountains

You can draw this map in cursesusing characters to represent
grass, mountains and water. Next, I describe how to do just that using
functions and how to create and play a similar adventure game in the Linux terminal.

Constructing the Program

In my last article, I mentioned that most curses programs start with the same
set of instructions to determine the terminal type and set up the


For this program, I add another statement:

keypad(stdscr, TRUE);

The TRUE flag allows curses to read the keypad and function keys from the
user’s terminal. If you want to use the up, down, left and right arrow
keys in your program, you need to use keypad(stdscr, TRUE) here.

Having done that, you now can start drawing to the terminal screen. The
functions include several ways to draw text on the screen. In my previous
article, I demonstrated the addch() and addstr() functions and their
associated mvaddch() and mvaddstr() counterparts that first moved to a
specific location on the screen before adding text. To create the adventure
game map on the terminal, you can use another set of functions:
vline() and
hline(), and their partner functions mvvline() and
mvhline(). These mv
functions accept screen coordinates, a character to draw and how many times
to repeat that character. For example, mvhline(1, 2, '-', 20) will draw a line
of 20 dashes starting at line 1, column 2.

To draw the map to the terminal screen programmatically, let’s define this
draw_map() function:

#define GRASS     ' '
#define EMPTY     '.'
#define WATER     '~'
#define MOUNTAIN  '^'
#define PLAYER    '*'

void draw_map(void)
    int y, x;

    /* draw the quest map */

    /* background */

    for (y = 0; y 

In drawing this map, note the use of mvvline() and
mvhline() to fill large
chunks of characters on the screen. I created the fields of grass by drawing
horizontal lines (mvhline) of characters starting at column 0, for the entire
height and width of the screen. I added the mountains on top of that by
drawing vertical lines (mvvline), starting at row 0, and a mountain path by
drawing a single horizontal line (mvhline). And, I created the lake by drawing
a series of short horizontal lines (mvhline). It may seem inefficient to draw
overlapping rectangles in this way, but remember that curses
actually update the screen until I call the refresh() function later.

Having drawn the map, all that remains to create the game is to enter a loop
where the program waits for the user to press one of the up, down, left or
right direction keys and then moves a player icon appropriately. If the space the
player wants to move into is unoccupied, it allows the player to go there.

You can use curses as a shortcut. Rather than having to instantiate a version
of the map in the program and replicate this map to the screen, you can let
the screen keep track of everything for you. The inch() function, and
associated mvinch() function, allow you to probe the contents of the screen.
This allows you to query curses to find out whether the space the player wants to
move into is already filled with water or blocked by mountains. To do this,
you’ll need a helper function that you’ll use later:

int is_move_okay(int y, int x)
    int testch;

    /* return true if the space is okay to move into */

    testch = mvinch(y, x);
    return ((testch == GRASS) || (testch == EMPTY));

As you can see, this function probes the location at column y, row x and
returns true if the space is suitably unoccupied, or false if not.

That makes it really easy to write a navigation loop: get a key from the
keyboard and move the user’s character around depending on the up, down,
left and right arrow keys. Here’s a simplified version of that loop:

    do {
        ch = getch();

        /* test inputted key and determine direction */

        switch (ch) {
        case KEY_UP:
            if ((y > 0) && is_move_okay(y - 1, x)) {
                y = y - 1;
        case KEY_DOWN:
            if ((y  0) && is_move_okay(y, x - 1)) {
                x = x - 1;
        case KEY_RIGHT
            if ((x 

To use this in a game, you’ll need to add some code inside the loop to
allow other keys (for example, the traditional WASD movement keys), provide a
method for the user to quit the game and move the player’s character
around the screen. Here’s the program in full:

/* quest.c */


#define GRASS  	  ' '
#define EMPTY     '.'
#define WATER	  '~'
#define MOUNTAIN  '^'
#define PLAYER	  '*'

int is_move_okay(int y, int x);
void draw_map(void);

int main(void)
    int y, x;
    int ch;

    /* initialize curses */

    keypad(stdscr, TRUE);


    /* initialize the quest map */


    /* start player at lower-left */

    y = LINES - 1;
    x = 0;

    do {
	/* by default, you get a blinking cursor - use it to indicate player */

	mvaddch(y, x, PLAYER);
	move(y, x);

	ch = getch();

	/* test inputted key and determine direction */

	switch (ch) {
	case KEY_UP:
	case 'w':
	case 'W':
	    if ((y > 0) && is_move_okay(y - 1, x)) {
		mvaddch(y, x, EMPTY);
		y = y - 1;
	case KEY_DOWN:
	case 's':
	case 'S':
	    if ((y  0) && is_move_okay(y, x - 1)) {
		mvaddch(y, x, EMPTY);
		x = x - 1;
	case KEY_RIGHT:
	case 'd':
	case 'D':
	    if ((x 

In the full program listing, you can see the complete arrangement of
functions to create the game:

1) Initialize the curses environment.

2) Draw the map.

3) Initialize the player coordinates (lower-left).

4) Loop:

  • Draw the player’s character.
  • Get a key from the keyboard.
  • Adjust the player’s coordinates up, down, left or right,
  • Repeat.

5) When done, close the curses environment and exit.

Let’s Play

When you run the game, the player’s character starts in the lower-left
corner. As the player moves around the play area, the program creates a
“trail” of dots. This helps show where the player has been before, so
the player can avoid crossing the path unnecessarily.

Figure 2. The player starts the game in the lower-left corner.

Figure 3. The player can move around the play area, such as around the lake and
through the mountain pass.

To create a complete adventure game on top of this, you might add random
encounters with various monsters as the player navigates his or her character
around the play area. You also could include special items the player could
discover or loot after defeating enemies, which would enhance the
player’s abilities further.

But to start, this is a good program for demonstrating how to use the
curses functions to read the keyboard and manipulate the screen.

Next Steps

This program is a simple example of how to use the curses functions to update
and read the screen and keyboard. You can do so much more with
depending on what you need your program to do. In a follow up article, I plan
show how to update this sample program to use colors. In the meantime, if you
are interested in learning more about curses, I encourage you to read Pradeep
Padala’s NCURSES
Programming HOWTO
at the Linux Documentation Project.


Creating an Application-Based Terminal Session

One of my first exposures to computers in a work environment was using a Wyse
terminal to access a console-based application for data entry. It wasn’t until a
while later that I learned about terminals in UNIX and how they work. Once I
understood it, I wanted to create my own self-contained application that was tied
to a telnet or SSH session.


Watermarking Images–from the Command Line

Us geeks mostly think of the command line as the best place for text
manipulation. It’s a natural with cat,
grep and shell scripts. But
although you can’t necessarily view your results from within a typical
terminal window, it turns out to be pretty darn easy to analyze and
manipulate images from within a shell script.