practicing techie

tech oriented notes to self and lessons learned

Category Archives: devops

Automating TLS certificate issuance with OpenSSL

Transport layer security (TLS) has long been the way to provide privacy, integrity and optionally authentication to protect TCP/IP traffic on the internet. TLS relies on public key infrastructure and trusted third-parties (Certificate Authority) to vouch for communicating parties correctness, which can complicate its use for the impatient software engineer.

Acquiring a X.509 server certificate from an established Certificate Authority (CA) is usually somewhat of a hassle. Certificate Authorities vouch for the legitimacy of the server certificate and the vouched party’s right to use it. In order to gain a level of trust in the vouched party Certificate Authorities use different kinds of authentication methods, which from a developer perspective add to the delivery delay, cost and effort.

As a result, for non-production or internal uses self-signed certificates are often used instead third-party signed certificates. Self-signed certificates can sometimes be good for ad-hoc use, but initially no shared chain of trust exists between the server and its clients. This means that for each client you either have to disable certificate validation or make the certificate trusted.

If you find yourself generating lots of self-signed certificates, a better solution may be to use a CA with fast turnaround time. These are usually also ones with an automated authentication process, and hence lower cost. Examples include commercial operators such as, as well as non-profits like Let’s Encrypt. Let’s Encrypt also allows certificate requests to be fulfilled using an automated process. It’s a security vs. fast turnaround / ease-of-use tradeoff: faster turnaround usually translates to less scrutiny in the authentication process.

Customer/partner-facing systems often warrant using a certificate from an established CA, but sometimes, particularly for internal purposes, when fast turnaround time is needed, as well as control over the certificate issuance process in order to automate it, the best solution may be to run your own CA. Running your own certificate authority is not difficult, technically. Keeping the system secure, on the other hand, will be hard work.

The big risk is that if the private key associated with your CA certificate is compromised, a malicious party can create and sign new certificates with your CA certificate. Also, if a private key associated with a certificate is compromised, a malicious party can intercept communication to/from the certificate owner or impersonate the owner. When a software-based crypto implementation is used, your best bet of securing your keys is applying operating system hardening practices. The full data lifecycle (including backups!) should be considered when planning to secure your keys. For an added level of security, specialised crypto hardware are able to store private keys in a secure manner.

OpenSSL based Certificate Authority

The OpenSSL project provides a basic tool for running your own CA, in the form of the ‘openssl ca’ command. OpenSSL also includes the wrapper script for making certificate management easier to use.

While Linux distributions and macOS ship with OpenSSL, it’s only newer versions of both OpenSSL and that support generating certificates using just command line tools and with no user interaction during the process. The simplest way to update the script is to download it from the project’s GitHub repository:

Make sure you get commit 022696cab014ffb94c8ef0bfc79c8955b9970eb6 or newer (not part of a released version at the time of this writing). Older version of openssl command should suffice. (It seems that ‘openssl ca’ has supported the required passin parameter for a while, but this was only documented in commit 16e1b281b2e16ff6deb8ca431dfc5743de31d0e2.)

With you can automate certificate generation with a simple shell script like the following:

pass=`openssl rand -hex 18`
pushd `dirname $CA_ROOT`

echo "issuing certificate for $cn"

# create certificate request
SUBJECT="/C=FI/L=Helsinki/O=Practicing techie/CN=$cn/"
OPENSSL=$OPENSSL $CAPL -newreq -extra-req "-passout pass:$pass -subj '$SUBJECT'"

# sign certificate request
OPENSSL=$OPENSSL $CAPL -sign -extra-ca "-passin file:$CA_ROOT/ca-cert-passphrase.txt -batch"
if [ "$?" -ne 0 ]; then
  echo "FATAL: failed to sign, aborting"
  exit 1

# export private key unencrypted, archive files
$OPENSSL rsa -in newkey.pem -out newkey-nodes.pem -passin pass:$pass
mkdir -p $CERT_BASE/$cn
mv new*.pem $CERT_BASE/$cn

See my Gist for a more details.


When you need to secure TCP/IP traffic in a non-production environment or non-customer facing system, running your own CA can be a very viable solution. Using certificates signed by a third-party can prove burdensome and costly when your architecture is based on microservices and TLS is being used to secure the traffic. In this case running your own CA to create certificates can help. Other uses include securing traffic between system management and monitoring software and remote agents (e.g. Nagios, JMX etc.).

OpenSSL tools can make automating certificate creation and signing quite easy. Keeping the CA keys secure will require considerable effort, so you need to carefully weigh the risks.


Extending a LVM based file system

A file system running out of space doesn’t always have to be a major catastrophe. The problem could be quite easily resolved, if you happen to be running Linux and were foresighted enough to create your file system on a LVM volume. If you also happen to have extra storage space available, the file system could likely be extended in a fairly straightforward manner. Despite the procedure’s relative simplicity, it’s quite easy to forget the details if you perform it very infrequently. So, this time I’m writing this down.

In order to extend a LVM based file system, a new block device needs to be allocated for LVM. This can be done by adding a new physical disk, partitioning an existing disk or adding a virtual disk. Typically, reboot is required to get Linux to detect a new physical or virtual disk.

Once the new block device is detected, you need to identify its device path. This information can be obtained e.g. by using lsblk or sfdisk commands. If you’ve ran out of disk space you know which file system is causing the problems, but you also need to determine the LVM logical volume and volume group underlying that file system. These can be determined using output from df and lvdisplay (pvscan, vgscan and lvscan may also sometimes be helpful). In the snippet below we assume the block device path to be /dev/sdb, volume group ubuntu-vg and logical volume /dev/ubuntu-vg/root.

After you learning these parameters the LVM based file system can be extended as follows:

Recently, I bumped into a tool called Asciinema, that allows recording and playback of terminal sessions. This differs from videos in that the terminal session’s input and output is recorded in a textual form, allowing you to e.g. copy-paste commands during playback. In order to try out the tool, I made a recording of extending a file system.