This is probably not the site you are looking for

Service discovery and authentication

Mon, 23 Nov 2015 00:00:00 +0000

ZooKeeper is a popular choice for service discovery in distributed systems. It is a hierarchical key-value store with very strong consistency guarantees. What makes it a good choice for service discovery is its capability to create temporary tree nodes that disappear when the session that created it is disconnected. Another important feature is sequential nodes - nodes that have a consistent counter appended to their name. You can use these for implementing distributed locking for example. This post is about the authentication problems that appear as a result of introducting service discovery into a distributed system.

Service discovery

The exact details of implementing a service discovery protocol using ZooKeeper is outside the scope of this document. I want to describe it in enough details so that we can understand its security properties. Each service that’s running and ready to respond to requests opens a persistent connection to Zookeeper. What usually happens is there is a root node, say /services, and it creates a temporary, sequential node:

/services/foo-service-1 = https://somehost.inthecloud.internal:6666 /services/foo-service-3 = https://otherhost.inthecloud.internal:12345

If you want to talk to this foo-service, you look in this root node and retrieve all the registered foo-service hostnames. You pick one randomly, or using round-robin, or what have you, and talk to that. If it fails, you try another. If you shut down foo-service, or it crashes, the temporary node is removed and other services will no longer try to talk to it.

Authenticating hostnames

In the example above, we are using HTTPS URLs, so you probably want some kind of internal CA to sign the certificates. (You might want to use some different protocol, Kerberos maybe.) You connect to the host and you can be sure it’s really that host, no one is eavesdropping, all the good things.

There are two problems with this. First, you are using Zookeeper because you don’t want to care about what host is running which service. You certainly don’t want to sign a new certificate for each one, deploy the private key to that host, etc. You can do this. It is possible. You can even automate it if you don’t mind having an online CA, that is not only easier to abuse, but also limits your availability.

The second, and even more important problem is authenticating the wrong thing. What if ZooKeeper lies to you? What if a compromised service publishes a hostname for some other service? You connect to that, verify that its certificate is correct, and proceed talking to the wrong service without noticing anything is wrong. Now, ZooKeeper is capable of controlling who can write what node, so you can create a structure like this:

/services/foo-service/foo-service-1 = https://a.inthecloud.internal:6666 /services/foo-service/foo-service-3 = https://b.inthecloud.internal:12345 /services/bar-service/bar-service-0 = https://c.inthecloud.internal:6666 /services/baz-service/baz-service-2 = https://d.inthecloud.internal:4444

You have to set up the permissions so that no one can write /services, the user running foo-service can only write /services/foo-service, etc. This still doesn’t help if Zookeeper itself is compromised.

This way we must trust Zookeeper and the CA.

Authenticating service names

There is a better way. We don’t care about the hostname, the same way we don’t care about the IP address. We don’t sign certificates for IP addresses. We can sign a certificate for the service name, as in, CN=foo-service. Actually, we don’t even have to sign it - the (self-signed) certificate can be included with the code of its clients.

The main disadvantage of this approach is having to write custom certificate validation code, which I have done in Java. In Java it’s actually pretty simple, the interfaces of interest are X509KeyManager and X509TrustManager. Some of the methods relating to CAs will be simple stubs. There are “extended” versions of both, with more methods you don’t need. I will publish the code, but I did this at work, and have not had time to reimplement it in the public domain.

This way we don’t have to trust Zookeeper, and we don’t have to trust the CA either (we don’t have one). The only thing we do have to trust is that the certificates included in the code are correct, but if someone can change that, they can change the code too.

There is still the matter of deploying the private keys to the hosts, but now it can be the same one for each instance of the same service. This could be on a NAS, or scp’d to the local machine before starting the service that needs it. Obviously we also have to trust that the ACLs will protect the private key but this is no worse than the previous method.

Authenticating the client

We must not forget to authenticate the client certificate as well. This is usually not done for browsers, but for inter-service traffic it is essential. You might want to use a different CN if the same service is sometimes the server and sometimes the client. Or you can use the same certificate but I don’t recommend it. If they are different you can use the CA system for client authentication and use the embedded cert system for server authentication. Either way, this means you have to embedd the certificate of the client in the server code.

There are all these certificates embedded everywhere. It doesn’t sound too maintainable, but what’s really going on here is each service consists of 3 files:

A server JAR that runs the server and doesn’t contain any certs.
A client JAR that’s a library imported by the clients, and contains the cert for the service.
A certificate, to be used for client authentication. (Or not, you could use a CA here.)

It’s not that bad.

Why cyber deterrence is bullshit

Fri, 07 Aug 2015 00:00:00 +0000

You may have heard of this brilliant idea. That’s right, “hack back” as foreign policy. It does not and will never work.

The reasons why this is utter bullshit should be all too obvious, but my Twitter account is locked and I have nothing better to do, so here goes:

The word cyber. Whenever you hear it, question the speaker’s competence.
Easy access. To build nuclear weapons, you need: relatively rare fissionable materials, huge enrichment facilities, knowledge, and ICBMs. To hack, you need: a computer, electricity, knowledge, and Internet access. You may be able to deter a few rational entities capable of building nukes, but not every single child capable of hacking OPM. There is nothing even close to perfect rationality on the Internet.
Attribution. Radars and satellites can clearly show where the missiles are coming from. No such device exists for detecting the origin of a breach. Attribution is purely guesswork, based on things like the language of strings found in the malware, source IPs, etc. It’s like sending a postcard, then trying to figure out who read it. Does that sound like perfect detection?
Working defense. Missile defense systems are actually more dangerous that the nukes they are trying to defend against. This is because they have a very interesting property: They reduce the second strike capability of one side, making counterforce first strike the logical choice for the other. The reason we are still all here, reading this post, is because covering large areas with such defenses is not feasible. This is not the case for defending computers against intrusion. Yes, attackers have a clear advantage over defenders, that is to say, hacking will always be possible. But it is significantly more difficult, and the effect less devastating, if the opponent just doesn’t have an OPM equivalent. There goes your second strike.
It is not a fucking weapon. The damage caused by even the worst breaches is nowhere near the damage caused by a thermonuclear explosion. Even conventional weapons cause more damage, and are more scary. This is also supported by the fact that no sane person would risk armed conflict over hacking incidents. No one is afraid of getting pwned.

Today’s attackers don’t even need their advantage. They get in because we let them. We should focus on exploit mitigation (Grsecurity), isolation (Qubes), sandboxed multiprocess architectures (Chrome, qmail), encryption, and fixing shit. Instead we are wasting time trying to make the attribution dice work, listening to fumbling idiots threaten the Chinese, and looking for a magic security device that solves everything if we just connect it to the network.

We know what to do and we know how to do it.

Rowhammering ElGamal

Thu, 12 Mar 2015 00:00:00 +0000

I had this idea of combining a new hardware attack with an old cryptanalysis paper. Specifically, using Rowhammer to flip bits in an ElGamal private key. The usual approach is to target executable memory or the page table and hope for a change that makes the system exploitable. I’m guessing this is not going to be stable because you need a change in particular and other changes may crash things. But we’ll see when Google releases their Chrome sandbox escape/kernel mode code execution details. The attack described in Fault Cryptanalysis of ElGamal Signature Scheme results in key recovery for any change in the private key.

ElGamal signatures

Just a quick reminder on how ElGamal works. Nothing new here, feel free to skip ahead to the attack. It’s an asymmetric signature scheme based on the Discrete Logarithm Problem.

A large prime number, p, and a generator, g, defines a multiplicative group. Choose a random integer, 1 < a < p - 2, as the private key, then compute the public key as y = g^a mod p.

Normally signing works like so (m is the message to be signed):

Choose a random integer unique for each signature, 1 < k < p, so that k and p - 1 are relative primes.
r = g^k mod p
s = k^-1 * (hash(m) - a * r) mod (p - 1)
If s == 0, then goto 1.
The pair (r, s) is the signature.

Verification looks kind of weird at first glance:

Check that 0 < r < p.
Check that 0 < s < p - 1.
Verify that g^hash(m) = y^r * r^s mod p

But why? Solve the equation in step 3 of signing:

hash(m) = a * r + s * k mod (p - 1)
g^hash(m) = g^{a * r} * g^{s * k} mod p
g^hash(m) = y^r * r^s mod p

Hey, how is that mod p, not mod (p - 1)? It’s Fermat’s fault.

Fault cryptanalysis attack

If for some reason one of the secret key bits gets flipped in memory, then an incorrect signature is computed. Not only will this signature fail to verify, it reveals the original value of the flipped bit. Since the original paper is behind a paywall I repeat some of the results here.

A single bit error at the i-th bit of the secret key can be expressed as

a’ = a ± 2ⁱ

This will result in an incorrect s:

s’ = k^-1 * (hash(m) - a’ * r) mod (p - 1)

From the incorrect signature (r, s’) the attacker can calculate r^s’. Let’s see why that’s a problem.

r^s’ = (g^k)^{k^-1 * (hash(m) - a’ * r)} mod (p - 1)

r^s’ = g^{k * k^-1 * (hash(m) - a’ * r)} mod (p - 1)

k * k^-1 is of course 1.

r^s’ = g^{hash(m) - r * a’} mod (p - 1)

Substituting a’ we get:

r^s’ = g^{hash(m) - r * (a ± 2ⁱ)} mod (p - 1)

r^s’ = g^{hash(m) - r * a ∓ r * 2ⁱ} mod (p - 1)

r^s’ = g^hash(m) * g^{-r * a} * g^{∓r * 2ⁱ} mod (p - 1)

Notice that g^{-r * a} = (g^a)^-r = y^-r.

r^s’ = g^hash(m) * y^-r * g^{∓r * 2ⁱ} mod (p - 1)

Now you can solve for g^{∓r * 2ⁱ}.

g^{∓r * 2ⁱ} = r^s’ / g^hash(m) / y^-r mod (p - 1)

OK, so you need to know i and whether it’s minus or plus. But you still have to solve the DLP. Or do you? Remember, i is between 0 an the bit length of the secret key so you can just try them all. Once you have e.g. +2⁵, you know that the fifth bit changed from 1 to 0.

You need to repeat this many times to recover most bit of the secret key. (Actually way more than the paper estimates, since rowhammer changes the same cell with high probability.) This is easy if the program is structured in such a way that you can start the attack with a fresh copy of the private key and corrupt just one bit. The attack still works if you keep flipping more and more bits in the already corrupted key. To understand this, let’s see how flipping the j-th bit would affect the above calculation. The more corrupt private key is

a’’ = a ± 2ⁱ ± 2^j

The two ± signs can be the same or opposite, it doesn’t matter. You can do the above calculation again with one extra term if you want, I typed enough sup tags for today. The final equation solved for g^{∓r * 2^j} will be:

g^{∓r * 2^j} = r^s’’ / g^hash(m) / y^-r / g^{∓r * 2ⁱ} mod (p - 1)

You already know i and one of the signs from the previous step, so you can try all the j values with both plus and minus again. Even if you don’t know i, it’s still brute forceable which might be useful if you accidentally corrupt two bits at once.

Rowhammer

The basic idea behind Rowhammer is that you can access certain DRAM cells repeatedly and sometimes the charge will leak into a nearby DRAM cell. Cells on a DRAM chip are organized into rows, discharging and recharging (hammering) both neighbours of the target row will likely change the memory stored in that row, hence the name. The neat or scary thing about this is the fact that it works regardless of any software or hardware access control.

To target anything, you need to know the memory layout of the process (/proc/$pid/maps) and how the virtual memory is mapped to physical memory (/proc/$pid/pagemaps). Grsecurity protects both of these but there could be other, unexpected ways I’m not thinking of. Obviously these mappings have to be randomized for hiding them to be meaningful, which /proc/$pid/maps is, /proc/$pid/pagemaps I don’t think so.

The attack doesn’t work on all RAM modules. It doesn’t work on ECC RAM but that’s extremely expensive. There’s a memtest fork that can test your hardware and a userspace testing utility that works as any user.

Secure Secure Shell

Sun, 04 Jan 2015 00:00:00 +0000

You may have heard that the NSA can decrypt SSH at least some of the time. If you have not, then read the latest batch of Snowden documents now. All of it. This post will still be here when you finish. My goal with this post here is to make NSA analysts sad.

TL;DR: Scan this post for fixed width fonts, these will be the config file snippets and commands you have to use.

Warning: You will need a recent OpenSSH version. It should work with 6.5 but I have only tested 6.7 and connections to Github. Here is a good compatibility matrix.

The crypto

Reading the documents, I have the feeling that the NSA can 1) decrypt weak crypto and 2) steal keys. Let’s focus on the crypto first. SSH supports different key exchange algorithms, ciphers and message authentication codes. The server and the client choose a set of algorithms supported by both, then proceed with the key exchange. Some of the supported algorithms are not so great and should be disabled completely. This hurts interoperability but everyone uses OpenSSH anyway. Fortunately, downgrade attacks are not possible because the supported algorithm lists are included in the key derivation. If a man in the middle were to change the lists, then the server and the client would calculate different keys.

Key exchange

There are basically two ways to do key exchange: Diffie-Hellman and Elliptic Curve Diffie-Hellman. Both provide forward secrecy which the NSA hates because they can’t use passive collection and key recovery later. The server and the client will end up with a shared secret number at the end without a passive eavesdropper learning anything about this number. After we have a shared secret we have to derive a cryptographic key from this using a key derivation function. In case of SSH, this is a hash function. Collision attacks on this hash function have been proven to allow downgrade attacks.

DH works with a multiplicative group of integers modulo a prime. Its security is based on the hardness of the discrete logarithm problem.

Alice           Bob
---------------------------
Sa = random
Pa = g^Sa   --> Pa
                Sb = random
Pb          <-- Pb = g^Sb
s = Pb^Sa       s = Pa^Sb
k = KDF(s)      k = KDF(s)

ECDH works with elliptic curves over finite fields. Its security is based on the hardness of the elliptic curve discrete logarithm problem.

Alice           Bob
---------------------------
Sa = random
Pa = Sa * G --> Pa
                Sb = random
Pb          <-- Pb = Sb * G
s = Sa * Pb     s = Sb * Pa
k = KDF(s)      k = KDF(s)

OpenSSH supports 11 key exchange protocols:

curve25519-sha256: ECDH over Curve25519 with SHA2
diffie-hellman-group1-sha1: 1024 bit DH with SHA1
diffie-hellman-group14-sha1: 2048 bit DH with SHA1
diffie-hellman-group14-sha256: 2048 bit DH with SHA2
diffie-hellman-group16-sha512: 4096 bit DH with SHA2
diffie-hellman-group18-sha512: 8192 bit DH with SHA2
diffie-hellman-group-exchange-sha1: Custom DH with SHA1
diffie-hellman-group-exchange-sha256: Custom DH with SHA2
ecdh-sha2-nistp256: ECDH over NIST P-256 with SHA2
ecdh-sha2-nistp384: ECDH over NIST P-384 with SHA2
ecdh-sha2-nistp521: ECDH over NIST P-521 with SHA2

We have to look at 3 things here:

ECDH curve choice: This eliminates 9-11 because NIST curves suck. They leak secrets through timing side channels and off-curve inputs. Also, NIST is considered harmful and cannot be trusted.
Bit size of the DH modulus: This eliminates 2 because the NSA has supercomputers and possibly unknown attacks. 1024 bits simply don’t offer sufficient security margin.
Security of the hash function: This eliminates 2, 3, and 7 because SHA1 is broken. We don’t have to wait for a second preimage attack that takes 10 minutes on a cellphone to disable it right now.

We are left with 1 and 8, as well as 4-6 which were added in OpenSSH 7.3. 1 is better and it’s perfectly OK to only support that but for interoperability (with Eclipse, WinSCP), 8 can be included.

Recommended /etc/ssh/sshd_config snippet:

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

Recommended /etc/ssh/ssh_config snippet:

# Github needs diffie-hellman-group-exchange-sha1 some of the time but not always.
#Host github.com
#    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
    
Host *
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

If you chose to enable 8, open /etc/ssh/moduli if exists, and delete lines where the 5th column is less than 2000.

awk '$5 > 2000' /etc/ssh/moduli > "${HOME}/moduli"
wc -l "${HOME}/moduli" # make sure there is something left
mv "${HOME}/moduli" /etc/ssh/moduli

If it does not exist, create it:

ssh-keygen -G /etc/ssh/moduli.all -b 4096
ssh-keygen -T /etc/ssh/moduli.safe -f /etc/ssh/moduli.all
mv /etc/ssh/moduli.safe /etc/ssh/moduli
rm /etc/ssh/moduli.all

This will take a while so continue while it’s running.

Authentication

The key exchange ensures that the server and the client shares a secret no one else knows. We also have to make sure that they share this secret with each other and not an NSA analyst.

Server authentication

The server proves its identity to the client by signing the key resulting from the key exchange. There are 4 public key algorithms for authentication:

DSA with SHA1
ECDSA with SHA256, SHA384 or SHA512 depending on key size
Ed25519 with SHA512
RSA with SHA1

DSA keys must be exactly 1024 bits so let’s disable that. Number 2 here involves NIST suckage and should be disabled as well. Another important disadvantage of DSA and ECDSA is that it uses randomness for each signature. If the random numbers are not the best quality, then it is possible to recover the secret key. Fortunately, RSA using SHA1 is not a problem here because the value being signed is actually a SHA2 hash. The hash function SHA1(SHA2(x)) is just as secure as SHA2 (it has less bits of course but no better attacks).

Protocol 2
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

The first time you connect to your server, you will be asked to accept the new fingerprint.

This will also disable the horribly broken v1 protocol that you should not have enabled in the first place. We should remove the unused keys and only generate a large RSA key and an Ed25519 key. Your init scripts may recreate the unused keys. If you don’t want that, remove any ssh-keygen commands from the init script.

cd /etc/ssh
rm ssh_host_*key*
ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N "" < /dev/null
ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N "" < /dev/null

Client authentication

The client must prove its identity to the server as well. There are various methods to do that.

The simplest is password authentication. This should be disabled immediately after setting up a more secure method because it allows compromised servers to steal passwords. Password authentication is also more vulnerable to online bruteforce attacks.

Recommended /etc/ssh/sshd_config snippet:

PasswordAuthentication no
ChallengeResponseAuthentication no

Recommended /etc/ssh/ssh_config snippet:

Host *
    PasswordAuthentication no
    ChallengeResponseAuthentication no

The most common and secure method is public key authentication, basically the same process as the server authentication.

Recommended /etc/ssh/sshd_config snippet:

PubkeyAuthentication yes

Recommended /etc/ssh/ssh_config snippet:

Host *
    PubkeyAuthentication yes
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa

Generate client keys using the following commands:

ssh-keygen -t ed25519 -o -a 100
ssh-keygen -t rsa -b 4096 -o -a 100

You can deploy your new client public keys using ssh-copy-id.

It is also possible to use OTP authentication to reduce the consequences of lost passwords. Google Authenticator is a nice implementation of TOTP, or Timebased One Time Password. You can also use a printed list of one time passwords or any other PAM module, really, if you enable ChallengeResponseAuthentication.

User Authentication

Even with Public Key authentication, you should only allow incoming connections from expected users. The AllowUsers setting in sshd_config lets you specify users who are allowed to connect, but this can get complicated with a large number of ssh users. Additionally, when deleting a user from the system, the username is not removed from sshd_config, which adds to maintenance requirements. The solution is to use the AllowGroups setting instead, and add users to an ssh-user group.

Recommended /etc/ssh/sshd_config snippet:

AllowGroups ssh-user

Create the ssh-user group with sudo groupadd ssh-user, then add each ssh user to the group with sudo usermod -a -G ssh-user <username>.

Symmetric ciphers

Symmetric ciphers are used to encrypt the data after the initial key exchange and authentication is complete.

Here we have quite a few algorithms (10-14 were removed in OpenSSH 7.6):

3des-cbc
aes128-cbc
aes192-cbc
aes256-cbc
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.com
aes256-gcm@openssh.com
arcfour
arcfour128
arcfour256
blowfish-cbc
cast128-cbc
chacha20-poly1305@openssh.com

We have to consider the following:

Security of the cipher algorithm: This eliminates 1 and 10-12 - both DES and RC4 are broken. Again, no need to wait for them to become even weaker, disable them now.
Key size: At least 128 bits, the more the better.
Block size: Does not apply to stream ciphers. At least 128 bits. This eliminates 13 and 14 because those have a 64 bit block size.
Cipher mode: The recommended approach here is to prefer AE modes and optionally allow CTR for compatibility. CTR with Encrypt-then-MAC is provably secure.

Chacha20-poly1305 is preferred over AES-GCM because the SSH protocol does not encrypt message sizes when GCM (or EtM) is in use. This allows some traffic analysis even without decrypting the data. We will deal with that soon.

Recommended /etc/ssh/sshd_config snippet:

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

Recommended /etc/ssh/ssh_config snippet:

Host *
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

Message authentication codes

Encryption provides confidentiality, message authentication code provides integrity. We need both. If an AE cipher mode is selected, then extra MACs are not used, the integrity is already given. If CTR is selected, then we need a MAC to calculate and attach a tag to every message.

There are multiple ways to combine ciphers and MACs - not all of these are useful. The 3 most common:

Encrypt-then-MAC: encrypt the message, then attach the MAC of the ciphertext.
MAC-then-encrypt: attach the MAC of the plaintext, then encrypt everything.
Encrypt-and-MAC: encrypt the message, then attach the MAC of the plaintext.

Only Encrypt-then-MAC should be used, period. Using MAC-then-encrypt have lead to many attacks on TLS while Encrypt-and-MAC have lead to not quite that many attacks on SSH. The reason for this is that the more you fiddle with an attacker provided message, the more chance the attacker has to gain information through side channels. In case of Encrypt-then-MAC, the MAC is verified and if incorrect, discarded. Boom, one step, no timing channels. In case of MAC-then-encrypt, first the attacker provided message has to be decrypted and only then can you verify it. Decryption failure (due to invalid CBC padding for example) may take less time than verification failure. Encrypt-and-MAC also has to be decrypted first, leading to the same kind of potential side channels. It’s even worse because no one said that a MAC’s output can’t leak what its input was. SSH by default, uses this method.

Here are the available MAC choices:

hmac-md5
hmac-md5-96
hmac-sha1
hmac-sha1-96
hmac-sha2-256
hmac-sha2-512
umac-64
umac-128
hmac-md5-etm@openssh.com
hmac-md5-96-etm@openssh.com
hmac-sha1-etm@openssh.com
hmac-sha1-96-etm@openssh.com
hmac-sha2-256-etm@openssh.com
hmac-sha2-512-etm@openssh.com
umac-64-etm@openssh.com
umac-128-etm@openssh.com

The selection considerations:

Security of the hash algorithm: No MD5 and SHA1. Yes, I know that HMAC-SHA1 does not need collision resistance but why wait? Disable weak crypto today.
Encrypt-then-MAC: I am not aware of a security proof for CTR-and-HMAC but I also don’t think CTR decryption can fail. Since there are no downgrade attacks, you can add them to the end of the list. You can also do this on a host by host basis so you know which ones are less safe.
Tag size: At least 128 bits. This eliminates umac-64-etm.
Key size: At least 128 bits. This doesn’t eliminate anything at this point.

Recommended /etc/ssh/sshd_config snippet:

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

Recommended /etc/ssh/ssh_config snippet:

Host *
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

Preventing key theft

Even with forward secrecy the secret keys must be kept secret. The NSA has a database of stolen keys - you do not want your key there.

System hardening

OpenSSH has some undocumented, and rarely used features. UseRoaming is one such feature with a known vulnerability.

Recommended /etc/ssh/ssh_config snippet:

Host *
   UseRoaming no

This post is not intended to be a comprehensive system security guide. Very briefly:

Don’t install what you don’t need: Every single line of code has a chance of containing a bug. Some of these bugs are security holes. Fewer lines, fewer holes.
Use free software: As in speech. You want to use code that’s actually reviewed or that you can review yourself. There is no way to achieve that without source code. Someone may have reviewed proprietary crap but who knows.
Keep your software up to date: New versions often fix critical security holes.
Exploit mitigation: Sad but true - there will always be security holes in your software. There are things you can do to prevent their exploitation, such as GCC’s -fstack-protector. One of the best security projects out there is Grsecurity. Use it or use OpenBSD.

Traffic analysis resistance

Set up Tor hidden services for your SSH servers. This has multiple advantages. It provides an additional layer of encryption and server authentication. People looking at your traffic will not know your IP, so they will be unable to scan and target other services running on the same server and client. Attackers can still attack these services but don’t know if it has anything to do with the observed traffic until they actually break in.

Now this is only true if you don’t disclose your SSH server’s fingerprint in any other way. You should only accept connections from the hidden service or from LAN, if required.

If you don’t need LAN access, you can add the following line to /etc/ssh/sshd_config:

ListenAddress 127.0.0.1:22

Add this to /etc/tor/torrc:

HiddenServiceDir /var/lib/tor/hidden_service/ssh
HiddenServicePort 22 127.0.0.1:22

You will find the hostname you have to use in /var/lib/tor/hidden_service/ssh/hostname. You also have to configure the client to use Tor. For this, socat will be needed. Add the following line to /etc/ssh/ssh_config:

Host *.onion
    ProxyCommand socat - SOCKS4A:localhost:%h:%p,socksport=9050

Host *
    ...

If you want to allow connections from LAN, don’t use the ListenAddress line, configure your firewall instead.

Key storage

You should encrypt your client key files using a strong password. Additionally, you can use ssh-keygen -o -a $number to slow down cracking attempts by iterating the hash function many times. You may want to store them on a pendrive and only plug it in when you want to use SSH. Are you more likely to lose your pendrive or have your system compromised? I don’t know.

Unfortunately, you can’t encrypt your server key and it must be always available, or else sshd won’t start. The only thing protecting it is OS access controls.

The end

It’s probably a good idea to test the changes. ssh -v will print the selected algorithms and also makes problems easier to spot. Be extremely careful when configuring SSH on a remote host. Always keep an active session, never restart sshd. Instead you can send the SIGHUP signal to reload the configuration without killing your session. You can be even more careful by starting a new sshd instance on a different port and testing that.

Can you make these changes? If the answer is yes, then…

If the answer is no, it’s probably due to compatibility problems. You can try to convince the other side to upgrade their security and turn it into a yes. I have created a wiki page where anyone can add config files for preserving compatibility with various SSH implementations and SSH based services.

If you work for a big company and change management doesn’t let you do it, I’m sorry. I’ve seen the v1 protocol enabled in such places. There is no chance of improvement. Give up to preseve your sanity.

Special thanks to the people of Twitter for the improvements.

ChangeLog

You may have noticed that this document changed since last time. I want to be very transparent about this. There were three major changes:

After some debate and going back and forth between including GCM or not, it’s now back again. The reason for dropping it was that SSH doesn’t encrypt packet sizes when using GCM. The reason for bringing it back is that SSH does the same with any EtM algorithms. There is no way around this unless you can live with chacha20-poly1305 only. Also, the leaked documents don’t sound like they can figure out the lengths or confirm presence of some things, more like straight up “send it to us and we’ll decrypt it for you”. Wrapping SSH in a Tor hidden service will take care of any traffic analysis concerns.
I’m now allowing Encrypt-and-MAC algorithms with CTR ciphers as a last resort. I initially thought it was possible to use downgrade attacks, I now think it is not.
I briefly disabled RSA because it uses SHA1, this turned out to be a non-issue because we’re signing SHA2 hashes.

You can see the full list of changes on github. I promise not to use git push -f.

What have I done

Fri, 14 Nov 2014 00:00:00 +0000

I did something no one should ever do: I invented my own key derivation function, BFKDF. It’s based on scrypt so it can’t be too bad. I tried to make sure it’s not worse than scrypt. There is also some brainfuck involved which might just make me the only person who ever unironically used the language.

Why would I do something like that? Well, it all started with a new Twitter app which scrapes the website, stores all the tweets, greps for suicidal thoughts and sends alerts to friends. Sounds like a good idea, excepts it’s really not and it raises privacy concerns. True, Twitter is a public website. Unless you have a private account every tweet you ever sent is preserved forever and publicly available. However, there is a slight difference between just having it out there and having it analyzed, flagged, and e-mailed out to people. Insurance companies could refuse contracts or ask for higher fees if they know you are a “riskier” customer.

Naturally, whenever Twitter does something stupid, the topic of writing a distributed replacement comes up. But how would we do better? We’d like to keep it public, that’s the best part of Twitter, but we don’t want to allow anyone to collect all tweets. The only difference between a normal user and a scraper is the number of followed users and, in turn, the number of tweets read per second. We could encrypt them with a mostly zero key and tune it so cracking one takes about a second on a cellphone. Not a problem for regular users but it at least makes cracking all of them a little more difficult. We can make it more difficult by using a key derivation function such as PBKDF2 or scrypt as encrypt(kdf(mostly_zeros), message). The problem is that a cracking farm can beat a billion cellphones with specialized hardware. The great thing about scrypt is that it adds a large memory requirement for parallel cracking but cellphones have some RAM anyway and they don’t have to crack in parallel. There are only a few tweets coming in every few seconds. Cellphones also have a CPU. If you ever tried to mine bitcoins you know that this is something you want to avoid in a cracking circuit. You want as many copies of a pipelined implementation as you can fit on a chip.

My idea was to force them to include a fully functional CPU by generating deterministic random Turing complete code, give it deterministic random input and include the output in the final hash. Brainfuck happens to be pretty easy to randomly generate. We can’t just generate any sequence of the 8 opcodes but almost. A brainfuck program is only valid if it contains an equal number of “[” and “]” characters and, counting from left to right, there are never more “]”s than “[”s. Let’s try to count how many n character programs there are and the way to generate them with equal probability will follow.

The number of 0 character programs is 1. It’s just the empty string.
The number of 1 character programs is 6 because it can’t be any of the brackets.
The number of n > 1 character programs can be expressed recursively.
- If the first character is not a bracket, then there are 6 * f(n - 1) possible programs.
- If the first character is “[”, then we have to choose where to put the matching “]”.
  - Let 0 < k < n be this position.
  - For each possible k, there are f(k - 1) * f(n - k - 1) possible programs.
  - Using python notation, that’s sum([ f(k-1) * f(n-k-1) for k in range(1, n) ]).
- The first character can’t be “]”.
- That’s 6 * f(n - 1) + sum([ f(k - 1) * f(n - k - 1) for k in range(1, n) ]). Cool.

It’s easy to see that this is between 6^n and 7^n but I can’t be arsed to give you an exact number. It’s not important, there are enough. By following the above description and choosing the first character randomly from the 7 possible, then choosing k from uniformly if needed, then repeat recursively, we will pick each program with the same probability.

Now, there are few things we have to pay attention to when writing cryptographic code. One of the most important things is making sure the runtime of the algorithm does not depend on secret values. If we would seed a regular random number generator directly with the input of our brand new KDF and use it to generate brainfuck and its input, then measuring the runtime would probably tell the attacker everything they need to break it. It would still be OK for the Distributed Twitter scenario but not much else. Instead, let’s scrypt the input, use that as a key and IV for AES CTR, then take the bitstream and use that.

The next problem is the quality of random numbers generated by brainfuck. It sucks. It’s true, all programs have the same probability but many of them are equivalent. Infinite loops without output are common and so are infinite loops that output a short repeating sequence. Brainfuck control flow only distinguishes zeros and non-zeros, therefore the output contains too many 00, 01 and ff bytes. We can repeat this process until the output matches some complicated randomness condition but it will still suck. So let’s scrypt the output again, that’s what it’s good at, it makes shitty passwords secure. Finally, we append the input to this and scrypt the whole thing again.