authorized key lime pie

Published: 2013-12-20
Last Updated: 2013-12-20 00:10:58 UTC
by Daniel Wesemann (Version: 1)
7 comment(s)

Recently, while conducting a review at a community college where I'm helping out with security, we tried to determine why they kept having remote logins to generic student accounts, even though they had just reset all the passwords and erased all the home directories, to prepare for a new course.

The probably good news is that they noticed. The not so good news is that they couldn't figure out what was going on. Turns out that some of the student home directores contained a .ssh folder, which in turn contained an authorized_keys file. Some of these files were from as far back as 2011, and others contained a dozen or more public keys. Nobody had any idea of where the corresponding private keys were, and who had access to them, but it is safe to assume that former students had added a key of their own, and some of them had kept occasionally using the lab accounts.

When the teacher prepped the systems for a new set of lab exercises, he reset all the passwords, and erased all the contents of the home directories.  All the contents?  No. The cleanup script that they used did a for-loop over the existing home folders, and basically ran

cd /home/user1
rm -rf *
cd /home/user2


which .. to their surprise .. left the ".ssh" folders and its contents in place. [Why? Well, "*" is a shell expansion, and customarily ignores "hidden" files that start with a "dot". But that's a different story :)]  Due to this, all the authorized_keys remained, and the corresponding keys continued to work, no matter how often the administration reset the passwords on the student lab accounts.


Once we had this figured out and resolved, we wondered how many more stale authorized_keys files they had elsewhere in their Unix environment, so we went hunting from box to box:

find / -name "authorized_keys" -exec ls -ald \{\} \;

The advantage of doing it this way is that the "ls" command conveniently lists both the file size and the file timestamp, like thusly:

-rw------- 1 theowrig users 605 Apr 25  2008 /home/theowrig/.ssh/authorized_keys
-rw------- 1 stuhouwe users 393 May 15  2010 /home/stuhouwe/.ssh/authorized_keys


which gives a first indication of possible issues. While user "theowrig" was still working at the college, he had no idea what or where the corresponding private key was. No abuse of this particular account was detected when we reviewed the logs, but the account basically was a sitting duck since 2008 if anyone else once had access to the corresponding private key.

Long story short: If you are running a Unix environment that permits key-based SSH login, maybe it is a good idea to check for stale authorized_keys files. You can go by date and file size for a first triage, but often will find that you need to look at the file contents themselves for an indication whose key in the past might have been authorized to do what.

Another worthwhile exercise is to check the SSH logs for as far back as you have them, and to extract which accounts are being connected to by means of key-based login:

debian:/var/log# cat sshd | grep -i "accepted publickey" | perl -pe 's/.*for (\S+) from (\S+).*/$1 $2/' | sort | uniq -c | sort -rn
   1008 rendwras x.x.63.79
    550 sablythe x.x.25.12
    263 markraji x.x.25.235
    223 rendwras x.x.10.141
    211 arfranci x.x.65.90

This example shows user "rendwras" as a frequent user of key-based login, and he is accessing from two different IPs. Unfortunately, SSH by default does not log the fingerprint of the key that is being used, but even without this information, a quick tally of the logs like shown here can help to spot issues. If you have any other tips on how to keep tabs of private and public keys authorized for ssh login, please share in the comments below.


7 comment(s)


In order to avoid this problem, in addition to disabling the user's password, I add disabled users to a "disabled" posix group. Then in sshd_config you just need this line:
DenyGroups disabled

Of course, this is only a clean solution if usernames are unique. I think it's a bit unadvisable/messy to recycle usernames for reasons like this - I'd avoid it if possible. But if you are going to recycle usernames, be sure that you also delete (I recommend disabling entirely) their crontab files. Otherwise, they could easily re-install a backdoor after your wipe script is run. There may also be other problems to watch out for...
Users probably shouldn't be using key-based auth in the first place in that environment.

I frequently use /etc/ssh/sshd_config config directives like this:

AuthorizedKeysFile /etc/sshkeys/%u.keys
X11Forwarding off
AllowTcpForwarding no
PermitTunnel no
PermitUserEnvironment no

I make /etc/sshkeys owned by root and chmod 711.

There is no need to allow users to make their own authorized keys file in their home directory; it is just the default.

Nobody can setup locally stored sshkeys; without initial permission from root, the users can only login using the ssh public keys separately associated with the user in the LDAP / FreeIPA central authentication system.

Thank you for the good, experience based, advice.
You'd be surprised how many times I hear companies say "we use SSH" so, by implication, 'we're ok'.
Sounds like the old adage... our network is behind a firewall, and the IP addresses are private, therefore we are secure.

Often coupled with.... No need to bother enabling SSL/HTTPS to manage various devices; behind the firewall, and it's on a switch - not a hub, so we don't need to worry about sniffers.
You should have performed a find for "authorized_keys*", since most SSH servers will also look for authorized_keys2, if it can't find a good public key in authorized_keys.
I like what Mysid suggests, put the admin in charge of the keys.

Using the users cookbook in Chef is a simple way to automate and mange the user accounts and associated keys.
Putting keys in LDAP would also be a great choice.
The way I usually disable an account is simple, will leave everything intact including password, and will prevent both console logins, ssh logins and cronjobs from running. How? - Just zero out the permissions on the home directory itself: chmod 0000 /home/<user>.

Diary Archives