My next class:

SSHD rootkit in the wild

Published: 2013-02-21. Last Updated: 2013-02-22 18:32:22 UTC
by Bojan Zdrnja (Version: 3)
19 comment(s)

There are a lot of discussions at the moment about a SSHD rootkit hitting mainly RPM based Linux distributions.
Thanks to our reader unSpawn, we received a bunch of samples of the rootkit. The rootkit is actually a trojanized library that links with SSHD and does *a lot* of nasty things to the system.

At this point in time we still do not know what the initial attack vector is – it is unknown how the attackers get root access on the compromised servers that is needed to change the legitimate libkeyutils library with a trojanized one. We are, of course, keeping an eye on the development and will post a new diary or update this one if we receive more information about the attack vectors.

The trojanized library is very, very nasty. Upon execution it performs a number of actions, as described below.

The code first deobfuscates the text strings needed for it to work. The original text is only XORed so this is very easy to retrieve and the deobfuscated strings have already been posted on a lot of sites.

Once that has been done, the library sets up everything needed for it to work. It resolves symbols for the following functions which are used later: PEM_write_RSAPrivateKey, PEM_write_DSAPrivateKey, MD5_Init, MD5_Update, and MD5_Final. As you can already see, it is definitely messing up with the authentication mechanism.

Besides resolving the symbols, the library also hooks the following functions: pam_authenticate, pam_start and crypt as well as audit_log_user_message and audit_log_acct_message. By hooking these functions, the rootkit can modify the flow of the SSHD – as you can see, this is a user-mode rootkit, as it does not affect the kernel.

The main activity of the rootkit consists in collection of credentials of authenticated users. Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys, so no matter which authentication mechanism you use, if the target host is infected it will successfully steal your information. The hooking of audit_log* functions was done to allow the attacker to stay as low profile as possible – if the attacker uses the hardcoded backdoor password to issue any commands to the rootkit, no logs will be created.

The current version of the rootkit supports three commands: Xver, Xcat and Xbnd. The first command just prints the rootkit’s version; the Xcat commands print the collected information back in the session for the attacker while the Xbnd command allows the attacker to setup a listener.

Besides this, the rootkit can automatically send collected credentials to the attacker. In order to do this the rootkit has a DGA (Domain Generation Algorithm) implemented that will create random looking domain names in the .biz, .info and .net domains (in that order). It will then send a DNS packet containing collected credentials to the target IP address, if it was able to resolve it (meaning the attacker has registered that day’s domain). If no domains have been resolved, the DNS packet is sent to the hard-coded IP address, which in all samples we received was 78.47.139.110.

The rootkit itself looks very similar to the Ebury trojan which was detected back in 2011. In fact, I’m pretty sure that a lot of the code has been directly copied, however, the Ebury trojan patched the whole SSHD and required the attacker to change it.

This was easier to detect and prone to being overwritten with patching. The libkeyutils library, which comes as part of the keyutils-libs package is not changed that often so the chance of it being overwritten automatically is much lower.

If you run a RPM based system you can check the integrity of the file with the rpm command:

# rpm -Vv keyutils-libs-1.2-1.el5
........    /lib/libkeyutils-1.2.so
S.5.....    /lib/libkeyutils.so.1
........    /usr/share/doc/keyutils-libs-1.2
........  d /usr/share/doc/keyutils-libs-1.2/LICENCE.LGPL

This will check a lot of things, the most important being the MD5 checksum so if you see the output as one above you have a trojanized library. Proper output should have all (and only) dots. Keep in mind that the RPM’s verification, of course, depends on the integrity of its database and the kernel itself.

We will keep an eye on the development and will update the diary accordingly – if you have samples or more information, especially on what the initial attack vector is please let us know.

I’d like to thanks again to unSpawn for supporting the SANS ISC.

UPDATE:

Over the night (depending on where you live :), a lot of things happened. Thanks to one of our readers, Steve, I got access to a compromised server and hopefully might shed more light on what is going on here.

cPanel also posted a notice to their users that they have been compromised. What’s even worse, it appears that one of the main support servers in cPanel has been compromised which again means that the attackers got *loads* of passwords. cPanel recommends that passwords are changed, but keep in mind – if your servers are infected with the SSHD rootkit, the attackers will get your passwords/keys *anyway* (see below). So make sure that you check if your server has been compromised and that you clean it accordingly.

While analyzing the rootkit I noticed that, similarly to the Ebury trojan, it uses shared memory for communication between processes (and Steve e-mailed about that too). On Linuxes, you can check the status of shared memory with the ipcs command:

# ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x7400845c 1769472    root      600        4          0
0x00000000 2129921    root      644        52         2
0x7400845b 1736706    root      600        4          0
0x00000000 2162691    root      644        16384      2
0x00000000 2195460    root      644        268        2
0x0052e2c1 2228229    postgres  600        10469376   16

In order to check the owner, use the –p flag:

# ipcs -mp

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid
1769472    root       1975       1975
2129921    root       2931       2940
1736706    root       1965       1965
2162691    root       2931       2940
2195460    root       2931       2940
2228229    postgres   4011       6813

Now you can verify if SSHD owns any of the shared memory segments – normally it shouldn’t, so if it does you can try to inspect the system further.

Finally, unSpawn also wrote a ClamAV signature (a logical signature – so store this in a .ldb file):

RKH_libkeyutils.so.1.9;Target:6;(((0)&(1)&(2))&(((3)&(4)&(5))|((6)&(7)&(8))));636f6e6e656374;73656e64;736f636b6574;62696e64;746d7066696c65;77616974706964;646c636c6f7365;737472636174;737472637079

It should help detect malicious libraries. Since the attackers can store it in a different directory, make sure to use the find command to locate all potential copies. Something like this will work:

# find / -name libkeyutils*

UPDATE2:

As you can see in the comments below, SJFriedl is right that the private key cannot be stolen on the server since effectively it is never sent.

The rootkit does, however, hook the PEM_write_RSAPrivateKey and PEM_write_DSAPrivateKey functions, which are used to dump the private key into a file. I'm not sure when this is used yet, but it is possible that they maybe also hook libraries of the ssh-keygen utility, which is used for generation of RSA/DSA keys, and is also linked with the libkeyutils library.


--
Bojan (@bojanz)
INFIGO IS

Keywords: rootkit sshd
19 comment(s)
My next class:

Comments

lots of speculation about the attack vector - local workstation.
This just in from cpanel: Salutations,

You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.

As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.
I wonder if a bit more clarification is in order.

I don't believe there is anything that SSHD can do, hacked or not, to get an inbound SSH user's private key - the private key never leaves the originating system (instead, SSHD asks the incoming SSH a question that only the holder of the private key could answer). So if you're coming into a hacked server from some other place (say, via the PuTTY or SecureCRT client), your private keys are not at risk.

The only private keys that SSHD has anything to do with, as far as I know, are *host* keys of the server it's on, but SSHD has access to those anyway.

However, if the badware is hacking libkeyutils, then that is *also* used by the ssh command-line client, so if you are on a hacked server and initiate an outbound SSH session, then it appears that it can grab your private keys there (and is bad news). But this doesn't appear to involve SSHD unless libkeyutils is somehow sending ssh-based intercepted keys to sshd for phoning home.

None of this applies to *passwords*, which SSHD of course has full access to.

Disclaimer: I don't know any more about this incident than I have read here.
Quote: "Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys, so no matter which authentication mechanism you use, if the target host is infected it will successfully steal your information."

and

Quote: "...the attackers will get your passwords/keys *anyway*."

They cannot steal what is not there. So if you create a key pair off server (using PuttyGen for instance) and upload the public key only (because all you need on the server is the public key) and authorise it, they have no private key to steal so cannot access using keys.
@SJFriedl You are right, I should have better checked what I wrote :)

They still hook the RSA/DSA key write functions so I am not sure why they do it. I see that ssh-keygen is also linked to libkeyutils so it might be that they try to steal private keys during the creation process.

Thanks for pointing to this, I'll update the diary shortly.
"Now you can verify if SSHD owns any of the shared memory segments"

SSHD runs as root on RHEL at least, so I'm not sure what
ipcs -mp
is supposed to show.

thanks
I mentioned yesterday on twitter, but I don't know if this is completely valid, hopefully you can help clarify this for me. You mentioned earlier that it hooks into MD5_Init, MD5_Update, and MD5_Final. So how can we trust the MD5 sum calculated by rpm -vV, as the checksum could potentially be altered by the MD5_ functions it hooks into?
Is it limited to hosts with the RPM-specific local escaliation hole https://isc.sans.edu/diary/Fedora+RedHat+Vulnerabilty+Released/15172 ?
seems to be related to this:
https://bbs.archlinux.org/viewtopic.php?pid=1234615
Quote: [some] got fresh systems installed just to find out that the file was there few minutes later. This might suggest attack from the administrators machine.
They claim: It was confirmed that problems exist in distros based on RHEL and with the cPanel, DirectAdmin and Plesk. One of the vulnerabilities used by the backdoor is CVE-2012-56-71, remote code execution in Exim.
and maybe this is also interesting (including some logs):
https://bbs.archlinux.org/viewtopic.php?pid=1234615

MiKa
sorry, second link should be (cpanel discussion):
https://forums.cpanel.net/f185/sshd-rootkit-323962.html#post1324702
Do you concur that, if the key pair is generated on a local system using something like PuttyGen and only the public key uploaded then the exploit cannot steal the private key.

Diary Archives