modsec might not be working correctly and hidden /proc dirs

[Spork Schivago]

Hi.

chkrootkit sent me an email saying it found hidden processes and it thinks it's discovered the LKM rootkit. It detected 2 hidden processes. I know chkrootkit doesn't do a very good job of trying to detect this rootkit, but why haven't I seen the message before? It gets weirder.

I have csf / LFD and modsec installed on my box. I was running v58.0.19 but I'm now upgrading to v58.0.20....

I run ./chkproc -v and I now see 79 processes hidden for readdir and ps.

I think, okay, I have stuff like Exim running, I have MariaDB. Maybe these are from that. So, I look at one of the PIDs...

PID 28060. I run:

ls -ld /proc/28*
/bin/ls: cannot access /proc/28*: No such file or directory

ls -ld /proc/28060*
/bin/ls: cannot access /proc/28060*: No such file or directory

ls -ld /proc/28060/
dr-xr-xr-x 7 nscd nscd 0 Aug 10 04:08 /proc/28060//

cat /proc/28060/cmdline
/usr/sbin/nscd

How come I cannot see the directory, but if I pretend it exists, I can access it? This worries me very much.

Did someone get into my system? I don't know but I'm very worried something is very wrong here.

Ken

 

[Spork Schivago]

Okay, a bit of an update. I edited the first post and removed all the information about the IP address / attacker.

First off, I'd like to apologize for being such an idiot. The attackers IP address was actually in csf.ignore. It's scanmysite.com, so it was doing what it was supposed to do. I can't believe I didn't realize that. I am so dumb.

However, the hidden /proc/ directories still worry me. Why can I not see the directories under /proc/ but I can access them if I type ls /proc/<pid>/ ?

I know I can hide files from ls if the filename or directory starts with a . However, these don't. I'm not aware of anyway, besides playing with the actual filesystem, to hide directories like that.

 

[Spork Schivago]

Okay, so I couldn't find an answer to the hidden files in the /proc directory. I've contacted various people, asked various questions on various forums and finally decided to run a test of my own.

I believe I figured out what's happening. I created a simple C program that creates two threads and prints a message to the screen with the thread ID. Then, I have the program wait for the user to hit enter. I've modified chkrootkit's chkproc source code to display the contents of the /proc/<hidden PID>/cmdline file so I can see which processes are hidden from readdir and ps. I've learned a good bit about the proc filesystem. I've learned it's a virtual file system and exists in memory.

I run my threads program and while it's waiting for user input, I run my modified chkproc program. Sure enough, two hidden directories in the proc system belong to my thread program.

So I've come to the conclusion that any program, such as MariaDB, nscd (name service cache daemon), etc, that creates threads or mutexes (if I correctly understand the definition of a mutex), has their PID hidden from programs such as ps and readdir. The /proc/<PID> for the thread is hidden, but can still be accessed.

If I run ps aux |grep mythreads (that's the name of my simple C program), I see it has a PID of 24127. If I go into the /proc/24127/task directory, I see the parent PID (24127) and the two threads that mythreads creates, 24128 and 24129. chkrootkit -v shows 24128 and 24129 as being hidden.

So, essentially, those hidden /proc entries, at least on my system, are just programs that use threads.

I felt I should update people here with the answer, in case anyone else runs into this issue.

 

[cPanelMichael]

Hello,

I'm happy to see you were able to find an answer to your question. Thank you for updating us with the outcome. The following external URL also helps to explain how /proc works for anyone that's interested:

How does /proc/* work?

Thank you.