The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Apache Processes

Discussion in 'Security' started by reggaebkk, Nov 4, 2014.

  1. reggaebkk

    reggaebkk Registered

    Nov 4, 2014
    Likes Received:
    Trophy Points:
    cPanel Access Level:
    Root Administrator
    I re-open this ticket because this issue has been tormenting me for quite some time.

    My host tells me that they are normal apache processes, and everywhere I read that they are normal.

    I have been looking at this issue very closely and in my case I could notice that:

    - When upgrading from centos 5.x to 6.x, these processes stopped for a few hours, then started to return 1 by 1, and now I have a dozen always open

    - if I kill them all and restart http service, then my load goes down for a while, up to a couple of hours, then returns.

    - they seem to be responsible for high load, but they usually show 0% usage of memory and CPU (or is it Maldet that uses so much resources to keep up with what they are doing)

    - from time to time I see one that is run by root, but most of the time they are run by nobody which is I think the normal way, so why the root one?

    but more concretely this is what bothers me:

    When I track these processes, some show this kind of result:

    "lstat("/home/XXXXXX/public_html/culture_reggae_afro", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    open("/home/XXXXXX/public_html/culture_reggae_afro/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/XXXXXX/public_html/culture_reggae_afro/artists", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    open("/home/XXXXXX/public_html/culture_reggae_afro/artists/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/XXXXXX/public_html/culture_reggae_afro/artists/fj", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    open("/home/XXXXXX/public_html/culture_reggae_afro/artists/fj/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/XXXXXX/public_html/culture_reggae_afro/artists/fj/him.jpg", {st_mode=S_IFREG|0644, st_size=36495, ...}) = 0
    Going on a website that has no php script and look for .htaccess files... fine
    "/home/hotelcom/public_html/crans-montana_directory/submit_useful_link.html", {st_mode=S_IFREG|0644, st_size=14714, ...}) = 0
    open("/.htaccess", O_RDONLY|O_CLOEXEC)  = -1 ENOENT (No such file or directory)
    lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
    open("/home/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/YYYYYYY", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
    open("/home/YYYYYYY/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/YYYYYYY/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
    open("/home/YYYYYYY/public_html/.htaccess", O_RDONLY|O_CLOEXEC) = 88
    going to another website that is not directly linked to that 1st one

    "stat("/home/ZZZZZZ/public_html/components", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    open("/home/ZZZZZZZ/public_html/components/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/ZZZZZZZ/public_html/components/com_jvotesystem", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    open("/home/ZZZZZZZ/public_html/components/com_jvotesystem/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    lstat("/home/ZZZZZZZ/public_html/components/com_jvotesystem/ajax.php", {st_mode=S_IFREG|0644, st_size=40315, ...}) = 0
    In the process mentioned above, it seems to be looking for .htaccess files everywhere
    I have attached a truncated process tracking output, of course I don't understand a word in it.
    it just goes on and on and on for hours or days.

    In other such processes it's looking for files that don't exist, here index.xhtml and index.shtml... extensions I never use:

    lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa/index.shtml", 0x7fff6c9da440) = -1 ENOENT (No such file or directory)
    access("/var/cpanel/bwlimited/", F_OK) = -1 ENOENT (No such file or directory)
    stat("/home/UUUUUUUUo/public_html/country_info/asia/east_asia/china/lhasa/index.xhtml", 0x7fff6c9da440) = -1 ENOENT (No such file or directory)
    lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUU", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUUU/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUU/public_html/country_info", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUUU/public_html/country_info/asia", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia/china", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
    lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
    When I kill them, restart apache, they return instantly.

    So here are my questions:

    1/ What are these processes actually doing apart from looking for .htaccess or vulnerable files everywhere?

    2/ why doesn't anything on the system controls processes that last so long and work their way from account to account? I have CSF setup pretty tight but it doesn't seem to matter.

    3/ is there a way to track what IP is actually using this process and block it?

    4/ why are these processes increase my load but don't seem to consume cpu and ram?

    5/ how can I setup cpanel to recognise these dodgy processes and kill them fast?

    6/ Is there a way to track eventual script that is exploited or exploiting the system this way? (all my scripts are up to date, but maybe I am carrying a parasite for some time and I don't know about it. I have used freelancers in the past and it's not impossible that one of them left a small gizmo in a corner.

    ... That's a long post for a 1st post... but this issue has been increasingly bothering me for several years now and I'd really like to hear what other people think about it.

    Attached Files:

  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Apr 11, 2011
    Likes Received:
    Trophy Points:
    cPanel Access Level:
    Root Administrator
  3. quizknows

    quizknows Well-Known Member

    Oct 20, 2009
    Likes Received:
    Trophy Points:
    cPanel Access Level:
    DataCenter Provider
    A couple things here.

    For gathering data and seeing what IP(s) are in question, I'd have a look at the output of "httpd fullstatus" as well as "lsof -i :80". These should help you track things a little better and help you associate IP addresses to requests and PIDs. Also review the domlogs (domain access logs) for the domain(s) in question.

    It's normal for apache to look for .htaccess anywhere under the user account. If you open a file that's X subdirectories deep, Apache will likely look for any .htaccess files in X subdirectories. For example if I make a web request for /home/UUUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa/index.xhtml, Apache is probably going to look for:

    (and so on).

    Also It is normal to have one httpd process as root and all the child processes as the Apache user ("nobody" on a cPanel system). You may also see /usr/local/cpanel/bin/splitlogs run as root.

    I hope this sheds a little light.

Share This Page