The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is this xmlrpc brute force amplification attack?

Discussion in 'Security' started by garconcn, Nov 2, 2015.

  1. garconcn

    garconcn Well-Known Member

    Joined:
    Oct 29, 2009
    Messages:
    98
    Likes Received:
    1
    Trophy Points:
    8
    Recently, I noticed that the "nobody" process was using 100% CPU for a long time on many cpanel servers. After lsof the process, I see the server was responding to the request from IP 50.93.203.147(and many other different IPs, most were VPS or Dedicated server). From the Apache access log, I see this IP issued only a few xmlrpc requests, but the request seems be different from before, as the POST size is larger and it always starts with "POST / HTTP/1.1" 2 or 3 mins before the "POST /xmlrpc.php HTTP/1.1". I guess the attacker passed a bunch of username and password into the 1st POST, then use xmlrpc to do brute force in 2nd POST continuously in one connection. Because they only do 1 or 2 connections at the same time, it bypass the old mod_sec rule for xmlrpc attack. Is there a way to stop this? Thank you.

    Brute Force Amplification Attacks Against WordPress XMLRPC - Sucuri Blog


    # lsof -p 182704
    Code:
    COMMAND  PID  USER  FD  TYPE  DEVICE  SIZE/OFF  NODE NAME
    httpd  182704 nobody  cwd  DIR  253,0  4096  2 /
    httpd  182704 nobody  rtd  DIR  253,0  4096  2 /
    httpd  182704 nobody  txt  REG  253,0  1809242  24909121 /usr/local/apache/bin/httpd
    httpd  182704 nobody  mem  REG  253,0  27424  72614015 /lib64/libnss_dns-2.12.so
    httpd  182704 nobody  mem  REG  253,0  65928  72614229 /lib64/libnss_files-2.12.so
    httpd  182704 nobody  mem  REG  253,0  206640  72613996 /lib64/libidn.so.11.6.1
    httpd  182704 nobody  mem  REG  253,0  439965  45876797 /opt/curlssl/lib/libcurl.so.4.3.0
    httpd  182704 nobody  mem  REG  253,0  2457613  24910725 /usr/local/apache/modules/mod_security2.so
    httpd  182704 nobody  mem  REG  253,0  4319965  45876366 /opt/xml2/lib/libxml2.so.2.9.2
    httpd  182704 nobody  mem  REG  253,0  90880  72613965 /lib64/libgcc_s-4.4.7-20120601.so.1
    httpd  182704 nobody  mem  REG  253,0  596272  72614059 /lib64/libm-2.12.so
    httpd  182704 nobody  mem  REG  253,0  987096  24908802 /usr/lib64/libstdc++.so.6.0.13
    httpd  182704 nobody  mem  REG  253,0  64956  24910723 /usr/local/apache/modules/mod_suphp.so
    httpd  182704 nobody  mem  REG  253,0  33138  24910724 /usr/local/apache/modules/mod_bw.so
    httpd  182704 nobody  mem  REG  253,0  9610  24910720 /usr/local/apache/modules/mod_bwlimited.so
    httpd  182704 nobody  mem  REG  253,0  79008  24905557 /usr/lib64/liblve.so.0.9.0
    httpd  182704 nobody  mem  REG  253,0  46048  24910722 /usr/local/apache/modules/mod_hostinglimits.so
    httpd  182704 nobody  mem  REG  253,0  122040  72614113 /lib64/libselinux.so.1
    httpd  182704 nobody  mem  REG  253,0  110960  72614235 /lib64/libresolv-2.12.so
    httpd  182704 nobody  mem  REG  253,0  10192  72614091 /lib64/libkeyutils.so.1.3
    httpd  182704 nobody  mem  REG  253,0  43728  72614033 /lib64/libkrb5support.so.0.1
    httpd  182704 nobody  mem  REG  253,0  10312  72614067 /lib64/libfreebl3.so
    httpd  182704 nobody  mem  REG  253,0  19536  72614048 /lib64/libdl-2.12.so
    httpd  182704 nobody  mem  REG  253,0  174840  72613927 /lib64/libk5crypto.so.3.1
    httpd  182704 nobody  mem  REG  253,0  14664  72614244 /lib64/libcom_err.so.2.1
    httpd  182704 nobody  mem  REG  253,0  946048  72613932 /lib64/libkrb5.so.3.3
    httpd  182704 nobody  mem  REG  253,0  277704  72613897 /lib64/libgssapi_krb5.so.2.2
    httpd  182704 nobody  mem  REG  253,0  1920936  72613905 /lib64/libc-2.12.so
    httpd  182704 nobody  mem  REG  253,0  142640  72614050 /lib64/libpthread-2.12.so
    httpd  182704 nobody  mem  REG  253,0  40400  72613974 /lib64/libcrypt-2.12.so
    httpd  182704 nobody  mem  REG  253,0  43880  72614238 /lib64/librt-2.12.so
    httpd  182704 nobody  mem  REG  253,0  294128  24909036 /usr/local/apache/lib/libapr-1.so.0.5.1
    httpd  182704 nobody  mem  REG  253,0  165264  72613984 /lib64/libexpat.so.1.5.2
    httpd  182704 nobody  mem  REG  253,0  232253  24909093 /usr/local/apache/lib/libaprutil-1.so.0.5.4
    httpd  182704 nobody  mem  REG  253,0  88600  72613952 /lib64/libz.so.1.2.3
    httpd  182704 nobody  mem  REG  253,0  690887  45876294 /opt/pcre/lib/libpcre.so.1.2.4
    httpd  182704 nobody  mem  REG  253,0  1963296  24905926 /usr/lib64/libcrypto.so.1.0.1e
    httpd  182704 nobody  mem  REG  253,0  441256  24908718 /usr/lib64/libssl.so.1.0.1e
    httpd  182704 nobody  mem  REG  253,0  154664  72613899 /lib64/ld-2.12.so
    httpd  182704 nobody  mem  REG  0,4  495107656 (deleted)/dev/zero (stat: No such file or directory)
    httpd  182704 nobody  mem  REG  0,4  496619447 (deleted)/dev/zero (stat: No such file or directory)
    httpd  182704 nobody  mem  REG  0,4  496619446 (deleted)/dev/zero (stat: No such file or directory)
    httpd  182704 nobody  mem  REG  0,4  496619451 (deleted)/dev/zero (stat: No such file or directory)
    httpd  182704 nobody  0r  CHR  1,3  0t0  4350 /dev/null
    httpd  182704 nobody  1w  CHR  1,3  0t0  4350 /dev/null
    httpd  182704 nobody  2w  REG  253,0  579156  28185318 /usr/local/apache/logs/error_log
    httpd  182704 nobody  3w  REG  253,0 108426411  28185555 /usr/local/apache/logs/modsec_audit.log
    httpd  182704 nobody  4r  CHR  10,57  0t0  12393 /dev/lve
    httpd  182704 nobody  5w  REG  253,0  15691127  28186096 /usr/local/apache/logs/modsec_debug.log
    httpd  182704 nobody  6u  IPv4 495104908  0t0  TCP *:http (LISTEN)
    httpd  182704 nobody  7u  IPv4 495104914  0t0  TCP *:https (LISTEN)
    httpd  182704 nobody  8r  FIFO  0,8  0t0 496619412 pipe
    httpd  182704 nobody  9w  FIFO  0,8  0t0 496619412 pipe
    httpd  182704 nobody  10w  REG  253,0 229383576  28185359 /usr/local/apache/logs/access_log
    httpd  182704 nobody  11u  0000  0,9  0  4348 [eventpoll]
    httpd  182704 nobody  12w  FIFO  0,8  0t0 496619414 pipe
    httpd  182704 nobody  13u  IPv4 497074376  0t0  TCP our_ip:http->50.93.203.147:46508 (CLOSE_WAIT)
    httpd  182704 nobody  14w  FIFO  0,8  0t0 496619417 pipe
    httpd  182704 nobody  15w  REG  253,0  0  28182908  (deleted)/usr/local/apache/logs/ssl-cache.60941
    httpd  182704 nobody  16w  REG  253,0  0  28185319  (deleted)/usr/local/apache/logs/rewrite-map.60944
    httpd  182704 nobody  18w  FIFO  0,8  0t0 496619449 pipe
    httpd  182704 nobody  19r  FIFO  0,8  0t0 496619450 pipe

    Here's the Apache log

    Code:
    50.93.203.147 - - [02/Nov/2015:13:10:52 -0800] "POST / HTTP/1.1" 200 domain.com 12744  "-"
    50.93.203.147 - - [02/Nov/2015:13:13:08 -0800] "POST /xmlrpc.php HTTP/1.1" 200 domain.com 6607  "-"
     
    #1 garconcn, Nov 2, 2015
    Last edited by a moderator: Nov 2, 2015
  2. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    940
    Likes Received:
    55
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    This should be addressed in the WP core in the next update:

    #34336 (Disable XML-RPC system.multicall authenticated requests on the first auth failure) – WordPress Trac

    In the mean time I have written some modsec rules but they might break some things like jetpack. Although I have tried to avoid this breaking any legitimate code or plugins, I offer no guarantee. As far as I can tell jetpack and other plugins very rarely use the multicall feature so these are probably safe but I have only used them on personal servers and not my customers servers.

    The first rule basically disables "system.multicall" entirely for properly formatted requests. The 2nd rule disables system.multicall for malformed requests that contain the two most commonly abused functions.

    Code:
    #would work for posts with the proper content type specified but may break legit apps :/
    SecRule REQUEST_HEADERS:Content-Type "^text/xml$" "phase:1,t:lowercase,nolog,pass,ctl:requestBodyProcessor=XML,id:3997"
    SecRule REQBODY_PROCESSOR "!^XML$" "nolog,pass,skip:1,id:3998"
    SecRule REQUEST_URI "xmlrpc.php" "t:lowercase,id:3999,deny,status:500,chain"
    SecRule XML:/methodCall/methodName/text() system.multicall
    
    #testing this because some POSTs are coming in as "application/x-www-form-urlencoded" not "text/XML"
    SecRule REQUEST_URI "xmlrpc.php" "deny,id:4000,chain"
    SecRule REQUEST_BODY "system.multicall" "chain"
    SecRule REQUEST_BODY "wp\.(getCategories|getUsersBlogs)"
    
    I recommend only using these where you are actually being attacked, and then removing them after the next WP core updates are installed on the majority of your users sites. If you do test them on servers with many sites I would be interested in receiving your feedback regarding if you get any complaints or false positives.
     
    #2 quizknows, Nov 2, 2015
    Last edited: Nov 2, 2015
  3. garconcn

    garconcn Well-Known Member

    Joined:
    Oct 29, 2009
    Messages:
    98
    Likes Received:
    1
    Trophy Points:
    8
    Thank you for this information. I will give it a try on my server and let you know if I have any result. :)
     
  4. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    940
    Likes Received:
    55
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    Good deal. I know the rules work and have blocked attacks with both of them, but just curious if there are false positives on busier servers. Cheers.
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  6. popeye

    popeye Well-Known Member

    Joined:
    May 23, 2013
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Hi could anyone please tell me how to block xmlrpc attack

    I use Free Modsecurity rules - Comodo Web Application Firewall but it does not stop them and the rule to stop them is enabled
     
  7. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,145
    Likes Received:
    34
    Trophy Points:
    48
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hello :),

    If you are getting this issues with only one site then you can add following code on your .htacces file to prevent this.

    Code:
    <Files xmlrpc.php>
        Order Deny,Allow
        Deny from all
    </Files>
     
  8. popeye

    popeye Well-Known Member

    Joined:
    May 23, 2013
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Hi no it's loads of sites.
     
  9. popeye

    popeye Well-Known Member

    Joined:
    May 23, 2013
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Hi how long until the next Wordpress update does anyone know.
     
  10. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  11. popeye

    popeye Well-Known Member

    Joined:
    May 23, 2013
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Hi thanks so i need to make sure all customers upgrade to new 4.4 and is this built in or do they have to enable it please.
     
  12. popeye

    popeye Well-Known Member

    Joined:
    May 23, 2013
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Texas
    cPanel Access Level:
    Root Administrator
    Hi i found out today that if you delete ip.pag file in var/tmp and restart web server it will create a new one and stop all attacks.

    This works with comodo mod security rules. My file was huge.
     
  13. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    940
    Likes Received:
    55
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    In 4.4, in a multicall request if auth fails the subsequent calls in that request will be denied. This is in the code and is nothing that has to be manually enabled. Once upgraded to 4.4 the site is protected from multicall requests being able to try multiple auth attempts.
     
  14. gryzli

    gryzli Active Member

    Joined:
    Jul 23, 2012
    Messages:
    44
    Likes Received:
    5
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    Twitter:
    The problem is that multicall requests are not the only trouble. In fact multicall request give the attacker better chance to guess your passwords with less number of requests and most of the times, these type of requests are not the ones breaking down your server.

    What is the real trouble is bruteforcing while using single request per user/password try, which is the case with the single xmlrpc.php calls like "wp.getUsersBlog" or "wp.getUsersCategories", because this way the attacker needs to issue much more requests to your servers (and they do that), which immediately leads to spike in your resource usage.

    In short, the bugfix of Worpress guys which is mentioned as:
    , wont fix your xmlrpc.php ddos load spikes problems at all !

    The effective protection to this attack is to try drop the request before they reach Wordpress, which could be done with ModSecurity.

    If you want to globally deny xmlrpc.php hits, you can do this with the following mod_Security rule:

    Code:
    SecRule REQUEST_URI "@pm xmlrpc.php" "phase:1, id:22, deny, status:406, msg:'Blocked xmlrpc.php hit'"
    Consider the following:
    - If some software relies on xmlrpc.php it will stop working (JetPack wp plugin is one example)
    - The current rule id is "22", make sure you have no other rules with same id (or change it to a free one)
    - Every request to xmlrpc.php or xmlrpc.php containing url, will end up with error code 406
     
    #14 gryzli, Jan 5, 2016
    Last edited by a moderator: Jan 5, 2016
  15. quizknows

    quizknows Well-Known Member

    Joined:
    Oct 20, 2009
    Messages:
    940
    Likes Received:
    55
    Trophy Points:
    28
    cPanel Access Level:
    DataCenter Provider
    The multicall requests cause significant load. WP uses a processing intensive hashing algorithm. Being able to check multiple passwords with one POST request causes that single request to make the server hash hundreds of passwords. This is by design because "hash cracking" will take longer should a database be leaked.

    You do not need to block xmlrpc entirely. This is an overly heavy solution and should only be used temporarily on sites under heavy attack (if at all). If you're going to use ModSecurity you can specifically block just the system.multicall requests, as well as "wp.getUsersBlog" or "wp.getUsersCategories" by doing post payload inspection. This allows things like Jetpack (which are extremely popular) to still function. You can also filter xmlrpc.php requests that are missing user agents or HTTP referers, which drops a significant portion of both multicall attacks and single calls from bogus bots. Some requests can also be caught because they contain mismatched or missing headers (such as containing a content-length header with no content-type specified).

    The goal of a good modsecurity rule should be to only block bad requests while still allowing software to function as intended. Rules that break legitimate features are one of the reasons customers so often demand modsecurity be disabled, causing them to forfeit their own protections.
     
    Infopro likes this.
  16. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,447
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    For anyone unsure of what xmlrpc actually is:
    Should You Disable XML-RPC on WordPress? - Wordfence
     
  17. sparek-3

    sparek-3 Well-Known Member

    Joined:
    Aug 10, 2002
    Messages:
    1,381
    Likes Received:
    23
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    But how many people actually use anything related to xmlrpc.php?

    I don't know about everyone else, but I'd factor a guess (conservatively) that 50% of the WordPress installs we have on our servers are single "install and never touch again" installations. At which point xmlrpc.php just becomes a venue that malicious users can easily exploit (or attempt to exploit) to hack into the installation and then do their bidding.

    There definitely are users that fully utilize everything regarding WordPress and xmlrpc.php, but I'm just not sure how many really are.

    My preferred method, for any model, is the principle of least privilege. Don't enable more privileges than are needed and if they are needed then those select users can enable them as needed.
     
  18. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,447
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Agree, just adding context to the thread. ;)
     
  19. sparek-3

    sparek-3 Well-Known Member

    Joined:
    Aug 10, 2002
    Messages:
    1,381
    Likes Received:
    23
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    Further to this fix for WordPress 4.4. Are they going to fix this for WordPress 4.3? 4.2? 4.1? 4.0? 3.9? 3.8? 3.7? See this is the problem with trying to support so many versions, like WordPress has tried to do.

    Up until 4.4, they were releasing new versions for 4.2, 4.1, etc. alongside 4.3 updates. And now that just suddenly stops?
     
  20. sparek-3

    sparek-3 Well-Known Member

    Joined:
    Aug 10, 2002
    Messages:
    1,381
    Likes Received:
    23
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    I would probably recommend using some form of xmlrpc.php blocking (ModSecurity is probably a good one) by default and exempting those VirtualHosts that really need access to xmlrpc.php. This is why ModSecurity is probably a good avenue for doing this, because you can exempt certain ids per VirtualHost.

    My real recommendation would be for WordPress to do an xmlrpc disable by default on new installs. But since I don't see that happening, taking matters into your own hands might be the only solution.
     
Loading...

Share This Page