The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

I found what causes FD_SETSIZE (1024) errors...

Discussion in 'General Discussion' started by dwayrynen, Jun 26, 2003.

  1. dwayrynen

    dwayrynen Member

    Joined:
    Jun 19, 2002
    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    1
    Like many other people we have been running into the FD_SETSIZE apache errors:

    [Wed Jun 18 21:47:41 2003] [warn] [csd] filedescriptor (1024) larger than FD_SETSIZE (1024) found, you probably need to rebuild Apache with a larger FD_SETSIZE
    [Wed Jun 18 21:49:21 2003] [warn] [csd] filedescriptor (1024) larger than FD_SETSIZE (1024) found, you probably need to rebuild Apache with a larger FD_SETSIZE
    [Wed Jun 18 21:50:00 2003] [warn] [csd] filedescriptor (1024) larger than FD_SETSIZE (1024) found, you probably need to rebuild Apache with a larger FD_SETSIZE

    The recommended solution has been to "recompile apache with a large FD_SETSIZE", which in my humble opinion is just a poor bandaid for fixing the real problem - and that is apache opening more and more and more file descriptors until it reachs 1024....


    How to replicate problem:

    When you run apache, you can do a "ps-ef | grep apache" to find the process id (call it xxx) and then "cd /proc/xxx/fd" - from there do a "ls | wc" to see the number of file descriptors open by apache.

    (You can also do a ls -l to see what files are opened - the log files, the config file, and a bunch of sockets).

    From there, you can try to control apache with SIGUSR1 or SIGHUP like:

    kill -SIGHUP `cat /usr/local/apache/logs/httpd.pid`

    or

    kill -SIGUSR1 `cat /usr/local/apache/logs/httpd.pid`

    Which are valid ways to control apache.

    after doing that a few times perform a "ls | wc" command again and you'll see that the number of open file descriptors has grown.

    If you do that enough so that the number of file descriptors grows to 1024 (or whatever you recompiled apache to use in /scripts/easyapache) then you get the errors in your error_log and your web sites that use cgis break, etc

    Since scripts like cpanellogd send a SIGUSR1 signal to apache on a constant basis, the number of file descriptors will grow and grow. Other tools (like manually running /usr/local/apache/apachectl graceful will also leave an extra file descriptor socket open on each invocation.


    Recompiling with FD_SETSIZE to 16000 will help stave off the problem, but the problem still remains. Perhaps Nick can poke around the apache code and kill this problem once and for all.

    We are getting screamed at by our customers. :-(

    I've verified that this does not occur with stock Apache installs.

    Darin

    cPanel.net Support Ticket Number:

    cPanel.net Support Ticket Number: 16675

    cPanel.net Support Ticket Number:
     
  2. Juanra

    Juanra Well-Known Member

    Joined:
    Sep 22, 2001
    Messages:
    777
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Spain
    Question. After you compile Apache with the FD_SETSIZE patches, what do you get in /proc/pid/status in the FDSize: line?

    cPanel.net Support Ticket Number:
     
  3. dwayrynen

    dwayrynen Member

    Joined:
    Jun 19, 2002
    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    1
    256, but I believe that is not the limit - it can grow a more file descriptors are used by a program

    root@para [/proc]# grep FDSize */status
    1/status:FDSize: 32
    11054/status:FDSize: 32
    11321/status:FDSize: 32
    11536/status:FDSize: 32
    12/status:FDSize: 32
    1240/status:FDSize: 32
    1258/status:FDSize: 32
    1276/status:FDSize: 32
    14819/status:FDSize: 32
    1597/status:FDSize: 32
    17282/status:FDSize: 32
    17284/status:FDSize: 32
    17285/status:FDSize: 256
    17313/status:FDSize: 256
    18248/status:FDSize: 32
    18250/status:FDSize: 32
    18251/status:FDSize: 256

    cPanel.net Support Ticket Number:
     
  4. DWHS.net

    DWHS.net Well-Known Member
    PartnerNOC

    Joined:
    Jul 28, 2002
    Messages:
    1,569
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    LA, Costa RIca
    cPanel Access Level:
    Root Administrator
    Any news on how to fix this without re-compiling apache?

    There has to be another way to fix it, doesn't apache have this limit set for a reason?

    More importantly how do you control these logs from going out of control?

    -Charles

    cPanel.net Support Ticket Number:
     
  5. nickn

    nickn Well-Known Member
    PartnerNOC

    Joined:
    Jun 15, 2003
    Messages:
    619
    Likes Received:
    1
    Trophy Points:
    18
  6. dysk

    dysk Well-Known Member

    Joined:
    Apr 22, 2003
    Messages:
    52
    Likes Received:
    0
    Trophy Points:
    6
    Hi-
    See http://forums.cpanel.net/showthread.php?s=&postid=44709
    It's actually a bug in the Zend optimizer that causes apache to open a new socket whenever it recieves a SIGHUP. One workaround is to disable the Zend optimizer, another is to write a cronjob that stops and starts apache every so often.
    I contacted Zend about this bug, and they have assured me that it will be fixed in their next version, so keep an eye on their website, it's possible that they already came out with the new version.
    Hope this helps.


    Regards,
    Erek Dyskant
    Unix Consultant

    cPanel.net Support Ticket Number:
     
  7. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    5
    Trophy Points:
    38
    Location:
    back woods of NC, USA
    I just had an idea and maybe it won't help .. maybe it would. I just ran this by Nick .. haven't heard back yet.

    Many of these files are open are the logs .. one for each site.

    Could we make apache write all logs to one file? ..then only have one file open (logs anyway). Could we then have a script that run before the webalizer and all that stuff ..have a script that sorts the big log file for each domain. then have it build a "baby" log for each user's domain and then chown that file to each users /home/user/ dir. That file would be flat and not open .. only built when this script runs. Stamp a file for each day.

    Sorting this huge log file may be resource intensive .. but I wonder how resource intensive writing to 600 of them at once is.. also doing this at once at 4am can be controled and isn't even mission critical if it had to be abandoned for more important stuff. Unlike now .. if the disk - system can't keep up with writing to 300 logs at the same time .. I am not sure what happens. I have a feeling this max open files is there for a reason. and bet it's preformance related ceilings we should pay attention to.

    I know this doesn't fix the zend issue. I have however seen apache run on 100K unique users a day sites run for 300 days with no restarts. ( plain RH box 1 site).

    Just throwing up dumb ideas ..see what you guys thought.

    cPanel.net Support Ticket Number:
     
  8. dwayrynen

    dwayrynen Member

    Joined:
    Jun 19, 2002
    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    1
    That would limit the initial number of file descriptors used, but the additional socket that gets opened is not associated with a log file. So consolidating logs would not stop the continuous increase in file descriptors being used.

    Dairn

    cPanel.net Support Ticket Number:
     
  9. dysk

    dysk Well-Known Member

    Joined:
    Apr 22, 2003
    Messages:
    52
    Likes Received:
    0
    Trophy Points:
    6
    Hi-
    I am certainly all for that change. It is that way that many non-webhosting companies handle log management, and I would have done that had I elected not to use cPanel. I'm of the mind that using large amounts of CPU time for a minute or two at 4AM is better than using small amounts of CPU time during peak hours. If it does turn out to be too intensive one solution is to have it process 1/3rd of the log at 1 am, another third at 2am, etc.
    You might still have to use a separate file for each byteslog, since they don't store the virtual host with the log entry.
    I remember vaguey that there is an open source file for splitting logs by domain, but I can't remember the name. I'm sure some googling would find it.
    I hope this helps.


    Regards,
    Erek Dyskant
    Unix Consultant

    cPanel.net Support Ticket Number:
     
  10. dysk

    dysk Well-Known Member

    Joined:
    Apr 22, 2003
    Messages:
    52
    Likes Received:
    0
    Trophy Points:
    6
    Re: Zend

    Hi-
    Just a quick note on Zend. I would just make sure that cPanel incorporates the newer Zend into their distribution when it comes out. Other than that I don't know that there's a whole lot the cPanel people can do about this since Zend is closed source.
    It might be worthwhile for the cPanel people to investigate some of the open source php optimizers, some of which have performance comperable to Zend.

    Regards,
    Erek Dyskant
    Unix Consultant

    cPanel.net Support Ticket Number:
     
  11. gleen

    gleen Member

    Joined:
    Nov 3, 2003
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    I am having the 'too many open files' errors, and I do not use Zend. This is a major issue..
     
  12. bjarne

    bjarne Well-Known Member

    Joined:
    Mar 23, 2002
    Messages:
    135
    Likes Received:
    0
    Trophy Points:
    16
    It's not just Zend I belive, id there is a lot of trafic I belive it will still happen. Create a cron job that does a apache stop and apache start (not graceful) and you will be ok.

    At least that's my experince, and I also posted a howto do it.

    Just stop apache, removed the pid file and start apache again. Some times on buzzy servers - if you stop apache and then start again apache will respond that it's running, becuse the pid file is still there. So eiter have 6-10 sek pause, or shorter pause and then remove the pid before restarting.


    BTW I belive that 2 GB is file size limit on most systems, so putting all the logs into one file most likly will create other less nice problems, like 50 gttpd defunkts :)
     
  13. gleen

    gleen Member

    Joined:
    Nov 3, 2003
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Apparently cPanel/WHM support is nonexistent.
     
  14. Website Rob

    Website Rob Well-Known Member

    Joined:
    Mar 23, 2002
    Messages:
    1,506
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    Alberta, Canada
    cPanel Access Level:
    Root Administrator
    #14 Website Rob, Dec 6, 2003
    Last edited: Dec 6, 2003
  15. RogelioH

    RogelioH Active Member

    Joined:
    Jun 2, 2003
    Messages:
    41
    Likes Received:
    0
    Trophy Points:
    6
    So where is the fix? i posted in the other thread and i havent heard nothing yet. My clients are starting to get pissed off because cgi is going down constantly .
     
Loading...

Share This Page