Segfaults httpd error 6 with libpcre

[k1ck3r]

Hello,

Well i'm having this issue for a while however the apache is running , but keeps segfaulting and many defunct proccesses and childs of apache dies.

after doing a debug on the coredumps

here is the backtrace

PCRE Version :

pcretest -C
PCRE version 8.12 2011-01-15
Compiled with
UTF-8 support
Unicode properties support
Just-in-time compiler support
Newline sequence is LF
\R matches all Unicode newlines
Internal link size = 2
POSIX malloc threshold = 10
Default match limit = 10000000
Default recursion depth limit = 10000000
Match recursion uses stack

i have also tried compiling versions : 8.10 , 8.20

Kernel version :
Linux xxx.xxx.com 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux


Apache version :

httpd -v
Server version: Apache/2.2.3
Server built: May 4 2011 06:51:15

php -v
PHP 5.2.17 (cli) (built: Nov 28 2011 19:35:16)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with XCache v1.3.2, Copyright (c) 2005-2011, by mOo
with Zend Optimizer v3.3.9, Copyright (c) 1998-2009, by Zend Technologies

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/apache/bin/httpd...(no debugging symbols found)...done.
BFD: Warning: /home/fatafat/public_html/forums/core.18880 is truncated: expected core file size >= 1370353664, found: 1024733184.
[New Thread 18880]
Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /opt/pcre/lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /opt/pcre/lib/libpcre.so.0
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/local/apache/lib/libaprutil-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/apache/lib/libaprutil-1.so.0
Reading symbols from /lib64/libexpat.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libexpat.so.0
Reading symbols from /usr/local/apache/lib/libapr-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/apache/lib/libapr-1.so.0
Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/local/apache/modules/mod_rpaf-2.0.so...done.
Loaded symbols for /usr/local/apache/modules/mod_rpaf-2.0.so
Reading symbols from /usr/local/apache/modules/libphp5.so...done.
Loaded symbols for /usr/local/apache/modules/libphp5.so
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /opt/libmcrypt/lib/libmcrypt.so.4...done.
Loaded symbols for /opt/libmcrypt/lib/libmcrypt.so.4
Reading symbols from /usr/lib64/libltdl.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libltdl.so.3
Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /usr/lib64/libXpm.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXpm.so.4
Reading symbols from /usr/lib64/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libjpeg.so.62...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libjpeg.so.62
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /opt/curlssl/lib/libcurl.so.4...(no debugging symbols found)...done.
Loaded symbols for /opt/curlssl/lib/libcurl.so.4
Reading symbols from /usr/lib64/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libidn.so.11
Reading symbols from /usr/lib64/libldap-2.3.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libldap-2.3.so.0
Reading symbols from /usr/lib64/libmysqlclient.so.16...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libmysqlclient.so.16
Reading symbols from /opt/xml2/lib/libxml2.so.2...done.
Loaded symbols for /opt/xml2/lib/libxml2.so.2
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libaudit.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Reading symbols from /usr/lib64/liblber-2.3.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/liblber-2.3.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so
Reading symbols from /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-zts-20060613/memcache.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-zts-20060613/memcache.so
Reading symbols from /usr/lib64/gconv/CP1256.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/gconv/CP1256.so
Failed to read a valid object file image from memory.
Core was generated by `/usr/local/apache/bin/httpd -k start -DSSL'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002ac971e458ca in match () from /opt/pcre/lib/libpcre.so.0
(gdb) bt full
Cannot access memory at address 0x7fff4b93c1b8

Please elaborate

Thank you

 

[cPanelTristan]

Could you try moving /opt/pcre:

mv /opt/pcre /root/pcre

At that point, please try recompiling Apache via WHM > EasyApache (Apache Update) or using /scripts/easyapache in command line to see the results. Otherwise, I'm uncertain how you tried compiling different versions for Apache (if those were done manually in /opt).

 

[k1ck3r]

Well i have already tried what you are saying

did move the /opt/pcre to /opt/pcre2
and did easyapache rebuild but nothing changed

it didn't work , so i tried recompiling it manually using --prefix=/opt and adding unicode support

tried several versions , downgrade to 8.10 and upgrade to 8.20

i tried downgrading cpanel from Release to stable and after rebuild the errors stopped showing for around 2 weeks but then again came back showing , the only thing i did is moving the xcache config from php.ini to /usr/local/lib/php.ini.d/xcache.ini

and changed how it load from extension= to zend_extension=

i have returned that now to how it was and restarted apache and waiting to see if any coredumps will showup again


however it showsup very randomly i tried checking which files or application is causing the coredump but i couldn't find it

not a cron as its very random in timing , maybe doesn't show up for hours and then suddenly i find coredumps

Well now its been 30 minutes since in restarted apache and its fine , i have hope that this did solve it as it was stable without coredumps for 2 weeks

 

[k1ck3r]

well it just generated 3 coredump files :

-rw-------  1 nobody  nobody  1024323584 Dec  6 21:49 core.14171
-rw-------  1 nobody  nobody  1024356352 Dec  6 21:50 core.24208
-rw-------  1 nobody  nobody  1025437696 Dec  6 21:50 core.5179

each one around : 978M

 

[k1ck3r]

well it just generated 3 coredump files :

-rw-------  1 nobody  nobody  1024323584 Dec  6 21:49 core.14171
-rw-------  1 nobody  nobody  1024356352 Dec  6 21:50 core.24208
-rw-------  1 nobody  nobody  1025437696 Dec  6 21:50 core.5179

each one around : 978M

 

[cPanelTristan]

Please open up a ticket for us to investigate. We are going to need deeper access to the system than we can obtain using the forum. You can submit a ticket in WHM > Support Center > Contact cPanel or using the link in my signature.

 

[k1ck3r]

ticket submitted
Your Request id is: 2050478.

 

[k1ck3r]

it seems i still didn't get any reply on the ticket , shall i resubmit it through the datacenter ?

 

[cPanelMichael]

Hello

I see the most recent reply to that ticket was sent by us on:

December 7th, 2011 at 01:31 PM CST

We have yet to receive a response back from you. You may want to check your SPAM folders in your email/webmail client to ensure it was not filtered out of your Inbox. Otherwise, feel free to open another ticket, and reference your previous ticket number.

Thank you.