SOLVED [EA-7822] mod_cpanel segfaults

[Rodrigo Gomes]

Moved from: SOLVED - [EA-7780] mod_cpanel CPU usage

Hello @cPanelMichael,

Thanks for the info.

It looks like CloudLinux released the update today.

[~]# rpm -qa|grep ea-apache24-mod_cpanel
ea-apache24-mod_cpanel-1.3-1.el7.cloudlinux.x86_64

But when I activate the module, I still get many errors like these below:

[Mon Sep 03 12:26:19.008713 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 279732 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:19.008736 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 279734 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:19.008742 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 279736 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:20.011438 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 279733 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:20.011482 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 279735 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:21.016491 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 280071 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:22.025867 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 280139 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:22.025916 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 280140 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:23.027432 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 280270 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:23.027466 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 280271 exit signal Segmentation fault (11)
[Mon Sep 03 12:26:23.027471 2018] [core:notice] [pid 279726:tid 140505417664640] AH00052: child pid 280273 exit signal Segmentation fault (11)

 

[cPanelMichael]

Hello @Rodrigo Gomes,

Can you verify that uninstalling mod_cpanel solves the problem? If so, could you open a support ticket so we can take a closer look at your system? It's possible this relates to a separate issue that wasn't addressed by the referenced case. Please post the ticket number here and we'll link this thread to the ticket.

Thank you.

 

[Rodrigo Gomes]

Hello, @cPanelMichael,

Can you verify that uninstalling mod_cpanel solves the problem? If so, could you open a support ticket so we can take a closer look at your system?

These errors only appear when mod_cpanel is enabled.

If so, could you open a support ticket so we can take a closer look at your system?

Unfortunately, I can not open a support ticket at this time.
But if you want, I can help you identify this problem. Just tell me what you need to know.

Thank you!

 

[cPanelMichael]

Unfortunately, I can not open a support ticket at this time.
But if you want, I can help you identify this problem. Just tell me what you need to know.

This type of issue is better addressed through a support ticket so we can access the affected system. Is there anything in-particular preventing you from opening the ticket that I can help with?

Thank you.

 

[Rodrigo Gomes]

This type of issue is better addressed through a support ticket so we can access the affected system. Is there anything in-particular preventing you from opening the ticket that I can help with?

Thank you.

Hello @cPanelMichael,

I'm just out of time now.
So it's more practical and quick for me to answer the forum.

I would like to be able to use mod_cpanel, but this is not the most urgent right now.

Thank you!

 

[cPanelMichael]

Hi @Rodrigo Gomes,

I've not seen any additional reports of mod_cpanel resulting in Segmentation fault (11) errors, but I'll continue to keep an eye out for any additional reports. I've also been unsuccessful at reproducing this behavior on a test system with mod_cpanel enabled. It's possible this relates to something specific to the affected server's environment. When you have the time, please open a support ticket to report this issue and post the ticket number here. I'll link this thread to the ticket and update this thread with the outcome.

Thank you.

 

[Rodrigo Gomes]

Hello @cPanelMichael,

I opened a support ticket: 10246561

Thank you for your attention!

 

[Rodrigo Gomes]

Hello @cPanelMichael,

An internal case of number EA-7822 was opened,
I would like to receive updates on this case through the forum, do I need to open a new topic?

 

[cPanelMichael]

Hello @Rodrigo Gomes,

I've moved discussion of this topic to a separate thread.

Thanks so much for opening a support ticket! With your help we were able to identify a potential issue where the suspended_user_handler() function in mod_cpanel can result in segfaults under some circumstances. Case EA-7822 is open to investigate this issue further. I'll update this thread as soon as more information is available.

Thank you.

 

[Rodrigo Gomes]

Hello @cPanelMichael,

I'm happy to help! And thanks for the update.
I'll follow this thread.

 

[Kyle Eadie]

Hello @cPanelMichael,

I just found this tread while trying to solve a similar issue after migrating from EA3 to EA4 last weekend. So far disabling mod_cpanel has fixed the problem. Turning it on brings it back.

1. Is it a problem if I leave that off?
2. Would you like more info or a ticket submitted?
3. Our hosting provider said they submitted a cPanel ticket, but later stated it had to do with our wordpress plugins.

Thanks!

 

[cPanelMichael]

Hello,

The issue referenced on this thread is solved in the latest version of the mod_cpanel RPM:

#  rpm -qa --changelog ea-apache24-mod_cpanel | head -7
* Tue Sep 18 2018 Tim Mullin - 1.5-1
- EA-7386: Eliminate warning when suspended account directory does not exist.

* Mon Sep 17 2018 Rishwanth Yeddula - 1.4-1
- EA-7821: Don't strip symbools from the debug package.
- EA-7822: Handle edge cases where request_rec->filename == NULL.
    * Some modules that hook into the map_to_storage process (eg. mod_pagespeed)

I just found this tread while trying to solve a similar issue after migrating from EA3 to EA4 last weekend. So far disabling mod_cpanel has fixed the problem. Turning it on brings it back.

Can you let me know the specific issue you are facing and how you are reproducing it?

Thank you.

 

[Kyle Eadie]

We are on the latest stable build, v74.0.9.

Core Dumps /var/log/apache2/error_log
[Mon Oct 29 11:35:36.980771 2018] [core:notice] [pid 17944] AH00051: child pid 5122 exit signal Segmentation fault (11), possible coredump in /tmp/apache2-gdb-dump
[Mon Oct 29 11:35:36.980826 2018] [core:notice] [pid 17944] AH00051: child pid 5522 exit signal Segmentation fault (11), possible coredump in /tmp/apache2-gdb-dump
[Mon Oct 29 11:35:47.076730 2018] [core:notice] [pid 17944] AH00051: child pid 5654 exit signal Segmentation fault (11), possible coredump in /tmp/apache2-gdb-dump
[Mon Oct 29 11:35:47.076870 2018] [core:notice] [pid 17944] AH00051: child pid 5665 exit signal Segmentation fault (11), possible coredump in /tmp/apache2-gdb-dump
[Mon Oct 29 11:35:47.076929 2018] [core:notice] [pid 17944] AH00051: child pid 5668 exit signal Segmentation fault (11), possible coredump in /tmp/apache2-gdb-dump

Restarting Apache /var/log/apache2/error_log
[Mon Oct 29 11:54:53.603062 2018] [mpm_prefork:notice] [pid 10988] AH00169: caught SIGTERM, shutting down
[Mon Oct 29 11:54:54.181869 2018] [core:notice] [pid 11652] core dump file size limit raised to 0 bytes
[Mon Oct 29 11:54:54.273052 2018] [suexec:notice] [pid 11652] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Oct 29 11:54:54.273087 2018] [:notice] [pid 11652] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured.
[Mon Oct 29 11:54:54.273091 2018] [:notice] [pid 11652] ModSecurity: APR compiled version="1.6.3"; loaded version="1.6.3"
[Mon Oct 29 11:54:54.273096 2018] [:notice] [pid 11652] ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"
[Mon Oct 29 11:54:54.273098 2018] [:notice] [pid 11652] ModSecurity: LUA compiled version="Lua 5.1"
[Mon Oct 29 11:54:54.273101 2018] [:notice] [pid 11652] ModSecurity: LIBXML compiled version="2.9.7"
[Mon Oct 29 11:54:54.273103 2018] [:notice] [pid 11652] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Mon Oct 29 11:54:54.545481 2018] [core:notice] [pid 11661] core dump file size limit raised to 0 bytes
[Mon Oct 29 11:54:54.638289 2018] [lbmethod_heartbeat:notice] [pid 11661] AH02282: No slotmem from mod_heartmonitor
[Mon Oct 29 11:54:54.640324 2018] [:notice] [pid 11661] mod_bw : Memory Allocated 0 bytes (each conf takes 48 bytes)
[Mon Oct 29 11:54:54.640331 2018] [:notice] [pid 11661] mod_bw : Version 0.92 - Initialized [0 Confs]
[Mon Oct 29 11:54:54.656938 2018] [mpm_prefork:notice] [pid 11661] AH00163: Apache/2.4.35 (cPanel) OpenSSL/1.0.2p mod_bwlimited/1.4 configured -- resuming normal operations
[Mon Oct 29 11:54:54.656967 2018] [core:notice] [pid 11661] AH00094: Command line: '/usr/sbin/httpd'


GDB Core Dump - without mod_cpanel
#0 find_entry (ht=0x7f5c58bbeec0, key=<value optimized out>, klen=46, val=0x0) at tables/apr_hash.c:301
hep = 0x7f5c5cb90800
he = 0x7f5c5cb90c00
hash = 1557230651
#1 0x00007f5c56b94b4b in apr_hash_get (ht=<value optimized out>, key=<value optimized out>, klen=<value optimized out>) at tables/apr_hash.c:367
he = <value optimized out>
#2 0x00007f5c42c8ef98 in ?? ()
No symbol table info available.
#3 0x0000000000000009 in ?? ()
No symbol table info available.
#4 0x00007f5c59c20838 in ?? ()
No symbol table info available.
#5 0x0000000000000000 in ?? ()
No symbol table info available.


GDB Core Dump - with mod_cpanel
#0 find_entry (ht=0x7f5c58bbeec0, key=<value optimized out>, klen=54, val=0x0) at tables/apr_hash.c:301
hep = 0x7f5c5cb90870
he = 0x7f5c5cb90c08
hash = 2088971884
#1 0x00007f5c56b94b4b in apr_hash_get (ht=<value optimized out>, key=<value optimized out>, klen=<value optimized out>) at tables/apr_hash.c:367
he = <value optimized out>
#2 0x00007f5c42c8ef98 in cpanel_open_htaccess () from /etc/apache2/modules/mod_cpanel.so
No symbol table info available.
#3 0x00007f5c57e89166 in ap_run_open_htaccess (r=0x7f5c5cb75c70, dir_name=0x7f5c5cbad398 "/home/orlando/public_html/wp-content/plugins/", access_name=0x7f5c5cd55ca0 ".htaccess", conffile=0x7ffcbb52a418, full_name=0x7ffcbb52a410)
at config.c:180
pHook = <value optimized out>
n = <value optimized out>
rv = 122000
#4 0x00007f5c57e8ca9a in ap_parse_htaccess (result=0x7ffcbb52a598, r=0x7f5c5cb75c70, override=31, override_opts=255, override_list=<value optimized out>, d=0x7f5c5cbad398 "/home/orlando/public_html/wp-content/plugins/",
access_names=0x7f5c57eacc99 "") at config.c:2214
access_name = <value optimized out>
f = 0x0
parms = {info = 0x0, override = 31, override_opts = 255, override_list = 0x0, limited = -1, limited_xmethods = 0x0, xlimited = 0x0, config_file = 0x0, directive = 0x0, pool = 0x7f5c5cb75bf8, temp_pool = 0x7f5c5cb75bf8,
server = 0x7f5c5997d910, path = 0x7f5c5cd55c70 "/home/orlando/public_html/wp-content/plugins/", cmd = 0x0, context = 0x0, err_directive = 0x0}
filename = 0x7f5c5cd55cb0 "/home/orlando/public_html/wp-content/plugins/.htaccess"
cache = <value optimized out>
new = <value optimized out>
dc = 0x0
status = <value optimized out>
#5 0x00007f5c57e80d8a in ap_directory_walk (r=<value optimized out>) at request.c:1026
htaccess_conf = 0x0
res = <value optimized out>
seg_name = <value optimized out>
delim = <value optimized out>
temp_slash = 1
opts = {opts = 110 'n', add = 0 '\000', remove = 1 '\001', override = 31, override_opts = 255, override_list = 0x0}
thisinfo = {pool = 0x7f5c5cb75bf8, valid = 7598960, protection = 1877, filetype = APR_DIR, user = 609, group = 619, inode = 11535170, device = 2049, nlink = 42, size = 4096, csize = 4096, atime = 1540574041247480,
mtime = 1540574039191480, ctime = 1540574039191480, fname = 0x7f5c5cbad398 "/home/orlando/public_html/wp-content/plugins/", name = 0x7f5c58f48da8 "\270\215\364X\\\177", filehand = 0x7ffcbb52a5cc}
save_path_info = 0x0
buf = <value optimized out>
startseg = 1
matches = 0
last_walk = 0x7f5c5cbad340
this_dir = <value optimized out>
seg = <value optimized out>
rxpool = 0x0
sec_idx = 13
cached_matches = 0
buflen = <value optimized out>
filename_len = 45
now_merged = 0x7f5c5cd45708
sconf = 0x7f5c58f43728
sec_ent = 0x7f5c58f437c8
num_sec = 13
cache = 0x7f5c5cbad2b8
entry_dir = 0x7f5c5cbad350 "/home/orlando/public_html/wp-content/plugins/wp-power-stats/"
rv = <value optimized out>
cached = 0
#6 0x00007f5c57e7b8c9 in core_map_to_storage (r=0x7f5c5cb75c70) at core.c:4675
access_status = 1555631112
#7 0x00007f5c57e7e7e0 in ap_run_map_to_storage (r=0x7f5c5cb75c70) at request.c:80
pHook = <value optimized out>
n = <value optimized out>
rv = -1
#8 0x00007f5c57e819c8 in ap_process_request_internal (r=0x7f5c5cb75c70) at request.c:221
file_req = 0
access_status = <value optimized out>
d = <value optimized out>
#9 0x00007f5c57ea2a68 in ap_process_async_request (r=0x7f5c5cb75c70) at http_request.c:451
c = 0x7f5c5aba8190
access_status = -1
#10 0x00007f5c57ea2bce in ap_process_request (r=<value optimized out>) at http_request.c:488
bb = <value optimized out>
b = <value optimized out>
c = 0x7f5c5aba8190
rv = <value optimized out>
#11 0x00007f5c57e9e9c5 in ap_process_http_sync_connection (c=0x7f5c5aba8190) at http_core.c:210
keep_alive_timeout = 10000000
r = 0x7f5c5cb75c70
cs = 0x0
csd = 0x7f5c5aba7fa0
mpm_state = 1
#12 ap_process_http_connection (c=0x7f5c5aba8190) at http_core.c:251
No locals.
#13 0x00007f5c57e94930 in ap_run_process_connection (c=0x7f5c5aba8190) at connection.c:42
pHook = <value optimized out>
n = <value optimized out>
rv = -1
#14 0x00007f5c55dbfb04 in child_main (child_num_arg=<value optimized out>, child_bucket=<value optimized out>) at prefork.c:615
current_conn = <value optimized out>
csd = 0x7f5c5aba7fa0
thd = 0x7f5c5aba5f90
osthd = 140034590844896
ptrans = 0x7f5c5aba7f28
allocator = 0x7f5c5aba1b90
status = <value optimized out>
i = <value optimized out>
lr = <value optimized out>
pollset = 0x7f5c5aba64e8
sbh = 0x7f5c5aba64e0
bucket_alloc = 0x7f5c5cb6de28
last_poll_idx = 1
lockfile = <value optimized out>
#15 0x00007f5c55dbfd4d in make_child (s=0x7f5c58bc63e0, slot=57, bucket=0) at prefork.c:716
pid = 0
#16 0x00007f5c55dc0815 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at prefork.c:824
i = <value optimized out>
idle_count = <value optimized out>
free_length = <value optimized out>
free_slots = {46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77}
last_non_dead = <value optimized out>
bucket_make_child_record = 0
bucket_kill_child_record = -1
total_non_dead = <value optimized out>
#17 prefork_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at prefork.c:1019
status = 27
pid = {pid = -1, in = 0x0, out = 0x7f5c5ab9a598, err = 0x1}
child_slot = <value optimized out>
exitwhy = 0
processed_status = <value optimized out>
index = <value optimized out>
remaining_children_to_start = 0
i = <value optimized out>
#18 0x00007f5c57e697ae in ap_run_mpm (pconf=0x7f5c58b9d138, plog=0x7f5c58bca358, s=0x7f5c58bc63e0) at mpm_common.c:96
pHook = <value optimized out>
n = <value optimized out>
rv = -1
#19 0x00007f5c57e62a71 in main (argc=3, argv=0x7ffcbb52abd8) at main.c:819
c = 0 '\000'
showcompile = 0
showdirectives = 0
confname = 0x7f5c57ea9bcc "conf/httpd.conf"
def_server_root = 0x7f5c57ea9bbf "/etc/apache2"
temp_error_log = 0x0
error = <value optimized out>
process = 0x7f5c58b9b218
pconf = 0x7f5c58b9d138
plog = 0x7f5c58bca358
ptemp = 0x7f5c58bc8348
pcommands = 0x7f5c58bbf248
opt = 0x7f5c58bbf338
rv = <value optimized out>
mod = <value optimized out>
opt_arg = 0x7f5c58b97250 ""
signal_server = <value optimized out>
rc = <value optimized out>

 

[cPanelMichael]

Hello @Kyle Eadie,

Our hosting provider said they submitted a cPanel ticket, but later stated it had to do with our wordpress plugins.

Could you let us know the ticket number?

Thank you.

 

[Kyle Eadie]

10575417

Thanks!

 

[cPanelMichael]

Hello @Kyle Eadie,

The investigation in that support tickets suggests the issue relates to one of the WordPress plugins installed on the affected accounts. Have you tried temporarily disabling the plugins on those accounts to see if the issue persists? If so, go ahead and open a new support ticket and reference the previous ticket number when doing so in order for us to take a closer look.

Thank you.

 

[Kyle Eadie]

I was not able to figure out how to get the core dumps to save again, they weren't saving to the tmp directory like before. I also don't want to risk bringing out sites down again like has happened. For now I think we will just leave mod_cpanel off.

Thanks!