apache down again and again, error reading response from OCSP server

[weblinks]

Hi,

My apache down again and again from last 3 hours. while check apache log, I am getting this

[Wed Jan 15 14:24:21.739530 2020] [ssl:error] [pid 131280:tid 47125616011008] (70007)The timeout specified has expired: [client 103.244.174.219:30373] AH01985: error reading response from OCSP server
[Wed Jan 15 14:24:21.739581 2020] [ssl:error] [pid 131280:tid 47125616011008] AH01941: stapling_renew_response: responder error
[Wed Jan 15 14:24:26.696528 2020] [ssl:error] [pid 131281:tid 47125620213504] (70007)The timeout specified has expired: [client 39.57.247.59:55964] AH01985: error reading response from OCSP server
[Wed Jan 15 14:24:26.696584 2020] [ssl:error] [pid 131281:tid 47125620213504] AH01941: stapling_renew_response: responder error
[Wed Jan 15 14:44:45.194211 2020] [ssl:error] [pid 196835:tid 47671576803072] (70007)The timeout specified has expired: [client 66.249.65.151:42099] AH01985: error reading response from OCSP server
[Wed Jan 15 14:44:45.194295 2020] [ssl:error] [pid 196835:tid 47671576803072] AH01941: stapling_renew_response: responder error
[Wed Jan 15 14:44:48.276517 2020] [ssl:error] [pid 196838:tid 47671585208064] (70007)The timeout specified has expired: [client 202.143.122.155:59477] AH01985: error reading response from OCSP server
[Wed Jan 15 14:44:48.276589 2020] [ssl:error] [pid 196838:tid 47671585208064] AH01941: stapling_renew_response: responder error
[Wed Jan 15 14:44:51.386510 2020] [ssl:error] [pid 196837:tid 47671570499328] (70007)The timeout specified has expired: [client 180.178.135.98:61544] AH01985: error reading response from OCSP server
[Wed Jan 15 14:44:51.386571 2020] [ssl:error] [pid 196837:tid 47671570499328] AH01941: stapling_renew_response: responder error

someone pls assist me, what happening

 

[keat63]

I found this thread from May last year. worth reading.

https://forums.cpanel.net/threads/sectigo-ocsp-outage-05-01-2019.652095/

 

[quietFinn]

@weblinks
Apache is not going down because of those errors, it must be something else.

 

[weblinks]

I found this thread from May last year. worth reading.

https://forums.cpanel.net/threads/sectigo-ocsp-outage-05-01-2019.652095/

I go through this thread and adopt temporary solution


To disable OCSP Stapling you can access WHM >> Service Configuration >> Apache Configuration >> Include Editor >> Pre VirtualHost Include >> All Versions and adding the following line:

SSLUseStapling off

but apache still going down again and again.

 

[weblinks]

@weblinks
Apache is not going down because of those errors, it must be something else.

yes, you are right

I found this thread from May last year. worth reading.

SOLVED - Sectigo OCSP Outage 05/01/2019

Is anyone else seeing issues with OCSP from Comodo currently? Getting these errors on all my cPanel servers since around 14:50 UTC today. [Wed May 01 15:45:05.022337 2019] [ssl:error] [pid 32448:tid 47455874840320] AH01941: stapling_renew_response: responder error [Wed May 01 15:45:09.434592...

forums.cpanel.net

I go through this thread and adopt temporary solution


To disable OCSP Stapling you can access WHM >> Service Configuration >> Apache Configuration >> Include Editor >> Pre VirtualHost Include >> All Versions and adding the following line:


SSLUseStapling off


but apache still going down again and again.


In error log now its showing


AH00288: scoreboard is full, not at MaxRequestWorkers


so i set maxrequestworkers from 150 to 600 issue still not resolved.


As I am checking apache status


[[email protected] wlsupport]# apachectl status

Apache Server Status for localhost (via ::1)


Server Version: Apache/2.4.41 (cPanel) OpenSSL/1.0.2t mod_bwlimited/1.4


Server MPM: worker


Server Built: Dec 25 2019 10:27:21


--------------------------------------------------------------------------


Current Time: Wednesday, 15-Jan-2020 23:28:59 PKT


Restart Time: Wednesday, 15-Jan-2020 22:47:12 PKT


Parent Server Config. Generation: 7


Parent Server MPM Generation: 6


Server uptime: 41 minutes 46 seconds


Server load: 0.47 1.18 1.90


Total accesses: 1653654 - Total Traffic: 1.6 GB - Total Duration: 11716000


CPU Usage: u290.51 s71.35 cu3527.48 cs1255.16 - 205% CPU load


660 requests/sec - 0.7 MB/second - 1040 B/request - 7.08492 ms/request


600 requests currently being processed, 0 idle workers


pls assist

 

[cPanelLauren]

Your issue is that you're reaching the limits for the scoreboard.

AH00288: scoreboard is full, not at MaxRequestWorkers

This error clearly states that the MaxRequestWorkers limit has not been reached so modifying that setting will have no value for you.

I'd wager this is either a timeout/keep-alive issue or a synflood attack.

Right below

600 requests currently being processed, 0 idle workers

when running apachectl status it shows the scoreboard. What's listed for the status? L's or G's?

 

[weblinks]

Your issue is that you're reaching the limits for the scoreboard.

AH00288: scoreboard is full, not at MaxRequestWorkers

This error clearly states that the MaxRequestWorkers limit has not been reached so modifying that setting will have no value for you.

I'd wager this is either a timeout/keep-alive issue or a synflood attack.

Right below

600 requests currently being processed, 0 idle workers

when running apachectl status it shows the scoreboard. What's listed for the status? L's or G's?

its shows :


apachectl status
Apache Server Status for localhost (via ::1)

Server Version: Apache/2.4.41 (cPanel) OpenSSL/1.0.2t mod_bwlimited/1.4

Server MPM: worker

Server Built: Dec 25 2019 10:27:21

--------------------------------------------------------------------------

Current Time: Wednesday, 15-Jan-2020 18:40:05 PKT

Restart Time: Wednesday, 15-Jan-2020 18:38:03 PKT

Parent Server Config. Generation: 1

Parent Server MPM Generation: 0

Server uptime: 2 minutes 2 seconds

Server load: 0.74 0.70 0.64

Total accesses: 17504 - Total Traffic: 32.4 MB - Total Duration: 175399

CPU Usage: u6.97 s2.58 cu0 cs0 - 7.83% CPU load

143 requests/sec - 271.6 kB/second - 1938 B/request - 10.0205 ms/request

150 requests currently being processed, 0 idle workers

RKKRRWRKRKKRRRKRRRRRRRRKRRRRKRCKRRRRRKKKKRKRRRRRRRRRKRKRKKRRRKRR
KRKRRRRKRRRRKRRRRRKCRRRRRRRKRRRRRRRKKRRKRRRRKKRCRRRKRKRRRRRKRRRK
KRRKRRKRRRWRRKKWKRCKKR

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

 

[cPanelLauren]

What do you have set for keepalive and KeepAlive timeout in the Apache configuration-> Global Configuration?