SOLVED Wordpress sending empty requests and killing Apache

Hi, our wordpress server has been choking itself seemingly randomly for the last 4 or 5 months now. After a long process of troubleshooting various things I finally pinpointed it to 4-6 requests that show up in apache. They're completely empty and they come from our own wordpress server IP. No VHost specified, no request path either.


These requests get stuck in Reading Request, and any subsequent legitimate requests get stuck in Sending Reply until the scoreboard is full. The site effectively goes down until we restart FPM and Apache.

I "fixed" this by banning our server's IP from making requests via htaccess. Our apache now stays up but we get error logs for access denied to 400.shtml (bad request). This works, but of course I really want to figure out what is causing these requests.

I've checked our cron jobs and removed all of the ones from old plugins but the requests still come in anywhere from every hour to every 3 hours. This doesn't match any of the remaining cron frequencies.

Any help is greatly appreciated, I'm at wits end.

Screenshot attached.

 

Anyone have any ideas? One thing I noticed is I was getting errors in nginx log before, trivial stuff, but since I banned our own IP the error log has remained empty.

 

I have the Easy Updates Manager to block the updates of one particular plugin, but all other updates are allowed.

I don't think this fits what I've described though, because the update would be outbound from my server to wordpress' servers. What I'm seeing is a request from my IP to my IP, and it has no actual request data. (see screenshot)

 

I actually figured this out, multiple bots were scanning the site, and occasionally hitting URLs that included "#" at the end of the URL. Instead of clipping this off, they were including it in their GET request, which either nginx or apache can't handle.

I'm guessing the reason apache ended up with our own IP is because of the reverse proxy and the bad request.

I've banned their IPs and banned any request with "#" in it (since requests shouldn't include it) and that seems to have stopped it.