Wordpress update hung up from all morning

[GoWilkes]

My server started throwing errors about a high server load and services failing all morning today. the server load went up over 30!

The only thing I can find in any of the logs is in /var/log/secure, where I have an error related to Wordpress over and over. The site isn't letting me post the error lines, I'll try to submit it in a second post.

I'm not sure if a Wordpress update was causing the load spike, or if the load spike was causing the update to fail.

The errors began at 10:53am, and end at 12:54pm. So I'm guessing that the WP update was set to time out after 2 hours, then all of the problems stopped.

Any thoughts or suggestions? I thought that all of my updates would run in the middle of the night, anyway, not at 11am!

 

[GoWilkes]

Sorry, the site keeps throwing an error, even when I just try to paste the first line of the error message. Clicking the "CODE" button opens up the entire forum list, too, so something's up with the site.

 

[cPRex]

Hey hey! We'd need some more details in order to provide you with anything useful. /var/log/secure is typically related to the SSH access on the server, so it would be interesting to see the lines that you're including.

I've pasted some code to the Forums today and haven't seen an issue. Let's do another test:

here's a random code block

Nope, that looks good as well.

 

[GoWilkes]

I still can't get it to paste, I sent you a PM, Rex. But here's a screenshot

 

[cPRex]

Do those repeat the whole two hours?

 

[GoWilkes]

Quick follow up, /var/log/messages has a few errors from this morning showing:

rsyslogd: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': No space left on device [v8.24.0-57.el7_9.1 try You searched for error 2013 - rsyslog ]

A few days ago my server did run out of storage, but I deleted about 30G of unnecessary files and now it's at 82%. Coincidence, or is there something somewhere that could still think that the server is full?

 

[GoWilkes]

Do those repeat the whole two hours?

Yes, the same block of errors over and over for 2 hours.

 

[cPRex]

The server's journal should not have any cached data. Could you open a ticket so we can check the machine directly? If there's something going on with the WPT update, we'd like to see that.

 

[GoWilkes]

Will do. Everything seems OK for now so it's not an emergency, but I'll submit a ticket later this evening.

I've also had an issue with mysqldump timing out in a weird way *, so it could be related.


* mysqldump --single-transaction --quick --lock-tables=false database_name | gzip > /backup/database_name.sql.gz

The original /var/mysql/ files are cumulatively about 15G, and the completed backup should be around 2G.

The last several times I've run it, though, SSH never shows that it's done and eventually just times out.

But it LOOKS to be done.

The last successful backup was a few weeks ago. It took 16 minutes to complete, and the GZIP file was was 2,037,494,108 bytes.

Last night, I began the backup at 12:54am. At 1:07am I had a file at /backup/database_name.sql.gz that was 2,051,999,839, so it appeared to be a valid backup. But mysqldump was still running in SSH, until it eventually timed out at 1:28am, saying:

Network error: Software caused connection abort

 

[cPRex]

Are you manually making that backup, or is the server using a remote MySQL machine?

 

[GoWilkes]

That one was just a manual command on my end. I have a separate shared server that I use to store backups remotely, too, but this particular mysqldump was manual.

 

[cPRex]

Thanks for the clarification - that error was actually the SSH session timing out, not the mysqldump itself.