SOLVED CPANEL-40507 - WHM/CPANEL Broke two sites on my server (that I know of) out of the blue today.

[RickKukiela]

Two sites had their document root paths crossed up out of the blue in the middle of the work day. No one on my team was even working on anything on the server. Apparently WHM did something to the HTTPD config that caused the issue.

The issue is that for both sites we have the following setup:

MainDomain -> MainDocRoot
AlternateDomain -> AlternateDocRoot

HTTP requests started being served instead as:

MainDomain -> AlternateDocRoot
AlternateDomain -> AlternateDocRoot

On one of the sites, AlternateDocRoot just had a simple .htaccess file with a redirect back to MainDomain.com and that caused that site to go down in a fire death of "too many redirects" for all of its visitors. The other site was just showing a "landing page" instead of their actual website due to the wrong document root.

What both sites have in common is that they both have their original document root folder "public_html" replace with a symlink to a project sub-directory, eg:
~/public_html -> ~/project/web

This configuration is NOT unique to these sites though. I have at least 5 other sites that use the same set up and did not have issues come up.

My managed server team over at liquid web is looking into this to see if there is anything in the cpanel/whm logs that can indicate to us what happened and why. I will report back when I get more information from them.

We were able to resolve the problem rebuilding the HTTPD conf file. However I am VERY concerned that this happened. I cant be having WHM/CPANEL just mangling site configs out of the blue.

 

[cPRex]

Hey there! Let's try and keep each thread limited to just one issue to avoid confusion. Can you let me know how the Apache customizations were implemented on the machine? If there are any direct edits made to the Apache configuration that aren't done through the include system, cPanel will overwrite those changes. It doesn't sound like this was the case since you mention a rebuild resolved the problem, but it would still be good to clarify.

Do you have a backup copy or a diff of the two Apache configuration files on the machine so we can see exactly what changed there?

 

[RickKukiela]

I move the 2nd issue to its own thread.

There is nothing custom configured on this server's apache config outside of the scope of the EasyApache system except for an ImageMagick Module for PHP but that's not really related directly to this.

Unfortunately the tech I was working with from the server company did not make a copy of the httpd.conf file before rebuilding it and since we fixed the issue within 30 minutes of it happening, the overnight back up would not have the "corrupted" version either Unless you know of a secret location that all httpd.conf file previous versions are stored during the rebuild process I think we are out of luck there.

So far the log search has not yielded anything interesting. I was provided these two entries, and the "corruption" would have happened some time between them:

[2022-04-01 12:15:02 -0500] info [autorepair] Successfully verified signature for cpanel (key types: release).
[2022-04-01 12:39:49 -0500] info [whostmgrd] Successfully verified signature for cpanel (key types: release).

Is there a specific log file that would contain information from a botched process that would involve rebuilding the apache config?

 

[mtindor]

I had the same issue yesterday. Only affected one site (to my knowledge) on the whole server. May have occurred at about the time AutoSSL was doing some things. The DNS zone had been touched at about the same time (which also led me to believe it may have been during some autossl stuff).

It was a domain (somedomain.com) that also had a subdomain associated with it (zixvpm.somedomain.com). After the site broke, if I went to www.somedomain.com it would end up redirecting to www.somedomain.com/zixvpm/zixvpm . I executed /scripts/rebuildhttpdconf and restarted Apache and all was well.

I don't have copies of the old (broken) vs new (fixed) httpd.conf. Was in too much of a hurry to get it fixed, just in case other sites may have been broken but not reported yet.

 

[cPRex]

Our team has confirmed the issue over the weekend, but we don't know for sure exactly what is causing this just yet:

https://support.cpanel.net/hc/en-us/articles/5041423360663

If you have a server where you're experiencing the issue and can manage to leave it in a broken state for a bit so our team can check it out, feel free to submit a ticket so we can examine that system.

 

[RickKukiela]

I'm glad you guys were able to confirm the issue. Hoping for a swift resolution and patch.

 

[RickKukiela]

Any word on progress with this issue? So far the issue has happened twice for me. Once last Friday and again on Monday. It's been OK since then but until I get confirmation this has been resolved I'm going to be nervous of this happening again.

 

[cPRex]

I don't have any interesting updates yet, other than our team is still working on the issue.

 

[Jose A. Garcia]

Hello,
On Saturday I lost 4 more main vhosts at httpd.conf
We have not discovered until this morning. It is a critical bug.

I think it is related to AutoSSL. I got an email notify about a reduced SSL coverage for a host... at exact same date time than my httpd.conf was last modified.

Any recent update?

 

[cPRex]

At this point we are still working to identify the root cause. I'll be sure to post an update once I have one.

 

[Babene7]

I had two reports of this today (and I'm expecting more in the coming days), at least it's easy to solve, let's hope the root cause is found and fixed soon.

 

[cagsmith]

I also had two reports of this. In one case the primary domain was loading a recently created staging site. That took a while to diagnose since I hadn't heard of this issue yet but finally found that rebuilding apache configuration and restarting the web server resolved it.

Second report there was no subdomain, so the domain was just returning 404s for every request but again, rebuilding apache config resolved it.

I don't believe it's related to AutoSSL because that runs during the night, and in the second case, it seemed to suddenly happen around 1500 UTC, but what do I know.

Don't believe it's related to EasyApache4 either, at least not the update from the 6th of April (EasyApache 4 Release Notes | cPanel & WHM Documentation), since one server was running the update, and one was not.

If it happens again, which I'm really hoping it does not because it's pretty annoying both for us and customers, I'll try and grab a copy of the httpd.conf file before and after the rebuild if that would be useful.

 

[JoseDieguez]

these past few days we have seen this error at least 5 times (5 clients reported it on different servers).

Opened a ticket, and the command in the article does fix the issue, but it just keeps happening..

 

[cenourinha]

We have been seeing this problem since the update to cPanel v102 and every day we need to rebuild rebuild httpd.conf and restart apache several times to solve this. You need to address and fix this bug urgently!

 

[cagsmith]

We have been seeing this problem since the update to cPanel v102 and every day we need to rebuild rebuild httpd.conf and restart apache several times to solve this. You need to address and fix this bug urgently!

It happened for us on a machine still running v100 unfortunately, so I don't think it's specific to that version.

Wish we could get some more information about it though

 

[Jamira40]

Hey, more than 8 days since reported it to support and no patch. Daily we are getting more and more reports with same issue. We need patch for this right now

 

[cPRex]

I wish I had a patch to give! At this point, our QA team is testing a possible fix caused by a race condition. I'm not sure if it will be approved, but if it does that would be today or over the weekend, and then it would get into a build next week most likely.

I'm sorry this hasn't been quicker, but this has been a tricky one to pin down.

 

[JoseDieguez]

still happening, today still. CPANEL-40348
websites randomly dissapearing and showing other website.

 

[cPRex]

As of this morning I see this is being tested in Edge with a backport request to version 102 if that testing goes well. I'll be sure to post an update once I have one.

 

[cPRex]

UPDATE - this is scheduled to be included in the 102.0.12 build when that is released.