SOLVED CPANEL-40507 - WHM/CPANEL Broke two sites on my server (that I know of) out of the blue today.

RickKukiela

Active Member
Jan 17, 2017
25
15
53
Chicago
cPanel Access Level
Root Administrator
Two sites had their document root paths crossed up out of the blue in the middle of the work day. No one on my team was even working on anything on the server. Apparently WHM did something to the HTTPD config that caused the issue.

The issue is that for both sites we have the following setup:

MainDomain -> MainDocRoot
AlternateDomain -> AlternateDocRoot

HTTP requests started being served instead as:

MainDomain -> AlternateDocRoot
AlternateDomain -> AlternateDocRoot

On one of the sites, AlternateDocRoot just had a simple .htaccess file with a redirect back to MainDomain.com and that caused that site to go down in a fire death of "too many redirects" for all of its visitors. The other site was just showing a "landing page" instead of their actual website due to the wrong document root.

What both sites have in common is that they both have their original document root folder "public_html" replace with a symlink to a project sub-directory, eg:
~/public_html -> ~/project/web

This configuration is NOT unique to these sites though. I have at least 5 other sites that use the same set up and did not have issues come up.

My managed server team over at liquid web is looking into this to see if there is anything in the cpanel/whm logs that can indicate to us what happened and why. I will report back when I get more information from them.

We were able to resolve the problem rebuilding the HTTPD conf file. However I am VERY concerned that this happened. I cant be having WHM/CPANEL just mangling site configs out of the blue.
 
Last edited:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,499
1,971
363
cPanel Access Level
Root Administrator
Hey there! Let's try and keep each thread limited to just one issue to avoid confusion. Can you let me know how the Apache customizations were implemented on the machine? If there are any direct edits made to the Apache configuration that aren't done through the include system, cPanel will overwrite those changes. It doesn't sound like this was the case since you mention a rebuild resolved the problem, but it would still be good to clarify.

Do you have a backup copy or a diff of the two Apache configuration files on the machine so we can see exactly what changed there?
 

RickKukiela

Active Member
Jan 17, 2017
25
15
53
Chicago
cPanel Access Level
Root Administrator
I move the 2nd issue to its own thread.

There is nothing custom configured on this server's apache config outside of the scope of the EasyApache system except for an ImageMagick Module for PHP but that's not really related directly to this.

Unfortunately the tech I was working with from the server company did not make a copy of the httpd.conf file before rebuilding it and since we fixed the issue within 30 minutes of it happening, the overnight back up would not have the "corrupted" version either :( Unless you know of a secret location that all httpd.conf file previous versions are stored during the rebuild process I think we are out of luck there.

So far the log search has not yielded anything interesting. I was provided these two entries, and the "corruption" would have happened some time between them:

[2022-04-01 12:15:02 -0500] info [autorepair] Successfully verified signature for cpanel (key types: release).
[2022-04-01 12:39:49 -0500] info [whostmgrd] Successfully verified signature for cpanel (key types: release).

Is there a specific log file that would contain information from a botched process that would involve rebuilding the apache config?
 

mtindor

Well-Known Member
Sep 14, 2004
1,456
112
193
inside a catfish
cPanel Access Level
Root Administrator
I had the same issue yesterday. Only affected one site (to my knowledge) on the whole server. May have occurred at about the time AutoSSL was doing some things. The DNS zone had been touched at about the same time (which also led me to believe it may have been during some autossl stuff).

It was a domain (somedomain.com) that also had a subdomain associated with it (zixvpm.somedomain.com). After the site broke, if I went to www.somedomain.com it would end up redirecting to www.somedomain.com/zixvpm/zixvpm . I executed /scripts/rebuildhttpdconf and restarted Apache and all was well.

I don't have copies of the old (broken) vs new (fixed) httpd.conf. Was in too much of a hurry to get it fixed, just in case other sites may have been broken but not reported yet.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,499
1,971
363
cPanel Access Level
Root Administrator
Our team has confirmed the issue over the weekend, but we don't know for sure exactly what is causing this just yet:


If you have a server where you're experiencing the issue and can manage to leave it in a broken state for a bit so our team can check it out, feel free to submit a ticket so we can examine that system.
 

RickKukiela

Active Member
Jan 17, 2017
25
15
53
Chicago
cPanel Access Level
Root Administrator
Any word on progress with this issue? So far the issue has happened twice for me. Once last Friday and again on Monday. It's been OK since then but until I get confirmation this has been resolved I'm going to be nervous of this happening again.
 

Jose A. Garcia

Registered
Apr 11, 2022
1
0
1
Spain
cPanel Access Level
Root Administrator
Hello,
On Saturday I lost 4 more main vhosts at httpd.conf
We have not discovered until this morning. It is a critical bug.

I think it is related to AutoSSL. I got an email notify about a reduced SSL coverage for a host... at exact same date time than my httpd.conf was last modified.

Any recent update?
 
Last edited:

Babene7

Active Member
Apr 13, 2017
41
9
58
Earth
cPanel Access Level
Root Administrator
I had two reports of this today (and I'm expecting more in the coming days), at least it's easy to solve, let's hope the root cause is found and fixed soon.
 

cagsmith

Member
Jul 21, 2021
12
8
3
Sweden
cPanel Access Level
Root Administrator
I also had two reports of this. In one case the primary domain was loading a recently created staging site. That took a while to diagnose since I hadn't heard of this issue yet but finally found that rebuilding apache configuration and restarting the web server resolved it.

Second report there was no subdomain, so the domain was just returning 404s for every request but again, rebuilding apache config resolved it.

I don't believe it's related to AutoSSL because that runs during the night, and in the second case, it seemed to suddenly happen around 1500 UTC, but what do I know.

Don't believe it's related to EasyApache4 either, at least not the update from the 6th of April (EasyApache 4 Release Notes | cPanel & WHM Documentation), since one server was running the update, and one was not.

If it happens again, which I'm really hoping it does not because it's pretty annoying both for us and customers, I'll try and grab a copy of the httpd.conf file before and after the rebuild if that would be useful.
 
  • Like
Reactions: cPRex

JoseDieguez

Well-Known Member
PartnerNOC
Jan 26, 2016
57
31
68
Chile
cPanel Access Level
Root Administrator
these past few days we have seen this error at least 5 times (5 clients reported it on different servers).

Opened a ticket, and the command in the article does fix the issue, but it just keeps happening..
 

cenourinha

Well-Known Member
PartnerNOC
Jan 8, 2011
50
2
58
Portugal
cPanel Access Level
DataCenter Provider
Twitter
We have been seeing this problem since the update to cPanel v102 and every day we need to rebuild rebuild httpd.conf and restart apache several times to solve this. You need to address and fix this bug urgently!
 

cagsmith

Member
Jul 21, 2021
12
8
3
Sweden
cPanel Access Level
Root Administrator
We have been seeing this problem since the update to cPanel v102 and every day we need to rebuild rebuild httpd.conf and restart apache several times to solve this. You need to address and fix this bug urgently!
It happened for us on a machine still running v100 unfortunately, so I don't think it's specific to that version.

Wish we could get some more information about it though :(
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
12,499
1,971
363
cPanel Access Level
Root Administrator
I wish I had a patch to give! At this point, our QA team is testing a possible fix caused by a race condition. I'm not sure if it will be approved, but if it does that would be today or over the weekend, and then it would get into a build next week most likely.

I'm sorry this hasn't been quicker, but this has been a tricky one to pin down.