Why Do I See "Apache is working on your cPanel® and WHM™ Server"?

[websissy]

I'm not a licensed control panel user. I haven't licensed ANY control panel software yet. cpanel or WHM are not installed on my server. Yet suddenly at 10:30 last night my domains began to disappear one-by-one and gradually got replaced with this default setup screen about Apache and Control Panel.

Mine is a dedicated server. The server is relatively new. I am the server owner and admin. None of my webhosting clients have admin-level access to the server. I am the server's ONLY authorized admin user. I wasn't even looking at admin tasks at the time this first started. I'm running Debian Etch 4.0r3. In fact, what first made me aware of the issue was one of my hosting clients reported the problem.

So, why did one of my hosting clients suddenly report they were getting this cpanel default setup display and why did it ripple out to all the other domains on my server over the next hour or two as I fought to try to stop it? I'm a 40 year computer veteran but still pretty much a novice at server administration at this level.

Can anyone explain to me what causes Apache to display this message and what would it mean when Apache suddenly started displaying it on a server that's not running cpanel OR whm? Where would I go looking for the cause of a problem like this?

Any help you can offer would be greatly appreciated. My server is completely shut down at the moment. Apache has been stopped and all 17 of my domains are turned off and inaccessible while I try to figure out the cause of this problem and how to FIX it. But, I'm completely baffled here!!

Can someone here please offer some advice?

Thanks VERY much!

[hydra]

Hi,

Probably a DNS issue.
Check your domains with a tool like intodns
Most likely your DNS is pointing to your hoster his servers.
I would advice you to contact your dedicated server supplier so they can check it for you.

Ronald.

[websissy]

Thanks a lot for the reply!

For all intents and purposes, I basically AM my own dedicated server supplier. There is noone upstream from me. The supplier who provides my server offers no support beyond the most basic of "Can you log into your server?" and provides no support beyond that.

As I configured it, my server acts as its own DNS host/server with a domain on the server dedicated to that purpose. I run maradns rather than Bind9 because it required less system resources and was supposed to be MUCH easier to set up than Bind9. This has worked fine for 3 months now.

For the record, "dig domainname" reports no issues with any of the domains on my server at the moment. However, following your suggestion, I did go to intodns.com and tried their tool to test two of my 17 domains. The only errors intodns reports at the moment are:

1. Nameservers are lame ERROR: looks like you have lame nameservers. The following nameservers are lame: 219.18.180.89
   The other IP address involved shows no errors. and from what I can tell in my research this is not a fatal error on its own.
2. Different subnets WARNING: Not all of your nameservers are in different subnets
   This is a yellow (cautionary) error
   
   
3. Different autonomous systems WARNING: Single point of failure
   This is a yellow (cautionary) error
   
   
4. SOA MNAME entry WARNING: SOA MNAME (domainname.org) is not listed as a primary nameserver at your parent nameserver!
   This is a yellow (cautionary) error

It reports the same errors for both of the domains I checked.

My research suggests that none of these errors should be deal-killers on their own.

Any other thoughts or suggestions?

[hydra]

Hi,

When you ping one of your domains, do you get your dedicated server IP address returned?
When you do a trace, with domainname instead of ip, do you reach your server?
Has whm previously been installed on your server and then removed?
That could explain the default Apache page with the addition of WHM.
Can you share one of your domain names so we can check?

Ronald.

ps Can you check your apache config if all your virtual domains are still listed.

[websissy]

Hi,
First things first. This server was brand new at the end of July. I'm the one and only admin. So I can say with some certainty that cpanel has never been installed on this server. Neither has WHM. We'd have had to pay for it, right? I KNOW we never did that. I will PM you a couple of domain names and our IP addresses assigned to the server shortly; but I'd rather not post them here when the server is already struggling.

When you ping one of your domains, do you get your dedicated server IP address returned?

Yes. I do always get one or the other of the two IP addresses assigned to our server returned. The server has ONLY two IP addresses and those two IPs are each allocated to one of our two name servers (which reside on the server as well). Yes, I KNOW that's not ideal. But that's how we set it up in the beginning to avoid having to dedicate two more machines to hosting DNS servers. It seemed a reasonable thing to do for a big-fast server that presently hosts just 17 domains.

If we assume our two IPs are 208.18.180.68 and 208.18.180.69 (IPs are fictitious), then ns1.myhost.com is pointed to 208.18.180.68 and ns2.myhost.com is pointed to 208.18.180.69.

However, the IP address I get when I ping varies from 1 request to the next. Since there are two name server with 2 IP addresses, our DNS software "load-balances" inbound domain requests and shares them between our two name servers. By trial and error, I learned I can force the inbound ping to be directed to a particular IP by adding or leaving off the www. at the beginning of the domain name.

When you do a trace, with domainname instead of ip, do you reach your server?

By trace, I assume you mean tracert. Yes. I do. I'm also able to force tracert to deliver one of the IP addresses or the other by adding the www. or leaving it off when I do the trace.

Worth noting here... when I used intodns.com to search test a couple of of our domain names, it consistently reported the second IP is "lame" and doesn't complain about the primary IP. However, once again, my tests show I'm more likely to get the web pages displayed when I'm directed to the second IP address than whn I'm directed to the first one

Has whm previously been installed on your server and then removed?

See my remarks at the beginning of this reply. No. neither cpanel nor whm have ever been installed on this server.

That could explain the default Apache page with the addition of WHM.

Can you share one of your domain names so we can check?

I will PM you the server IP addresses and domain names for 3 of the sites involved

Ronald.

ps Can you check your apache config if all your virtual domains are still listed.

I did double-check the apache config and yes they do all seem to be listed there now. Again, one of my experiments in trying to isolate the cause of the problem had been to disable a few of the 17 sites on the server. That made things a bit better.

I've developed an operating "what's happening" theory here that I can't prove or disprove yet. It says: I think what I'm seeing is the effects of a ddos attack. My hunch is the behavior I'm seeing on the Apache side is caused by an incoming mail flood and possibly a ping overload as well. My anecdotal proofs of that so far are:

1. postfix recorded 40,000 mail errors in 8 hours and during that 8 between 10.m. last night and 6 this morning. The hosted domains began disappearing 1 by 1 around 9:00 our time last night. I was still able to get to a few of them until around 10:45 p.m. After 10:30 I could not get to any of our domains.

2. For all that time I was unable to display the pages of any of the domains hosted on the server at all. However, once I shut postfix down, I was able to get some of the web sites to display occasionally. with better results when the requests were directed to the secondary IP address rather than the first.

3. Even now, I still seem to be unable to get the domains to display consistently by surfing to or or pinging the domain name. It's like 1 moment it will be there and a few seconds later it can't be found. Sometimes the www.domain.com will display for a site in the browser (or the site will be found by ping) and the very next time it's requested the same domain can't be found by ping AND won't appear in the browser. It's maddening. This behavior is being exhibited with the email server NOT running. I'll bet if I bring it back up I won't see the sites at all and ping will begin to fail EVERY time....

4. My 'noobs' theory about the Apache, cpanel and WHM message is when the load is very high the upstream DNS server my be directing the failure request to its local default Apache page rather than to ours and it seems likely that server IS running cpanel and WHM. Could that explain why I'm getting that Apache/cpanel/WHM message or am I totally nuts?

I'm not sure how to PROVE what I'm seeing is the effects of a dos attack, but if it walks, looks, talks, smells and acts like a skunk, I figure it's likely it IS one.

Am I on the right track? Or am I smoking loco weed? Though I'm a 40 year IT veteran, I'm new to the ddos attack phenomena. So, I'm still trying to figure out how to prove that what I'm seeing is caused by an attack and I haven't a figured out yet how to do that or how to fight it either.

Any suggestions you can offer along those lines would be MUCH appreciated and very helpful.

Thanks again for your help. I'll now PM you those IP addresses and domain names to you. I'm extremely interested to hear what you have to say after you've looked for yourself.

[websissy]

Hello, hydra!

I haven't heard back from you (I was wrong about this. You did PM me. I didn't notice until later.) I have concluded this was caused by a DDOS attack. Here is what I know now.

The problem first started to appear around 8- 8:30pm server time on Wednesday. By 7:30pm Thursday night the problem had disappeared and things were back to normal.

Working at first on the assumption that it was caused by a hacker attack and only realizing after several hours that it was actually a DDOS attack, we did what made sense to combat it. Here is my diary of events and actions taken over the 23 hour period of the attack.

• 10pm-12am Wed/Thurs - 2 hours after attack started and an hour before it reached its peak, changed all user passwords on server to 12 character alphanumerics. Average number of attempts per password required to crack these passwords: 1,106,838,475,932,200,000,000 -- just over 1 sextillion. If this was an attack on any of the server's domains or by email our enemies were in for work.
  
• 12am-1am Thur - Conducted a site-by-site security and stability check of all sites on server. All sites appeared stable and secure. Server's domains are still inaccessible due to what appear to be DNS timeouts. Server seems to be VERY busy handling traffic. DNS server is holding its own. Experienced 3 or 4 (puTTY/ssh) server connection losses.
  
• 1am-1:30am - With passwords secure and server sites stable, we shut down the most vulnerable and attackable apps on server -- including guestbooks, mysql, etc. Database passwords to all mysql apps were also changed. If domains can't be accessed, there's no sense inviting trouble! Still seeing 1 or 2 server connection losses per hour with (puTTY/ssh).
  
• 1:30am-2:30am - Posted inquiries and requests to trusted Linux support sites seeking advice on how to analyze, diagnose and solve the problem.
  
• 2:30am - Went to bed exhausted. This was going to take a while!
  
• 6:30am-7:30am - Rebooted server and conducted review of overnight traffic and logs. 40,000 errors had occurred overnight in email system alone due to password failures. The locally-installed mailman listserv app also experienced many errors but no successful security breaches could be identified. All user domains still seem intact and undamaged. This was when we began to believe a DDOS attack was the cause of the problem. Domains on server are still inaccessible -- apparently due to heavy net traffic. Despite the server password changes, attack had not subsided. Other than full log files damage seemed limited. The security barriers were holding up well. Checked web advice requests. No responses yet.
  
• 7:30am-11:30am - Posted a few more web advice requests. Responded to questions, suggestions and advice from hosting clients and web contacts. Conducted web-research on as-yet-unidentified server vulnerabilities, tools and methods to identify, analyze and fight DDOS attacks. Also researched server hardening strategies, techniques, tools and options. The news isn't good. Even the world's top DDOS experts say these attacks are tough to identify, hard to fight and can take many forms. The tools available to fight them are also limited and expensive. Ran a few tests, but made no server changes. There's no sense being caught with our pants down while we're under attack!
  
• 11:30am-12:30pm - Domains on server still inaccessible due to what appears to be heavy traffic. A second error review showed thousands of new errors from email server. These guys weren't giving up easy! Decided to shutdown Apache, email server, mailman listserv and the DNS server.
  
• 1:30pm-3:30pm - Traffic loads continued to fall. Server connection losses are not as frequent. Some local domain home pages do occasionally appear now. However, attack has not completely subsided. Left key server apps shutdown during this period.
  
• 3:30pm-6:30pm - Either the attack mitigation strategies were successful or the attack was timed to last 24 hours. Over these hours the intensity of the attack gradually subsided. As it did, we brought more of the server's apps back online. By 5:00pm, 50% of local domain requests were successful in displaying the domain's home page. Those that were MOST successful were the ones routed to the IP address allocated to the second DNS. A server damage assessment conducted after 18 hours showed no visible user domain or server damage or penetration.
  
• By 8:30pm, roughly 24 hours after the attack began, all domains were working again and the Apache/cpanel/WHM screen had disappeared.

Conclusion: I now believe the Apache/cpanel/WHM screen we were seeing was being displayed by our server supplier's upstream DNS server because our local DNS server was failing to respond fast enough. The fact that we heard nothing from our server supplier during this period suggests ours was not the only server in their datacenter that was under attack...

What say you, hydra? Did I get my situational, strategic and tactical analysis and combat techniques right or did I screw up somewhere?

Thanks again for your comments, thoughts, insights and suggestions.

[dave9000]

Nice job tracking and handling it