SOLVED Server wont boot but data accessable through Rescue Mode

[Nizzo]

Hi,

One of our OVH servers that run cPanel crashed and did not boot into the OS.

I can still access the full data if i boot from OVH Rescue Mode. I have about 250GB of data.

Before i try to repair or attempt anything i like to copy the 250gb to another server (remote or local) before i proceed.

What is the best way to do that? because its 250gb of data i want to make sure it copies in full.

*IF* i copy the data and all attempts to fix the server fails and i have to do a new installation, would i be able to restore the data i backed up to a new cPanel installation?

Thank you in advance.

 

[Eminds]

Try to fix the boot errors from the console that OVH provides , if the errors are not fixed boot the server in rescue mode , mount the drive where the data resides. Try to chroot to the system and try running the cpanel backup script.

 

[cPanelMichael]

Hello @Nizzo,

The following document is a good place to start:

Full Disaster Recovery - cPanel Knowledge Base - cPanel Documentation

documentation.cpanel.net

Let me know if this helps.

Thank you.

 

[Nizzo]

Hello @Nizzo,

The following document is a good place to start:

Full Disaster Recovery - cPanel Knowledge Base - cPanel Documentation

documentation.cpanel.net

Let me know if this helps.

Thank you.

So from what i understand, i have to run 'the pkgacct script', from what i understand also is if i do that i will be able to back up cPanel accounts. Will this be a full backup of the account? including the databases, mail and everything in between?

What about the WHM settings? like accounts info, DNS, IP's, NameServer settings etc...? is there a way to back this up?

 

[Nizzo]

Try to fix the boot errors from the console that OVH provides , if the errors are not fixed boot the server in rescue mode , mount the drive where the data resides. Try to chroot to the system and try running the cpanel backup script.

I think thats what cPanelMichael suggested too.. question remains, can this be used to backup WHM settings etc... or only account backups?

 

[cPanelMichael]

Hello @Nizzo,

Were backups enabled on this server? If so, do recent backups of your accounts exist on a remote backup server, or do recent backups exist on the local cPanel & WHM server (the one that's not booting)?

Thank you.

 

[Nizzo]

Hi @cPanelMichael,

Thanks for the reply.

We do have regular cPanel backups that we took manually from about 2 months ago.

The after many attempts unfortunately the OS failed to boot and I have to reinstall CentOS and cPanel.

I can access the harddrive through Rescue Mode and all the WHM/cPanel data is accessable.

So in the documents you provided i found out on how to take individual backups for the cPanel accounts of the "clients". But what about a backup of the WHM settings, DNS, CNAME's, IP addresses etc.. how can i make a back up of that? and Restore it on a fresh installed server.

 

[cPanelMichael]

So in the documents you provided i found out on how to take individual backups for the cPanel accounts of the "clients". But what about a backup of the WHM settings, DNS, CNAME's, IP addresses etc.. how can i make a back up of that? and Restore it on a fresh installed server.

Hello,

The DNS zone data and cPanel-specific data such as assigned IP address is included in the individual account backup. The service configuration settings are covered under Section 5 (Step 7) of the linked article:

Back up service configurations from the old hard drive.

Let me know if that helps.

Thank you.

 

[Nizzo]

@cPanelMichael,

Thanks. Im testing all this right now and will report back here on how it goes.

Fingers crossed.

 

[Nizzo]

@cPanelMichael,

I have a little problem.

All the users data was in /mnt/chroot/home

When i ran this:

6. Back up accounts from the old hard drive.


Run the following commands to back up the accounts from the old hard drive:

1 cd /var/cpanel/users
2 for i in `ls -1 *`; do /scripts/pkgacct $i; done

This will store the backups in the /mnt/chroot/home directory. This will also store the feature lists and packages settings.

The command runs and then it breaks down when it cant find dnsadminsn running.

After quitting i found out it actually over wrote the "backups" it was trying to created in /mnt/chroot/home so all the user data is gone and not listed there anymore!

Is this normal? It doent look normal but i had to ask

 

[cPanelMichael]

Hello @Nizzo,

Can you share step-by-step the commands you ran on your system? Ensure to replace any identifying information about your server or accounts with examples when sharing the output here.

Additionally, while we're happy to help point you in the right direction, note the warnings on the document linked below:

Full Disaster Recovery - cPanel Knowledge Base - cPanel Documentation

documentation.cpanel.net

• Only experienced system administrators should use the procedures in this document.
• cPanel, L.L.C. cannot support failed hard drives or hard drive recovery. We are not responsible for any data loss.

We provide a list of system administration service providers with SafeAdmin Certification status on the link below should you require more in-depth investigation/assistance:

System Administration Services

System Administration Services Listings Have No Affiliation to cPanel What is SafeAdmin Accreditation? SafeAdmin Certified Service Providers have ensured that a minimum of 90% of their support technicians have successfully completed a...

go.cpanel.net

Thank you.

 

[Nizzo]

@cPanelMichael

Hi,

So back to creating the backups.. I was wrong when i though it overwritten the data and everything still exists.

Following the Full Disaster Recovery, everything goes well except when i try to run a Service like mysql or anything else.

Here is what i get when I try to run a Service:

{chrooted}[[email protected] home]# service mysql start
Redirecting to /bin/systemctl start mysql.service
Running in chroot, ignoring request.
{chrooted}[[email protected] home]#

Any ideas why its doing that?

 

[cPanelMichael]

Following the Full Disaster Recovery, everything goes well except when i try to run a Service like mysql or anything else.

Hello @Nizzo,

Can you confirm that you are at Step 5 of Section 5 at Full Disaster Recovery - cPanel Knowledge Base - cPanel Documentation ? If so, please post the output from the command below:

/scripts/restartsrv_mysql

Ensure to remove the references to your server's hostname when pasting the output here.

Thank you.

 

[Nizzo]

Hello @Nizzo,

Can you confirm that you are at Step 5 of Section 5 at Full Disaster Recovery - cPanel Knowledge Base - cPanel Documentation ? If so, please post the output from the command below:

/scripts/restartsrv_mysql

Ensure to remove the references to your server's hostname when pasting the output here.

Thank you.

@cPanelMichael

Thanks for the reply.

Yes, i can confirm im in Step 5 of Section 5 of that Knowledge Base

Here is the result of what you requested:

{chrooted}[[email protected] home]# /scripts/restartsrv_mysql
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
warn [restartsrv_mysql] The 'mysql' service's PID file '/var/lib/mysql/rescue.ovh.net.pid' did not appear after 10 seconds.

Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Waiting for “mysql” to start ………Running in chroot, ignoring request.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
warn [restartsrv_mysql] The 'mysql' service's PID file '/var/lib/mysql/rescue.ovh.net.pid' did not appear after 10 seconds.

Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
…Running in chroot, ignoring request.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 132.
warn [restartsrv_mysql] The 'mysql' service's PID file '/var/lib/mysql/rescue.ovh.net.pid' did not appear after 10 seconds.

…failed.

Cpanel::Exception::Services::StartError
Service Status

Service Error
        (XID aqehz5) The “mysql” service failed to start.

Use of uninitialized value in string eq at /usr/local/cpanel/Cpanel/RestartSrv/Systemd.pm line 111.
Running in chroot, ignoring request.
mysql has failed. Contact your system administrator if the service does not automagically recover.
{chrooted}[[email protected] home]#

 

[cPanelMichael]

Hello @Nizzo,

Can you submit a support ticket so we can take a closer look? You can post the ticket number here once it's opened and I'll link this thread to it.

Note that while we can take a general look at the affected server to determine if there's an obvious fix for the error in your last post, failed hard drives and hard drive recovery falls outside the scope of technical support.

Thank you.

 

[Nizzo]

@cPanelMichael

Thanks, ill do that.

And the HDD's did not fail actually, booting the OS is whats failing, the data is still intact.

If the Services can be started then we can take the backups we need and reinstall the OS from scratch.

Thanks again

 

[Nizzo]

@cPanelMichael

Your support request ID: 13356717

Thanks.

 

[cPanelMichael]

booting the OS is whats failing

Can you confirm if you tried any of the workarounds posted on the thread below?

SOLVED - Kernel conflicts on OVH servers due to missing exclude line in /etc/yum.repos.d/OVH-kernel.repo

Hello everyone, OVH support is horrible, I had the same problem on four servers with CentOS 7. They replied (2 days later) that even being the OVH kernel, they do not support distributions and cannot help since "it's a unmanaged server"... OVH is very good, stable and good prices, but have the...

forums.cpanel.net

Thank you.

 

[Nizzo]

Can you confirm if you tried any of the workarounds posted on the thread below?

SOLVED - Kernel conflicts on OVH servers due to missing exclude line in /etc/yum.repos.d/OVH-kernel.repo

Hello everyone, OVH support is horrible, I had the same problem on four servers with CentOS 7. They replied (2 days later) that even being the OVH kernel, they do not support distributions and cannot help since "it's a unmanaged server"... OVH is very good, stable and good prices, but have the...

forums.cpanel.net

Thank you.

@cPanelMichael

I did not try these steps specifically, but I was going to try some other method that involves OVH Netbook but the server does not respond when i do that and both times it had to be rebooted to Rescue Mode by OVH.

If i test and have to force to reboot to another Rescue Mode the root command will change and the guys working on the Support Ticket wont be able to log in because the root passwd would change.

 

[cPanelMichael]

@cPanelMichael

I did not try these steps specifically, but I was going to try some other method that involves OVH Netbook but the server does not respond when i do that and both times it had to be rebooted to Rescue Mode by OVH.

If i test and have to force to reboot to another Rescue Mode the root command will change and the guys working on the Support Ticket wont be able to log in because the root passwd would change.

You can reply to the ticket to let us know you're performing the test and we'll hold off on the investigation until we have confirmation from you that your test is complete.

Thank you.