Unble to Restore Database from R1soft Backup

[uk01]

Hi all,

I'm just putting this out there as I"'ve not found an answer anywhere else and R1soft have not offered any solution. Also the license provider have seen the issue before but have no idea how to resolve.

There will be Cpanel hosts who also offer R1soft backups. Have any of you started experiencing the errors below recently?
Our problem started a few weeks ago out of the blue, it's been fine for ages, then suddenly we can't restore databases. Whatever we do, the restore fails.

Logs below.
Maybe others looking for a solution will find this thread and we'll get some input.

Logs

----------------------------------------
Message Time         Level    Source               Message
5/22/19 3:47:28 AM    Info     Manager    Attempting to connect to agent ********** at port ****
5/22/19 3:47:28 AM    Info     Manager    Connected to agent *********** at port **** successfully
5/22/19 3:47:29 AM    Info     Protected Machine    Agent Version 6.10.0.14
5/22/19 3:47:29 AM    Info     Protected Machine    Connection authenticated; Waiting for command.
5/22/19 3:47:29 AM    Info     Protected Machine    Executing get_mysql_instance_configuration_request request...
5/22/19 3:47:29 AM    Info     Manager    Restoring files to Agent required for selected databases
5/22/19 3:47:30 AM    Info     Manager    Calculating restore size
5/22/19 3:47:30 AM    Info     Manager    Calculation complete
5/22/19 3:47:30 AM    Info     Manager    Calculated 4.1 KB to restore (2 files)
5/22/19 3:47:30 AM    Info     Manager    Calculating restore size
5/22/19 3:47:30 AM    Info     Manager    Calculation complete
5/22/19 3:47:30 AM    Info     Manager    Calculated 76.0 MB to restore (1 files)
5/22/19 3:47:30 AM    Info     Manager    Calculating restore size
5/22/19 3:47:30 AM    Error    Manager    Failed to read filesystem for /var/lib/mysql/./ib_buffer_pool
5/22/19 3:47:30 AM    Error    Manager    Path not found ([var, lib, mysql, ., ib_buffer_pool])
5/22/19 3:47:30 AM    Error    Manager    Failed to read filesystem for /var/lib/mysql/./mysql.ibd
5/22/19 3:47:30 AM    Error    Manager    Path not found ([var, lib, mysql, ., mysql.ibd])
5/22/19 3:47:30 AM    Info     Manager    Calculation complete
5/22/19 3:47:30 AM    Info     Manager    Calculated 96.0 MB to restore (2 files)
5/22/19 3:47:30 AM    Warn     Manager    Errors encountered during calculations
5/22/19 3:47:30 AM    Warn     Manager    Estimated size and file count may not be accurate
5/22/19 3:47:32 AM    Info     Manager    Restoring MySQL database files to '/var/lib/r1soft/tmp/r1soft-mysql-restore-tmp-********************'
5/22/19 3:47:32 AM    Info     Protected Machine    Command completed; Waiting for next command.
5/22/19 3:47:32 AM    Info     Protected Machine    Executing file restore request...
5/22/19 3:47:33 AM    Error    Manager    Errors encountered during file restore
5/22/19 3:48:38 AM    Info     Manager    File restore complete
5/22/19 3:48:38 AM    Info     Manager    Setting up temporary MySQL instance
5/22/19 3:48:38 AM    Info     Protected Machine    Command completed; Waiting for next command.
5/22/19 3:48:38 AM    Info     Protected Machine    Executing mysql_start_restore request...
5/22/19 3:48:43 AM    Info     Protected Machine    Secondary MySQL instance process terminated
5/22/19 3:48:45 AM    Warn     Protected Machine    Secondary MySQL instance is not shutdown; you must do so manually!
5/22/19 3:48:45 AM    Error    Protected Machine    An exception occurred during the request. Unable to start the secondary MySQL instance.
5/22/19 3:48:45 AM    Error    Manager    Failed to start temporary database instance
5/22/19 3:48:45 AM    Error    Manager    Agent reported error during requested operation
5/22/19 3:48:45 AM    Error    Manager    Failed to start temporary database instance
5/22/19 3:48:46 AM    Error    Manager    Failed to start temporary database instance
5/22/19 3:48:46 AM    Error    Manager    Agent reported error during requested operation
5/22/19 3:48:46 AM    Info     Manager    Task Finished

 

[GOT]

Yes actually, there is a problem with their latest agent. Their current advice is to roll back the agent version.

This is what they told us:

There are issues w/ the 6.10 agent that prevents the temp instance from starting up. Before proceeding with the db restore, please downgrade the agent to 6.8.2 release.
The steps for downgrade are:

1. To remove the 6.10 agent packages from the server. The package manager can do this.. i.e.

rpm -qa | grep serverbackup (remove all the packages that are returned)
yum remove serverbackup-enterprise-agent

1. Afterwards, delete/backup the /usr/sbin/r1soft/* directory and all subdirs.

rm -rf /usr/sbin/r1soft/*

1. Reinstall the agent.. you can edit the repo to point to the 6.8.2 repository.. (link provided)

i.e.
change /etc/yum.repos.d/r1soft.repo
set baseurl=Index of /release/6.8.2/55/yum/stable$basearch

Afterwards -

yum clean all
yum install serverbackup-enterprise-agent
After reinstall, please attempt a db restore with the innodb_force_recovery option at your convenience.

instructions for adding force recovery

1) Perform a restore like normal
2) Look for the section called "Additional Options" on the page "Temporary Instance"
3) Under Options enter : innodb_force_recovery
4) under Value enter : 1
5) press the "Add Another" button to save the option

 

[uk01]

Fantastic, thanks for your reply. I will check this out!

 

[cPanelLauren]

Thanks @GOT for the information!

 

[mlweber]

Rolling back did not work for me. I have been having this issue since migrating to Centos 7. The iWeb Data Center System Administrators have been working with R1Soft for a solution, and I am told that R1Soft claims that the issue is with Centos 7 and mySQL 5.7 compatibility, and that they supposedly need to write a script to make the database restore function correctly.

 

[uk01]

Admittably we've not actually tried the roll back yet due to workload. However, we've been on Centos 7 for ages and never had any issues so it's not directly related to that. We use MariaDB 10.1 and it worked before the latest updates (either r1 or otherwise, it just stopped one day!).
R1 need to get their act together and resolve it rather than look for excuses otherwise people are just going to start looking for better back up solutions, after all R1 is very slow restoring even on a gig connection.

 

[uk01]

I would say that the agents etc tend to relate to the kernel version, so there may well be a relation between the latest centos 7 kernel and agent but still, if they are going to charge for software, it's their responsibility to resolve these issues quickly, the problem is R1 take forever to do everything.

 

[GOT]

We did not actually try the roll back either, so I cannot confirm it works either.

R1 has been really disappointing. We rolled out an ubuntu 18 server which ahs been out a year and had to use a beta driver for it...

On my agenda is looking for alternate options. Suggestions welcomed.

 

[wintech2003]

We have switched most of our customers to Acronis Backup Cloud since the beginning of the year and are quite happy.
As for the above solution @GOT suggested, I can confirm it's working on two CloudLinux 6 servers that were experiencing this issue. Haven't tried with it with CentOS/CloudLinux 7 though.

 

[gvard]

Hello,

A temporary workaround is in the additional options (during restore) to set innodb_force_recovery to 1, if that fails try setting it to 2 and then to 3 (in a specific case it wanted "3", however in other case it wanted "2"). The database can be restored this way, you may get a warning message during restore but if you check it gets restored.

They are still working for a permanent solution, this is a temporary workaround. Unfortunately restores have to be made via the admin (not via the cPanel plugin as they can't enter additional restore options there).

 

[puneet_1234]

I upgraded to the 6.10.2 version of the Agent and the Maria Db restores were working fine. Faced issues while restoring MySQL 5.7 with cPanels. So resorted to file restores for the specific MySQL dbs and was good to go.