Faster cPanel Backup by replacing gzip with pigz [Case 55535]

[webmastergreg]

+1 really impressive
Thanks for the tests

can't wait for this...

[InterServed]

+1 from me also. Would love to see cPanel taking advantage of this on their backup/archiving functions

[Jay M]

This looks mighty impressive, definitely a +1 for me.

[crazyaboutlinux]

4* star from be to this feature

[Kent Brockman]

Very good! +1 for this.

Some questions for cPanel staff:

As pigz performs better with multi core processors. What happens in those old servers with one or two cores? Is this script able to do its magic too?

Also, as pigz does not come by default with common OS installations, what considerations should be taken in account? or cPanel would just install it as "third party" software? Then cPanel can offer pigz as a Tweak Setting or in the Configure Backup screen.

[Compubuster]

Kent,

As pigz performs better with multi core processors. What happens in those old servers with one or two cores? Is this script able to do its magic too?

pigz on uni-core processors will perform the same as gzip. Also, pigz isn't a script, it's a full in-place replacement to the gzip compression program/binary.

Also, as pigz does not come by default with common OS installations, what considerations should be taken in account? or cPanel would just install it as "third party" software? Then cPanel can offer pigz as a Tweak Setting or in the Configure Backup screen.

This is totally left for cPanel developers, when they approve the idea.

For all the other supporters, thank you guys

[Kent Brockman]

Great, would be an interesting add to cPanel.

[eva2000]

yeah pigz and pbzip2 are what i have incorporated in a backup script i am writing mysqlmybackup.sh

pbzip2 is the multi-threaded version of bzip2 Parallel BZIP2 (PBZIP2)

[ikillbill]

impressive!

looks like it is easy to replace gzip? could we have this ASAP?
current backup gzip always overload server....

[Compubuster]

current backup gzip always overload server....

Actually, pigz will raise the load, as it utilizes more cores. That's why we suggested a configuration option for the number of processors/cores flag.

On the other hand, pigz will tremendously reduce the time taken for backups, thus reducing the "load" time.

[eva2000]

guys remember to also take into account cpu and memory usage as pigz can use up to 6-7x times more memory than gzip

using my gziptest.sh script on 605,530 KB sized mysql sql backup file for

gzip vs bzip2 vs pigz vs pbzip2 vs lzip (forgot plzip but will add that to my script soon)

compression levesl 1 to 4 + decompression test on level 4

note, ran on virtualbox instance so timer might not be that accurate

• CentOS 5.7 64bit VirtualBOX
• Xeon W3540 @3.5Ghz - 4 core allocation
• 2GB allocated
• 20GB EXT3 VDI disk

./gziptest.sh leveltest
------------------------------------------------
gzip -1 /root/sqlbackup/vb387.sql
real: 12.46 seconds cpu: 75% max mem: 2688KB
compression level: 1
compress: 24% 147230/605530 KB
------------------------------------------------
gzip -2 /root/sqlbackup/vb387.sql
real: 10.64 seconds cpu: 92% max mem: 2720KB
compression level: 2
compress: 23% 142356/605530 KB
------------------------------------------------
gzip -3 /root/sqlbackup/vb387.sql
real: 14.42 seconds cpu: 76% max mem: 2704KB
compression level: 3
compress: 22% 138441/605530 KB
------------------------------------------------
gzip -4 /root/sqlbackup/vb387.sql
real: 15.95 seconds cpu: 79% max mem: 2672KB
compression level: 4
compress: 21% 127571/605530 KB
------------------------------------------------
gzip -d /root/sqlbackup/vb387.sql.gz
real: 4.42 seconds cpu: 99% max mem: 2240KB
------------------------------------------------
bzip2 -1 /root/sqlbackup/vb387.sql
real: 84.71 seconds cpu: 97% max mem: 6384KB
compression level: 1
compress: 17% 106378/605530 KB
------------------------------------------------
bzip2 -2 /root/sqlbackup/vb387.sql
real: 95.93 seconds cpu: 92% max mem: 9552KB
compression level: 2
compress: 15% 92240/605530 KB
------------------------------------------------
bzip2 -3 /root/sqlbackup/vb387.sql
real: 101.55 seconds cpu: 92% max mem: 12672KB
compression level: 3
compress: 14% 85639/605530 KB
------------------------------------------------
bzip2 -4 /root/sqlbackup/vb387.sql
real: 105.26 seconds cpu: 94% max mem: 15792KB
compression level: 4
compress: 13% 81459/605530 KB
------------------------------------------------
bzip2 -d /root/sqlbackup/vb387.sql.bz2
real: 17.92 seconds cpu: 99% max mem: 8448KB
------------------------------------------------
pigz -1 /root/sqlbackup/vb387.sql
real: 9.24 seconds cpu: 126% max mem: 14080KB
compression level: 1
compress: 24% 146626/605530 KB
------------------------------------------------
pigz -2 /root/sqlbackup/vb387.sql
real: 9.96 seconds cpu: 126% max mem: 14352KB
compression level: 2
compress: 23% 141627/605530 KB
------------------------------------------------
pigz -3 /root/sqlbackup/vb387.sql
real: 9.21 seconds cpu: 149% max mem: 14256KB
compression level: 3
compress: 22% 137511/605530 KB
------------------------------------------------
pigz -4 /root/sqlbackup/vb387.sql
real: 8.43 seconds cpu: 201% max mem: 14096KB
compression level: 4
compress: 21% 127267/605530 KB
------------------------------------------------
pigz -d /root/sqlbackup/vb387.sql.gz
real: 2.70 seconds cpu: 139% max mem: 2960KB
------------------------------------------------
pbzip2 -1 /root/sqlbackup/vb387.sql
real: 22.75 seconds cpu: 387% max mem: 98592KB
compression level: 1
compress: 17% 106356/605530 KB
------------------------------------------------
pbzip2 -2 /root/sqlbackup/vb387.sql
real: 24.71 seconds cpu: 388% max mem: 89680KB
compression level: 2
compress: 15% 93768/605530 KB
------------------------------------------------
pbzip2 -3 /root/sqlbackup/vb387.sql
real: 28.90 seconds cpu: 363% max mem: 107568KB
compression level: 3
compress: 14% 85683/605530 KB
------------------------------------------------
pbzip2 -4 /root/sqlbackup/vb387.sql
real: 28.76 seconds cpu: 370% max mem: 133744KB
compression level: 4
compress: 13% 84088/605530 KB
------------------------------------------------
pbzip2 -d /root/sqlbackup/vb387.sql.bz2
real: 6.26 seconds cpu: 307% max mem: 455072KB
------------------------------------------------
lzip -1 /root/sqlbackup/vb387.sql
real: 58.16 seconds cpu: 90% max mem: 67456KB
compression level: 1
compress: 9% 57413/605530 KB
------------------------------------------------
lzip -2 /root/sqlbackup/vb387.sql
real: 66.53 seconds cpu: 95% max mem: 87952KB
compression level: 2
compress: 8% 50561/605530 KB
------------------------------------------------
lzip -3 /root/sqlbackup/vb387.sql
real: 84.68 seconds cpu: 95% max mem: 108416KB
compression level: 3
compress: 7% 45938/605530 KB
------------------------------------------------
lzip -4 /root/sqlbackup/vb387.sql
real: 116.57 seconds cpu: 97% max mem: 149392KB
compression level: 4
compress: 6% 42204/605530 KB
------------------------------------------------
lzip -d /root/sqlbackup/vb387.sql.lz
real: 7.26 seconds cpu: 84% max mem: 16384KB
------------------------------------------------

[Infopro]

Thanks for posting those results George, very important Info.

[eva2000]

Cleaned up my gziptest.sh script and added plzip as well parallel version of lzip

For speed = pigz
For compression = lzip
For speedier compression = plzip (but huge memory usage)

gzip vs bzip2 vs pigz vs pbzip2 vs lzip vs plzip

compression levels 1 to 3

./gziptest.sh leveltest
------------------------------------------------------------
gziptest.sh 0.07 by George Liu (eva2000)
http://vbtechsupport.com
------------------------------------------------------------

------------------------------------------------
gzip -1 /root/sqlbackup/vb387.sql
real: 22.19 seconds cpu: 43% max mem: 2688KB
compression level: 1 compress: 24% 147230/605530 KB
------------------------------------------------
gzip -2 /root/sqlbackup/vb387.sql
real: 12.99 seconds cpu: 76% max mem: 2704KB
compression level: 2 compress: 23% 142356/605530 KB
------------------------------------------------
gzip -3 /root/sqlbackup/vb387.sql
real: 14.45 seconds cpu: 76% max mem: 2688KB
compression level: 3 compress: 22% 138441/605530 KB
------------------------------------------------
gzip -d /root/sqlbackup/vb387.sql.gz
real: 4.39 seconds cpu: 99% max mem: 2240KB
------------------------------------------------
bzip2 -1 /root/sqlbackup/vb387.sql
real: 84.90 seconds cpu: 97% max mem: 6400KB
compression level: 1 compress: 17% 106378/605530 KB
------------------------------------------------
bzip2 -2 /root/sqlbackup/vb387.sql
real: 93.05 seconds cpu: 95% max mem: 9552KB
compression level: 2 compress: 15% 92240/605530 KB
------------------------------------------------
bzip2 -3 /root/sqlbackup/vb387.sql
real: 102.50 seconds cpu: 92% max mem: 12656KB
compression level: 3 compress: 14% 85639/605530 KB
------------------------------------------------
bzip2 -d /root/sqlbackup/vb387.sql.bz2
real: 17.78 seconds cpu: 99% max mem: 6880KB
------------------------------------------------
pigz -1 /root/sqlbackup/vb387.sql
real: 13.87 seconds cpu: 85% max mem: 14320KB
compression level: 1 compress: 24% 146626/605530 KB
------------------------------------------------
pigz -2 /root/sqlbackup/vb387.sql
real: 6.27 seconds cpu: 197% max mem: 14064KB
compression level: 2 compress: 23% 141627/605530 KB
------------------------------------------------
pigz -3 /root/sqlbackup/vb387.sql
real: 6.39 seconds cpu: 211% max mem: 14256KB
compression level: 3 compress: 22% 137511/605530 KB
------------------------------------------------
pigz -d /root/sqlbackup/vb387.sql.gz
real: 3.02 seconds cpu: 127% max mem: 2976KB
------------------------------------------------
pbzip2 -1 /root/sqlbackup/vb387.sql
real: 24.12 seconds cpu: 363% max mem: 93456KB
compression level: 1 compress: 17% 106356/605530 KB
------------------------------------------------
pbzip2 -2 /root/sqlbackup/vb387.sql
real: 24.64 seconds cpu: 387% max mem: 96064KB
compression level: 2 compress: 15% 93768/605530 KB
------------------------------------------------
pbzip2 -3 /root/sqlbackup/vb387.sql
real: 27.84 seconds cpu: 369% max mem: 107472KB
compression level: 3 compress: 14% 85683/605530 KB
------------------------------------------------
pbzip2 -d /root/sqlbackup/vb387.sql.bz2
real: 6.77 seconds cpu: 278% max mem: 443888KB
------------------------------------------------
lzip -1 /root/sqlbackup/vb387.sql
real: 55.51 seconds cpu: 94% max mem: 67456KB
compression level: 1 compress: 9% 57413/605530 KB
------------------------------------------------
lzip -2 /root/sqlbackup/vb387.sql
real: 66.81 seconds cpu: 95% max mem: 87952KB
compression level: 2 compress: 8% 50561/605530 KB
------------------------------------------------
lzip -3 /root/sqlbackup/vb387.sql
real: 84.24 seconds cpu: 95% max mem: 108432KB
compression level: 3 compress: 7% 45938/605530 KB
------------------------------------------------
lzip -d /root/sqlbackup/vb387.sql.lz
real: 6.25 seconds cpu: 99% max mem: 12272KB
------------------------------------------------
plzip -1 /root/sqlbackup/vb387.sql
real: 16.15 seconds cpu: 337% max mem: 496992KB
compression level: 1 compress: 11% 66885/605530 KB
------------------------------------------------
plzip -2 /root/sqlbackup/vb387.sql
real: 18.16 seconds cpu: 360% max mem: 593776KB
compression level: 2 compress: 9% 58445/605530 KB
------------------------------------------------
plzip -3 /root/sqlbackup/vb387.sql
real: 22.64 seconds cpu: 375% max mem: 712720KB
compression level: 3 compress: 8% 52533/605530 KB
------------------------------------------------
plzip -d /root/sqlbackup/vb387.sql.lz
real: 2.06 seconds cpu: 369% max mem: 405920KB

gzip looks dated

compression levels 1 to 9

[root@centos57a ~]# ./gziptest.sh levelgzip
------------------------------------------------------------
gziptest.sh 0.07 by George Liu (eva2000)
http://vbtechsupport.com
------------------------------------------------------------

------------------------------------------------
gzip -1 /root/sqlbackup/vb387.sql
real: 15.47 seconds cpu: 60% max mem: 2704KB
compression level: 1 compress: 24% 147230/605530 KB
------------------------------------------------
gzip -2 /root/sqlbackup/vb387.sql
real: 13.06 seconds cpu: 75% max mem: 2704KB
compression level: 2 compress: 23% 142356/605530 KB
------------------------------------------------
gzip -3 /root/sqlbackup/vb387.sql
real: 12.71 seconds cpu: 85% max mem: 2704KB
compression level: 3 compress: 22% 138441/605530 KB
------------------------------------------------
gzip -4 /root/sqlbackup/vb387.sql
real: 15.52 seconds cpu: 81% max mem: 2688KB
compression level: 4 compress: 21% 127571/605530 KB
------------------------------------------------
gzip -5 /root/sqlbackup/vb387.sql
real: 19.54 seconds cpu: 80% max mem: 2656KB
compression level: 5 compress: 20% 122217/605530 KB
------------------------------------------------
gzip -6 /root/sqlbackup/vb387.sql
real: 23.54 seconds cpu: 85% max mem: 2624KB
compression level: 6 compress: 19% 120392/605530 KB
------------------------------------------------
gzip -7 /root/sqlbackup/vb387.sql
real: 32.24 seconds cpu: 72% max mem: 2640KB
compression level: 7 compress: 19% 119340/605530 KB
------------------------------------------------
gzip -8 /root/sqlbackup/vb387.sql
real: 43.68 seconds cpu: 96% max mem: 2624KB
compression level: 8 compress: 19% 118054/605530 KB
------------------------------------------------
gzip -9 /root/sqlbackup/vb387.sql
real: 53.93 seconds cpu: 93% max mem: 2640KB
compression level: 9 compress: 19% 117995/605530 KB
------------------------------------------------
gzip -d /root/sqlbackup/vb387.sql.gz
real: 4.19 seconds cpu: 99% max mem: 2256KB

[eva2000]

Parallel Compression tests

• compression level = 1 to 3
• number threads tests = 1 to 4 (Xeon W3540 4 core allocated would be 2 physical + 2 virtual)

pigz
vs
pbzip2
vs
plzip

[root@centos57a ~]# ./gziptest.sh threadtest
------------------------------------------------------------
gziptest.sh 0.07 by George Liu (eva2000)
http://vbtechsupport.com
------------------------------------------------------------

------------------------------------------------
pigz -p1 -1 /root/sqlbackup/vb387.sql
real: 26.13 seconds cpu: 36% max mem: 4560KB
number of threads: 1
compression level: 1 compress: 24% 147451/605530 KB
------------------------------------------------
pigz -p1 -2 /root/sqlbackup/vb387.sql
real: 12.94 seconds cpu: 81% max mem: 4544KB
number of threads: 1
compression level: 2 compress: 23% 142545/605530 KB
------------------------------------------------
pigz -p1 -3 /root/sqlbackup/vb387.sql
real: 14.36 seconds cpu: 81% max mem: 4560KB
number of threads: 1
compression level: 3 compress: 22% 138618/605530 KB
------------------------------------------------
pigz -p2 -1 /root/sqlbackup/vb387.sql
real: 8.60 seconds cpu: 131% max mem: 9120KB
number of threads: 2
compression level: 1 compress: 24% 146626/605530 KB
------------------------------------------------
pigz -p2 -2 /root/sqlbackup/vb387.sql
real: 9.55 seconds cpu: 126% max mem: 8880KB
number of threads: 2
compression level: 2 compress: 23% 141627/605530 KB
------------------------------------------------
pigz -p2 -3 /root/sqlbackup/vb387.sql
real: 8.35 seconds cpu: 161% max mem: 9120KB
number of threads: 2
compression level: 3 compress: 22% 137511/605530 KB
------------------------------------------------
pigz -p3 -1 /root/sqlbackup/vb387.sql
real: 7.03 seconds cpu: 164% max mem: 11520KB
number of threads: 3
compression level: 1 compress: 24% 146626/605530 KB
------------------------------------------------
pigz -p3 -2 /root/sqlbackup/vb387.sql
real: 8.85 seconds cpu: 136% max mem: 11696KB
number of threads: 3
compression level: 2 compress: 23% 141627/605530 KB
------------------------------------------------
pigz -p3 -3 /root/sqlbackup/vb387.sql
real: 4.99 seconds cpu: 268% max mem: 11456KB
number of threads: 3
compression level: 3 compress: 22% 137511/605530 KB
------------------------------------------------
pigz -p4 -1 /root/sqlbackup/vb387.sql
real: 7.43 seconds cpu: 157% max mem: 14368KB
number of threads: 4
compression level: 1 compress: 24% 146626/605530 KB
------------------------------------------------
pigz -p4 -2 /root/sqlbackup/vb387.sql
real: 4.18 seconds cpu: 294% max mem: 14224KB
number of threads: 4
compression level: 2 compress: 23% 141627/605530 KB
------------------------------------------------
pigz -p4 -3 /root/sqlbackup/vb387.sql
real: 5.50 seconds cpu: 244% max mem: 14176KB
number of threads: 4
compression level: 3 compress: 22% 137511/605530 KB
------------------------------------------------
pbzip2 -p1 -1 /root/sqlbackup/vb387.sql
real: 85.18 seconds cpu: 96% max mem: 12208KB
number of threads: 1
compression level: 1 compress: 17% 106356/605530 KB
------------------------------------------------
pbzip2 -p1 -2 /root/sqlbackup/vb387.sql
real: 93.46 seconds cpu: 94% max mem: 15200KB
number of threads: 1
compression level: 2 compress: 15% 93768/605530 KB
------------------------------------------------
pbzip2 -p1 -3 /root/sqlbackup/vb387.sql
real: 97.09 seconds cpu: 97% max mem: 18864KB
number of threads: 1
compression level: 3 compress: 14% 85683/605530 KB
------------------------------------------------
pbzip2 -p2 -1 /root/sqlbackup/vb387.sql
real: 42.14 seconds cpu: 198% max mem: 52320KB
number of threads: 2
compression level: 1 compress: 17% 106356/605530 KB
------------------------------------------------
pbzip2 -p2 -2 /root/sqlbackup/vb387.sql
real: 45.47 seconds cpu: 198% max mem: 51952KB
number of threads: 2
compression level: 2 compress: 15% 93768/605530 KB
------------------------------------------------
pbzip2 -p2 -3 /root/sqlbackup/vb387.sql
real: 49.28 seconds cpu: 197% max mem: 75472KB
number of threads: 2
compression level: 3 compress: 14% 85683/605530 KB
------------------------------------------------
pbzip2 -p3 -1 /root/sqlbackup/vb387.sql
real: 29.35 seconds cpu: 294% max mem: 70608KB
number of threads: 3
compression level: 1 compress: 17% 106356/605530 KB
------------------------------------------------
pbzip2 -p3 -2 /root/sqlbackup/vb387.sql
real: 31.53 seconds cpu: 295% max mem: 96016KB
number of threads: 3
compression level: 2 compress: 15% 93768/605530 KB
------------------------------------------------
pbzip2 -p3 -3 /root/sqlbackup/vb387.sql
real: 34.73 seconds cpu: 295% max mem: 90944KB
number of threads: 3
compression level: 3 compress: 14% 85683/605530 KB
------------------------------------------------
pbzip2 -p4 -1 /root/sqlbackup/vb387.sql
real: 24.95 seconds cpu: 357% max mem: 74960KB
number of threads: 4
compression level: 1 compress: 17% 106356/605530 KB
------------------------------------------------
pbzip2 -p4 -2 /root/sqlbackup/vb387.sql
real: 25.24 seconds cpu: 381% max mem: 116864KB
number of threads: 4
compression level: 2 compress: 15% 93768/605530 KB
------------------------------------------------
pbzip2 -p4 -3 /root/sqlbackup/vb387.sql
real: 28.83 seconds cpu: 359% max mem: 144128KB
number of threads: 4
compression level: 3 compress: 14% 85683/605530 KB
------------------------------------------------
plzip -n1 -1 /root/sqlbackup/vb387.sql
real: 50.21 seconds cpu: 98% max mem: 110336KB
number of threads: 1
compression level: 1 compress: 11% 66885/605530 KB
------------------------------------------------
plzip -n1 -2 /root/sqlbackup/vb387.sql
real: 59.29 seconds cpu: 99% max mem: 153088KB
number of threads: 1
compression level: 2 compress: 9% 58445/605530 KB
------------------------------------------------
plzip -n1 -3 /root/sqlbackup/vb387.sql
real: 80.05 seconds cpu: 95% max mem: 258976KB
number of threads: 1
compression level: 3 compress: 8% 52533/605530 KB
------------------------------------------------
plzip -n2 -1 /root/sqlbackup/vb387.sql
real: 26.04 seconds cpu: 196% max mem: 281728KB
number of threads: 2
compression level: 1 compress: 11% 66885/605530 KB
------------------------------------------------
plzip -n2 -2 /root/sqlbackup/vb387.sql
real: 30.82 seconds cpu: 196% max mem: 314704KB
number of threads: 2
compression level: 2 compress: 9% 58445/605530 KB
------------------------------------------------
plzip -n2 -3 /root/sqlbackup/vb387.sql
real: 40.22 seconds cpu: 194% max mem: 405808KB
number of threads: 2
compression level: 3 compress: 8% 52533/605530 KB
------------------------------------------------
plzip -n3 -1 /root/sqlbackup/vb387.sql
real: 19.49 seconds cpu: 291% max mem: 332720KB
number of threads: 3
compression level: 1 compress: 11% 66885/605530 KB
------------------------------------------------
plzip -n3 -2 /root/sqlbackup/vb387.sql
real: 22.74 seconds cpu: 290% max mem: 508352KB
number of threads: 3
compression level: 2 compress: 9% 58445/605530 KB
------------------------------------------------
plzip -n3 -3 /root/sqlbackup/vb387.sql
real: 29.47 seconds cpu: 288% max mem: 558928KB
number of threads: 3
compression level: 3 compress: 8% 52533/605530 KB
------------------------------------------------
plzip -n4 -1 /root/sqlbackup/vb387.sql
real: 17.92 seconds cpu: 333% max mem: 508352KB
number of threads: 4
compression level: 1 compress: 11% 66885/605530 KB
------------------------------------------------
plzip -n4 -2 /root/sqlbackup/vb387.sql
real: 18.37 seconds cpu: 366% max mem: 546304KB
number of threads: 4
compression level: 2 compress: 9% 58445/605530 KB
------------------------------------------------
plzip -n4 -3 /root/sqlbackup/vb387.sql
real: 28.80 seconds cpu: 311% max mem: 683472KB
number of threads: 4
compression level: 3 compress: 8% 52533/605530 KB

[ikillbill]

could anyone try to use pigz to zip 5G and 10G files please?