Used /scripts/securetmp, now /usr is 100%

Citizen

Active Member

Sep 3, 2012

#1

So, I'm moving from my old server to my new server using WHM's multi account copy. Everything moved over except one account, which has a huge database and it was filling /tmp to 100%.

So, I followed this guide:

http://forums.cpanel.net/f5/drive-critical-dev-loop0-var-tmp-97-full-190772.html#post782021

And attempted to increase the /tmp space to 4gb on the old server. This didn't work and now my drives look like this:

/dev/sda2 /var 45% (4,367,568 of 10,154,020)
/usr/tmpDSK /tmp 16% (1,079 of 7,272)
/dev/sda5 / 7% (512,992 of 8,123,168)
/dev/sda8 /home 24% (46,782,644 of 206,970,004)
/tmp /var/tmp 16% (1,079 of 7,272)
/dev/sda1 /boot 14% (16,490 of 124,427)
/dev/sda3 /usr 100% (7,871,400 of 8,123,200)

du -h /usr | grep ^[2-9][0-9][0-9][0-9.]*M shows this:

331M /usr/bin
201M /usr/local/cpanel/bin
454M /usr/local/cpanel/src/3rdparty/gpl
608M /usr/local/cpanel/src/3rdparty
614M /usr/local/cpanel/src
340M /usr/local/cpanel/logs
373M /usr/local/cpanel/base
577M /usr/local/apache/logs
618M /usr/local/apache
229M /usr/share/doc
983M /usr/share
697M /usr/lib

Really at a loss here at what to do. Rerunning securetmp doesn't work, in fact each time I run it the /tmp directory gets smaller.

Any ideas?

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#2

Rather than resizing /tmp, why not simply set MySQL tmpdir to another location with more space? That's also mentioned in the same thread in this post:

http://forums.cpanel.net/f5/drive-critical-dev-loop0-var-tmp-97-full-190772.html#post799162

C

Citizen

Active Member

Sep 3, 2012

#3

Wow. Ok I feel stupid now. Yes I'll do that.

But how do I fix my tmp directories? (edit: to clarify, it appears my /tmp directory is 15kb as detailed above, just making sure you saw that) I'm thinking that /usr partittion didn't have room for it, so I've cleared my logs and now my directories look like this:

du -sk *| sort -nr

2124028 local
1005720 share
713248 lib
338296 bin
103800 sbin
100864 src
28260 include
24116 i686-pc-linux-gnu
21932 libexec
7524 tmpDSK
2104 info
1656 kerberos
536 man
76 doc
20 quota.user
20 etc
16 X11R6
16 lost+found
16 aquota.user
8 games
0 tmp

du -h /usr | grep ^[1-9][0-9][0-9][0-9.]*M

331M /usr/bin
201M /usr/local/cpanel/bin
121M /usr/local/cpanel/src/3rdparty/other
454M /usr/local/cpanel/src/3rdparty/gpl
608M /usr/local/cpanel/src/3rdparty
614M /usr/local/cpanel/src
119M /usr/local/cpanel/base/horde
150M /usr/local/cpanel/base/3rdparty
373M /usr/local/cpanel/base
140M /usr/local/cpanel/3rdparty
169M /usr/local/cpanel/whostmgr/bin
192M /usr/local/cpanel/whostmgr
180M /usr/local/apache/logs/archive
180M /usr/local/apache/logs
221M /usr/local/apache
171M /usr/share/locale
229M /usr/share/doc
120M /usr/share/mysql-test
983M /usr/share
102M /usr/sbin
124M /usr/lib/perl5
121M /usr/lib/locale
697M /usr/lib

Last edited: Sep 3, 2012

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#4

Can we have a df -kh or df -h on the machine? The output is far easier for me to read than what I'm seeing.

C

Citizen

Active Member

Sep 3, 2012

#5

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 7.8G 501M 6.9G 7% /
/dev/sda8 198G 45G 143G 24% /home
/dev/sda3 7.8G 6.9G 505M 94% /usr
/dev/sda2 9.7G 4.2G 5.1G 46% /var
/dev/sda1 122M 17M 100M 14% /boot
none 3.0G 0 3.0G 0% /dev/shm
/usr/tmpDSK 7.2M 1.1M 5.7M 16% /tmp

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#6

This new server only has 7.8GB for /usr partition? That seems a bit of an issue, since it's against our recommended partition sizes guide for installation:

Advanced Options: Pre-Installation

The very minimum is 8GB for /usr in the guide. This machine should not be used until it has the 16GB recommended for /usr partition. /var partition also needs to be 16GB, especially if you have an account with a large database. The machine is new and will shortly have issues with space constraints because it wasn't setup properly for partition sizes.

As for /tmp, what do you have in your /scripts/securetmp file for the size? Can you post it so we can ensure you aren't actually putting the wrong number for resizing it:

Code:

grep tmpdsksize /scripts/securetmp

C

Citizen

Active Member

Sep 3, 2012

#7

This is the old server, not the new one. The new one is running fine. The old one was crashing when packaging the last account (which is big) when moving to the new server.

my $tmpdsksize = 1048576; # Must be larger than 250000
if ( $one_percent_of_available > $tmpdsksize ) {
$tmpdsksize = $one_percent_of_available;
if ( $tmpdsksize > $FOUR_GIG_k ) {
$tmpdsksize = $FOUR_GIG_k;
$tmpdsksize = int($tmpdsksize);
$tmpdsksize = $tmpdsksize - ( $tmpdsksize % 1024 );
my $tmpdsksize_megs = ( $tmpdsksize / 1024 );
print "/tmp calculated to be $tmpdsksize_megs M based on available disk space in /usr\n";
elsif ( ( -s '/usr/tmpDSK' ) < ( $tmpdsksize * 1024 ) ) {
system( "dd", "if=/dev/zero", "of=/usr/tmpDSK", "bs=1024", "count=${tmpdsksize}" );

Btw, thank you so much for your help on this, I'm lost without you

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#8

Could you change the file to have the amount recommended in the prior post, which is 2097152 instead?

Also, if it's the old server, I'm uncertain why we care about getting /tmp re-sized at this point, since changing MySQL's tmpdir should work fine.

C

Citizen

Active Member

Sep 3, 2012

#9

cPanelTristan said:
Could you change the file to have the amount recommended in the prior post, which is 2097152 instead?

Ok I just ran it but now my /usr drive is full again

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 7.8G 501M 6.9G 7% /
/dev/sda8 198G 45G 143G 24% /home
/dev/sda3 7.8G 7.8G 0 100% /usr
/dev/sda2 9.7G 4.3G 5.0G 46% /var
/dev/sda1 122M 17M 100M 14% /boot
none 3.0G 0 3.0G 0% /dev/shm
/usr/tmpDSK 882M 17M 821M 3% /tmp

du -h /usr | grep ^[1-9][0-9][0-9][0-9.]*M

331M /usr/bin
201M /usr/local/cpanel/bin
121M /usr/local/cpanel/src/3rdparty/other
454M /usr/local/cpanel/src/3rdparty/gpl
608M /usr/local/cpanel/src/3rdparty
614M /usr/local/cpanel/src
119M /usr/local/cpanel/base/horde
150M /usr/local/cpanel/base/3rdparty
373M /usr/local/cpanel/base
140M /usr/local/cpanel/3rdparty
169M /usr/local/cpanel/whostmgr/bin
191M /usr/local/cpanel/whostmgr
180M /usr/local/apache/logs/archive
180M /usr/local/apache/logs
221M /usr/local/apache
171M /usr/share/locale
229M /usr/share/doc
120M /usr/share/mysql-test
983M /usr/share
102M /usr/sbin
124M /usr/lib/perl5
121M /usr/lib/locale
697M /usr/lib

Also, if it's the old server, I'm uncertain why we care about getting /tmp re-sized at this point, since changing MySQL's tmpdir should work fine.

Right that's exactly what I'm going to do since you recommended it, but I tried this first and it jacked up my /tmp directories. All I need to do is get this working enough that I can run WHM's copy. Are my tmp drives good to go now? Did they even get set up properly by securetmp? They have funny names now and tmp is only 800m, not 2g like securetmp is set to. Should I just go ahead now with the my.cnf change and see if I can migrate the site? Really lost here.

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#10

It doesn't have enough space to resize it appears. You could clear /usr like the domlogs could be moved or removed, what size is /usr/local/apache/domlogs?

Otherwise, just change it back to the default after clearing some space in /usr and then please move MySQL's tmpdir to another location.

C

Citizen

Active Member

Sep 3, 2012

#11

I cleared all the logs, here's my directory size report:

331M /usr/bin
201M /usr/local/cpanel/bin
121M /usr/local/cpanel/src/3rdparty/other
454M /usr/local/cpanel/src/3rdparty/gpl
608M /usr/local/cpanel/src/3rdparty
614M /usr/local/cpanel/src
119M /usr/local/cpanel/base/horde
150M /usr/local/cpanel/base/3rdparty
373M /usr/local/cpanel/base
140M /usr/local/cpanel/3rdparty
169M /usr/local/cpanel/whostmgr/bin
190M /usr/local/cpanel/whostmgr
171M /usr/share/locale
229M /usr/share/doc
120M /usr/share/mysql-test
983M /usr/share
102M /usr/sbin
124M /usr/lib/perl5
121M /usr/lib/locale
697M /usr/lib

Still at 100%. Anything else I can remove from here?

Any ideas?

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#12

You could move /usr/local/cpanel/src contents to another location. Those are source files that will be redownloaded whenever cPanel needs to do that. Since this is an old server, you don't need the files to exist short-term.

Put /tmp back to 500MB after you've moved /usr/local/cpanel/src and re-run /scripts/securetmp again with the 500MB set in it (again, refer to the prior post where the original size is indicated).

C

Citizen

Active Member

Sep 3, 2012

#13

Ok I moved src and reran it at 512000 and here's the results:

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 7.8G 501M 6.9G 7% /
/dev/sda8 198G 46G 142G 25% /home
/dev/sda3 7.8G 7.8G 0 100% /usr
/dev/sda2 9.7G 4.3G 5.0G 46% /var
/dev/sda1 122M 17M 100M 14% /boot
none 3.0G 0 3.0G 0% /dev/shm

What should I do?

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#14

Where did you move /usr/local/cpanel/src directory? It doesn't seem to show as being moved, since it was over 1GB large and yet that doesn't show as available now.

Also, can you restart Apache on the machine. Sometimes, it caches data that's been cleared, although I don't see how that would apply here.

C

Citizen

Active Member

Sep 3, 2012

#15

It's definitley gone. I moved it to /home/cpaneltmp.

du -h /usr | grep ^[1-9][0-9][0-9][0-9.]*M

331M /usr/bin
201M /usr/local/cpanel/bin
119M /usr/local/cpanel/base/horde
150M /usr/local/cpanel/base/3rdparty
373M /usr/local/cpanel/base
140M /usr/local/cpanel/3rdparty
169M /usr/local/cpanel/whostmgr/bin
190M /usr/local/cpanel/whostmgr
171M /usr/share/locale
229M /usr/share/doc
120M /usr/share/mysql-test
983M /usr/share
102M /usr/sbin
124M /usr/lib/perl5
121M /usr/lib/locale
697M /usr/lib

When I moved it, it did clear up a ton of space in /usr, but as soon as I ran securetmp, it filled right back up again to full.

What should I do?

C

Citizen

Active Member

Sep 3, 2012

#16

Also, not to mix things up too much here but I changed the mysql tmpdir and ran the whm copy account and I'm getting the exact same problem, so I'm not sure if this tmp folder stuff is even the problem.

redacted

Any ideas?

Last edited: Sep 3, 2012

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#17

What is the issue you are having at this point? It didn't fail from what I can see. The only errors are that some tables couldn't be checked because of the table engine type ("note : The storage engine for the table doesn't support repair"), which indicates they are InnoDB.

The account is going to take awhile to package if it has a large database or databases.

C

Citizen

Active Member

Sep 3, 2012

#18

cPanelTristan said:
What is the issue you are having at this point? It didn't fail from what I can see. The only errors are that some tables couldn't be checked because of the table engine type ("note : The storage engine for the table doesn't support repair"), which indicates they are InnoDB.

The account is going to take awhile to package if it has a large database or databases.

Last time it stalled and then crashed. I'll give it time, hopefully it will work.

cPanelTristan

Quality Assurance Analyst

Staff member

Sep 3, 2012

#19

If it doesn't work, please submit a ticket to ask us to manually package it for you due to the issues you are having with getting it moved.

C

Citizen

Active Member

Sep 3, 2012

#20

It worked, thank you for your help

Thread starter	Similar threads	Forum	Replies	Date
H	Telnet connection refused.	File Management	7	Jul 12, 2021
M	ProFtpd refused connection	File Management	1	Mar 26, 2021
	Disabling mod_mpm_itk caused lots of permissions problems	File Management	18	Dec 13, 2020
P	/tmp and /bar/tmp disk used warning	File Management	1	May 26, 2020
L	cPanel and WHM showing 0 MB Disk used.	File Management	1	Oct 3, 2019