Drive Critical: /dev/loop0 (/var/tmp) is 97% full

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
38
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Did you try running the script a second time after receiving that initial failure?
 

ronald.prism

Member
Dec 26, 2011
5
0
51
cPanel Access Level
Website Owner
Did you try running the script a second time after receiving that initial failure?
Yes. Doing so outputs the same notice -


*** Notice *** No loop module detected
If the loopback block device is built as a module, try running `modprobe loop` as root via ssh and running this script again.
If the loopback block device is built into the kernel itself, you can ignore this message.
*** Notice *** No working loopback device files found. Try running `modprobe loop` as root via ssh and running this script again.

Nothing else changes.


Thanks
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
38
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Is your system Virtuozzo or OpenVZ? If so, then /scripts/securetmp will not work on those systems due to not having a loopback device on them.
 

ronald.prism

Member
Dec 26, 2011
5
0
51
cPanel Access Level
Website Owner
Is your system Virtuozzo or OpenVZ? If so, then /scripts/securetmp will not work on those systems due to not having a loopback device on them.
It's a straight-up dedicated server.

I did an -

# /sbin/lsmod | grep loop
loop 18633 2

but /scripts/securetmp terminates on that notice regardless.

Looking at the securetmp script, both checks below fail to detect the loop device.

Code:
    if ( !grep /loop/, @modules ) {
        print "*** Notice *** No loop module detected\n";    # could be built into kernel, so don't bail out yet
        print "If the loopback block device is built as a module, try running `modprobe loop` as root via ssh and running this script again.\n";
        print "If the loopback block device is built into the kernel itself, you can ignore this message.\n";
    }
    if ( !$has_loop_device ) {
        print "*** Notice *** No working loopback device files found. Try running `modprobe loop` as root via ssh and running this script again$
How risky would it be to run the script with those checks disabled?
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
38
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
I wouldn't try running it without the checks. Could you please submit a ticket to us about this issue? You can use WHM > Support Center > Contact cPanel or use the link in my signature.
 

DreamPhysix

Well-Known Member
Sep 30, 2009
78
0
56
Today I noticed that my tmp folder /usr/tmpDSK is relatively high today. In crontab I have 0 * * * * /usr/sbin/tmpwatch --mtime --all 24 /tmp so I figured I could run /usr/sbin/tmpwatch --mtime --all 24 /tmp in cli to clean up the partition, but the disk usage didn't change. Does this mean that tmpwatch is not working properly? No output was given when issuing the "/usr/sbin/tmpwatch --mtime --all 24 /tmp" command
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,203
363
No output was given when issuing the "/usr/sbin/tmpwatch --mtime --all 24 /tmp" command
You can use the verbose flag when running the command manually. EX:

Code:
/usr/sbin/tmpwatch --verbose --mtime --all 24 /tmp
This will let you know which files and directories are removed. You can try decreasing the "mtime" value if you want to remove newer files and directories.

Thank you.
 

TapanB

Active Member
Nov 29, 2004
40
0
156
Hello,

What size is your /tmp partition? You can resize it if it isn't large enough. Many servers have only 500MB for /tmp which isn't large enough and should be around 2GB instead. You could edit /scripts/securetmp for this line:

Code:
my $tmpdsksize     = 512000;    # Must be larger than 250000
Change to this line for 2GB:

Code:
my $tmpdsksize     = 2097152;    # Must be larger than 250000
Save the file :)wq), then run these commands to resize /tmp partition:

Code:
umount -l /tmp
umount -l /var/tmp
rm -fv /usr/tmpDSK
/scripts/securetmp
Unfortunately, if you aren't able to catch it when it has the high usage, it's difficult for us to state why it's alerting. It could be high session activity (those sessions do clear on their own) for some site during the times it hits the high disk space level. It could be MySQL having the temporary MyISAM tables filling it up. It could be EAccelerator writing files to it (if you use that, it writes some temporary files there). There are many reasons it could be happening. If you aren't certain, you can simply resize it to be larger. Otherwise, you'll have to watch it closely until you do catch it and then provide the details on what you are seeing.

There are also some scripts that will periodically clear tmp such as tmpwatch, but you need to be careful not to cron clear /tmp as EasyApache writes to /tmp during build, so you cannot simply clear files every 15 minutes or something, since you'll then break EasyApache from recompiling properly.

Thanks.
Hi,

I got same email as the op and I followed your instructions and now my server is down. I am getting error:

/tmp/crontab.XXXXJRVkLb: Read-only file system

How to fix this ???

Please help my server is down.

Thanks.
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
38
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Running the commands indicated in this thread would not cause the machine to go down and submitting a ticket to cPanel would be unable to fix this error:

/tmp/crontab.XXXXJRVkLb: Read-only file system
The /tmp partition is read only and indicates file system corruption or disk issues. You would need to contact your provider, datacenter or NOC to reboot the machine and have them run an fsck on it. This is outside of anything to do with cPanel and is a hardware level issue.
 

gkgcpanel

Well-Known Member
Jun 6, 2007
214
1
166
cPanel Access Level
DataCenter Provider
Tristan,

Ok, so I followed the instructions below and moved mysqltmp to our backup drive (which has plenty of space).

Only question I have is, what is to prevent a user from escaping to a MySQL Shell prompt (which can be easily done), and be able to list all of our customers backups?



You can use the following steps to move the MySQL /tmp directory location:

1. Edit /etc/my.cnf file to set new tmpdir for MySQL

Code:
vi /etc/my.cnf
Place the following line into the file under the [mysqld] section:

Code:
tmpdir=/mysqltmp
Save the file with :wq

2. Create the /mysqltmp directory

Code:
mkdir /mysqltmp
chmod 1777 /mysqltmp
3. Restart MySQL and check that the tmpdir is now /mysqltmp

Code:
/etc/init.d/mysql restart
mysqladmin var | grep tmpdir
You should see the following return:

Code:
# mysqladmin var | grep tmpdir
| slave_load_tmpdir                       | /mysqltmp
| tmpdir                                  | /mysqltmp
This presumes / partition itself has sufficient space. I would like to hope / partition would have more space than /tmp was provided on most systems.
 

SitexMedia

Member
Feb 28, 2013
12
0
1
cPanel Access Level
Root Administrator
I have decided not to install tempwatch for now as I hope the advice here will cure the tmp directory getting full

After changing my /scripts/securetmp to

my $tmpdsksize = 2097152; # Must be larger than 250000

I then

umount -l /tmp
umount -l /var/tmp
rm -fv /usr/tmpDSK
/scripts/securetmp

and received this warning notice

*** Notice *** No loop module detected
If the loopback block device is built as a module, try running `modprobe loop` as root via ssh and running this script again.
If the loopback block device is built into the kernel itself, you can ignore this message.
*** Notice *** No working loopback device files found. Try running `modprobe loop` as root via ssh and running this script again.

Would you like to secure /tmp & /var/tmp at boot time? (y/n) y
Would you like to secure /tmp & /var/tmp now? (y/n) y

Securing /tmp & /var/tmp
/tmp is already secure
/var/tmp is already secure
Checking fstab for entries ...Done
Logrotate TMPDIR already configured
Process Complete

Can anybody tell me does the above look right? as I did not notice on other posts the two lines asking to secure the tmp folders

I ran the above a couple of times and always the same. I did not run using modprobe loop as unsure what that is

Thanks
 
Last edited:

SitexMedia

Member
Feb 28, 2013
12
0
1
cPanel Access Level
Root Administrator
Once again, since I did the update above i have had an email

The file system /usr/tmpDSK, which is mounted at /tmp, has reached warn status because it is 89% full

Used: 1.66GB
Available: 0.21GB
Total: 1.97GB

I can increase the tmp size to 4gb and see what happens, I have looked at the files and see

drwxrwxrwt. 4 root root 110592 Mar 19 00:49 ./
dr-xr-xr-x. 22 root root 4096 Mar 17 11:49 ../
drwxr-xr-x. 2 mijasfor mijasfor 4096 Mar 18 16:47 drush-env/
-rw-------. 1 costapro costapro 0 Mar 18 09:40 file0U0O1C - <<<---- lots of these

drwx------. 2 root root 16384 Mar 17 11:49 lost+found/


-rw-------. 1 ukcityes ukcityes 359 Mar 19 00:28 sess_04b4c368056bdade9cf162ffaaebb8d9
-rw-------. 1 bedinbri bedinbri 105 Mar 19 00:39 sess_04cad4e85272e3c82cc878b79029efa9
-rw-------. 1 ukcityes ukcityes 248 Mar 19 00:28 sess_055946f36d4b89ae09c52d74c558dcc5

various of the above type lines

I will increase the size of the tmp to see what happens but may need to do more than that

Any thoughts?
 

DomineauX

Well-Known Member
PartnerNOC
Apr 12, 2003
429
11
168
Houston, TX
cPanel Access Level
Root Administrator
Try the following to see what large files are in /tmp:
ls -lahS /tmp/ | head

If disk space is fine when you check it though it might have been a MySQL tmp table that was quite large but gone once it was no longer needed (query completed or failed) by MySQL.

Better to find what is consuming space and resolve it than just increasing /tmp until it doesn't run low.
 

irwebco

Member
Dec 28, 2013
8
0
1
cPanel Access Level
Root Administrator
Hello,

What size is your /tmp partition? You can resize it if it isn't large enough. Many servers have only 500MB for /tmp which isn't large enough and should be around 2GB instead. You could edit /scripts/securetmp for this line:

Code:
my $tmpdsksize     = 512000;    # Must be larger than 250000
Change to this line for 2GB:

Code:
my $tmpdsksize     = 2097152;    # Must be larger than 250000
Save the file :)wq), then run these commands to resize /tmp partition:

Code:
umount -l /tmp
umount -l /var/tmp
rm -fv /usr/tmpDSK
/scripts/securetmp
Unfortunately, if you aren't able to catch it when it has the high usage, it's difficult for us to state why it's alerting. It could be high session activity (those sessions do clear on their own) for some site during the times it hits the high disk space level. It could be MySQL having the temporary MyISAM tables filling it up. It could be EAccelerator writing files to it (if you use that, it writes some temporary files there). There are many reasons it could be happening. If you aren't certain, you can simply resize it to be larger. Otherwise, you'll have to watch it closely until you do catch it and then provide the details on what you are seeing.

There are also some scripts that will periodically clear tmp such as tmpwatch, but you need to be careful not to cron clear /tmp as EasyApache writes to /tmp during build, so you cannot simply clear files every 15 minutes or something, since you'll then break EasyApache from recompiling properly.

Thanks.
hello dear

This is the wrong way

for resize /tmp partition you can use :
Code:
cp /scripts/securetmp /scripts/securetmp.bck
find this line in the /scripts/securetmp
Code:
 $tmpdsksize = int($tmpdsksize);
after find go to five line top you can see similar this line
Code:
my $FOUR_GIG_k = ( 1024 * 1024 * 2 );
this mean is 4G byte size of /tmp partition
You can modify the numbers to Favorites , for example i will build /tmp with 16G byte space i change this line to
Code:
    my $FOUR_GIG_k = ( 2048 * 2048 * 4 );
2048 * 2048 * 4 = 16G Byte space
after change save and exite from file and run another command on ssh area
Code:
service cpanel stop
service httpd stop
service mysql stop
service exim stop
cp -r /tmp /root/tmp.backup
cp -r /usr/tmpDSK /root/tmpDSK.backup
umount -l /tmp
umount -l /var/tmp
rm -fv /usr/tmpDSK
/scripts/securetmp
finish your tmp partition is resized now
run again all service
Code:
service cpanel start
service httpd start
service mysql start
service exim start