shacker23

Well-Known Member
Feb 20, 2005
263
1
168
I've been reading various threads here on Drive Space Critical messages but can't find the answer to this one.

All data is stored in various partitions - there is no data in "/" proper. If I were running out of space in a particular partition, wouldn't the error say that? My partitions appear to be fine, space-wise:

Code:
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      2.0G  1.8G  117M  94% /
/dev/mapper/VolGroup00-LogVol02
                      992M   37M  905M   4% /tmp
/dev/mapper/VolGroup00-LogVol04
                       15G  5.5G  8.1G  41% /var
/dev/mapper/VolGroup00-LogVol03
                       12G  6.1G  4.8G  56% /usr
/dev/mapper/VolGroup00-LogVol05
                       96G   70G   22G  77% /home
/dev/sda1              99M   31M   64M  33% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm
Here's output of du -sh /* and # /scripts/fixquotas:

Code:
# du -sh /* 
12K	/aquota.user
7.0M	/bin
25M	/boot
112K	/dev
4.0K	/error_log
137M	/etc
du: cannot read directory `/home/virtfs/africa/proc/23297': No such file or directory
78G	/home
351M	/lib
23M	/lib64
16K	/lost+found
8.0K	/media
0	/misc
12K	/mnt
0	/net
209M	/opt
4.0K	/perl
du: cannot access `/proc/17908/task/17908/fd/4681': No such file or directory
du: cannot read directory `/proc/24460': No such file or directory
0	/proc
4.0K	/quota.user
941M	/root
31M	/sbin
7.9M	/scripts
8.0K	/selinux
8.0K	/srv
0	/sys
3.2M	/tmp
5.8G	/usr
5.3G	/var
============

Code:
# /scripts/fixquotas
Installing Default Quota Databases...../aquota.user..../quota.user..../tmp/aquota.user..../tmp/quota.user..../var/aquota.user..../var/quota.user..../usr/aquota.user..../usr/quota.user..../home/aquota.user..../home/quota.user..../boot/aquota.user..../boot/quota.user.....Done
Linux Quota Support
Quotas are now on
Warning virtual file systems are mounted. Quota's may be counted as double
for users who are currently logged in. Please have jailshell users logout
before running quotacheck in the future!

All jailed users will be logged out in 60 seconds!


Broadcast message from root (Wed Apr  7 15:20:46 2010):

Warning! The system is about to perform quota maintenance.
All users will be logged out in 60 seconds.
PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
 account.
jailshell: no process killed
Warning : Your system does not have a seperate filesystem for backups.  This will cause performance degradation during the backup process.
Updating Quota Files......
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol00 [/] quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 2832 directories and 22395 files
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol04 [/var] done
	quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 938 directories and 16524 files
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol03 [/usr] done
	quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 18536 directories and 189102 files
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol05 [/home] done
	quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 222302 directories and 852355 files
	done
	

	quotacheck: WARNING - Quotafile //quota.user was probably truncated. Can't save quota settings...
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol00 [/] quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 2832 directories and 22395 files
	quotacheck: WARNING - Quotafile /var/quota.user was probably truncated. Can't save quota settings...
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol04 [/var] done
	quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 938 directories and 16524 files
	quotacheck: WARNING - Quotafile /usr/quota.user was probably truncated. Can't save quota settings...
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol03 [/usr] done
	quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 18536 directories and 189100 files
	quotacheck: WARNING - Quotafile /home/quota.user was probably truncated. Can't save quota settings...
	quotacheck: Scanning /dev/mapper/VolGroup00-LogVol05 [/home] done
	quotacheck: Old group file not found. Usage will not be substracted.
	quotacheck: Checked 222320 directories and 852384 files
	done
	
....Done
Resetting quota for africa to 20000 M
Resetting quota for albany to 700 M
Resetting quota for barthood to 20000 M
Resetting quota for berkblog to 20000 M
Resetting quota for bigstory to 20000 M
Resetting quota for bnn to 20000 M
Resetting quota for bpug to 20000 M
Resetting quota for calnews to 20000 M
Resetting quota for calpress to 20000 M
Resetting quota for charris to 20000 M
Resetting quota for citylab to 20000 M
Resetting quota for cpantech to 20000 M
Resetting quota for ebaywest to 20000 M
Resetting quota for ecfocus to 20000 M
Resetting quota for emery to 20000 M
Resetting quota for git to 20000 M
Resetting quota for intra to 20000 M
Resetting quota for jazz to 20000 M
Resetting quota for journ to 20000 M
Resetting quota for keystone to 25000 M
Resetting quota for knight to 20000 M
Resetting quota for loclrprt to 20000 M
Resetting quota for mandric to 20000 M
Resetting quota for mission to 20000 M
Resetting quota for news21 to 20000 M
Resetting quota for news21n to 20000 M
Resetting quota for now51st to 20000 M
Resetting quota for nowoak to 20000 M
Resetting quota for oaknorth to 20000 M
Resetting quota for richcon to 20000 M
Resetting quota for rprt510 to 20000 M
Resetting quota for rue to 20000 M
Resetting quota for shacker to 20000 M
Resetting quota for tvlives to 20000 M
Resetting quota for vlab to 20000 M
Thanks for any tips.
 

shacker23

Well-Known Member
Feb 20, 2005
263
1
168
OK, I figured it out. My mistake was in saying that all data lives off in other partitions. That's mostly true, but there's a 2GB "/" partition that contains a small amount of data - kernels, system-level stuff, and the /root account. What threw me is this distinction:

df -h <-- Shows all partitions mounted at /
ls -la / <-- Shows all directories living in / including the mounted partitions.

So if you want to know what directories truly live at / you need to subtract the set of partition names from the list you get with ls -la / . Then you'll have a list of directories to investigate when the system reports "Drive Space Critical at /".

Once I understood this distinction, finding the offending files was easy (we had stored some temp files in root's home, which is one of the actual directories living at /).

Since the / partition is typically so small (ours is 2GB), it doesn't take many files to push it to the critical stage.

Lesson: Don't use the /root directory as a file storage location :)