HELP! - WHM wont start - 500 error with disk quota exceeded message

[doulos61]

Let me start this off by stating that there should be no mass emailings being sent from this server. The only emails that should come from this server is admin notifications and such. I received notification from the single client for this VPS that server was down. I get an "Error 500 - internal server error" trying to connect to the WHM. I then rebooted the server vps control panel. Once I was able to connect to the server via WHM port login, I got this message -

Internal Server Error
can't write into /var/cpanel/configs.cache/_var_cpanel_sessions_jmGVHIpNCN3hFA3TeE9X4Ks4en75gOvWCwdNTjkY6taIuJu_8IqNkXKIrmS96lYP___1ef8a188423cdc2808a41e4a265ac8b4: Disk quota exceeded at (eval 11) line 1 

 at /usr/lib/perl5/5.6.2/Carp.pm line 119
	Carp::croak('can\'t write into /var/cpanel/configs.cache/_var_cpanel_sessions...') called at /usr/lib/perl5/site_perl/5.6.2/i686-linux/Storable.pm line 76
	Storable::logcroak('can\'t write into /var/cpanel/configs.cache/_var_cpanel_sessions...') called at /usr/lib/perl5/site_perl/5.6.2/i686-linux/Storable.pm line 248
	Storable::_store('CODE(0x8e71f4c)', 'HASH(0x8f51cf4)', '/var/cpanel/configs.cache/_var_cpanel_sessions_jmGVHIpNCN3hFA3Te...', 1) called at /usr/lib/perl5/site_perl/5.6.2/i686-linux/Storable.pm line 236
	Storable::lock_nstore('HASH(0x8f51cf4)', '/var/cpanel/configs.cache/_var_cpanel_sessions_jmGVHIpNCN3hFA3Te...') called at (eval 11) line 1
	eval 'Storable::lock_nstore( $conf_ref, $cache_file  );' called at /usr/local/cpanel/Cpanel/Config/LoadConfig.pm line 248
	Cpanel::Config::LoadConfig::loadConfig('/var/cpanel/sessions/jmGVHIpNCN3hFA3TeE9X4Ks4en75gOvWCwdNTjkY6ta...', 'HASH(0x8f51cf4)', '=', '^\s*[#]', 0, 0, 'HASH(0x8f51ce8)') called at cpsrvd-ssl line 7051
	main::loadSession('jmGVHIpNCN3hFA3TeE9X4Ks4en75gOvWCwdNTjkY6taIuJu_8IqNkXKIrmS96lYP') called at cpsrvd-ssl line 4268
	main::handle_auth() called at cpsrvd-ssl line 809

When I login to the SSH and get this -

/tmp/RsMkBIts: Disk quota exceeded 
-bash: echo: write error: Broken pipe 
/tmp/Rs9W32Zq: Disk quota exceeded

So I then check the disk quota, and I get this -

[email protected] [~]# df -h 
Filesystem Size Used Avail Use% Mounted on 
/dev/simfs 19G 14G 5.6G 71% / 
none 250M 0 250M 0% /dev/shm

To follow up I check the cpanel logs under /usr/local/cpanel/logs/error_log and see this repeat

[2011-07-16 07:35:01 -0500] warn [tailwatchd] Lock /var/cpanel/cpanel.config.lock lost! at /usr/local/cpanel/Cpanel/SafeFile.pm line 149
        Cpanel::SafeFile::safeunlock(ARRAY(0x9449808)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 78
        Cpanel::SafeFile::safeclose(GLOB(0x94298ac), ARRAY(0x9449808)) called at /usr/local/cpanel/Cpanel/Config/LoadConfig.pm line 144
        Cpanel::Config::LoadConfig::loadConfig('/var/cpanel/cpanel.config', HASH(0x9434130), undef, undef, undef, 1, HASH(0x94498d4)) called
        Cpanel::Config::LoadCpConf::loadcpconf() called at /usr/local/cpanel/Cpanel/TailWatch.pm line 332
        Cpanel::TailWatch::init_global_share(Cpanel::TailWatch=HASH(0x93069ac)) called at /usr/local/cpanel/Cpanel/TailWatch.pm line 70
        Cpanel::TailWatch::new('Cpanel::TailWatch') called at /usr/local/cpanel/libexec/tailwatch/tailwatchd line 22
Internal Server Error: "GET / HTTP/1.1" 500 more than 400 attempts to create a session failed
Internal Server Error: "GET / HTTP/1.1" 500 more than 400 attempts to create a session failed
Internal Server Error: "GET /favicon.ico HTTP/1.1" 500 more than 400 attempts to create a session failed
Internal Server Error: "GET / HTTP/1.1" 500 more than 400 attempts to create a session failed[2011-07-16 08:35:01 -0500] warn [restartsrv_tailwatchd] Lock /var/cpanel/cpanel.config.lock lost! at /usr/local/cpanel/Cpanel/SafeFile.pm l
        Cpanel::SafeFile::safeunlock(ARRAY(0x9018c00)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 78
        Cpanel::SafeFile::safeclose(GLOB(0x9001404), ARRAY(0x9018c00)) called at /usr/local/cpanel/Cpanel/Config/LoadConfig.pm line 144
        Cpanel::Config::LoadConfig::loadConfig('/var/cpanel/cpanel.config', HASH(0x900c458), undef, undef, undef, 1, HASH(0x902c320)) called
        Cpanel::Config::LoadCpConf::loadcpconf() called at /usr/local/cpanel/Cpanel/Rlimit.pm line 122
        Cpanel::Rlimit::get_server_setting_or_default() called at /usr/local/cpanel/Cpanel/Rlimit.pm line 102
        Cpanel::Rlimit::set_rlimit_to_infinity() called at /usr/local/cpanel/scripts/restartsrv_tailwatchd line 15
        eval {...} called at /usr/local/cpanel/scripts/restartsrv_tailwatchd line 12

How can I fix this and get my WHM back online?
Please assist

 

[Bdzzld]

Internal Server Error
can't write into /var/cpanel/configs.cache/_var_cpanel_sessions_jmGVHIpNCN3hFA3TeE9X4Ks4en75gOvWCwdNTjkY6taIuJu_8IqNkXKIrmS96lYP___1ef8a188423cdc2808a41e4a265ac8b4: Disk quota exceeded at (eval 11) line 1

You /var partition is full or in readonly mode. Execute

mount

to see for yourself.

 

[doulos61]

Thanks for the response - I did the "mount" command. This is what I got -

[email protected] [/]# mount
/dev/simfs on / type reiserfs (rw,usrquota,grpquota)
/proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
none on /dev/pts type devpts (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

If you don't mind, how do I fix this?
Thnx

 

[doulos61]

Just thought I'd follow up - The problem was the Inodes were maxed out.

[email protected] [~]# df -ih
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/simfs              195K    195K       0  100% /
none                     63K       1     63K    1% /dev/shm

I contacted VPS provider, they corrected the Inodes and the issue was resolved.

df -ih : 
Filesystem Inodes IUsed IFree IUse% Mounted on 
/dev/simfs 5.9M 195K 5.7M 4% / 
none 63K 1 63K 1% /dev/shm

The WHM launched without issue.
I appreciate the input
D