The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Got error 28 from storage engine

Discussion in 'General Discussion' started by quicklyweb, Oct 27, 2009.

  1. quicklyweb

    quicklyweb Member

    Joined:
    Apr 24, 2005
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    Got error 28 from storage engine. Harddisk is only on 36% and /tmp directory is emptied. But i am still getting the same error message. Restarting the server solves the issue for about a day. But error comes back again. I am stuck.
     
  2. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    It will help to know a few more details about the circumstances involved so that we can better diagnose the pinpoint the cause.

    Here are a few questions to start with:

    1.) What activity occurs before and up to when the error is experienced?

    2.) Is the server running in a virtual environment (i.e., a VPS system, like Xen or Virtuozzo/OpenVZ)?

    3.) What specific sources display the error? For example, is it seen in a specific log file, and or displayed by a specific account and script using MySQL?

    4.) Do multiple accounts exhibit the same error?

    5.) What version of MySQL is in-use? The following two commands entered via root SSH access will help to determine the MySQL client and daemon (server) versions:
    Code:
    # mysql -V
    # mysqld -V
    The following page from the vendor web site may help to narrow the investigation:
    MySQL :: Error 28, how to avoid it.
     
  3. quicklyweb

    quicklyweb Member

    Joined:
    Apr 24, 2005
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    1.) What activity occurs before and up to when the error is experienced?

    The problems started when the /tmp directory was full a few weeks ago. For some reason i couldn't delete the files in the tmp folder with rm -rf * so i deleted the whole folder and created new one. But that didn't solve the issue. Seems like after reboot it is going fine for hours and starting to give errors again

    2.) Is the server running in a virtual environment (i.e., a VPS system, like Xen or Virtuozzo/OpenVZ)?

    No virtual environment. It is dedicated server.

    3.) What specific sources display the error? For example, is it seen in a specific log file, and or displayed by a specific account and script using MySQL?

    I get the error with every account using mysql now in their scripts. Including phpmyadmin.


    4.) Do multiple accounts exhibit the same error?

    I get the error practically with every account using mysql.


    5.) What version of MySQL is in-use?


    mysql Ver 14.12 Distrib 5.0.81, for pc-linux-gnu (i686) using readline 5.1
    mysqld Ver 5.0.81-community for pc-linux-gnu on i686 (MySQL Community Edition (GPL))
     
  4. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    What are the current permissions on the "/tmp" directory (i.e.,. what is the output displayed by the following command)?
    Code:
    # stat /tmp
    The line showing the permissions should appear similar to the following:
    Code:
    Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
    If the access mode is not "1777" please try the following to apply corrected permissions, followed by rechecking output from "stat":
    Code:
    # chmod -v 1777 /tmp
    If that alone does not correct the problem, please also provide output from the following two commands via root SSH access:
    Code:
    # df -h
    # du -hsx /tmp
    # mount
    As another information resource, I would also check the MySQL error log, checking for any additional hints to a more specific cause of the current issue. The following command may help to locate the MySQL error log path:
    Code:
    # mysqladmin variables | grep -i "log_error"
    If the above "log_error" variable is not set, here are two additional commands to help locate the MySQL error logs in two commonly-used paths:
    Code:
    # find /var/lib/mysql -iname "*.err"
    # find /var/log -iname "mysql*"
    In the directory "/var/lib/mysql" the MySQL error log may be named after the hostname and end in a file extension of ".err".

    Once the log file is located, simply open it in your preferred text viewer (e.g., less, view); here are a few examples (where "/path/to/log" is replaced by the actual path found in the above step):
    Code:
    # less /path/to/log
    # view /path/to/log
    For less, pressing "q" on the keyboard will exit.
    For view, like vi and vim, entering a ":" (colon) then typing "q" and pressing Enter will exit.
     
  5. quicklyweb

    quicklyweb Member

    Joined:
    Apr 24, 2005
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    root@latte [~]# stat /tmp
    File: `/tmp'
    Size: 81920 Blocks: 162 IO Block: 4096 directory
    Device: 700h/1792d Inode: 2 Links: 4
    Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root)
    Access: 2009-10-28 22:39:38.000000000 +0000
    Modify: 2009-10-28 22:39:35.000000000 +0000
    Change: 2009-10-28 22:39:35.000000000 +0000

    root@latte [~]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda1 143G 42G 94G 31% /
    tmpfs 502M 0 502M 0% /dev/shm
    /usr/tmpDSK 485M 12M 449M 3% /tmp

    root@latte [~]# du -hsx /tmp
    1.2M /tmp

    root@latte [~]# mysqladmin variables | grep -i "log_error"
    | log_error | |

    root@latte [~]# find /var/lib/mysql -iname "*.err"
    /var/lib/mysql/latte.quicklyweb.net.err

    less /var/lib/mysql/latte.quicklyweb.net.err
    091028 2:13:57 [ERROR] /usr/sbin/mysqld: Disk is full writing '/tmp/STH2JAx1' (Errcode: 28). Waiting for someone to free space... (Expect u
    p to 60 secs delay for server to continue after freeing disk space)
    091028 2:13:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 2:23:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 2:33:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 2:43:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 2:53:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 3:03:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 3:13:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 3:23:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 3:33:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 3:43:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 3:53:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 4:02:57 [ERROR] /usr/sbin/mysqld: Disk is full writing '/tmp/STH2JAx1' (Errcode: 28). Waiting for someone to free space... (Expect u
    p to 60 secs delay for server to continue after freeing disk space)
    091028 4:02:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 4:12:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 4:22:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 4:32:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 4:42:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 4:52:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 5:02:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 5:12:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 5:22:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 5:32:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 5:42:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 5:52:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 6:02:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 6:12:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 6:22:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 6:32:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 6:42:57 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    091028 6:51:19 [Note] /usr/sbin/mysqld: Normal shutdown

    091028 6:51:21 [Warning] /usr/sbin/mysqld: Forcing close of thread 22596 user: 'root'

    091028 6:52:09 [Note] Retrying repair of: './jazzex_nuke1/nuke_bbsearch_wordmatch' with keycache
    091028 6:52:09 InnoDB: Starting shutdown...
    091028 6:52:10 InnoDB: Shutdown completed; log sequence number 0 86741786
    091028 6:52:10 [Note] /usr/sbin/mysqld: Shutdown complete

    091028 06:52:10 mysqld ended

    091028 06:53:39 mysqld started
    091028 6:53:39 InnoDB: Started; log sequence number 0 86741786
    091028 6:53:40 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.0.81-community' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition (GPL)
    091028 22:51:01 [ERROR] /usr/sbin/mysqld: Disk is full writing '/tmp/STtHQv9e' (Errcode: 28). Waiting for someone to free space... (Expect u
    p to 60 secs delay for server to continue after freeing disk space)
    091028 22:51:02 [ERROR] /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
    (END)
     
  6. cPanelDon

    cPanelDon cPanel Quality Assurance Analyst
    Staff Member

    Joined:
    Nov 5, 2008
    Messages:
    2,557
    Likes Received:
    7
    Trophy Points:
    38
    Location:
    Houston, Texas, U.S.A.
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Please submit a ticket so that we can take a closer look at this issue; if possible, please also PM me the ticket ID so I may follow-up with this internally. If needed, you may use the link in my forums signature to initiate the ticket submission.
     
Loading...

Share This Page