The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

MySQL Monitoring failed to detect partial outage

Discussion in 'Database Discussions' started by ottdev, Aug 27, 2016.

  1. ottdev

    ottdev Well-Known Member

    Joined:
    Oct 1, 2013
    Messages:
    63
    Likes Received:
    1
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    Twice today MySQL crashed, recovered fine the first, but not after the second. No cPanel server monitoring detected the condition, so no automated recovery took place. There are a few log entries that perhaps should be monitored for ... why wouldn't something like 'Fatal error' be detected ? These in combination indicate an issue despite MySQL appearing to be up.

    Here are the relevant lines from the second entry indicating an issue
    (from /var/lib/mysql/%HOSTNAME%.err )
    InnoDB: mmap(136806400 bytes) failed; errno 12
    160827 10:38:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
    160827 10:38:10 [ERROR] Plugin 'InnoDB' init function returned error.
    160827 10:38:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

    Here are the 2 full log entries:
    Code:
    160827  9:20:05160827  9:20:05 InnoDB: Unable to allocate memory of size 14909544.
    160827  9:20:05  InnoDB: Assertion failure in thread 2751060848 in file mem0mem.c line 361
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to [URL='http://bugs.mysql.com']MySQL Bugs[/URL].
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: [URL='http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html']MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery[/URL]
    InnoDB: about forcing recovery.
    InnoDB: Unable to allocate memory of size 14909544.
    160827  9:20:05  InnoDB: Assertion failure in thread 2755455856 in file mem0mem.c line 361
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to [URL='http://bugs.mysql.com']MySQL Bugs[/URL].
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: [URL='http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html']MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery[/URL]
    InnoDB: about forcing recovery.
    160827 09:20:05 mysqld_safe Number of processes running now: 0
    160827 09:20:05 mysqld_safe mysqld restarted
    160827  9:20:05 [Note] /usr/sbin/mysqld (mysqld 5.5.50-cll) starting as process 15494 ...
    160827  9:20:05 [Note] Plugin 'FEDERATED' is disabled.
    160827  9:20:05 InnoDB: The InnoDB memory heap is disabled
    160827  9:20:05 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
    160827  9:20:05 InnoDB: Compressed tables use zlib 1.2.3
    160827  9:20:05 InnoDB: Using Linux native AIO
    160827  9:20:05 InnoDB: Initializing buffer pool, size = 128.0M
    160827  9:20:06 InnoDB: Completed initialization of buffer pool
    160827  9:20:06 InnoDB: highest supported file format is Barracuda.
    InnoDB: Log scan progressed past the checkpoint lsn 21388284244
    160827  9:20:06  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 21388305604
    160827  9:20:13  InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
    InnoDB: Apply batch completed
    160827  9:20:14  InnoDB: Waiting for the background threads to start
    160827  9:20:15 InnoDB: 5.5.50 started; log sequence number 21388305604
    160827  9:20:15 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
    160827  9:20:15 [Note]  - '0.0.0.0' resolves to '0.0.0.0';
    160827  9:20:15 [Note] Server socket created on IP: '0.0.0.0'.
    160827  9:20:15 [Note] Event Scheduler: Loaded 0 events
    160827  9:20:15 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.5.50-cll'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
    
    =====================================================================
    
    160827 10:38:10160827 10:38:10 InnoDB: Unable to allocate memory of size 14909544.
    160827 10:38:10  InnoDB: Assertion failure in thread 2673179504 in file mem0mem.c line 361
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to [URL='http://bugs.mysql.com']MySQL Bugs[/URL].
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: [URL='http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html']MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery[/URL]
    InnoDB: about forcing recovery.
    InnoDB: Unable to allocate memory of size 14909544.
    160827 10:38:10  InnoDB: Assertion failure in thread 2743319408 in file mem0mem.c line 361
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to [URL='http://bugs.mysql.com']MySQL Bugs[/URL].
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: [URL='http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html']MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery[/URL]
    InnoDB: about forcing recovery.
    160827 10:38:10 mysqld_safe Number of processes running now: 0
    160827 10:38:10 mysqld_safe mysqld restarted
    160827 10:38:10 [Note] /usr/sbin/mysqld (mysqld 5.5.50-cll) starting as process 27807 ...
    160827 10:38:10 [Note] Plugin 'FEDERATED' is disabled.
    160827 10:38:10 InnoDB: The InnoDB memory heap is disabled
    160827 10:38:10 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
    160827 10:38:10 InnoDB: Compressed tables use zlib 1.2.3
    160827 10:38:10 InnoDB: Using Linux native AIO
    160827 10:38:10 InnoDB: Initializing buffer pool, size = 128.0M
    InnoDB: mmap(136806400 bytes) failed; errno 12
    160827 10:38:10 InnoDB: Completed initialization of buffer pool
    160827 10:38:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
    160827 10:38:10 [ERROR] Plugin 'InnoDB' init function returned error.
    160827 10:38:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    160827 10:38:10 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
    160827 10:38:10 [Note]  - '0.0.0.0' resolves to '0.0.0.0';
    160827 10:38:10 [Note] Server socket created on IP: '0.0.0.0'.
    160827 10:38:11 [Note] Event Scheduler: Loaded 0 events
    160827 10:38:11 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.5.50-cll'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
     
    #1 ottdev, Aug 27, 2016
    Last edited: Aug 28, 2016
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello,

    The log output shows that MySQL started, but with InnoDB corruption. Chkservd currently determines if the MySQL process is running, but it doesn't issue notifications when InnoDB is corrupted. The following thread aims to help users resolve issues with InnoDB:

    InnoDB Corruption Repair Guide

    We are currently exploring options to detect and notify users about InnoDB corruption, but there's no specific time frame to offer on that feature at this time.

    Thank you.
     
Loading...

Share This Page