The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

MySQL restarts without InnoDB

Discussion in 'Database Discussions' started by robertjw, Aug 21, 2015.

  1. robertjw

    robertjw Member

    Joined:
    Oct 18, 2013
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Occasionally, once a week or so, my server will be hit with a brute force attack that sucks up all it's memory. When this happens MySQL will crash and restart.

    The problem is that when it restarts there often won't be enough memory available to restart InnoDB, but MySQL will restart without InnoDB running.

    Here is an excerpt from the log file:


    150821 12:07:59 [Note] InnoDB: Initializing buffer pool, size = 119.0M
    InnoDB: mmap(127713280 bytes) failed; errno 12
    150821 12:07:59 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
    150821 12:07:59 [ERROR] Plugin 'InnoDB' init function returned error.
    150821 12:07:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    150821 12:07:59 [ERROR] mysqld: Out of memory (Needed 128909312 bytes)
    150821 12:07:59 [Note] Plugin 'FEEDBACK' is disabled.
    150821 12:07:59 [Note] Server socket created on IP: '::'.


    I have one client using a plugin that requires InnoDB, otherwise almost everything else humms along without a problem.

    So, why doesn't MySQL fail? Can I control this? Can I setup some kind of watchdog that will check if InnoDB is running and either notify me or restart MySQL if it's not?
     
  2. 24x7server

    24x7server Well-Known Member

    Joined:
    Apr 17, 2013
    Messages:
    1,146
    Likes Received:
    34
    Trophy Points:
    48
    Location:
    India
    cPanel Access Level:
    Root Administrator
    Hello,

    Can you please provide use "free -m" command output from your server, And try to increase your server swap space and check if you are getting same issues.
     
  3. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    I believe the best way to address this issue is to implement firewall rules to prevent the brute force attacks that are taking up your memory. Have you installed a firewall management utility such as CSF to help prevent these types of attacks?

    Thank you.
     
  4. robertjw

    robertjw Member

    Joined:
    Oct 18, 2013
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Yes, I have CSF installed and ModSecurity enabled with the Comodo WAF plugin. This has almost eliminated the problem, but it still happens about once a week.

    The server isn't configured with any swap space.
    free -m
    total used free shared buffers cached
    Mem: 3832 3582 250 4 82 2201
    -/+ buffers/cache:1298 2533
    Swap: 0 0 0

    My hosting company set it up this way, and I'm not sure that adding swap is a good idea. If an attack uses up all the swap space too and MySQL crashes, won't I have the same issue?
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You may simply need to add additional physical memory to your system if it's current available memory is not sufficient to handle these types of attacks. Or, you may want to consult with a system administrator to see if these attacks are preventable through other custom firewall rules.

    Thank you.
     
  6. robertjw

    robertjw Member

    Joined:
    Oct 18, 2013
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    My problem isn't really with the attacks. They are a rare occurrence, and I'm not sure how much additional memory would be needed to make sure this never happens.

    My problem is why MySQL comes up and runs in a failed state?

    Is there a configuration for MySQL where it would crash if it can't start InnoDB?

    Is there a way to test for this externally?
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You should be able to add the following line in your /etc/my.cnf file if you want MySQL to fail to start if InnoDB fails:

    Code:
    innodb=force
    Thank you.
     
Loading...

Share This Page