The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Swap pages per second in/out

Discussion in 'General Discussion' started by GoWilkes, Mar 24, 2014.

  1. GoWilkes

    GoWilkes Well-Known Member

    Joined:
    Sep 26, 2006
    Messages:
    367
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    I'm looking at the Munin report, under "Swap in/out - by day".

    What is considered "good" when it comes to this section? I just migrated to a new server, and it's showing several times throughout the day that I'm either between 0 and -4.0, or between 0 and +3.0. I can't look at the stats on my old server (they have already aged out of Munin), but I don't remember it having ANY swap.

    The CPU and RAM usage are well within normal limits. I have 4G of RAM, and "apps" usually stays between 1.6G and 3.0G. I also have two 2.4GHz CPUs, and the load usually stays between 0.4 and 0.5 (with a max today of 1.17 during peak hours).

    With this hardware, should I be using swap at all? If not, what's causing it?
     
  2. Sys Admin

    Sys Admin Well-Known Member

    Joined:
    Apr 29, 2007
    Messages:
    67
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Do you have any type of RAID on the new server? Please provide the full output of:

     
  3. GoWilkes

    GoWilkes Well-Known Member

    Joined:
    Sep 26, 2006
    Messages:
    367
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    No, to my knowledge there's no RAID.

    Here is the output you requested. I'm doing this at 3:30am, when the traffic is typically the lowest; do you need it during peak hours, instead?:

    Code:
    # free -mt
                 total       used       free     shared    buffers     cached
    Mem:          3881       3810         71          0        271       2206
    -/+ buffers/cache:       1331       2549
    Swap:         2047        121       1925
    Total:        5928       3932       1996
    
    Code:
    # cat /proc/cpuinfo
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 42
    model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz
    stepping        : 7
    cpu MHz         : 3392.412
    cache size      : 8192 KB
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm rep_good aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer hypervisor lahf_lm arat epb pln pts dts
    bogomips        : 6784.82
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
    processor       : 1
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 42
    model name      : Intel(R) Xeon(R) CPU E31270 @ 3.40GHz
    stepping        : 7
    cpu MHz         : 3392.412
    cache size      : 8192 KB
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm rep_good aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer hypervisor lahf_lm arat epb pln pts dts
    bogomips        : 6784.82
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
    Code:
    # lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                2
    On-line CPU(s) list:   0,1
    Thread(s) per core:    1
    Core(s) per socket:    1
    Socket(s):             2
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 42
    Stepping:              7
    CPU MHz:               3392.412
    BogoMIPS:              6784.82
    Hypervisor vendor:     Xen
    Virtualization type:   para
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0,1
    
    Code:
    # sar -q
    Linux 2.6.32-431.5.1.el6.x86_64 (server1.example.com)    03/26/2014      _x86_64_(2 CPU)
    
    12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    12:10:01 AM         2       204      0.02      0.11      0.09
    12:20:01 AM         2       197      0.21      0.14      0.10
    12:30:01 AM         5       205      0.33      0.21      0.12
    12:40:01 AM         2       198      0.07      0.15      0.13
    12:50:01 AM         2       197      0.04      0.10      0.12
    01:00:01 AM         3       218      0.07      0.10      0.10
    01:10:01 AM         4       195      0.22      0.14      0.10
    01:20:01 AM         2       196      0.15      0.16      0.12
    01:30:01 AM         3       200      0.07      0.18      0.17
    01:40:01 AM         2       212      0.19      0.21      0.18
    01:50:01 AM         2       206      0.04      0.14      0.16
    02:00:01 AM         3       202      0.12      0.13      0.15
    02:10:01 AM         3       195      0.16      0.13      0.14
    02:20:01 AM         2       198      0.08      0.14      0.15
    02:30:01 AM         4       196      0.03      0.09      0.13
    02:40:01 AM         2       196      0.03      0.04      0.08
    02:50:01 AM         2       208      0.25      0.16      0.10
    03:00:01 AM         3       190      0.12      0.14      0.09
    03:10:01 AM         2       197      0.09      0.09      0.09
    03:20:01 AM         2       188      0.25      0.15      0.10
    03:30:01 AM         3       192      0.05      0.08      0.08
    Average:            3       200      0.12      0.13      0.12
    
    Code:
    # sar -r
    12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commi                                                                                                 t
    12:10:01 AM    117856   3856504     97.03    178260   2292152   1864560     30.7                                                                                                 1
    12:20:01 AM    128532   3845828     96.77    180780   2298528   1727312     28.4                                                                                                 5
    12:30:01 AM    110456   3863904     97.22    182816   2308996   1772864     29.2                                                                                                 0
    12:40:01 AM     62140   3912220     98.44    185252   2352588   1705716     28.1                                                                                                 0
    12:50:01 AM     66372   3907988     98.33    186092   2356304   1680684     27.6                                                                                                 8
    01:00:01 AM     34952   3939408     99.12    188968   2328928   2035512     33.5                                                                                                 3
    01:10:01 AM    116472   3857888     97.07    182364   2312368   1638760     26.9                                                                                                 9
    01:20:01 AM     52800   3921560     98.67    184788   2370712   1692752     27.8                                                                                                 8
    01:30:01 AM     53616   3920744     98.65    186752   2364744   1708268     28.1                                                                                                 4
    01:40:01 AM     97264   3877096     97.55    176848   2326128   1922004     31.6                                                                                                 6
    01:50:01 AM     79656   3894704     98.00    179012   2331968   1794340     29.5                                                                                                 6
    02:00:01 AM    108632   3865728     97.27    178092   2313560   1762932     29.0                                                                                                 4
    02:10:01 AM    127436   3846924     96.79    180128   2318684   1682480     27.7                                                                                                 1
    02:20:01 AM     93912   3880448     97.64    181788   2328084   1722140     28.3                                                                                                 7
    02:30:01 AM     83464   3890896     97.90    183540   2336868   1688508     27.8                                                                                                 1
    02:40:01 AM     93792   3880568     97.64    185412   2340540   1687460     27.8                                                                                                 0
    02:50:01 AM     42456   3931904     98.93    184364   2334248   1932212     31.8                                                                                                 3
    03:00:01 AM     95196   3879164     97.60    184612   2344428   1609236     26.5                                                                                                 1
    03:10:01 AM     90688   3883672     97.72    186528   2354392   1692308     27.8                                                                                                 8
    03:20:01 AM     57448   3916912     98.55    279416   2271124   1530716     25.2                                                                                                 1
    03:30:01 AM     53464   3920896     98.65    278072   2260352   1618000     26.6                                                                                                 5
    Average:        84124   3890236     97.88    192090   2325986   1736608     28.6                                                                                                 1
    
    Code:
    # iostat
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               6.75    0.63    4.41    1.42    0.08   86.71
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    xvdb              0.41         1.05         2.63     145852     366576
    xvda             53.89       183.23       631.54   25501168   87895698
    
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    654
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  5. GoWilkes

    GoWilkes Well-Known Member

    Joined:
    Sep 26, 2006
    Messages:
    367
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    For whatever reason, at around 8:50am EST today my disk latency, disk utilization, RAM, and CPU usage skyrocketed. I usually have a CPU load of around 0.5, and it's currently 77!

    I had made a few minor Exim changes through the WHM Basic Configuration editor, so I thought that might have been the culprit. I restored, and even turned off Exim entirely, with no change, so I guess that wasn't it.

    You can see exactly what I mean here:

    Code:
    # sar -q
    Linux 2.6.32-431.5.1.el6.x86_64 (server1.example.com)    04/02/2014      _x86_64_(2 CPU)
    
    12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    12:10:01 AM         1       200      0.06      0.17      0.17
    12:20:01 AM         2       195      0.12      0.21      0.18
    12:30:01 AM         4       195      0.06      0.17      0.17
    12:40:01 AM         2       198      0.27      0.19      0.18
    12:50:01 AM         2       206      0.21      0.19      0.18
    01:00:01 AM         4       196      0.11      0.12      0.15
    01:10:01 AM         2       194      0.01      0.13      0.16
    01:20:01 AM         2       190      0.06      0.15      0.16
    01:30:01 AM         3       189      0.06      0.14      0.16
    01:40:01 AM         2       199      0.02      0.08      0.13
    01:50:01 AM         2       197      0.25      0.13      0.10
    02:00:01 AM         2       196      0.03      0.06      0.08
    02:10:01 AM         3       202      0.15      0.12      0.09
    02:20:01 AM         2       185      0.09      0.05      0.06
    02:30:01 AM         4       191      0.18      0.08      0.04
    02:40:01 AM         2       190      0.00      0.01      0.01
    02:50:01 AM         2       191      0.03      0.03      0.00
    03:00:01 AM         2       184      0.07      0.11      0.03
    03:10:01 AM         2       187      0.19      0.16      0.10
    03:20:01 AM         2       187      0.06      0.10      0.09
    03:30:01 AM         3       183      0.02      0.08      0.08
    03:40:02 AM         2       200      0.04      0.06      0.07
    
    03:40:02 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    03:50:01 AM         2       195      0.08      0.09      0.08
    04:00:01 AM         3       193      0.07      0.20      0.17
    04:10:01 AM         2       190      0.05      0.09      0.12
    04:20:01 AM         2       200      0.42      0.32      0.20
    04:30:01 AM         3       205      0.18      0.20      0.18
    04:40:01 AM         2       192      0.16      0.21      0.18
    04:50:01 AM         2       198      0.22      0.18      0.16
    05:00:01 AM         2       192      0.16      0.16      0.15
    05:10:01 AM         2       184      0.17      0.11      0.10
    05:20:01 AM         2       186      0.18      0.11      0.09
    05:30:01 AM         3       195      0.07      0.06      0.06
    05:40:01 AM         2       194      0.12      0.14      0.11
    05:50:01 AM         2       206      0.19      0.13      0.10
    06:00:01 AM         3       201      0.04      0.12      0.10
    06:10:01 AM         2       201      0.11      0.08      0.08
    06:20:01 AM         4       213      0.06      0.12      0.10
    06:30:01 AM         3       210      0.13      0.19      0.13
    06:40:01 AM         2       206      0.24      0.22      0.17
    06:50:01 AM         2       205      0.05      0.14      0.16
    07:00:01 AM         2       210      0.17      0.19      0.18
    07:10:01 AM         3       207      0.10      0.17      0.17
    07:20:01 AM         2       228      0.41      0.27      0.20
    
    07:20:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    07:30:01 AM         2       212      0.23      0.18      0.17
    07:40:01 AM         4       213      0.03      0.18      0.18
    07:50:01 AM         2       231      0.18      0.24      0.20
    08:00:01 AM         4       246      0.09      0.27      0.25
    08:10:01 AM         2       231      0.48      0.28      0.22
    08:20:01 AM         1       299      0.54      0.68      0.50
    08:30:01 AM         5       241      0.80      1.16      0.90
    08:40:01 AM         2       286      0.52      0.95      0.96
    08:50:01 AM         2       255     10.68      7.60      3.85
    09:00:01 AM         1       347     15.27      8.13      5.00
    09:10:01 AM         2       224      5.09      5.79      5.44
    09:20:01 AM         2       313     33.26     15.00      8.44
    09:30:01 AM         2       289     13.55     25.06     19.79
    09:40:01 AM         2       295     25.29     31.82     27.66
    09:50:01 AM         1       312     37.19     28.17     26.51
    10:00:01 AM         2       232      4.27     11.82     19.89
    10:10:01 AM         2       267     11.29      8.02     13.50
    10:20:01 AM         2       288     28.31     35.00     25.58
    10:30:02 AM         3       222      2.95      9.96     17.47
    10:40:02 AM         3       235      4.73      4.73     11.03
    10:50:01 AM         3       215      1.26      4.04      8.30
    11:00:01 AM         0       220      5.29      7.98      8.47
    
    11:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    11:10:01 AM         2       231      4.63      4.48      6.49
    11:20:01 AM         2       309     73.26     54.97     27.27
    11:30:01 AM         2       271      1.68     11.20     17.52
    11:40:01 AM         3       244      5.08      5.15     11.27
    11:50:01 AM         2       313     53.20     29.30     18.71
    12:00:01 PM         2       223      1.43     10.97     16.00
    12:10:01 PM         3       254      3.21      3.42      9.33
    12:20:01 PM         2       232      3.04      2.55      6.05
    12:30:01 PM         4       214      4.32      3.39      4.64
    12:40:13 PM         5       313     27.70     24.75     14.27
    12:50:01 PM         2       288     10.56     15.85     16.26
    01:00:01 PM         3       216      1.05      3.70      9.50
    01:10:01 PM         1       241      1.92      5.92      8.35
    01:20:01 PM         2       316     54.19     34.36     18.99
    01:30:01 PM         2       213      3.33     10.49     14.00
    01:40:01 PM         3       234      4.86      4.99      9.14
    01:50:01 PM         2       243      9.73     11.92     11.20
    02:00:01 PM         2       220      5.78      8.41      9.21
    02:10:01 PM         2       231      3.87      4.82      6.95
    02:20:01 PM         2       224      4.76      4.61      5.62
    02:30:01 PM         3       228      1.41      3.70      5.37
    02:40:01 PM         3       252      3.10      4.01      4.60
    
    02:40:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    02:50:01 PM         2       231      3.07      3.79      4.53
    03:00:03 PM         2       319     27.38     14.55      8.95
    03:10:01 PM         2       226      4.17      8.28      9.53
    03:20:01 PM         2       358     16.61     16.15     12.25
    03:30:01 PM         2       293     30.77     39.20     26.62
    03:40:01 PM         2       238      3.64     16.06     21.69
    03:50:01 PM         2       249     11.61     24.09     23.27
    04:00:03 PM         1       349     40.80     33.00     25.51
    04:10:01 PM         2       289     24.49     26.66     24.34
    Average:            2       231      6.72      6.78      6.36
    
    Iostat doesn't look much different than when I first posted, though:

    Code:
    # iostat
    Linux 2.6.32-431.5.1.el6.x86_64 (server1.example.com)    04/02/2014      _x86_64_(2 CPU)
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               5.89    0.63    3.92    2.66    0.07   86.84
    
    Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
    xvdb              0.27         1.24         1.02     976420     805592
    xvda             48.77       169.81       557.79  134086558  440438298
    
    In Munin, all of the graphs are super high. VMStat shows "I/O Sleep" at 9.52, where it's usually at 0. CPU Usage - iowait is at 149.2, and it's usually at 0.5.

    IO Service Time also had a major shift on the chart, but I don't know how to read or explain it; in Munin, the red line is usually stable halfway between 1e03 and 1e04, but now it is erratic and halfway between 1e06 and 1e07.

    Disk Utilization in Munin is usually at around 500m, but it's currently at 95.54 (not 95m, but 95; so, 200 times higher than usual).

    Disk Latency is usually around 180m, but is currently at 2.49.

    If it matters, I'm using WHM 11.42.0 (build 23).

    Any suggestions? Any changes to cPanel at 8:50am that could cause this?
     
  6. GoWilkes

    GoWilkes Well-Known Member

    Joined:
    Sep 26, 2006
    Messages:
    367
    Likes Received:
    1
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    We think that this problem was hardware failure, not software, so this is all probably irrelevant now :-(

    Man, this has been a bad day...
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    654
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Thank you for updating us with the outcome.
     
Loading...

Share This Page