Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Tracking IMAP bandwidth use

Discussion in 'E-mail Discussion' started by LoadFactor, Apr 3, 2019.

  1. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    I have a user that blew our their bandwidth allocation overnight. Over 3GB in the span of a few minutes.

    There's one IMAP account with just 24MB of mail, so a massive new device resync isn't the issue.

    I've gone through the mail delivery reports and there's two 5MB messages and a normal level of sapm that didn't get past greylisting. Grepped /var/log/maillog matching the domain for anything unusual, nothing obvious came up.

    LFD reports nothing in terms of some brute force attempt.

    I'm stumped. Where else should I be looking?
     
  2. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    I also just looked at munin... there's no corresponding spike in network traffic, no unusual change in system load. Nothing. Meanwhile the cPanel bandwidth report shows two narrow spikes, one over 300 MB/min.
     
  3. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,815
    Likes Received:
    443
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    cPanel (3.23 GB)
    WHM (3.23 GB)

    It's not a question of the statistic, it's tracking down what it is in IMAP that used it. cPanel is reporting these spikes in traffic but munin's network traffic is normal. With no spikes in system load or reports from LFD, it suggests that it wasn't a DOS attack. And there's nothing even close to 3GB in the mail delivery logs. Total mail volume on that account looks like it's under 25 MB... I would have been getting mail queue size alerts anyway.
    Clipboard 1.jpg Clipboard 2.jpg
     
  5. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,815
    Likes Received:
    443
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    I did mention I had grepped maillog. This is why I'm stumped. Bytes in: 37,862, bytes out: 1,074,256
    Code:
    Mar 31 03:56:11 <server> dovecot: imap(<client-email>)<28277><GXIdPF+FMcFBXE5Y>: Logged out in=572, out=1623, bytes=572/1623
    Mar 31 03:56:11 <server> dovecot: imap(<client-email>)<28278><s3okPF+FMsFBXE5Y>: Logged out in=417, out=1440, bytes=417/1440
    Mar 31 05:03:38 <server> dovecot: imap(<client-email>)<3649><G2QELWCFM8FBXE5Y>: Logged out in=417, out=1443, bytes=417/1443
    Mar 31 05:03:38 <server> dovecot: imap(<client-email>)<3650><oaUELWCFNMFBXE5Y>: Logged out in=572, out=1626, bytes=572/1626
    Mar 31 05:59:25 <server> dovecot: imap(<client-email>)<6376><XSRT9WCFN8FBXE5Y>: Logged out in=417, out=1440, bytes=417/1440
    Mar 31 05:59:25 <server> dovecot: imap(<client-email>)<6375><YyJT9WCFOMFBXE5Y>: Logged out in=572, out=1631, bytes=572/1631
    Mar 31 06:59:36 <server> dovecot: imap(<client-email>)<12317><iwsWzGGFOsFBXE5Y>: Logged out in=572, out=1631, bytes=572/1631
    Mar 31 06:59:36 <server> dovecot: imap(<client-email>)<12315><5u4VzGGFOcFBXE5Y>: Logged out in=417, out=1440, bytes=417/1440
    Mar 31 07:58:33 <server> dovecot: imap(<client-email>)<18411><H1h0n2KFO8FBXE5Y>: Logged out in=551, out=1555, bytes=551/1555
    Mar 31 07:58:33 <server> dovecot: imap(<client-email>)<18414><Ue18n2KFPMFBXE5Y>: Logged out in=135, out=950, bytes=135/950
    Mar 31 07:58:38 <server> dovecot: imap(<client-email>)<18487><jc6kn2KFPcFBXE5Y>: Logged out in=417, out=1432, bytes=417/1432
    Mar 31 08:31:06 <server> dovecot: imap(<client-email>)<8446><WI7rE2OFQcFBXE5Y>: Logged out in=587, out=2212, bytes=587/2212
    Mar 31 08:31:06 <server> dovecot: imap(<client-email>)<8457><drD1E2OFQsFBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
    Mar 31 08:31:07 <server> dovecot: imap(<client-email>)<8463><pa39E2OFQ8FBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
    Mar 31 08:31:08 <server> dovecot: imap(<client-email>)<8473><+5EKFGOFRMFBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
    Mar 31 09:14:55 <server> dovecot: imap(<client-email>)<3764><CBapsGOFXMFBXE5Y>: Logged out in=183, out=6707, bytes=183/6707
    Mar 31 09:14:56 <server> dovecot: imap(<client-email>)<3765><36CqsGOFXcFBXE5Y>: Logged out in=277, out=2019, bytes=277/2019
    Mar 31 09:15:00 <server> dovecot: imap(<client-email>)<3799><gbL1sGOFXsFBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Mar 31 09:20:04 <server> dovecot: imap(<client-email>)<3757><AyCisGOFW8FBXE5Y>: Logged out in=872, out=66405, bytes=872/66405
    Mar 31 09:20:04 <server> dovecot: imap(<client-email>)<3747><xgmasGOFWsFBXE5Y>: Logged out in=508, out=2265, bytes=508/2265
    Mar 31 17:36:33 <server> dovecot: imap(<client-email>)<19814><aJeXsmqFtMFBXE5Y>: Logged out in=535, out=2043, bytes=535/2043
    Mar 31 17:36:33 <server> dovecot: imap(<client-email>)<19819><1i6ksmqFtsFBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
    Mar 31 17:36:34 <server> dovecot: imap(<client-email>)<19831><BdyqsmqFt8FBXE5Y>: Logged out in=267, out=1147, bytes=267/1147
    Mar 31 17:36:37 <server> dovecot: imap(<client-email>)<19873><YCvasmqFuMFBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Mar 31 17:46:03 <server> dovecot: imap(<client-email>)<27101><JNOW1GqFzMFBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Mar 31 17:47:12 <server> dovecot: imap(<client-email>)<27030><lj861GqFycFBXE5Y>: Logged out in=598, out=2486, bytes=598/2486
    Mar 31 17:47:12 <server> dovecot: imap(<client-email>)<27032><YqFA1GqFysFBXE5Y>: Logged out in=536, out=1607, bytes=536/1607
    Mar 31 17:47:12 <server> dovecot: imap(<client-email>)<27034><MGhD1GqFy8FBXE5Y>: Logged out in=380, out=2940, bytes=380/2940
    Mar 31 19:50:59 <server> dovecot: imap(<client-email>)<11919><f5pXk2yFAMJBXE5Y>: Logged out in=271, out=7407, bytes=271/7407
    Mar 31 19:51:00 <server> dovecot: imap(<client-email>)<11936><LNZrk2yFA8JBXE5Y>: Logged out in=409, out=1431, bytes=409/1431
    Mar 31 19:51:00 <server> dovecot: imap(<client-email>)<11938><e9txk2yFBMJBXE5Y>: Logged out in=180, out=985, bytes=180/985
    Mar 31 19:51:01 <server> dovecot: imap(<client-email>)<11943><DK96k2yFB8JBXE5Y>: Logged out in=308, out=1217, bytes=308/1217
    Mar 31 19:51:01 <server> dovecot: imap(<client-email>)<11942><u9R1k2yFBcJBXE5Y>: Logged out in=178, out=993, bytes=178/993
    Mar 31 19:51:04 <server> dovecot: imap(<client-email>)<11982><Lcyzk2yFCMJBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  1 07:42:34 <server> dovecot: imap(<client-email>)<28346><CX0qhHaFG8JBXE5Y>: Logged out in=271, out=6257, bytes=271/6257
    Apr  1 07:42:36 <server> dovecot: imap(<client-email>)<28378><jKdEhHaFH8JBXE5Y>: Logged out in=582, out=2135, bytes=582/2135
    Apr  1 07:42:36 <server> dovecot: imap(<client-email>)<28452><73ZWhHaFIMJBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
    Apr  1 07:42:39 <server> dovecot: imap(<client-email>)<28487><P7R8hHaFIsJBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  1 07:42:39 <server> dovecot: imap(<client-email>)<28467><N31hhHaFIcJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
    Apr  1 07:42:40 <server> dovecot: imap(<client-email>)<28503><r82JhHaFI8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
    Apr  1 07:46:42 <server> dovecot: imap(<client-email>)<28347><r7MqhHaFHMJBXE5Y>: Logged out in=1264, out=26999, bytes=1264/26999
    Apr  1 07:46:42 <server> dovecot: imap(<client-email>)<28355><WZcyhHaFHsJBXE5Y>: Logged out in=874, out=3412, bytes=874/3412
    Apr  1 18:57:51 <server> dovecot: imap(<client-email>)<533><OAYz83+FUsJBXE5Y>: Logged out in=243, out=6804, bytes=243/6804
    Apr  1 18:57:51 <server> dovecot: imap(<client-email>)<536><GbM683+FU8JBXE5Y>: Logged out in=171, out=31495, bytes=171/31495
    Apr  1 18:57:52 <server> dovecot: imap(<client-email>)<539><BehB83+FVcJBXE5Y>: Logged out in=173, out=19850, bytes=173/19850
    Apr  1 18:57:52 <server> dovecot: imap(<client-email>)<544><RaZI83+FWMJBXE5Y>: Logged out in=183, out=6025, bytes=183/6025
    Apr  1 18:57:52 <server> dovecot: imap(<client-email>)<543><Vk9I83+FV8JBXE5Y>: Logged out in=304, out=1223, bytes=304/1223
    Apr  1 18:57:53 <server> dovecot: imap(<client-email>)<556><CQ5O83+FWcJBXE5Y>: Logged out in=262, out=1141, bytes=262/1141
    Apr  1 18:57:53 <server> dovecot: imap(<client-email>)<570><9i1S83+FWsJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
    Apr  1 18:57:53 <server> dovecot: imap(<client-email>)<574><1ttY83+FW8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
    Apr  1 18:57:56 <server> dovecot: imap(<client-email>)<598><kk6C83+FXMJBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  1 18:59:20 <server> dovecot: imap(<client-email>)<532><4/Iy83+FUcJBXE5Y>: Logged out in=2267, out=413637, bytes=2267/413637
    Apr  1 18:59:20 <server> dovecot: imap(<client-email>)<537><IE8783+FVMJBXE5Y>: Logged out in=292, out=2081, bytes=292/2081
    Apr  1 18:59:20 <server> dovecot: imap(<client-email>)<540><XPtB83+FVsJBXE5Y>: Logged out in=417, out=1460, bytes=417/1460
    Apr  1 19:59:20 <server> dovecot: imap(<client-email>)<11500><aLoZz4CFcsJBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  1 20:00:40 <server> dovecot: imap(<client-email>)<11434><8J/FzoCFccJBXE5Y>: Logged out in=1018, out=3200, bytes=1018/3200
    Apr  1 20:00:40 <server> dovecot: imap(<client-email>)<11433><fgbFzoCFcMJBXE5Y>: Logged out in=956, out=4071, bytes=956/4071
    Apr  2 15:09:26 <server> dovecot: imap(<client-email>)<12672><hNYl4JCFmMJBXE5Y>: Logged out in=273, out=8594, bytes=273/8594
    Apr  2 15:09:27 <server> dovecot: imap(<client-email>)<12683><Q2c24JCFm8JBXE5Y>: Logged out in=183, out=8527, bytes=183/8527
    Apr  2 15:09:27 <server> dovecot: imap(<client-email>)<12678><0Xwv4JCFmsJBXE5Y>: Logged out in=717, out=6323, bytes=717/6323
    Apr  2 15:09:28 <server> dovecot: imap(<client-email>)<12687><AVBE4JCFncJBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
    Apr  2 15:09:28 <server> dovecot: imap(<client-email>)<12690><++FI4JCFnsJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
    Apr  2 15:09:28 <server> dovecot: imap(<client-email>)<12691><+kBO4JCFn8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
    Apr  2 15:09:31 <server> dovecot: imap(<client-email>)<12734><p+N74JCFoMJBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  2 15:10:42 <server> dovecot: imap(<client-email>)<12675><hN0m4JCFmcJBXE5Y>: Logged out in=1470, out=138647, bytes=1470/138647
    Apr  2 15:10:42 <server> dovecot: imap(<client-email>)<12684><YYk34JCFnMJBXE5Y>: Logged out in=417, out=1443, bytes=417/1443
    Apr  2 17:28:26 <server> dovecot: imap(<client-email>)<17270><xGs70ZKFr8JBXE5Y>: Logged out in=271, out=9034, bytes=271/9034
    Apr  2 17:28:27 <server> dovecot: imap(<client-email>)<17286><4WhT0ZKFssJBXE5Y>: Logged out in=372, out=1356, bytes=372/1356
    Apr  2 17:28:27 <server> dovecot: imap(<client-email>)<17295><zxpZ0ZKFs8JBXE5Y>: Logged out in=262, out=1141, bytes=262/1141
    Apr  2 17:28:28 <server> dovecot: imap(<client-email>)<17301><w29e0ZKFtMJBXE5Y>: Logged out in=307, out=1233, bytes=307/1233
    Apr  2 17:28:28 <server> dovecot: imap(<client-email>)<17304><X0Bk0ZKFtcJBXE5Y>: Logged out in=267, out=1139, bytes=267/1139
    Apr  2 17:28:31 <server> dovecot: imap(<client-email>)<17361><DbmY0ZKFtsJBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  2 19:11:51 <server> dovecot: imap(<client-email>)<18274><amchQ5SF28JBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  2 19:14:48 <server> dovecot: imap(<client-email>)<18243><dyjSQpSF2sJBXE5Y>: Logged out in=1071, out=2656, bytes=1071/2656
    Apr  2 19:14:48 <server> dovecot: imap(<client-email>)<18242><Ow3SQpSF2cJBXE5Y>: Logged out in=927, out=4013, bytes=927/4013
    Apr  2 19:32:50 <server> dovecot: imap(<client-email>)<7128><YZYqjpSF6sJBXE5Y>: Logged out in=330, out=1269, bytes=330/1269
    Apr  2 19:32:50 <server> dovecot: imap(<client-email>)<7132><DSsvjpSF68JBXE5Y>: Logged out in=180, out=993, bytes=180/993
    Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7144><IQ00jpSF7cJBXE5Y>: Logged out in=225, out=1085, bytes=225/1085
    Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7149><9Rg6jpSF78JBXE5Y>: Logged out in=226, out=1069, bytes=226/1069
    Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7148><EP05jpSF8MJBXE5Y>: Logged out in=179, out=1007, bytes=179/1007
    Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7134><5CwvjpSF7MJBXE5Y>: Logged out in=242, out=1477, bytes=242/1477
    Apr  2 19:34:03 <server> dovecot: imap(<client-email>)<7114><Y0McjpSF6MJBXE5Y>: Logged out in=595, out=2220, bytes=595/2220
    Apr  2 19:34:03 <server> dovecot: imap(<client-email>)<7120><lboijpSF6cJBXE5Y>: Logged out in=417, out=1443, bytes=417/1443
    Apr  2 19:34:03 <server> dovecot: imap(<client-email>)<7145><9g80jpSF7sJBXE5Y>: Logged out in=178, out=993, bytes=178/993
    Apr  3 07:56:19 <server> dovecot: imap(<client-email>)<4007><nnIY8Z6FKMNBXE5Y>: Logged out in=114, out=582, bytes=114/582
    Apr  3 07:56:20 <server> dovecot: imap(<client-email>)<4016><kIkh8Z6FK8NBXE5Y>: Logged out in=171, out=39835, bytes=171/39835
    Apr  3 07:56:21 <server> dovecot: imap(<client-email>)<4031><tuEl8Z6FLMNBXE5Y>: Logged out in=414, out=1436, bytes=414/1436
    Apr  3 07:56:21 <server> dovecot: imap(<client-email>)<4039><URAw8Z6FLcNBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
    Apr  3 07:56:21 <server> dovecot: imap(<client-email>)<4014><8IQe8Z6FKsNBXE5Y>: Logged out in=564, out=3301, bytes=564/3301
    Apr  3 07:56:22 <server> dovecot: imap(<client-email>)<4046><h0I68Z6FLsNBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
    Apr  3 07:57:29 <server> dovecot: imap(<client-email>)<3967><oCC/8J6FJ8NBXE5Y>: Logged out in=598, out=2468, bytes=598/2468
    Apr  3 07:57:29 <server> dovecot: imap(<client-email>)<4008><LRsZ8Z6FKcNBXE5Y>: Logged out in=1204, out=160361, bytes=1204/160361
     
    #6 LoadFactor, Apr 3, 2019
    Last edited by a moderator: Apr 3, 2019
  7. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,815
    Likes Received:
    443
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    You did but you didn't include what you looked for specifically.


    Feel free to open a ticket so that our analysts can track down the source of the discrepancy and possibly shed some light on the issue for you. You can open the ticket using the link in my signature and once open please update here with the ticket ID so we can update this thread with the outcome.

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    Done. The Support Request ID is: 11848183
     
  9. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,815
    Likes Received:
    443
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @LoadFactor

    Great, I'm watching that ticket and will update here with the findings as soon as available.


    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    With the caveat that I still need to work with the user to figure out why this is happening, it seems that the secret is in the grep! My initial grep was on the user's email and the volume of data returned made it hard to find the issue. The linked grep only reported a subset of bandwidth consuming commands. In particular just one UID SEARCH seems to have used a gigabyte!

    A better command sees to be:

    cat /var/log/maillog | grep "account@domain.com" | grep "bytes="

    I've also posted this on the SOLVED - One account transfered 50 Gb of data on IMAP in one month thread.
     
    cPanelLauren likes this.
  11. LoadFactor

    LoadFactor Active Member

    Joined:
    Jul 12, 2013
    Messages:
    42
    Likes Received:
    6
    Trophy Points:
    83
    cPanel Access Level:
    Root Administrator
    I have finally identified what happened here. The end user's Android phone updated and for some reason the mail client on their phone started making multiple UID SEARCH requests to IMAP. For some reason, dovecot saw fit to return anything from the usual <2 KB though to responses over 1 GB, often in the order of 40 MB. This on a mailbox with only 21 MB of mail. The problem continued overnight, so IMAP traffic ran up to about 40GB total. I do not want to see that end user's phone bill!

    We took a scorched earth approach to this: wiped out everything in the mail account, deleted the mail app, wiped data, and reinstalled. Hopefully this puts an end to it.
     
  12. cPanelLauren

    cPanelLauren Forums Analyst II Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    5,815
    Likes Received:
    443
    Trophy Points:
    233
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @LoadFactor

    Wow, that's a pretty interesting case though, I'm glad you found the cause of the issue and I believe it will be good to have this as a reference in the event someone else runs into this issue as well.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice