Tracking IMAP bandwidth use

[LoadFactor]

I have a user that blew our their bandwidth allocation overnight. Over 3GB in the span of a few minutes.

There's one IMAP account with just 24MB of mail, so a massive new device resync isn't the issue.

I've gone through the mail delivery reports and there's two 5MB messages and a normal level of sapm that didn't get past greylisting. Grepped /var/log/maillog matching the domain for anything unusual, nothing obvious came up.

LFD reports nothing in terms of some brute force attempt.

I'm stumped. Where else should I be looking?

 

[LoadFactor]

I also just looked at munin... there's no corresponding spike in network traffic, no unusual change in system load. Nothing. Meanwhile the cPanel bandwidth report shows two narrow spikes, one over 300 MB/min.

 

[cPanelLauren]

Hi @LoadFactor

Where are you looking for bandwidth usage for this account specifically? Within WHM or cPanel?

Depending on where you're seeing the usage the following documentation might be helpful:

Apparent Discrepancies in Bandwidth Usage Statistics - Version 78 Documentation - cPanel Documentation
View Bandwidth Usage - Version 78 Documentation - cPanel Documentation
Bandwidth - Version 78 Documentation - cPanel Documentation

 

[LoadFactor]

cPanel (3.23 GB)
WHM (3.23 GB)

It's not a question of the statistic, it's tracking down what it is in IMAP that used it. cPanel is reporting these spikes in traffic but munin's network traffic is normal. With no spikes in system load or reports from LFD, it suggests that it wasn't a DOS attack. And there's nothing even close to 3GB in the mail delivery logs. Total mail volume on that account looks like it's under 25 MB... I would have been getting mail queue size alerts anyway.

 

[cPanelLauren]

Hello,

The following threads go over how to identify the usage:

SOLVED - One account transfered 50 Gb of data on IMAP in one month
IMAP consuming so much bandwidth


Specifically what to look for in /var/log/maillog

 

[LoadFactor]

I did mention I had grepped maillog. This is why I'm stumped. Bytes in: 37,862, bytes out: 1,074,256

Mar 31 03:56:11 <server> dovecot: imap(<client-email>)<28277><GXIdPF+FMcFBXE5Y>: Logged out in=572, out=1623, bytes=572/1623
Mar 31 03:56:11 <server> dovecot: imap(<client-email>)<28278><s3okPF+FMsFBXE5Y>: Logged out in=417, out=1440, bytes=417/1440
Mar 31 05:03:38 <server> dovecot: imap(<client-email>)<3649><G2QELWCFM8FBXE5Y>: Logged out in=417, out=1443, bytes=417/1443
Mar 31 05:03:38 <server> dovecot: imap(<client-email>)<3650><oaUELWCFNMFBXE5Y>: Logged out in=572, out=1626, bytes=572/1626
Mar 31 05:59:25 <server> dovecot: imap(<client-email>)<6376><XSRT9WCFN8FBXE5Y>: Logged out in=417, out=1440, bytes=417/1440
Mar 31 05:59:25 <server> dovecot: imap(<client-email>)<6375><YyJT9WCFOMFBXE5Y>: Logged out in=572, out=1631, bytes=572/1631
Mar 31 06:59:36 <server> dovecot: imap(<client-email>)<12317><iwsWzGGFOsFBXE5Y>: Logged out in=572, out=1631, bytes=572/1631
Mar 31 06:59:36 <server> dovecot: imap(<client-email>)<12315><5u4VzGGFOcFBXE5Y>: Logged out in=417, out=1440, bytes=417/1440
Mar 31 07:58:33 <server> dovecot: imap(<client-email>)<18411><H1h0n2KFO8FBXE5Y>: Logged out in=551, out=1555, bytes=551/1555
Mar 31 07:58:33 <server> dovecot: imap(<client-email>)<18414><Ue18n2KFPMFBXE5Y>: Logged out in=135, out=950, bytes=135/950
Mar 31 07:58:38 <server> dovecot: imap(<client-email>)<18487><jc6kn2KFPcFBXE5Y>: Logged out in=417, out=1432, bytes=417/1432
Mar 31 08:31:06 <server> dovecot: imap(<client-email>)<8446><WI7rE2OFQcFBXE5Y>: Logged out in=587, out=2212, bytes=587/2212
Mar 31 08:31:06 <server> dovecot: imap(<client-email>)<8457><drD1E2OFQsFBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
Mar 31 08:31:07 <server> dovecot: imap(<client-email>)<8463><pa39E2OFQ8FBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
Mar 31 08:31:08 <server> dovecot: imap(<client-email>)<8473><+5EKFGOFRMFBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
Mar 31 09:14:55 <server> dovecot: imap(<client-email>)<3764><CBapsGOFXMFBXE5Y>: Logged out in=183, out=6707, bytes=183/6707
Mar 31 09:14:56 <server> dovecot: imap(<client-email>)<3765><36CqsGOFXcFBXE5Y>: Logged out in=277, out=2019, bytes=277/2019
Mar 31 09:15:00 <server> dovecot: imap(<client-email>)<3799><gbL1sGOFXsFBXE5Y>: Logged out in=114, out=582, bytes=114/582
Mar 31 09:20:04 <server> dovecot: imap(<client-email>)<3757><AyCisGOFW8FBXE5Y>: Logged out in=872, out=66405, bytes=872/66405
Mar 31 09:20:04 <server> dovecot: imap(<client-email>)<3747><xgmasGOFWsFBXE5Y>: Logged out in=508, out=2265, bytes=508/2265
Mar 31 17:36:33 <server> dovecot: imap(<client-email>)<19814><aJeXsmqFtMFBXE5Y>: Logged out in=535, out=2043, bytes=535/2043
Mar 31 17:36:33 <server> dovecot: imap(<client-email>)<19819><1i6ksmqFtsFBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
Mar 31 17:36:34 <server> dovecot: imap(<client-email>)<19831><BdyqsmqFt8FBXE5Y>: Logged out in=267, out=1147, bytes=267/1147
Mar 31 17:36:37 <server> dovecot: imap(<client-email>)<19873><YCvasmqFuMFBXE5Y>: Logged out in=114, out=582, bytes=114/582
Mar 31 17:46:03 <server> dovecot: imap(<client-email>)<27101><JNOW1GqFzMFBXE5Y>: Logged out in=114, out=582, bytes=114/582
Mar 31 17:47:12 <server> dovecot: imap(<client-email>)<27030><lj861GqFycFBXE5Y>: Logged out in=598, out=2486, bytes=598/2486
Mar 31 17:47:12 <server> dovecot: imap(<client-email>)<27032><YqFA1GqFysFBXE5Y>: Logged out in=536, out=1607, bytes=536/1607
Mar 31 17:47:12 <server> dovecot: imap(<client-email>)<27034><MGhD1GqFy8FBXE5Y>: Logged out in=380, out=2940, bytes=380/2940
Mar 31 19:50:59 <server> dovecot: imap(<client-email>)<11919><f5pXk2yFAMJBXE5Y>: Logged out in=271, out=7407, bytes=271/7407
Mar 31 19:51:00 <server> dovecot: imap(<client-email>)<11936><LNZrk2yFA8JBXE5Y>: Logged out in=409, out=1431, bytes=409/1431
Mar 31 19:51:00 <server> dovecot: imap(<client-email>)<11938><e9txk2yFBMJBXE5Y>: Logged out in=180, out=985, bytes=180/985
Mar 31 19:51:01 <server> dovecot: imap(<client-email>)<11943><DK96k2yFB8JBXE5Y>: Logged out in=308, out=1217, bytes=308/1217
Mar 31 19:51:01 <server> dovecot: imap(<client-email>)<11942><u9R1k2yFBcJBXE5Y>: Logged out in=178, out=993, bytes=178/993
Mar 31 19:51:04 <server> dovecot: imap(<client-email>)<11982><Lcyzk2yFCMJBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  1 07:42:34 <server> dovecot: imap(<client-email>)<28346><CX0qhHaFG8JBXE5Y>: Logged out in=271, out=6257, bytes=271/6257
Apr  1 07:42:36 <server> dovecot: imap(<client-email>)<28378><jKdEhHaFH8JBXE5Y>: Logged out in=582, out=2135, bytes=582/2135
Apr  1 07:42:36 <server> dovecot: imap(<client-email>)<28452><73ZWhHaFIMJBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
Apr  1 07:42:39 <server> dovecot: imap(<client-email>)<28487><P7R8hHaFIsJBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  1 07:42:39 <server> dovecot: imap(<client-email>)<28467><N31hhHaFIcJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
Apr  1 07:42:40 <server> dovecot: imap(<client-email>)<28503><r82JhHaFI8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
Apr  1 07:46:42 <server> dovecot: imap(<client-email>)<28347><r7MqhHaFHMJBXE5Y>: Logged out in=1264, out=26999, bytes=1264/26999
Apr  1 07:46:42 <server> dovecot: imap(<client-email>)<28355><WZcyhHaFHsJBXE5Y>: Logged out in=874, out=3412, bytes=874/3412
Apr  1 18:57:51 <server> dovecot: imap(<client-email>)<533><OAYz83+FUsJBXE5Y>: Logged out in=243, out=6804, bytes=243/6804
Apr  1 18:57:51 <server> dovecot: imap(<client-email>)<536><GbM683+FU8JBXE5Y>: Logged out in=171, out=31495, bytes=171/31495
Apr  1 18:57:52 <server> dovecot: imap(<client-email>)<539><BehB83+FVcJBXE5Y>: Logged out in=173, out=19850, bytes=173/19850
Apr  1 18:57:52 <server> dovecot: imap(<client-email>)<544><RaZI83+FWMJBXE5Y>: Logged out in=183, out=6025, bytes=183/6025
Apr  1 18:57:52 <server> dovecot: imap(<client-email>)<543><Vk9I83+FV8JBXE5Y>: Logged out in=304, out=1223, bytes=304/1223
Apr  1 18:57:53 <server> dovecot: imap(<client-email>)<556><CQ5O83+FWcJBXE5Y>: Logged out in=262, out=1141, bytes=262/1141
Apr  1 18:57:53 <server> dovecot: imap(<client-email>)<570><9i1S83+FWsJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
Apr  1 18:57:53 <server> dovecot: imap(<client-email>)<574><1ttY83+FW8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
Apr  1 18:57:56 <server> dovecot: imap(<client-email>)<598><kk6C83+FXMJBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  1 18:59:20 <server> dovecot: imap(<client-email>)<532><4/Iy83+FUcJBXE5Y>: Logged out in=2267, out=413637, bytes=2267/413637
Apr  1 18:59:20 <server> dovecot: imap(<client-email>)<537><IE8783+FVMJBXE5Y>: Logged out in=292, out=2081, bytes=292/2081
Apr  1 18:59:20 <server> dovecot: imap(<client-email>)<540><XPtB83+FVsJBXE5Y>: Logged out in=417, out=1460, bytes=417/1460
Apr  1 19:59:20 <server> dovecot: imap(<client-email>)<11500><aLoZz4CFcsJBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  1 20:00:40 <server> dovecot: imap(<client-email>)<11434><8J/FzoCFccJBXE5Y>: Logged out in=1018, out=3200, bytes=1018/3200
Apr  1 20:00:40 <server> dovecot: imap(<client-email>)<11433><fgbFzoCFcMJBXE5Y>: Logged out in=956, out=4071, bytes=956/4071
Apr  2 15:09:26 <server> dovecot: imap(<client-email>)<12672><hNYl4JCFmMJBXE5Y>: Logged out in=273, out=8594, bytes=273/8594
Apr  2 15:09:27 <server> dovecot: imap(<client-email>)<12683><Q2c24JCFm8JBXE5Y>: Logged out in=183, out=8527, bytes=183/8527
Apr  2 15:09:27 <server> dovecot: imap(<client-email>)<12678><0Xwv4JCFmsJBXE5Y>: Logged out in=717, out=6323, bytes=717/6323
Apr  2 15:09:28 <server> dovecot: imap(<client-email>)<12687><AVBE4JCFncJBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
Apr  2 15:09:28 <server> dovecot: imap(<client-email>)<12690><++FI4JCFnsJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241
Apr  2 15:09:28 <server> dovecot: imap(<client-email>)<12691><+kBO4JCFn8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
Apr  2 15:09:31 <server> dovecot: imap(<client-email>)<12734><p+N74JCFoMJBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  2 15:10:42 <server> dovecot: imap(<client-email>)<12675><hN0m4JCFmcJBXE5Y>: Logged out in=1470, out=138647, bytes=1470/138647
Apr  2 15:10:42 <server> dovecot: imap(<client-email>)<12684><YYk34JCFnMJBXE5Y>: Logged out in=417, out=1443, bytes=417/1443
Apr  2 17:28:26 <server> dovecot: imap(<client-email>)<17270><xGs70ZKFr8JBXE5Y>: Logged out in=271, out=9034, bytes=271/9034
Apr  2 17:28:27 <server> dovecot: imap(<client-email>)<17286><4WhT0ZKFssJBXE5Y>: Logged out in=372, out=1356, bytes=372/1356
Apr  2 17:28:27 <server> dovecot: imap(<client-email>)<17295><zxpZ0ZKFs8JBXE5Y>: Logged out in=262, out=1141, bytes=262/1141
Apr  2 17:28:28 <server> dovecot: imap(<client-email>)<17301><w29e0ZKFtMJBXE5Y>: Logged out in=307, out=1233, bytes=307/1233
Apr  2 17:28:28 <server> dovecot: imap(<client-email>)<17304><X0Bk0ZKFtcJBXE5Y>: Logged out in=267, out=1139, bytes=267/1139
Apr  2 17:28:31 <server> dovecot: imap(<client-email>)<17361><DbmY0ZKFtsJBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  2 19:11:51 <server> dovecot: imap(<client-email>)<18274><amchQ5SF28JBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  2 19:14:48 <server> dovecot: imap(<client-email>)<18243><dyjSQpSF2sJBXE5Y>: Logged out in=1071, out=2656, bytes=1071/2656
Apr  2 19:14:48 <server> dovecot: imap(<client-email>)<18242><Ow3SQpSF2cJBXE5Y>: Logged out in=927, out=4013, bytes=927/4013
Apr  2 19:32:50 <server> dovecot: imap(<client-email>)<7128><YZYqjpSF6sJBXE5Y>: Logged out in=330, out=1269, bytes=330/1269
Apr  2 19:32:50 <server> dovecot: imap(<client-email>)<7132><DSsvjpSF68JBXE5Y>: Logged out in=180, out=993, bytes=180/993
Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7144><IQ00jpSF7cJBXE5Y>: Logged out in=225, out=1085, bytes=225/1085
Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7149><9Rg6jpSF78JBXE5Y>: Logged out in=226, out=1069, bytes=226/1069
Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7148><EP05jpSF8MJBXE5Y>: Logged out in=179, out=1007, bytes=179/1007
Apr  2 19:32:51 <server> dovecot: imap(<client-email>)<7134><5CwvjpSF7MJBXE5Y>: Logged out in=242, out=1477, bytes=242/1477
Apr  2 19:34:03 <server> dovecot: imap(<client-email>)<7114><Y0McjpSF6MJBXE5Y>: Logged out in=595, out=2220, bytes=595/2220
Apr  2 19:34:03 <server> dovecot: imap(<client-email>)<7120><lboijpSF6cJBXE5Y>: Logged out in=417, out=1443, bytes=417/1443
Apr  2 19:34:03 <server> dovecot: imap(<client-email>)<7145><9g80jpSF7sJBXE5Y>: Logged out in=178, out=993, bytes=178/993
Apr  3 07:56:19 <server> dovecot: imap(<client-email>)<4007><nnIY8Z6FKMNBXE5Y>: Logged out in=114, out=582, bytes=114/582
Apr  3 07:56:20 <server> dovecot: imap(<client-email>)<4016><kIkh8Z6FK8NBXE5Y>: Logged out in=171, out=39835, bytes=171/39835
Apr  3 07:56:21 <server> dovecot: imap(<client-email>)<4031><tuEl8Z6FLMNBXE5Y>: Logged out in=414, out=1436, bytes=414/1436
Apr  3 07:56:21 <server> dovecot: imap(<client-email>)<4039><URAw8Z6FLcNBXE5Y>: Logged out in=262, out=1149, bytes=262/1149
Apr  3 07:56:21 <server> dovecot: imap(<client-email>)<4014><8IQe8Z6FKsNBXE5Y>: Logged out in=564, out=3301, bytes=564/3301
Apr  3 07:56:22 <server> dovecot: imap(<client-email>)<4046><h0I68Z6FLsNBXE5Y>: Logged out in=308, out=1225, bytes=308/1225
Apr  3 07:57:29 <server> dovecot: imap(<client-email>)<3967><oCC/8J6FJ8NBXE5Y>: Logged out in=598, out=2468, bytes=598/2468
Apr  3 07:57:29 <server> dovecot: imap(<client-email>)<4008><LRsZ8Z6FKcNBXE5Y>: Logged out in=1204, out=160361, bytes=1204/160361

 

[cPanelLauren]

I did mention I had grepped maillog

You did but you didn't include what you looked for specifically.


Feel free to open a ticket so that our analysts can track down the source of the discrepancy and possibly shed some light on the issue for you. You can open the ticket using the link in my signature and once open please update here with the ticket ID so we can update this thread with the outcome.

Thanks!

 

[LoadFactor]

Done. The Support Request ID is: 11848183

 

[cPanelLauren]

Hi @LoadFactor

Great, I'm watching that ticket and will update here with the findings as soon as available.


Thanks!

 

[LoadFactor]

With the caveat that I still need to work with the user to figure out why this is happening, it seems that the secret is in the grep! My initial grep was on the user's email and the volume of data returned made it hard to find the issue. The linked grep only reported a subset of bandwidth consuming commands. In particular just one UID SEARCH seems to have used a gigabyte!

A better command sees to be:

cat /var/log/maillog | grep "[email protected]" | grep "bytes="

I've also posted this on the SOLVED - One account transfered 50 Gb of data on IMAP in one month thread.

 

[LoadFactor]

I have finally identified what happened here. The end user's Android phone updated and for some reason the mail client on their phone started making multiple UID SEARCH requests to IMAP. For some reason, dovecot saw fit to return anything from the usual <2 KB though to responses over 1 GB, often in the order of 40 MB. This on a mailbox with only 21 MB of mail. The problem continued overnight, so IMAP traffic ran up to about 40GB total. I do not want to see that end user's phone bill!

We took a scorched earth approach to this: wiped out everything in the mail account, deleted the mail app, wiped data, and reinstalled. Hopefully this puts an end to it.

 

[cPanelLauren]

Hi @LoadFactor

Wow, that's a pretty interesting case though, I'm glad you found the cause of the issue and I believe it will be good to have this as a reference in the event someone else runs into this issue as well.

 

[Metro2]

After noticing that I have a customer who in recent months has been averaging over 1GB per day in just IMAP bandwidth despite the fact that he's really only doing about 50MB a day of data transfer, I came here to the trusty forum just now to see if anyone else ever encountered such a thing and found this thread right away. (Thank you to the OP and others who responded!)

So using grep commands and sifting through his /var/log/maillog entries, I discovered at least a clue to the source of the problem and I'll be following up with the customer today to see what we can determine what's really happening, but so far here's what I discovered and might be helpful to others who end up finding this thread as well...

When he logs in to his mail from his normal devices through his ISP connection, from what I can tell, *looks like* his IMAP mail transactions are normal / expected sizes like this - "Logged out in=1462, out=19339, bytes=1462/19339" (so basically typical 5kb to 50kb emails)

But then I noticed a pattern which *looks like* may be coming from an Outlook.com or Exchange IP, since there are also steady connections coming from a prod.outlook.com IP address IP address - every 30 minutes an IMAP connection to his account occurs , and shows a 54MB transaction, almost all like this - "Logged out in=2336, out=54564254, bytes=2336/54564254"

One thing I can see for sure is this - every 30 minutes an IMAP connection occurs with a 54MB outbound transaction (yet there are NO signs of anything even close to that large in Mailscanner > MailControl search of his email address - largest one showing in there during the past 3 days is only 5MB).

Now, I'm a bit sleep-deprived at the moment so I'm not 100% sure if it's actually the outlook.com login connection or his regular ISP login connections that are generating the out = 54MB transactions, but now at least I have a starting point based on finding this thread (and other linked within) and more closely inspecting the maillog.

When I do finally figure out which connection is causing it and manage to resolve the issue, I'll come back here and post an update, hopefully more helpful than the info above so far.

Big thanks to everyone here on the forums who has posted about this! First time I've ever encountered this "IMAP using tons of bandwidth for no obvious reason" issue.