Disk 100% full, but still multiple GB left on device

[DennisMidjord]

Hi,

We have this odd problem where df reports the disk as 100% full, but we could still have 20GB free on that disk. When the disk is 100% full, we can no longer write anything to the disk.
Any idea why this happens?

 

[cPanelLauren]

Hi @DennisMidjord

Can you show me an example of this? as well as the output of the following:

cd /
du -h --max-depth=1

 

[DennisMidjord]

Hi @cPanelLauren
I don't have an example right now - we usually extend the disk when we see there's an issue. The 'df -h' command will just report 100% full on /home, even though it has multiple GB left.

 

[cPanelLauren]

Hi @DennisMidjord

There was a point I was going to make with that. This is a common concern and the best example is shown If you use df and du you'll a lot of times encounter a situation where they show differing amounts. This is due to the way that they each calculate disk space and would account for why you are seeing available disk space when df is telling you there's nothing available.

df measures disk space in 512-byte disk sectors and on some file systems like zFS df is sometimes inaccurate in providing enough information to determine if the file system is running out of space. df is measuring File System usage

The great comparison is du which a lot of times can take a lot longer because it shows File Space usage

Another thing to keep in mind as well is that df adds deleted items still held in memory to the calculation as they are by all purposes still present until they're no longer held in memory. You can see this usage by running something like:

lsof |grep '(deleted)'

 

[DennisMidjord]

du -h shows this: 614G ./home
df -h shows this: /dev/mapper/cloudlinux-home 676G 614G 29G 96% /home

 

[cPanelLauren]

du -h shows this: 614G ./home
df -h shows this: /dev/mapper/cloudlinux-home 676G 614G 29G 96% /home

So that difference there is what I was referring to. The difference between File Space usage and File System usage. DF Is calculating File System usage in blocks as well as counting deleted items held in memory where DU isn't. Sometimes this will lead to potentially showing more disk space used.

 

[DennisMidjord]

But it showed the same amount used? :D Both commands showed 614GB used.

 

[cPanelLauren]

Oh! I misread it without the bar present:

Filesystem      Size  Used Avail Use% Mounted on

In this case then, it does show you have available space, but I'm assuming this isn't an example where it's reporting the server is out of space?

 

[DennisMidjord]

Correct - but it did last week. I didn't save any information from back then, unfortunately.

 

[DennisMidjord]

Hello, @cPanelLauren

We now have the issue again. Could you please take a look at ticket 13798409? I've added a temporary login for FTP so you can check the issue.

 

[DennisMidjord]

Hi,

I think I figured out why. PureFTP has a setting called MaxDiskUsage set to 99 (%). On large partitions, this can be many gigabytes. I'm gonna set it to 100.

 

[cPanelLauren]

Hi @DennisMidjord

That's interesting, were you only seeing the usage in FTP?

 

[DennisMidjord]

No - we were only seeing the error message with FTP (apparently). I had understood that it was an issue with writing anything, but it seems like it was just FTP.
Writing anything to the disk from cPanel or uploading a database didn't cause any issues which led me to investigate whether it was actually just the FTP server.

 

[cPanelLauren]

Hi @DennisMidjord

Ahhh ok, that makes sense. Nice catch there!