If you're using swap memory then you have a real risk of disk thrashing (assuming these are spindle disks and not SSD).
Just my opinion... but if you are using more than 1 or 2GB of swap memory, then your server is overloaded. You shouldn't be swapping out memory like that. If you're using more than 24GB of memory, then the server is overloaded.
If you have 8GB of RAM (maybe as little as 4GB) then having swap memory is less of a concern. If your server is needing more RAM than that, then you need a bigger server.
About the only time swap space makes sense is for low memory VPSs that don't necessarily see a lot of activity but need to temporarily hold something in memory. Such as ClamAV. ClamAV can consume 1GB of memory on its own. If a VPS only has 1GB or 2GB of memory, then having swap space available for ClamAV to sit in might be beneficial. But if that VPS is receiving a lot of mail or otherwise using ClamAV a lot then it's going to be constantly swapping that memory in and out of swap space - disk thrashing.