Do you have "usrquota" set up for the partition your sites are on? Check /etc/fstab for something like the following - where you can see /dev/sda2 has "usrquota" enabled:
If not, I've found the least complicated way to enable quotas is to edit /etc/fstab manually, adding "usrquota" for the partition(s) needed, reboot the server and then run "/scripts/initquotas; /scripts/fixquotas" from shell once it has rebooted.
# /etc/fstab: static file system information.
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/dev/sda1 none swap sw 0 0
/dev/sda2 / ext3 noatime,nodiratime,usrquota,errors=remount-ro 0 1
/usr/tmpDSK /tmp ext3 defaults,noauto 0 0
(Disclaimer: Take caution when editing /etc/fstab if you're not 100% sure of what you're doing. As with any significant system file, you are best advised to back it up before making any changes. Should failure result, you should be able to reboot in rescue mode and restore the backup)