It mostly depends of the type of hosting environment you are serving.
It should be set to ON if you are serving shared hosting accounts, resellers.
The main issue is that a good amount of *old* PHP applications require safe_mode to be set to Off.
If you dont experience any customer complain while having safe_mode set to ON I would suggest to keep it that way.
Fortunately safe_mode will be gone on PHP6 so you wouldn't have to worry about it anymore.