When edititing MX records is it ok to use an IP address for MX record, or MUST it be a hostname?
Nope, sure isn't. It's "illegal" per the RFC. Don't even bother to try it, because it simply won't work.
I have a client who wants to host their site on my cpanel server, but host their mail with another provider.
If the client is using cPanel, you can use WHM to edit their MX record for their zone (Edit a DNS Zone feature). After you've edited the zone, make sure you remove the domain from /etc/localdomains and place it into /etc/remotedomains
When editing the zone, don't forget to increment the serial, and don't forget your periods. It's a pretty simple task overall, not much to it. If you need more info, just let us know.