Let me first outline your requirements to ensure I fully understand.
1) The root or a reseller user use WHM to create an account.
2) Account creation is dependent on registering the new account's intended domain with a Registar (via their API)
3) If the domain registration is not successful, then account creation is unsuccessful. An error should be displayed within WHM stating the specific reason.
Well, short answer is that the various hooks systems
either spawn separate processes (which couldn't preempt account creation, ie Function Hooks, Script Hooks) or receives their information on a level that WHM don't report, ( cPanel API level, ie CustomEventHandler). Your real option will require some significate coding.
I think the best opinion for you would be a WHM Plugin
. WHM Plugins are CGI scripts, so is can be Perl or PHP. Your plugin would be a wrapper of sorts; maybe more like a mini application. You'd present a New Account form, similar to the one under Account Functions, but when the form is submitted it first attempts to register the domain, then tries to create the new account locally (either by executing /scripts/wwwacct or via the XML-API).
Even if there was a way to accomplish the 'do other stuff first and possibly fail' scenario with one of our hooks, I think you're best served doing a plugin if only for the reason that domain registration could be a rather lengthy and sophisticated bit of code. If you're customization was more application like, then you could actually do some very useful things, like besides just say there was a problem. For instance you could fully leverage either the domain registar's API or a payment gateway to ensure secure and PCI compliant transactions for the new domain.
Take a look at that stuff and post back if you have any questions,