Create accounts for 6,000 domains, Bulk Load In?

[SteveInSpain]

Hello again:

Back in March '10 I posted a request for assistance to be able to do a bulk creation of accounts and then a load in of 1,000s of domains in one go (see: Bulk Load In of many domains possible?)

The time is finally here, and I do need some help. This post will be much easier than the one in March.

One issue:

Can I create 6,000 accounts and then assign each one of the 6,000 domains to an account. In one single bulk operation?

These domains/URLs will resolve to what is essentially a parked "holding" one-page website. There will be no sub domains, no email, nothing except very standard log file/traffic analysis package. Very simple no frills config, as these holding sites are temporary.

cPanelDavidG kindly suggested that I look at developer resources, and that there was an API that would do just this function.

I have looked through the resource and can't seem to find this particular API.

So, DavidG or anyone else in the know, if you are reading this, could you point me to the API?

Many thanks.

Steve

[cPanelDavidG]

Hello again:

Back in March '10 I posted a request for assistance to be able to do a bulk creation of accounts and then a load in of 1,000s of domains in one go (see: Bulk Load In of many domains possible?)

The time is finally here, and I do need some help. This post will be much easier than the one in March.

One issue:

Can I create 6,000 accounts and then assign each one of the 6,000 domains to an account. In one single bulk operation?

These domains/URLs will resolve to what is essentially a parked "holding" one-page website. There will be no sub domains, no email, nothing except very standard log file/traffic analysis package. Very simple no frills config, as these holding sites are temporary.

cPanelDavidG kindly suggested that I look at developer resources, and that there was an API that would do just this function.

I have looked through the resource and can't seem to find this particular API.

So, DavidG or anyone else in the know, if you are reading this, could you point me to the API?

Many thanks.

Steve

It sounds like you're looking for the createacct API function:

CreateAccount < AllDocumentation/AutomationIntegration < TWiki

I've moved your thread to the Developer Discussions section so you are likely to receive relevant followups to any questions you ask about the APIs.

If you are not familiar with our APIs yet, I recommend visiting Developer Resources

[SteveInSpain]

It sounds like you're looking for the createacct API function:

CreateAccount < AllDocumentation/AutomationIntegration < TWiki

I've moved your thread to the Developer Discussions section so you are likely to receive relevant followups to any questions you ask about the APIs.

If you are not familiar with our APIs yet, I recommend visiting Developer Resources

Hey David...

Many thanks. This might just do the trick.

I will pass this on to my devs and it will make much more sense to them. I am not the developer, but it is my project, so I am trying to help them out by doing the research to lighten their load.

I would be ecstatic if we could do this all in one go in a very simplistic set up. Beggars can't be choosers.

But then, I did see several parameters in this API that could be very interesting if they can be tweaked with some creative and clever coding to fit my ultimate requirement:

I have the portfolio organized into 21 functional Categories, inside of each Category are Sub Categories (10 to 100 sub cats, depending). It would be incredibly convenient if I could have this same organization in cPanel, as it would make management so much more easy.

Could I use the "plan (string)" parameter for each of the Categories, and either "pkgname (string)" or some form of nested "plan (string)" for the Sub Categories?

Then, after creation and load in, in cPanel I would be able to see and access domains — in a hierarchical folder tree format or similar: Category > Sub Category > domain accounts, by name*, in this Cat/Sub Cat?

*The domain URLs are composed of a brand term and an unique generic term, e.g., www.brandgeneric1.com, www.brandgeneric2.com... I would be happy to see only the generic term as title of each account, if this is possible... don't need the www, or the brand, or the domain's gTLD.

As always, your assistance is greatly appreciated.

All the best,

Steve

[SteveInSpain]

I posted an hour or so ago, and got a message that it's awaiting moderation.

Did my response get lost in the shuffle?

Thanks,

Steve

[sirdopes]

You don't even need to use the api to create an account. You can just use /scripts/createacct. Just loop through something like the following 6000 times and you've got your accounts. If you already have the skeleton directory setup, it should put the files in the public_html folder you when the account is created.

/scripts/createacct $domain $username $pass


This is assuming that you have full root access. If not, then you would need to use the api. The following is some perl that can be used. This would need to be looped through to generate each account.


use LWP::UserAgent;
use MIME::Base64;

my $user = "user";
my $pass = "password";
my $auth = "Basic " . MIME::Base64::encode( $user . ":" . $pass );

my $ua = LWP::UserAgent->new;
my $request =
HTTP::Request->new( GET =>
"http://domain.com:2086/xml-api/createacct?username=$user&ip=n&cpmod=x3&password=$pass&domain=$domain&useregns=0&reseller=0&forcedns=1"
);
$request->header( Authorization => $auth );
my $response = $ua->request($request);
print $response->content;
}

[SS-Maddy]

Even a simple shell script with wget-ting WHM and its' support for POST method also can assist you in this process Plugins like LiveHTTP Headers and Firebug will be handy in this project. It is all just looping.

[SteveInSpain]

You don't even need to use the api to create an account. You can just use /scripts/createacct. Just loop through something like the following 6000 times and you've got your accounts. If you already have the skeleton directory setup, it should put the files in the public_html folder you when the account is created.

/scripts/createacct $domain $username $pass


This is assuming that you have full root access. If not, then you would need to use the api. The following is some perl that can be used. This would need to be looped through to generate each account.


use LWP::UserAgent;
use MIME::Base64;

my $user = "user";
my $pass = "password";
my $auth = "Basic " . MIME::Base64::encode( $user . ":" . $pass );

my $ua = LWP::UserAgent->new;
my $request =
HTTP::Request->new( GET =>
"http://domain.com:2086/xml-api/createacct?username=$user&ip=n&cpmod=x3&password=$pass&domain=$domain&useregns=0&reseller=0&forcedns=1"
);
$request->header( Authorization => $auth );
my $response = $ua->request($request);
print $response->content;
}

Yes, we do have full root access. We have our own dedicated server.

And, looking at the Create Account API, it seems rather straightforward because this is exactly the kind of automation we are looking for.

Now I have to ask a few dumb questions, so please bear with me:

1. My dev doesn't know PERL, can this script be rewritten in PHP and still accomplish the same goal?

2. Uh, where do we put the script, and then how do we run it, after we've created it?

3. (Optional) We have multiple IPs that come with our server. One private, one public and 6 more publics at our disposal. We want to distribute the portfolio of domains across all of the public IPs if the hosting company will let us. Could we batch the script for each IP? Or, with each run of the loop the script just goes down the list of available IPs?

Thanks...

Steve

[cPanelDavidN]

Hi Steve,

Looks like you're getting alot of good advice, so I won't repeat or sway you to use one method over the other.

However, I want to make sure and point out that /scripts/createacct (which is the same as /scripts/wwwacct) takes slightly different parameters than the XML-API's createacct. Both work just fine, they're just a little different and you can't assume their input is the same.

Also, if I recall correctly, the default values may differ slightly too. So, if you're not using packages/plans, then you might need to test descrepancies between the two methods for 'optional' parameters.

Don't hold me to it, but I think 'locale' and 'language' are not completely synonymous...All docs I'm seeing ATM say 'language' is what you're suppose to use for both methods...but I've seen code in the past that was scripted as such, *shrug*.

Here's a snippet from some code I had lying around; it's Perl. The context is I had several scripts that used the XML-API to create accounts, but I needed to alter a few of them to create the account via CLI. So, this is part of the function that transforms my new_user_hash that was designed as XML-API input into input for a /scripts/wwwacct. It should be enough to clue you in on what I'm talking about:

    #handle XMLAPI->CLI value diffs
    # $acct is the hash ref I used for XMLAPI
    # @ordered_values is what I concatenate and
    #      pass to /scripts/wwwacct
    #  NOTE: this is just the exception differs
    #   and may not be inclusive of all value diffs
    #   it's just an example...

if($_ eq 'cgi' && $acct->{$_} =~ /\d/){
	push( @ordered_values, ($acct->{$_} )? 'y':'n' );
}
elsif($_ eq 'ip' && $acct->{$_} =~ /\d/){
	push( @ordered_values, ($acct->{$_} )? 'y':'n' );
}
elsif($_ eq 'frontpage' && $acct->{$_} =~ /\d/){
	push( @ordered_values, ($acct->{$_} )? 'y':'n' );
}
elsif($_ eq 'hasshell' && $acct->{$_} =~ /\d/){
	push( @ordered_values, ($acct->{$_} )? 'y':'n' );
}
elsif($_ =~ qr/max.*/ && $acct->{$_} =~ qr/unlimited/){
	push( @ordered_values, 'default');

}

Good Luck,
-DavidN

[SteveInSpain]

Hello again all:

I have communicated all of your great comments and suggestions to my cPanel Authorized Partner NOC via a support thread I started with them some days ago.

Their response, after days of wrangling with them is: We installed it on your server for you, but that's all we're going to do.

I don't mind telling you who they are either, because this level of atrocious service — and scam — needs to be exposed.

100TB is the hosting company that leased me my fully "managed" server, and I also paid for their "Supreme Support" service, which promised: "expert assistance... you just sit back and relax...we do everything for you"

I have been in email communication with 100TB's General Manager who promised a day or so ago that he would "get it sorted". Uh, that was then, and since then this line of communication has gone dead, as has the support thread.

So folks, I am completely on my own here, and I really need some help.

What's more, when I open my cPanel, I get an error message (and daily emails) that the installation is not correct!

I have reviewed cPanels's Admin, and it is very well organised, but I can't even begin to know where to start.

I am a complete newb, my field is in business operations, not coding or the intricacies of server management. My developer's speciality is in CMS and PHP coding, also not in server setup.

So I repeat my plea.... can someone help me?

Here's what I think I know:

1. For my "Categories" I can set up "reseller" accounts as a cheat. PLEASE NOTE: I HAVE ABANDONED THE IDEA OF SUB CATEGORIES as described above in one of my earlier posts. We have to go KISS.

2. Somehow, I can then setup "add on" accounts under each Category/reseller accounts for my domains.

3. Then, my developer can somehow code a script (PHP) to take the Drupal nodes, which are generated in XHTML and created to be stand alone websites each with a simple URL, and somehow import them into the appropriate folder in each Add On account.

But that's as far as I understand right now.

Thank you guys for the PERL scripts coding, but my dev doesn't know PERL, so we're lost there too.

As to opinions and other advice... I wouldn't know what you were saying to me, and I really need to focus on topic right now, and get this thing done.

I'm not a dummy, so if someone can give me instruction in easy to digest, baby steps, paint-by-the-numbers, then maybe I can do this thing myself.

Please help.

Steve

[cPanelDavidN]

Hi Steve,

I'm sorry to hear that your NOC is not meeting your expectations.

As far as how you intend to implement a business structure around domain creation/existence, I'm at a lost to give adequate guidance. Maybe it's because I fail to fully understand the full scope of the business model.

Administration of cPanel accounts and parts of WHM are totally doable in PHP. You and your developer should scour the various pages over at sdk.cpanel.net.

The first method I'd look at is the XMLAPI (as mentioned previously). You're in luck because there's an XMLAPI PHP client class. You can download it here, http://sdk.cpanel.net/lib/xmlapi/php..._v1.0.6.tar.gz , and find general documentation on the WHM/cPanel XMLAPI interface here, XML and JSON APIs . This PHP class is a simple client interface that allows you to make HTTP/HTTPS requests to perform WHM or cPanel actions and retrieve information. Most anything that is done natively in the cPanel interface can be emulated with the proper API1/API2 call via the XMLAPI. There are a few examples in the client class tarball. But basically, any single cPanel action is just one URL POST away! You can extend this class or snip out the parts that work for you (say as pluggin for Drupal)

If you need to code PHP in a more direct manor within cPanel served pages, you should look at LivePHP and cPanelPHP.

LivePHP is a PHP class designed to be used on you cPanel box that essentially gives you a direct pipe to the cPanel API parser. You instantiate you PHP object and feed it a cPanel tag and will receive an XML response. You can find more on LivePHP here, LivePHP -PHP Class for cPanel-. LivePHP class is also extendable. You could reimplemented this class entirely, but I've be careful altering the actual file as it's possible to get overwritten with the original version when cPanel system files are synced.

cPanelPHP is built right into the cPanel PHP interpretor and can be considered a bonus to using PHP files within the standard cPanel web serving paths. cPanelPHP leverages PHP's interoperability and allows you to parse cPanel API tags first, then PHP (within PHP tags: <?php ?>), and lastly HTML (at the browser). It's also possible to have the PHP parsed first, followed by the cPanel API tags. All you have to do for cPanelPHP is have the file in a theme directory (or one of it's descendants) and have the proper file extension (.php or phpcp, respectively). You can find more on cPanelPHP here, CpanelPhp < AllDocumentation/AutomationIntegration < TWiki

If you have specific questions, let me know. Your PHP developer should be able to integrate with one or more of the methods above.

Regards,
-DavidN

[Eric]

Howdy,

I ask you submit a support request.

I have my conserns about the warning emails your getting. I'd hate for us to do all of this work and/or the server suffer some catastrophic error and we loose all of our hard work.

Feel free to mark the ticket attn Eric and I'll be happy to serve as your advocate.

Thanks!

[SteveInSpain]

Howdy,

I ask you submit a support request.

I have my conserns about the warning emails your getting. I'd hate for us to do all of this work and/or the server suffer some catastrophic error and we loose all of our hard work.

Feel free to mark the ticket attn Eric and I'll be happy to serve as your advocate.

Thanks!

Hey Eric:

By support ticket, do you mean through cPanel or 100TB/SupremeSupport?

Because my cPanel license comes through 100TB, I can't create a support ticket directly with cPanel.net.

The first thing I am asked when I do try a support ticket with CP.net is the IP of my server. Then I get a message saying "contact Authorized Partner"

Here is the warning email I get:

IMPORTANT: Do not ignore this email.
The hostname (xxx.xxxxxxx.com) resolves to . It should resolve to xxx.xxx.xxx.xxx. Please be sure to correct /etc/hosts as well as the 'A' entry in zone file for the domain.

Some are all of these problems can be caused by
/etc/resolv.conf being setup incorrectly. Please check this file if you
believe everything else is correct.


You may be able to automatically correct this problem by using the ' Add an A entry for your hostname ' under ' Dns Functions ' in your Web Host Manager

Thanks,

Steve

[SteveInSpain]

Hi Steve,

I'm sorry to hear that your NOC is not meeting your expectations.

As far as how you intend to implement a business structure around domain creation/existence, I'm at a lost to give adequate guidance. Maybe it's because I fail to fully understand the full scope of the business model.

Regards,
-DavidN

Hi David:

Yeah, I'm pretty sorry about my NOC, too.

Let me boil down my business model to its essence, as regards cPanel. I'll spare you all of the value proposition and marketing message, and I'll keep my explanation in simple non-tech terms, because I'm a non-tech:

1. I have an extensive portfolio of domains that will be eventually licensed to 3rd parties. These domains are organised into 21 Categories (discussed in second part below)

2. Until they are licensed, I have created one-page "holding" websites for them which are published to the web as stand alone websites, each with it's unique URL. Each website has unique content.

3. Clearly, I need to host these websites to get them on the web. I am using cPanel more as a management tool, rather than its traditional role of managing a full blown hosting company with myriad customers and variable hosting requirements.

4. These websites have been created from a Drupal CMS. Once dynamically generated, they can be cached, as content changes very rarely.

5. If I do need to modify content, I do this in Drupal, and simply "refresh". So, this refreshed page does need to be sent also to the hosting account.

5.1 Or the cPanel instance will call the Drupal instance for the page. Don't know which is better and affords less server overhead.

6. IMPORTANT: Once they are licensed, the licensee will develop an extensive, full-featured website, which will not be hosted on my server or through my cPanel instance.

7. That is, my server, my Drupal instance and my cPanel instance are only for the internal management my domain portfolio. Once a particular domain is licensed, all hosting activities for that domain will take place off my server. That is, the licensed domain will be hosted on the Licensee's hosting package. Nothing to do with me.

8. All of these "holding" domains and websites have minimal hosting requirements: NO email accounts (all incoming email, spam or otherwise to be bounced or "disappeared", and there will never be outgoing email). None of the other frills that a hosting customer needs. Bare bones...just enough to get them published to the web. Although I do need DNS management.

9. I will continue to use cPanel as my permanent internal management tool, to "point" the licensed domain to the new licensee's host DNS, to add and delete domains, to deny access to the Licensee due to non-payment or violation of TOS, etc..

NOW, STRUCTURE FOR CPANEL:

As a "cheat" it has been suggested:

1. I set up 21 "reseller" accounts, one for each of my Categories.

2. Because I am using cPanel for my internal organisation, I am the contact for all of the reseller/Category accounts. Don't need email.

3. Inside of each of these Category (reseller) accounts, I create "add on" accounts, one for each of the individual domains/websites described above.

4. Inside each "add on" account, I place the Drupal-generated XHTML page in the public_html folder. This is what is suggested, anyway, and it makes sense to my non-techie, pretzel logic.

5. To create all of these "add on" accounts we use the Create Account API, so we can do it in bulk.

6. Then, and only then, when these "add on" accounts are pre-populated with the necessary settings, my developer creates a script (still unsure how to do) to "import" or "link" the Drupal page into the correct "add on" account.

7. As to IPs, I have 7 available, and the Category/Reseller accounts will be spread across them.

--------

Addendum: WHMCMS will be integrated into WHM for billing purposes. I mention this only as a heads up. WHMCMS integrates beautifully with cPanel/WHM

--------


This is how I understand it, folks, in my non-tech mind.

Hope this clears up the issue...

Steve

[cPanelDavidN]

Hi Steve,

Thanks for the clarification. I see only one issue with the structure you've proposed. WHM/cPanel accounts are designed to have their content isolated from one another. You have three options


1) alter vhosts in Apache to essentially ignore cPanel structure and just point to you various Drupal nodes

2) push your Drupal pages into individual cPanel account's "public_html" directory

3) push your Drupal pages, grouped, into a cPanel account and use the Addon Domain feature.

Item one is a bit drastic and kind of works against the idea of you getting a cPanel box, so I don't think it's a valid solution.

Item two would be the simplest only because it's a one to one relationship. Foreach Drupal generated website there's one cPanel account instance. All things considered, I think I'd choose this method if I had a box that was adequate for thousands of accounts. (since there won't be any activity other than single page web serving, I don't see any problem with it)

Item three is probably the closest to your original idea. The 21 accounts do not need to be reseller accounts to utilize Addon Domains, but they can be if you want. Addons will make a vhost entry for the domain and point that domain to a subdirectory within the cPanel account public_html. You'll still need to push you Drupal content into these subdirectories.

So, to do method #2 you have to make a XMLAPI createacct call and migrate the Drupal page for each domain.

To do method #3 you need to make the XMLAPI createacct call 21 times (or just create 21 accounts manually), and you'll need to make the API2 AddonDomain::addaddondomain call (via XMLAPI) 6000 times, varying the cPanel account user appropriately for each "category" of the domain and supplying a valid subdirectory (possibly just the domain name). Then, you can migrate the Drupal content into those subdirectories. This is pretty much just a re-hash of what you said in the second part of your last post.

As far as "import" or "link" to the Drupal content. That's kind of an interest issue. I assume that the Drupal CMS is not creating dynamic sites as much as it's rendering dynamic content. That is, it's not creating and saving a .html file but instead will populate a template based on the node or some mod_rewrite content. I could be assuming to much. But if that's true, and like you said it can be cached, then I'd write a simple bash script that uses something like wget to fetch the page into an output file, index.html, and move that into the proper directory. Alternatively, you PHP developer could use cURL and fopen to fetch and save the pages. Anytime the content changes in Drupal, you fire off this "poll and populate" script. Heck, if you do it in PHP, then you should be able to write it as a hook based plugin so that it happens 'automagically'. That said, if you have any AJAX or relative based resources linked in your HTML or JS, then you'll have problems. In that case, you might just clone the entire Drupal application to each account or subdirectory instead of just the respective domain content page. At that point, you'd just do a few updates in the the Drupal config to make such you're got the correct base directory, etc.

I think you've got a few choices to make concerning the final implementation, but let us know if you have any specific questions.

-DavidN

[Eric]

Hey Eric:

By support ticket, do you mean through cPanel or 100TB/SupremeSupport?

Because my cPanel license comes through 100TB, I can't create a support ticket directly with cPanel.net.

The first thing I am asked when I do try a support ticket with CP.net is the IP of my server. Then I get a message saying "contact Authorized Partner"

Here is the warning email I get:



Thanks,

Steve

PM me with your server's IP when you get a chance.

Thanks!