Creating accounts slow on new server

[tomsie1000]

Hello all,

I work for as a sysadmin at a University and am in the process of building a cPanel instance on CentOS 7 to host students' coursework.

Since setting up the system, I have noticed creating accounts is seemingly quite slow, and it looks like something is waiting for a response and timing out, but I'm not seeing any errors.. We've done this previously on CentOS 6 and had no issues..

I've tried via the JSON api, as well as the UI and at the cli - the script always pauses at 'Validating Username......Done' for approx 2 minutes.. This delay also happens when you park a domain for some reason, so I'm feeling like it could be DNS related. DNS is disabled on this server as we have DNS servers already and no need for accounts to manage their own zones.

Running the 'wwwacct' script directly behind 'time' shows the following:

# time ./wwwacct somesite.com someuser passwrd0101
+===================================+
| New Account Info                  |
+===================================+
| Domain: somesite.com
| UserName: someuser
| PassWord: passwrd0101
+===================================+

This ok? y
Checking input data...Validating Username......Done
***2m pause here***
Validating IP......Done
Validating Contact Email......Done
...Done
Validating system setup......Done
Checking for database conflicts......Done
WWWAcct 12.6.0 (c) 2018 cPanel, Inc.......Done
Running pre creation script (/usr/local/cpanel/scripts/prewwwacct)......Done
Adding User......Done
Adding Entries to userdata......Done
Setting up Mail & Local Domains......Done
Configuring DNS......Done
Verifying MX Records and Setting up Databases......Done
Setting up Proxy Subdomains......Done
Updating Authentication Databases......Done
Adding Entries to httpd.conf......Done
Setting passwords......Done
Updating the userdata cache......Done
Setting up Horde database in the background.......Done
Creating bandwidth datastore......Done
Updating the dedicated IP address usage cache......Done
Generating and installing DKIM keys......Done
Enabling Apache SpamAssassin™......Done
Sending Account Information......Done
Running post creation scripts (/usr/local/cpanel/scripts/legacypostwwwacct, /usr/local/cpanel/scripts/postwwwacct, /usr/local/cpanel/scripts/postwwwacctuser)......Done
Setting up Domain Pointers......Done
Setting Reseller Privs......Done
Account Creation Complete!!!......Done
Account Creation Ok
Dns Zone check is enabled.


+===================================+
| New Account Info                  |
+===================================+
| Domain: somesite.com
| Ip: xxx.xxx.xxx.xxx (n)
| HasCgi: y
| UserName: someuser
| PassWord: passwrd0101
| CpanelMod: paper_lantern
| HomeRoot: /cpanel-sites
| Quota: 0 bytes
| NameServer1: xxx.ac.uk
| NameServer2: xxx.ac.uk
| NameServer3:
| NameServer4:
| Contact Email:
| Package: default
| Feature List: default
| Language: en
+===================================+
Custom Account Data Provided: no
Adding Shell Access (y)
Successlocaldomains...valiases ...vdomainaliases ...vfilters ...Zone somesite.com has been successfully added
Reconfiguring Mail Routing:
LOCAL MAIL EXCHANGER: This server will serve as a primary mail exchanger for somesite.com's mail.:
 This configuration has been manually selected.

nameserver: reload skipped on [serverhostname] (nameserver disabled)
wwwacct creation finished
Account Creation Ok

real    2m11.066s
user    0m0.504s
sys     0m0.162s

Not a huge issue but it does slow down account creation when we provision new users..

Should also note that CSF is installed and configured, however disabling it complete makes no difference to these times, so I'm fairly sure it's not a misconfigured firewall.

Appreciate any tips/thoughts/etc. as I'd like to get that time down a bit if possible..

Cheers

Tom

 

[cPanelLauren]

HI @tomsie1000

Is the server by chance using an NFS mount somewhere? If you're unsure running something like

cat /etc/fstab

or

 cat /proc/mounts

Would answer that.

 

[tomsie1000]

Hi Lauren, thanks, sorry for the delay..

I have since rebuilt the server to be absolutely sure it wasn't something I did wrong earlier. The same pause on account creation persists..

There are no NFS mounts present:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        37G  5.7G   32G  16% /
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G  4.0K  2.0G   1% /dev/shm
tmpfs           2.0G  9.1M  2.0G   1% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sdb1        40G   33M   40G   1% /cpanel-sites
/dev/sda1       976M  138M  787M  15% /boot
/dev/loop0      1.6G  2.6M  1.5G   1% /tmp
tmpfs           396M     0  396M   0% /run/user/1000

# cat /etc/fstab
UUID=a19043ee-7101-4410-a162-3eaabdecf0e9       /       xfs     defaults,uquota 0       0
UUID=f2f39603-0723-44c1-9298-a355436cde6b /boot                   ext3    defaults        1 2
UUID=1365405e-ac9a-443d-bc49-5b4a470ece69 swap                    swap    defaults        0 0
/dev/sdb1       /cpanel-sites   xfs     defaults,uquota 0       0
/usr/tmpDSK             /tmp                    ext3    defaults,noauto        0 0

 

[cPanelLauren]

Hi @tomsie1000

Thanks for the response! What about the resolv.conf can you tell me the nameservers listed there? Also, can you tell me is the server NAT routed? If so can you confirm it's a 1:1 configuration?

 

[tomsie1000]

Sure. Just to note that the local BIND instance is disabled as we have no intent to use it

Here's resolv.conf - these are our DNS servers of course. (I have redacted the actual names and addresses for obvious reasons)

# cat /etc/resolv.conf

search domain.somewhere, sub.domain.somewhere
nameserver xxx.xxx.98.15
nameserver xxx.xxx.28.59

And yes, the server has a dedicated public IP - no NAT is involved.

 

[cPanelLauren]

HI @tomsie1000


Are you aware if you're required to use those nameservers? Can you try replacing those with something like google's DNS resolvers?

8.8.8.8
8.8.4.4

And let me know if the issue persists?

Thanks!

 

[tomsie1000]

Thanks. It looks like DNS resolution is the problem, but not with the resolver per-se. Under normal circumstances we block outbound DNS queries to any other servers than our internal ones. It looks like during account creation the script is trying to work back the path of the account domain by querying the root servers directly for NS data, specifically not through the configured resolver. This explains why I was also seeing pauses when parking domains.

Allowing DNS requests to pass the firewall got rid of the pause completely, but I wanted to do some more digging.. See tcpdump output:

16:43:18.117258 IP this.server.com.46707 > d.root-servers.net.domain: 29302 [1au] NS? uk. (31)
16:43:18.129418 IP this.server.com.52837 > nsa.nic.uk.domain: 40239 [1au] NS? ac.uk. (34)
16:43:18.125975 IP d.root-servers.net.domain > this.server.com.46707: 29302- 0/8/14 (451)
16:43:18.169360 IP this.server.com.37810 > d.root-servers.net.domain: 30862 [1au] NS? nl. (31)
..etc..

So, I assume the short answer is "don't block outgoing DNS requests" however I have to ask the question; is there a cPanel option I can set to force it to query DNS using ONLY the configured resolvers or is it hard-coded to go directly?

 

[cPanelLauren]

Hi @tomsie1000

Unfortunately, because that's hardcoded I don't see a way that would work. I believe it queries the root nameservers when it does this. You might try going to WHM>>Server Configuration>>Tweak Settings and enabling Allow Remote Domains and Allow unregistered domains though I'm not confident that will stop the DNS lookup that occurs.

 

[tomsie1000]

Thanks @cPanelLauren for the confirmation and your help. I guess I could submit it as a feature request but I'd guess it's not likely to be considered. I'll just make sure that this host can query directly.

 

[cPanelLauren]

Hi @tomsie1000

You're welcome, just out of curiosity what is the specific configuration you've done to block this? I'd like to try a couple of things on my test environments, no guarantees but I'll try none the less.

Thanks!

 

[tomsie1000]

Our perimeter firewall blocks any DNS requests outbound from clients inside our network. This forces them to use our internal resolvers for recursive lookups, where we run some DNS-RPZ for anti-malware, etc.. Therefore, by default, clients on our network (including servers in our DCs) cannot query the root zone directly, only recursively through our resolvers. This works fine, so long as the applications/clients in question honour their configured resolvers.

i.e.:

under normal circumstances, I'd expect to be able to do this:

$ dig @a.root-servers.net uk

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @a.root-servers.net uk
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57965
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 8, ADDITIONAL: 14
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;uk.                            IN      A

;; AUTHORITY SECTION:
uk.                     172800  IN      NS      dns1.nic.uk.
uk.                     172800  IN      NS      dns4.nic.uk.
uk.                     172800  IN      NS      nsa.nic.uk.
uk.                     172800  IN      NS      nsd.nic.uk.
uk.                     172800  IN      NS      nsc.nic.uk.
....etc....
;; Query time: 11 msec
;; SERVER: 198.41.0.4#53(198.41.0.4)
;; WHEN: Thu Aug 23 14:46:56 BST 2018
;; MSG SIZE  rcvd: 451

however due to our firewall restrictions you'd see this:

$ dig @a.root-servers.net uk

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @a.root-servers.net uk
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

So if the server/client in question is resolving only via its configured resolvers, it would work fine just doing this:

$ dig uk

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38043
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;uk.                            IN      A

;; AUTHORITY SECTION:
uk.                     10800   IN      SOA     dns1.nic.uk. hostmaster.nic.uk. 1403801968 7200 900 2419200 10800

;; Query time: 14 msec
;; SERVER: my.server.com#53(1.2.3.4)
;; WHEN: Thu Aug 23 14:41:50 BST 2018
;; MSG SIZE  rcvd: 87

Honestly I can see the rationale for querying the root servers directly to do this sort of validation. You're asking the authoritative source of information what the current state of play is. It avoids any caching issues. It just slows my particular use case down that's all :P

To test, you could set up iptables rules on your server to block outbound DNS to any servers but your local resolvers. You should then be able to reproduce the same results I get when doing so.

 

[cPanelLauren]

Hi @tomsie1000

So specifically the issue occurs when UDP port 53 outbound is blocked you could block TCP requests and still be ok. Otherwise, that's the only workaround to that I'm finding.

 

[tomsie1000]

Pretty much, yeah. That's what I'm going with anyway!