UPDATE
I want to start by acknowledging your frustration with the time that it has taken to resolve this ongoing issue, and assure you that we are making every effort to resolve this issue as soon as possible. I'm hoping this explanation will show why this took longer to resolve than a typical high-priority case.
The process that we use to update the Apache configuration on a cPanel has gotten quite complex through the years, and there are now several different subsystems of code that come into play. To name a few:
1. The domain/account creation logic
2. The task queue (queueprocd) that we use to asynchronously perform tasks
3. The file locking logic that we use ensure that multiple processes do not attempt to read/write to httpd.conf at the same time
4. The logic to surgically insert VirtualHost's into the apache configuration
5. The logic to restart apache after the configuration has been built
Unfortunately, we have had difficulty replicating this issue in even a semi-infrequent capacity in our internal testing infrastrure, which has made it far more difficult for us to track the root cause of the issue.
After looking at the data from affected servers, we were able to identify that this was occurring when a new domain/account was added to the server, and that the issue itself was a race condition that occurred from asynchronous tasks that are offloaded to queueprocd during domain/account creation.
The first attempt to resolve this issue (102.0.12) did three things:
1. It updated queueprocd's logic to force certain processes to perform sequentially in order to remove the chances of the race condition occurring
2. It updated the domain/account creation logic so that the certain processes were offloaded to queueprocd faster
3. It added some debug logging to help us analyze the issue further in the event that the first two changes did not resolve the issue
The second attempt to resolve this issue (102.0.14) was a targeted change that added logic to rebuild the userdata in the event that it was missing (the race condition occurred). Due to the complexity of the logic that we use to surgically insert VirtualHost's into the apache configuration, it turned out that this code path was not reached every single time that a new domain/account is created on a server.
The third attempt to resolve this issue, which will be released in 102.0.15, does two things:
1. It resolves a specific race condition that was found in the logic we use to update the userdata during domain and account creation. This race condition was identified as a direct result of the additional debug logging that we added during our first attempt to resolve this issue.
2. In the event the issue is not completely resolved with the 102.0.15 update, although we do hope this takes care of things, we've added additional debug logging to one of the code path's that is taken when new domains are added to the server to help us analyze this issue further.
We plan to release 102.0.15 soon after it goes through final testing on our end.
