Getting to know rpm.versions

[cybermonk]

I had a DNS server last night fire a high load for a hour. It was stuck running apache distiller. Solved for now as killing the distiller allowed upcp to proceed. A forced upcp then succeeded without problem.

Now I have multiple DNS-only's reported :-

"Can't locate httpd.conf at /usr/local/bandmin/ipaddrmap line 37"

These DNS-only's also now have mysql installed and it's not used. Mysql is not selected in the service manager and a DNS-only can't run apache anyway.

All is okay but I'm trying to get to grips with the new rpm.versions system and would like to suggest an improvement that might help us all migrate to this new feature.

So we can make sure targets are set to what we want, it would be nice to have a option :-

update_local_rpm_versions --status
also a target setting of "prohibit" would be good to force removal of a target if it got installed an/or even if another "3rd party?" file turned it on.

The status option would list every possible target and it's status. This status would be the final determination and also which file in /var/cpanel/rpm.versions.d/ has led to this decision.

If we had this and it used the same mechanism as upcp uses to determine what to do, then we would be able to confidently set a cpanel server up to install/remove/unmanage the targets we want/need.

Right now I feel reluctant to mess with the rpm.versions settings but I want the necessary targets maintained and I have others that I want removed altogether. I'm not sure of all the target names, there doesn't seem to be one for apache for instance, and also why any target is now getting installed when there is no setting in local.versions for it and it's not enabled to run.

As I say, all is okay at the moment so no need to raise a ticket but I wanted to suggest it here as there must be others that are having similar concerns.

 

[cPanelKenneth]

Thank you for your feedback. We are investigating a number of improvements, including UIs, for the RPM Version system. One of the improvements is very similar to your --status proposal; a way of telling whether a given target will be fulfilled by cPanel, or by a third party (and which 3rd party if possible).

 

[cybermonk]

Super Kenneth, thanks for the update. Your product is excellent keep up the good work. With the improvements and watching some of the agressive comments (re feature requests) it's been a worry of mine that WHM would become restricted and loose the flexibility I've been used too. It's good to know that we are still going to be able to control whether it just runs email or http etc and also know what will happen on an update.
Many thanks again.

 

[Rubas]

We see the '"Can't locate httpd.conf at /usr/local/bandmin/ipaddrmap line 37" error too with 11.36.0.20 DNS only.

 

[cybermonk]

Yes Rubas,
still getting "Can't locate httpd.conf at /usr/local/bandmin/ipaddrmap line 37." on my two DNS-Only's. I thought 11.36.0.18 was supposed to fix that but I'm still getting it ever night.

Also a high load on the Full-CP every time it's up's to a new target :-

My setup has a full CP as main DNS controller but it only runs DNS, mysql and exim, all other services are shutdown, FTP, httpd etc, . There are two slave servers that are CP-DNS-Only. The high load occurs only on the full CP master DNS controller. The high load does not occur on the CP-DNS-Only servers.
Happened again last night as the up to 11.36.0.21 occurred.
Logged in an killed /usr/local/cpanel/bin/apache_conf_distiller
Then the update proceeded and load drop to normal. Anyone know what I should/could put in /var/cpanel/rpm.versions.d/local.versions to maybe stop it trying to distill apache?

Cheers

 

[cybermonk]

Just realising that I have other Full-CP's that don't run apache either and they are not complaining. Maybe 11.36.0.18 only fixed if for them and the fix didn't get passed to the CP-DNS-Only fork?

 

[springdc]

We are getting the same error every night at midnight (via the cron running)

Can't locate httpd.conf at /usr/local/bandmin/ipaddrmap line 37.

Did cpanel support ever fix this for you? Any updates on this from cPanel by chance?

It is only affecting coming from our DNS-ONLY boxes.

 

[cybermonk]

No springdc still getting them here too.

Yes, I'm fairly sure I got them originally on Full-CP's not running apache but now (as with you) )it's only the DNS-Only's.

It's only a notice email and not affecting performance etc, so I'm not as concerned as I am about the high load caused by distiller on a Full-CP's not running apache after every new target release . That does affect performance so I'd rather see that remedied first, or at least a way to tell rpm-versions to not do it.

Cheers.

 

[fbau]

Also getting Can't locate httpd.conf at /usr/local/bandmin/ipaddrmap line 37 on 3 different Dnsonly istalls

 

[cPanelKenneth]

The issue with bandmin being noisy about missing httpd.conf is fixed in 11.36.1.4.

cybermonk, if you have zero need for the apache_conf_distiller to run, I can offer the following (untested) work around:

1. Edit /usr/local/cpanel/bin/apache_conf_distiller
2. Immediately after the copyright header section, add the following:

exit;

3. Save your changes
4. Add the apache_conf_distiller to the cpanelsync exclude file:

echo /usr/local/cpanel/bin/apache_conf_distiller >> /etc/cpanelsync.exclude

Now each time the distiller is executed, it will immediately exit. More info on cpanelsync.exclude is here: Exclude Functionality for cpanelsync

Please note that there is no telling what may not work within cPanel & WHM by preventing the distiller from running. This is not a condition we support or test.

 

[cybermonk]

Thanks Kenneth,

Hmm...your answer however poses me a problem. I understand the work around but you say "if I have zero need for the apache_conf_distiller to run" !!

Well I don't like the idea of doing something that's unsupported. I haven't done so yet with any CP setup and don't want to start doing so now if I can help it. So I find myself saying, if I need to leave apache_conf_distiller to run to keep my install supported then ... I need it.

I've added IPV6 support to apache and loads of other refinements but all have been done as per your recommendations on the docs/twiki with a lot of attention paid to your advice on having the customisations survive upgrades.

CP has always allowed me to do what I need. Not always via the GUI interface but there has always been some way of doing it. I'm beginning to think that this may be just a problem with one machine.
No one else is asking how to have rpm-versions exclude apache as a lack of it's use is causing high load on an target version change.

I also have realised that I have full CP's with only FTP running and also MX's and MTA's that are not running apache and indeed easyapache wasn't run on any of these ones either (except what may be auto-default done during the actual initial install).
At the last up from 11.36.0.20 to 11.36.0.21 it was only this one DNS master that fired the high load with a (seemingly) hung distiller process.

I don't like daemons running that are not needed or used so I disable anything redundant. E.g. mysql doesn't run on the DNS-only's for instance.

Should I raise a ticket for this machine or should I maybe try to run an easyapache config (pretending to setup a minimum apache set up) then shut apache back down to see if that in some way clears the problem on the next cpup ?
If I try the latter I could also, this time, tail the update log before I kill the distiller to see at what stage the update says it's getting stuck at. That might give a clue as to what cpup is upping and what distiller might be waiting for or attempting to do.

I'd appreciate your advice and thanks again for once more coming back to us.

 

[cybermonk]

Okay we upped to 11.36.1.5 last night and a few things happened.

The master DNS (Full-CP) fired a high load again as before and
my CP-DNS-Only's started their mysqld.

Now I realise this is probably starting to get off topic a little so if you feel we should start a new topic here then I'll understand. But I feel the questions are related.

First I think I may have a clue as to what's happened here. This MasterDNS was originally spec'd as such and I knew it would never be running apache or anything else for that matter. I believe I used the advanced "disable_easyapache_builds method" (or similar) to prevent apache being built at install time. From memory I think it was a option in /root/cpanel_profile/cpanel.config but that file no longer exists so I can't be certain.

Both this DNS-Master and all my other CP's that are not running apache have /etc/httpddisable on there.

This DNS-master however is the only one that has this problem and it seems to be due to missing files.

here is the relevant section of the update log rom last night :

[20130415.215314]      [19366] [Starting zonetemplates]
[20130415.215314]      [19366] [Finished zonetemplates (0 seconds)]
[20130415.215314]      [19366] [Starting apache]
[20130415.215315]      [19366] Enabled scgi-bin since suexec is not installed
[20130415.215316]      [19366] Could not determine apache version at /usr/local/cpanel/scripts/ensure_includes line 14.
[20130415.233911]      [19366] Failed to create symlink to apxs at /usr/local/cpanel/install/Apache.pm line 133.
[20130415.233911]      [19366] Disabling Alias /interchange-5 /usr/local/cpanel/3rdparty/interchange/share/interchange-5/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?kpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?cpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAlias /scgi-bin /usr/local/cpanel/cgi-sys/scgiwrap
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling Alias /interchange /usr/local/cpanel/3rdparty/interchange/share/interchange/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling Alias /java-sys /usr/local/cpanel/java-sys/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling Alias /img-sys /usr/local/cpanel/img-sys/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAlias /cgi-sys /usr/local/cpanel/cgi-sys/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?whm/?$ /usr/local/cpanel/cgi-sys/whmredirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?securewhm/?$ /usr/local/cpanel/cgi-sys/swhmredirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling Alias /bandwidth /usr/local/bandmin/htdocs/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?securecontrolpanel/?$ /usr/local/cpanel/cgi-sys/sredirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling Alias /sys_cpanel /usr/local/cpanel/sys_cpanel/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?securecpanel/?$ /usr/local/cpanel/cgi-sys/sredirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?controlpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling Alias /akopia /usr/local/cpanel/3rdparty/interchange/share/akopia/
[20130415.233911]      [19366] 
[20130415.233911]      [19366] Disabling ScriptAliasMatch ^/?webmail/?$ /usr/local/cpanel/cgi-sys/wredirect.cgi
[20130415.233911]      [19366] 
[20130415.233911]      [19366] [Finished apache (6357 seconds)]
[20130415.233911]      [19366] [Starting bandmin-pass]
[20130415.233911]      [19366] Already performed.
[20130415.233911]      [19366] [Finished bandmin-pass (0 seconds)]
[20130415.233911]      [19366] [Starting bandwidth]
[20130415.233911]      [19366] Already performed.
[20130415.233911]      [19366] [Finished bandwidth (0 seconds)]
[20130415.233911]      [19366] [Starting boxtrapper]
[20130415.233911]      [19366] You must use --force to run this utility once it has already been run.
[20130415.233911]      [19366] [Finished boxtrapper (0 seconds)]
[20130415.233911]      [19366] [Starting chkservdsetup]
[20130415.233913]      [19366] [Finished chkservdsetup (2 seconds)]
[20130415.233913]      [19366] [Starting chshsecurity]

As before I simply killed the distiller process and the upcp proceeded.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     19927 99.6  3.2  68432 24904 ?        R    21:53 104:37 /usr/local/cpanel/bin/apache_conf_distiller

Now here is the same relevant section from another server that is not running apache either but it doesn't exhibit the same problem :

[20130416.002304]      [24034] [Starting zonetemplates]
[20130416.002304]      [24034] [Finished zonetemplates (0 seconds)]
[20130416.002304]      [24034] [Starting apache]
[20130416.002305]      [24034] Disabled scgi-bin since suexec is installed
[20130416.002307]      [24034] Updating default: pidfile
[20130416.002307]      [24034] Updating default: lockfile
[20130416.002307]      [24034] Updating default: listen
[20130416.002307]      [24034] Updating default: group
[20130416.002307]      [24034] Updating default: extendedstatus
[20130416.002307]      [24034] Updating default: serveradmin
[20130416.002307]      [24034] Updating default: servername
[20130416.002307]      [24034] Updating default: loglevel
[20130416.002307]      [24034] Updating default: serversignature
[20130416.002307]      [24034] Updating default: directoryindex
[20130416.002307]      [24034] Updating default: listen
[20130416.002307]      [24034] Dropping template item for default pidfile
[20130416.002307]      [24034] Dropping template item for default lockfile
[20130416.002307]      [24034] Dropping template item for default listen
[20130416.002307]      [24034] Dropping template item for default group
[20130416.002307]      [24034] Dropping template item for default extendedstatus
[20130416.002307]      [24034] Dropping template item for default serveradmin
[20130416.002307]      [24034] Dropping template item for default servername
[20130416.002307]      [24034] Dropping template item for default loglevel
[20130416.002307]      [24034] Dropping template item for default serversignature
[20130416.002307]      [24034] Dropping template item for default directoryindex
[20130416.002307]      [24034] Dropping template item for default documentroot
[20130416.002307]      [24034] Dropping template item for default serverroot
[20130416.002307]      [24034] Dropping template item for default timeout
[20130416.002307]      [24034] Dropping template item for default user
[20130416.002307]      [24034] Distilled successfully
[20130416.002307]      [24034] [Finished apache (3 seconds)]
[20130416.002307]      [24034] [Starting bandmin-pass]
[20130416.002307]      [24034] Already performed.
[20130416.002307]      [24034] [Finished bandmin-pass (0 seconds)]
[20130416.002307]      [24034] [Starting bandwidth]
[20130416.002307]      [24034] Already performed.
[20130416.002307]      [24034] [Finished bandwidth (0 seconds)]
[20130416.002307]      [24034] [Starting boxtrapper]
[20130416.002307]      [24034] You must use --force to run this utility once it has already been run.
[20130416.002308]      [24034] [Finished boxtrapper (1 seconds)]
[20130416.002308]      [24034] [Starting chkservdsetup]
[20130416.002309]      [24034] [Finished chkservdsetup (1 seconds)]
[20130416.002309]      [24034] [Starting chshsecurity]
[20130416.002309]      [24034] [Finished chshsecurity (0 seconds)]

Now I know this thread was originally about rpm.versions but my question is this:

rpm.versions as I understand it controls the installation and update of rpm's. How do we now control whether or not a service will be installed at install time and also whether or not it will be enabled. The CP-DNS-Only servers for instance have MySQL un-ticked under "Server manager" and yet they all fired their MySQL daemons up last night during the upgrade. MySQL (again as I understand it) is not needed for DNS-Only and so (as those of you that know me might expect) I have it turned off. Please correct me if I'm wrong there.
I've also checked :

root@dnsonly ~# chkconfig --list mysql
mysql          	0:off	1:off	2:off	3:off	4:off	5:off	6:off
root@dnsonly ~#

So how do we have happen what we need. At both install time and during updates

Lastly, it is not really efficient to have a server spec'd to build apache and have it rebuild all that when is just a DNS server. The same goes for other servers that maybe only run exim for example. Even if this machine does have the resources needed and manages to build apache (which I'm not sure it does), I don't really want it tied up when an major update comes out, churning away rebuilding something it will never run.

I've tried to find the advanced install option to prevent easyapache running at install time and can't find it. Has it been deprecated. Did I dream it. I know I didn't do anything that wasn't on All Documentation or not recommended. Since I have another similar cluster to do soon, what is the rpm.versions equivalent for selecting what daemons/services to do at install and also why is the Service manager not doing what it says it will.

At the moment I don't know if this is a bug, the initial use of a deprecated option or something I've done wrong. I know I didn't have any of these issues before 11.34.1.12 so something has changed and quite a few of my servers haven't handled the transition. It's as if they all want to be full multi-service servers instead of the single dedicated service servers that I planned.

All was happy before with 11.34.1.12, if the setting up of this structure has changed then how do I now implement it again, properly and supported.

Many thanks.

 

[cPanelKenneth]

cybermonk,

Your posts are rather long, I apologize if I miss something

Apache is not delivered via RPM, hence it is not managed by rpm.versions. A lot of things that happen outside the context of an RPM are also not managed by rpm.versions.

There are at least two ways to prevent EasyApache from running during cPanel & WHM installation:

1. Run the installer (latest) with --skipapache
2. Touch /root/skipapache prior to running the installer

Neither of the above methods prevent someone from running EasyApache post-installation. To prevent that, create /scripts/preeasyapache with the following contents:

#!/bin/bash
exit 1

The installer will automatically skip running EasyApache if it detects that it is installing DNSONLY.

The apache conf distiller sole purpose is to attempt to preserve manual modifications to httpd.conf. Since you are not using Apache on your machine, you should have no need for the distiller to run. The distiller updates templates in /var/cpanel, used to generate httpd.conf

MySQL was added to DNSONLY a few versions ago when we added support for cPHulkd. Also if you desired to use MyDNS, MySQL is required.

Without examining your servers it is difficult for me to offer suggestions why the behavior (with Apache items) during upcp is different. My suggestion is to look at how they differ from the Apache, as cPanel & WHM view apache, perspective. For example the contents of /usr/local/apache and how they are the same or different between the two servers.

The behavior of Service Manager changed somewhat for 11.36. It will still manage whether a service is started or not (started by chkservd, that is). In prior versions it could also be used (unintentionally) to prevent installation of RPMs. That bit of functionality was removed.

For RPMs that were provided prior to 11.36 (e.g. dovecot, pure-ftpd) you can still change whether they are installed during cPanel & WHM installation by using the information here Advanced Options: Pre-Installation

For the newer RPMs make appropriate entries in /var/cpanel/rpm.versions.d. On already installed systems you can create the appropriate changes using scripts/update_local_rpm_versions, then copy the file(s) created in /var/cpanel/rpm.versions.d to new servers prior to installing cPanel & WHM. Some additional information is here How to Customize and Override the rpm.versions System

 

[cybermonk]

Many thanks, very informative and no need to apologise. I apologise for the long post but it kinda all got tangled together.

All understood but a few questions :-

First, just for anyone reading re: the DNS-Only/CPHulkd : Yes I knew mysql would be needed for CPHulkd but I don't think CPHulkd is of any use/protection to a DNS server that only has port 53 open. That's the reason I have both off. Does CPHulk provide any use in that scenario ?
If you were running anything else on there dovecot, exim etc. then yes you'd want CPHulkd on (and MySQL).

You explained the change in the service manager behaviour, no longer (unintentionally), prevents installation (cool didn't know that); but why did it start mysqld when it was ticked off ?

You said that

Neither of the above methods prevent someone from running EasyApache post-installation.

Easyapache won't be run (manually by anyone) on any of these servers as they aren't and never will serve up web pages. Will a upcp ever do that as part of a run ; and if so will the /root/skipapache method prevent that? Or do I need to implement the /scripts/preeasyapache method to prevent that?

I remember reading somewhere (another thread perhaps) discussions about Easyapache vs YUM. I thought "eventually" even apache was going over to rpm ? (Custom ones from CPanel not the OS repo ones). If so will the /root/skipapache automatically update rpm.versions so that doesn't happen if it's not wanted ?

I'll start checking that server against the others and see if I can figure it out. I'll also possibly try the installer with --skipapache on a test server to see what the files look like with that option and how it passes upcp target change without issue. If I discover the answer/solution I'll post back here.

Again Kenneth, many thanks
Starting to get a feel of it now and more confident about future upcp's.

 

[cPanelKenneth]

Use of cPHulkd is up to you, regardless of what services you are running. In your specific scenario I agree it doesn't make sense to use cPHulkd.

I'm not 100% confident service manager (which is really chkservd) started MySQL. Rather I suspect the post section in the MySQL RPM configured MySQL to start. It may be possible to narrow down by examining the update logs (/var/cpanel/updatelogs) and the chkservd log (/var/log/chkservd.log) for entries around the time MySQL was updated. Other system logs (e.g. the MySQL log) may also shed light on what happened.


/root/skipapache is only used by the installer. Nothing else uses it.

upcp generally won't run easyapache. We explicitly prevent upcp and easyapache from running at the same time. If you use the MySQL Upgrade wizard in WHM, that can trigger a run of EasyApache. Using the /scripts/preeasyapache hook will prevent all unintentional executions of EasyApache.

You are correct. The long term goal of EasyApache is to provide the applications via prebuilt RPMs. In fact the EasyApache team is currently working on Tomcat 7 RPMs as their first deliverable.

When we introduce new ways of doing things, we make attempts to preserve prior behavior in the initial version where we introduce the new thing. Then in the next, or future, version, we remove the prior methods of doing things. This allows us to continue evolving our product lines, and provides a means for people to transition from old to new.

I don't want to make any promises, since we are not at a point in development where we would consider preservation (or even changing) the behavior of /root/skipapache, however it will be brought up in our discussions when we reach that point in development.

 

[cybermonk]

Super Kenneth, many thanks.
Okay I'll start hunting for server differences and a possible rectification of this one non-apache install.
I'll also hunt through the logs as well for the point at which MySQL got started.
If I find any clues for either or even a fix I'll post back.