Switching from compile-on-demand to binary packages

[cPanelKenneth]

For many years installing and using cPanel & WHM has involved compiling software on-demand. Want Apache and PHP? Run /scripts/easyapache, which builds those and all dependencies from source. Want perl installed? Download and run the perl installer from httpupdate.cpanel.net, which installs perl from source. While compiling from source has its benefits, it also has its drawbacks.

The drawbacks are more obvious when you consider what cPanel & WHM attempts to do with compile-on-demand. We are automating this on thousands of servers, each with their own subtle differences (and some no-so-subtle). A non-trivial portion of our development effort, bug fixing and support requests are consumed by getting source compilation working in environments under other peoples (yours) control. We decided this needed to change. Hence in 11.34 we are switching our model from compile-on-demand to delivery of pre-compiled RPM packages.

What is involved in this change

For many years we've provided applications via RPM. For example MySQL, Dovecot, and Exim. In 11.34 we are adding over 800 additional RPMs. These RPMs provide:

• perl 5.14
• all the perl modules that scripts/checkperlmodules formerly installed
• PHP 5.3 for cPanel & WHM
• rrdtool
• all the pear packages needed for Horde, Roundcube, phpMyAdmin, Logaholic and Squirrelmail
• the encoders used by internal PHP
• nearly everything in /usr/local/cpanel/src

The vast majority of the RPMs are for perl and CPAN modules (801 at last count).

New RPMs will have the naming convention of cpanel-APPLICATION-APPMAJORVERSION-APPVERSION-CPANELPATCH-CPANELVERSION. For example:

cpanel-perl-514-5.14.2-0.cp1134

Broken into its parts we have:

• cpanel-perl: The application is perl
• 514: perl major version of 5.14
• 5.14.2: The specific version of perl in the RPM
• -0: We have not applied any patches, therefore we use 0
• cp1134: Built for cPanel & WHM 11.34

All the new RPMs will be installed to /usr/local/cpanel, rather than to the operating system locations (e.g. /usr ). Whenever possible we fulfill dependencies (e.g. libraries) using what the operating system provides. If that is not possible, we provide the dependency ourselves with an RPM.

Along with an expanded set of RPMs we are also providing a tool you can use to reinstall RPMs. This is useful if something is modifying files managed by a package. The tool will tell you the packages that need reinstalled and allows you to perform the reinstall at your convenience.

How Something is installed

Does this mean you will find scripts like perlup, internalphpup and so forth in 11.34? No. In fact all the scripts that install the current RPMs (e.g mysqlup, eximup) are gone in 11.34. Instead RPMs are tied to the product version and are installed by our "rpm.versions" system. This system behaves somewhat like yum, or apt-get, in that you are able to configure repositories and install software from them. The rpm.versions system was introduced in 11.30, but for 11.34 is completely overhauled. More information on that later.

By and large RPM installation is handled by upcp. For now this is acceptable as the only RPMs we are delivering are required by the product. Eventually we want to provide optional RPMs (e.g. git, subversion, etc) that you can elect to install. When we reach that stage tools, both command line and graphical, will be developed to facilitate package installation outside of upcp.

Source Compiles

All the source RPMs will be provided on our mirrors. Customers that want to apply patches to the software we provide, or just like compiling software they use, will need to obtain the source RPM and rebuild it. If you want to do this for more than one server, then you will be able to use our rpm.version system so your cPanel & WHM servers install your custom package from a central location.

What is not Involved in this Change

Not everything will switch to delivery via RPM packages. EasyApache will still build Apache et al. from source. cPanel & WHM itself will not be delivered via RPM in this release.

The Benefits

Fresh installs will see a 30% reduction (or better) in installation time. This is due to installing perl, and necessary CPAN modules, from RPM rather than building them from source.

Upgrades should also see a reduction in time, in the long term, as new applications and dependency updates will be fulfilled via RPM packages, rather than building from source during the upgrade.

The rpm.version system gives you a simple way of deploying custom software directly as part of the cPanel & WHM installation and upgrade.

Removal of problems stemming from automated source builds in wildly diverse environments.

Re-installing perl is a matter of re-installing a package, rather than building from source and forcing the reinstallation of a bunch of CPAN module.

Applications and dependencies delivered with cPanel & WHM will be in one location, /usr/local/cpanel, rather than strewn all over the file system(s). The long term goal is to have everything installed to /usr/local/cpanel. For 11.34 only the new RPMs adhere to this.

More

There is far more to these changes than what is detailed here. In the coming weeks we'll share more information on:

• perl 5.14 in cPanel & WHM
• Changes to the internal PHP
• rpm.versions system
• Changes to how cPanel & WHM are updated to make it more robust

[jestep]

All the source RPMs will be provided on our mirrors. Customers that want to apply patches to the software we provide, or just like compiling software they use, will need to obtain the source RPM and rebuild it.

What's the implications of this as far as security related patches go? Are we going to need to apply patches to cpanel provided software ourselves?

[cPanelKenneth]

What's the implications of this as far as security related patches go? Are we going to need to apply patches to cpanel provided software ourselves?

Only if you are custom compiling the software. Otherwise we'll provide the update as we currently do, which is update the software in question then release a new version of cPanel & WHM.

[chirpy]

How will we install additional perl modules from cpan that we currently do through /scripts/perlinstaller?

How will this apply to scripts that run within cPanel itself, i.e. through the cPanel user UI and the WHM root UI?

How will this apply to scripts that run outside of cPanel?

Currently any installation of a perl module through /scripts/perlinstaller means that it is available in all 3 scenarios at present.

[cPanelKenneth]

How will we install additional perl modules from cpan that we currently do through /scripts/perlinstaller?

How will this apply to scripts that run within cPanel itself, i.e. through the cPanel user UI and the WHM root UI?

How will this apply to scripts that run outside of cPanel?

Obviously, at present, any installation of a perl module through /scripts/perlinstaller means that it is available in all 3 scenarios at present.

I'll go into this in more detail in a future post, since it's quite a lengthy topic.

In short, the perl RPM will be for use with cpsrvd. This means that all the CPAN RPMs we provide will be for use in cpsrvd. You will be able to install your own modules for use with cpsrvd using the standard perl tools in /usr/local/cpanel. Our preference is if you need such modules please let us know. If possible we'd like to provide them as RPMs.

Support for use with Apache or command line scripts is a bit murkier. I'd like to defer my answer until the article on this.

Your questions are very welcome, so please don't take my non-committal as dismissal

[twhiting9275]

Not everything will switch to delivery via RPM packages. EasyApache will still build Apache et al. from source.

For good, or will you eventually be forcing everyone to use your rpm's for this too?
Remember, not everyone WANTS to have an RPM based system. Some actually want to control their own systems! When you're talking apache/php, this is especially true

[cPanelKenneth]

For good, or will you eventually be forcing everyone to use your rpm's for this too?
Remember, not everyone WANTS to have an RPM based system. Some actually want to control their own systems! When you're talking apache/php, this is especially true

The long term goal is to provide pre-compiled binary packages for all operating systems, or linux distros, we support. In 11.34 the main focus of these packages is the perl stack.

We understand some people like the ability to custom compile applications, for a variety of reasons. This is why we are developing a flexible deployment system that allows such people to custom compile their own RPMs and deploy them to cPanel & WHM in a way that's accounted for by cPanel & WHM during upgrade, installation, etc.

Transitioning EasyApache to a RPM-based system is the most complex part of the long term goal. It is also the system people want to influence the most, as far as custom compile, patching and so forth. We will do EasyApache last so as to learn from the experience of transitioning other parts of the product to pre-compiled packages. We also want an open dialog with users and customers while designing the RPM-based EasyApache system so we deliver something that meets their needs.

[chirpy]

The concept seems fine to me, but I cannot really comment further until you do outline how perl modules can be installed both for cpsrvd and external at CLI.

The former sounds straightforward as you will have a contained environment, however the latter is probably going to be the the most important and crucial as it really does need to be as simple and transparent as the current process of doing so, otherwise I can see that a lot is going to break in third party product provision.

Either that, or we have to modify external scripts to use the cPanel provided perl binary in /usr/local/cpanel/, which could be a can of worms.

[deth4uall]

So essentially there will be a move from compiled on server to rpms, except for EasyApache? You mentioned that Perl would be RPMed, does this mean that you will finally move away from bloody Perl 5.8.8 at the core? Also with everything going to RPM what would we do for installing modules via CPAN? Would they be overwritten or what? Just trying to get an idea of how you guys are thinking of doing this.

[cPanelKenneth]

So essentially there will be a move from compiled on server to rpms, except for EasyApache? You mentioned that Perl would be RPMed, does this mean that you will finally move away from bloody Perl 5.8.8 at the core? Also with everything going to RPM what would we do for installing modules via CPAN? Would they be overwritten or what? Just trying to get an idea of how you guys are thinking of doing this.

I'm working on the Perl article now, which will hopefully address you questions.

There is a short list of things not provided via RPM in 11.34. These include (but are not limited to):

• The webmail clients
• phpMyAdmin
• phpPgAdmin
• ruby
• rails

These will be converted in future versions of cPanel & WHM, as will EasyApache.

[GIANT_CRAB]

Waiting for OP to deliver...

[Infopro]

Waiting for OP to deliver...

I'm not sure your comment is clear, these changes are for future releases of cPanel, 11.34 and beyond.

[GIANT_CRAB]

I'm not sure your comment is clear, these changes are for future releases of cPanel, 11.34 and beyond.

That's several months away.

Understandable since these are big changes.

[rlshosting]

You need to have an interface to it. I don't think that everyone wants everything command line based. It's just going to complicate things and make things more time consuming.

Edit: It looks like it's going to be much better this way since a graphical interface will be included.

[cPanelKenneth]

You need to have an interface to it. I don't think that everyone wants everything command line based. It's just going to complicate things and make things more time consuming.

Edit: It looks like it's going to be much better this way since a graphical interface will be included.

An interface to what?

The majority of RPMs being delivered with cPanel & WHM 11.36 will be installed by default. The ones that aren't already have interfaces to choose which RPM gets installed. e.g. install courier versus dovecot.