The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Switching from compile-on-demand to binary packages

Discussion in 'cPanel Announcements' started by cPanelKenneth, Jun 6, 2012.

  1. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    For many years installing and using cPanel & WHM has involved compiling software on-demand. Want Apache and PHP? Run /scripts/easyapache, which builds those and all dependencies from source. Want perl installed? Download and run the perl installer from httpupdate.cpanel.net, which installs perl from source. While compiling from source has its benefits, it also has its drawbacks.

    The drawbacks are more obvious when you consider what cPanel & WHM attempts to do with compile-on-demand. We are automating this on thousands of servers, each with their own subtle differences (and some no-so-subtle). A non-trivial portion of our development effort, bug fixing and support requests are consumed by getting source compilation working in environments under other peoples (yours) control. We decided this needed to change. Hence in 11.34 we are switching our model from compile-on-demand to delivery of pre-compiled RPM packages.

    What is involved in this change

    For many years we've provided applications via RPM. For example MySQL, Dovecot, and Exim. In 11.34 we are adding over 800 additional RPMs. These RPMs provide:

    • perl 5.14
    • all the perl modules that scripts/checkperlmodules formerly installed
    • PHP 5.3 for cPanel & WHM
    • rrdtool
    • all the pear packages needed for Horde, Roundcube, phpMyAdmin, Logaholic and Squirrelmail
    • the encoders used by internal PHP
    • nearly everything in /usr/local/cpanel/src

    The vast majority of the RPMs are for perl and CPAN modules (801 at last count).

    New RPMs will have the naming convention of cpanel-APPLICATION-APPMAJORVERSION-APPVERSION-CPANELPATCH-CPANELVERSION. For example:

    cpanel-perl-514-5.14.2-0.cp1134

    Broken into its parts we have:
    • cpanel-perl: The application is perl
    • 514: perl major version of 5.14
    • 5.14.2: The specific version of perl in the RPM
    • -0: We have not applied any patches, therefore we use 0
    • cp1134: Built for cPanel & WHM 11.34
    All the new RPMs will be installed to /usr/local/cpanel, rather than to the operating system locations (e.g. /usr ). Whenever possible we fulfill dependencies (e.g. libraries) using what the operating system provides. If that is not possible, we provide the dependency ourselves with an RPM.

    Along with an expanded set of RPMs we are also providing a tool you can use to reinstall RPMs. This is useful if something is modifying files managed by a package. The tool will tell you the packages that need reinstalled and allows you to perform the reinstall at your convenience.

    How Something is installed

    Does this mean you will find scripts like perlup, internalphpup and so forth in 11.34? No. In fact all the scripts that install the current RPMs (e.g mysqlup, eximup) are gone in 11.34. Instead RPMs are tied to the product version and are installed by our "rpm.versions" system. This system behaves somewhat like yum, or apt-get, in that you are able to configure repositories and install software from them. The rpm.versions system was introduced in 11.30, but for 11.34 is completely overhauled. More information on that later.

    By and large RPM installation is handled by upcp. For now this is acceptable as the only RPMs we are delivering are required by the product. Eventually we want to provide optional RPMs (e.g. git, subversion, etc) that you can elect to install. When we reach that stage tools, both command line and graphical, will be developed to facilitate package installation outside of upcp.

    Source Compiles

    All the source RPMs will be provided on our mirrors. Customers that want to apply patches to the software we provide, or just like compiling software they use, will need to obtain the source RPM and rebuild it. If you want to do this for more than one server, then you will be able to use our rpm.version system so your cPanel & WHM servers install your custom package from a central location.

    What is not Involved in this Change

    Not everything will switch to delivery via RPM packages. EasyApache will still build Apache et al. from source. cPanel & WHM itself will not be delivered via RPM in this release.

    The Benefits

    Fresh installs will see a 30% reduction (or better) in installation time. This is due to installing perl, and necessary CPAN modules, from RPM rather than building them from source.

    Upgrades should also see a reduction in time, in the long term, as new applications and dependency updates will be fulfilled via RPM packages, rather than building from source during the upgrade.

    The rpm.version system gives you a simple way of deploying custom software directly as part of the cPanel & WHM installation and upgrade.

    Removal of problems stemming from automated source builds in wildly diverse environments.

    Re-installing perl is a matter of re-installing a package, rather than building from source and forcing the reinstallation of a bunch of CPAN module.

    Applications and dependencies delivered with cPanel & WHM will be in one location, /usr/local/cpanel, rather than strewn all over the file system(s). The long term goal is to have everything installed to /usr/local/cpanel. For 11.34 only the new RPMs adhere to this.

    More

    There is far more to these changes than what is detailed here. In the coming weeks we'll share more information on:
    • perl 5.14 in cPanel & WHM
    • Changes to the internal PHP
    • rpm.versions system
    • Changes to how cPanel & WHM are updated to make it more robust
     
  2. jestep

    jestep Active Member

    Joined:
    Dec 18, 2006
    Messages:
    41
    Likes Received:
    0
    Trophy Points:
    6
    What's the implications of this as far as security related patches go? Are we going to need to apply patches to cpanel provided software ourselves?
     
  3. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    Only if you are custom compiling the software. Otherwise we'll provide the update as we currently do, which is update the software in question then release a new version of cPanel & WHM.
     
  4. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    How will we install additional perl modules from cpan that we currently do through /scripts/perlinstaller?

    How will this apply to scripts that run within cPanel itself, i.e. through the cPanel user UI and the WHM root UI?

    How will this apply to scripts that run outside of cPanel?

    Currently any installation of a perl module through /scripts/perlinstaller means that it is available in all 3 scenarios at present.
     
    #4 chirpy, Jun 6, 2012
    Last edited: Jun 6, 2012
  5. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    I'll go into this in more detail in a future post, since it's quite a lengthy topic.

    In short, the perl RPM will be for use with cpsrvd. This means that all the CPAN RPMs we provide will be for use in cpsrvd. You will be able to install your own modules for use with cpsrvd using the standard perl tools in /usr/local/cpanel. Our preference is if you need such modules please let us know. If possible we'd like to provide them as RPMs.

    Support for use with Apache or command line scripts is a bit murkier. I'd like to defer my answer until the article on this.

    Your questions are very welcome, so please don't take my non-committal as dismissal :)
     
  6. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    For good, or will you eventually be forcing everyone to use your rpm's for this too?
    Remember, not everyone WANTS to have an RPM based system. Some actually want to control their own systems! When you're talking apache/php, this is especially true
     
  7. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    The long term goal is to provide pre-compiled binary packages for all operating systems, or linux distros, we support. In 11.34 the main focus of these packages is the perl stack.

    We understand some people like the ability to custom compile applications, for a variety of reasons. This is why we are developing a flexible deployment system that allows such people to custom compile their own RPMs and deploy them to cPanel & WHM in a way that's accounted for by cPanel & WHM during upgrade, installation, etc.

    Transitioning EasyApache to a RPM-based system is the most complex part of the long term goal. It is also the system people want to influence the most, as far as custom compile, patching and so forth. We will do EasyApache last so as to learn from the experience of transitioning other parts of the product to pre-compiled packages. We also want an open dialog with users and customers while designing the RPM-based EasyApache system so we deliver something that meets their needs.
     
  8. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    The concept seems fine to me, but I cannot really comment further until you do outline how perl modules can be installed both for cpsrvd and external at CLI.

    The former sounds straightforward as you will have a contained environment, however the latter is probably going to be the the most important and crucial as it really does need to be as simple and transparent as the current process of doing so, otherwise I can see that a lot is going to break in third party product provision.

    Either that, or we have to modify external scripts to use the cPanel provided perl binary in /usr/local/cpanel/, which could be a can of worms.
     
  9. deth4uall

    deth4uall Well-Known Member

    Joined:
    Sep 30, 2011
    Messages:
    45
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Austin, Texas, United States
    cPanel Access Level:
    Root Administrator
    So essentially there will be a move from compiled on server to rpms, except for EasyApache? You mentioned that Perl would be RPMed, does this mean that you will finally move away from bloody Perl 5.8.8 at the core? Also with everything going to RPM what would we do for installing modules via CPAN? Would they be overwritten or what? Just trying to get an idea of how you guys are thinking of doing this.
     
  10. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    I'm working on the Perl article now, which will hopefully address you questions.

    There is a short list of things not provided via RPM in 11.34. These include (but are not limited to):

    • The webmail clients
    • phpMyAdmin
    • phpPgAdmin
    • ruby
    • rails

    These will be converted in future versions of cPanel & WHM, as will EasyApache.
     
  11. GIANT_CRAB

    GIANT_CRAB Well-Known Member

    Joined:
    Mar 23, 2012
    Messages:
    89
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Waiting for OP to deliver...
     
  12. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,453
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:

    I'm not sure your comment is clear, these changes are for future releases of cPanel, 11.34 and beyond.
     
  13. GIANT_CRAB

    GIANT_CRAB Well-Known Member

    Joined:
    Mar 23, 2012
    Messages:
    89
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    That's several months away.

    Understandable since these are big changes.
     
  14. rlshosting

    rlshosting Well-Known Member

    Joined:
    Apr 23, 2009
    Messages:
    170
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    You need to have an interface to it. I don't think that everyone wants everything command line based. It's just going to complicate things and make things more time consuming.

    Edit: It looks like it's going to be much better this way since a graphical interface will be included.
     
    #14 rlshosting, Jan 8, 2013
    Last edited: Jan 8, 2013
  15. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    An interface to what?

    The majority of RPMs being delivered with cPanel & WHM 11.36 will be installed by default. The ones that aren't already have interfaces to choose which RPM gets installed. e.g. install courier versus dovecot.
     
  16. rlshosting

    rlshosting Well-Known Member

    Joined:
    Apr 23, 2009
    Messages:
    170
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    Interface to install rpms without easy apache. Isn't your goal to remove easy apache? If I have to install all of those rpms manually, I am not going to be happy and most people aren't either.
     
  17. rlshosting

    rlshosting Well-Known Member

    Joined:
    Apr 23, 2009
    Messages:
    170
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    United States
    cPanel Access Level:
    Root Administrator
    Are you going to eventually remove Easy Apache or not? I hope not. I hope that the only thing that changes is everything else but Easy Apache in the future.
     
    #17 rlshosting, Jan 9, 2013
    Last edited: Jan 9, 2013

Share This Page