Solutions for handling symlink attacks

[Arvand]

I hate cpanels stance on this so we have created a patch and have been using for some time. It turns FollowSymLinks into SymLinksIfOwnerMatch at the apache source code level.

We currently are working on rewriting the patch, and part of apache to take care of some possible race conditions. But given the rare race condition possibility, this is by far a better option than causing everyone to have to reconfigure their .htaccess files or allowing your server to be wide open to attack.

How to install our patch (apache 2.2 only):




If you have any issues, let us know, we would be interested in hearing it.
If you want to thank us, your free to do that aswell.

When trying to access a file located in another account via a symlink, you will see this in the error log:



Also, find out if your already a victim:



---

How to remove?:



Enjoy.

This does work for us. Thank you.

What is however unfortunate is that by having this public, it may further re-enforce the false notion that this has nothing to do with cpanel and that they should not be concerned about this. (I hope this isn't the case)

At this point in time, tens of thousands of servers are currently vulnerable. cPanel is getting paid a monthly premium to provide a secure management experience for the administrators of those servers. And in this case, greatly failing to do so.

To cPanel - Here is a simple analogy which may better describe why I feel you should at least be notifying your customers. If you were a grocery store and you sold meat, then a customer came in and said that meat over there killed my son because it has E coli. Would you sit there and say, well, that's not our meat - it comes from the so and so farm. We just package it and sell it. Go talk to the farm?!?

This sort of behavior, when this issue has been repeatedly brought up to Cpanel, is exactly whats expected of Parallels not cPanel...

 

[StevenC]

Atleast patch is not working for us. If someone need step by step guide to replicate please contact.

You run the wget command, and then run easyapache choosing apache 2.2. There is nothing more.

 

[StevenC]

Indeed setting up SymLinksIfOwnerMatch in apache conf improve the symlink protection and hence server security but that affect server performance. However server security can not be compromised against server performance.

Out of the few hundred servers we have this on, we noticed minimal load increase.

 

[brianoz]

Indeed setting up SymLinksIfOwnerMatch in apache conf improve the symlink protection and hence server security but that affect server performance. However server security can not be compromised against server performance.

So you'd rather have a fast server that doesn't work because it got hacked?? Really?? And you don't care about your customers getting hacked?

I guess the question is - how much does this really compromise server performance? Boiled down, as far as I can see, it's simply using an extra stat every time there's a symlink - do you have symlinks everywhere? If not, then you wouldn't notice the difference.

The impact of an extra stat is almost (not quite, but almost) negligible. But I guess it's a trade-off between having a secure and supportable server and not. This is already starting to happen in the wild, so it's of real concern.

Atleast patch is not working for us. If someone need step by step guide to replicate please contact.

Sorry - do you mean it is working or not working? If not working, what is happening, exactly? It compiles here, and works as far as I can see? Details would be useful.

 

[DomineauX]

What is however unfortunate is that by having this public, it may further re-enforce the false notion that this has nothing to do with cpanel and that they should not be concerned about this. (I hope this isn't the case)

At this point in time, tens of thousands of servers are currently vulnerable. cPanel is getting paid a monthly premium to provide a secure management experience for the administrators of those servers. And in this case, greatly failing to do so.

To cPanel - Here is a simple analogy which may better describe why I feel you should at least be notifying your customers. If you were a grocery store and you sold meat, then a customer came in and said that meat over there killed my son because it has E coli. Would you sit there and say, well, that's not our meat - it comes from the so and so farm. We just package it and sell it. Go talk to the farm?!?

This sort of behavior, when this issue has been repeatedly brought up to Cpanel, is exactly whats expected of Parallels not cPanel...

Sorry but I cannot agree with your blaming cPanel on this. The fact is that Apache includes two options and allows the insecure one by default. cPanel has included the ability to disable it in the Global Apache Configuration options via WHM. This is as much as they should be asked to do in my opinion.
This is also enough to ensure security, unless your users enable the insecure option whether on purpose or accident such as by installing a script that enables "FollowSymLinks" in the .htaccess file.

The solutions posted by StevenC and myself are simply extra measures to prevent the enabling of "FollowSymLinks" by users and script installations.

 

[ethix]

Enjoy.

Thank you for sharing this. It is appreciated

Brendan

 

[hostnex]

Below are the steps which any typical hacker can use to hack your website even after applying the patch.


1- The first thing any hacker need to access your server is to upload its roothsell through some vulnerable website.

2- Once he is successful to upload root shell he will try to disable safe mode from different ways one way is to place php.ini in root of user account.

3- Now he will make a new folder under public_html assume folder name is helo.

4- In helo folder he will upload or make a new .htaccess file and put the following code in it

Options Indexes FollowSymLinks

DirectoryIndex ssssss.htm

AddType txt .php

AddHandler txt .php

5- Now from root shell he will symlink with / which will give him access to browse whole server through rootshell

ln -s / root

6- Now he will browse website http://domain.com/helo/ which will show him the following contents

/Parent Directory
/root

Clicking root will let him browse whole server.

-8 Now he will get accounts list running on the server. There are different ways to get it. One way is to edit /etc/passwd file.

-9 But still hacker cant access users folders within /home/account to gain access of accounts root folder he will run the following command

ls -la /etc/valiases/helo.com

8- now he will go to the path /home/helo/public_html and can browse any file within it through rootshell


Even after running your patch this method can be used to access any website through command line and all contents of website will be browseable. Below are the recommendation to secure server.

-1 do not allow users to change php settings through php.ini or .htaccess file

2- Enable Apache mod_userdir Tweak but do not exclude any of the hostname as through http://domain.com/~username php.ini global settings can be overridden.

-2 Disable symlink for all users.

-3 disable following functions globally in php.ini

show_source,system,shell_exec,passthru,exec,proc_open,allow_url_fopen,symlink,exec,proc_close,dl,escap,eshellarg,escapeshellcmd,popen

4- Enable safe_mode

5-run maldet a malware detection tool which is very effective to catch most of roothsell.

 

[k-planethost]

the patch doesnt work on your servers?

 

[Estiny]

We have a bunch of servers that we would want to roll your patch out on. Rather then doing the following:

wget http://layer1.rack911.com/before_apache_make -O /scripts/before_apache_make
chmod 700 /scripts/before_apache_make
/scripts/easyapache --build


Could we do this on one server and then copy over X files to Y server to avoid running easyapache on each and every server? It would save us a bunch of issues I think. Do you know what files are changed by easyapache to incorporate your patch so we could say, copy over X files or X binaries to make it work on server Y.

 

[StevenC]

Below are the steps which any typical hacker can use to hack your website even after applying the patch.


1- The first thing any hacker need to access your server is to upload its roothsell through some vulnerable website.

2- Once he is successful to upload root shell he will try to disable safe mode from different ways one way is to place php.ini in root of user account.

3- Now he will make a new folder under public_html assume folder name is helo.

4- In helo folder he will upload or make a new .htaccess file and put the following code in it

Options Indexes FollowSymLinks

DirectoryIndex ssssss.htm

AddType txt .php

AddHandler txt .php

5- Now from root shell he will symlink with / which will give him access to browse whole server through rootshell

ln -s / root

6- Now he will browse website http://domain.com/helo/ which will show him the following contents

/Parent Directory
/root

Clicking root will let him browse whole server.

-8 Now he will get accounts list running on the server. There are different ways to get it. One way is to edit /etc/passwd file.

-9 But still hacker cant access users folders within /home/account to gain access of accounts root folder he will run the following command

ls -la /etc/valiases/helo.com

8- now he will go to the path /home/helo/public_html and can browse any file within it through rootshell


Even after running your patch this method can be used to access any website through command line and all contents of website will be browseable. Below are the recommendation to secure server.

-1 do not allow users to change php settings through php.ini or .htaccess file

2- Enable Apache mod_userdir Tweak but do not exclude any of the hostname as through http://domain.com/~username php.ini global settings can be overridden.

-2 Disable symlink for all users.

-3 disable following functions globally in php.ini

show_source,system,shell_exec,passthru,exec,proc_open,allow_url_fopen,symlink,exec,proc_close,dl,escap,eshellarg,escapeshellcmd,popen

4- Enable safe_mode

5-run maldet a malware detection tool which is very effective to catch most of roothsell.

In your method you mention:

'Options Indexes FollowSymLinks'

FollowSymLinks does not work as FollowSymLinks with my patch.

The hack you described is the same exact hack we see every day. The exact way you described it is blocked by the the patch. It sounds like the patch is not installed correctly on your server.

It sounds like you did NOT compile apache after running the wget command.

 

[StevenC]

I believe you just need to copy /usr/local/apache/bin/httpd, but I have not confirmed it.

 

[StevenC]

-2 Disable symlink for all users.

It is impossible to disable symlink for all users, Its a kernel based function. A perl script or even a compiled C binary that is uploaded can still access symlink.

In case you are not aware, you can execute C binaries as cgi scripts.

 

[Mitio]

Hello,

Solutions for this issue:

Symlink Protect cPanel EasyApache modul:

1. First download this files /http://spasov.us/patch/Apache.zip

Login as root and go to /var/cpanel/easy/apache/custom_opt_mods/Cpanel/Easy/Apache

upload SymlinkProtection.pm and SymlinkProtection.pm.tar.gz on this directory:

/var/cpanel/easy/apache/custom_opt_mods/Cpanel/Easy/Apache

2. Run /scripts/easyapache, and select SymlinkProtection from the "Exhaustive Options" list .

Note: I found that many servers have this problem.

Do not hesitate to contact us again for any further questions or need of assistance.

Best Regards,
Dimitar Spasov,

 

[hostnex]

It is impossible to disable symlink for all users, Its a kernel based function. A perl script or even a compiled C binary that is uploaded can still access symlink.

In case you are not aware, you can execute C binaries as cgi scripts.

Currently we have included following code in Apache include files to avoid symlink. When someone try to use following symlink option in .htaccess he got an Internal server error.

<Directory "/">
Options -ExecCGI -FollowSymLinks Includes IncludesNOEXEC Indexes -MultiViews SymLinksIfOwnerMatch
AllowOverride All
</Directory>

<Directory "/usr/local/apache/htdocs">
Options IncludesNOEXEC Indexes -FollowSymLinks +SymLinksIfOwnerMatch -ExecCGI
AllowOverride None
Order allow,deny
Allow from all

</Directory>


<Directory "/home">
Options All -ExecCGI -Indexes -FollowSymLinks -Includes +IncludesNOEXEC -MultiViews +SymLinksIfOwnerMatch
AllowOverride AuthConfig Indexes Limit Fileinfo options=Indexes,MultiViews

</Directory>



Are you saying its not effective ?. Also after using your patch do we still need to include above code in Apache ? It has broken most of our websites which are using followsymlink option in .htaccess.

We will try your patch on separate server and update you .

 

[Mitio]

Hello,

Solutions for this issue:

Symlink Protect cPanel EasyApache module:

1. /http://spasov.us/patch/Apache.zip

Login as root go to /var/cpanel/easy/apache/custom_opt_mods/Cpanel/Easy/Apache

upload this files SymlinkProtection.pm SymlinkProtection.pm.tar.gz on this directory /var/cpanel/easy/apache/custom_opt_mods/Cpanel/Easy/Apache

2. Run /scripts/easyapache, and select SymlinkProtection from the "Exhaustive Options" list

Do not hesitate to contact us again for any further questions or need of assistance.

Best Regards,
Dimitar Spasov,

 

[hostnex]

Hello,

Solutions for this issue:

Symlink Protect cPanel EasyApache module:

1. /http://spasov.us/patch/Apache.zip

Login as root go to /var/cpanel/easy/apache/custom_opt_mods/Cpanel/Easy/Apache

upload this files SymlinkProtection.pm SymlinkProtection.pm.tar.gz on this directory /var/cpanel/easy/apache/custom_opt_mods/Cpanel/Easy/Apache

2. Run /scripts/easyapache, and select SymlinkProtection from the "Exhaustive Options" list

Do not hesitate to contact us again for any further questions or need of assistance.

Best Regards,
Dimitar Spasov,

May I know who are you as I could not find any kind of information about you in your profile. So nice of you that you made a new forum account just to inform us that how to fix it

 

[Mitio]

Dear hostnex,

Thank you. My name is Dimitar Spasov from Bulgaria this is my email address - Removed - and my facebook:

- Removed -

Skype name: - Removed -

I administer a few linux servers. I'm just trying to help

Best Regards,
Dimitar Spasov,

 

[StevenC]

Currently we have included following code in Apache include files to avoid symlink. When someone try to use following symlink option in .htaccess he got an Internal server error.

<Directory "/">
Options -ExecCGI -FollowSymLinks Includes IncludesNOEXEC Indexes -MultiViews SymLinksIfOwnerMatch
AllowOverride All
</Directory>

<Directory "/usr/local/apache/htdocs">
Options IncludesNOEXEC Indexes -FollowSymLinks +SymLinksIfOwnerMatch -ExecCGI
AllowOverride None
Order allow,deny
Allow from all

</Directory>


<Directory "/home">
Options All -ExecCGI -Indexes -FollowSymLinks -Includes +IncludesNOEXEC -MultiViews +SymLinksIfOwnerMatch
AllowOverride AuthConfig Indexes Limit Fileinfo options=Indexes,MultiViews

</Directory>



Are you saying its not effective ?. Also after using your patch do we still need to include above code in Apache ? It has broken most of our websites which are using followsymlink option in .htaccess.

We will try your patch on separate server and update you .

If with my patch your having issues, then it did not compile correctly. My patch is intended to allow it to remain secure AND not give a internal server message if someone uses FollowSymLinks. It makes FollowSymLinks perform like SymLinksIfOwnerMatch. I have it running on several hundred servers. On a daily basis we see people attempting to perform this hack and fail (through our modified suhosin extension we use, we are able to monitor this).

 

[StevenC]

Nice patch Mitio.

 

[hostnex]

So which patch we should use yours or Mitio