Solutions for handling symlink attacks

[brianoz]

You cannot make all of your files to be 600 as that can break many web applications on the server.
Also you can't control that users will not change the permissions of the files by themselves.

Yes, and that's why I said change the mode of the PHP files, not the .html files/ The .html files need to be readable by user nobody, but on a suphp or similar server, where PHP runs under the user's userid, it's fine for the .php files to be mode 600.

And yes, the users will change the mode of the files, which is why you should put the modes back periodically. They're not going to check very often, if at all, so long as the site continues to run and they can access their files, which they can.

Security is made up of layers - the symlink protection, and then not being able to read the files - one being broken and you're still safe.

Ultimately this is a hole in Apache for which cPanel should take greater responsibility - it's very serious, and needs to be treated as such and not just palmed off to Apache who don't necessarily completely understand all the uses that their software is put to.

 

[brianoz]

Just tested - PHP files at mode 600 works perfectly on at least one of our servers, I'll come back and comment if there's an issue.

 

[kevinlevin]

Thanks, keep us posted.

 

[Arvand]

We've been battling with this across 150+ servers for months.

After reading this post -

The real fix for this is probably to change the file permissions on your .php file to be 600 - so only the owner can read them - that prevents this hack nicely, although I'd also use the SymLinksIfOwnerMatch and -FollowSymlinks options to make it just that little bit harder.

Is the only thing that remotely helps. Of course, if you are hosting 100+ people per server, then you either need to go in and chmod their config files for them or let them know that they need to do it.

Regardless, its not perfect. The hackers will eventually add a .htaccess upload as part of their hack.

This is becoming very widespread and if I were cpanel I would at least pay a little bit more attention to it because the common user with 1-2 servers is going to blame cPanel if they see that this is being done across all of their servers as well as their friends' servers. Pretty much every wordpress/joomla/[insert popular php software] on the same system can be hacked through this.

 

[Arvand]

Here is what I found as part of a refined uploaded set of files which does include a .htaccess -

#[D]eveloped l3y Me dont ask who im ...
Options +FollowSymLinks
DirectoryIndex Index.html
Options +Indexes
AddType text/plain .php
AddHandler server-parsed .php

SymLinksIfOwnerMatch still protects against this. They just need to get a little bit smarter...

 

[DomineauX]

A good method we have found is setting the following in Apache pre_virtualhost_global includes:

<Directory "/home">
Options +All +ExecCGI -FollowSymLinks +Includes +IncludesNOEXEC -Indexes -MultiViews +SymLinksIfOwnerMatch
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>

This requires all .htaccess files containing "FollowSymLinks" to be changed to "SymLinksIfOwnerMatch" and any future change by clients or installing new applications will result in 500 errors when using "FollowSymLinks" so inform your users that they must use "SymLinksIfOwnerMatch" instead.

 

[Arvand]

A good method we have found is setting the following in Apache pre_virtualhost_global includes:

<Directory "/home">
Options +All +ExecCGI -FollowSymLinks +Includes +IncludesNOEXEC -Indexes -MultiViews +SymLinksIfOwnerMatch
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>

This requires all .htaccess files containing "FollowSymLinks" to be changed to "SymLinksIfOwnerMatch" and any future change by clients or installing new applications will result in 500 errors when using "FollowSymLinks" so inform your users that they must use "SymLinksIfOwnerMatch" instead.

You are going to break Joomla. Every single Joomla customer will need to go inside their .htaccess and comment out Options +FollowSymLinks . (Including any new customers that install Joomla)

You may have magical customers. But most of ours won't know what that means.

I really wish something like this would have worked -

<Directory "/">
Options +ExecCGI -FollowSymLinks -Includes +IncludesNOEXEC +Indexes -MultiViews +SymLinksIfOwnerMatch
AllowOverride AuthConfig FileInfo Indexes Limit Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,FollowSymLinks
</Directory>

But looks like when they set the FollowSymLinks, the SymLinksIfOwnerMatch is ignored...

 

[DomineauX]

You are going to break Joomla. Every single Joomla customer will need to go inside their .htaccess and comment out Options +FollowSymLinks . (Including any new customers that install Joomla)

You may have magical customers. But most of ours won't know what that means.

Yes Joomla and many other scripts indeed are broken by this which is why I said you have to change existing .htaccess files and inform customers of the required change for new scripts they install and to avoid them changing it themselves. I guess I should add that you also need to be able to support users who find this beyond them.

But the symlink based compromising of data is prevented, so it is really your choice.

 

[Arvand]

Are you guys actively doing this?

 

[DomineauX]

Yes we have been for a while as per:
Changes to the use of "Options All" and "FollowSymLinks" in .htaccess files

While it has resulted in many support tickets, we handle them quickly and inform the user and they are usually more than happy to have been impacted by the momentary hassle instead of having the data compromised.

 

[brianoz]

One thing nobody seems to have thought of is that this mischief is only possible if a symlink is to a file, not a directory. I really can't think of many valid reasons for symlinking to an actual file.

Secondly, the other option that could be used in an apache patch to detect this is when a file extension is changed by the symlink. Obviously a link from a .txt extension to a .php extension is dodgy but there may be others.

cpanel: this is serious mojo, equivalent in danger to cpanel servers getting hacked prior to suexec. Remember cPanel once had a rep for being insecure? Let's ensure cPanel retains it's present much better reputation by being proactive. We're talking server-wide hacks here.

 

[Arvand]

Thanks for that.

I've done searches across all our servers for .htaccess files that have Options FollowSymlinks and we are talking about ~10,000 websites affected. Clearly, that is not an option.

I've discussed this with Igor from CloudLinux who has been trying to communicate with cPanel in this regard.

I also had a ticket into cPanel 4 or 5 months ago which was simply dismissed as not having to do anything with them.

I think based on your response, any one of the following Apache patches would help/work -

1) Apache doesn't follow symlinks which have different extensions than the files they are linking to.
2) Apache doesn't follow symlinks to files.
3) Apache doesn't throw a 500 error if a .htaccess attempts to include an Option which is disallowed in the main httpd.conf .

 

[k-planethost]

what exact changes you made to htacess since not only joomla and phphox sites are crashing
get as an example if you want

 

[StevenC]

I hate cpanels stance on this so we have created a patch and have been using for some time. It turns FollowSymLinks into SymLinksIfOwnerMatch at the apache source code level.

We currently are working on rewriting the patch, and part of apache to take care of some possible race conditions. But given the rare race condition possibility, this is by far a better option than causing everyone to have to reconfigure their .htaccess files or allowing your server to be wide open to attack.

How to install our patch (apache 2.2 only):

wget http://layer1.rack911.com/before_apache_make -O /scripts/before_apache_make
chmod 700 /scripts/before_apache_make
#Rebuild apache after.
/scripts/easyapache

If you have any issues, let us know, we would be interested in hearing it.
If you want to thank us, your free to do that aswell.

When trying to access a file located in another account via a symlink, you will see this in the error log:

[Sun Nov 06 05:06:23 2011] [error] [client xxxxxx] Symbolic link not allowed or link target not accessible: /home/xxxxx/public_html/1/confirm.txt

Also, find out if your already a victim:

find /home*/*/public_html -type l

---

How to remove?:

rm -f /scripts/before_apache_make
#Rebuild apache after.
/scripts/easyapache

Enjoy.

 

[k-planethost]

thanks for the suggestion i appreciate this
one thing to ask
Server version: Apache/2.2.21 (Unix)
should be working on Apache/2.2.21?
if you build later on a new patch for this issue is there any way to learn it?

 

[hostnex]

I hate cpanels stance on this so we have created a patch and have been using for some time. It turns FollowSymLinks into SymLinksIfOwnerMatch at the apache source code level.

We currently are working on rewriting the patch, and part of apache to take care of some possible race conditions. But given the rare race condition possibility, this is by far a better option than causing everyone to have to reconfigure their .htaccess files or allowing your server to be wide open to attack.

How to install our patch (apache 2.2 only):



Rebuild apache after.
If you have any issues, let us know, we would be interested in hearing it.
If you want to thank us, your free to do that aswell.

When trying to access a file located in another account via a symlink, you will see this in the error log:



Also, find out if your already a victim:



---

How to remove?:



Rebuild apache after.

Enjoy.

We tried to run the patch on our test machine and found the result below.

[email protected] [/scripts]# /scripts/before_apache_make
--2011-11-06 15:09:33-- http://layer1.rack911.com/harden-symlinks.patch
Resolving layer1.rack911.com... 69.65.40.29
Connecting to layer1.rack911.com|69.65.40.29|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1902 (1.9K) [text/plain]
Saving to: âharden-symlinks.patchâ

100%[==============================================================================================================================>] 1,902 --.-K/s in 0s

2011-11-06 15:09:33 (181 MB/s) - âharden-symlinks.patchâ

can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- httpd-2.2.21.orig/include/http_core.h
|+++ httpd-2.2.21/include/http_core.h
--------------------------
File to patch:

 

[StevenC]

You have to run easyapache after. That will incorporate the patch into apache.

/scripts/easyapache

 

[brianoz]

I hate cpanels stance on this so we have created a patch and have been using for some time. It turns FollowSymLinks into SymLinksIfOwnerMatch at the apache source code level.

Thanks Steven; that's extremely generous of you to share this with the community at no charge.

Nice, simple, idea! Symlinks aren't usable by hackers without FollowSymLinks, and if it checks for an owner match always, there's no security issue. And the use of /scripts/before_apache_make means it's a few seconds work to install. Thanks again!

 

[storminternet]

Thanks Steven; that's extremely generous of you to share this with the community at no charge.

Nice, simple, idea! Symlinks aren't usable by hackers without FollowSymLinks, and if it checks for an owner match always, there's no security issue. And the use of /scripts/before_apache_make means it's a few seconds work to install. Thanks again!

Indeed setting up SymLinksIfOwnerMatch in apache conf improve the symlink protection and hence server security but that affect server performance. However server security can not be compromised against server performance.

 

[hostnex]

Atleast patch is not working for us. If someone need step by step guide to replicate please contact.