Thoughts on HTTP Parameter Pollution

Luca Carettoni and Stefan di Paola presented at OWASP Appsec Poland, speaking about HTTP Parameter Pollution. The short version is this- by mangling the inputs for a HTTP request, you can cause all kinds of strange behaviors. On a high level, this isn't too different from SQLi, XSS, and plenty of other attacks. What sets HPP apart is that the quirks you can generate aren't all data sanitization issues- they are often logic bugs, which are much harder to test for, easier to introduce into an application, and more fun to exploit.

The presentation involved a few very different attacks, which makes it a bit difficult to summarize, but essentially, it involves overwriting HTTP parameters. A few examples:

PHP applications (and others) primarily get their user inputs from COOKIE, GET, and POST. If the application reads input from the generic REQUEST parameter, the data could conceivably come from any of these, and developers don't always have a clear understanding of how that decision is made. By abusing the order in which inputs are handled, an attacker can generate unexpected behavior in the application- expecting data to come from POST but instead reading it from GET, for example.

This really isn't new- the register_globals mess in PHP stems from a very similar issue. While register_globals is dying a well-deserved death, similar attacks can still be made against the $_REQUEST parameter, which many developers (usually the same ones that used register_globals) rely on. Java, perl, and ASP have all seen the same issues, and are at varying levels of fixed-ness.

The more interesting and sneaky attack: If you pass a query string with multiple copies of the same parameter (e.g. index.php?foo=bar&foo=baz), the application may handle it in unexpected ways. PHP will use the last occurence of that parameter, JSP will use the first occurence, mod_perl converts it to an array, and ASP concatenates the parameters into a comma-separated string. Custom-buit parsers on embedded hardware can be even more surprising.

Now, if the developer is doing his job right- being paranoid about where his inputs come from, validating, and escaping, this should not be a major problem. However, no developer is perfect, and this being a fairly unknown behavior, it is quite likely to pop up. Quirks are edge cases, and edge cases are where you find security holes.

I recommend you go over the slides, and the whitepaper when it comes out (which should soon). Some very interesting, very cool real-world attacks are outlined there (I especially like the ASP SQL reconstruction), but I won't go into them here. In summary, you can leverage these parameter processing quirks against both the server and the client, in a lot of interesting ways:

• Bypassing input validation


• Bypassing WAFs


• Manipulating application flow


• Manipulating mod_rewrites


• Forcing/spoofing cookies


• Manipulating client-side applications


• Manipulating and polluting page content

As a pentester, I absolutely love these kinds of attacks- they exploit seemingly-benign quirks in the application. They rely on the attacker knowing more about how the application works than the person who wrote it. Besides that, they're wicked fun to play with. I see HPP as yet-another useful tool in my arsenal. Like CSRF and XSS, it is (and will continue to be) extremely common in poorly built applications. Well-designed apps will have minimal attack surface, defense in depth will further limit their exposure, and what holes do exist will be quickly remediated.

HTTP Parameter Pollution isn't an entirely new attack, but it is a much more in-depth revisiting of older flow-control and input parsing issues, specifically at the border between the application and its back-end engine. I expect to see a surge of HPP-related vulnerability reports in the future. From the defender's perspective, though, it is the same story: Know your inputs, know where data flows through the app, validate before you use it, and escape it on the way out.

Owning Ubuntu with CSRF

When the latest version of Ubuntu was released, it came with a new version of the Transmission bit torrent client. One of the new features of Transmission was a web interface, which would allow users to manage torrents from their web browsers. The service is ip-restricted to localhost.

Why the service should be enabled by default, or why the local user (the only one able to access it by default) would ever want it, I'll never know, but it should be no surprise to anybody that I immediately poked around for CSRF holes.

As it turns out, the holes were there, and though a bit tricky to exploit- it required mangling the form inputs and using the text/plain encoding, Transmission was quite vulnerable. Here's a proof-of-concept. Basically, it allows me to first change the download destination to one of my choosing, then download a torrent of my choosing. This is bad- I could force a user to download a new .bashrc file or overwrite their entire home directory (I actually did this by accident while testing the exploit. Backups are great).

Other problems do make it harder to exploit. First, when you start the torrent, it pops up a box asking the administrator what he wants to do with it. Many people, including myself, have it configured to skip this step, as it's extremely annoying.

When you make the CSRF requests, the result is returned in JSON format, and browsers without handlers for it will pop up a box asking if you'd like to save the file. When this box pops up, it is already too late, as the request has already been processed by the server, but it does at least keep the attack from being silent.

This isn't really a new exploit- Nate McFeters talked about doing the same thing through an XSS hole in the Azureus web interface in his talk, "the Internet is Broken", at Black Hat 2008.

The hole is fixed in today's release, so go ahead and upgrade, then disable that web server.

Please, let's start taking CSRF seriously.

PCI Smackdown: Looking Forward

I've been publishing a lot of holes in security companies' websites, and more are coming next week. While I'll continue to point out these flaws, I'm hoping that we can turn these disclosures into industrywide changes.

I know I've pissed some people off. I'm pretty sure McAfee isn't going to be giving me much love in the future. I personally think that too much attention has been paid to them in particular. Though their official response was pretty lame and not entirely true, they definitely aren't the worst offenders. I must admit, however, if it hadn't been for that issue making big headlines, I don't think as many people would be listening to me now.

At this point, I want to issue an industry-wide challenge- any ASV wants me to evaluate their PCI scanning service need only contact me. I will perform rigorous tests, find whatever issues I can, and publish them in brutally honest fashion. I will provide recommendations to the vendors on how to improve the services, and if I'm impressed with them, I will certainly make that known publicly as well. I will give credit where it is due, but I will happily call you out if you've done something stupid. I have no love or hate for any PCI companies outside my own experiences, and I have no agenda beyond improving the industry.

Do I expect any vendors to take me up on this challenge? No. I am skeptical by nature, but I would love to have somebody take the lead and prove me wrong.

I'm out for the weekend, but I'll leave you with one last batch of screenshots. These are error messages from a handful more PCI ASVs, none of which are critical security issues, but all of which disclose information about the internal workings, directory structure, or configuration of their respective applications. As before, all of these would be found with a simple website audit.

An SQL error in ControlScan's trustmark validation script


Counterpane CGI Error with file path disclosure
Thanks to Russ McRee for this one


onestoppciscan.com Error Page


secureconnect.com ASP errors with stack trace


Symantec included file manipulation and file path disclosure
This script actually throws about half a dozen different errors, depending on how you manipulate it

How to Fix the ASVs

McAfee has been hit very hard this week, and the vulnerabilities in the other ASVs that I published are starting to make waves as well. While it's fun (and it really is) to point a finger and say "LOL! McAfee Fail!", this really isn't helpful. It's time we take this as an opportunity to look at things that are seriously wrong with PCI and the way that it is implemented.

I don't claim to be an expert on the PCI standards. I am just one of many people who is trying to improve the security of his company and the internet as a whole. In my view, achieving compliance should be an afterthought. While any business executive worth his paycheck should understand and value security from a risk-management perspective, the fact is that most respond better to industry requirements. Regulation is a dirty hack, but it at least gets the attention it needs.

Since the service providers already understand that they need to comply with these regulations, we may as well fix them so that they are effective. In my view, here are the major changes that need to be made:

Better Education
For many small service providers, the PCI testing process is the first time they look seriously at their own security. Having them sign a few forms and running a scan on their servers does not educate them, and it does not prepare them to deal with security properly. This sets a poor precedent for a growing company, and in my experience, it invariably causes problems down the road. Having an expert consultant educate the service provider about how to design a security program would do more to protect cardholder, customer and company data than any set of prescriptive security regulations. This phase of the process is usually skimmed over by a company trying to get the not-so-elusive compliance certificate. In my opinion, it may be the most important part.

Better documentation
Without clear definitions of the standards and recommendations for compliance, we'll never get far. Most service providers do not even have a clear understanding of which standards they are expected to comply with, much less how to do so. Instead, they speak with the sales department at the cheapest ASV they can find, perform a scan, and send the vulnerability reports to their system administrators.

To be fair, there is documentation out there, but there's a lot of misinformation too. The official PCI FAQ (which is itself vulnerable to XSS) doesn't provide much useful information beyond the standards themselves and a few supporting documents. Several unofficial forums have sprung up, but the information contained within isn't always reliable.

More Responsibility
To my knowledge, this week's McAfee issues, as well as the issues with other ASVs, have not affected their status as a PCI scanning vendor at all. Given that some of the ASVs' XSS holes have still not been fixed, I find this extremely shocking. PCI companies need to treat customer data responsibly, and need to be held responsible when they do not. Revoking their ability to perform PCI scans would be disastrous to their business, but the stated goal of the PCI standards isn't to keep poorly managed companies in business. If the true goal of these standards is to "help organizations proactively protect customer account data", then the ASVs need to be held responsible when they do not.

PCI ASVs should be held to much more rigorous standards than the Service Providers they certify. I was shocked to find out that regular third-party penetration tests are not a requirement for ASVs. From what I understand (and I may be misinformed), they essentially fill out paperwork and assure the PCI board that they are themselves compliant. Does this strike anybody else as insane?

More Transparency
While the PCI Council provides documentation of proper scanning procedures, they do not go into much depth. In particular, they barely touch on web application scans, while in my biased opinion, this is one of the most vulnerable and therefore critical parts of most Service Providers' systems.

No data is available to the public regarding the relative efficacy of the scans. I'm not aware of anybody with a truly effective web app scan, but I've run several on the same servers (in a rather unscientific fashion), and come up with vastly different results. The free PCI scan offered by nCircle took less than an hour, and only did a cursory inspection of my web application. By contrast, McAfee's scan took close to 3 hours, most of which was spent testing the web application. Neither found the rather basic XSS and SQLi holes that I deliberately put in.

While neither app was up to par, the McAfee scan was clearly superior- at least at finding web application holes. At the time, I was more interested in finding problems with the scanning services, rather than doing a direct comparison of the two. However, it would not be difficult to design a system for measuring the effectiveness of ASV scans. The PCI council should do so. They should make the results of these tests public, and should make scanners' status as an ASV dependent upon them.

Better Enforcement
An end user has a right to know whether the company he is doing business with is PCI compliant or not. A verification process, even as simple as a list of compliant companies on the PCI website, is essential to an effective certification program. Of course, when problems are found, the Service Provider in question should have their certification revoked.

I have come across many companies with poor security practices, and many simply do not care. The financial institution that I bank with used to have its access logs publicly accessible. I first alerted them to the issue over a year ago, and no action was taken. In the course of my PCI research, I coincidentally found out who their auditor was, and after alerting them to the issue, the logs were removed almost immediately.

Ideally, I should not be able to find out who performs the auditing on a company (lest I perform attacks through the ASV). However, only by reporting this issue to the auditor was I able to get it fixed. If I were able to report it directly to the PCI board, and a system were in place for getting that report to the right people, this would not have been a problem.

Better Testing
Finally, we arrive at the elephant in the room: the testing performed by the ASVs sucks.

Scanning has a place in security- it can find unpatched software, poor configuration, and other issues that often go unnoticed. However, until we have strong AI in the scanners, it will never have the ability to detect logic flaws or reliably fuzz for application vulnerabilities.

James Lester (Senior Security Analyst for McAfee, of all places) touched on this last month, and I think he's right- scanning is not effective in and of itself, and should not be relied on as the sole method of testing for PCI compliance. A full-scale audit will find issues that no scanner can, and more importantly, it will give auditors a reliable "feel" for the security of a Service Provider. Any provider that builds their own applications for payment and customer data management should be required to perform a QSA audit, regardless of the scale of the application.

Those Service Providers that cannot afford a full audit should not be allowed to build their own applications. Instead, they should use one of the already approved applications, on a sandboxed or private system. Scanning should still be required, but this way we can limit the scope of this scan to finding out-of-date software, unpatched holes, and configuration issues- things that the scanners are already very good at.

An Apology to nCircle

In yesterday's post on ASV vulnerabilities, I said that nCircle did not respond to my report and belonged in the doghouse. In fact, they were one of the best, and had the vulnerability fixed within mere minutes of my report. I had lists of good and bad ASVs and they accidentally ended up in the wrong one.

Apologies to nCircle, and thanks to Andrew Storms for pointing it out. I've corrected yesterday's post.

Abusing PCI Scanners

My company has a client with a small but successful ecommerce site. This site is a prime example of what I'd like all ecommerce sites to look like- thoroughly tested code, FTP disabled and no other unnecessary services running. The client is careful to make sure that patches stay up to date, and system logs are regularly reviewed.

It's one of the few companies that I would trust to handle security themselves. Unfortunately, this client needs to have his website tested for PCI compliance. He signed up with an ASV, and they started running scans. When the results came back, it showed every port on the server as being closed. A bit of research determined that the IDS had kicked in during the initial portscan and blocked the scanner from accessing his website at all- exactly as it should have. I received an email from the client:

They told me that [their server] needs to be 'whitelisted' - Their IP address needs to be listed as 'friendly'.

Essentially, we had to whitelist their IP on both IDS and firewall in order to get the PCI compliance certificate. This is pretty common throughout the industry, and I've heard this request from clients using a variety of ASVs.

Any security person should notice the problem though- the scanning system could not get past the IDS, so we opened the door and invited it to come in and break things.

I don't think many people have looked into the implications of this. Security administrators routinely poke holes in their own systems for the purpose of achieving PCI compliance. If that scanner were to be compromised in any way, it would leave my client's system (and thousands of others) vulnerable.

On Monday, I published details about a critical McAfee CSRF hole. Yesterday, I disclosed vulnerabilities in a handful of PCI ASVs' web sites and customer portals. Compromising the security administrator's account with web exploits is just one way to abuse the scanning service. Here are a few more-

• Client credentials can be compromised through offline attacks. In a recent weekend trip, I was driving by an ASV's office. On the top of their dumpster was a stack of customer signup forms with usernames, passwords and contact info- faxed in, entered into the system and discarded without even a trip to the paper shredder.


• The auditors themselves can be compromised in the same way- next to that customer information were printed pages from the administrator side of the ASV's website. Sales scripts, emails, and everything necessary to perform a social engineering attack were there too.


• Once the client or auditor accounts are compromised, the attacker gets lists of servers-including "secret" database, backup, infrastructure, development servers and others that shouldn't be publicly known. Along with the servers, they get nicely-formatted reports of all the vulnerabilities found


• In general, A few servers will perform the scans on all an ASV's clients. If compromising accounts isn't to your taste, you could just sign up and pay for the service. In most cases, all that's keeping you from scanning systems other than your own is a service agreement. As one employee at an ASV noted: "I am...waiting for a rogue account to use an ASV to scan sites that they don't have privileges to scan...to assist them with identifying vulns." Would they know if it had happened? With IDS disabled and scans already performed at random intervals, who would notice? It may not have happened yet, but it's bound to happen eventually.


• Knowing the source IP of the scanner (and knowing that it has full access to the target system), packets may be spoofed and a malicious payload delivered.


• Many ASVs use Nessus to perform these scans. It would be theoretically possible to piggyback a targeted, malicious payload with a normal plugin update. This probably wouldn't be worth the effort, but the idea of subverting the entire security community to execute a single attack appeals to me in a sick way.

I understand the good intentions and even the necessity of performing website scans. I should be clear: I am all for the testing of applications and systems, and I think that by and large, most systems are better off because they perform these PCI scans. The problem is that when we are trying to get that compliance certification, we approach it with an engineer's mindset ("How can I make it work?") rather than an auditor's mindset ("How can I make it fail?"). Even in the process of securing our systems, we need to stop at every step and consider the implications of our actions.

Assuming that the PCI ASVs treat clients and data with due care, these attacks are mostly not practical. Considering how irresponsible these security companies often are, inviting them to attack your server is likely a Bad Idea.

Risky.biz Podcast Interview

I just finished an interview with Patrick at Risky.biz for a podcast about my McAfee CSRF and the other PCI ASV vulnerabilities. If you're into this kind of thing (and the traffic spikes I've seen this week indicate you probably are), go give it a listen.

http://risky.biz/netcasts/rb2/rb2-mcafee-bug-finder-mike-bailey-speaks-riskybiz

Most PCI Companies Are Insecure

The McAfee XSS got slashdotted. I think that all this attention is a good thing, putting a spotlight on XSS issues, but I have to say, I'm surprised by it. It's not like XSS attacks are news anymore, and it's not as if this is the first McAfee XSS to be published. Last night, I found an XSS hole in the verification script for their SiteAdvisor service (for extra irony).



But really, focusing on these XSS holes is missing the point. I never thought I'd say this, but in my experience, McAfee is one of the better ASVs out there. This isn't a compliment to them, it's an insult to the entire industry. Here are a few examples of other ASVs.

Until last week, atsec.com was vulnerable to XSS.

Until last week, secureconnect.com was vulnerable to XSS.

Until last week, ncircle.com was still vulnerable to XSS.

sungard.com is still vulnerable to XSS.

controlcase.com is still vulnerable to XSS.

support.foundstone.com (McAfee's premium brand) is still vulnerable to Cross-site Framing.

Up until a few weeks ago, there were also open redirects on the websites of Qualys, SecurityMetrics, and others. Is it any wonder I'm not at all shocked at a few XSS holes in McAfee's web site?

Some of these companies should be commended for handling the vulnerabilities correctly- nCircle, SecureConnect, Qualys, and even McAfee responded admirably- sometimes the issue was fixed within minutes of my vulnerability report. Others- Foundstone, ControlCase, and Sungard, belong in the doghouse- none of them even responded.

However, the glaring fact is that the entire PCI scanning industry is, frankly, bad at scanning for vulnerabilities. Most of these websites use their own scanning service on their own websites. While I still hold that in-depth audits for these sites should have taken place long ago, the scanners should have caught the problems as well. Some of these domains contain the portals for customers to manage their PCI compliance scans.

People, let's take the focus off of McAfee, and put it where it belongs. The PCI scanning industry as a whole is a joke, and across the board, these Web Security companies are themselves bad at security.

Edit 5-6-2009: nCircle was one of the fast-responders.
I mistakenly listed them as one of the "doghouse" ASVs.


Epic Failure from McAfee

When you outsource all your PCI scanning to another company, you'd expect that company to be careful with your data. First off, there's the fact that security is what they do- if that data were to get leaked, you'd expect the damage to reputation alone to be crippling.

Second, (and probably the least important, really), the PCI ASV validation requirements (section 4.5.1, if you're interested) make it pretty clear that client data should be kept very confidential.

Finally, the data contained on those servers- lists of vulnerabilities on all their clients' systems, along with security administrators' passwords and more, is pretty high on the list of Things You Don't Want Getting Out.

You know what's coming.

We've found problems with McAfee time and time again. I was saving this for a post later this week, but given recent events, I think the time is right.

Until last week, McAfee Secure was vulnerable to critical CSRF holes. Not little ones, or ones that were difficult to exploit- basic, zero-kowledge, classic GET-based total-account-compromise holes. I think the pictures tell the story:









I sent a vulnerability report to a contact at McAfee and they responded quickly, communicated with me, and fixed the issue. They also began reviewing their entire codebase for similar holes. I commend them for that, but I cannot overlook the fact that this vulnerability never should have existed in the first place.

It's not going to be a surprise to anybody, but this is solid proof that McAfee Secure is mis-named. Their failure here is on an epic scale:

• They did not comply with PCI requirements for ASVs


• They did not use a secure software development lifecycle in building this application


• An in-depth penetration test should catch an issue like this, so I presume that no such audit has taken place


• Until I reported it, they had never performed a full code review for web vulnerabilities


• As you can see in the 1st and 3rd screenshots, the application was itself certified as McAfee Secure when I performed this demonstration.


• At no point in five weeks from me finding the vulnerability and it being fixed was that McAfee Secure logo removed from their own website.

The ultimate and obvious irony, however, is that McAfee Secure is in the business of testing others' web applications. I'll be the first to say that they are not equipped to do so.

I have a lot more information coming on this and related issues. Try the veal, tip your waitress, I'll be here all week.

Mass Hosting && PCI: A Case Study

Just as a quick follow up to my earlier post, here is a prime example of a "PCI Compliant" company who is probably not.

I attempted responsible disclosure several weeks ago and received no response from this company, a New York based web hosting and development company. I'm not trying to get them in trouble, so I redacted the screenshots and won't refer to them by name, but I have no doubt that anybody who cared could find them quickly enough.

As part of my research, I made a phone call to this company. I inquired about the PCI compliance of their servers, and the sales associate assured me that their servers are scanned regularly and are compliant. He did not tell me who their ASV is, and while I have my theories, I won't speculate publicly.



Regardless, this company has a login portal on their main web site. I know one of their customers, so I can tell you that the credentials used to log into this portal are used for customer cPanel accounts, billing system, and webmail. Those credentials are probably pretty valuable, right?



How did this site ever pass any kind of PCI testing? It's only a theory, but I suspect this is because it's on a mass-hosted server. When you hit the server directly by IP, it brings up the default cPanel page.



But this company was, by their claim, 100% PCI compliant. They told me that some customers also get their sites tested and are in turn PCI compliant.

Is Mass Hosting PCI Compliant? (Biting the hand that feeds me)

If anybody actually pays attention to what I have to say, there's a good chance that this post is going to make a lot of people awfully mad. It needs to be said.

Most web sites built by my company are mass-hosted on cPanel servers. We're not the only company around that does this- there are many more. On the internet, there are hundreds of thousands of mass hosting servers, with hundreds of users on each server. On our systems, about 1/3 of those accounts have some kind of ecommerce software. I don't know whether that fits with industry averages, but it paints a pretty interesting picture.

The internet has a massive number of ecommerce web sites, all processing credit card transactions, hosted on servers with at least a few hundred other users. Let's presume one thing for the sake of this post: breach of one account can lead to compromising another account through local attacks (world-readable configuration files, local file inclusion, and a whole host of others that I haven't publicly disclosed yet) as well as web-based attacks (see my previous post on server and client attacks via mod_userdir). I'll be posting a lot more ways to attack shared hosting in the future, but humor me and take it for granted if you must.

I've seen some debate as to whether PCI standards apply to ecommerce apps that outsource the payment processing to a third party like PayPal or Authorize.net, but I honestly don't think it should matter. First off, any site that handles CC data, even if they are not stored on the system, can be abused. If I can break into your Zen Cart installation and 'customize' your payment module to email those numbers to me, it doesn't matter who does the payment processing. If you have a payment form, the PCI standards should apply to you.

But here's the thing: every one of my company's servers gets scanned for PCI compliance. Every one of them passes, and I've got the certificate to prove it. When you hit a cPanel server directly (via IP address), only one website will come up, and it is usually a default cPanel page, which is pretty minimal. Each of these servers has hundreds more accounts that don't even get touched by McAfee's web app scanner.

Finding XSS holes on at least one of those accounts is practically guaranteed. In fact, when I first started in my current position, I was able to gain shell access on any given server in less than 5 minutes. I've since been able to make incredible progress with mass patchings and my own vulnerability research, but there are enough unknown issues and custom code that I'll never be able to fix them all.

So why are these servers being certified as PCI compliant?

Now, my company don't actually advertise that our shared servers are PCI compliant, but we do test them as part of our own merchant agreements. In addition, we've greatly improved our processes and sales practices to make sure that new code is reviewed and apps with sensitive data go on VPSes, but that does nothing for all the old clients, and even less for the millions of other sites out there.

The main point here, of course, is that these servers are consistently tested as PCI compliant. This isn't really a attack on McAfee. Many of our clients use other ASVs for their own sites, and none have a problem with the fact that these sites are mass-hosted, nor the fact that other sites on the same server are not PCI compliant (though if they tried to scan the other sites on the server, I'd put a stop to it pretty quickly, for obvious reasons).

Can somebody with more background in PCI weigh in on this? Is this just a massive oversight on the part of every ASV out there? When my company gets big enough that they need a PCI QSA audit rather than the ineffective ASV scans, are we going to be have to rebuild all of our customers' websites? Are we going to be completely screwed compliance-wise?