How to Fix the ASVs

McAfee has been hit very hard this week, and the vulnerabilities in the other ASVs that I published are starting to make waves as well. While it's fun (and it really is) to point a finger and say "LOL! McAfee Fail!", this really isn't helpful. It's time we take this as an opportunity to look at things that are seriously wrong with PCI and the way that it is implemented.

I don't claim to be an expert on the PCI standards. I am just one of many people who is trying to improve the security of his company and the internet as a whole. In my view, achieving compliance should be an afterthought. While any business executive worth his paycheck should understand and value security from a risk-management perspective, the fact is that most respond better to industry requirements. Regulation is a dirty hack, but it at least gets the attention it needs.

Since the service providers already understand that they need to comply with these regulations, we may as well fix them so that they are effective. In my view, here are the major changes that need to be made:

Better Education
For many small service providers, the PCI testing process is the first time they look seriously at their own security. Having them sign a few forms and running a scan on their servers does not educate them, and it does not prepare them to deal with security properly. This sets a poor precedent for a growing company, and in my experience, it invariably causes problems down the road. Having an expert consultant educate the service provider about how to design a security program would do more to protect cardholder, customer and company data than any set of prescriptive security regulations. This phase of the process is usually skimmed over by a company trying to get the not-so-elusive compliance certificate. In my opinion, it may be the most important part.

Better documentation
Without clear definitions of the standards and recommendations for compliance, we'll never get far. Most service providers do not even have a clear understanding of which standards they are expected to comply with, much less how to do so. Instead, they speak with the sales department at the cheapest ASV they can find, perform a scan, and send the vulnerability reports to their system administrators.

To be fair, there is documentation out there, but there's a lot of misinformation too. The official PCI FAQ (which is itself vulnerable to XSS) doesn't provide much useful information beyond the standards themselves and a few supporting documents. Several unofficial forums have sprung up, but the information contained within isn't always reliable.

More Responsibility
To my knowledge, this week's McAfee issues, as well as the issues with other ASVs, have not affected their status as a PCI scanning vendor at all. Given that some of the ASVs' XSS holes have still not been fixed, I find this extremely shocking. PCI companies need to treat customer data responsibly, and need to be held responsible when they do not. Revoking their ability to perform PCI scans would be disastrous to their business, but the stated goal of the PCI standards isn't to keep poorly managed companies in business. If the true goal of these standards is to "help organizations proactively protect customer account data", then the ASVs need to be held responsible when they do not.

PCI ASVs should be held to much more rigorous standards than the Service Providers they certify. I was shocked to find out that regular third-party penetration tests are not a requirement for ASVs. From what I understand (and I may be misinformed), they essentially fill out paperwork and assure the PCI board that they are themselves compliant. Does this strike anybody else as insane?

More Transparency
While the PCI Council provides documentation of proper scanning procedures, they do not go into much depth. In particular, they barely touch on web application scans, while in my biased opinion, this is one of the most vulnerable and therefore critical parts of most Service Providers' systems.

No data is available to the public regarding the relative efficacy of the scans. I'm not aware of anybody with a truly effective web app scan, but I've run several on the same servers (in a rather unscientific fashion), and come up with vastly different results. The free PCI scan offered by nCircle took less than an hour, and only did a cursory inspection of my web application. By contrast, McAfee's scan took close to 3 hours, most of which was spent testing the web application. Neither found the rather basic XSS and SQLi holes that I deliberately put in.

While neither app was up to par, the McAfee scan was clearly superior- at least at finding web application holes. At the time, I was more interested in finding problems with the scanning services, rather than doing a direct comparison of the two. However, it would not be difficult to design a system for measuring the effectiveness of ASV scans. The PCI council should do so. They should make the results of these tests public, and should make scanners' status as an ASV dependent upon them.

Better Enforcement
An end user has a right to know whether the company he is doing business with is PCI compliant or not. A verification process, even as simple as a list of compliant companies on the PCI website, is essential to an effective certification program. Of course, when problems are found, the Service Provider in question should have their certification revoked.

I have come across many companies with poor security practices, and many simply do not care. The financial institution that I bank with used to have its access logs publicly accessible. I first alerted them to the issue over a year ago, and no action was taken. In the course of my PCI research, I coincidentally found out who their auditor was, and after alerting them to the issue, the logs were removed almost immediately.

Ideally, I should not be able to find out who performs the auditing on a company (lest I perform attacks through the ASV). However, only by reporting this issue to the auditor was I able to get it fixed. If I were able to report it directly to the PCI board, and a system were in place for getting that report to the right people, this would not have been a problem.

Better Testing
Finally, we arrive at the elephant in the room: the testing performed by the ASVs sucks.

Scanning has a place in security- it can find unpatched software, poor configuration, and other issues that often go unnoticed. However, until we have strong AI in the scanners, it will never have the ability to detect logic flaws or reliably fuzz for application vulnerabilities.

James Lester (Senior Security Analyst for McAfee, of all places) touched on this last month, and I think he's right- scanning is not effective in and of itself, and should not be relied on as the sole method of testing for PCI compliance. A full-scale audit will find issues that no scanner can, and more importantly, it will give auditors a reliable "feel" for the security of a Service Provider. Any provider that builds their own applications for payment and customer data management should be required to perform a QSA audit, regardless of the scale of the application.

Those Service Providers that cannot afford a full audit should not be allowed to build their own applications. Instead, they should use one of the already approved applications, on a sandboxed or private system. Scanning should still be required, but this way we can limit the scope of this scan to finding out-of-date software, unpatched holes, and configuration issues- things that the scanners are already very good at.

Abusing PCI Scanners

My company has a client with a small but successful ecommerce site. This site is a prime example of what I'd like all ecommerce sites to look like- thoroughly tested code, FTP disabled and no other unnecessary services running. The client is careful to make sure that patches stay up to date, and system logs are regularly reviewed.

It's one of the few companies that I would trust to handle security themselves. Unfortunately, this client needs to have his website tested for PCI compliance. He signed up with an ASV, and they started running scans. When the results came back, it showed every port on the server as being closed. A bit of research determined that the IDS had kicked in during the initial portscan and blocked the scanner from accessing his website at all- exactly as it should have. I received an email from the client:

They told me that [their server] needs to be 'whitelisted' - Their IP address needs to be listed as 'friendly'.

Essentially, we had to whitelist their IP on both IDS and firewall in order to get the PCI compliance certificate. This is pretty common throughout the industry, and I've heard this request from clients using a variety of ASVs.

Any security person should notice the problem though- the scanning system could not get past the IDS, so we opened the door and invited it to come in and break things.

I don't think many people have looked into the implications of this. Security administrators routinely poke holes in their own systems for the purpose of achieving PCI compliance. If that scanner were to be compromised in any way, it would leave my client's system (and thousands of others) vulnerable.

On Monday, I published details about a critical McAfee CSRF hole. Yesterday, I disclosed vulnerabilities in a handful of PCI ASVs' web sites and customer portals. Compromising the security administrator's account with web exploits is just one way to abuse the scanning service. Here are a few more-

• Client credentials can be compromised through offline attacks. In a recent weekend trip, I was driving by an ASV's office. On the top of their dumpster was a stack of customer signup forms with usernames, passwords and contact info- faxed in, entered into the system and discarded without even a trip to the paper shredder.


• The auditors themselves can be compromised in the same way- next to that customer information were printed pages from the administrator side of the ASV's website. Sales scripts, emails, and everything necessary to perform a social engineering attack were there too.


• Once the client or auditor accounts are compromised, the attacker gets lists of servers-including "secret" database, backup, infrastructure, development servers and others that shouldn't be publicly known. Along with the servers, they get nicely-formatted reports of all the vulnerabilities found


• In general, A few servers will perform the scans on all an ASV's clients. If compromising accounts isn't to your taste, you could just sign up and pay for the service. In most cases, all that's keeping you from scanning systems other than your own is a service agreement. As one employee at an ASV noted: "I am...waiting for a rogue account to use an ASV to scan sites that they don't have privileges to scan...to assist them with identifying vulns." Would they know if it had happened? With IDS disabled and scans already performed at random intervals, who would notice? It may not have happened yet, but it's bound to happen eventually.


• Knowing the source IP of the scanner (and knowing that it has full access to the target system), packets may be spoofed and a malicious payload delivered.


• Many ASVs use Nessus to perform these scans. It would be theoretically possible to piggyback a targeted, malicious payload with a normal plugin update. This probably wouldn't be worth the effort, but the idea of subverting the entire security community to execute a single attack appeals to me in a sick way.

I understand the good intentions and even the necessity of performing website scans. I should be clear: I am all for the testing of applications and systems, and I think that by and large, most systems are better off because they perform these PCI scans. The problem is that when we are trying to get that compliance certification, we approach it with an engineer's mindset ("How can I make it work?") rather than an auditor's mindset ("How can I make it fail?"). Even in the process of securing our systems, we need to stop at every step and consider the implications of our actions.

Assuming that the PCI ASVs treat clients and data with due care, these attacks are mostly not practical. Considering how irresponsible these security companies often are, inviting them to attack your server is likely a Bad Idea.

Epic Failure from McAfee

When you outsource all your PCI scanning to another company, you'd expect that company to be careful with your data. First off, there's the fact that security is what they do- if that data were to get leaked, you'd expect the damage to reputation alone to be crippling.

Second, (and probably the least important, really), the PCI ASV validation requirements (section 4.5.1, if you're interested) make it pretty clear that client data should be kept very confidential.

Finally, the data contained on those servers- lists of vulnerabilities on all their clients' systems, along with security administrators' passwords and more, is pretty high on the list of Things You Don't Want Getting Out.

You know what's coming.

We've found problems with McAfee time and time again. I was saving this for a post later this week, but given recent events, I think the time is right.

Until last week, McAfee Secure was vulnerable to critical CSRF holes. Not little ones, or ones that were difficult to exploit- basic, zero-kowledge, classic GET-based total-account-compromise holes. I think the pictures tell the story:









I sent a vulnerability report to a contact at McAfee and they responded quickly, communicated with me, and fixed the issue. They also began reviewing their entire codebase for similar holes. I commend them for that, but I cannot overlook the fact that this vulnerability never should have existed in the first place.

It's not going to be a surprise to anybody, but this is solid proof that McAfee Secure is mis-named. Their failure here is on an epic scale:

• They did not comply with PCI requirements for ASVs


• They did not use a secure software development lifecycle in building this application


• An in-depth penetration test should catch an issue like this, so I presume that no such audit has taken place


• Until I reported it, they had never performed a full code review for web vulnerabilities


• As you can see in the 1st and 3rd screenshots, the application was itself certified as McAfee Secure when I performed this demonstration.


• At no point in five weeks from me finding the vulnerability and it being fixed was that McAfee Secure logo removed from their own website.

The ultimate and obvious irony, however, is that McAfee Secure is in the business of testing others' web applications. I'll be the first to say that they are not equipped to do so.

I have a lot more information coming on this and related issues. Try the veal, tip your waitress, I'll be here all week.

Is Mass Hosting PCI Compliant? (Biting the hand that feeds me)

If anybody actually pays attention to what I have to say, there's a good chance that this post is going to make a lot of people awfully mad. It needs to be said.

Most web sites built by my company are mass-hosted on cPanel servers. We're not the only company around that does this- there are many more. On the internet, there are hundreds of thousands of mass hosting servers, with hundreds of users on each server. On our systems, about 1/3 of those accounts have some kind of ecommerce software. I don't know whether that fits with industry averages, but it paints a pretty interesting picture.

The internet has a massive number of ecommerce web sites, all processing credit card transactions, hosted on servers with at least a few hundred other users. Let's presume one thing for the sake of this post: breach of one account can lead to compromising another account through local attacks (world-readable configuration files, local file inclusion, and a whole host of others that I haven't publicly disclosed yet) as well as web-based attacks (see my previous post on server and client attacks via mod_userdir). I'll be posting a lot more ways to attack shared hosting in the future, but humor me and take it for granted if you must.

I've seen some debate as to whether PCI standards apply to ecommerce apps that outsource the payment processing to a third party like PayPal or Authorize.net, but I honestly don't think it should matter. First off, any site that handles CC data, even if they are not stored on the system, can be abused. If I can break into your Zen Cart installation and 'customize' your payment module to email those numbers to me, it doesn't matter who does the payment processing. If you have a payment form, the PCI standards should apply to you.

But here's the thing: every one of my company's servers gets scanned for PCI compliance. Every one of them passes, and I've got the certificate to prove it. When you hit a cPanel server directly (via IP address), only one website will come up, and it is usually a default cPanel page, which is pretty minimal. Each of these servers has hundreds more accounts that don't even get touched by McAfee's web app scanner.

Finding XSS holes on at least one of those accounts is practically guaranteed. In fact, when I first started in my current position, I was able to gain shell access on any given server in less than 5 minutes. I've since been able to make incredible progress with mass patchings and my own vulnerability research, but there are enough unknown issues and custom code that I'll never be able to fix them all.

So why are these servers being certified as PCI compliant?

Now, my company don't actually advertise that our shared servers are PCI compliant, but we do test them as part of our own merchant agreements. In addition, we've greatly improved our processes and sales practices to make sure that new code is reviewed and apps with sensitive data go on VPSes, but that does nothing for all the old clients, and even less for the millions of other sites out there.

The main point here, of course, is that these servers are consistently tested as PCI compliant. This isn't really a attack on McAfee. Many of our clients use other ASVs for their own sites, and none have a problem with the fact that these sites are mass-hosted, nor the fact that other sites on the same server are not PCI compliant (though if they tried to scan the other sites on the server, I'd put a stop to it pretty quickly, for obvious reasons).

Can somebody with more background in PCI weigh in on this? Is this just a massive oversight on the part of every ASV out there? When my company gets big enough that they need a PCI QSA audit rather than the ineffective ASV scans, are we going to be have to rebuild all of our customers' websites? Are we going to be completely screwed compliance-wise?

Diary of a Hacker - Part 1

Here's the first of many writeups for my "Hacker Challenge". Keep in mind, this isn't a penetration test- it's just a train-of-thought writeup of how I cracked an application. The idea is to demonstrate instances of various attacks in use, rather than the theoretical proof-of-concepts that appear in most security web sites.

The site is a file repository- basically customers can pay a monthly fee to have access to a virtual filing cabinet-type-thing. I don't think that's a fabulous business model, but that's not the point of this blog. After familiarizing myself with the site, my first attempts were file upload attacks (naturally, given the purpose of the site). Fortunately, the site used several layers of protection for uploaded files- first, it only allowed a whitelist of filetypes, preventing malicious files from being uploaded. Next, it stored those files in a non-web-accessible directory. Even if I could get a malicious file uploaded, I could not access it directly. Finally, the filenames were randomly generated- if I found a way to circumvent the access control, I still would not know the name of my malicious file.

Basically, this type of attack didn't work; I moved on.

Throughout the site, inputs were validated thoroughly (excessively in some cases). No matter where I looked and how much I mangled the inputs, I could not find an SQL injection hole. I did find a reflected XSS hole in an obscure stock script, so I decided to work with that. My goal was to
steal an admin session using Javascript, and after setting up my evil little javascript-session-stealing kit (I'm quite proud of it), found out that this also wouldn't work so easily. The developer had added the following directive to php.ini:

session.cookie_httponly = 1

This directive made it so that session IDs were inaccessible to client-side Javascript. I could have still used the XSS hole for other things, such as forcing the administrator to perform actions on my behalf, but this would require work and inside knowledge to code, and would have a relatively low chance of success.

I'm rather lazy, so I moved on.

I went to myipneighbors.com and found a list of sites on the server. It didn't take me long to find one with some known vulnerabilities, and next thing I knew, I had filesystem access. Faithful readers already know where I'm going with this, but mass-hosted servers are a Bad Idea. Even so, it is possible to configure file permissions and code a site so that an attacker can't simply walk in and read your configuration files.

I whipped up a quick script and ran it on the compromised account (The server runs FreeBSD and cPanel). This script monitored the output of `ps -aux` while fetching a page from the target account. Because of suexec, I immediately knew the username of the targeted website.

With this information, I tried to read files from the public_html directory, but the permissions were set up (mostly) correctly and prevented access. While I couldn't list the files in the directory, I could guess at their filenames, and eventually landed on one file that was readable- php.ini.

Reviewing the site's php.ini file, I could tell the server was relatively locked down, but one line in particular caught my attention:

session.save_path = /home/theSiteUsername/tmp/

I tried to list the files in that directory and was able to view a list of open sessions' IDs. This was the hole I needed to compromise the account:

$ ls -la /home/theSiteUsername/tmp/ | grep sess
-rw-------   1 theSiteUsername  theSiteUsername  5068 Sep 18 09:20 sess_1a1aexxxxxxx5e282e1a5747694ec8e2
-rw-------   1 theSiteUsername  theSiteUsername  5023 Sep 18 09:19 sess_462fexxxxxxx13dafa331bc15e962e22
-rw-------   1 theSiteUsername  theSiteUsername  5109 Sep 18 10:51 sess_717edxxxxxxx653df341f91d096e0484
-rw-------   1 theSiteUsername  theSiteUsername  5079 Sep 18 09:14 sess_91850xxxxxxxef480524c23412b23dc1
-rw-------   1 theSiteUsername  theSiteUsername  5011 Sep 18 10:38 sess_9246dxxxxxxxe6e710a9e623af23ca95
-rw-------   1 theSiteUsername  theSiteUsername  5079 Sep 18 09:25 sess_a4cdcxxxxxxx065ede6ecb062d443f86

I then manually set the SESSID cookie in my browser to those found in the session filenames, and confirmed that I could hijack other accounts' sessions. Unfortunately, none of those sessions were administrators. I wrote a shell script to watch the directory in question and send a text message to my cell phone when a new session is created, but no admins logged in over the 50-something
hours that I waited.

While this tactic would eventually work, I wanted to speed things up. I sent the following email to the programmer:

Log into the admin section and check the 'edit_users.php' page.

;)

The developer and I have done a fair amount of trash-talking about this account in the past, so worried that I had successfully hacked the site, he logged in to check the admin section. As soon as his new session was created, I hijacked it, accessed the admin page, and snagged a screenshot as proof.

So what could have been done to prevent this attack? A few little things.

First, making the ~/tmp directory globally unreadable would have prevented me finding session IDs. This isn't failproof, as there are other ways I could have found them, but every little bit helps.

Second, adding further session validation would have made hijacking the session very difficult. This can be as simple as storing the source IP address in the $_SESSION array on login, then comparing it on each pageload to make sure the user isn't suddenly coming from the other side of the world. Admittedly, there are legitimate uses that would have issues (web proxies and Tor, specifically), but it would have closed the hole. If the site's user base are likely to be behind proxies, there is other information that can be used to identify the browser uniquely- client's user agent, supported protocols, OS, connection latency, etc.

The biggest lesson here, though, is to follow the 'defense in depth' mantra.

Diary of a Hacker - Introduction

I work for a web development and hosting firm, where have an ongoing competition. Our webapp developers submit their newly-built sites for a voluntary cracking attempt. If I cannot in some way compromise, abuse, DOS, or gain escalated privileges, the developer in question gets a monetary prize.

So far, I have not failed, and the prize keeps getting bigger.

Of course, it isn't really a code audit or even a penetration test. Primarily, it's a way for me to hone my own skills while informing our developers about web security issues. The contest has a somewhat limited scope- I must get in by abusing a hole in the code or configuration, so social engineering, server configuration, and other attacks are out (at least as the primary attack vector, though I may use them as leverage against another hole). It also is limited to single-developer applications, so they tend to be small in scale.

I'm writing about this here for two reasons. First, I think it's a very good idea. We have had an extremely positive response from the developers. They code better than ever. They are actively engaged in maintaining the security of the applications, rather than building and then forgetting about them. The contest has sparked many insightful discussions about various security concepts, including CAPTCHA, PHP session management, and XSS filtering. If anybody out there manages a team of developers, I highly recommend holding a similar competition.

The second reason is because I've received permission from the higher-ups to publish some of the post-exploitation writeups on this blog. I hardly consider myself to be the best web application hacker out there, and I don't think I'm a definitive authority on the subject, but I've found (and I'm not alone) that very few non-security people, particularly the developers in charge of building the applications, know how an attacker works.

I'm hoping that by showing the thought process and techniques that I use, I can enlighten a few developers, project managers, and maybe even a few other security-folk.

A Grain of Salt

From time to time, I'm asked to evaluate a particular application for use on my company our our customers' servers. This is often depressing, as most of the ready-built web apps out there are poorly written at best. This is pretty much the status quo, it would seem.

I evaluated V3 Chat, a web-based instant messenger service, about a year ago for a client. Based on the high number of XSS and other holes in the software, I strongly recommended against it. The client found another solution, and life went on.

Today I was asked to look at it again, as another client wanted to use it and a new version has since been released. Within seconds, I found the first of many holes in their demo.

Normally, this wouldn't irritate me so much- recommend that they not use it, move on with life, etc. But the vendor's web site states the following as a bullet point:

"New! - Improved User Security. Greater safety and security for private chat conversations. Increased protection against XSS and MySQL attacks."

The really irritating part to me here is the fact that they are using improved user security as a sales point, when not just the original, but the "improved" version of the software would not pass basic PCI compliance testing. They haven't even fixed the XSS hole that I emailed to the vendor several weeks before sending it to XSSed.com about a month ago (this was partially the inspiration for my Open Letter to the Internet at about the same time). While this software is, in my professional opinion, lousy, this is nothing new to the web. Unfortunately, there is a lot of broken software out there, and very few vendors, much less their customers, realize this.

What would it take to fix the web? A lot, I'm afraid. Bruce Schneier keeps telling people that it starts with legal accountability on the vendor's part, which I'm inclined to agree with. In the meantime, you can secure your own web services with a healthy dose of skepticism (hence, the title of my blog). It's common enough that it's not even news anymore, but I've seen far too many instances of encrypted, unbreakable, or "improved" software turning out to be insecure.

Whenever my company is looking to use some commercial software, they run it by me first. I talk to the sales representative and am always told that the product is perfectly secure, will solve all my problems and whiten my teeth while I sleep. Invariably, I get a demo copy and find holes in the software.

This in itself isn't always enough to get a negative recommendation; I send the vendor my findings, and watch how quickly the issues are resolved. After evaluating the vendor's response to the problems, I can finally make a recommendation. Usually, we end up building whatever we needed in-house. The times that we don't, we usually wish we had.

Often, the vendor is the most surprised to find out that their magical software is full of holes. At least, they act surprised. Don't trust the marketing, don't trust the reviews in ad-supported-magazine weekly. If it's going to be used somewhere important, it's worth your time to look for yourself. If the vendor won't let you demo a copy, move on. If you haven't the time or the knowledge to evaluate it, I have both, but less money than I would prefer.

That's my rant for the day.