The Patch Paradox - To Patch or Not to Patch

Via GCN -

Security administrators faced a familiar if uncomfortable position: Just one day after Microsoft released an out-of-band patch to fix vulnerabilities in several versions of Windows, exploit code appeared in the wild.

The latest vulnerability -- with its near-same-day exploit code -- was too serious to go unpatched, experts warned, but it put administrators in a bad spot: Damned if they patched and damned if they didn't.

Industry watcher Gartner, for example, urged IT administrators to bite the bullet and patch their systems. "Microsoft made the right decision to issue an out-of-cycle patch for this vulnerability, given the evidence of active attacks against the Windows Server service and the ease at which exploits can spread across the majority of Windows systems," wrote Gartner analysts John Pescatore and Neil MacDonald. "Although Microsoft reports that these privately reported attacks have been limited and targeted, [it] is being very aggressive in pushing these patches out to prevent large-scale downtime for businesses around the world."

Gartner argued that the stakes were just too high for IT administrators to do otherwise.

[...]

All the same, risk-averse IT administrators probably still approach the prospect of installing an untested Microsoft patch with some degree of trepidation. More to the point, the systems management and software development lifecycles in most shops simply aren't designed with rapid (that is, untested) patch deployments in mind.

A Certified Information Systems Security Professional (CISSP) with consultancy and services firm Booz Allen Hamilton put it best: "The threat that comes from the outside is an unknown. In a case like this, it might never materialize. [So] if your systems go down because of an external threat...you're not off the hook, but it's more understandable than if you install [a patch] and it takes them down.

"If in the commission of updating and applying a patch you broke your own system, you're kind of stuck on the hook for that. Chances are that [management] won't buy it if you said you were only doing it to protect your systems."

That's why this security pro said it's still important to test -- even in cases where exploit code is in the wild.

"The best you can do is compress your testing schedule. The most immediate threat is that the patch could break your application, so you have to test it, even if the testing is compressed," the CISSP said.

It's a rule for which few, if any, exceptions are made. "When SQL Slammer appeared [in January of 2003], that was the only time in my life when I know that a patch was installed immediately. Other than that, the management process of the application system in question dictates how quickly a patch will be applied," the CISSP said. "You have to weigh the risk of the patch taking down your system versus whatever damage could be done by the exploit."

Approaches to patching can vary from organization to organization, too. Shops that are certified according to the rigorous capability maturity model index tend to be much charier about how or when they install patches.

"A CMMI organization would very much prefer to run a full regression test rather than just throw it on the server and hope for the best," the CISSP said, "so the typical approach [in CMMI shops] is to thoroughly test it and then to apply it -- but to apply it not right away but during the next scheduled patch cycle."

---------------------------------------------

As a disclaimer I would like to say that I am a former Microsoft SMS Patch Administrator. I was responsible for patching corporate desktop clients (over 800)....ranging from NT4 to Windows XP.

During that time, I was quoted multiple times in the press on the subject of patch management.
http://searchsecurity.techtarget.com/news/article/0,289142,sid14_gci1115378,00.html
http://news.cnet.com/Patching-up-problems/2100-7347_3-5553945.html

However, I am no patch management expert...nor do I claim to be one. In addition, I am also a CISSP holder...for whatever that means.

When those responsible choose not to patch their systems for a known vulnerability, they take several factors into account. How bad is the vulnerability? How many systems in our network are exposed? How critical are those systems? How important are those systems to our business and what type of information is stored on said systems? The answers to these questions can determine how fast the patch are pushed out (and how much testing is required beforehand).

However, anyone in the patch management knows that once in a while....a patch is going to kill a system. This is a given and should be bad clear to management. Patches are not perfect and testing will most likely never cover any possible system configuration. Testing is critical and you should always test as much as possible..but it is a balancing game of risk in the end.

But on to my main point....

While I agree with the general view of this "unnamed CISSP"...there is another view which was not discussed in this article. It is how exploitation and Incident Response (IR) fit into the mix.

If you apply a patch and the system crashes...then you know why the system crashed and the system can be considered clean (not infected or pwned). If you don't patch and the system is comprised, the system could be infected with malware which takes the situation into a different realm. A comprised system could be infected with unknown rootkits or other backdoors which can be extremely difficult to detect and remove.

In addition, if a worm enters the network, it can start to infect other unpatched computers...causing network issues and perhaps bringing the company business to a halt. This situation can result in employee downtime and reduced productivity (which translates to lost wage pay and other headcount-type cost).