Hackers accessed city infrastructure via SCADA

(This article was originally written by Hal Hodsen on November 29, 2011 via Information Age and has been copied here for reference purposes only.)

The deputy assistant director of the FBI's Cyber Division says hackers recently accessed the infrastructure of three cities through SCADA systems

Hackers recently accessed the critical infrastructure of three unnamed cities by compromising their SCADA (supervisory control and data acquisition) systems, the deputy assistant director of the FBI's Cyber Division said today.

Gleg releases Ver 1.8 of the SCADA+ Exploit Pack for Immunity Canvas

On November 24, Gleg released version 1.8 of the SCADA+ Exploit Pack for the Immunity Canvas framework, along with a corresponding version 2.7 of the Agora Exploit Pack.

In SCADA+ 1.8 there are modules for several fresh public SCADA/ICS vulnerabilities, most of which were recently disclosed by Luigi Auriemma.  Many of these exploits appear to be denial-of-service (DoS) exploits, so this really is not something that I think is worth the money at this time.

UPDATED: Hackers Independently Attack Two Different Water Utility Districts

Updated: November 23, 2011

News reports broke on November 18, 2011 (Attack on City Water Station Destroys Pump - Wired) when fellow security specialist Joe Weiss blogged about a report released on November 8, 2011 that a water utility district in Springfield, IL (later identified as Curran-Gardner Public Water District) suffered what looked like a "blended attack". The first phase focused on compromising a supplier's internal system which contained remote access credentials not only the target, but several other yet "unnamed" sites. The second phase allowed the attackers to simply "turn the key and walk in the front door" gaining complete access to the industrial control system. The end result was a failure of one of the process pumps.

Are Web Services a Dumb Idea???

I recently read a blog post by Reid Wightman on the @DigitalBond site entitled "When Web Services are a Dumb Idea". It seems that the folks at Digital Bond are on some kind of mission to create a list of "insecure ICS products" which might not necessary be a bad idea, but at least we need to be sure that everyone is being evaluated against the same criteria.

First off, I have to apologize to Dale in my comment to this post, as I did not see that it was written by Reid, and incorrectly referenced Dale in my response.  I have copied my "edited" response from the @DigitalBond site below:

SCADAhacker publishes Duqu Reference Page

Based on the success of the Stuxnet Resource Page on SCADAhacker.com, today I launched a similar page consolidating the useful information and material relating to the new "Son of Stuxnet" malware known as "Duqu".

There are currently multiple researchers analyzing this relatively unknown piece of malware, and all of them appear to be coming up with different conclusions. I felt that it would be useful to share my bookmarks and some of the interesting references that I come across in performing my own open-source research and analysis.

Please bookmark your browser and visit this page often.

I am currently consolidating information.  If you have anything you would like to share, please pass it along.

SCADAhacker to Offer ICS / SCADA "Blue Team" Security Training and Awareness Course in 2012

Having been involved in the industry for several years, I have realize that there is a lack of specific training to address "how to secure" industrial control systems. There are several very good courses currently available, including those offered by InfoSec Institute (which I will teach until early 2012), Red Tiger Security, Digital Bond, SANS and Idaho National Labs. However, when reviewing the syllabi of these courses, I feel that they tend to focus too much on either (1) theoretical aspects of the problem, or (2) the "hacking" or "red team" side of ICS security.

Knowing this, and not trying to duplicate what is currently available, I have decided to launch my own course entitled "Understanding and Security Industrial Control Systems". This course will be primarily focused on "securing" or "blue teaming" the ICS and will involve several labs that reinforce the selection and implementation of security controls relating specifically to ICS.

Duqu: ICS experts weigh in on protecting against zero-day threats - Oct. 25, 2011 Webcast

On October 18, 2011, ICS-CERT issued an advisory related to the discovery of new malware – W32.Duqu – targeting industrial control systems. One year after revelations of Stuxnet came to light, the emergence of Duqu points to the continued need for vigilance in protecting critical infrastructure. What does Duqu – and future zero-day threats – mean to your organization? Join an interactive panel discussion with experts from Industrial Defender, Red Tiger Security and The SCADAhacker on Tuesday, October 25, 2011 at 11 am ET. In this session, you'll gain insight into how you can ready your organization to sustain security in the face of today's threat environment.

Does Anyone Want the Source Code to Stuxnet? Come and Get It!!! (update 1)

After reading report-after-report, blog-after-blog during the past 24 hours, I have decided that rather than comment to each of these individually to offer some additional information which should help set the record straight on who the author is ... or maybe better ... who it is NOT ... in this new variant to our old friend Stuxnet.

Gleg releases Ver 1.7 of the SCADA+ Exploit Pack for Immunity Canvas

On October 20, Gleg released version 1.7 of the SCADA+ Exploit Pack for the Immunity Canvas framework, though this time around, I do not see a lot of unique value in the code updates.

Microsoft and other AV Vendors offer signatures for W32.Duqu

As recently communicated via the SCADASec forum, Microsoft and others have made available anti-virus signature updates for the W32.Duqu trogan, covering at least three variants.  The links below are to the Microsoft Malware Protection Center, and provide some useful background information:

• Variant "A"
• Variant "B"
• Variant "C"

Interesting enough are the details contained in the Variant "C" summary which identifies the IP addressed used for the C&C server - 206.183.111.97, which is registered to WebWerks India Pvt. in Mumbai. This should not lead you to believe that the attackers originate within India, but rather that this site could be used as a proxy.


Bob Radvanovsky also provided a link which highlights the updates of a large number of AV vendors relating to Duqu. This list is available by clicking here.

Son of Stuxnet has Surfaced in Europe According to Symantec Report (update 1)

According to a blog posted by Symantec on October 18, and as reported by Homeland Security News Wire on October 19, a research lab with "strong international connections" alerted Symantec to sample code that appears to be very similar to Stuxnet. This new threat has been named "Duqu" (pronounced dyü-kyü) because it creats files with the prefix "~DQ".  (A copy of the complete Symantec report is available by clicking here).  Samples given to Symantec were obtained from systems located in Europe.

SCADAhacker to Speak at Information Security Trends Meeting in Columbia

I will be speaking on current issues facing industrial control system (ICS) cyber security issues at the Digiware Information Security Trends Meeting scheduled for October 12, 2011 at the Marriott Bogota, Columbia.

My talk will focus on the issues facing ICS/SCADA systems used to control a vast majority of a country's infrastructure, including electric generation (fossil, hydro, nuclear), water/wastewater treatment, energy distribution (pipelines), transportation (rail, traffic), process industries (pharma, oil, gas, refining), and discrete manufacturing.  One point of special attention will be on recent attacks and how to address the new "insider threats" where a malicious outside gains inside access via various tools and then "poses" as a valid user with appropriate credentials!  Identifying and stopping these attacks presents unique challenges that many are not completely aware.

I hope to provide live updates of the conference via my Twitter feed at @SCADAhacker.

Gleg releases Ver 1.6 of the SCADA+ Exploit Pack for Immunity Canvas

On September 26, Gleg released version 1.6 of their SCADA+ exploit pack for Immunity Canvas. This release includes several new modules including many found by Luigi Auriemma. Note that Metasploit has also incorporate a large number of these exploit modules in their free framework.

Oil and Gas Cyber Security Forum 2011 - London - Nov. 21-22

SCADAhacker is proud to be a key member of the speaker roster at the launch of SMI's inaugural Oil and Gas Cyber Security Forum 2011. This conference takes place in London on November 21-22, bringing together cyber security professions from across the world to discuss, network and analyze key cyber security issues facing the oil and gas industry today.

Security researcher Luigi Auriemma again discloses publicly numerous vulnerabilities targeting multiple SCADA/ICS systems

On September 13, 2011, Italian Security Research Luigi Auriemma (web site) disclosed a laundry list of vulnerabilities that target six (6) different Industrial Control Systems, including United States market leader Rockwell Automation.

Gleg releases Ver 1.5 of the SCADA+ Exploit Pack for Immunity Canvas

Today (August 25, 2011), Gleg announced the availability of Version 1.5 of the SCADA+ add-on exploit pack for Immunity's CANVAS exploitation framework (much like the Metasploit Framework). As we have seen over the past few months, this release contains several new automated SCADA exploits, including several zero days.

Gleg releases Version 1.4 of the SCADA+ Pack for Canvas

On July 21, Gleg Ltd. annouced the availability of Release 1.4 of the SCADA+ pack for Immunity's Canvas.  This confirms a trend by which Gleg appears to be offering an updated SCADA+ pack about every month. Details of v1.2 - 1.3 are also provided below.

ICS-CERT also released an alert ICS-ALERT-11-230-01 on August 18 which provides some additional details on the SCADA+ Pack.  Though there were no alerts or updates for SCADA+ Versions 1.2 and 1.3, the ICS-CERT update and this blog should provide good revision control.

Offensive Security Releases Backtrack 5 R1

On August 18, Offensive Security released BackTrack 5 R1. This release contains over 120 bug fixes, 30 new tools and 70 tool updates. They plan to roll out new how-to's on their website's wiki in the coming weeks.  Topics to be covered include VMware tool installation, alternate compat-wireless setups, etc.

The kernel was updated to 2.6.39.4 and includes the relevant injection patches.

As with Backtrack 5, choices exist for either the GNOME or KDE GUI, and include both 32- and 64-bit versions.  A VMware image is available in 32-bit GNOME only.

Download available directly from SCADAhacker.com using the Tools section, or through the normal Offensive Security website.

Comments on Langner post: "ICS-CERT on Beresford Vulns: Flawed Analysis, Misleading Advice"

On August 20, 2011, Ralph Langner posted a very insightful blog on the recent security work of NSS Labs' Dillon Beresford (Twitter @D1N) and the report that ICS-CERT released regarding this research. This was a very well written article, which I have to say I agree with most of the document. In particular, I am a bit disappointed in how ICS-CERT is handling these reports in general especially in the way of offering sound, practical, ICS-based guidance on dealing with these threats.

There are a couple of points that Ralph mentions that I feel deserve mention that would require more than 140 characters in a tweet to discuss!

Gleg releases Ver 1.1 of the SCADA+ Pack for Canvas

Gleg Ltd. annouced the availability of Release 1.1 of the SCADA+ pack for Immunity's Canvas.

Here are the details of the release contents:

White Phosphorus Exploit Pack Ver 1.11 Released for Immunity Canvas

Version 1.11 of the White Phosphorus exploit pack is now ready, and contains
5 new exploit modules, including one for SCADA.

CIP-002-4 “Bright Line” Secures 163 Plants, Max

(Copied from blog "Findings from the Field" posted April 3, 2011 by Andrew Ginter)

In the 2009 statistics, the latest available, NERC tracked some 10,500 generators with a nameplate capacity of 0.1 MW or higher, at about 5700 sites. The new NERC CIP-002 version 4 “bright line” rule says NERC-CIP applies to only those generating sites with “an aggregate highest rated net Real Power capability of the preceding 12 months equal to or exceeding 1500 MW in a single Interconnection.” How many locations/plants is that? According to NERC, only 163 sites have a nameplate generating capability of 1500 MW or greater, and there is no word yet on how many of those plants are exempt because they feed less than 1500 MW into any one interconnection.

Russian Security Team to Upgrade Agora SCADA+ Exploit Tool for Canvas

(Originally posted in PC World, March 25 by Jeremy Kirk, IDG News and edited by SCADAhacker) 

A Russian security company plans to release an upgraded exploit pack for industrial control software that incorporates a raft of new vulnerabilities released by an Italian security researcher [on March 21].

More SCADA Security Threats: Where There’s Smoke, There’s Fire

One of the unfortunate facts about security is that if you can find one vulnerability, you can usually find lots more. Vulnerabilities are not just bad luck – they are caused by a poor Software Security Assurance (SSA) process (or a complete lack of one). Next in line for blame are experienced professionals who do little in terms of security assessments prior to commissioning systems in actual production facilities.

So what about the four SCADA / HMI products that have Luigi Auriemma’s 34 Zero-day vulnerabilities? Would any of those have additional vulnerabilities, just waiting to be exposed to the world? After all, Luigi claims to have spent only two days per product. That isn’t much time – what if someone else started looking harder. So we decided to give it a shot.

Protecting your ICONICS GENESIS SCADA HMI System from Security Vulnerabilities (plus White Paper)

As mentioned in a blog article we wrote earlier this week, an Italian “Security Researcher” named Luigi Auriemma published thirty-four SCADA product vulnerabilities against four SCADA products (the complete list of vulnerabilities and companies is provided in the earlier article).

Eric Byres and I have tested the vulnerabilities and today we are releasing a White Paper that analyses the ones regarding ICONICS GENESIS32 and GENESIS64 products.  The paper summarizes both the current known facts about the vulnerabilities and the actions that operators of SCADA and ICS systems can take to protect critical systems.

The Italian Job – Multiple SCADA/ICS Vulnerabilities Go Public

(Originally posted by Eric Byres on March 21, 2011 @ Practical SCADA Security)

Selling the concept of security for SCADA and ICS might still be struggling, but publishing vulnerabilities for SCADA and ICS equipment seems to be a growth industry.

Summing up Stuxnet in 4 Easy Sections (plus Handy Presentation)

(Originally posted by Eric Byres on March 21, 2011 @ Practical SCADA Security)

There has been a lot of media coverage and discussion of the Stuxnet malware, and its impact on industrial control system (ICS) and SCADA security. We are one of the groups guilty of creating a Stuxnet publishing industry.

A few weeks ago we issued a White Paper “How Stuxnet Spreads – A Study of Infection Paths in Best Practice Systems” written by Andrew Ginter, Joel Langill and I. The paper is a 26 page deep dive into how Stuxnet could migrate from the Internet to an isolated ICS, including a discussion of what can be learned from Stuxnet’s pathways.

Now, frankly, I am sick of Stuxnet. If you are one of our regular readers, you might be as well. However, being sick of a threat doesn’t make it go away. So far, very little has changed in our industry; we still need to address the issues that Stuxnet exposed.

Furthermore, I am constantly surprised how little upper management in the industry knows about the worm. I am not expecting that the CEO of “Real Big Corporation” knows the technical details, but it would be good if he or she at least understood the basics. Otherwise, it is hard to get ICS security the attention it deserves.

So today, we are publishing a presentation that abridges the findings of the "How Stuxnet Spreads" White Paper, and is a summarization of a lot of information on Stuxnet. If you need a crash course on Stuxnet, or a presentation for management, this may come in handy.  Below is a synopsis of the presentation, and a link to the download for it.

Agora+ SCADA Exploit Pack for CANVAS

GLEG ltd. is pleased to announce Agora SCADA+ exploit pack that is entirely focused on industrial software and hardware vulnerabilities.

Are the NERC CIPs a roadmap for attacking the electric grid?

By Joe Weiss

The NERC CIPs have a number of characteristics that make them a roadmap for attacking the electric grid. - They were developed by the NERC consensus process. The process is long, arduous, and inherently a “low bar”. As such, the process results in trying to make it easier on the “attackee” than trying to make it more difficult on the attacker.

How Stuxnet Spreads - A Study of Infection Paths in Best Practice Systems

A new White Paper, "How Stuxnet Spreads – A Study of Infection Paths in Best Practice Systems" has just been released by Eric Byres (Byres Security), Andrew Ginter (Abterra Techologies), and myself. The paper details how Stuxnet could infect a control system site protected by a high security architecture. It shows that current best practices are insufficient to block advanced threats and it discusses what operators of control and SCADA systems need to do to protect critical systems from future threats like Stuxnet.

McAfee: Data-theft attack hits oil industry

McAfee: data-theft attack hits oil industry

by Stephen Shankland (CNET News)

For years, companies in the oil and energy industry have been the victims of attempts to steal e-mail and other sensitive information from hackers believed to be in China, McAfee said yesterday.
The attacks, to which McAfee gave the sinister name "Night Dragon," penetrated company networks through Web servers, compromised desktop computers, bypassed safeguards by misusing administrative credentials, and used remote administration tools to obtain the information, the security firm said. McAfee and other security companies now have identified the method and can provide a defense,

DHS Best Practice for Remote Access Falls Short

DHS's Control Systems Security Program (CSSP) group recently released their best practice on securing remote access to trusted networks like control systems and their semi-trusted demilitarized zones (DMZ).  The best practice can be viewed by clicking here.  Unfortunately, if we maintain the "think like a hacker" mentality, this document still falls very short of expectations, and is not far off of the NIST guidance document produced in September 2010.

In order to provide a comprehensive defense-in-depth strategy for remote access that addresses both internal (company) and external (non-company or contract) personnel, the focus has to be expanded from just providing basic authentication and confidentiality.  The solution needs to address the health of the endpoint, as well as provide a mechanism to restrict access to the trusted system once access is granted.

February Issue of Hakin9: Network Security

Click here to view the February issue.

Hakin9 has released their February Free Issue of Hakin9 Magazine. This month the magazine has articles focused in Network Security.

Stuxnet Demonstration Videos now available on YouTube

Due to the success of the demonstration videos that I produced on the Stuxnet worm, I have posted them on YouTube for the general public to view including relevant keywords. The links for the videos that have been produced to date are as follows:

Part 1: Stuxnet Introduction, Installation and Infection
http://www.youtube.com/watch?v=sEfqtET13SY

Part 2: Stuxnet Mitigation: Using Software Restriction Policies
http://www.youtube.com/watch?v=YjlShzQEWOo

Immunity Releases CANVAS 6.66

This release should be promising, as it provides updates to my personal favorite exploits that revolve around the SMB services on Windows hosts using port 445.

Bandolier Baselines: Windows 7 and 2008 Server

The use of a vulnerability scanner in assessing the overall security posture of an integrated ICS is critical no more than ever. A common misconception is that these scanners will not detect ICS-related vulnerabilities like those used on the recent Stuxnet attack, and that these scans can often cause ICS equipment to fail. These misconceptions are in fact, quite false. Digital Bond has provided some very good guidance on the "proper" use of the Nessus vulnerability scanner within ICS environment. As for the comment on vulnerabilities ... well this is just because those individuals do not realize the power and flexiblity of perform ICS scans and audits using the authenticated scan features of Nessus, coupled with the use of specialized Nessus audit files.

New York Times: Stuxnet Worm Used Against Iran Was Tested in Israel

Normally, I would not copy-and-paste an entire article in this blog, but this article provides views and references on Stuxnet that is worthy or retaining. Enjoy!

(Click here to download the referenced presentation given by Siemens and Idaho National Lab (INL) during the Siemens Automation Summit Users Conference at Chicago's Navy Pier in 2008.)

A Different Approach to ICS Security Controls and Stuxnet Mitigation Strategies

There are a lot of experts, some with and some without any relevant control systems experience, who are today offering advice regarding how to handle Stuxnet and Stuxnet-like attacks. One thing is pretty much agreed to by all: while no single solution will block an attack like Stuxnet, a comprehensive solution of countermeasures including process and policy can significantly reduce the negative consequences that result from such an attack.

Knowing this in advance means that any mitigation strategy needs to be based on a solid defense-in-depth strategy that utilizes multiple, independent layers of protection. The members of the CSFI Stuxnet Project agree that while it will always be possible to find flaws in any one solution it should be increasingly difficult to find and exploit flaws in a comprehensive solution that depends on multiple protective measures.

UPDATED: Nearly Instant Exploit when MS Releases Patch

I teach my SCADA security students that we always are faced with a double-edged sword when dealing with the patching of security holes within a common platform. In the days of the SQL Slammer worm (2003), an exploit could take up to six (6) months to develop using reverse engineering techniques once Microsoft released their security hotfixes.

Today, that time is now on the order of hours! Take, for example, MS11-002 (Microsoft Data Access Components Vulnerability - [CVE: 2011-0027]) which was just released this morning (January 12, 2011) by Microsoft. It was just a matter of hours before an exploit was available for download for script kiddies and experienced pen testers to begin using (http://www.exploit-db.com/exploits/15984/).

When we consider control systems, and the fact that at best, security patches are approved by the vendor within 7-14 days, we have a pretty wide window of opportunity to exploit these critical systems. Using my "think like a hacker" approach to security, the best time to exploit a targeted control system is during the first few days following the publication of the MS Security Advisories (which are released on the second Tuesday of each month). In reality, we have even longer, as many control systems do not utilize any form of automated patch management system that deploys these updates as soon as they are approved by the vendor.

If I was planning an attack, I would complete my reconnaissance phase, and wait until the days immediately following the MS announcement to commence the actual attack using the latest vulnerabilities that will be sure to evade both the OS and the security protections that are in place.

I also thought that it would be useful to share the updated schedules from a few of the other major vendors. Of course, vendors are free to release out-of-cycle updates for vulnerabilities which they feel are too critical to wait for the normal cycle.

• Microsoft
  Monthly
  2nd Tuesday
  
  
• Oracle
  Quarterly (Jan, Apr, Jul, Oct)
  Tuesday closest to 17th of the Month
  
  
• Cisco (Internetwork Operating System)
  Bi-Annual (Mar, Sep)
  4th Wednesday
  
  
• Adobe
  Quarterly (Feb, May, Aug, Nov)
  2nd Tuesday
  

I compiled some data for Stuxnet that is interesting and worth sharing, comparing the date of discovery (let's just say July 16 for all practical purposes), and the date of patch and exploit being released:

• MS10-046 (Propagation)
  SecurityFocus releases exploit July 15
  Metasploit releases exploit July 19
  Microsoft releases patch August 2 (out-of-band)
  Immunity releases exploit September 27
  
  
• MS10-061 (Propagation)
  Microsoft releases patch September 14
  SecurityFocus releases exploit September 14
  Metasploit releases exploit September 17
  
  
• MS10-073 (EoP)
  SecurityFocus releases proof-of-concept July 1
  Immunity releases exploit October 5
  Microsoft releases patch October 12
  SecurityFocus releases exploit October 12
  
  
• MS10-092 (EoP)
  Immunity releases exploit October 5
  SecurityFocus releases exploit October 18
  Microsoft releases patch December 14
  

I found an interesting statistic from a 2009 article in ComputerWorld:

"The increase in the number of flaws being discovered comes at a time when attackers are getting much faster at exploiting them. A survey by security vendor Qualys earlier this year [2009] showed that 80% of vulnerability exploits are available within 10 days of the vulnerability's disclosure. Nearly 50% of the vulnerabilities patched by Microsoft in its security updates for April [2009] already had known exploits by the time the patches were available."

As you can see, we all need to be diligent in addressing patch management within our control system networks. Next month (February 2011), I will be asking the wider community to participate in a survey to collect some real-world data regarding patch management implementations.

If you are interested in exploring any of the Stuxnet exploits that have been published, a list is available at http://www.stuxnetcure.com.

China Sleeps On A Stuxnet-Like SCADA Bug

A vulnerability has been identified in Wellintech KingView, which could be exploited by remote attackers to take complete control of a vulnerable system. This issue is caused by a buffer overflow error in the "HistorySvr.exe" module when processing packets sent to port 777/TCP, which could be exploited by remote unauthenticated attackers to crash an affected application or execute arbitrary code.

I was tipped off by a article in ThreatPost today. You can also read the full vulnerability advisory from Vupen Security.

Security Onion - Ubuntu-based Live CD to facility network monitoring, IDS, etc.

After reading a tweet from Richard Bejtlich over at Tao Security regarding the Security Onion Live CD, I felt that this package was a "must have" for any SCADA hacker looking to build or expand their current tool kit needed for comprehensive system assessments and pen testing.

Doug Burks has just released a new version of Security Onion, which if you have not played with this in the past, is an Ubuntu-based live CD that is used to facilitate network security monitoring. Since network vulnerabilities lead the list of common vulnerabilities seen on most control systems, this tool is worth your time and effort.

Since I am a huge proponent of the addition of Intrusion Detection Systems (IDS) to control system networks, I am a big fan of Security Onion. The current distribution includes the standby Snort release 2.9.0.3, but it also contains the Open Information Security Foundation (OISF) Suricata IDS project funded in part by the U.S. Dept. of Homeland Security chartered with building the next generation IDS/IPS engine. You can visit the OISF site for more information on Suricata by clicking here.

Some of the other packages you will find in Security Onion include:

• Vortex IDS
  
• Bro IDS
  
• ABCIP
  
• Dumbpig
  
• NSMnow (includes Sguil, Barnyard2, Sancp, etc)
  
• OSSEC
  
• Squert
  
• Xplico
  
• hogger
  
• SnortValidator
  
• Metasploit

To install, I just created downloaded the ISO and then created a new virtual machine that boots from the ISO image. Total setup time ... less than 5 minutes!

This package is one that every hacker should have in their tool kit in order to completely evaluate the networks used with industrial control systems.

Industrial Control System Cybersecurity Advanced Training - Feb. 14-18

The United States Department of Homeland Security Control Systems Security Program (CSSP) is pleased to announce the next Industrial Control Systems (ICS) Cybersecurity Advanced Training, scheduled for February 14-18, 2011, in Idaho Falls, Idaho.

This week long training course will provide intensive hands-on training on protecting and securing control systems from cyber attacks. The event will also include a Red Team - Blue Team exercise that will be conducted within an actual control systems environment. It will provide participants an opportunity to network and collaborate with other colleagues involved in operating and protecting control systems networks.

There is no tuition fee; however, travel, lodging, and meals are at the expense of the participant. Please see the attached flyer for additional registration information. You can also find this information on the CSSP website: http://www.us-cert.gov/control_systems/cscalendar.html

SCADA Security Professionals are Hard to Find

There have been a couple articles recently published that I have mentioned on my Twitter feed that are worth mention here. Both the public and private sectors are seeing that we have a significant deficiency when it comes to training security professionals that can help protect our nation as well as our critical infrastructure. This problem is further compounded when we look at the highly specialized aspects of security services that are targeted at industrial control systems like SCADA and DCS systems that are used throughout our infrastructure.

I know that many of you have attended a training course like the one that I currently teach (SCADA Security by InfoSec Institute), and I only encourage you to continue to learn more about the highly specialized field of ICS security. There are so many opportunities that await you, and if you are a customer seeking training professionals, please feel free to contact me and allow me to help you match your particular needs with those of us that can provide these services.

I want to direct you to a couple articles that I recently came across that are worth reading:

Pentagon, Industry to Swap Cybersecurity Experts
Shortage of Skilled Information Security Professionals Looms

I also want to point you to another article that I previously commented on via the SCADA Security Professionals group of LinkedIn:

Security Firms Scramble for SCADA Talent after Stuxnet

This also means that the "bad guys" are also looking for talent to help them launch attacks against commercial and industrial targets. This confirms that we need to remain diligent with information security and in particular, InfoSec focused on protecting our infrastructure depending on industrial control systems.

Job outlook improving for cybercrooks

UPDATED: Virtualization can actually Improve the Security Posture of existing Control Systems

For those of you that are members of the LinkedIn group "Process Control System Forum Members", you may have seen some recent dialogue relating to moving some control system components to the cloud (or let's just say "virtualizing" these components). If you have not had the opportunity to read this discussion, I have copied it below. The reason I have posted this conversation is that it provides some strong issues around the cloud, virtualization, security, and "thinking like a hacker" to make sure the implementation is based on a solid and secure design. As always, I appreciate and respect all comments, criticisms and suggestions ... after all ... we all have the same goal in mind.

INITIAL POST BY RAPHAEL PEREIRA:
"I am writing a paper about the benefits and problems OS use of cloud technology in Control Systems. Does anyone have experience with this use? I am looking for use of virtualization technology, syncronous data tranfers, DR sites and other applications that we could use to help to be control systems more realible."

Jake Brodsky • I have several things to point out:

First: The primary purpose of virtualization in this arena is not to save on the number of servers, but to make existing systems more resilient and to reduce recovery time from software crashes.

Second, if you're using these servers to consolidate all processing to one place, you're doing it wrong. The point of this exercise is resiliency. Make sure you have at least one backup server room on the other side of the plant with backup servers capable of running the whole show from a separate power source.

Third, network design is key. Make certain that the switches (and routers, if any) are capable of handling the traffic of moving images from server to server without affecting plant operations. In other words, don't buy a cheap layer 3 switch and then get surprised when the backplane saturates.

Fourth, make sure the network infrastructure is also distributed. In other words, even if you diversify the servers, it won't do you much good if everything is focused through one great big switch of everything.

Fifth, virtualizing is great --as long as the applications support it. Make certain that the HMI vendor, driver vendors, historian vendors, and any other application vendors all support this.

Finally, in case you haven't already figured this out: Just because you've virtualized the control systems apps doesn't mean you can put office applications on the same server. Security issues aside, office application performance management is very different from control systems application performance management As with networks, while it is theoretically possible to mix them, in practice it is more trouble and more risky than any potential savings one might see.

Andrew West • I know of a utility that is using virtualization in much the manner that Jake describes: The benefit of virtualizaiton is to manage version control (it is easy to roll back to the previous configuration) and facilitiate failover to a backup system if a fault takes one of the servers down: The process image is checkpointed to a backup machine and is made active with almost no downtime. This system also keeps the disaster recovery image at a second control centre current in a similar way.

Ron Southworth • I know people are using virtualised environments for testing. Some vendors are starting to certify and support virtualised environments. Two vendors in the power systes space spring to mind. I know of a few (less than the fingers on one hand) owners and operators that are perhaps bravely (hapily) using a virtualised environment. Early days for control systems lots of advantages and challanges to sort out as Jake has mentioned.

Perry Pederson • Perhaps I am going to the same place as the dinosaurs, but I would never EVER put my safety systems in the cloud. Take whatever threats and vulnerabilities there may be and multiply them by some huge unknown number and then try to sleep at night. Fugetaboutit!

Joel (the SCADAhacker) Langill • I have lots of experience with virtualization, and am completely behind this movement in order to help isolate the dependence on hardware from the functionality of the control system software. Coming from a vendor, too much time was spent on compatibility issues with hardware, where it should have been spent on validation of software functionality and its inherent security!

I have been using virtualization for several years in certain aspects of the project lifecycle and system architecture. Let's expand on this, starting with the high value, low risk areas first.

One of the best most obvious locations for virtualization is within the test and development environment of a facility. With the risk presented by installing untested or "lightly" tested patches and updates, virtualization eliminates most of the hurdles that were common in the past relating to building and maintaining a separate test bed for such a purpose. Patches need to be stressed more against the software installed, than the hardware platform upon which they are installed.

Next, virtualization is perfectly aligned with high-fidelity training simulation facilities that are becoming more common as manufacturing facilities are required to demonstrate that operating personal are regularly tested on their ability to operate and control the facility under a variety of planned and unplanned events.

We also are beginning to see more virtualization show up on the application level within the control system architecture, including historization and advanced application platforms. This are commonly applications that are not critical to maintaining production levels, and the vendors that are entering the cloud are beginning at this level.

I have also begin to work with virtualization on several of the hosts that would typically reside within 1 or more DMZs, such as web servers and jump hosts. Virtualization effectively allows us to create an architecture with multiple functional DMZs all directly connected to dedicated virtual platforms. This all but removes many of the common concerns sites had with "too many" DMZs and how this is managed with a traditional firewall. What some of the more progressive designs have also considered is the use of virtual firewall appliances as well that are used "downstream" of a traditional dedicated appliance, further allowing these sites to be built with very restricted and dedicated functional DMZs.

The one contraint that is going to continue to hinder the deployment of virtual technologies within the lower levels of the control system architecture will be the lack of flexibility in terms of peripheral support. It would be next to impossible, for example, to create a Profibus interface adapter that would be certified by both the vendor and the virtual software platform. As long as we have "non-standard" or proprietary technologies, it will be difficult to completely migrate the traditional level 1 and level 2 applications to the cloud.

Hope this provides some insight ... it certainly has given me some thoughts for a blog entry of my own!!! (blog.SCADAhacker.com).

Ron Southworth • G'day Perry & Joel.

I don't think you are going the way of the dinosaurs.

(If you are then call me a Muttaburrasaurus )

Perry you are reflecting what I would say is my present risk appetite, especially when I look at many industrial process control systems designs in the face of modern targeted malware.

Joel I am yet to see an implimented cloud that isn't more flawed than a legacy control system to be completely frank. Most of the cloud offerings I have seen are really more to do with outsourcing. I hope that folks will learn that this actually costs an enterprise up to 2.5 times the cost of operating and maintaining your own non core buisness enterprise technologies.

There is a big difference between running systtems in a virtualised environment and what all this cloud computing is all about. Please don't misunderstand me it isnt a question of if it is more a question of when I guess, and I think that folks are racing too fast towards embracing and merging technologies. Profi- Bus is about providing deteriministic operation something that is an issue not being given enough attention. We have to be careful about what we standardise on or how we impliment technologies.

Hopefully many other folks will see or have a better understanding or aprecaiation of this aversion to operational risk that I am speaking about. Many folk when they talk about mitigation techniques to permit connectivity from the board room to the plant floor I don't think really have a firm operational understanding of Protection and Safety Systems fragility.

Ralph Langner prior to all this malware hyperbole was developing a great paper on the subject - well on it's way to being a book actually and he was using fragility as a means to close and explain this gap of understanding.

I still stand by what I said at a conference a few years ago now. Sometimes the only effective mitigation we have to cyber threats with all it's limitations is physical segregation. It is by no means perfect but it removes a whole heap of "cyber" problems off the table providing you have your house in oder on your human elements.

As you say Joel there are some levels of an enterprise that might be capable of being supported in the cloud however I will need a lot of convincing as to when this should happen.

Joel (the SCADAhacker) Langill • Ron ... excellent points ... but let me be clear ... my implementations and examples provided in my comment above are not for the "enterprise", but for the control system domain. These have been implemented, and when implemented by individuals who understand how to effectively implement and secure virtual environments, they are quite reliable in practice. This means that those individuals have actual experience in implementing a true virtual environment based on a hypervisor, and not simply taking some casual backoffice experience with a product like VMware Workstation or Server and trying to move this into a production environment.

Maybe we need to talk further about what you have observed in your implementations versus mine. Since I prefer to make security a base requirement, these systems are significantly more secure than any legacy control system I have personally used. However, since my first love is control systems, and I have grown into security over the years, my design approach is very different than most!

I too have seen very poor cloud implementations, however, these should not discredit the solution, but rather discredit the individuals responsible for its implementation. For starters, most fail in their virtual implementations with the poor "built-in" and "default" configurations relating to virtual networks. This then is further exaggerated with less-than-optimal designs around virtual management.

I read the ISA article on HMI in the cloud, and personally, think this really misses the true value proposition of virtualization to both the owner-operator (end-user) and vendor. With nearly 18 years experience as a vendor, the real value lies not in the HMI nodes, but the server nodes. There may be some performance gains with the HMI nodes, and these tend to be easier due to their lack of non-standard hardware, however, the nodes that cause use the greatest headache continue to the those that are based on a server operating system (primary/backup system servers, historians, applications, etc.).

A solid virtualization platform, like vSphere from VMware, allow vendors to implement hardware redundancy on nodes that they have not been successful in the past in providing high-availability solutions. The features provided by products like vMotion offer an opportunity to proper vendors into a new domain of reliability without really investing much from a product development perspective. Take common application nodes like those used for batch management, multivariable control, optimization, and web services ... these are non-existent in a cost effective redundant configuration (barring something like a Marathon product which defeats the benefits of COTS hardware), but are fairly straightforward in a virtual world.

I only used Profibus as an example of what virtualization cannot be used everywhere, however, please focus on the main point of my comment which shows significant benefit of virtualization relating to two components of any control system architecture that impacts the overall security posture of the system: patch management; application development, testing and migration; and DMZ applications.

I paper on this topic is a great idea, and is something I will definitely add to my 2011 goals and objectives! Any suggestions would be greatly appreciated and respected.

Jake Brodsky • Following our design, we are currently testing a virtualized HMI and Historian system for a water filtration plant. As you say Joel, there are many pitfalls. Too many are selling office oriented systems, treating this application as if it were just another web server. I have no patience for such idiots.

By doing this we are treading a very find line between complexity and usability with this technology. Remember, people will have to use this system during times of stress and fatigue. It is difficult for some IT experts who live and breath this stuff to understand that on a plant, in the wee hours of the morning, with the superintendent breathing down your neck and the plant radio system squawking away, most of the HMI or historian systems dead in the water, and potential hazards ready to engulf, explode, or burn someone --that someone (a 24 hour duty engineer like me) has to remember how to bring this stuff online!

Yeah, when things are routine, when you're managing stuff on a planned schedule, it is a wonderful technology. However, Murphy's law says that things will fail in the worst possible way at the worst possible time. We're trying to find simple instructions to diagnose and repair these systems so that a tired duty engineer can talk an operator through this problem in a matter of minutes.

Sometimes, even though a solution may take longer to recover, the simplicity and predictability may make a simpler system more desirable. Ultimately, the goal is to get back up and running in minimal time. The fewer opportunities to make mistakes, the more likely it is that that recovery from an outage will happen sooner.

That said, we see a value in this technology and we are implementing it with a look toward using our experience to push solutions of this sort elsewhere. However, support from our vendors has been tepid; the tools, particularly the licenses, are confusing; and the costs, while reasonable, aren't exactly small change. There are lots hazards to navigate on this still poorly traveled road. Those who casually wave their hands about this while glossing over the details clearly haven't done this before or do not have to service this creature after it has been built.

Virtualization has future for control systems design. I think it will be a bright one. But as with many early adopters, there are still many lessons to be learned.

Joel Langill • Excellent points, Jake. Again, I think this group dialogue will result in
excellent material for a paper, and I hope no one objects to its use. (I
wonder is the original author of the post from 7 months is still
following.)

I agree completely, but am somewhat disappointed that the bad reputation of
virtualization is tied more to the quality of the implementation than the
actual technology of virtualization and what it offers.

Good luck with your project. I could see virtualizing an HMI for a SCADA
type workstation like Wonderware, but this would not be my first choice
with a system like Centum, DeltaV or Experion. I will stick to level 3
nodes and development / patch management / training systems for now. I
would be interested in feedback in the future.

Jake Brodsky • And as if on cue here is a Dilbert Cartoon to illustrate my point:
http://dilbert.com/strips/comic/2011-01-07/