Waste Water

I had an unusual opportunity today. My days are generally filled with all sorts of unusual things. Very weird seatmates on airplanes, strange mechanical mishaps and, sometimes, most unusually, a pretty girl smiling at me. (Alright that nearly never happens but I continue to hope to run into a pretty girl with unusually poor eyesight).

Well, last night I had the opportunity to wander around a huge Waste Water Treatment plant. I really can’t tell you about the smells at the input end of the process. I’ll would leave that to your imagination but believe me, you can’t imagine it anyway.

Over the last 15 years or so the plant has automated. Prior to that time they had a single PLC 5 running a very small part of the process. When there was a “weather event”, which I found means “it’s raining”, a bunch of people would run all over the plant opening, closing and adjusting valves.

It’s not unusual for their input to go from 20 million gallons to 200 million gallons during one of those “weather events”. That necessitates lots of adjusting and manipulating. So about 15 years ago they started to automate the process. They choose AB PLCs, ControlLogix and CompactLogix , for the job. They now have a lot of the pumps, drives, valves connected to those PLCs on EtherNet/IP and can much more easily handle a weather event. In fact, they staff the plant with only two people on off hours but could just as easily get by with one.

One of the key points made to me was that both the Ethernet control network and the Ethernet office network are both disconnected from the Internet. They feel that they have such an important responsibility to the citizens of the area that they just can’t take a chance at a virus infecting either their business systems or their process control systems. They have gone as far as to give everyone two computers; one on the plant networks and one on the internet.

They follow that plan with their lift stations. Most of the smelly “inputs” come to them via gravity. But in a number of areas the geography is such that they need to lift these inputs from a lower area to a higher area to continue the journey to the plant. Each of these lift stations is totally independent. Prior to a few years ago they had a dedicated phone line to each station. Now they’ve replaced those telephone lines with satellite links and constantly upload information from the stations. Little to no control information is sent to the lift stations as they operate automatically. They also perceive this as a security measure.

My impressions.

My first thought is that they are very naïve about the threats to their waste water plant. Most successful attacks on industrial facilities are not carried out over the internet. In fact, that is the most difficult way to attack a facility. There are scads of software and systems designed to detect and prevent those attacks. The air gap is where there problem really is. It’s an employee picking up a “free” USB stick, CD from somewhere and using it at work. Or a contractor installing something malicious or a cleaning person planting a virus.

They’ve completely forgotten about physical security at this place and I think that is where they are vulnerable. For example, in San Jose, California, on April 16, 2013, someone cut the fiber electric cables at a PG&E substation and then riddled the transformers with gunfire hoping, I think, for an explosion. They attack was apparently ill-designed but with the intent to cause a widespread outage.

At this waste water facility I drove into the facility at night not passing through any kind of security, parked my car and could have had free reign to wander anywhere I choose, doing who knows what. And if they really only have two employees during off hours in a massive physical plant, I bet I could have walked around undetected all night long. This is their vulnerability and since half of their task is to supply clean water to hundreds of thousands of customers, the threat is very real.

My impressions about their technology.

What they have seems reasonable. I am just not a fan anymore of physically static HMIs. The outside of every control cabinet has an AB PanelView. I view that as outdated, if not actually quaint. Especially when you consider the massive size of their plant, I’m talking miles. How many times is somebody going to be in the evaporator control room to look at that HMI? I think that over the next few years we will come to realize how useless that has become.

All physical plants are growing larger. The number of employees to run them is growing smaller. Those people have to be on the move solving problems and physically checking on the process. As my host indicated, there is a lot of value in a human being walking around visually checking on things, listening for odd sounds or smelling something abnormal.

If you are going to do that you need to have HTML5 web servers and equip your people with the portable tools (phones, pads…etc) that they can use to monitor the plant on the go. The static PanelView panels just don’t cut it anymore in the kind of process environment we have today.

They’ve done well over the past few years just moving into control systems. But I don’t know that they realize that there is another hurdle that they have to climb. In fact, it’s a hurdle that my businesses have to climb. The tools and technologies to serve up data to mobile devices would vastly improve many systems.

John