Intermittent Failures: The Reason Control Engineers Have Gray Hair (And How Historians Help)

My life on the factory floor consisted of working at two massive consumer products companies, both in the same consumer products industry. At both, I was on the engineering team that designed and built converting machinery to make diapers, feminine pads and Kleenex. I thought it was a very cool job.

My area of expertise was the positioning systems (we called it product registration). It is a key component of the machine. To accurately apply a line of glue, attach a tab or activate a saw to make a cut, the positioning system needed to have the product positioned precisely, not just once but every time. I really enjoyed fighting that battle with the limited technology of the day.

The Challenge of Intermittent Failures

What really sucked were the intermittent failures. For no apparent reason, the machine would reject a long series of units. Some ten, twenty or even 200 units would be discarded, after which the machine would return to normal. I experienced this quite a few times, with failures happening at rates from every five minutes to every five days. There was little diagnostic technology available at that time. We made a lot of guesses.

It’s different today. I just heard from a control engineer with a very complex machine that isn’t new. It’s been around for a while and just like my packaging line, he has an intermittent condition. Instead of rejecting product, his machine simply stops. It might be a code bug, a raw material issue, a failing mechanical component – he just doesn’t know. He could put a camera on the machine to get visual evidence, but what he really needs is to save the entire machine state when the failure occurs.
He needs a Historian.

Key Considerations for Data Historians

There are two key considerations when evaluating Historians for diagnostic applications like this. One, how rapidly it can collect time-series data and, two, how much storage it has to retain that data. In this instance, the time series data will clearly show where data collection ended, making it easy to examine the data leading up to the failure.

Timestamping and Clock Synchronization

Timestamping will be critical to this application. This requires the Historian to know the correct time. Some of the various mechanisms Historians use to maintain current time include Real-Time Clocks (RTC), the Network Time Protocol (NTP) and the Precision Time Protocol (PTP). Time is important here, but any of these mechanisms will suffice in this application.

Data analysis and Export Tools

The other consideration is the tools you will use to analyze data. Excel is a favorite for applications like this. Most diagnostic applications support ingesting CSV (Comma-Separated-Values) files from an FTP session or even on USB sticks. They often also support SQL. Know the diagnostic tool you will use before picking the historian.

Enterprise Historians vs. Cell-Level Historians

Historian applications fall into two main categories: enterprise-level and cell-level. Enterprise-level historians are typically deployed in a Virtual Machine(VM) or Cloud and are designed for ingestion and organization of massive volumes of data. Cell-level historians are typically used for real-time process monitoring or performance optimization. These are a good fit for applications that don’t require the volume or speed of the enterprise-level historians.

An example of a device that can solve intermittent failures is the RTA Historian[EW6.1] from Real Time Automation.

Capturing Time-Series Data with the RTA Historian

The RTA Historian is easy to install and quickly configurable, offering the ideal set of features for most time-series data collection applications like the diagnostic application above.

Data from multiple Allen-Bradley PLCs and Modbus devices can be captured and published on demand, without subscriptions, licensing constraints or reliance on third-party middleware. With configurable storage of up to 1 TB, a comprehensive suite of publishing protocols (SQL, HTTP, FTP, WebSockets, USB, MQTT and email) and direct integration with InfluxDB for visualization and analytics, the RTA Historian is an invaluable tool for plant floor operations, maintenance and process engineers.

Learn more about the RTA Data Historian or email us at solutions@rtautomation.com or call and speak to an Enginerd 262-436-9299.