The cell stopped at 3:12 in the morning. The early shift acknowledges the alarm, restarts, and everything runs again. Ask why it stopped and nobody has a solid answer. Two weeks later, the same picture. In many plants this is the norm rather than the exception, and it is rarely down to the skill of the maintenance team: it is down to the fact that the moment of the incident cannot be reconstructed. This article shows which information you need and how to get it.
Why reconstruction so often fails
An alarm is rarely an explanation. "Error 1042" or "emergency stop triggered" describes the outcome, not the path that led there. What happened in the seconds before, which signal changed first and whether the gripper had even picked up the part is recorded nowhere.
On top of that comes a practical problem: the person who witnessed the event went home hours ago. Reconstruction happens from memory, and memory is no reliable tool for an event that lasted half a second. Anyone who then lines up log files from the robot controller, the PLC and the test rig is fighting three devices, three clocks and three formats.
The four questions every incident must answer
Regardless of vendor and machine type, a solid root cause analysis needs four answers. Miss one of them and what remains is a guess:
- When exactly? Not "during the night shift" but to the millisecond, and on every device involved using the same clock.
- What was the state? Which inputs and outputs were set how, and in which order they changed.
- What was the program doing? Which work step was running, which program line was active, with which parameters.
- What could be seen? An image of the cell at the moment of the stop answers questions no signal trace ever will.
Why log files alone fall short
Classic logging fails in three places. First, the time base: without a shared time source the clocks of your devices drift apart, and even one second of offset turns cause and effect into guesswork. Second, context: "DI 3 = 0" is worthless later on, while "vacuum lost" explains itself. The difference is naming your signals once.
Third, the time window. Most systems only start recording once the fault is there. What matters are the seconds before, because that is where the cause sits. If you do not capture the run-up, you document the symptom only.
What you need to truly reconstruct an incident
The following five building blocks are vendor neutral. They can be implemented with a central log server, and in principle by other means too, as long as all five come together:
- A shared time base: every device synchronised via NTP, every event stamped by the receiver.
- Signals in plain language: name the pins once, and from then on everyone reads "safety door closed" instead of "DI 7".
- Program context: the client reports the work step and program line, not just the error text.
- A buffered camera image: the cell camera keeps the last seconds, and on an incident the relevant clip is uploaded automatically.
- A good run as reference: only the comparison with a fault-free cycle shows which deviation is genuinely new.
What this looks like in practice
In Leif, our log and asset server, this building block is called the timeline. An incident there is not a single log entry but a replay: you drag a cursor along the time axis and the camera image, signal traces and program line move in sync. Where a reference trace from a good run is stored, the deviation is visible immediately.
The practical effect is less about the technology than about the basis for discussion. Instead of "the cell jams sometimes" there is an exportable incident package on the table that maintenance and supplier can review together.
What changes day to day
- Recurring faults become visible because comparable events sit side by side instead of in memories.
- Supplier conversations run on facts rather than assumptions, including a file to take away.
- Shift handovers lose the item "did not run, nobody knows why".
- Individual cases turn into statistics, and from those comes the decision which fault is worth fixing.
Conclusion
The cause of an incident is almost never untraceable. It simply was not recorded. Bringing together time base, signal names, program context and an image cuts root cause analysis from hours to minutes, permanently and regardless of who is on shift.
If you would like to see what such an incident replay looks like with your own machines, we will show you Leif in 30 minutes, on request with your floor plan and sample data from your production.
Frequently asked questions
Do I need new machines for this?
No. What matters is that your existing devices can hand over their events, be it via TCP, REST, MQTT, syslog or OPC-style routes. Older machines without an interface of their own can be connected through small add-on devices that read existing signals without interfering with the control system.
How long should I keep incident data?
A few weeks are enough for troubleshooting, while several months make sense for quality records and recurrence analysis. Video attachments are the biggest cost factor, so it pays to set a shorter retention period for images than for the log entries themselves.
What about privacy if a camera is running?
The camera shows the cell, not the workplace. Camera framing, retention period and access rights belong on the table with your works council before rollout. Technically it helps that footage is only stored permanently in the event of an incident, while the running buffer is discarded otherwise.
Is this worth it with only a few machines?
The effort pays off less through the number of machines than through the cost of a stoppage. Where an hour of downtime is expensive or faults recur, the analysis pays for itself with a single cell.