Learning from the incident in which data from 18 temperature sensors were lost suddenly.
Today, we are going to discuss an incident in which data from 18 temperature transmitters (or tags) were lost on PLC for about half an hour and started showing 0 deg centigrade temperature instead of the real-field value.
Temperature Sensors Fault
Let us see this incident.
In one area of a plant, suddenly readings of 18 temperature tags went 0 degrees centigrade. Already, 18 tags were showing 0 degrees centigrade because of issues in some instrumentation systems. So, now a total of 36 tags were showing 0 degrees centigrade.
Background of the Plant Shutdown
A total of 18 temperature tags were going 0 degrees centigrade and coming back normal within a few seconds. This issue was seen frequently for the last 2 days. For the last few minutes, these tags were showing continuously 0 degrees centigrade.
So, the Operation Engineer asked the Instrumentation Engineer to check for the issue and resolve it. The instrumentation Engineer took a permit to check/replace a temperature multiplexer (MUX) which had these 18 temperature tags and started doing the job.
After 20 minutes, the other 18 temperature tags also went 0 degrees centigrade Instrumentation Engineer suddenly rushed to another temperature mux to see its status. It was working ok and no abnormality was found.
The instrumentation Engineer checked everything in that temperature mux but everything was found normal.
Plant Shutdown Analysis
Now let us try to figure out what happened and how the issue was resolved.
- The Operation Engineer informed the Instrumentation Engineer about the problem with the 18 temperature tags and issued a permit for checking the job.
- The instrumentation Engineer went to the field to check the temperature MUX as all 18 tags were in the same temperature mux and found that the mux is in power ON condition (This is area 1 temperature mux).
- Red LED on the temperature mux was ON which indicates fault. So, the Instrumentation Engineer tried to power reboot the temperature mux. On power reboot also, the red LED was ON which indicated that the temperature mux is faulty.
- The instrumentation Engineer arranged a spare temperature mux for replacement. This temperature mux was already pre-configured and was ready to be used in case of emergency.
- The instrumentation Engineer removed the faulty temperature mux and installed a spare temperature mux and powered it ON.
- After a few seconds, the Instrumentation Engineer received a contact from the Operation Engineer. The Operation Engineer said that some other 18 tags also went to 00 centigrade (This is area 2 temperature mux).
- Instrumentation Engineer rushed to area 2 temperature mux. On seeing the temperature mux, everything was found healthy.
- He tried to power reboot the area 2 temperature mux. When Instrumentation powered off the 2 temperature mux, he received contact from the operation team that all 18 temperature tag readings were coming but there was something wrong. These readings are of area 1.
- The Instrumentation Engineer understood everything and rushed to area 1 temperature mux and changed the Modbus address to 1 from 2.
- Then he went to area 2 temperature mux and powered it ON. All data became healthy.
What happened and why it was happened?
When the instrumentation Engineer arranged a spare for the temperature mux, it was installed on the test bench. A few other engineers were working on it for some reason.
At that time, they might have given it the Modbus address as 2. So, when the Instrumentation Engineer installed this temperature mux, 2 temperature mux with the same Modbus addresses were present on the system and so, PLC started showing 0 degrees centigrade for all 36 tags.
When the Instrumentation Engineer powered off the area 2 temperature mux, at that time the temperature mux in the area 1 was present online with Modbus address 2. So, all temperature tags of area 2 started showing the value of area 1. After changing the area 1 temperature mux’s Modbus address to 1, issue got resolved.
So, learning from the incident is that whenever an addressable device is there, then give it an address that does not match with already installed systems in the plant. Or write the address on the device.
If you liked this article, then please subscribe to our YouTube Channel for Instrumentation, Electrical, PLC, and SCADA video tutorials.
You can also follow us on Facebook and Twitter to receive daily updates.
Read Next:
- Solenoid Valve Failure in Start-Up
- Root Cause Analysis for Plant Trip
- Malfunctioning Inlet Check Valves
- BP Refinery Explosion at Texas City
- Junction Box Problems and Solutions
