Do You Have Single Points of Failure?
Plans need to be in place before a catastrophic event occurs.
by David Wynn
Director of Technical Services, PMPA
Published October 1, 2024
The CrowdStrike event showed how having a single point of failure (in this case, a coding mistake) can have cascading effects. Services of companies worldwide were halted until issues could be resolved. We are in no-fail industries, so it begs the question … Do you have single points of failure in your shop?
Failure
Failure can come in many forms. Production can stop in a shop due to running out of coolant, lost tools or late arrival of raw materials. All these are failures that affect one job or one machine that can be remedied. We can always try alternate tooling. Jobs can be moved to another machine. Parts could even be made from larger diameter material for expedited production. They are not catastrophic single points of failure because contingencies are already in place. At the very least, solutions can be found quickly.
However, an event that takes days, weeks or longer to recover from is considered catastrophic. Solutions may not be readily apparent, so when catastrophic events occur, plans need to already be in place; plans that were communicated to all parties involved to get mission-critical processes moving.
Power Outages
Power outages can stop production. Do you have alternate sources of power in case of emergency? Companies use battery backup on electronics all the time. Have you ever thought about using them on a machine? A small power outage for just a few minutes can interrupt production across a shop. The air compressor needs to be restarted. All the machines and bar loaders must be reset. If machines were in the cut, they must be carefully backed out and restarted. There might even be scrap associated with the outage, because a machine was in the middle of an operation that can’t easily be restarted — I am thinking of threading. Who is responsible for restarting the shop during a power outage? Is there a procedure? What happens if this outage continues for days or weeks? I know of a member shop whose area was devastated by a tornado. They were first told that the power would be out for a month. Fortunately, this shop already had a plan in place to rent a large generator from a local electrician. Luckily, power was restored within a week, but what would have happened if they did not have a plan?
Equipment
Our equipment can fall into this category as well. Think about having only one of a top-of-the-line Swiss machine running critical parts. What happens when that machine is down? Parts cannot be shipped to the customer because of this single point of failure. Having at least two of every machine is the best solution; however that is not always economically feasible. At least have an alternate process to produce parts until the machine can be repaired. It is better to make production doing three operations than to fail our customers. It is important to have contingencies on all aspects of producing parts down to having alternate outside vendors. What if a plater has a massive fire? Are there other vendors pre-production approval processed and ready to fill the void?
Physical Plant
A shop could be a single point of failure in and of itself. Are there backup plans to serve customers if a catastrophic event destroys the building and equipment? For instance, a tornado, hurricane or earthquake could prevent a shop from operating for months. How will the customers be served during this time? Do you have backup suppliers that can fill the void until production is up and running?
Planning for disaster is important. Hopefully, we never have to use the plans. If there are no plans for disaster, high-pressure scenarios can result in fatal failure. Build plans around the worst-case scenarios. Know what to do when an event happens. Building out disaster plans gets to the root of what our FMEA process should look like. Go out and build real FMEAs that have actionable solutions.
Author
David Wynn is the PMPA Director of Technical Services with over 20 years of experience in the areas of manufacturing, quality, ownership, IT and economics. Email David