Manufacturing Challenges - Bringing In New Machinery Into the Plant

A common challenge for manufacturing facilities is bringing in new equipment.

Before Manufacturing Equipment is Acquired

Depending on your facility, you’ve either designed your own custom machinery or you’ve acquired standard machines specific to your process. An example of a standard machine would be a case packer, palletizer, filler, etc. They’re standard because they’re needed in most facilities and multiple groups have been through numerous iterations to optimize the process.

There’s little value in going down the path of creating a palletizer from scratch - it’s going to be expensive and it’s rarely going to add any advantage to your process.

Note that this doesn’t mean that you can’t customize a specific portion of a machine to better suit your needs. If we continue with the example of a palletizer, there are groups that will create custom end-of-arm-tooling (EOAT) that will be used for custom picking and placing your product onto a pallet.

So, what are the challenges in acquiring custom equipment and “standard” equipment?

Pre-Purchase Custom Equipment Challenges

It’s very easy to overspend on custom equipment. Engineering time can balloon as change requests and adjustments are made to the original scope. It’s critical to understand your requirements, the skillset of the group you’ve chosen to execute the project, and the guardrails in place that will either help you change the scope or limit you from building out your vision.

What does this boil down to?

• The completeness of the engineering design is key.
• The alignment of expectations when it comes to change requests is key.
• Understanding the deliverables (timeline) is key.

Pre-Purchase Standard Equipment Challenges

Standard equipment is much easier to purchase. You should be able to find multiple vendors that would be able to build the machine you’re looking for, and based on the following criteria select the right one:

1. What does the vendor provide beyond the machine? - Will you be getting on-site support for the VAT / FAT? Will you be able to call them for a specific period in time? Can you get them to come into the facility when your line is down? Can they customize certain components of the machine to suit your needs?
2. Does the machine fit what you currently have on site? - I’ve seen this mistake time and time again… A manufacturing plant standardized on Rockwell Automation products brings in an OEM that will only sell them a machine that runs on Siemens. This is a bad idea…
3. What are the details of the agreement between you and the OEM? - Are they liable for a certain run rate before you sign off on the machine? Are they liable for the installation and commissioning? Are they liable for training your staff on how to operate and maintain the asset?

What does this boil down to?

• You need to shop around; standard equipment is typically manufactured by multiple OEMs.
• You should understand what the OEM provides beyond the asset.
• You need to understand how the machine will integrate into your facility.

Vendor Acceptance Tests (VATs) / Factory Acceptance Tests (FATs)

A significant amount of money and effort is often spent on VATs / FATs. These initiatives aren’t just boondoggles for the engineers. These activities are key to understand the issues and rectification steps before you sign off on the machine.

Here’s a simple scenario:

• The OEM agrees to build a case packer using a vision system specified by the end-user.
• During the part acquisition process, the OEM realized that the parts they needed weren’t available.
• Without informing the end-user, the OEM decides to proceed using a different vision system their team is familiar with.
• The end-user engineering team shows up at the OEM for the VAT and realizes that the vision system can’t be integrated with the rest of their current line.
• The end-user engineering team mandates that the OEM purchases the correct parts, which will be installed at the end-user site to match what was agreed upon.
• The above saved the end-user $30,000 in engineering integration costs to retrofit the different brand vision systems AND saved the need to train and stock the new parts.

The moral of the story is that a VAT / FAT is used to detect anomalies BEFORE they make it to the plant floor. By investing the resources to validate the equipment before it’s signed off on, end-users can ensure that it has been built to spec, has no issues running at the specified rates, etc.

Integrating Machinery Into Your Network

In the current smart manufacturing (Industry 4.0, IIoT) world, it’s extremely rare not to have some sort of connectivity between various assets and a central location. Depending on how the OEM handles their control system design and how your IT / OT engineers decide to bring this machine onto the network, you may run into challenges.

There’s always more than one way to “skin a cat,” so let’s take a look at a few options below:

Architecture #1 - Flat Network

In small facilities and those that haven’t been modernized, you’ll often find what’s called a “flat network.” These topologies will frequently involve multiple unmanaged switches (or managed ones set to be unmanaged (separate conversation)) scattered throughout the facility.

The engineering / maintenance teams on the OT side will often maintain a log of devices in a spreadsheet with an IP address for every PLC, HMI, VFD, IO block, etc.

If your facility is small enough, it’s possible to operate this way…

Once a new machine arrives, your team can simply all all the network devices onto their spreadsheet, re-assign IP addresses based on the facility table, and move onto operating the machine.

Generally speaking, here’s what you’ll end up with in this case:

192.168.0.10 - Line 1 Filler PLC

192.168.0.11 - Line 1 Filler HMI

192.168.0.12 - Line 1 Filler VFD1

192.168.0.13 - Line 1 Filler VFD2

192.168.0.14 - Line 1 Capper PLC

192.168.0.15 - Line 1 Capper HMI

192.168.0.16 - Line 1 Capper IO

…

192.168.0.116 - Line 6 Palletizer PLC (new)

192.168.0.117 - Line 6 Palletizer HMI (new)

192.168.0.118 - Line 6 Palletizer VFD1 (new)

192.168.0.119 - Line 6 Palletizer VFD2 (new)

192.168.0.120 - Line 6 Palletizer Vision System (new)

Is this a problem?

In general, it’s not a problem for a small operation; it starts to become a problem as you add more devices onto the network. It becomes more of a problem if you add “traffic heavy” devices (vision systems).

Architecture #2 - Separate VLANs / NAT Via Managed Switch

Managed switches come in many different price points in the industrial automation space. It’s often difficult to figure out what you need without consulting an expert. The world of IT has dealt with many of these challenges for decades - there’s a different feature set for a facility of every size.

Using a managed switch, you can isolate the machine you’ve just brought in from the rest of the facility.

Wait a minute; why would we want to isolate? I thought we wanted to connect…?

Well, do you really need the traffic of the VFDs, vision systems, IO modules, safety modules, and many other peripherals on the network? Chances are, your engineers only need to connect to the PLC and HMI. Furthermore, if they’ve gotten there, and they’re using standard equipment, they should be able to “tunnel” to anything else they might need. Ex - You can access a PowerFlex drive through a ControlLogix PLC.

In short, the managed switch can be configured to only allow certain traffic through.

Architecture #3 - Segmentation Using PLC Ports

Many “modern” PLCs (not all) feature two Ethernet ports. It’s important to note that some of them will be the same exact port, while others can have each one configured separately.

Why is this important?

Well, it allows the machine builder / OEM to keep the entire machine control system standardized and allows the end-user to simply set an IP address on the PLC port for communication purposes.

Although the nuances between the approaches are small, it’s important to understand the trade-offs.

The first architecture should only be used in facilities with little equipment on the network and with low IT / OT networking skills. This approach is going to “keep things simple,” but isn’t going to provide any network resiliency should there be excessive traffic.

The second architecture is a solid approach. It requires OT / IT networking knowledge and an expert who would configure the switch properly. Keep in mind that a robust managed switch can be expensive depending on what you need - how many devices are connected, which vendor / platform you’re using, how much control you need, etc. The simple approach is to purchase a switch from the same OEM as the control system. Notice that there might be better alternatives.

The third approach leverages the OT infrastructure to create different access points into the PLC. This is usually simple to implement and segments the machine very well from external access. Note that this provides less “flexibility” in terms of being able to access all devices on that network - Ex: You won’t always be able to tunnel through the PLC to all devices using this architecture. If you need access to a specific network devices using scheme #2, you can always add them to the NAT / VLAN tables.

Post Deployment Challenges

Once your machine shows up on site, the time starts ticking for the following activities:

1. Assembly & Installation | Engineers and technicians need to piece the various parts of the machine based on OEM specifications.
2. Pre-power-on checks | Engineers and technicians need to validate the connections made by the construction crews, check grounds, etc.
3. Power-on checks | The machine is powered on and engineers / technicians validate voltages to breakers, motors, etc.
4. Dry Run | The machine is ran without any materials.
5. Wet Run | The machine is fed materials to “assemble” the final product.
6. OEE | It’s extremely rare to see a machine run at the specified rate on day one. Once it has been shown to be able to run the product, engineers and technicians will need to “tweak” and adjust to meet OEE targets.

At this stage, the exercise is purely operational - a skilled project manager will keep a list of various issues that need to be addressed by each group, ensure that there’s a date & owner, and follow-up with the parties that need to handle the specific tasks - Ex: process engineer, mechanical engineer, electrical engineer, QA supervisor, technician, construction crew, OEM, etc.

Personal Machinery Deployment Experience

We’ve certainly been part of multiple new machine start-up activities. I’ve personally traveled to Italy, Japan, and Germany for VAT equipment before it was brought on-site in the US, commissioned, and run at a manufacturing site. Here are some of our learnings:

Pre-Purchase Design & Validation

Before a commitment is made to purchase new equipment, it’s important to understand what is needed and what will be delivered. For some of the projects I’ve been on, assets similar to the ones at the facility were purchased. It’s important to note however, that requirements change - the machines need to run faster, provide higher performance, and be able to run different SKUs than originally purchased for.

The above ultimately means that a lot of time should be invested in creating these specifications and clearly communicating them to the OEM (in writing!)

I’ve learned that if you fail to mention any of these requirements, you’re likely to be disappointed with the machine, or spend a lot of engineering time retrofitting it once it is delivered.

FATs & VATs Learnings

Different groups, OEMs, vendors approach FATs and VATs differently. As you build the relationship, you can expect the number of “surprises,” or deficiencies to decrease. However, as you start working with a new group, it’s well worth your money to send a team of engineers for a week or two (depending on the process complexity) to validate the equipment. As discussed above, we’d often find deviations from the prints, which if not rectified, would cost the end-users thousands, if not hundreds of dollars.

Commissioning Learnings

The OEM, or machine builder, has no responsibility beyond their machine. They aren’t going to help you integrate the machine into your facility (with some small exceptions). They’re going to ensure that it runs, and that it runs to your agreed upon specifications. It will be up to you and your team to get it connected to the plant SCADA, MES, ERP, etc. The challenges at this stage lie primarily on the networking side (there are electrical challenges as well, but that’s a different story…) - you can manage this multiple ways as specified above. It’s easy to fall in the trap of “wrong” integration that makes it difficult to alter later on. Get IT and OT involved with the right experts to make this happen.

Conclusion

Involving the right groups into conversations around new equipment purchases is critical to every manufacturer. Big, medium, or large, every plant will face challenges when it comes to designing, purchasing, and commissioning equipment. Utilization of various Root Cause Analysis Tools such as the 5 Whys can help with getting these machines onto the plant floor.