Inductive Automation Blog

In an industry like construction or mining, when something goes wrong, even cleaning up the mess can require some heavy lifting. Having a SCADA solution that improves operator awareness and general safety (not to mention decision-making and sustainability) is the Industry 4.0 version of lifting with your legs, not your back.

George Reed — a Modesto-based company involved in asphalt production, mining, and general heavy civil construction — was looking to further automate the processes at its quarries with the help of Factory Technologies, Inc. (FTI), a systems integrator located in Northern California primarily focused on the food and beverage industry, but with a portfolio that includes other projects like mining and aggregates.

The companies have a history that stretches back over 20 years, a collaboration that predates FTI. “One of my mentors, Dave Hiti [Production Manager at FTI], actually worked with George Reed back in the mid-90s, automating one of their other facilities. He was building the relationship between himself and the George Reed company,” said Cory Mercer, Senior Controls Engineer at FTI.

In late 2019, FTI built on that working relationship by implementing a system made in Ignition — an industrial automation platform for SCADA, HMI, IIoT, and more — to modernize George Reed’s processes. Since then, George Reed has worked to make Ignition its go-to standard.

Initial Success

George Reed runs a material processing site at its shot rock quarries where it blasts and produces asphalt aggregates or other construction materials, harvesting rock out of the mountainside with explosives, then running the rock through a series of crushers and conveyors.

Originally, George Reed had no automation at all, with the entire facility controlled via push buttons from a console housed inside an enclosed room that offered no visibility of the actual process. The reason for initial Ignition implementation was to simply transfer control of the crushing circuit from push buttons to automation. For this, FTI leveraged Ignition's freshly released Perspective Module for native mobile-responsive visualization in what would be the first of FTI’s many Perspective projects.

The system’s overview screen represented a literal overview of the circuit’s conveyors, giving operators the ability to monitor and control all operations from the watchtower, 30 feet up. The Ignition system streamlined the process of starting and stopping every conveyor, replacing a complicated manual procedure with a single touchscreen button. The Ignition system runs through the startup sequence for the entire plant, with a similar, yet reversed, procedure for system shutdown, allowing each conveyor to run out individually.

Where design was concerned, FTI focused on the fundamentals, putting a premium on scalability and ease of use. The overview screen includes tabs for basic hand, off, auto, reset, feed conveyor bypass, and conveyor speed control. Operators can view the status of important indicators like return temperature and clamp pressure through a reporting screen.

Whereas previously managers needed to make a phone call to know whether or not the plants were running, now they can pull the information up on a mobile device and see real-time status of everything.

Building On A Foundation

After that initial success, FTI and George Reed decided to collaborate on two further Ignition implementations. The concept behind these new systems was to dive deeper into analytics, taking advantage of the new features that Inductive Automation has continually added to Perspective since its release.

“Now what we are looking to do is implement a data acquisition side of things and have a more robust operator-friendly platform so we could scale out across all processes that we do,” said Kenneth Cooper, Plant Engineer at George Reed.

The new system can aggregate data, generate reports and trends based on historized data, perform production planning, and display real-time status through a combination of Ignition’s Tag Historian, SQL Bridge, and Reporting modules.

Standardization Through Templatization

For these new projects, FTI learned from the previous implementation to lay a groundwork for scalability and take advantage of new quality-of-life features in Perspective.

Everything that FTI builds in Ignition focuses on templatization, embedded assets, and structured data groupings called User Defined Types (UDTs). From a developer standpoint, templates and UDTs have allowed FTI to build the new systems much more efficiently, creating reusable assets for all of George Reed’s individual facilities while maintaining the same look, feel, and sequence of operation from plant to plant.

George Reed’s goal was to have a system that was robust, but simple enough that it could set its own standards. “Every operator is going to be looking at a similar screen no matter where they're at. So when we train one operator to do a task in one location, it's not a completely new process training them at a different location,” said Cooper. “You can follow the flow of the material through the plant and follow the logic.”

Safety First

Another goal of the new implementation was to simplify the operation, making it easier, and safer, for less-experienced operators to run the system. The pre-Ignition system required operators to press a specific sequence of buttons when a problem occurred, forcing them to quickly react and run down several flights of stairs to the control panel.

In contrast, the Ignition system has made this process much safer and far less laborious. Ignition captures and handles the logic of halting production, allowing operators to view the alarm and quickly ascertain the cause of the fault. “What Ignition has helped us do is catch those events before the operator even knows that event has happened. Mostly now the operator knows that there's a problem because his feeds have stopped,” said Cooper.

Not only has the Ignition system reduced downtime, it is also a literal weight off operators’ backs. George Reed’s plants process hundreds of tons of material per hour; a spillage or overflow at a transfer point could create some very large spills. As Cooper puts it, “You end up shoveling a whole lot of product because of that.”

George Reed is also leveraging Ignition to help tighten preventative maintenance windows, which is especially critical because the rock wears down the steel in the conveyors and crushers over time. “It's actual data that we have that we're collecting that we're able to prove out, day in and day out. So I'm collecting things like tonnages, hours ran, stuff like that to kind of trigger internal mechanisms to make sure that PMs are being done,” said Cooper.

Data-Driven Decisions

The new Ignition system has improved efficiency, delivered better insights and analytics, and resulted in an all-around better operation for George Reed. All of this is due to Ignition’s data-collection capabilities, which have allowed George Reed to replace an “old-school way of doing things” with data-driven decision-making.

With this new level of data accessibility, FTI is making sure to set George Reed up for success. “[FTI] has been historicizing data since its implementation, things like motor amps, motor runtime, fault history, things of that nature, with the intent that as [George Reed] decide[s] to come up with reports that seem relevant to them, we will already have the data setup ready for them,” said Mercer.

“When we have large projects coming up, we can look at actual performance of the plant, actual capacities, capabilities, and we can schedule and plan our production accordingly to make sure that we can not only meet the goals of some one-off projects, but still maintain the current customers and product inventories that we have from a day to day,” said Cooper.

Crushing It

While the projects themselves are not overly complex, the benefits for George Reed are massive. Leveraging the data collected in Ignition, George Reed can continue to yield more efficient results, increasing overall sustainability, with metrics to back it up.

The Ignition system has also made it easier than ever to train new operators, taking the process from a week at minimum to a matter of hours. “The operators really like the system,” said Mercer. “So far, the operators that have had exposure to other SCADA systems say that they like what we've implemented with Ignition significantly more than what they had seen in the past.”

The great partnership between FTI and George Reed, combined with Ignition’s unlimited licensing model, means that there are plenty of new projects in the future, including plans to expand the amount of sites running on Ignition.

No matter what comes next, FTI is confident Ignition can handle the task. “Ignition is such a flexible software, it's really applicable to every industry. As a systems integrator, I feel confident that I could pitch Ignition as a SCADA of choice for any industry I walk into, and it would do the job and excel at that job,” said Mercer.

Project Scope:

• Tags: 1,000
• Screens: 5
• Clients: Many company associates
• Alarms: 0
• Devices used: Web Browser, iPhone, iPad, iPad Mini 
• Architectures used: Ignition Perspective running in Azure. Plants utilize Ignition Edge with MQTT Sparkplug B.
• Databases used: Microsoft SQL Service
• Historical data logged: Yes

Project Summary:

Dolese Bros. is a large provider of aggregate material and cement throughout the state of Oklahoma. Over the past several years, Dolese has upgraded many of its quarries with advanced automation, networking, and operational systems. A key component of this strategy is the deployment of Inductive Automation’s Ignition platform at each quarry, and then a cloud-hosted enterprise-level Ignition system to provide reporting, visualization, and business system integration at a corporate level. Dolese enlisted the help of Industrial Networking Solutions (INS) to accomplish their goals. 

The project includes tools and features such as edge computing, MQTT Sparkplug B, store-and-forward, cellular SD-WAN, private LTE, and cloud-hosted Ignition Perspective. 

Problem:

As the Ignition projects were rolled out for each quarry, Dolese wanted to take advantage of the Perspective Module’s mobile-responsive capabilities to allow plant operators to view the status of quarry operations anywhere. The initial goal of the project was to focus on downtime, OEE, and real-time production reporting. As more information became available, and Dolese began to see the power of providing information to everyone, they requested the ability to view even more. The plant’s legacy SCADA solutions allowed only a limited number of viewers to see what was going on with the equipment. Dolese asked INS to provide them with the capability for anyone in the organization to monitor processes from anywhere, and that everything be mobile-responsive.

Solution:

INS delivered Inductive Automation’s Ignition Perspective Module to create a fully web-based, dynamic system with fast response time and with enough flexibility to include mobile devices. INS looked at the spreadsheet that the plants were using to gather production data, and built a dynamic reusable page that allows the user to enter data that could not be automatically collected, such as item codes and downtime data. With this page, operators can view the data gathered from scales during shifts. 

INS built the page so that it could be reused from site to site. All that is needed is to populate a database table with the information about the various entities within the system, such as work centers, conveyors, or environmental parameters. Once the data table is populated, the page essentially builds itself with the use of query bindings that populate the table, and accordion and flex repeater components. Once the production run is started, the data is sent via MQTT up into the cloud where it is integrated into their ERP system in real time. At the enterprise level, other users can view all this data, including downtime data, over time using the downtime report. This report allows the user to select a work center or entity from the chart, and further drill into things such as a particular production run or a piece of equipment to view downtime data for that selection.    

Results: 

Dolese initially began their Ignition journey wanting to capture downtime events and replace reporting spreadsheets with real-time reporting. Since then, multiple areas of integration were identified, transforming the original proof of concept into a holistic enterprise solution. This includes real-time inventory and production data, downtime tracking and reporting, and plant monitoring in real time for mobile interfaces. As Dolese scales their operations, they plan on using Ignition to integrate plant-to-plant transfer systems with RFID, real-time integrations to their data warehouse to help facilitate ticketing and sales data more efficiently, and begin migrating legacy control systems into Perspective HMIs.

Project Scope

• Tags: 1.1 million
• Screens: approx. 200
• Clients: approx. 60
• Alarms: 160+ on UDTs, 53k+ instances
• Devices used: 9AF OPC UA Device (Custom), 1 OPC DA
• Architectures used: Redundant
• Databases used: SQL Cluster, Denodo VSQL
• Historical data logged: 53,000+ tags

Project Overview

This project is a high-performance human-machine interface (HMI) which assists technicians at BHP’s Technology Remote Operations Centre (TROC) for Minerals Australia in monitoring and managing the status of critical technology infrastructure and components across the operations.

BHP has provided visibility and context of PLCs, switches, servers, applications, substations, data centers and more, using its internally developed OPC UA module for OSI PI to bring in hundreds of thousands of data points.

BHP has also integrated with many of its other enterprise platforms to provide quick, intuitive, and easy-to-digest context at a glance to enable TROC technicians.

The BHP HMI solution is a significant innovation that uniquely integrates the IT world into the OT space for the mining industry. BHP has used new technology and modern integration to combine huge volumes of real-time data and context — presenting this in an actionable form to enhance safety and productivity. BHP’s initial research found no solutions in the market that met its needs, so it chose to develop an innovative solution.

Problem

The TROC operates 24x7 supporting critical technology across assets in Australia. Its purpose is to prevent and minimize safety and production impacts to the assets, triggered by failure or degradation of critical technology. As such, a TROC technician occupies a control desk for a 12-hour shift, monitoring and responding to all issues impacting critical technology services. 

A TROC technician must therefore watch, absorb, and interpret large amounts of complex data to determine if any action is required to prevent or mitigate an impact to operations. 

Prior to the deployment of HMI, the TROC technicians performed this process manually, by searching through alerts across multiple isolated systems. These alerts were presented in the form of thousands of uncontextualized emails, manual and automated phone calls, or simply through “eyes on glass,” monitoring multiple application screens and logs. 

With an ever-increasing reliance on technology to underpin critical safety and production functions across operations, a missed alert or delayed response could result in the shutdown or evacuation of an entire operation, or worse, someone being injured. To avoid this, BHP needed to improve the way TROC was presented with vast amounts of information. 

Solution

After extensive research, reviewing best practices, and interviewing/observing TROC technicians, BHP defined “what good looks like.” This would be TROC technicians receiving ONLY critical data, presented in near-real time, in a way that provides actionable context, displayed on a single pane of glass.

Developing the solution was complex, as BHP’s Australian operations consist of: 

• 30+ sites
• 20+ network domains
• Tens of thousands of devices
• Hundreds of thousands of data-monitoring points

For the software backbone of HMI, BHP chose Ignition Perspective. This solution provides the means to present information in a modern browser interface, provides a zero-install environment, is mobile-device-friendly, and allows BHP to present all the information to the TROC technicians in a traditional SCADA environment. Importantly, it allows the team to leverage an off-the-shelf solution with traditional SCADA functionality, alongside modern IoT connectivity options.

The development team has spanned BHP’s Australian assets to collect device health and process data via the existing OSI PI historian infrastructure, collecting and contextualizing 1.1 million data points every few seconds. BHP achieved this by developing its own OPC UA device modules for Ignition. This integrates with the OSI PI data infrastructure using "data pipes" which ingest the internal OSI PI data stream from Interface/Collectors before it is written to disk in each Historian Archive. Through research and development, BHP found that this is the most efficient and scalable way to collect and feed information into the Ignition SCADA platform. Inductive Automation provided excellent guidance and assistance in this development.

This data is automatically mapped back to data models, while bringing in ServiceNow data (via API) on current incidents, problems, and changes, to provide more context in an alarm situation. BHP also hooks into predictive Splunk alerts via API and consolidates this along with all the asset downtime into alarms using Denodo VSQL calls.

BHP has configured for monitoring:

• 800+ servers
• 1,400+ switches
• 79 GE PLCs
• 257 AB PLCs
• Many services and applications

Ignition Perspective allowed BHP to leverage scripting, automation, and modern integration to quickly deploy at scale. Most of the data models built for these devices in Ignition are scripted and automated, so BHP can quickly deploy at scale and provide the ability to refresh when new devices are added or removed from the landscape.

What made this unique is the scale of the scope across Australia, the application of Technology Health Monitoring, the application of ASM and Alarm Management to Technology Monitoring and the data connectivity to OSI PI, along with how much of the solution is scripted or automated in creation.

Result

The system is providing better visibility and context to operational personnel in the field, as well as a better experience when the TROC is contacted, with the TROC technician already aware of the issue and actively resolving it. Usually, issues are being proactively resolved prior to the operations noticing any effect or impact. As the TROC HMI is accessible to all employees, others outside of TROC watch and respond to alarms, preventing issues earlier than in the past.

BHP is working on adding more context into the HMI to provide TROC Technicians with the visibility and quick fault-finding they require. Pages already developed include:

Server Health Monitoring

PLC Health Monitoring (GE and AB)

• Layer 2 physical network layouts
• Layer 3 physical network layouts
• Power distribution overviews (from Power Station to UPS and devices)
• Sub Station and server room overviews
• Application health overviews (Historians, Fleet Management, SCADA)
• Alarm status page side panel drill-in with trend and ServiceNow data
• Specific Material Risk views showing critical equipment health (Ship Line Tension, Chlorine Monitoring)

There is also a cultural benefit, as BHP is providing a solution that is “fit for purpose” for TROC, rather than presenting the team with multiple siloed solutions which do not provide the context required.

The HMI system results in fewer outages and IT failures. Over time, this will result in significant cost savings from disruptions that have been avoided across all of BHP’s operations in Australia — and, if replicated to other operations, potentially globally.

Additional Information

The team applied international best practices, standards, and learnings from industry in terms of Alarm Management (ISA 18.2), High Performance HMI Design (Abnormal Situation Management) and modern data collection via OPC UA and API Integration to monitor critical technology across Australia in near-real time.

In the same way that virtualization changed the way we develop and deploy industrial software, container computing presses on as the next enabler for scalable, robust, and efficient deployments and developer workflows. Containers can help solve some of the challenges presented by today’s complex and distributed architectures. This session will demonstrate how you can leverage Ignition on Docker to model complex architectures quickly, facilitate multi-version Ignition environments, and increase density on your development systems. Join Kymera & EN Engineering as they show how you can use container concepts to achieve better simulation and load testing for your MQTT deployments. They’ll also share what they’ve learned in using Ignition on Docker and help you envision what you can create with this exciting technology!

Project Scope

• Tags: 5,700
• Screens: 26
• Clients: 2 (Vancouver, Canada and Freiburg, Germany)
• Alarms: 0
• Devices used: 0
• Architectures used: Standard, deployed on MS Azure cloud VMs
• Databases used: 1 Microsoft Azure SQL Server
• Historical data logged: 225 tags

Project Overview

ANDRITZ used Ignition to create a platform for training an artificial intelligence (AI) controller to run an industrial plant. The company built a software application in Ignition that integrates process simulation software and machine learning components.

ANDRITZ built a prototype as an internal development project using its own funds. The company then entered and won the 2019 GoldCorp #DisruptMining competition, a contest that focuses on digital transformation of the mining industry to optimize profitability and competitiveness. The prize is a 1-million-dollar (Canadian) project to execute a pilot at one of GoldCorp’s sites.

Problem

ANDRITZ developed a concept of using its proprietary process simulation software, IDEAS, to be the training ground for machine learning algorithms that, once trained, would be used for automatic plant control. ANDRITZ needed to build software that could control its IDEAS software and pass data back and forth between the virtual plant and the AI. Eventually, this same software would need to pass data back and forth with the running plant, using industrial communication protocols.

The new software also needed to function as an administration interface for training the AI. This new software needed to allow a user to set up training scenarios, like a particular plant malfunction that the AI should learn. The software would then manage the training, recording results to history and reporting on the outcome.

This was a research and development project for ANDRITZ, so it needed a platform that would allow it to create a robust prototype quickly with minimal cost or time spent creating the fundamental building blocks.

Solution

ANDRITZ took advantage of a wide range of Ignition capabilities in building this solution. Communication between the IDEAS model or the real plant and the AI engine was a classic IT/OT bridge, with OPC on one side and scripted HTTP calls on the other. Scenario configuration is a very standard database application using a SQL connection for the database backend and Vision clients to build the front end. Monitoring training performance uses Vision, Historian, and Reporting functions.

Dispatch of the training scenarios themselves is accomplished using sequential function charts and a significant amount of Python code.

The whole AI training environment was deployed on Microsoft Azure Europe cloud virtual machines, using an Azure SQL Server for database functions. There were separate VMs for the Ignition Gateway, PSIORI AI engine, and 5 IDEAS process models. During the AI training process, users in Germany and Canada simultaneously connected Vision clients to the Ignition cloud gateway.

This solution is, as far as ANDRITZ knows, the first application of reinforcement learning to control continuous industrial processes. This AI uses the same principles as well-publicized successes like AlphaGo. It is particularly applicable to sequential decision problems where an agent must take multiple discrete actions in sequence in order to achieve some future target state. That strength makes it applicable to a wide variety of control tasks that otherwise could only be controlled manually, like plant startup or upset recovery where the operator must start several motors or swing valves in a particular order that is defined by present plant conditions. The AI has also demonstrated itself suitable for continuous optimization problems where a plant might otherwise use model-predictive control or expert systems.

Result

The media exposure from winning the contest has been very significant for ANDRITZ. The #DisruptMining contest has a high profile in the mining industry, so ANDRITZ has been interviewed on TV, radio, and for several magazines. The award for ANDRITZ was the cover story for an issue of Canadian Mining & Energy magazine. The win has given ANDRITZ credibility in the sales phase with other potential customers both inside and outside the mining industry. Following this win, sales interest in this technology has been tremendous.

The process of prototyping and executing with Ignition as a platform demonstrated Ignition’s capabilities beyond being simple SCADA software. It is software to build software. Ignition enabled ANDRITZ to build a complex application at a small fraction of the cost of using any other method.

Project Scope

• For a typical project:
• 425 tags
• 8 screens, 6 popups
• 1 client
• 0 alarms
• 1 OPC DA device (Kepware)
• Standard Architecture
• MSSQL Server, single instance
• Approx. 3,700 points of historical data logged

Project Overview

For this project, ANDRITZ Automation used Ignition to create an engineering tool that performs detailed analysis of the preliminary design of a plant. ANDRITZ creates an engineering design model of the plant in its proprietary simulation software called IDEAS. Then, Ignition controls the execution of a multitude of design scenarios and aggregates the results for the user, graphically presenting the impacts of design choices on parameters like plant throughput and energy consumption.

Automating scenario execution and reporting has allowed ANDRITZ to assess process design options in more detail and at a much earlier stage in the project than would have been possible using traditional engineering methods. Having deployed this application on several projects in the pre-feasibility stage, ANDRTIZ has been able to demonstrate conclusively why certain design choices are superior and other options should not be used.

Being able to subject a design to a wide range of operating conditions (like ore type and plant throughput) gives a greater degree of certainty at a very early project phase. By excluding inferior design options early, customers can prioritize their engineering hours and avoid costly changes later in the project.

Process simulation software has long been an effective tool for validating plant design in many industries; a model is constructed in software and process engineers use it to verify a design. Typically, engineers use a model to test a design hypothesis by setting a value (like a crusher size) and then running the model under various ore types and throughputs, monitoring the impact on the process to determine the optimal design choice.

Historically, running scenarios and gathering data involved a number of manual steps which, for budget and schedule reasons, significantly constrained the number of design scenarios that could be run.

To solve this problem, ANDRITZ Automation used Ignition to create a supervisory layer which sits on top of ANDRITZ’s proprietary IDEAS simulation software. The Ignition layer allows the user to define which variables in the model they are interested in varying, and the range for each. Ignition then automates the scenario execution, accumulating results in a database. After the scenario set is complete, tables, charts, and reports allow the engineer to assess the impacts on the process of each design variable.

Once the user has entered the variables of interest, scripts create the scenarios to run and load them into a database. A typical scenario set has between 40 and 200 scenarios and will take 2-10 hours to run.

Sequential Function Charts, or SFCs, manage scenario execution. When a scenario set is started by the user, the SFC fetches each individual scenario’s parameters from the database, commands IDEAS to run, waits for the simulation to converge, gathers the results, and then moves on to the next scenario. In order to automate the execution of IDEAS software, Python scripting in Ignition uses XML-RPC (remote procedure call) to command IDEAS via the IDEAS Application Programming Interface.

Ignition collects process data from the model using OPC. A multi-transmitter OPC object in the model transmits flow, density, pressure, temperature, particle size (D50/D80), and percent solids from the inflow and outflow of every piece of equipment to UDTs in Ignition. Along with equipment power, these values are stored to a database at the completion of each scenario. For convenience, the Ignition tags are built automatically from an XML import of IDEAS model contents.

When the system is used for dynamic instead of steady-state analysis, a time series of the process data is stored. However, each scenario set refers to simulated time instead of wall clock time, such as 0-20,000 seconds after an instigating event. A custom historian is necessary so that simulation time and scenario identification can be used to compare the results of different scenarios by overlaying them on the same time scale.

Certain measured values, typically equipment power and any flows into or out of the plant, are assigned material types from a material cost database. This feature allows aggregation of, for example, total plant power usage or water consumption for each scenario.

Equipment details from the model are imported into the database via XML files and scripting. Tags are associated with database objects so that the operating conditions (minimum and maximum flow, for example) for each piece of equipment are saved. This data forms the core of equipment specification sheets, which are created using PDF reporting and exported to distribute to vendors for quotation as part of the capital cost-estimating process.

Ignition allowed us to efficiently create an application that replaces engineering hours with computation. Ignition’s suitability for rapid software prototyping, particularly the combination of visualization with powerful scripting and database integration, was key to the success of this project.

This solution is one of several unique pieces of software for which ANDRITZ has used Ignition as a platform. Beyond its capacity to create powerful HMIs, Ignition really is a powerful platform for creating a wide range of applications.

The Ignition community is made up of some of the most forward-thinking industrial professionals around. In this closing session of the conference, we will celebrate their spirit of innovation by honoring the best projects from the Discover Gallery with Ignition Firebrand Awards. At this rousing and informative panel discussion, hear from the people behind these award-winning projects as they discuss how they did it and what they are doing next with Ignition.