Flow Exhibitor Demo: Stop Coding, Start Scaling: Optimize Data Transformation for KPIs, Batch Reporting, OEE, and Beyond

Using our OEE template as an example, we'll demonstrate how you can streamline your Ignition projects by avoiding complex coding and scripting. This is all about scaling your data processing while adding centralized data and engineering governance. Every new KPI we calculate, event we detect, and batch we process, will be served back to Ignition, an MQTT broker, and to the enterprise data warehouse.

Transcript:

00:01
Jeff Knepper: Data is the new oil. If we're digging in our backyard and a bunch of crude comes up out of the ground, we are not rich. We have a huge liability on our hands. Oil is valuable only after it's been captured, after it's been processed, cleansed, transported, distributed, and value has been obtained from it. Data is no different. Let's talk about data a little bit. Our engineers, engineers, engineers, engineers, anybody in the room, an engineer? You've seen a trend before. That's actually not data, by the way. If we took just one point off of that trend and took away all of the context of what's happened before and what's happened after, that's data. This is information. But there's other types of data that we work with. We have transactional records. And what do we keep doing in our goal of obtaining information. We look up at our trends. We say, "Oh, I see I've moved from a state of zero up to a state of 10 to a state of 20."

01:12
Jeff Knepper: And I want to know what's going on with this other data when these state changes are happening. So we draw little lines down our trend charts and we write a bunch of code and we try to figure out how that value has changed during that state. Or we write a bunch of code to try to figure out what's been happening across a couple of shifts. Or we write a bunch of code to figure out what's been happening every single hour, or every single minute, or every single quarter, or how it compares this period to that period, or what it was like when that operator was using it compared to when this operator was using it.

01:49
Jeff Knepper: The point is, in order to try to turn data into information, there is a common factor, and it is we write a bunch of code. It is not easy to turn raw data into information. Our goal is to help you model, transform, and distribute information. My name is Jeff Knepper. This is my partner, Leonard Smit. And today we wanna spend a little bit of time to introduce you to the problems that Flow Software helps you solve. So what is Flow? Flow is a pre-built solution to help you connect to the data sources that matter most to you and your operation. So what are those? Well, in this case, I think we can agree, Ignition. And what is Ignition? Ignition has time series data, Historian. It has transactional data, SQL Bridge. It has manually entered data, Web Forms. And then there's those other sources that Ignition connects to. My quality systems, maybe that's not been built in Ignition, my ERP system, my MES, my CMMS, my enterprise asset management system.

03:06
Jeff Knepper: These are all sources of data that your engineers and your team and your customers really would like to have access to and be able to work with without writing a bunch of code. So we connect to those systems and we do a really neat thing. We normalize those various data formats into one easy to work with format and do a point in time normalization. Now, that's a bit of a measure of words. But what is a point in time normalization? Well, think about those trend charts. We've got pressure that is just moving all over the place. And I've got a transactional record for a batch.

03:48
Jeff Knepper: So batch changed here. Pressure has been moving all over the place. How do I link a point in time on pressure change to a batch record when the timestamps don't match? What Flow does is carry the change from the transactional record forward and assign a virtual timestamp to it, every time I have another timestamp change in my system. So I get a wide table format perfectly set up for row-based calculations so that I have no fear of loss of data when I start to do my transformations. With the data in this type of format ready for me to work with, then I apply my process events, my calendar events, anything that my operations team wants to identify as being important to them. And that is shift patterns, it's financial quarters, it's state changes, it's customer changes, lot runs, batch changes, whatever it is that you need. Maybe safety incidents. It's not limited to machines. And then I aggregate in my model, I write rules in my model on how I'm gonna bring that data together. At that point, Flow's engines execute all of the rules of this model and output the results to a database. Now you have pre-processed and contextualized data ready to be shipped elsewhere. So here's an example of the information that was on the screen before, the trend with the states, the energy usage now contextualized across a batch run. In our little mock simulation here, we're producing water for the fine prisons in the surrounding area.

05:42
Jeff Knepper: Johnny Cash said the water in Folsom wasn't very good, so we're trying to help fix that. We can see that we had a lot. We had a batch start, we had a batch end, we had a duration, we have a customer ID, we have a SKU, and we have a quality check. So now that I've sliced the data by batch runs, I can further slice it. I can say, well, give me lot number one and break lot #1 down into every single stage of that process. I wanna see how much energy did we use during my startup. I wanna see every single downtime. I wanna know why we were down. All of this can be done inside of an information model. Once we've abstracted the underlying data sources, their namespace, and giving your people a really easy namespace that they can interact with. They can code information, they can add comments, they can manually enter data that's missing because it hasn't been automated yet. All of this so that now you have information that's ready to send to any other application, that you wanna send it to. So Lenny, why would I do this in Flow? Why wouldn't I just build it myself?

07:00
Leonard Smit: Well, let's think a little bit of what you need to do to actually get all of those slices, processed and captured. Well, obviously, I need to be able to collect a whole bunch of different data sources, time series data sources, relational data sources. I need a way to actually ingest that into a solution wherever I'm gonna build it. I need to do this point normalization that Jeff has spoken about to get me the boundary values for each and every timestamp whenever the data changes either in the time series data by the source or the relational data source. I need to be able to detect my events. When do I detect an event? When does a batch start? When does a downtime occur? So I need to be able to write all of those rules on when to go and slice it based on my event information.

07:43
Leonard Smit: I need to be able to contextualize that information with data coming from all of these different data sources. And don't forget about the human 'cause the human can also give me context to that data as well. I need to be able to figure out well, we spoke about contextualizing or slicing it per shift per day per hour what is a shift? What is a day? I know it sounds silly to ask you what a day is, but a day means different things for different people 6:00 to 6:00 in the morning or 7:00 to 7:00 or midnight to midnight how do you cater for all of that. How do you not take that information and aggregate it up how do I do a shift to date or a day-to-day total or roll up a whole bunch of production to give me a line production KPI.

08:23
Leonard Smit: I need to be able to version this. If something changed from where I'm getting the data and I'm rerunning the calculations, I need to be able to version it. I need to be able to rubber timestamp it for auditing and governance compliance as well. And then I need to, well, I need to show it somehow. I need to train it. I need to put it on a dashboard. I need to put it on a visualization. And I need to send it out potentially to the hyperscalers, I wanna push this result of this KPI to Snowflake, or I wanna push it to a BI report. So this is just some of the little steps that you need to be able to do, just to be able to get that contextualized data going. I don't know about you, but when I see this list, I just see fingers going crazy and people just coding their life away.

09:04
Jeff Knepper: I could hear the keyboard clicks, and I could see the lines of code starting to develop. And what's the issue with writing code? Maintaining it. The more you write, the more you maintain. The more you write, the less it scales. How do I take the current data project that I'm working on, when I'm done with it, pick it up, and bring it over and put it on this site and reuse it without editing a bunch of code? And if I have to edit a bunch of code, now how do I take it and go put it on 20 other sites? You won't. And if you do, congratulations, you've built a product and now you get to maintain it for the rest of your career. So what does Flow do? Flow starts right after data collection and helps you templatize and model all of these steps while then executing the transformations and the publishing of data according to the rules of your model. Stopping just before you include the information in front of your people or back into other applications to take action on your decisions.

10:13
Jeff Knepper: And you might look at this and say, well, yeah, Jeff, but that's what I do with Ignition. And I'd say, yes, that's what you do with Ignition. That's what you've been doing with Ignition. And we absolutely love Ignition. Our goal is to take your Ignition project and give you a dedicated resource to templatize and execute all of these data transformations and then feed it right back into Ignition so that you continue to build on your Ignition work. And if you're an integrator, to take that work that you've just done, put it into your template library and bring it with you to your next job, standing on the work of, standing on the shoulders of previous work.

10:55
Jeff Knepper: So we know that coding affects us negatively, yet we keep doing it. Maybe it's just time that we say, no more custom code, no more scripting, no more burying this work in applications where I don't have access to it again, no more Excel worksheets that I'm afraid to open up because they're so fragile. And most importantly, centralizing all of this so that I can add governance to my transformations. I wanna make sure that when I define a KPI, that that KPI does not change as I move from one application to the other. I wanna make sure the way I cleanse data is universal across all of my sites, not just based on how one engineer wrote one program over here. Last point, to do all of this work and to do it all in code and do it all in scripting requires a team, it requires people, and it requires people that cost a lot because it is a skill set the more code that you write. Okay, so if you wanna do this and you want to do it well and you wanna get away from code, there's three things that we've identified and built into Flow that will help you do it.

12:05
Jeff Knepper: The very first thing is the ability to write templatized information models. Information models are not, are not master data models. You're not trying to represent every single point inside of your process. An information model is more use case driven. It is the key information that your organization cares about, and it's your ability to govern how that information is brought together over and over and over again before getting shipped out to other applications. Intelligent execution engines, these are simply the bots that do the work that you write into the model. But manufacturing has a ton of challenges. Lenny already touched on it. Data comes in late.

12:51
Jeff Knepper: Data values change. I have to rerun KPIs. If I do secondary aggregation, like rate of change of one KPI over another, if I change an underlying KPI, now I need to rerun those KPIs. That's why it's really important to focus on having complete relational dependencies inside of your model, which of course we've accounted for, and always monitoring the underlying databases. So if data arrives late, or if underlying values change, it triggers the rerunning of all of these calculations, even the republishing of all of those calculations. So if you're publishing this data back to a SQL database or up to a cloud infrastructure like Snowflake or AWS into a Kafka stream, our engines are smart enough to know something's changed, republish. And of course, we're versioning all of the results internally so you always have access to what it was and what it is now. Finally, the last piece is universal information access. Nothing that you build inside of this model, none of the results that get published can stay locked away. I get asked all the time, how are you different than this analytics tool or this analytics tool? And my question is always, well, what you've built in that analytics tool, can you ship it anywhere you want?

14:14
Jeff Knepper: Can you search the model that you've built? Can you move the data freely out of it? And the answer is almost always, no, I can't. And that is the missing piece on almost every single information model and strategy that we come across. So in order for it to scale, we know it needs to stay platform agnostic, be templatized, be API driven, stay away from code as much as possible, be flexible, run on different OSs, and give you universal governance. With that, how about we watch Lenny build a model in real time? You wanna do it in real time?

14:50
Leonard Smit: Let's do it.

14:51
Jeff Knepper: Of course.

14:55
Leonard Smit: So we're gonna cover a little bit of use cases today. Obviously, we're gonna cover a little bit of a batching use case. And then we'll likely touch around OEE as well. But let's get cracking and building out a little bit of a model. Again, the visualization that we started off with today, I've got my state engine, I've got my quality samples, and I've got my energy usage. And I wanna try and make sense of all of this data by contextualizing all of these different data points with one another. So what I'm gonna do is I'm gonna go to our configuration environment. So this is the environment that I'm gonna go and create my information model with all of my KPIs. I'm also gonna use the same tool to hook up to all of my different data sources. Now, those data sources can be time series data as well as our relational databases that we spoke about. Let's quickly connect to a SQL database.

15:44
Leonard Smit: What I'm gonna do, I'm gonna go to my data sources on the right-hand side here, and I'm gonna go and create a new connection to a Microsoft SQL database. Inside of the SQL database will be a lot of MES transactional type of data that I would like to have in context and sliced by my energy information that normally sits in my time series historian. So let's do that let's quickly create a connection there. I'm gonna populate this is gonna be my MES data that I'm gonna connect to and I'm gonna just populate the server name of where that database resides and give the name of the database. So my database is called ACME MES so that will now create a connection to that SQL database.

16:28
Leonard Smit: Test the connection, make sure that it can connect to that database yes it can, hit the save button and that will now create a new database connection. Flow will go and deploy that so that my engines will be able to get the data out of that transactional database. Now, when I select this database out of my data sources, what it's gonna do is it's actually gonna go and browse that SQL database and it's gonna see what is the tables and the structure that lives inside of it. So there we go. I've got all of my different tables. I've got my order history, I've got my SKU that I'm making, my work order number, the customer, all of that data is now available for me. Jeff, can you help me please write a joint statement on all of these tables to get it in context with my time series data?

17:18
Jeff Knepper: And this is where it falls apart for me because no, that's not my job. I'm an engineer. I've got knowledge of my process, but I don't know how to do all this database work, but I need access to this data. So absolutely not. No, I cannot. No, I will not.

17:31
Leonard Smit: Okay, cool. So we've got you covered. We can write a definition file for a SQL schema. And what that means is that I can work either with the vendor or with someone, a champion in my organization that understands that database, to make it easier for us as the engineers to actually interact with that data in relation to my time series data potentially. So I've done this before. I've created another connection to this. But in this case, I've loaded what we call a definition file to that SQL database. Now when I browse the namespace, I get a complete different view. I get a tag-like structure that will tell me what is my customer name, what is the SKU that I'm running for all of the different work orders that's loaded within that database. I don't have to write a single piece of SQL code to do all of the joins to that database. And literally, I can now take this, drag it into my data preview here, and that will now do the point normalization for me, and give me what is the work order that I'm running in context to all the other relational data that I have. So I've got my energy usage, I know what the state of the machine is, I know what my sample is, and look at that, I've got the work order number now in context to that data as well.

18:47
Jeff Knepper: And this is where all of the timestamps have already been prepared and aligned perfectly. So that I don't have to worry about looking backwards to try to figure out, well, what was the work order change when I'm looking at this period of time. We've essentially, take in all of the timestamps, align them, ready to go. Now I can work with this information.

19:09
Leonard Smit: Correct. All right, so job number one, done connecting to data sources, doing the point normalization for me, making it easy to use the data that I've got in my production floor. Let's quickly extend my model now. So I wanna go and be able to track this batch, all the different states that my batch goes through. And I wanna do that within the model environment within Flow. So let's go and create a new folder here. I'm just gonna extend my model. This is my ICC live demo. And within that I'm gonna go and create a metric where I'm gonna go and add all of these different inputs that I'm gonna use to add context to my batch of it. So this is gonna be all my inputs and literally all that I have to do is I would go and create tags that represent all of this data for me inside of my model.

19:56
Leonard Smit: So I would go and create a tag that tells me what is the work order and literally drag it across from the namespace into the model. What is the SKU that I'm running? And let's go and marry that with some data from my time series historian. So I've got a little simulator running where I actually now have the state of what this machine is. So I'm gonna add that to my namespace as well. So there we go. This is the actual state of the machine. Let's add that to the inputs and we can also go and add some or the actual energy that I'm using for that as well. And why not marry that with the data from my LIMS system that does all the QC checks for me. Click on that, drag it across and there's my QC check for that as well.

20:44
Jeff Knepper: So three different systems?

20:47
Leonard Smit: Three different systems.

20:49
Jeff Knepper: Represented in one model?

20:49
Leonard Smit: Represented in one model normalized. And what's the best part about this? I don't have to keep this tag name within what is this in the time names historian. I can normalize these names and this I can just, can simply say this is my energy usage. So give it nice normalized names within my model. Okay. I can now use these tags to go and slice the data to all bunch of different ways. And the first way that we're gonna do it is we're gonna slice it per batch. So I'm gonna use what we call an event within Flow, drag that event across this is gonna be my ICC batch tracker and now I've got rules on when do I need to start the event, stop the event and what's the different context that I need to add to that event?

21:29
Leonard Smit: Let's open that up. And it's simply a drag and drop environment. I say, "Oh I wanna go and trigger this batch every time my state value changes from the control load." And that's it. There's my rule for starting a batch. Okay, what context do I wanna add to this batch? Well let's add some attributes to this batch. I wanna know what is the context of the SKU and literally I go and drag the tag onto that attribute to add the context to it. What is the context of my work order number? Again, drag the work order from my tags or my inputs and drag that as context. So by adding more and more attributes, by adding more and more context to the data that I have available. And I can go on to add the quality check as well. Now the model needs to be able to execute live, no point in just building the model without me having the capability to execute these rules in runtime. So deploy it out. What that means is Flows engines will now starting to execute the model. We will backfill on the historical data if there is in the database and we already start to create these batch events for us. There we go. Sliced batches with information and that will give me the context of my different work orders that's now available for you.

22:51
Jeff Knepper: So now written into Flow's database are each of these events with the context from three different databases stored with each of them.

23:02
Leonard Smit: So let's slice it even more. Let's slice it per hour. So I'm gonna take my energy usage, slice it per hour and I'm also gonna slice it per the batch. So I'm gonna add more and more context as we go along. So I'm gonna slice it per the batch and I'm gonna give it context of the work order. So all of those lines that you saw Jeff drew on the PowerPoint of slicing all of this data, that's exactly what I've done here. Slicing it per my calendar, which will give me context of time, batches, hours, days, weeks, and I'm slicing it by my batch information as well. And again, deploy this out and Flow, the Flow engine will do all of this work in the backend. Okay, adding that context, saving it in the database, having that available for everybody to share.

23:46
Leonard Smit: Okay, now that's just a very simple little batch example, how we slice it, how we add the context to that and how we can get that information now sliced not only per the work order but then also per my hour and having that work order number available as well. Cool. We have a little bit extended the model as well just to show you a little bit of what we can do from potentially an OEE kind of example. So here I can see I've got an OEE example built out and I've got all the typical kind of KPIs that I track from an OEE perspective. I've got my schedule, I've got my available time, I've got all the production information. And you'll notice a little bit different in the model. Those guys are blue. The one that I've built is like this grey kind because this all comes from a template. All right, so we've templatized exactly how a line OEE should look. And we utilize that inside of our model here.

24:43
Jeff Knepper: And when we say we've templatized it, what we mean is as engineers, we've templatized it. Flow, has not told you how to define OEE. In fact, I promised Rick Pallotta that I wouldn't even say OEE today. But you get to define how you do calculations and KPIs according to what your organization has structured those rules and those expressions as, you're not forced to our definitions of them.

25:11
Leonard Smit: Now how can I use this model to even extend my KPI definitions? Well I've got four lines for the one, two, three and four. I know what is the total production per line for the hour, but now I wanna do a line roll up. So what is the total production for all four of the lines? Add it together. Okay. Now the nice thing about having a nice normalized model is I can utilize the model for Flow to go and automatically discover all the measures that I need to KPIs that I need to add together to get me to that point. So I can very simply go and say, okay, underneath production I would like a metric for total production and inside of my total production I'm gonna go and create a new hourly KPI and a calculation for that. This is gonna be my total. And inside here I can now define rules based on what is happening within my model definition.

26:06
Leonard Smit: So I can go and create a new collection of measures that I'm gonna filter within my model. And I'm gonna say, "You know what? I would like to know all the measures that's got a UOM within my model. And that UMM is gonna be units." Okay, so what am I doing? I'm searching the model inside here for anything that's got the unit of measure. So that I will go and search the model and by now the definition you'll see that guy's got units, it's gonna bring back all the measures that's got that associated with it. I can extend this obviously to now also go and search for anything in the model by its name. And again, it will go and filter the model and bring back all of those definitions for me. So I can easily do those type of roll-ups within the solution as well. I made a mistake with my spelling there. I'll fix that now.

27:00
Jeff Knepper: So the beauty of this is that as you move to different pieces of equipment or different processes that have different counts, maybe one side has four, the other side has seven, you're not hard coding these numbers in. Flow is finding that automatically for you based on the logic that you've built into Flow, identifying the correct number of measures or metrics and then aggregating them according to the rules of the collection. Lenny, for time.

27:27
Leonard Smit: Yes.

27:28
Jeff Knepper: I think we should talk about how we get information out of Flow.

27:32
Leonard Smit: 100%. So there's a few ways. What we can do is obviously we like Ignition, we play well, we like Perspective. It's a very good tool to create very rich dashboarding capability. So I just played around a little bit and we are introducing the first search engine for industrial data. We call it Floogle.

27:52
Jeff Knepper: It's Floogle.

27:54
Leonard Smit: Yes.

27:55
Jeff Knepper: Floogle.

27:57
Leonard Smit: I haven't heard about Floogle before. It is a information search site and this is a little Floogle doogle.

28:01
Jeff Knepper: Sure.

28:02
Leonard Smit: And if you see Travis, that's Travis by the way. He's celebrating his twentieth year at Inductive Automation. So if you do see him in the passage, just give him a pat on the back and congratulate him with twenty years at IA. But the point is, I can now search my model. All right, so I can search that Flow model for anything that's got bad in it as an example. So there we go. I've got my actual bad inputs. I've got my five minutely bad production figure, so I'm not searching it by tag name in the historian who can tell me what the tag name for the historian was for that point when we did the demo. Nobody knows. Nobody cares.

28:39
Leonard Smit: Of course we've now normalized this nice model where I can search thing by nice proper names. So click on that, it will pull the data from Flow's API and I can use trending capability within perspective to get my information available as well. I can also go and embed the entire Flow dashboard with this using iframe technology and I can have a complete dashboard that's been built within Flow with all my states and all my downtimes already capable as well or populated as well. One thing that we wanna do is also we need to be able to get human interaction into this. We talked about getting data from transactional data, time series data, but what about human factors? Now we do have the capability to go and do calculation or capturing of frames based on our batches.

29:26
Leonard Smit: In this case, well I wanna know this downtime, what was the state of the downtime? It was an electrical fault based on the electrical guys. So again, humans can go and enter the state classifications for us, but we can also go and add data just based on time. You've got a meter out in the field. It's not historicized, I don't know why it's not historicized as yet. But the point is, I can go and manually add that data in there and again, if I do change the value then it will go and do a full audit trail on that value. Flow will store that data point and we'll have a full history of who changed it, when was it changed, and by whom it was changed as well.

30:08
Jeff Knepper: Thank you. That was a lot to try to show, but I appreciate it. Hopefully you were following along and seeing if nothing else. The fact that this is a drag and drop tool that allows anyone with a little bit of experience to be able to build models. Lenny touched on a point and it's something that's really hit home to our team and that is that in an age where I'm building art using Midjourney, there are still operations and processes that do not have data historians. Worse yet, there's operations and processes that have data historians that have locked data away and made it almost impossible to get data out of or licensing models that are frankly a bit abusive. We do not like that. It actually keeps us from being able to do our job in helping build an information model. And so today I'm really pleased to present on behalf of our founder Graham Walton and the entire Flow team, a new product I'd like to introduce you to Timebase.

31:07
Jeff Knepper: Timebase is a completely free purpose-built industrial historian. We are opening up beta testing for it in two weeks and we are releasing Timebase on December the second. It is performant, it is secure and it is easy to get data out of. We're developing a trending tool currently that will I think change the way you trend information. And of course we have ensured that there is a full rest API to be able to pull the data out of and a licensing model that frankly lets you grab Timebase and install it, whether it's in Docker or in Windows and use it across an organization however you see fit. So if you could help us out and be beta testers, that would be wonderful. We'd love to get your experience with the UI. We'd love to be able to tweak the product. And with a December 2nd release date, it is coming up on us fast. So thank you everybody for coming in today. Our next action would be if you'd like to book a consult or see a live demo for yourself, we'd be happy to do so. You can scan the QR code to get more information about our solution. Yeah, I think we probably wanna end there, but our booth is right outside if you'd like to come ask us some questions, but please do not be late for the technical keynote that kicks off at 1 o'clock.