Event driven systems in manufacturing

In manufacturing, where a single factory can produce multiple products to order, it seems logical to execute production orders according to a certain time schedule. However, as people experienced in such environments will immediately recognise, during the execution of a single order, or production run, hundreds of things will happen that require more or less attention from either the production system or its operators. All these things that happen we can refer to as events, and each event may require some form of handling. An event can be anything, from unplanned things like a machine reaching its maximum operating temperature to planned things like a production order completing succesfully.

This is not uncommon to systems in other domains than manufacturing, but I will stick to the single manufacturing example for readability.

Not just objects

I’ve been teaching software design and architecture for over 12 years now, next to doing projects, and applying it for over 12 more. A lot of that work has over time centered around objects, components, and services, all of which are more or less structural elements of software systems. A key notion, that has been around since the early 70s, in all this is that our world consists of objects and that it makes sense to represent these objects one-to-one in software structures. This has been stressed even more the past 20 years in Domain Driven Design.

However, over time, both my own experience and that of the advocates of Domain Driven Design, has made it clear that the world does not exists solely of objects, but that most of these objects react to things that happen: events.

As a simple example, let’s consider what happens when a customer calls our factory, because he wants to put in a new order. Looking from an object oriented perspective, we model the customer as an object, the factory employee that answers the phone as an object, and the phone itself as well.

Traditionally, these objects call operations on each other. Better said, using the terminology use in Smalltalk, the first real object oriented programming language, they send messages to each other.

A naief design would result in the diagram below. The customer sends a dial() message to the phone, the phone connect()s to the phone in the factory and the phone in the factory alerts() the operator. Now that is weird. In real live, my phone does not alert me, it just rings and I decide whether I’m alerted or not. Better, I may not even be in the vicinity of the phone, so I cannot be alerted, but the phone still rings. Yes, Schroedinger’s cat might disagree, but let’s not go there now.

Of course, we know that in reality the phone is not going to invoke a pickUp operaton on the Employee. The Employee is picking up the phone (or not) as a reaction to it ringing. In that case ‘PhoneStartedRinging’ is the actual event, and the Employee handles that even by picking up the phone.

Similar reasoning can be applied to the call coming in at the factory phone, and the customer dialing a number on his phone. So, we end up with objects that trigger events, and objects that handle events instead of just sending messages to each other. Events are then defined as things that happen in the system, and which are directly connected to a change in the state of the sytem. The events that are raised in the system contain data that related to the nature of that change.

This implies that object communication is a lot more asynchronous than invocation of operations that we we see in ‘classic’ object oriented systems.

Event driven architecture

This is the basis of event driven architecture, an architectural style that considers a system to be a set of components that react to change by handling events and/or raising events.

Each component can then be an event source, or an event handler. In many cases they are actually both.

As an example let’s look at a manufacturing system. Let’s say a sensor detects a problem with a machine. It raises an event MachineIssueDetected. This issue is then handled by the AlarmSystem to notify the operators, since problems with machines often require immediate attention. At the same time, the event is also handled by the OrderSystem of the supplier of the machine. This triggers an order for relevant spareparts in the supplier’s warehouse.

In a similar way, an event driven architecture makes it very easy to track the flow of materials or orders through a workshop, or trace the use of containers, trays, and mobile equipment throught the production process. In this way, operational processes and statistical analysis can be handled in the same way.

Decoupling

One strong point of an event driven architecture is that with the right technology in place, different components can be much more decoupled from each other. Distribution and handling of events can be based on standard technology and simple data structures, removing the need for big, product specific APIs.

It’s these APIs that are now often getting in the way of making manufacturing more flexible and more open. A lot of factories operate a software stack that consists from top to bottom of an ERP system, a MES system and often a SCADA system and multiple PLCs. Each of these communicate with the layer above or below via predefined interfaces. Every change in the process that requires new or different data leads to (expensive) changes in these interfaces.

Raising a new event and publishing it on an event bus requires only a change in the component that raises the event, and over time components interested in that event can be modified as needed. An event bus in this case is a piece of communications software build exactly for the purpose of exchange events - often in the form of messages. Examples are MQTT brokers, RabbitMQ and Apache Kafka.

Having this in place thus brings a lot of flexibility to the shopfloor and the systems that control it, and it allows different parts of the manufacturing system to respond to these events in real time.

Order in the chaos

While event driven architecture clearly has benefits, it also has some disadvantages if not managed properly. A person may get overpowered by the amount of stimuli one can encounter in a big city or a crowded amusement park. In the same way an event driven system, and its maintainers, can get overwhelmed by the large amount and variety of events.

This is where the combination of an event driven architecture with a unified name space comes into play. A unified namespace or UNS can be compared to a well organised folder structure on a PC. Just like every file goes into a specific folder, in a folder hierarchy, every event can be placed in a specific location in an event hierarchy. A UNS does exactly that: it defines a structure that allows us to position events where they belong. In manufacturing the ISA-95 hierarchy is often used as a starting point for this, with factory specific modifications being introduced over time.

Now with this structure in place as part of the event handling system, different components can focus on (i.e. subscribe to) events in the parts of the hierarchy they are interested in and ignore the rest. It also allows for the introduction of an extra component, a historian, that subscribes to all events, and stores them in a database for future analysis. This allows for offline analysis and improvement of production processes.

But it’s still not simple

Although the combination of EDA and UNS offers significant advantages, it also introduces complexity. Even with the UNS structure in place, the amount of events that occurs is still big. The event processing system that is used should be able to deal with all of them, and make sure that none get lost, or duplicated. Missing an event can be disastrous for production, and not only if a machine overheats, duplicating an event equally so.

While the UNS helps structure the events and their related data, and the event driven architecture ensures that system components can automatically handle events, there is still always a need for monitoring. What needs to be monitored is often not individual events, but a derivative of multiple events. Getting this arranged in real time dashboards, reports and process analysis is not a simple thing. Think for example of the amount of events that need to be combined to say something useful about the operational efficiency (OEE) of a production line. Every ‘item changed’, ‘item completed’, ‘item discarded’ etc event needs to be taken into account there over a given period.

Yet it scales and is resilient

To compensate for this, the combination of event driven architecture and UNS does allow us more easily scale systems and make the more resilient, robust to failure.

The event system will record all events, including possible failure of an element of the production line. This allows other components to react to this and reroute production or start a graceful shutdown of production.

When system needs to grow, the combination of the two allows for that as well. Additional components can be added as additional machines or production lines are added, raising and handling the already know event types, and being added to the UNS in the appropriate place in the structure.

Future proof

Combining event driven architecture and unified namespace is a powerful mechanism to make smart manufacturing work. The event driven architecture ensures that importent events are handled in real time. At the same time, the unified namespace provides a manageable data and event structure to manage and monitor the whole evniroment.