This clause describes the general OPC UA PubSub concepts.

The DataSet constitutes the payload of messages provided by the Publisher and consumed by the Subscriber. The DataSet is described in 5.2. The mapping to messages is described in 5.3. The participating entities like Publisher and Subscriber are described in 5.4.

The abstract communication parameters are described in clause 6.

The mapping of this model to concrete message and transport protocol mappings is defined in clause 7.

The OPC UA Information Model for PubSub configuration in clause 9 specifies the standard Objects in an OPC UA AddressSpace used to create, modify and expose an OPC UA PubSub configuration.

Figure 2 provides an overview of the Publisher and Subscriber entities. It illustrates the flow of messages from a Publisher to one or more Subscribers. The PubSub communication model supports many other scenarios; for example, a Publisher may send a DataSet to multiple Message Oriented Middleware and a Subscriber may receive messages from multiple Publishers.

image005.png

Figure 2 – Publisher and Subscriber entities

Publishers and Subscribers are loosely coupled. They often will not even know each other. Their primary relation is the shared understanding of specific types of data (DataSets), the publish characteristics of messages that include these data, and the Message Oriented Middleware.

The “messages” in Figure 2 represent NetworkMessages. Each NetworkMessage includes header information (e.g. identification and security data) and one or more DataSetMessages (the payload). The DataSetMessages may be signed and encrypted in accordance with the configured message security. A Security Key Server is responsible for the distribution of the security keys needed for message security.

Each DataSetMessage is created from a DataSet. A component of a Publisher called DataSetWriter generates a continuous sequence of DataSetMessages. Syntax and semantics of DataSets are described by DataSetMetaData. The selection of information for a DataSet in the Publisher and the data acquisition parameters are called PublishedDataSet. DataSet, DataSetMetaData and PublishedDataSet are detailed in 5.2.

Note 1: The PubSub directory is an optional entity that allows Publishers to advertise their PublishedDataSets and their communication parameters. This directory functionality is planned for a future release of this specification.

A DataSet can be thought of as a list of name and value pairs representing an Event or a list of Variable Values.

A DataSet can be created from an Event or from a sample of Variable Values. The configuration of this application-data collector is called PublishedDataSet. DataSet fields can be defined to represent any information, for example, they could be internal Variables in the Publisher, Events from the Publisher or collected by the Publisher, network data, or data from sub-devices.

DataSetMetaData described in 5.2.3 defines the structure and content of a DataSet.

For publishing, a DataSet will be encoded into a DataSetMessage. One or more DataSetMessages are combined to form the payload of a NetworkMessage.

Figure 3 illustrates the use of DataSets for publishing.

image006.png

Figure 3 – DataSet in the process of publishing

A PublishedDataSet is similar to either an Event MonitoredItem or a list of data MonitoredItems in the Client Server Subscription model. Similar to an Event MonitoredItem, a PublishedDataSet can select a list of Event fields. Similar to data MonitoredItems, the PublishedDataSet can contain a list of Variables.

A DataSet does not define the mechanism to encode, secure and transport it. A DataSetWriter handles the creation of a DataSetMessage for a DataSet. The DataSetWriter contains settings for the encoding and transport of a DataSetMessage. Most of these settings depend on the selected Message Oriented Middleware.

The configuration of DataSets and the way the data is obtained for publishing can be configured using the PubSub configuration model defined in clause 8.2 or with vendor specific configuration tools.

DataSets can be individual for a Publisher or they can be derived from a DataSetClass. Such a DataSetClass acts as template declaring the content of a DataSet. The DataSetClass is identified by a globally unique id – the DataSetClassId (see 6.2.2.2).

The DataSetMetaData is identical for all PublishedDataSets that are configured based on this DataSetClass. The DataSetClassId shall be in the corresponding field of the DataSetMetaData.

When all DataSetMessages of a NetworkMessage are created from DataSets that are instances of the same DataSetClass, the DataSetClassId of this class can be provided in the NetworkMessage header.

DataSetMetaData describes the content and semantic of a DataSet. The structure description includes overall DataSet attributes (e.g. name and version) and a set of fields with their name and data type. The order of the fields in the DataSetMetaData shall match the order of values in the published DataSetMessages.

The DataSetMetaDataType is defined in 6.2.2.1.2.

Example description (simplified, in pseudo-language):

Name: Temperature-Sensor Measurement

Fields: [1] Name=DeviceName, Type=String

[2] Name=Temperature, Type=Float, Unit=Celsius, Range={1,100}

Subscribers use the DataSetMetaData for decoding the values of a DataSetMessage to a DataSet. Subscribers may use name and data type for further processing or display of the published data.

Each DataSetMessage also includes the version of the DataSetMetaData that it complies with. This allows Subscribers to verify if they have the corresponding DataSetMetaData. The related ConfigurationVersionDataType is defined in 6.2.2.1.5.

DataSetMetaData may be specific to a single PublishedDataSet or identical for all PublishedDataSets that are configured based on a DataSetClass (see 5.2.2).

There are multiple options for Subscribers to get the initial DataSetMetaData:

There are multiple options to exchange the DataSetMetaData between Publisher and Subscriber if the configuration changes.

The term message is used with various intentions in the messaging world. It sometimes only refers to the payload (the application data) and sometimes to the network packet that also includes protocol-, security-, or encoding-specific data. To avoid confusion, this specification formally defines the term DataSetMessage  to mean the application data (the payload) supplied by the Publisher and the term NetworkMessage  to mean the message handed off and received from a specific Message Oriented Middleware. DataSetMessages  are embedded in NetworkMessages. Figure 4 shows the relationship of these message types.

image007.png

Figure 4 – OPC UA PubSub Message Layers

The transport protocol-specific headers and definitions are described in 7.3.

Following is an abstract definition of DataSetMessage and NetworkMessage. The concrete structure depends on the message mapping and is described in 7.2.

A DataSetMessage field is the representation of a DataSet field in a DataSetMessage.

A DataSet field contains the actual value as well as additional information about the value like status and timestamp.

A DataSet field can be represented as a DataValue, as a Variant or as a RawData in the DataSetMessage field. The representation depends on the DataSetFieldContentMask defined in 6.2.3.2.

The representation as a DataValue is used if value, status and timestamp should be included in the DataSetMessage.

The representation as Variant is used if value or bad status should be included in the DataSetMessage.

The representation as RawData is the most efficient format and is used if a common status and timestamp per DataSet is sufficient.

A DataSetMessage is created from a DataSet. It consists of a header and the encoded fields of the DataSet.

Depending on the configured DataSetMessageContentMask, a DataSetMessage may exist in different forms and with varying detail. DataSetMessages do not contain any information about the data acquisition or information source in the Publisher.

Additional header information includes:

DataSetWriterId Identifies the DataSetWriter and indirectly the PublishedDataSet.

Sequence number A number that is incremented for each DataSetMessage. Can be used to verify the ordering and to detect missing messages.

Timestamp A timestamp describing when the data in this DataSetMessage was obtained.

Version Version information about the configuration of the DataSetMetaData.

Status Status information about the data in this DataSetMessage.

Keep alive When no DataSetMessages are sent for a configured time period, a keep alive DataSetMessage is sent to signal the Subscribers that the Publisher is still alive.

Some encodings differentiate between key frame DataSetMessages and delta frame DataSetMessages. A key frame DataSetMessage includes values for all fields of the DataSet. A delta frame DataSetMessage only contains the subset that changed since the previous DataSetMessage.

A key frame DataSetMessage is sent after a configured number of DataSetMessages.

The NetworkMessage is a container for DataSetMessages and includes information shared between DataSetMessages. This information consists of:

PublisherId Identifies the Publisher.

Security data Only available for encodings that support message security. The relevant information is specified in the message mapping.

Promoted fields Selected fields out of the DataSet also sent in the header.

Payload One or more DataSetMessages.

The payload, consisting of the DataSetMessages will be encrypted in accordance with the configured message security. Individual fields of a DataSetMessage can be marked as being “promoted fields”. Such fields are intended for filtering or routing and therefore are never encrypted. How and where the values for promoted fields are inserted depends on the NetworkMessage format and the used protocol. The NetworkMessage header is not encrypted to enable efficient filtering.

Message security in PubSub concerns integrity and confidentiality of the published message payload. The level of security can be:

  • No security
  • Signing but no encryption
  • Signing and encryption

Message security is end-to-end security (from Publisher to Subscriber) and requires common knowledge of the cryptographic keys necessary to sign and encrypt on the Publisher side as well as validate signature and decrypt on the Subscriber side.

The keys used for message security are managed in the context of a SecurityGroup. The basic concepts of a SecurityGroup are described in 5.3.7.

This standard defines a general distribution framework for cryptographic keys. This framework is introduced in 5.4.3.

All parameters that are relevant for message security are described in 6.2.4. These parameters are independent of any Broker level transport security.

The message security for PubSub is independent of the transport protocol mapping and is completely defined by OPC UA.

The transport security is specific to the transport protocol mapping.

When using a broker-based middleware (see 5.4.4.2.2), confidentiality and integrity can be ensured with the transport security between Publishers and the Broker as well as Subscribers and the Broker. The Broker level security in addition requires all Publishers and Subscribers to have credentials that grant them access to a Broker resource.

Transport security may be hop-by-hop security with some risk of man-in-the-middle attacks. It also requires trusting the Broker since the Broker can read the messages. Combining transport security with message security reduces this risk.

A SecurityGroup is an abstraction that represents the message security settings and security keys for a subset of NetworkMessages exchanged between Publishers and Subscribers. The security keys are used to encrypt and decrypt NetworkMessages and to generate and check signatures on a NetworkMessage.

A Security Key Service (SKS) manages SecurityGroups and maintains a mapping between Roles and their access Permissions for a SecurityGroup. This mapping defines if a Publisher or Subscriber has access to the security keys of a SecurityGroup. The SKS is described in more detail in 5.4.3.

A SecurityGroup is identified with a unique identifier called the SecurityGroupId. It is unique within the SKS. A Publisher for its PublishedDataSets must know the SecurityGroupId. For Subscribers the SecurityGroupId is distributed as metadata together with the DataSetMetaData. The metadata for a SecurityGroupId includes the EndpointDescription of the responsible SKS. Publishers and Subscribers use the EndpointDescription to access the SKS and the SecurityGroupId to obtain the security keys for a SecurityGroup.

The Publisher is the PubSub entity that sends NetworkMessages to a Message Oriented Middleware. It represents a certain information source, for example, a control device, a manufacturing process, a weather station, or a stock exchange.

Commonly, a Publisher is also an OPC UA Server. For the abstract PubSub concepts, however, it is an arbitrary entity and should not be assumed to be an individual or even a specific network node (an IP or a MAC address) or a specific application. Figure 5 illustrates a Publisher with data collection, encoding and message sending.

image008.png

Figure 5 – Publisher details

A single Publisher may support multiple PublishedDataSets and multiple DataSetWriters to one or more Message Oriented Middleware. A DataSetWriter is a logical component of a Publisher. See 5.4.1.2 for further information about the DataSet writing process.

If the Publisher is an OPC UA Server, it can expose the Publisher configuration in its AddressSpace. This information may be created through product specific configuration tools or through the OPC UA defined Methods. The OPC UA Information Model for PubSub configuration is specified in clause 9.

Figure 6 illustrates the process inside a Publisher when creating and sending messages and the parameters required to accomplish it. The components, like DataSet collection or DataSetWriter should be considered abstract. They may not exist in every Publisher as independent entities. However, comparable processes have to exist to generate the OPC UA PubSub messages.

image009.png

Figure 6 – Publisher message sending sequence

The sending process is guided by different parameters for different logical steps. The parameters define for example when and how often to trigger the sending sequence and the encoding and security of the messages. The PubSub communication parameters are defined in 6.

The first step is the collection of data (DataSet) to be published. The configuration for such a collection is called PublishedDataSet. The PublishedDataSet also defines the DataSetMetaData. Collection is a generic expression for various different options, like monitoring of Variables in an OPC UA Server AddressSpace, processing OPC UA Events, or for example reading data from network packets. In the end, the collection process produces values for the individual fields of a DataSet.

In the next step, a DataSetWriter takes the DataSet and creates a DataSetMessage. DataSetMessages from DataSetWriters in one WriterGroup can be inserted into a single NetworkMessage. The creation of a DataSetMessage is guided by the following parameters:

The resulting DataSetMessage is passed on to the next step together with the DataSetWriterId (see 6.2.3.1), the DataSetClassId (see 6.2.2.2), the ConfigurationVersion of the DataSetMetaData (see 6.2.2.1.5), and a list of values that match the configured propagated fields.

Next is the creation of the NetworkMessage. It uses the data provided from the previous step together with the PublisherId (see 6.2.6.1) defined on the WriterGroup. The structure of this message is protocol specific. If the SecurityMode (see 6.2.4.2) requires message security, the SecurityGroupId (see 6.2.4.3) is used to fetch the SecurityPolicy and the security keys from the SKS (see 5.4.3). This information is used to encrypt and/or sign the NetworkMessage as required by the SecurityMode.

The final step is delivery of the NetworkMessage to the Message Oriented Middleware through the configured Address.

Subscribers are the consumers of NetworkMessages from the Message Oriented Middleware. They may be OPC UA Clients, OPC UA Servers or applications that are neither Client nor Server but only understand the structure of OPC UA PubSub messages. Figure 7 illustrates a Subscriber with filtering, decoding and dispatching of NetworkMessages.

image010.png

Figure 7 – Subscriber details

To determine for which DataSetMessages and on which Message Oriented Middleware to subscribe, the Subscriber has to be configured and/or use discovery mechanisms.

Subscribers shall be prepared to receive messages that they do not understand or are irrelevant. Each NetworkMessage provides unencrypted data in the NetworkMessage header to support identifying and filtering of relevant Publishers, DataSetMessages, DataSetClasses or other relevant message content (see 5.3).

If a NetworkMessage is signed or signed and encrypted, the Subscriber will need the proper security keys (see 5.3.5) to verify the signature and decrypt the relevant DataSetMessages.

Once a DataSetMessage has been selected as relevant, it will be forwarded to the corresponding DataSetReader for decoding into a DataSet. See 5.4.2.2 for further information about this DataSet reading process. The resulting DataSet is then further processed or dispatched in the Subscriber.

If the Subscriber is an OPC UA Server, it can expose the reader configuration in its AddressSpace. This information may be created through product specific configuration tools or through the OPC UA defined configuration model. The OPC UA Information Model for PubSub configuration is specified in clause 9.

Figure 8 illustrates the process inside a Subscriber when receiving, decoding and interpreting messages and the parameter model required for accomplishing it. As for the Publisher, the components should be considered abstract.

image011.png

Figure 8 – Subscriber message reception sequence

The Subscriber has to select the required Message Oriented Middleware and establish a connection to it using the provided Address. Such a connection may simply be a multi-cast address when using OPC UA UDP or a connection to a message Broker when using MQTT or AMQP. Once subscribed, the Subscriber will start listening. The sequence starts when a NetworkMessage is received. The Subscriber may have configured filters (like a PublisherId, DataSetWriterId or a DataSetClassId) so that it can drop all messages that do not match the filter.

Once a NetworkMessage has been accepted, it has to be decrypted and decoded. The security parameters are the same as for the Publisher.

Each DataSetMessage of interest is passed on to a DataSetReader. Here, the DataSetMetaData is used to decode the DataSetMessage content to a DataSet. The DataSetMetaData in particular provides the complete field syntax including the name, data type, and other relevant Properties like engineering units. Version information that exists in both the DataSetMessage and the DataSetMetaData allows the Subscriber to detect version changes. If a major change occurs, the Subscriber needs to get an updated DataSetMetaData.

Any further processing is application-specific. For example, an additional dispatching step may map the received values to Nodes in the Subscribers OPC UA AddressSpace. The configuration for such a dispatching is called SubscribedDataSet.

A Security Key Service (SKS) provides keys for message security that can be used by the Publisher to sign and encrypt NetworkMessages and by the Subscriber to verify the signature of NetworkMessages and to decrypt them.

The SKS is responsible for managing the keys used to publish or consume PubSub NetworkMessages. Separate keys are associated with each SecurityGroupId in the system. The GetSecurityKeys Method exposed by the SKS shall be called to receive necessary key material for a SecurityGroupId. GetSecurityKeys can return more than one key. In this case the next key can be used when the current key is outdated without calling GetSecurityKeys for every key needed. The PubSubKeyServiceType defined in 8.2 specifies the GetSecurityKeys Method.

The GetSecurityKeys Method can be implemented by a Publisher or by a central SKS. In both cases, the well-known NodeIds for the PublishSubscribe Object and the related GetSecurityKeys Method are used to call the GetSecurityKeys Method. The PublishSubscribe Object is defined in 8.4.

The SetSecurityKeys Method is typically used by a central SKS to push the security keys for a SecurityGroup into a Publisher or Subscriber. The Method is exposed by Publishers or Subscribers that have no OPC UA Client functionality. The Method is part of the PublishSubscribeType defined in 9.1.3.2.

The SKS is the entity with knowledge of SecurityGroups and it maintains a mapping between Roles and SecurityGroups. The related User Authorization model is defined in OPC 10000-3. The User Authorization model defines the mapping of identities to Roles and the mechanism to set Permissions for Roles on a Node. The Permissions on a SecurityGroup Object is used to determine if a Role has access to the keys for the SecurityGroup.

An example for setting up a SecurityGroup and the configuration of affected Publishers and Subscribers is shown in Figure 9.

image012.png

Figure 9 – SecurityGroup Management Sequence

To secure NetworkMessages, the NetworkMessages must be secured with keys provided in the context of a SecurityGroup. A SecurityGroup is created on a SKS using the Method AddSecurityGroup.

To limit access to the SecurityGroup and therefore to the security keys, Permissions must be set on the SecurityGroup Object. This requires the management of Roles and Permissions in the SKS.

To set the SecurityGroup relation on the Publishers and Subscribers, the SecurityGroupId and the SKS EndpointDescriptions are configured in a PubSub groups.

The Publisher or Subscriber use keys provided by an SKS to secure messages exchanged via the Message Oriented Middleware. The handshake to pull the keys from a SKS is shown in Figure 10. The handshake to push the keys from a SKS to Publishers and Subscribers is shown in Figure 11.

image013.png

Figure 10 – Handshake used to pull keys from SKS

To pull keys, the Publisher or Subscriber creates an encrypted connection and provides credentials that allow it access to the SecurityGroup. Then it passes the identifier of the SecurityGroup to the GetSecurityKeys Method that verifies the identity and returns the keys used to secure messages for the PubSubGroup. The GetSecurityKeys Method is defined in 8.4.

The access to the GetSecurityKeys Method may use SessionlessInvoke Service calls. These calls typically use an Access Token that is retrieved from an Authorization Service. Both concepts are defined in OPC 10000-4.

image014.png

Figure 11 – Handshake used to push keys to Publishers and Subscribers

To push keys, the SKS creates an encrypted connection to a Publisher or Subscriber and provides credentials that allow it to provide keys for a SecurityGroup. Then it passes the identifier of the SecurityGroup and the keys used to secure messages for the SecurityGroup to the SetSecurityKeys Method. The SetSecurityKeys Method is defined in 9.1.3.3.

Access to the SKS can be managed by an Authorization Service as shown in Figure 12.

image015.png

Figure 12 – Handshake with a Security Key Service

The SKS is a Server that exposes a Method called GetSecurityKeys. The Access Token is used to determine if the calling application is allowed to access the keys. One way to do this would be to check the Permissions assigned to the SecurityGroup Object identified by the GetSecurityKeys Method arguments. Publishers and Subscribers can request keys if the Access Token they provide is mapped to Roles that have been granted Permission to Browse the SecurityGroup Object.

Message Oriented Middleware as used in this specification is any infrastructure supporting sending and receiving NetworkMessages between distributed applications. OPC UA does not define a Message Oriented Middleware, rather it uses protocols that allow connecting, sending and receiving data. The transport protocol mappings for PubSub are described in 7.3.

This part describes two general types of Message Oriented Middleware to cover a large number of use cases. The two types, broker-less and broker-based middleware are described in 5.4.4.2 and 5.4.4.3.

With this option, OPC UA PubSub relies on the network infrastructure to deliver NetworkMessages to one or more receivers. Network devices – like network routers, switches, or bridges – are typically used for this purpose.

One example is a switched network and the use of UDP with unicast or multicast messages shown in Figure 13.

image016.png

Figure 13 – PubSub using network infrastructure

Advantages of this model include:

Figure 14 depicts the applications, entities and messages involved in peer to peer communication using UDP as a protocol that does not require a Broker.

image017.png

Figure 14 – UDP Multicast Overview

The PublishSubscribe Object contains a connection Object for each address like an IP multicast address. The connection can have one or more groups with DataSetWriters. A group can publish DataSets at the defined publishing interval.

In each publishing interval, a DataSet is collected for a PublishedDataSet which can be a list of sampled data items in the Publisher OPC UA Address Space. For each DataSet a DataSetMessage is created. The DataSetMessages are sent in a NetworkMessage to the IP multicast address.

OPC UA Applications like HMI applications would use the values of the DataSetMessage that they are interested in.

An OPC UA Application that maps data fields from UADP DataSetMessages to internal Variables can be configured through the DataSetReader Object and dispatcher in the Subscriber. The configuration of a DataSetReader defines how to decode the DataSetMessage to a DataSet. The SubscribedDataSet defines which field in the DataSet is mapped to which Variable in the OPC UA Application.

With OPC UA UDP there is no guarantee of timeliness, delivery, ordering, or duplicate protection. The sequence numbers in DataSetMessages provide a solution for ordering and duplicate detection. The reliability is improved by the option to send the complete DataSet in every DataSetMessage or with the option to repeat NetworkMessages.

Other transport protocol mappings used with the broker-less model could provide guarantee of timeliness, delivery, ordering, or duplicate protection.

This option assumes a messaging Broker in the middle as shown in Figure 15. No application is speaking directly to other applications. All the communication is passed through the Broker. The Broker routes the NetworkMessages to the right applications based on business criteria ("queue name", "routing key", "topic" etc.) rather than on physical topology (IP addresses, host names).

image018.png

Figure 15 – PubSub using broker

Advantages of this model (partly depending on used Broker and its configuration) include:

In addition, the Broker model is to some extent resistant to the application failure. So, if the application is buggy and prone to failure, the NetworkMessages that are already in the Broker will be retained even if the application fails.

Figure 16 depicts the applications, entities and messages involved in typical communication scenarios with a Broker. It requires use of messaging protocols that a Broker understands, like AMQP defined in ISO/IEC 19464:2014 or MQTT defined in ISO/IEC 20922:2016. In this model the Message Oriented Middleware will be a Broker that relays NetworkMessages from Publishers to Subscribers. The Broker may also be able to queue messages and send the same message to multiple Subscribers.

Note that the Broker functionality is outside the scope of this specification. In terms of the messaging protocols, the Broker is a messaging server (the OPC UA Publisher and the OPC UA Subscriber are messaging clients). The messaging protocols define how to connect to a messaging server and what fields in a message influence the Broker functionality.

image019.png

Figure 16 – Broker Overview

An OPC UA Publisher that publishes data may be configured through the PubSub configuration model. It contains a connection Object per Broker. The Broker is configured through an URL in the connection. The connection can have one or more groups which identity specific queues or topics. Each group may have one or more DataSetWriters that format a DataSet as required for the messaging protocol. A DataSet can be collected from a list of Event fields and/or selected Variables. Such a configuration is called PublishedDataSet.

Each DataSet is sent as a separate DataSetMessage serialized with a format that depends on the DataSetWriter. One DataSetMessage format is the JSON message mapping which represents the DataSet in a format which Subscribers can understand without knowledge of OPC UA. Another DataSetMessage format is the UADP message mapping.

Message confidentiality and integrity with the Broker based communication model can be ensured at two levels:

The Broker level security requires all Publishers and Subscribers to have credentials that grant them access to the necessary queue or topic. In addition, all communication with the Broker uses transport level security to ensure confidentiality. The security parameters are specified on the connection and group.

The message security provided by the Publisher is only defined for the UADP message mapping.