Perhaps one of the single most important considerations in any modern IT/OT system deployment is networking.  With all the myriad of terminology and hype around terms like IoT, IIoT, Cloud, SaaS, IaaS, Industry 4.0 and IPv6, it is easy to get overwhelmed with the subject matter.  However, there are a few guiding principles that can be applied to IT/OT projects when it comes to networking. The following are a few high-level considerations to help guide the discussion with your IT/OT team or service provider:

Standards

To ensure all participants (e.g. IT/OT team and suppliers) design and implement networking equipment correctly, a good (and necessary) practice is to have or develop a standards document that applies to staff and vendors with respect to system change management and new projects.  As a guide, the document should include at minimum:

• Network Architecture Standards
• Data Standards/Best Practices
• Video Standards/Best Practices
• Voice Standards/Best Practices
• Wireless Standards
• Monitoring requirements
• Hardware specifications
• Protocol Standards
• IP Addressing schema
• VLAN schema
• Structured Cable Standards

This is not intended to be an exhaustive list, but to get your team and/or service provider thinking about Standards.

Architecture/Infrastructure (Design)

The typical best practice for network architecture follows variations of either the recommendations from Purdue Enterprise Reference Architecture (PERA) or ISA-95/ IEC-62443 architecture (see References below and figure above).  In simple terms, both these recommendations follow a layered network approach to separating data flow by function starting from the connection of simple I/O or devices up through to the enterprise systems.  With the ubiquitous use of Ethernet and IP, these layers have started to become more blurred and often several layers are compressed into one in some cases.  The bottom line is – develop an architecture and keep the documentation of this current.

Another key consideration, as part of the architecture, is the performance requirements at each layer within the network.  Consideration needs to be made to ensure time sensitive systems data is not impacted by other less time sensitive data (e.g. consider Quality of Service standards – QOS).  Also, as more and more wireless systems are being deployed, architectures need to scale to support and segregate this traffic and growth.

Monitoring

As the network is considered critical infrastructure and will scale considerably over time, software monitoring tool(s) will be required.  There are protocol standards in place for network monitoring (e.g. SNMP) which integrate with monitoring systems.  Consideration should be made to centralizing the monitoring to provide integrated visualization and notification within the application. Gone are the days when an individual with “tribal knowledge” can run around and monitor these complex systems.  Typical monitoring software packages are considered “reactive” monitoring as they inform users of an issue which may or may not result in downtime. Going forward, software and appliances providing reactive monitoring (e.g. Network Traffic Statistical Trends) to detect issues before there is unplanned downtime will be key to managing the complexity and scale of these network infrastructures.

Hardware

Finally, on the topic of selecting hardware, there are differences between what is typically used in IT versus OT applications.  Typically, core switches for main backbones are rack mounted, in a server room under environmentally controlled systems. On the contrary, Industrial Automation and Control Systems (IACS) generally require fanless, diskless technology and are rail mount in non-environmentally controlled panels (e.g. require wider operating temperature ranges).  Be sure to get your team or service provider to provide the right specifications to ensure the performance requirements are met at the various network layers.  Redundancy and reliability should be considered for the level of uptime required by the system (e.g. 99.999% uptime – Five Nines).  Typically in IT/OT projects uptime is the priority.

Another consideration regarding hardware selection is standardization and reduction of spare parts inventory.  This has an obvious cost advantage and reduces the learning required and complexity of implementation.  In addition, there should be some consideration on the level of configuration necessary before hardware can be deployed to ensure that any spares are either preconfigured or do not require configuration to deploy in the event of unplanned failures in the middle of the night.

Wrap Up

The above information should serve as a quick list of things to consider and discuss with your IT/OT team or services provider and other stakeholders when embarking on initiatives or projects with respect to Networking.  Noticeably absent in this topic of discussion is security, which we will leave for another article as it deserves its own topic.

» Read more from the IT/OT Executive Series

References

ISA95 – Enterprise-Control System Integration
SolarWinds – Network Monitoring Design Philosophy
Cisco – Securely Traversing IACS Data Across the Industrial Demilitarized Zone
Automation.com – Simplifying Automation System Hierarchies

This post was written by Chris Gray, P.Eng. Chris is the VP of Technology Solutions at Cogent Industrial Technologies, a recognized leader in IT/OT solutions and project management.