Utility Network Migration

Utility companies are some of the most asset-intensive and complex industries, requiring a well-monitored and controlled infrastructure to produce a service to customers. Achieving modernization goals for a utility requires the large-scale integration of energy sources and technologies. It also involves a transformation of advanced telecommunications networks. A modern electric utility must combine two rapidly evolving large system infrastructures: electric grids and telecom networks.

IT vs. OT: What’s the difference?

Utility companies have traditionally divided their network needs into two separate organizations:

Operational technology (OT) helps a utility manage and obtain a specific line of business goals. OT frameworks consist of the physical grids, monitoring systems, automated Supervisory Control and Data Acquisition (SCADA) devices, and other equipment, including communications equipment. These networks include devices, sensors, and software that identify or trigger a change by monitoring or controlling physical devices, processes, and events in the enterprise. More complex, IP-based networked systems can replace legacy or proprietary OT systems. Maintaining separate technology stacks, protocols, standards, and organizational units can impede innovation for a utility.

Information technology (IT) helps a utility enable more efficient functional capabilities of the enterprise such as finance, customer support, human resources, etc. Systems built upon commercially available equipment, services, and best practices help contain IT costs while increasing efficiency.

IT/OT Convergence

A convergence of IT and OT offers clear advantages and significant opportunities to drive business process improvement for a utility company. The network must constitute a well-coordinated, maintainable, stable, and standard-based infrastructure delivering a predictable and secure communication service for the operational mission-critical applications of the power system. The evolution of the power grid further requires the network to be extensible to future requirements while assuring the optimal operation of legacy systems during the long phase of migration.

Many utilities must upgrade the OT infrastructure to enable their systems to communicate and converge their previously separate IT and OT. This isn’t an easy task.

More complex, networked devices are now replacing legacy hard-wired or proprietary systems.   A fiber optic-based gigabit backhaul for a distribution network is the primary building block of the architecture. Network services such as switched Gigabit Ethernet and MPLS are well-understood technologies, but they represent a significant migration for legacy OT networks.  A utility can leverage broadband power lines to create an easy-to-deploy multi-megabit Ethernet infrastructure.   In some cases, a utility will leverage wireless mesh systems to provide highly secure IP networks.

A modern utility’s telecommunications network must consider these requirements:

1. High Availability: With a minimum uptime of three-nines (99.9%), the expectation for availability increases to four-nines (99.99%) and five-nines (99.9995%) as dependency on the network grows. The network must consider high operational service availability in harsh conditions, as well as power autonomy for reliable communications in case of a power outage.
2. Latency: Critical applications involving safety require low latency (< 6 milliseconds round trip delay) network. Protection and control services require low latency and low delay variation.
3. Security: A utility can be considered a high-value attack surface for hackers. It increases exposure to unwanted attempts to disable critical infrastructure.
4. Coverage: The need for ubiquitous coverage as a significant part of utility infrastructure resides in rural areas for customers and increasingly distributed energy sources such as solar or wind turbines.
5. Scale: While the throughput is generally low compared with other applications, the number of devices requiring similar functionality and encryption only increases the magnitude of the scale of the network.
6. Traffic Prioritization: For critical monitoring and control (SCADA), traffic prioritization is needed.
7. Deterministic Operation: Keeping tight control of time and path across the network and delivering fast restoration.
8. Field Complexity: The complexity of the substation telecom network at a level compatible with field-worker skills, as well as the lifecycle stability for network components installed at substations to avoid continual upgrades and updates disturbing critical applications.

Network Migration Challenges

Leveraging IT network design for an appropriate OT solution involves more than just assembling blocks of existing technology. It is critical for implementors to have a proper understanding of regulatory requirements, applications, constraints, their evolution in time, and following adequate migration paths to avoid any loss of service. Some high-level considerations when making decisions include:

• Leveraging mature technologies to allow a utility complete control of its network
• Products adapted to harsh electromagnetic and environmental conditions
• Reliable and secure building blocks

Our Client’s Scope: A Multi-site, Multi-year Project

A large utility company servicing electricity and natural gas to millions of people over a multi-state region has a network migration requirement to address upgrades to substations in specific locations. This project will cover hundreds of sites over many years, requiring upgraded routing and switching equipment across different states. The project is managed through a site upgrade/completion schedule to support planning, designing, testing, and implementing enterprise network solutions to a new architecture. The client asked TSG to create a proposal, team structure, and plan to complete this project to support their requirements.

Project Engagement

TSG Managed Solutions

Our client looked for a partner that understood the complexities of large-scale network design, utility requirements, substation lifecycle, Route/Switch Configuration, OSPF, BGP, cybersecurity, and access controls. After spending time with the client’s team to understand their needs and craft a comprehensive solution, the TSG Managed Solutions Team took responsibility for the client’s IT/OT network migration project. Leveraging our scoping process, we learned the engineers need a separate background check completed due to NRC CP compliance. Any delays to the background check can impact initial training and productivity.

Discovery and Analysis: Gaps in Team Roles

The Select Group’s experience of supporting our own teams across many market verticals has allowed us to refine an approach for working with our clients to build successful teams.

Our discovery phase consists of a small group focused on understanding the project roles and responsibilities, teamwork, and culture. This preliminary phase allows our team to properly scope and scale the solution and provide recommendations. While no two clients or projects are the same, their teams have similar functional roles, and our discovery phase helps us identify gaps to help the client be successful.

The following is a summary of roles that are important on most projects:

Domain Expert: A knowledgeable, skilled person capable of solving problems in a specific area related to the project’s requirements. They must be willing to communicate this knowledge and commit significant time to the project’s success. This individual might be considered the visionary leader for the project, and the team’s job is to turn these ideas into reality. It could be a product owner, CTO, or other roles as defined by the company.

Knowledge Leader: A person capable of designing, building, and testing a specific project, such as an architect or technical lead. They determine which tools, systems, languages, processes, etc., are required to build the project. They also help establish reasonable methods to handle facts and rules and decide how to represent them in the project.

Implementor: These individuals are responsible for the actual day-to-day building of the project. Depending on the project, this role could be the software developer, DBA, network engineers, full-stack developers, customer support representative, etc.

Project Manager: The expected leader of the project, responsible for keeping it on track.  As the expected leader of the project, the Project Manager must make sure that all deliverables and milestones are met. They are responsible for keeping the project on track and interacting with the expert(s), knowledge leader(s), implementor(s), and stakeholders.

Stakeholders: The people who utilize the outcome of the project. They should not only confident in the project performance, but also feel comfortable using it.

Proposal: A Smooth, Gradual Network Migration

The proposed solution emphasizes the importance of smooth and gradual network migration of network services. Our planning process identified the different capabilities required to create a complete team for the client:

1. 1x Project Manager
2. 11x Network Engineers

Our rapid fulfillment of an entire team and deliverables-based approach allows us to stay engaged at scale with our clients. Our process begins with appointing a long-term engagement manager to be the single point of contact, ensuring service-level agreements (SLAs), deliverables, and milestones are met. The engagement manager also identifies and manages the project’s scope as dictated in the SOW, including all financial performances like invoicing and budget tracking and service level agreements, in addition to team performance issues.

Network Migration Results

Increased Deployment Efficiency

The client’s decision to partner with The Select Group delivered the network migration with significant efficiency increases. It allowed our client to meet their overall internal business objectives.

Benefits to the client included:

• An improved network migration delivery schedule.
• Leverage industry expertise to help perform highly reliable upgrades.
• Reduced time spent hiring and providing personnel oversight.
• Improved business agility for the client.

Looking to advance your networks?

Our experts are here to help.