Stable networks need it.
If you want to have a stable network, you need to have an effective change management process. Why bother? Changes are the most common source of instability in your network. Sure hardware fails now and again, but in the end the most common source of network failures is the hands of those that are running the network.
As an example, we recently completed a large IP telephony installation at one of our customers. They system had be fully tested and was in the process of being handed off to the customer’s operations team. As I was heading to the airport, I get a call telling me that they’d just lost all of their telephone circuits. As our team dug into the problem, we discovered that one of the customer’s network engineers had just made what he thought was an innocuous change in the system configuration of each of the voice gateways. The result, the gateways disassociated themselves with the network and stopped taking calls. The result, the customer was down, everyone lost time troubleshooting something that didn’t need to happen.
So what went wrong here? A change happened, it seemed simple, but it wasn’t. The customer has a change management process and technically it was followed. In this case it’s purely a “process oriented” change management process. A ticket was written, in this case as a blanket order to cover a multitude of changes associated with this particular project. The change itself was never reviewed. After all, it was “just housekeeping”. Without review the individual making the change had no idea what could happen.
To get value from the change management process, you need to review the change with those that have a stake in the results of the change and who have sufficient understanding of the change itself that they can provide oversight and peer review.
These questions have to be answered:
- Did I get the proposed change right?
- Am I sure I understand who this change matters to? Have they been consulted?
Asking “Did I get it right?”, presupposes that you’ve actually planned out the change and the steps and proposed configuration changes are in a form that others can look at and judge the quality of. It also presupposes that it’s been determined that this is a necessary and proper change to make in the network. Getting the change right is only half the task.
Knowing who cares about the change is the first step in determining when and if the change should be made. Once those who care about the change have been consulted a priority can be set and the change can be scheduled at the appropriate time.
Working collaboratively and honoring the process are the first steps in building a successful change management process.
Can change management get out of hand? Of course. I’ve seen countless instances in large organizations where changes were unnecessarily delayed because those involved in the process were not able to effectively assess the risks associated with the change. In some cases too many people become involved in the process. The usual result is changes are delayed, schedules slip, costs increase and the overall benefits of the changes that are in the queue are delayed.
The solution is to have the right people in the room, who understand the impact and can assess both the technical and business risk of the change. Getting the individuals with the right level of authority and skill in the room (or on the call) can be challenging, but the results will speak for themselves. It’s to use a good group process that ensures everyone gets heard, and no one is railroaded into accepting a change. It has to be a process that’s the right size for your particular organization. Too light, and you’re going to miss mistakes that will take down your network. Too heavy and not only will you never get anything done, you’ll demoralize your team and risk losing hard won talent.
