Change is coming to your organization. But things are not as dismal as they first appear when an unforeseen change is not for the better.
Every day networks around the world go down, causing untold losses in productivity and revenues. Despite highly publicized cases of hackers, viruses and worms, the majority of these outages are the result of changes made to network configurations by people within the company.
Industry analysts IDC and Gartner Group estimate that as much as 70 to 80 percent of all changes resulting in downtime or reduced operational capabilities are in fact initiated by people within the organization–and most of those changes are accidental or unintentional.
While some of these inadvertent changes come from general users, the most notable outages occur on production servers where the IT department is the culprit. In the midst of implementing workarounds, quick fixes, or patch updates, changes are often made that negatively impact network performance and unintentionally cause a chain reaction across the network.
Enterprise network infrastructures are becoming exponentially more complex, and budgets are dwindling. IT teams are stretched with increased demands, costs, requirements, and operational risks. Some bigger organizations have the added distraction of a “cold war” between different IT departments.
Establishing processes for implementing changes can be difficult to enforce, and groups within the IT department–such as security and IT operations–sometimes appear to be working toward opposing goals. One group implements a policy, only to have it completely ignored by the other, and soon the situation quickly degrades to finger-pointing and arguing.
Unauthorized changes to a server, router, or switch configuration can have a major impact on the level and quality of IT services. Without a way to know when change occurs (and whether or not it was called for, malicious, intentional, or even internal), IT teams have few options for preventing negative consequences and minimizing damage.
But don’t worry–things are not as dismal as they first appear when an unforeseen change is not for the better.
A need for IT controls
The need for effective change management is driven by two factors: regulatory compliance and the drive toward operational excellence. Regulations are becoming increasingly complex and specific, making compliance a difficult and resource-intensive process. Auditing for compliance with a variety of these new regulations (including Sarbanes-Oxley Act, Grimm-Leach-Bliley, and HIPAA) is a growing business need. Sarbanes-Oxley Section 404 actually requires a company’s independent auditor to sign off on the client’s internal controls.
In most large businesses, critical financial processes run automatically on a vast, complex computing and networking infrastructure. Most executives presume (and maybe even hope) that this infrastructure is a monolithic, unchanging entity, and that once policies are established and the systems are running, everything is fine. In fact, IT operations are surprisingly and alarmingly fluid.
Auditors understand that financial applications reside on infrastructure managed by IT departments, and they look for integrated processes and controls ensuring that the underlying systems are managed responsibly. In an auditor’s view of the world, symptoms of poor service levels typically point to control issues. An ERP system may be running flawlessly, but if nobody can reach it due to a network failure, failed system upgrade, or improperly tested business rule change, the whole system needs to be examined. For today’s organizations to operate effectively, defined service levels must be maintained throughout, not just at the system level.
The change management dilemma
In a recent Meta Group report, Dan Vogel and Homan Farahmand wrote, “Change management is among the most complex and challenging processes Global 2000 organizations face. Indeed, many IT service quality problems can be linked directly to poor change-management procedures.”
In spite of the difficulties associated with change management, many of these organizations have implemented costly, often ineffective change management processes. Few feel their processes are working at an optimum level; a frustration often voiced by companies is that they have solid policies and documented processes in place for change management, but their people don’t consistently follow them. Furthermore, they have no way to really put “teeth” in the process to hold their people accountable to the policy.
Despite these challenges, a growing number of high-performing organizations have effectively implemented change management strategies that translate to their bottom line.
In these organizations, change management works by observing three key practices: rigorous enforcement of change management processes; a “culture of causality;” and making sure that workers involved in IT security adhere to and help enforce change management policies.
Establishing change management processes
Typically, the first question of someone diagnosing an IT problem is “What changed?” With a change management process in place, that question is far easier to answer. Change management is a process made up of people, software, and procedures. When properly followed, the process results in many benefits including increased staff efficiency and reduced server and network device downtime. Change management can also increase security and provide trusted audit data, all of which leads to reduced IT costs.
Change management is critical for maintaining reliable systems. To this end, best organizations are pushing all changes back into the build-and-test phases so that only rare emergency changes are actually performed on production systems. The whole network device change process must become formalized and should incorporate security, testing, and documentation.
Automating the change management process means addressing the six steps in an effective change management process:
1. A change is requested–for example, install a security patch to a Windows XP server.
2. Requested changes are reviewed, the impact assessed, and resources estimated and assigned.
3. Changes are either approved or rejected.
4. If approved, changes are developed and tested in a preproduction environment.
5. Changes are implemented into production.
6. Changes are verified and reconciled by someone else in the organization.
The final step is the critical missing piece in many organizations. In order to effectively manage change, you need to “close the loop”: Conduct a final verification confirming that the requested change was implemented properly, verify that change was implemented on all target systems, and, finally, have the ability to see if the change control process was circumvented.
Without this step, the change management loop remains open-ended, and it’s impossible to tell the difference between authorized, successful changes and unauthorized (or unsuccessful) ones.
The experts agree that reducing service outages from human error through automated processes provide IT savings and a more efficient business. Eighty percent of IT budgets is used to maintain the status quo.
By implementing enforceable change management process, IT gains control of the infrastructure. By gaining visibility in what changed, IT closes the loop on change management and improves availability, improves audit performance, and lowers IT operational costs.
Dwayne Melancon is vice president of Services and Support for Portland, Ore.-based Tripwire, a provider of change monitoring and analysis software.