We Have Automation and AI — Why Should Engineers Still Know Change Management?

My honest take is that good change management skills (I mean knowing how to change a production system safely) has become a lost art. Between infrastructure automation, agentic code generation and an industry that tries to re-invent the same paradigms every two years, the ability to plan and execute a change to a live system has left the industry (at least circa 2026).

This is for the future - documentation on what I would expect an engineer to know and do when planning to change a live system (no matter the size).

Make a Plan and Review It

I was an early adopter of CI/CD and IaC automation and I never want to go back to manual anything, but we have to understand that automation can destroy a system in seconds. All it takes is for us to do the wrong thing and the system can be wiped out with a few simple commands.

Any time you want to make a change (big or small), pause and write down a step-by-step implementation plan. The magic is not so much in what you write - but the act of just writing something down should make you think through the process. This is where you should think about what-if scenarios.

  • What if the index takes too long to build?
  • When is the best time to do this?
  • Will my change impact anything or anyone else?
  • If this goes bad, how will I roll back to a good state?
  • Do I have backups?
  • Do I have permissions and approvals?

Then, if you can, find a friend to review it with you. Or, use AI to review it. Or, put it aside and come back to it with a fresh mind the next day. First drafts of anything are just that - unrefined thoughts.

The nice by-product of this is that it sets you up nicely for the next phase - practicing.

Practice the Plan

Many engineers treat pre-prod as just a formality to get to the production environment. But, I would strongly recommend that when you need to make a change to a live system that you practice the change step-by-step in a non-production environment exactly how you intend to do it in production.

Be honest:

  • Did you start making your change without a written plan?
  • Did you have a rollback prepared in non-prod before you started?
  • Are you doing the change with a similar load/traffic profile on the system as you would in production?
  • Are you watching the system to see what impact your changes have?

Why have pre-prod environments if you aren’t going to use them to validate your change processes?

After you practice, the plan should be refined, solid, and ready to go — with all contingencies mapped out and mitigations defined. Now you are ready to go to production.

Performing the Change

Assuming you have approvals and all the pre-conditions are lined up, there are 4 steps you should always take:

  1. Environment Pre-validation: Make sure the system you are about to change is actually in a known state. There is nothing worse than making a change, going to test it and finding everything is broken. Did you do it? Was it broken before? Did someone else just make another change that impacted yours? You can avoid most of this drama by knowing the state of the system beforehand. Hopefully your system has dashboards that show you how healthy it is, but ultimately I still like doing a manual smoke test myself before touching something in production. However you do it know the state of the system before you change it.

  2. Follow the Plan: Start the change, follow the plan step-by-step. If you had written down things that would make you roll back, then roll back when you get there. If the plan says to do steps 1, 2 and 3 in that order, then do them in that order. It seems so simple, but I could have avoided many late-night scares in my career had people just carefully followed the plan.

  3. Validation: This is the step most engineers skip (because validation is boring), but don’t walk away until you know the change worked. How do you know the change worked? It should have been part of the plan.

  4. Rollback / Validation: Finally, if you have to roll back for any reason (stuff does happen, and distributed systems are not 100% predictable — they run in a physical, imperfect world) you must validate that the system gets back to its original state. Not just on paper (ie, I restored the database) but in practice (the user can still login and do this thing). Rollback plans, even when rehearsed, aren’t 100% bulletproof. I have seen issues where data in a restored database causes the system to fail, or a key rotation gone wrong basically makes rollback impossible and you have to roll forward. Don’t assume that a rollback works - test it before you leave the system.

Wrapping Up (Documentation)

And, when it is all said and done, document what actually happened. Proper change management means keeping an accurate log of all system changes. I have had too many changes where something unexpected happens, an engineer fixes it, and then everyone forgets until a few months later it surfaces again.

It seems like such simple advice, but when you have been up all night because a change went bad, the absolute last thing you want to do is document stuff. My solution I have found (the hard way) is to write down things as they happen during a change (ie, take good notes) and then the following day with a little bit of distance, clarity and rest, I update whatever documentation is needed.

Technologies and methodologies come and go. Good change management doesn’t — it’s one of the few skills that compounds over your career, benefiting both the engineer and the team long after the tools they use have been replaced.