Measuring DevOps success

A key part of your DevOps programme, and instrumental in its success, is how DevOps and your applications are measured. If this measurement is done well, it will provide accurate ROI, help you to manage progress towards your DevOps goals, and be a great help in evangelising DevOps success.

 

Metrics supporting the business

Well-defined and relevant metrics also help to focus the organisation on delivering the right business outcomes. They make sure that every change implemented is a step towards achieving a defined business goal. However, metrics used badly can stall or even reverse the cultural changes in progress. This will delay your DevOps ambitions and disrupt your long term goals. Bad metrics can for example be ones that put different teams in direct conflict – which is the very state we are trying to move away from.

For the most accurate information on your progress, start measuring from day 1 of your DevOps journey – or earlier, if possible. This allows you to compare your current state with your starting point throughout the entire process.

 

Measuring issue resolution

A good metric to start measuring might be MTTR – Mean Time To Recovery. This measures how long it takes for an issue to be resolved in production. High MTTR generally translates to long downtime for the customer or business users, which results in loss of revenue and direct business impact. Low MTTR corresponds with high availability, fast responsiveness and minimal customer impact.

With low automation, high MTTR is generally down to the manual nature of a lot of the work required for releases. When a manual release goes wrong, a manual rollback or hot fix is often rushed in. This change can bring increased risk, as the rollback process is not likely to be thoroughly tested under the added time pressure. If a quick hot fix is needed, the developer will write the code quickly and try to get as much testing as possible done before the hot fix deadline. But without continuous integration and continuous delivery, the full set of tests (including regression tests) will inevitably not get run.

Reducing MTTR directly translates to better reliability and better customer service. It creates cohesion within the entire application team, as there is no conflicting goal between any groups.

 

Encouraging innovation

Something which goes hand in hand with measuring MTTR is not being afraid to innovate and make mistakes. With a low MTTR we have greater confidence that any problem can be resolved quickly, so the impact of issues in production is not so high. This means that each issue becomes less critical. This attitude relies heavily on underlying cultural change to encourage innovation and not focus the blame on incidents.

When creating goals and metrics for tracking progress, the final key characteristic I want to highlight is the creation of goals that are applicable to each stakeholder. This means that everyone can take ownership for achieving them. Having individual goals for each team member should encourage success, as everyone has a vested interest.

 

What works for your business?

In this blog, I have only scratched the surface of the vast topic of metrics and measurement in DevOps success, but I will be discussing this more in future blogs. Please let me know if you agree or disagree, and if you have any other useful metrics that you have seen used successfully – or badly, for that matter!

I have touched on one particular metric here, which is useful for some organisations adopting DevOps. Our recent white paper goes into much more detail about KPIs for DevOps success from a broader perspective, including tips and tricks to build a KPI framework suited to your organisation.

The following two tabs change content below.

Bronwyn Davies

Bronwyn Davies is a senior DevOps consultant at Sandhata Technologies. She brings more than 12 years' experience in application development to help clients achieve their DevOps ambitions. With in-depth understanding of the financial industry, she is able to draw on her experiences and knowledge to provide best practise guidance and practical tips to organisations looking to reinvent their application development processes.