Beware of vanity metrics in digital transformation
Not all data is equal. Do you know how to measure the true business impact your digital transformation efforts and investments?
Meet Billy Smith. Billy’s 6 month ERP implementation just went live this month and a deep sigh of relief was in the air. 7,159 people just attended a training session in the past week and the project even came in on budget. Even the initial in-app CSAT survey scores are coming back positive. Billy reaches out for a celebratory glass of Moët when a catastrophic realization bubbles up amidst the celebration. Billy has no idea whether the project actually worked. It turns out, Billy is the latest victim of vanity metrics.
What are vanity metrics?
Vanity metrics exist in plain sight – buried in reports, dashboards and even OKRs. Look closely, and you’ll be able to identify the tell-tale signs, the gorgeous looking metric highlighted in bold, may not be that good looking after all.
Vanity metrics aren’t actionable
The heart of data-driven teams is the understanding that if something can’t be measured, it can’t be affected. The days of running blind off of gut feelings should be as far behind us as faxing sales contracts and cashing checks.
Whether you’re building WalkMe solutions, or sending an action-oriented email, we can all take a page of marketing optimization wisdom by thinking about a call to action (or CTA.) v Pop quiz – what metric would you use to optimize and report on? Well, it’s surely not the click – it’s the action. Measure that. If you’re measuring something else. It’s probably a vanity metric.
Vanity metrics look impressive, but might not be repeatable.
Is that dashboard that came pre-loaded in your ERP providing data around how many people logged into your system every month? Good – that information is super critical for understanding whether or not it’s being found and accessed, but it may not tell you whether or not you’ve had any impact on fulfillment times. Perhaps it’s a number that looks great, but if it’s not tied to the work your team is doing and iterating, it might not be the best metric to look at.
If you’re leading a Digital Adoption effort, you’ll want to dig into more than just the reporting you’re getting out of the box with your software, or that stale BI dashboard your team might still be using from 2020, and instead think about what you need to know about your in-platform user behavior to measure success.
The problem with how we measure digital transformation projects.
So what does this have to do with your digital transformation project? Well, if you’re working on one, go take a look at your latest status update. I’ll wait. It’s just us friends here… go ahead and give this article 5 stars if you’re tracking go-live date, SDLC milestones, or maybe training and enablement dates.
Bravo! Well, here’s the thing, all of those data points are absolutely critical in measuring the project, but ask yourself if they’re providing any visibility into your actual transformation. If you’re scratching your head, let’s talk about how value was determined back when your mammoth project was a wee slide in your executive team’s planning off-site.
What organizations really want to do
This one is easy… drive growth and maximize revenue. That’s clear, but how it’s done usually falls into a few common categories.
- Reduce operating costs (improve margins)
- Improve employee satisfaction and engagement (increase productivity)
- Accelerate rate of distribution (maximize earnings)
- Accelerate deal cycles (boost revenue)
That’s why projects that get funded at organizations large and small are often tied to the types of efficiencies that impact the company’s bottom line. The ability to move faster, more efficiently, and with greater satisfaction is the cornerstone of every digital transformation effort. So why do so many transformation projects end up measuring go-live dates and delivery costs?
It’s how we forecast risk and costs
The realization of intended benefits can’t be had if there’s no platform to dole out said benefits. As a result, it’s all hands on deck to get new software in place. This pushes a ton of focus on the service delivery teams and project management teams to ensure new software tools and platforms are ready on time. Attention shifts from benefits to risk, and teams concentrate on what they know best.
- Expected return versus project costs
- Platform and integration costs
- Training costs
- CSAT
What should we measure instead?
If we think about our friend Billy Smith, part of the reason they’re up at night is because they’re caught up in a risk mindset instead of a transformation mindset. So let’s help Billy sleep better on their next project so they can focus on real impact metrics instead of vanity metrics.
Go Live Date → Value Realization Period
System go-live is easy to measure. Did we push to production? Congrats! But that doesn’t inform whether the investments paid off. Identify the motivations behind the system in the first place.
Is your goal to expedite contract signing in order to accelerate deal cycles? Measure that, and start to calculate how long it takes to see the value of the system you mapped out when you first purchased it. Measuring Value Realization Period, or time-to-value, will help ensure the Moët is twice as nice once you open it.
CSAT → Process Abandonment Rate
CSAT is an incredible metric used by almost every organization to measure whether the people that use your system are happy. The funny thing about CSAT is that while it looks like a quantitative measurement because it’s a number, CSAT is actually entirely subjective and therefore easy to measure, but difficult to replicate. More so, it’s less correlated to positive outcomes than ever before.
So instead, measure what your users are actually doing. Are they starting a fulfillment order and stopping midway? Are your sales people generating a proposal but getting confused along the way? This metric gives you a near real-time insight into how successful that shiny new platform is when it comes to driving outcomes so that you can provide help along the way when you need it – not when support tickets start piling up.
Training Participation Rate → Process Initiation Rate
Oftentimes, companies may rely on time-consuming training, webinars, or the more advanced, in-app guidance. Teams will often measure how many people joined, engaged or RSVPd. This is a great way to measure the reach of your training, but it misses the mark when it comes to measuring whether people know what they need to do.
Instead, measure how your employees or customers are actually starting the process they’ve been trained on. Perhaps it’s step one of benefits enrollment, or using a macro in the contact center. Being able to quantify and measure process initiation rate is your signal into what’s working and what’s not.
Are you ready to look past that shiny vanity metric?
As you look ahead to your next transformation project, make sure to examine the data you’re using to validate your investments. Are you measuring outcomes that you have an effect on? Are you connecting those to repeatable and measurable business impact? If you are, you might just have what it takes to harvest the real promise of digital transformation in your organization. Good luck!