Misleading Metrics, A Case Study
…
In the past, I worked for several large software companies, usually in the pre-sales consulting organization.  One of them was particularly keen on metrics… and striving to be “metrics-drivenâ€.  So, as you would expect, we had our objectives tied to Key Performance Indicators.  The most important Key Performance Indicator was for the consultants to spend at least  240 hours of time in customer-facing situations (good idea actually!). The top people in each region were then awarded points which could be exchanged for pretty nice gifts from the company store (no, really they were  nice).  If you won, you could expect to receive @$2,500/year in addition to your normal pay. In general, this was a great concept which incentivizes the consultant to get out of his/her shell to help the people that pay your salary. In my opinion, the KPI was even set at the right amount of  hours (about 50% of total available work time), because it was quite a stretch to spend that much time in customer activities, as we had plenty of paperwork, training and technical work to accomplish in that same period.  Personally, I felt like I had a real advantage in this situation because of the  relationships I had built in the region over the years.  So, to me this was pretty invigorating.
Then, reality came knocking… here’s how it set in:
– There were definitely 1 or 2 consultants who didn’t have a problem fudging their number.  We can categorize this as an integrity concern.
– Every consultant had a different definition of “customer-facing†time – Did prep time count? Time thinking about the meeting? Travel time to and from the meeting?
– Also, the manager was awarded points for total time in the field for all of his/her consultants, so he/she didn’t really have a reason to argue against people who were optimistic about their time spent.
– This information was entered into an sales force automation system (part of CRM) by the consultant. We were kind of biased – right?
– The manager had favorites – Now, let’s be honest. We all have favorites… Of course we do. You can’t just turn off your human nature and stop liking/respecting people that you actually like and respect. This favoritism ensured that tough questions weren’t asked to certain individuals.
So what happened? I busted my hump to make it into the yellow (KPI could be Red, Green Yellow… gotta love the traffic lights). Then, I was raked over the coals for being yellow. To this day, in my opinion, I believe the 2 consultants who won the 1st quarterly contest were absolutely, positively not in front of customers as much as I was. This was a pretty small group, so I had a fairly good idea. Was this demoralizing? Definitely. Did I stop trying as hard? No, but I kept on bringing up the topic on conference calls to the annoyance of everyone (including myself).
So what?
What are my takeaways from this situation given my chosen career in analytics?  Well, I’ve always been attracted to objective subjects (having direct reports and 7 children, makes me painfully aware of this reality every day), because people problems are much more complicated to solve.
– Having a good metric and goal remain valuable, but the people really matter most.
– Beware of bias in a metric at multiple levels in the organization.
– Beware of carrots and sticks based on key metrics. Don’t avoid them, but be certain that proper controls are in place to ensure compliance. Governance can certainly help.
– Ensure that the systems in place to capture the metric have a chance of getting the data without being subject to influence – External/Objective (this is covered in our first blog in greater detail.) Northcraft’s products give you an advantage in this area.
– Transparency – This metric provided cover for the department and gave the appearance of transparency, but the truth was that 2 people were rewarded with incentives that shouldn’t have been given to them.
How does this apply to IT?
Remember these concepts when calculating Availability SLAs… and feel free to call us for help.
If you’re interested…
We’re been noticing through our own analytics that the most popular Northcraft content recently is:
#1: Coffee KPIs – https://www.northcraftanalytics.com/coffee-kpis-for-it/
#2: Service Management Success in Action Conference – https://www.northcraftanalytics.com/success-in-action/
#3: Metrics for Mom’s Minivan – https://www.northcraftanalytics.com/metrics-for-itsm-and-moms-mini-van/