As I’ve mentioned in previous blogs, many companies struggle with getting their metrics right. Examples of these struggles abound but usually fall into three general groups. Last time I talked about collecting too much data. Now I will discuss the second one, poor/misaligned data or lagging metrics.
Each company is different and must decide for itself which metrics, based on which data, will provide a roadmap to success. For example, Continental Airlines focused on cost-cutting in the mid-1990s after coming out from bankruptcy. This focus led to rewarding pilots for saving fuel. Because of that, many pilots flew slower and behind schedule and skimped on air conditioning. The result – valuable business customers ran into the arms of competitors.
Clearly, Continental’s business performance doesn’t center on saving fuel. It centers on filling up as many planes with passengers as it can. This was a costly lesson for the airline. This lesson repeats on different scales across pharmaceutical R&D organizations. Many clinical operations groups focus on “small picture” metrics such as first subject screened and departmental budget targets. The moral of the Continental story is clear: when it comes to metrics, focus on using the right data by keeping the big picture in mind.
Ideal metrics should:
- provide leading indicators
- pair financial and non-financial indicators
- display central tendencies with a companion measure of variability
Leading indicators help personnel understand where the business is going, not just where it’s been. In clinical operations, for example, too many reports focus on delivering cycle-time reports. But during a study what is the point of knowing what has already happened?
Ideally, the reports would center on what would need to happen to hit agreed-upon targets. Instead of (or in addition to) cycle times for completed milestones, reports should provide cycle times in the form of “X days and counting” for open items, and add alarms for when days in progress passes a certain point. This will focus the users on what needs completing rather than what has already happened.
Pairing financial and non-financial indicators is important. It ensures a company’s action, based on a particular metric, is (literally) paying off. In an effort to cut costs, Continental Airlines failed to do this. Instead, they sacrificed significant revenue and market share of business travelers.
It wouldn’t hurt if clinical operations groups started to use this approach when implementing process improvement initiatives. How else will they (or management) know if it was really worth it? If proper sources of data are not used, then having “ideal” metrics in place will not be helpful. Completeness and accuracy of data are crucial. Equally important is the place you are getting your data from.
Case in point, a company by the name of InterFirst Mortgage was getting an inaccurate read on “turn time” (the time its staff took to close a loan). This is because it asked its sales force, not its customers (mortgage brokers), for information. The company later discovered through an online feedback tool that its turn time was as good as, if not better than, its competition.
InterFirst adjusted its sources of data and got a more accurate read from its valued customers. They thus avoided focusing on what was a non-issue and focused instead on improving efficiencies elsewhere.
In another example, a company called eePulse once consulted with a software company that had measured the impact of a merger on employee morale. They concluded that it was a smooth transition and a major success in the eyes of employees. Upon closer examination, it turned out the post-merger data came from workers who joined the company after the merger. Most of the employees present during the merger had left the company.
Regardless of what you’re measuring, without authoritative data and information sources (especially given the increased use of outsourcing these days), the interpretation of results can be easily compromised.