Are key performance indicators a true measure?

Some managers love to look solely at numbers when assessing performance, and key performance indicators (KPIs) are right up their street. But how useful are they in the context of software development?

Strictly speaking, KPIs should not be confused with metrics in general. The term should be reserved for factors that really are essential for an organisation's success, and that can be directly influenced by the individual or group being evaluated. Just because some aspect of a job can be measured, it is not necessarily worth measuring.

While it may be tempting to regard simple measures such as the number of lines of code written and debugged per day as KPIs, that could be seen as an early twentieth century perspective that focuses on efficiency rather than effectiveness. In most cases, developers are not the contemporary equivalent of stevedores unloading ships by hand. The role of the modern Australian developer typically has a creative side that is not amenable to such mechanical measurements.

"It's not practical to have KPIs for software developers," said Guy Harrison, chief architect for database solutions at Quest Software. "We don't have any form of KPIs ... for individuals."

The best developers are more like craftsmen than process workers, Harrison said. Creativity is needed to produce highly desirable software, and there is no good way to measure that. If you rated the best developers on function points per day, for example, they would probably get a low score, largely because they are likely to be assigned the most difficult parts of the project.

One problem with KPIs is that they can be self-defeating if they are not truly aligned with overall business objectives. Developers are smart, said Harrison, and will optimise their efforts to whatever you measure. That probably will not give the best outcome if you measure the wrong thing. A key goal is a gelled, highly productive team, and that involves helping each other and sometimes working on foundations that are not amenable to measurement. Unsophisticated measures of individual output may work against that.

Around 10 years ago, software metrics were a hot topic and much effort went into finding which were important, but it turned out that none were, said Harrison. It might be appropriate to use KPIs for routine work such as report writing, but successful software companies are "trying to foster collaboration and excellence."

Shanker Krishnan, general manager, products and services at Mastersoft, has racked up over 20 years in the industry and believes metrics such as function point analysis and lines of code might have been worthwhile in the COBOL days, but are less suitable today.

Owen Batt, technical director and co-founder of Smartpath said, "it's not a smart move" to use KPIs such as lines of code, as it is better to use fewer lines to achieve the same end.

However, "all of our staff members would have some sort of KPI" with associated bonuses or other benefits, he said, but they relate to longer-term targets such as completing a module within an agreed timeframe rather than day-to-day considerations.

Traditional KPIs are not relevant, agreed Greg Brady, general manager operations at VeCommerce. "They're probably not leading to the objective of reaching the [desired] outcomes of the business" and they tend to squash innovation and creativity.

Yet he is a believer in KPIs, as long as they are the right ones. VeCommerce -- which develops speech and telephony software -- focuses on customer benefit and business outcomes, and therefore sets KPIs accordingly. They are negotiated with the customer, and then rolled down to development teams and other groups within the company. A prime example is meeting completion dates, with the opportunity to earn "early achievement bonuses."

Another involves the degree of automation delivered by the products, which is important to VeCommerce's customers. An organisational-level KPI might be that a certain percentage of calls should be handled by the self-service system without resorting to a contact centre agent, and that trickles down to developer KPIs.

The idea of a "software factory" that is highly process driven and amenable to simple metrics is probably appropriate for maintenance and other predictable situations, Krishnan observes, but without predictability there can be no benchmarks, rendering KPIs meaningless.

Straughan Schofield, Tower Software's general manager, product development and support, made the converse point, suggesting that the "management by observation" approach he expects from team leaders and senior developers would be inappropriate for highly specified projects where staff have little or no opportunity to innovate. "I don't think this would work at all."

US and European organisations tend to adopt formulaic approaches simply because "that's the way you do it," he said, whereas Australian companies take a more pragmatic approach.

When Tower's CEO wanted to introduce KPIs across the organisation, Schofield resisted their application to his group. Software development is about innovation, he explains, and that requires the freedom to follow opportunities as they arise. It would be counterproductive to give incentives that work against that principle.

One suggestion was that the QA team could be judged on the number of units of work that they signed off, but Schofield argues that quality testing is more important than raw volume, and he wants QA staff to think about what they should be doing to deliver value to the organisation and its customers.

In place of KPIs, he favours formal lines of communication and administration, with (for example) team leaders and senior developers expected to know who is performing to the expected level.

Ben Wells, chief technical officer of Altium is another executive who questions the applicability of KPIs to the development process.

"At Altium we create complex, highly technical software. Of all the challenges we face, the most difficult is finding familiar metaphors, and designing cool GUIs around these metaphors, for complex technical procedures to make our software user friendly and intuitive," he said.

Altium regards this as part of the developer's job. "Our developers are not given specifications but are given projects characterised by high level problems which our users face or opportunities we can give them," said Wells.

"Our best developers will naturally focus on this level, and will often experiment with the software -- over many iterations -- to find something which meets all of the objectives. I cannot think of any KPI which could measure performance in this area: creativity and ideas are impossible to quantify."

Deadlines

On-time completion is the KPI most commonly mentioned by the executives we interviewed.

Schofield said his developers are expected to estimate how long a job will take, and then they are judged by how well they stick to it. This is important because the rest of the company works around the output of the development team.

While you might expect people to pad their estimates to provide some wriggle room, "most developers are show-offs" and are more likely to underestimate, he said. "People really want to do a good day's work."

Consequently, it is up to managers to ensure that all the necessary work has been factored in, not just the interesting pieces. Once a realistic schedule has been set, "finishing early is seen ... as inaccurate as underestimating [the time needed]," said Schofield, as it can result in a product being later to market than necessary as marketing and other associated activities are planned around the expected completion date.

Quest tracks the estimated time to complete a unit of work against reality, said Harrison, and the variance is fed back into future estimate "in a non-judgemental way". The best developers may take the longest and have the worst estimates because they work on the parts of the project that require the most creativity and are therefore the least predictable.

Krishnan summarises this by saying what is important is to understand individual and team performance, and to use that to plan well.