Who's Interested in Benchmarks?

Besides, how closely do CEOs and CIOs align in their interests in benchmarks? Let's find out using data gathered by Capers Jones in 2013.

You can also download the results presented here as a PDF.

What the Data Looks Like

  • The data set consists of 60 business and IT benchmarks and 10 audiences
  • For each benchmark (e.g. ROI) and each audience type (e.g. CIOs), a level of interest in the benchmark is recorded
  • The interest level scale goes from 1 (no interest) to 10 (extremely interested)
  • There are 10 audiences in the data set: Shareholders, CEOs, CFOs, CIOs, CTOs, Business Managers, Project Managers, Technical Staff, SQA (software quality assurance) Staff, and Clients

Key Questions

Looking at the raw data, a few questions popped up right away:
  • Which audiences are interested (or not interested) in the same benchmarks? 
  • How strong are these shared interests or disinterests?
  • On which benchmarks do we have a strong divergence of interest amongst the groups?
  • How do specific pairs of audiences compare? For example, how do the interests of CEOs overlap with those of CIOs? 

The Approach

To study these questions, I went ahead and:
  • Segmented the benchmarks into 6 groups: Business, Process Maturity, Risk, Quality, Cost, and Delivery benchmarks
  • Used correlation tables to quantify the extent to which different audiences are similar (or different) for each of the benchmark groups
  • Focused on CEOs and CIOs and identified the metrics that each group is interested in, somewhat interested in, or not interested in. Seeing these side by side gives both CIOs and CEOs actionable information on how to work in concert to provide and receive the information they need. 

Figure 1: Benchmark Groups


Let's see what we found. 

Strength of Alignment of Stakeholders' Interests Across All Benchmarks


Figure 2: Similarity and Dissimilarity of Stakeholders' Interests Across All Benchmarks

Figure 2 shows the correlation between interest scores for each pair of stakeholders across all the benchmarks. The oranges and dark yellows indicate high levels of similarity in interest. For example, CFOs and Shareholders (row 3, column 1) are highly similar in the benchmarks they're interested (or not interested) in (correlation = 0.9). In contrast, Business Managers and Tech Staff are highly dissimilar in their interests.

Strength of Alignment of Stakeholders' Interests in Business Benchmarks


Figure 3: Similarity and Dissimilarity of Stakeholders' Interests Across Business Benchmarks

Figure 3 shows the correlation between interest scores for each pair of stakeholders across all the benchmarks in the Business group. The oranges and dark yellows indicate high levels of similarity in interest and disinterest. In contrast, blue and pale blue indicate dissimilarity of interests. The oranges and dark yellows indicate high levels of similarity in interest or disinterest. In contrast, the blue shades indicate dissimilarity – CEOs are quite different from all other stakeholders as seen in row 2 (the same information is also in column 2). CFOs, CIOs, and CTOs are quite similar in their interests (or lack thereof) in the business benchmarks.


Strength of Alignment of Stakeholders' Interests in Delivery Benchmarks


Figure 4: Similarity and Dissimilarity of Stakeholders' Interests in Delivery Benchmarks

Looking for rectangular patches of orange and dark yellow in the correlation scores in figure 4, we see that CEOs, CFOs, and CIOs are quite aligned in their interests in delivery benchmarks. So are project managers, tech staff, and SQA staff.


Strength of Alignment of Stakeholders' Interests in Risk Benchmarks

Figure 5: Similarity and Dissimilarity of Stakeholders' Interests in Risk Benchmarks
It's clear from the swath of red on the top left of figure 5 that shareholders, CEOs, CFOs, CIOs, and CTOs are united int their interest levels in risk benchmarks. Tech staff and clients share some similarity with all other stakeholders.

Strength of Alignment of Stakeholders' Interests in Process Maturity Benchmarks

Figure 6: Similarity and Dissimilarity of Stakeholders' Interests in Process Maturity Benchmarks

It's clear from the swath of red on the top left of figure 6 that shareholders, CEOs, CFOs, and to a large extent, CIOs, are united in their interest in the process maturity benchmarks.


Strength of Alignment of Stakeholders' Interests in Quality Benchmarks

Figure 7: Similarity and Dissimilarity of Stakeholders' Interests in Quality Benchmarks

Once again, the swath of red on the top left of figure 7 shows strong alignment between shareholders, CEOs, CFOs, and to some extent CIOs and CTOs, united in their interest (or lack thereof) in benchmarks on quality. Tech staff are quite different from all other stakeholders as the blue and paler shades indicate along the Tech Staff row and column in figure 7.


Strength of Alignment of Stakeholders' Interests in Cost Benchmarks


Figure 8: Similarity and Dissimilarity of Stakeholders' Interests in Cost Benchmarks
Cost benchmarks provide cases of similarity as well as dissimilarity between the stakeholders as seen in figure 8. Notice in particular that CEOs are quite different from CFOs, CIOs, CTOs, and Business Managers in their interest in cost benchmarks.

We've looked at the interest levels across all stakeholders. Let's now focus on a couple of pairs of stakeholders, comparing CIOs with CEOs and with CFOs.


Comparing CIOs with CEOs and CFOs

(Sorry the print is so small -- it might be easier to read if you download the PDF version).
Figure 9: Comparing CEOs with CIOs

CIOs are sometimes said to not think like CEOs – indeed, this is cited as a reason for why IT has not fulfilled its potential to deliver business value in many organizations. How do the interests of CIOs and CEOs align across our benchmarks?
Quite nicely! There are lots of benchmarks deep into the IT space that both CEOs and CIOs find interesting. They are also nicely aligned on what they both find somewhat interesting. These metrics like EVA are quite specific to project and delivery management – they’re reliable early warning indicators and both CIOs and CEOs should have them on their dashboards.
CIOs can readily get CEOs metrics on application delivery productivity by industry. When stated in terms of function points, this information is available in many of the books written by Capers Jones.


Figure 10: Comparing CFOs with CIOs


Not surprisingly, CIOs and CFOs are even more aligned than CEOs and CIOs. They share a number of interests; and as the prevalence of “X”s show, their interests don’t diverge for the most part.

There are three main questions that prompt further action:
  1. Do some benchmarks that are somewhat interesting merit promotion to “Interesting” or demotion to “Not Interesting” status?
  2. Do some benchmarks that are interesting merit demotion to “Somewhat Interesting” or “Not Interesting”? 
  3. Do some benchmarks that are not interesting merit promotion to “Somewhat Interesting” or “Interesting”? 

The answers to these questions will depend on the operational characteristics of your business, your current dashboard, and the way in which you prefer to manage your domain. Use the benchmarks here as a trigger to detect gaps and make improvements.

Finally, how similar are stakeholders to each other? In other words, do they cluster in any natural way?

Figure 11: How Stakeholders Cluster Based on their Interest in Benchmarks

Figure 11 shows how “far apart” any two stakeholders are. The lower the number, the closer they are to each other in their interest in benchmarks. 
  • Shareholders, CEOs, and CFOs (the green path on the top left) are quite similar. 
  • CIOs are more similar to CTOs and Business Managers. 
  • The red patch on the top right shows that Project Managers, Tech Staff, SQAs and Clients are not similar to any of the other stakeholders. 
  • In particular, it’s surprising to see how much distance there is between Clients and the business.

And that concludes our analysis for now. You can also download the results presented here as a PDF.