Technique 7.3: Assessing the Creation of Value

Measures for Individuals and Teams

Now that we understand what we are trying to create (content standard) and how we are going to create it (structured workflow), we are ready to develop a measurement model to assess how we are doing. We want to know how we are doing as individuals as well as teams. We have found the approach described in the book The Balanced Scorecard by Norton and Kaplan to be very helpful. It describes a number of important concepts that we have embraced in the KCS Performance Assessment model.

• Link individual goals to departmental and organizational goals to help people see how their performance is related to the bigger picture.
• Look at performance from multiple points of view. The typical scorecard considers the key stakeholders: customers, employees, and the business.
• Distinguish leading indicators (activities) from lagging indicators (outcomes).

In this section, we show examples of how we apply the scorecard methodology to establish and maintain appropriate measures for both individuals and teams:

• Leading indicators (activities) and lagging indicators (outcomes): a very important distinction
• Triangulation—looking at things from at least three perspectives to see who is creating value
• Radar charts—a value footprint: a presentation/visualization technique
• Sample scorecards for knowledge workers and team leaders (managers)

Our first important concept distinguishes between activities and outcomes. If we put goals on activities (leading indicators), we will get what we ask for. Unfortunately, the activity by itself is not an indicator of value nor does it necessarily lead to the outcome we are after. For example, if we set a goal for each knowledge worker to create ten KCS articles per month, we will get ten articles a month. However, we will find that these articles are often created on the last few days of the month and they contain little or no valuable information (things like "fixed customer problem"). They were created to meet the goal, without consideration of the desired outcome. Goals on activities seldom, if ever, generate the desired outcome. In fact, in a knowledge management environment, if we put goals on activities it will corrupt the knowledge base. In the example of a goal on article creation, it's not just that the useless articles are a waste of time - their presence in the knowledge base is actually damaging to the health of the whole KCS system. This is why the distinction between activities and outcomes is so important, and why we say often: do not put goals on activities!

In our example above, the outcome we want is articles that are findable and usable by a target audience. We need to do the "create" activity in the context of the outcome. This reinforces the KCS Principle of Create Value: work tasks, think big picture. The outcome needs to be the focus, not the activity.

Goals placed on activities will corrupt the knowledge base.

Putting goals on activities will:

• Create unwanted results
• Destroy the value of the measure as an indicator of behavior
• Distract people from the real objective
• Relieve people from using judgment
• Make leadership look dull
• Disenfranchise people

A very helpful concept from The Balanced Scorecard distinguishes performance drivers (motivators— covered in the Leadership section) from leading indicators (activities) from lagging indicators (the results or outcomes). While each of these three elements is important, the role each plays in the measurement system is different. Making a distinction between them is crucial.

We need to pay attention to the trends of the activities and their correlation to the outcomes.

• Are the activity measures heading in the right direction?
• How rapidly are they changing?
• Do knowledge workers have timely visibility to their performance indicators?

While the distinction between activity and outcome measures is critical, we find people struggle with identifying which indicators are activities (leading indicators) and which are outcomes (lagging). Here are some helpful ways to test an indicator:

• Easy to measure and easy to manipulate or game — it is probably an activity (do not put a goal on it)
• Hard to measure and hard to manipulate or game — it is probably an outcome
• Only measurable after the fact (when the event completed)— it is probably an outcome

Whenever we are having a discussion about an indicator or measure, we must be clear as to whether it is an activity or an outcome. See the Metrics Matrix section for examples of activity and outcome metrics.

Triangulation—Who is Creating Value?

The distinction between activities and outcomes is only part of the picture. Effective performance assessment in KCS comes from the integration of multiple perspectives. Because there is no one measure that indicates value creation, we assess it by correlating at least three different perspectives. The basic model includes trends in activities (trend over time), key outcomes (measured against goals), and the KCS Content Standard Checklist (discussed in Content Health). These three perspectives consider measures that are both objective (quantifiable) and subjective (qualitative) to assess value creation by individuals and teams.

The concept of triangulation reflects the idea that the creation of value cannot be directly measured or counted—value is intangible. We believe the best way to assess the creation of value is through a process of triangulation. As with GPS (global positioning system) devices that calculate our location on the earth based on input from multiple satellites, an effective performance assessment model incorporates multiple views to assess the creation of value.

We offer, as an example, a collection of measures to create an initial assessment model. Every organization must be thoughtful about developing its own set of metrics that align with their organization's goals (documented in the strategic framework).

The choice of measures for KCS must focus on the attributes that create value for the organization. The integration of the following dimensions creates a comprehensive view of performance, which in turn gives us confidence in assessing who is creating value and who might benefit from some help from a Coach.

Aligning to Business Objectives: Balanced Scorecard Example

The balanced scorecard format helps ensure that we have encompassed the full range of objectives. We recommend referring to the book for guidance in the process of scorecard creation.

Make Trends Visible to the Knowledge Workers

Consider a driving analogy: we want to go from San Francisco to Yosemite National Park. We could reasonably expect to make the 180-mile drive in three to four hours at an average speed of 55 miles per hour. Our desired outcome is to reach Yosemite in a reasonable period, but we will not know if we have been successful until we arrive. What would we need for the trip? We need a car, a driver's license, and some gas, but a successful trip requires that we also pay attention to many other factors (leading indicators) along the way. Because we would like to average 55 mph, we want to pay attention to how fast we are going. Because we have determined three to four hours is the acceptable period, we want to be aware of the passage of time at different speeds and how much gas we have in order to avoid refueling delays.

The dashboard in the car is very helpful in informing us about the enabling factors for a successful trip. In KCS, the trends in the leading indicators are the dashboard that let the knowledge worker and the organization know the status of the enabling factors. They must be visible to the people who are driving the KCS system: the knowledge workers.

We emphasize this visibility because we have seen multiple organizations implement KCS and not provide the knowledge worker with the feedback they need to adjust their behavior and create optimal outcomes.

Goals for Outcomes, Not Activities!

Because leading indicators are quantifiable activities, they are often easier to measure than outcomes. This creates a seemingly irresistible urge to put goals on the activities that are required for the outcome. As we have mentioned a few times, this is... counterproductive.

Consider the trip to Yosemite. If the stated goal were solely maintaining an average of 55 mph, it could be done. But in the absence of understanding the objective (Yosemite) the driver will choose roads that allow him to maintain the average speed regardless of destination. We might end up in Chico! Not that Chico is a bad place; it just is not where we wanted to go.

During the KCS adoption process, we have seen organizations put goals on KCS article creation (everyone should create five KCS articles a week) or KCS article reuse (knowledge workers will be measured on how often they link KCS articles). The goals for these leading indicators may have been met, but the quality of the knowledge base has been seriously compromised. Invalid and duplicate KCS articles are created, because the focus is on the activity, not the outcome. Worse, emphasis can shift to gaming the system rather than generating real value. Inevitably, quality and morale suffer, management looks less competent, and the value of the knowledge is diminished.

However, the trends in the activities (link, modify, create) give us valuable insights about knowledge worker behavior, so long as (we will say it again) there are not goals on the activities. If we put goals on these activities, the trends become meaningless as indicators of behavior and we will have lost valuable insights. The activities are also an early indicator of how effective the organization's leadership has been in describing the purpose and benefits of KCS and inspiring knowledge workers to embrace the Solve Loop practices. If people understand why they are doing it and what is in it for them (WIIFM), the likelihood that they will contribute appropriately is greatly increased. If the knowledge workers know we are trying to get to Yosemite, they will make good judgments about the activities it will take to achieve that outcome. If they don't know where we want to go, they may meet all the activity goals but who knows where we will end up. Refer to the Leadership and Communication practice for more details on communication and motivation.

Knowledge Worker Visibility to Measures

Knowledge worker visibility to measures is a delicate thing. We have conditioned them to expect goals on measures. In some cases if leadership does not put a goal on the activities, the knowledge worker will self impose one - which is as dysfunctional as leadership putting goals on activities. It is critical that knowledge workers get feedback on how they are doing and coaching on how they could improve. The conversation with both coaches and leaders needs to be focused on behaviors and outcomes. While the activity trends and a comparison of activities to peers doing similar work can provide helpful insight to the knowledge worker behavior, the conversation needs to be about their understanding of the Solve Loop and the content standard - not about the numbers.

On the other hand, numbers can have a positive impact when talking about outcomes. A key responsibility of leadership is providing knowledge workers with visibility to the impact their contribution to the knowledge base is having on the outcomes. Quantifying these benefits, using numbers and percent improvement on outcomes or progress toward a goal is very beneficial. If the knowledge workers can not see the impact of their contribution, they will stop contributing.

A Scenario—Examples of KCS Reports

The example to the right is for the first six months of an organization's adoption of KCS.

Radar Charts - Creating a Value Footprint

The scenario with Hector and Kim requires a great deal of data and analysis with multiple charts for multiple people. A leader with a team of fifteen knowledge workers is not likely to have the time to routinely do that level of analysis. Can we make it easier to quickly identify who is creating value and who needs help? For rapid assessment, we use a tool called the Radar Chart.

Creating a Radar Chart requires some thought. First, we want to be sure we get a balanced view. Our key metrics should reflect a balance of:

• Leading and lagging indicators (activities and outcomes)
• Quantity and quality

The leading indicators (activities) are compared to the team average (not a goal) and lagging indicators (outcomes) are compared to the goal.

Secondly, we have to normalize the values to a common scale—for this example we will normalize to 1, so anything less than 1 is not meeting the team average or the goal, and anything greater than 1 is better than the team average or exceeds the goal. In the case of the leading indicators (activities), we will normalize it to the team average (do not put goals on activities). For the lagging indicators (outcomes), we will normalize the goal to 1.

We have to decide what measures to use in the radar chart. Organizations that use radar charts each have their own set of measures, usually defined by the KCS Council team.

For our example we will use Hector and Kim's data from the scenario and we will use the following measures:

• Customer satisfaction index (assuming this is captured at the individual level)
  • Based on post incident closure surveys, the goal is normalized to 1
• Knowledge contribution
  • Content Standard Checklist; based on sampling of articles, the goal is normalized to 1
  • Citations, (peer's use of articles, # per month, 1 = team avg.)
• Process and Operations
  • Incidents handled; the number of incidents handled/month, the team average is normalized to 1
  • Avg TTR: Average time to relief (average minutes to provide relief/answer), the team average is normalized to 1 (note that the individual values for Avg TTR have to be inverted; a shorter TTR than the team average has to have a value greater than 1 and longer is less than 1)
  • Participation rate; the % of cases closed with a resolution linked, the team average is normalized to 1

Once we have decided on the measures we want to include in the radar chart and the calculations for normalizing them to 1 we can plot the chart. Following are Hector and Kim's value footprint. We can see that these charts are much easier to read than the array of graphs we used in the scenario.

 

 

 

 

 

 

By comparing an individual's performance to the team averages for leading indicators and the goals for the lagging indicators, we can quickly see that Hector is creating value, and Kim needs some help. This is a helpful way to view measures so long as we have a balanced view of leading and lagging indicators as well as qualitative and quantitative measures. However, no measure or collection of measures can be meaningful without an understanding of the context in which the individual works and the role of the individual. Assessing the data in the context of the environment is a key responsibility of the team manager. We find that the assessment of value creation in a KCS environment is so different from the transaction and activity based measures we have conditioned first and second line managers to use that the managers need training on how to interpret and use value based measures.

Focus Shift Through Phases of Adoption

During the KCS Adoption, we want to focus on indicators for individual development, adoption of the Solve Loop practices, and adherence to the content standard:

• Learning
  • KCS competency levels across the organization (% of the organization in each of the levels: KCS Candidate, KCS Contributor, KCS Publisher)
  • Time to KCS proficiency (number of days to reach each competency level)
  • Knowledge contribution
    • KCS article creation rate (people are creating KCS articles as they solve problems)
    • KCS article modify rate (people are improving KCS articles as they use them)
    • KCS article reuse rate (people are using KCS articles they find in the knowledge base to solve problems)
    • Knowledge base participation (% of incidents handled using the knowledge base)
    • KCS article rework rate (KCS article flagged as needing attention because it could not be understood or fixed by the person who found it)
    • KCS article cycle time (rate at which KCS articles move through their life cycle)
    • KCS Content Standard Checklist (random sampling of articles)

• Process and Operations

  • Incidents handled, individual (# of incidents handled/month, 1= team average)
  • Average time to relief, individual (average minutes to provide relief/answer, 1 = team average)

As the organization matures and KCS becomes second nature for knowledge workers, we shift our focus from individual measures to a balance of individual and team or collaboration measures:

• Collaboration and teamwork
  • Reputation and peer feedback
  • Invitation rate (number of times invited to collaborate)
  • Opt-in rate (number of times the invitation is accepted)
  • Knowledge contribution—reuse by others (citations)
  • KCS Content Standard Checklist for the team
  • Citations or feedback from customers
  • Customer success on the web (index = of % customers using web first x % success)

Sample KCS lagging indicators:

• Loyalty/satisfaction index (team)
  • Based on surveys, team (post incident and periodic)
  • Retention rate/renewal rate

For a complete list of all the KCS measures the Consortium has considered please see the Appendix - Metrics Matrix

Summary: Performance Assessment

Performance Assessment for KCS represents a departure from traditional management practices. It focuses on collaboration, not competition, and assesses the creation of value, not activity. Job descriptions and expectations must shift to include the capture and maintenance of knowledge in the workflow (the Solve Loop). The measures must reflect the concept of collective ownership of the knowledge base.

Here are the key points to remember:

• Use a license model (KCS Candidate, KCS Contributor, KCS Publisher) to manage and encourage proficiency
• Align individual and department goals to the higher level company goals (strategic framework)
• Distinguish activities from measures
  • Look at trends and performance against team averages for the activities and create goals for the outcomes
• Use both the Content Standard Checklist and PAR to create a comprehensive view
• Be thoughtful about what measures you focus on and create a balanced view
  • Objective and subjective measures
  • Individual and team measures
  • Trends in activities and attainment of goals for outcomes
• Enable timely feedback to the people doing the work
• Conversations with knowledge workers must focus on behavior, process, and understanding, not on the numbers 
• Plan to evolve the measures as the organization matures