Technique 5.10: Content Health Indicators

Knowledge Sampling for the Content Standard Checklist

To complete a Content Standard Checklist spreadsheet, a group of qualified reviewers (usually the KCS Coaches) participate in regular knowledge sampling of articles from the knowledge base. While the articles are selected randomly (and assigned randomly to reviewers, so coaches are not often evaluating their coachee's articles), it is important to be sure to sample articles from each individual.

Here is a typical process:

• Develop a Content Standard Checklist and criteria
• Evaluate a sample of articles
• Calculate the content standard percentage met and develop summary reports
• Provide regular feedback to the knowledge workers on possible content standard areas of focus with comments from the Coach who did the evaluation
• Provide periodic feedback to leadership

The Content Standard Checklist should be used for learning and growth of the knowledge workers. During rollout and training, the frequency of this monitoring should be weekly. It will take more time due to the high number of KCS Candidates (people learning KCS). Once the organization has matured, the frequency is typically monthly and should not consume more than a few hours of time per month per reviewer. Note that what the organization focuses on around the content standard should change over time. The elements for assessment at the beginning of a KCS adoption will be more basic than those things the organization will focus on two years into the knowledge journey.

There are a number of considerations for monitoring knowledge health in organizations. With the Content Standard Checklist, we are concerned with an article's compliance with the content standard. Other areas to consider include the Process Adherence Review (PAR), case documentation and handling, customer interaction, and technical accuracy.  Organizations have various ways to monitor the health and quality of these important elements of the process. 

As organizations reflect on their processes in the Evolve Loop, they are identifying key monitoring elements and ways to integrate monitoring across the processes.  One element that is emerging as critical to monitor on a regular basis is link accuracy. This is part of the PAR, which is also done using a sampling technique and can be integrated into the Content Standard Checklist process.  Assessing link accuracy also shows up in the New vs. Known Analysis.  

Link rates (percentage of cases closed with an article linked) and link accuracy (the article resolves the issue raised in the case) are the key enabling elements for identifying the top issues that are driving support cost (case volume) and user disruption. In order to provide credible and actionable input to product management and development about the highest impact issues, we need to have link rates that exceed 60% and link accuracy that exceeds 90%.  Link accuracy is more important than link rate.       

Reviewing Articles Through Use

The KCS Principle of demand driven and the Core Concept of collective ownership combine to create efficiency.  The idea that people feel a sense of responsibility for the quality of the articles they interact with is critical. The cost and delay of the alternative: that someone else owns article quality and that it is someone else's responsibility to review it, is prohibitive. This sense of collective responsibility is reinforced through coaching, the competency/licensing program, communications from the leaders, the performance assessment program, and the recognition programs. The new hero in the organization is the person who creates value through their contribution to the knowledge base, not the person who knows the most and has the longest line outside their cube.

Feedback to the Knowledge Worker

Knowledge workers must have visibility to their Content Standard Checklist evaluations so they understand where to self-correct. Again, this is as part of a conversation about understanding and behavior - not about a percentage or number.  Content Standard Checklist results are also a key tool for the Coaches as it helps them identify opportunities for learning and growth.  

Article Value Based on Reuse 

The value of any particular article can be measured by the number of times it is used to resolve an issue. If we are linking articles to incidents, we can easily calculate the reuse count.  As we expand into the Build Proficiency phase of the KCS transformation, measuring the reuse of articles becomes much more difficult because customers using the article through self-service do not link articles to incidents nor do they show much interest in answering the oft-asked question, "Was this article helpful?"  To assess the value of individual articles in a self-service model, we have to infer value based on a number of factors.

A few Consortium Members have developed article value calculators that take into account the following:

• Page views
• Internal links
• Customer feedback (Member experience indicates that customers provide feedback on a tiny percent of articles viewed: 1-2%)

Article Value Based on Reference 

The second perspective is the value of the collection of articles.  Even though a specific article may not be the resolution to the issue, an article about a similar issue may provide some insight or remind us of an approach or diagnostic technique that we know but had not thought about. The frequency of reference is extremely valuable and hard to measure. 

The Value of the Collection of Articles 

The indicators for the value of the collection of content can be calculated based on the rate of customers' use and success with self-service.  More specifically, support organizations often look at the subset of the self-service success rate that represents issues for which the customer would have opened an incident had they not found an answer through self-service. This is often referred to with the unfortunate vocabulary of  "call avoidance" or "case deflection."   This avoidance or deflection view represents a vendor-centric view of support, not a customer-centric view.  A customer-centric view does not avoid or deflect customers; it promotes customer success through the path of least resistance and greatest success - for the customer!

Getting the Basics Right

To begin with, we examined the basic content requirement—the material that must be included in the KCS article. Respondents chose the following, mostly in the category of "accuracy," as "very important." Responses are listed in priority order:

• Technically accurate and relevant
• Problem and solution description
• Cause of problem
• Complete information
• Quickly found
• Clarity of content
• Valid hyperlinks
• Configuration information
• Vendor's sense of confidence in the answer

Considered "somewhat important," mostly in the category of "editing and format," were:

• Compound vs. single thoughts
• Complete sentences vs. short statements
• Date created
• Correct spelling
• Grammar
• Last modified
• No duplication of information
• Frequency of usage
• Punctuation

Of "least importance," perhaps not surprising in a technical audience, were the attributes:

• Legal disclosures
• Correct trademarks
• Date last used

Impact on Company Image

Most respondents considered editorial format somewhat important. Since the process involved in achieving editorial perfection can be time-consuming and delay access to information, we decided to assess the impact on corporate image of publishing KCS article information at various levels of editorial quality. The results were revealing. The majority of respondents:

• Disagreed with or were neutral to the statement: "I have a lesser image of a company that withholds support information access in order to technically validate it."  (In other words, the majority of respondents did not fault a company for withholding information that was not technically validated.)
• Agreed with the statement: " I have a lesser image of a company that withholds support information access in order to achieve editorial perfection."
• Agreed with the statement: "To gain knowledge faster, I would like an option to select to see support information that has not been fully validated."
• Agreed with the statement: "To gain knowledge faster, I would be willing to take responsibility for using any of the incomplete information should there be mistakes." Note: To mitigate risk from sharing this knowledge, many support organizations require customers to accept a disclosure agreement before seeing the KCS article.
• Would have a higher or at least the same opinion when asked: "If the support information were marked as being in draft format, what opinion would you have of a company that shared everything they know, even if it had editorial mistakes?"

Time/Value Tradeoff: KCS Recommendations

From this survey feedback coupled with other experience implementing KCS, the Consortium feels confident recommending that organizations invest in content speed and accuracy over presentation and format. We should strive for timely and accurate knowledge, ensure we are investing appropriately in training, have a good balance of competencies, develop a licensing model (see the roles section in Practice 7, Performance Assessment), and follow the recommendations for maintaining just-in-time KCS article quality through a sampling process and the creation of the Content Standard Checklist.

When it comes to information completeness and degree of validation, organizations must individually assess the risk-benefit tradeoff of sharing information early. The Consortium's findings should not be used as a substitute for asking customers about their needs in this area. In our experience, the just-in-time information model has become increasingly accepted as the business community has embraced open source, monthly and quarterly software releases, and extended and open beta-testing programs. Appropriate disclaimers, click-to-acknowledge interfaces, and a clear indication of KCS article status (confidence) are all ways to make the KCS article visible earlier and let the customer determine their own risk profile for the situation.