Measures of Success
Measures are always a challenge, even for a business we fully understand - let alone an emerging business model. Swarming does not change the highest-level goals of the organization. We are still interested in reducing customer effort, and increasing customer success, productivity, and loyalty, as well as the speed, consistency, and accuracy of our response to requests.
We are big fans of Kaplan and Norton's The Balanced Scorecard as a framework for measures; it helps us think about measures from different perspectives. It also highlights the critical distinction of activity measures from outcome measures. For Intelligent Swarming, we propose the following high-level framework.
- Customer
- Effort
- NPS
- Retention
- Escalations to management
- Business
- Revenue
- Margin
- Market share
- Brand promise realization
- Operational
- First touch resolution
- Avg time to resolve
- Productivity, capacity
- Queue bouncing, callbacks
- Employee
- Understanding and buy-in (engagement)
- Effort
- Loyalty
- Job satisfaction
- Organizational Learning
- Time to proficiency
- T-shaped people
- Key ONA indicators of health
- Collaboration health indicators
Intelligent Swarming changes how we measure team health and contribution. To assess the health of the Intelligent Swarming model, there are a number of leading indicators or activities that are interesting to monitor. These measures should not have goals, but trends in the activities can be an early indicator of the desired behaviors. And, in fact if we put goals on any of these activities, the trend becomes meaningless as an indicator of behavior. Do not put goals on activities!
- Frequency of use of analyst’s expertise (how often do we go to an individual?)
- How long to ask for help (case open to raise hand)
- Too long? Too quick?
- Frequency of requests for help from any one individual compared to the average for the team
- Time to respond: how long for individual to respond
- Frequency of request for help from a specific person
- Frequency of response by individual
- Unanswered requests
- Feedback from the requestor of assistance about the responder
- Feedback from responder about the requestor
- Frequency of unsolicited offers of help
- Knowledge articles linked, improved, or created as a result of collaboration
- Updating or linking to Agile stories
- Participation in daily standup, posing good questions, and offering help
The Challenge of Assessing Contribution
Contribution assessment is an element of the Intelligent Swarming framework that is still a frontier. Assessing an individual's contribution in a highly collaborative environment is challenging, in part because it is so dramatically different from how we have assessed the contribution of individuals in the past. The traditional transaction-based model of counting activity has conditioned knowledge workers to focus on numbers rather than value. Our goal is to assess who is creating value, not who is busy. Value is abstract and therefore not countable in any discrete way.
Consider the following scenario:
Two knowledge workers are working on a request. They have spent the last two hours trying to understand the issue and doing analysis and research. A colleague overhears the conversation (or in a swarming environment, has visibility to the open issue because they have experience with the topic at hand) and makes a suggestion on something to look into. Following the recommendation of their colleague the two knowledge workers solve the issue in the next 10 minutes.
How do we assess individual contribution in a scenario like this? It cannot be based on work minutes, or the number of events. Did the two knowledge workers wait too long to seek help? Did the work they did in analyzing a difficult situation enable their colleague to easily offer a helpful suggestion? What is the value of problem determination vs problem resolution?
The discussions we have had on this challenging topic have led us to the idea that a reputation model that reflects the history of value creation for an individual is the key. And that reputation model is fed by those who realize the value created. The combination of feedback (explicit) and the behaviors of the value recipients (implicit) will, over time, reflect the value created. This is a dramatically different way to think about assessing contribution. The concept is: value should be assessed by those who realize it, not by a third party (the manager) who is not part of the value creation or realization flow. It takes the manager out of the role of judge and puts them in the role of coach, facilitator, and exception manager. Leadership is responsible for creating and facilitating the value-capture mechanisms in the environment. Management has to give up their obsession with numbers and focus on people's behaviors and their ability to make good judgments. If people's behaviors align with the desired outcome, the numbers will take care of themselves. If knowledge workers have the information they need, an understanding of the bigger picture (why we are doing this), and a clear understanding of the outcomes, they will make good judgments based on the situation. This is not to say we ignore the numbers; we still have to pay attention, but a focus on the people has to come first.
Intelligent Swarming requires judgment on the part of the people swarming. If we don't trust the knowledge workers to make good judgments, perhaps we have not given them the perspective and information they need, or perhaps we have not hired the right people? Both are leadership's responsibility.
Two good reference points for the viability of such a model are the contribution measures used in Agile and the reputation model that exists in healthy communities such as Stack Overflow.
There is more to come on this topic. We continue to learn a lot about this topic as a number of the Consortium members are working on and implementing reputation models. Two interesting lessons learned in this area:
- Recognition is more powerful than rewards. Attaching tangible rewards (money) to the reputation model does not change people's behavior in a desirable way. Rewards are not a motivator in cognitive or intellectual work and in fact, tangible rewards can diminish the impact of recognition. (Video: summary of Drive by Daniel Pink)
- Do not do leaderboards as they promote competition, not collaboration. We should do recognition boards that recognize people's mastery in critical skill areas (there should be lots of skill areas defined). Recognition is criteria-based, not ranking-based. It is not about recognizing the one person who is the best at this skill, it is about recognizing all the knowledge workers who meet the criteria for mastery in that skill area.