For many practitioners in the field of learning, ROI—return on investment—has taken on a negative connotation. Evaluating ROI for learning initiatives is viewed as time-consuming, expensive, and difficult to do, akin to setting up a big science experiment in an organization with variables, statistics, equations, and working with dreaded spreadsheets. Now, while it’s true that some in the field of evaluation do approach ROI in this way, it does not have to be this way.
In fact, taking a story-based approach to ROI evaluation not only offers an alternative approach to the “science experiment,” it is a more powerful approach that has proven to be faster, cheaper, and easier to do.
What does taking a story-based approach mean? It means conducting a series of conversations with learners, asking questions, and constructing a story of value creation based on their answers. In doing so, evaluation methodology becomes a structured process of reflection for telling the value story through these conversations.
Here are the five basic questions to include in conversations with learners.
1. What were your key learnings? This open-ended question is intended to have the learners reflect and, in their own words, describe which part of what they had learned was most important for them.
2. How did you apply what you learned? Take the conversation a step further to learn how the learner’s behavior changed. Probe for details, for example: what was done specifically, who else was involved, and where did the actions take place?
3. What impact did these actions have? Impact can be both intangible and tangible in nature. Intangibles include areas such as improved teamwork, upward communications, decision-making, etc. More tangible impact areas include improved productivity and service quality, increased sales, and reduced costs.
4. How much of this impact (on a percentage basis) would you attribute directly to the actions you took? The actions of the learners may have improved service quality, however, there may have been other potential influencing factors on service quality. The intention of this question is to isolate the effects of the learning experience on the improvements in service quality.
5. How confident are you (on a percentage basis) in this estimate? In question 4, the learners estimated the percent of impact their actions had on service quality. Since no estimate is perfect, the answer to this question is intended to account for the error of the estimate.
Several conversations with the learners are conducted—generally about 10 to 20, depending upon the size of the learner group. These conversations are written into concise narratives. General themes from these stories are captured and written into an overall report. These stories balance the “head with the heart,” conveying the value of the learning initiative to stakeholders, sponsors, business leaders, and others.
As an option, the data from questions 3, 4 and 5 can be analyzed so tangible benefits can be converted into monetary value. The ROI of the learning initiative can be calculated by factoring in the full cost of the initiative.
For example, one learner noted that service quality was improved by the actions she took as a result of the learning initiative:
• (Question 3) She noted an annual benefit of $40,000 (later verified)
• (Question 4) Attributed 80 percent of this benefit to the learning initiative
• (Question 5) And was 75 percent confident in her estimate
•$40,000 x 80% x 75% = $24,000
A similar analysis is completed for all learners: Their monetary benefits are tallied, program costs identified, and ROI calculated. So, for example, if the tallied benefits were $500,000, and the program cost was $300,000, then the ROI would be:
ROI = (($500,000 – $300,000) / $300,000) x 100 = 67%
Three key points regarding this story-based approach to ROI:
1. The power of the data comes from the stories the learners shared. The evaluators do not create the data—they merely ask questions, listen to the answers, collect the data, and organize it into a master narrative. The credibility of the data comes in large measure from the credibility of the learners providing the data and the veracity of the stories they share. The linkage from the learnings to the actions taken, and then to the impact these actions had in the organization, are written to be both transparent and believable.
2. Conducting a story-based analysis of a learning initiative does not have to be onerous or time consuming. True, conducting 10 to 20 conversations does take time, as does writing up the data. However, the story-based approach is, in and of itself, a participatory change process that reinforces the learning. Story-based evaluation may be positioned as a follow-up activity to the learning initiative and an additional opportunity for learners to make sense of their own learning experiences and to share their insights with others.
3. Taking a story-based approach takes into account environmental and business factors. The nature of the open-ended questions allows for the expected to enter the conversations. A wide variety of contextual factors that influence the behaviors and the impact of these behaviors can be surfaced and explored further. This information further adds to the richness of the stories.
Content by Merrill Andersen
)