Why Kirkpatrick Matters

Donald L. Kirkpatrick’s passing last week at age 90 caused a moment of reflection for me. I never met the man, but his ideas have been an ongoing subject of discussion and debate in my professional life.

In a dark and smoke-filled office at a place called Chemical Bank, near Wall Street in Manhattan, I was first told of Kirkpatrick. In my first months as a “training” person, coming off an unsuccessful stint as a high school English teacher, I was taking instructions from a senior member of the training staff on his expectations for me to design the evaluation of a new-hire training program. The man (whose name is long-forgotten) explained to me the four levels of the Kirkpatrick Model:

• Level 1: Reaction. Did people like the course?
• Level 2: Learning. Did they learn the stuff they were supposed to learn?
• Level 3: Behavior. Were they able to perform the skills on the job?
• Level 4: Results. Did the application of the skills lead to improved business results?

The beautiful order and simplicity of the system immediately appealed to me and set off a long-lasting moment of clarity. It was then that I first understood the difference between what we do as school teachers (teach people content, for no apparent reason, other than knowing all of the stuff that all educated people know) and what we do in corporate training functions (teach people skills, for the purpose of doing a job).

Over the years I participated in a number of task teams charged with implementing a Kirkpatrick-esque system of evaluation. Typically, the task team would spend 6-12 months creating a meticulously-crafted standard Level 1 Evaluation instrument (smile sheet, if you will), before running out of steam and disbanding. This very tendency points to the peril of Kirkpatrick – people get so focused on following the Levels one at a time, starting with Level 1, that they lose sight of why they were pursuing them in the first place. My experience is supported by copious research that repeatedly shows that almost all organizations have Level 1 instruments in place, and almost none of them have Level 4 systems in place. This only serves to reinforce the instinct within training types to expend too much energy thinking first of how to make their classes or eLearnings aesthetically appealing or, even better, fun!

My conception of why we train people has evolved considerably in the intervening 20 years. Rather than a sequence, I prefer to think of the Kirkpatrick Levels as a taxonomy. All evaluative activities fall in to two categories: Measures that Training people care about and measures that Business people care about. While there is value in the former, we should emphasize the latter.

I know that Kirkpatrick himself understood this critique and tried to get out in front of it. Today, his family’s company, Kirkpatrick Partners, has consciously clarified and updated the presentation of the original concept in the form of the New World Kirkpatrick Model.

Being misunderstood and over-simplified is a fate that Donald Kirkpatrick shares with Winston Royce, the “inventor” of the waterfall method of managing large software development projects. In 1970, Royce wrote a highly influential paper that is often cited as the basis of the Software Development Life Cycle (SDLC), which today is routinely massacred by agile development enthusiasts. His paper explicitly denounces the end-to-end, one-way street that is waterfall planning, yet it spawned a multi-million dollar cottage industry. Royce and Kirkpatrick both gave the world ideas that were almost too elegant for their own good, so much so that many of those espousing the method only considered the ideas themselves in a superficial manner.

By any measure, Donald Kirkpatrick was a giant in the field of learning and development. His evaluation concept was first presented it as his PhD dissertation topic in 1959, and it was refined in many articles and books in the ensuing 50 years. To those of us steadfastly pursuing the holy grail of correlating training to business outcomes, he has been a constant inspiration.

One Man, Two Conferences

I’m fairly certain that I’m the only person who attended both THE Performance Improvement Conference #ISPI14 and the Global Scrum Gathering #SGNOLA this spring. I hold professional designations from both the International Society for Performance Improvement (CPT - Certified Performance Technologist) and the ScrumAlliance (CSPO - Certified Scrum Product Owner). As one who works as a education person among technologists, I’m interested in considering the distinction between the two conference crowds and the ethos of the attendees.

The conferences themselves were comparable in size, agenda structure, and cost. Both were more of an educational event than a trade show. Both had a lot more good than bad, and both experiences filled my little head with a boat load of information and ideas.

Below are two lists that show how they are distinct. I would love to hear your comments.

Conference versus Conference

Theme	ScrumAlliance	ISPI
Typical profession	External Agile Scrum Consultant	Internal performance consultant
Touted Credentials	ScrumAlliance certifications	Academic degrees
Gender mix	80% male, 20% female	50% male, 50% female
Conference locations	World-class cities: New Orleans, Paris, Berlin	2nd tier cities: Indianapolis, San Antonio, Reno
Book you need to have read or pretend to have read before attending	The Scrum Guide, by Schwaber and Sutherland	Human Competence: Engineering Worthy Performance, by Thomas F. Gilbert
Venue-based Metaphor	Music (New Orleans)	Auto racing (Indianapolis)
Conference groove	Networking	Forming bonds
Most popular tweet	A slide	A selfie
Awesome keynote speaker that had everyone buzzing	Kenny Rubin	David Maxfield
Book giveaway to support the awesome keynote presentation	Essential Scrum, A Practice Guide to the Most Popular Agile Process	Influencer, the New Science of Leading Change
Gadget giveaway	Portable smart phone charger	Thumb drive
Laughing, crying	Almost none	Almost constant
Typical model	Circular and repeating	Horizontals and verticals
Involvement of forefathers of the profession at the conference	None of the original Agile Manifesto signatories present	Almost every living performance improvement guru was present

Ethos versus Ethos

Theme	ScrumAlliance	ISPI
Foundation	Experience	Research
Holy grail	Delivering value	Measuring value
Common reference	Failed software projects	Skinnerian behavioral science
Before becoming a consultant, I spent years as a(n)….	Software engineer	Instructional designer
It all starts with articulation of….	An idea	An outcome
People are….	Resources	The most important thing
Management are….	Adversaries who don’t understand us and how we want to work	Leaders who don’t yet realize how much we can help them
Love/hate relationships	Hate for project managers. Disdain for HR.	Love everyone
About measurement	A lot of talk about things that should not be measured (defects, release cadence)	A lot of talk about what can and should be measured
On Training	One capable person can pollinate specific skills within a specific team	Scalable solutions and support structures need to be put in place
Don’t forget to….	Follow applicable scrum rules	Measure
People love to talk about how nobody talks about….	Continuous integration	Root cause analysis
View of the future	#noestimates	Predictive evaluation
They are snarky about….	Manual testing processes	Training as a standalone solution
Language of success	defect-free, minimum viable, value flow	greater discretionary effort, improved performance, employee satisfaction
Fancy word to describe most problems	Recursive	Unaligned
Fancy word to describe solutions	Automated	Holistic
What needs to be scaled	Coaching	Performance support
What attendees would learn if they attended the opposite conference.	An appreciation for the human component of success	An appreciation for the great learning agility that exists within work teams
And they would also learn	In order to quantify value, you need more sophisticated measurement techniques	People on teams don’t care about corporate interventions unless they have immediate prima facie value

Are there any drawbacks to agile?

I was taken aback with my response to a perfectly reasonable question. Are there any drawbacks to agile?

A binary question, to which I responded, appropriately.  “Well, yes there are.” Then I paused. The audience was not satisfied, and my head was swelling with the conversations I have on a daily basis about the strategic challenges of agile. So, I went for a divergent and long-winded (for me) answer:

“The software development community is divided on the ultimate efficacy of agile. There has been research that shows that agile development projects improve speed and quality. But, there has not been enough research over a long period of time to convince everyone.

“The popularity of Scrum, which has become a cottage industry for charismatic, book-writing consultants, has inculcated some misconceptions which have undermined, ironically, the case for agile. People focus so much on Scrum, which is merely a set of rules for the planning game, that they tend to ignore the enabling engineering practices that ultimately fuel effective development teams.

“The Agile Manifesto and current thought leadership are notably silent about the role of the manager and leadership, so important to any transformational effort. This again depowers any transformative oomph that such an initiative might have. I have seen leaders at all levels underestimate the cultural component of such an effort, and they tend to think that agile is a training thing. “Get the people trained on agile, and we will be agile.

“Top-down governance, a necessary evil in big and publicly-traded companies, is usually highly disruptive to agile teams, unless all of the supporting and operational functions are also adopting an agile mindset. The same can be said of the support functions, which would include the lawyers, the bean counters, and yes, the people people.”

Other than that, it’s just great.

The person who asked the question had more questions for me after the session. “What are you saying? Are you a champion of agile or a critic?”

Sigh.

I believe in agile. I think the elements of the Manifesto, if followed as a value system, can lead to work output that is more focused on outcomes. I love that so many smart people, like the Original Signatories and others with whom I work, are so invested in making agile work, because they think it’s the right thing to do. I also see potential for agile ways of thinking to transform work and the world. There are a myriad of people who are making rain with agile. I believe agile can accelerate our contribution as human performance technologists.

What Does Test-Driven Development Mean to Education?

I work at CA Technologies, a large software development company that is seeking to maintain a competitive edge through “innovation, execution, and speed.” As a company we subscribe to agile as a principle-based approach, and we play the planning game with Scrum. But the enablers of higher quality software within faster release cycles are the engineering practices beyond just the agile planning methodology.

One of the most powerful practices used in software development is Test-Driven Development (TDD), where the test is written and run (and fails) before the code is written. The goal of writing the code is to pass the test (and nothing more). In an Agile environment, this is done in increasingly small increments. Ultimately, tests are run every time a piece of source code is checked back into its repository home. With small increments, bugs are isolated and fixed immediately. The full fruit of this strategy is Continuous Integration, where the software always works and always improves. (Massive disclaimer: The preceding paragraph was written by an corporate education person living in a software development world. Stay with me!)

So what does this have to do with anything in the realm of human performance improvement? How often are we presented with requests to create learning assets with no reference to the business problem being solved? (Answer: For me, several times in the past week.) How often do we create things without an evaluation component, and then someone later asks if it is having an impact? (Answer: Too often.) If we can start with the outcome, if we can write the test that will be satisfied before we design the intervention, then we significantly improve the odds that our solution solves the business problem. Or, at a minimum, we know whether or not the solution answers the test question.

From a performance consulting standpoint, this is related to our practice of front-end analysis. When encountering an “order-taking” situation, there are simple questions we ask to reframe the request, such as: What is the desired outcome? What will change as a result of us implementing something? What will happen if we do nothing? What is the root cause of the performance problem?

Do not be surprised when a frustrating, circular, rhetorical conversation ensues with your client, where everyone needs to acknowledge that only rarely is the answer to these questions related to training. After all, if it was a pure training need, as the original request assumed, we would be able to, using training in isolation, enable the development of employee capabilities that are aligned with business objectives. No surprises here, since in the vast majority of cases (around 85% of the time – people have studied this), training is not the solution to human performance problems. This state of affairs will often be perceived as unfortunate for our client requestors, because they were hoping that we could help them tick the box by developing and rolling out a training “solution.”

Now, as we continue along this line of discussion (often root cause analysis) we have to be the bearer of bad news: You have a problem, but it’s not related to training. Maybe people don’t understand the expectations? Maybe the process is broken? Maybe the interface on your application is not intuitive? Maybe people are not motivated or incented to do what you want them to do. In these cases, training is not the answer. In fact, if you look at the possible causes of issues that inspire training requests, it is really difficult to imagine a case where training by itself is the answer. (That’s a topic for another blog post…for more information, check out Thomas F. Gilbert’s Behavioral Engineering Model.)

Back in the real world: Do not let the client walk away; we can still help them! Yes, even the Education people can help them with their business problem.

If we responded to every training request with diligent and dutiful fulfillment, as I did very well for many years, we would be very busy creating things that add no real business value. On the other hand, if we restricted our efforts to projects that met some sort of academic definition of training/education, we would be doing almost nothing. Both of those situations are bad places to be. The sweet spot is doing things to help our clients succeed, regardless of whether or the instructional designer in us thinks it meets our definition of training/education.

Almost any work we do in response to this type of analysis is OK as long as it’s adding value. Adding value means being able to demonstrate that you’ve added value after the thing is implemented. And your chances of adding value expand exponentially if you can come to an agreement with your client, before the design and development starts, on what adding value means. That is, if what we do works, what test will be satisfied?

Now that we are in the business of adding value, rather than the business of creating useless training, we have to think differently about how we create tests. Many of us are familiar with how to create training tests in the realm of the classic Kirkpatrick model, such as knowledge reviews and behavior observations. But by applying TDD to Education (ETDD?!?), we will need to invent different types of tests that answer different types of questions: How to you measure awareness? Buy-in? Teamwork? Innovation? Strategic thinking? These questions might be out of our comfort zone, and indeed, out of scope for what we think of when we think of Education, but that doesn’t mean we shouldn’t attempt to work with our clients to write tests for them. It will take persistence, creativity, and innovation, because every business problem is unique. Certainly our clients are often out of their comfort zone with these questions as well. That is why they called us in the first place.

The Logical Fallacy of Training

A classic scene from The West Wing provides a pithy definition of post hoc ergo propter hoc, a Latin phrase which defines a logical fallacy translated as "after this, therefore because of this." The fallacious argument is that if X occurs after Y, then Y must have be caused by X.

We see this argument often when consulting with clients and business leaders, and we even see it slip into the thinking of Learning people. I recently read an otherwise very intelligent article that provided a perfect example: employees are not following a procedure, and the assumption is that a lack of knowledge is causing the employees’ failure to follow the procedure. Leaping off from there, an elaborate business case follows: if we could just improve retention of learning, improved performance ensues.

Correlating a performance failure (“the employee who fails to follow proper customer-service principles and ends up driving a loyal customer away”) with poor learning transfer is fallacious, since the root cause of that fail is much more likely to be related to non-training factors. How many times do we see training promised as a panacea for performance problems? Excellent training is provided to employees, and Kirkpatrick-esque multi-level evaluation is used to establish the efficacy of the training. Then, after the performance indicators don’t move, the failure is attributed to the training, instead of the business executive, who is ultimately accountable for the result. Now we face a lose-lose situation, where not only did the performance not improve, but the training department is the scapegoat.

In Shakespeare’s Julius Caesar, the nobleman Cassius says to Brutus, “The fault, dear Brutus, is not in our stars, / But in ourselves.” We could adapt that notion for an executive to say: “the fault, dear comrades, is not in our supporting staff functions, but in ourselves” Fans of Thomas Gilbert, the “father of human performance technology,” recall that the point of his Performance Engineering Model is that the root cause of employee performance problems is far more likely to be non-training factors, such as flaws in process design, incentives, expectations-setting and other work environment issues. “Before we throw training at a problem,” the enlightened business leader continues, “let’s make sure that we understand the root cause of the performance issue.”

Learning professionals, take heed and seek the root causes of the performance issues before moving towards a (very expensive, very time-consuming) training solution. Business people don’t automatically think this way, so we, as advisers, need to teach them. When training is actually implemented, a useful metric beyond knowledge transfer would be to see a correlation between the effectiveness of knowledge transfer (Kirkpatrick Level III, if you will) and performance. Don’t make an assumption about root cause. Otherwise, you fall into the trap of post hoc ergo propter hoc.

This type of thinking can change the game. The game for most learning professionals has traditionally been to drive usage of training, creating the illusion of Learning as an indispensable function. The new game will be to create as much business value with as little training as possible.

Schools of Thought About "Training"

Do you ‘♥’ Training, or do you do everything possible to avoid it?

Recently, I was searching for a simple example to delineate the differences between what I think of as two schools of thought about training: (1) the conventional view of training (in my company, we talk in terms of “order-taking”) and (2) the (presumably more enlightened) view that adds emphasis to front-end analysis and a focus on outcomes. The former I will refer to as the “I ‘♥’ Training” school, and the latter, the “I do everything I can to avoid instruction" school.

The “I ‘♥’ Training” School

At its annual convention, the American Society for Training & Development (ASTD), the largest association for our profession, often distributes keepsakes (buttons, keychains and the like) that proclaim “I ‘♥’ Training.” This swag represents the most common view of training, among education professionals and the clients they serve. This school of thought reinforces mantras like, ”training is fun!” and “all curious and inquisitive people ‘♥’ training!”“I ‘♥’ Training” is an understandable, and even healthy, point of view for classroom facilitators and, to some degree, instructional designers. It is also common among executives (training and otherwise) who measure the success of training by volume of content produced, course evaluation feedback, and participation. In terms of the Kirkpatrick model, I refer to these metrics as 'Level 0' evaluation, that is, usage.

Even a cursory look at the accomplishments of Chief Learning Officers profiled in CLO magazine shows that many of the most successful practitioners in our field quantify their accomplishments in terms of spending and usage (Level 0 metrics), such as number of people trained, amount of money spent building training infrastructure, or number of courses put in place as part of a curriculum.

The “I do everything I can to avoid instruction" School

When HPT pioneer Joe Harless passed away a couple of years ago, I was reminded of the other school of thought. In an article in the Performance Improvement journal, author William Coscarelli recalls Harless opening a 1977 talk with "I do everything I can to avoid instruction." He continued to explain how front-end analysis directs clients to the root causes of their issues, and those root causes are only occasionally matters of “training.”

This "avoid instruction" attitude carries with it a whole other set of assumptions. Instead of training for the sake of training, with a goal of butts in seats, the measure of training success is business impact. Instead of more training being better, adherents to this school recognize that training is expensive to produce and deploy. Instead of trumpeting 4.5 mean responses on smile sheets, the Harlessians (yes, I made that up) know that training has limited utility- it can only be relied on to help build skills.

So What?

Carrying the Harliessian torch is much easier said than done. Even with the best of intentions and rigorous discipline, one cannot reasonably expect to demonstrate measurable business impact on even half of training projects. Alas, sometimes training is just training.

But we have to fight the fight.  I'll always fight the good fight in seeking demonstrable performance improvement over the development of useless training. And I'll always seek true partnership with clients to help them see how training can support their goals. If we can never demonstrate value and a focus on outcomes, our function will cease to exist.