Assessing to Learn and Learning to Assess

Assessing to Learn
and
Learning to Assess

Society for Organizational Learning

Assessment for Learning Research Initiative
Report of the First Research Forum
January 14-16, 1998

Assessing to Learn & Learning to Assess: Main Report

A Practitioners' Frustrations
Afterword from "A Recovering Management Accountant"
Framework for Thinking about Assessment
What is a "Learning Initiative?"
Assessment for Evaluation and Assessment for Learning
Different Types of Assessment and Different Audiences
Next Steps
The Organizing Committee
Participants

Appendix A: Reflections

Karen Ayas
Per Bastoe
Michael Beer
David Berdish
Anders Bröms
Juanita Brown
John Carroll
Megan Clark
Bill Easterday
Stella Humphries
Joe Jaworski
Tom Johnson
Katrin Kaeufer
John Knutson
Dawna Markova
David Obstfeld
Bob Putnam
Jean Redfield
Louann Reilly
George Roth
Peter Senge
Bill Torbert

Appendix B: Group Reports

Listening For The Music
Bridging
Health Chart
New Traditions In Assessment

Appendix C: Homework

Homework Letter

Karen Ayas
Mike Beer
Linda Booth-Sweeney
Lori Breslow
Juanita Brown
Megan Clark
Joyce Fletcher
Joe Jaworski
Tom Johnson
Rick Karash
Katrin Kaeufer
John Knutson
Dawna Markova
David Obstfeld
Bob Putnam
Louann Reilly
George Roth
Dennis Sandow
Tim Savino
Bill Torbert
Etienne Wenger

Appendices D and E: Guest Talks

"Nine Frustrations Of A CEO: Connecting Values And Business Performance" - Bill O'Brien

"Reflections of a Recovering Management Accountant" - H. Thomas Johnson

Assessing to Learn and Learning to Assess

Society for Organizational Learning
Assessment for Learning Research Initiative¹

Report of the First Research Forum
January 14-16, 1998

Organizations around the world are making investments in developing improved learning capabilities, with little guidance as to how to assess the effectiveness of these investments. It is the goal of this long-term initiative to contribute to clearer thinking and improved practice so that the consequences of learning investments can be better understood and so that such efforts can become more effective.

The Assessment Initiative is a multi-faceted research undertaking, drawing together diverse researchers, practitioners, and consultants. It is our belief that there is no one single "lens" that can clarify everything that is important in complex organization change processes. There is no one "story" that tells the whole story, no one "proof" that something works or not, no one definitive measure of how much improvement has been achieved. Understanding the consequences of learning investments requires understanding changes in individual and collective skills, changes in attitudes and norms, shifts in informal social networks and work practices, as well as financial results and overall changes in business performance. This initial workshop included researchers, practitioners, and consultants concerned with collaborative inquiry and systems thinking skills, learning histories, "communities of practice" and autopoeitic social networks, leading large scale change, and accounting and performance measurement.

The aims of this initial workshop were threefold:

1. begin building a diverse study group by enabling the participants to start learning about how one another thinks about assessment;

2. begin to develop a language for thinking about assessment, and;

3. establish next steps for moving forward into field studies and ongoing collaboration.

The first evening, Bill O'Brien and Tom Johnson set the tone for the meeting, raising some of the core questions and issues which, they argued, should be the concern of this initiative over the next decade.

¹ We gratefully acknowledge the support of the Shell Foundation for funding to support this research.

A Practitioners' Frustrations

Bill O'Brien launched the workshop with reflections on his 20 year career in developing a "values-based, vision-driven" organization. Bill was marketing Vice President and then CEO during a period when Hanover Insurance went from the bottom to one of the top performing property and liability insurers in the U.S.²

Bill's comments were organized around "nine frustrations": deep problems that he felt plague all managers attempting to foster significant innovations.³

There is so much fog in seeing real business performance against the ebbs and tides of business cycles. This means that it is very difficult to tell how a business is doing in less that 5-10 years. What are the differences in attitude between those businesses whose performance declines over time and those that continue to prosper?
We don't understand gestation periods: this makes it difficult to judge the effectiveness of basic innovations in culture, processes, and capabilities. We need to shed light on lag times between cause and effect - too much energy is lost on unrealistic expectations.
There is a war between the short-term and the long-term. Pressures to make short-term results look good are especially pernicious given that companies are, basically, their "own scorekeepers" and can typically manipulate short-term business results plus or minus 20%, based on standard discretionary reporting decisions. "Independent" CPA firms are insufficient protection, given that they are paid by the same managers they are reporting on. The people on the front lines of an enterprise typically know when "disinvestment" is occurring, but this can be covered up for many years.
Is the problem lack of knowledge or lack of virtue? "I believe that for many of the problems that afflict corporations, they are 70% virtue. We need to reinstate virtue."
Self score keeping: Is it more temptation than management can handle?
How do we embed "leanness" as a basic virtue, starting at the top? "I worry about companies that can lay off one thousand people but not one person." There needs to be a healthy return for every dollar spent and that needs to be monitored as a company grows; the temptation to get "fat" in good times leads to the large scale lay-offs.
Much damage is done by trying to quantify that which shouldn't be quantified: we often use proxies when we are too lazy to develop an understanding of the context for people's actions.
Much improvement can be achieved by simply avoiding dumb things that everyone says you must do. Avoid fads - choose what has long-term value.
Is it possible, through measurement devices, to develop a "legacy mentality" in corporations? A core dilemma driving large organizations today is the desire of those at the top to "put their stamp" on the organization: if few significant changes can be realized in 5 years, efforts of the current CEO to guarantee that significant improvement occur "on my watch" (or to make sure that no decline occur) can be counterproductive.

² Two excerpts from Hanover Annual Reports, "The Philosphy-Performance Link", and "The Connection Between Learning and Competitiveness at Hanover" are available from the Assessment Initiative Web Page.

³ The entirety of Bill O'Brien's comments, along with those of Tom Johnson, will be available in a SoL working paper in the coming months.

Afterword from "A Recovering Management Accountant"

Tom Johnson closed the first evening session with reflections on Bill's comments from his long career in managerial accounting. Tom is widely known as a pioneer in accounting, notably as the co-inventor, along with Robert Kaplan, of activity based costing (ABC). Their acclaimed 1987 book, Relevance Lost, showed how standard accounting practices had lost much of their usefulness to managers.

"Dr. W. Edwards Deming said that 97% of what matters cannot be measured. Today, it often seems that 97% of management attention is on measures. This means that we are spending most of our time on what doesn't matter." Why do we measure?

The history of western approaches to measurement started with Galileo, who was trying to verify Kepler's idea that the earth was moving about the sun, rather than being at the center of the solar system. Galileo, breaking from Aristotelian tradition, determined that it was necessary to separate the motion of an object from the object itself, in order to measure that motion. This led to the scientific practice of separating things which are connected, in order to measure distinct properties. In the process, we gradually began to sever the connections in our mind, to the point that we lost awareness of the connections altogether. We began to work with abstractions.

Our fondness for quantifiable measures became a foundation for the mechanical world, and for a mechanical viewpoint that gradually extended to how we see everything.

"It wasn't until the 20th century that we started to see ourselves in our organizations in this light. By this time, we had lost awareness that the process of measurement destroys what is natural. Nature does not measure. Nature recognizes patterns."

For the past ten years, Tom has been studying a small number of corporations that seem to not be caught in this measurement trap. Toyota, widely seen as the world's top car company, has no standardized cost accounting system. Scania, a leading Swedish financial services firm, has a uniquely successful approach to new product development. In both cases, it seems that people have developed the capacity to focus on the means rather than the ends in complex human processes. Toyota manufacturing people, in Japan and elsewhere, spend years learning how to assess how the work is flowing, and to learn ways to quickly detect and correct errors as work proceeds. This involves measurement but measurement in the service of learning and enhancing a rich context of tacit knowledge.

"Nature does not focus on ends. Nature knows only means." Can we understand the deeper logic of nature -- can we learn to trust that focusing on the means will suffice?

There was an interesting connection between Tom Johnson's comments and Bill O'Brien's regarding profits. O'Brien emphasized the importance of building wealth, as a primary way the corporation contributes to society. But he warned that "there is a difference between profit and creating wealth". "Wealth production" he said "is an honorable goal - profit is about looking good". He suggested that there is so much latitude in financial reporting that short-term profit figures can be greatly distorted. Tom Johnson argued similarly that, for assessing the health of an enterprise, "profit is an almost useless measure." The challenge, of course, is how to develop improved ways to assess health -- something Tom believes a few firms like Toyota and Scania are doing.

A Framework for Thinking about Assessment

In order to develop some common language and perspectives, a simple framework was suggested for beginning to compare experiences and questions. Efforts to bring about change and enhance learning capacity can be seen in three lights: the initiative itself, "first order consequences," and longer-term business consequences. This leads to three types of questions which have a bearing on what we assess, how we assess, and why we assess (Figure 1):

What are the conditions present which influence receptivity and predisposition to learn? We called these "preconditions," but they might alternatively just be called "environmental conditions." If, for example, a learning initiative is counter-cultural and/or was introduced insensitively - then assessment of its effect will be distorted. This suggests that assessment of an initiative per se needs to be done within the broader context of the local environment.
How can we discern "first order" changes in work practices broadly construed ? (e.g. new skills, news ways of working together, new social networks, new ways of organizing tasks and distributing resources, new ways of resolving conflicts,...). What we choose to observe is colored by the answers we seek -- so we need to be aware of who is asking the question and why. Is the assessment being done (designed) to enhance the long-term benefit of the business taking into account the dual agenda of corporate and worker needs?
How can we link changes in work practices to business consequences, given the complexities and indirect nature of these linkages as well as influences from external (e.g. market) forces?

What is a "Learning Initiative?"

Investments in developing new learning capabilities take many forms. Those that are the most important have the following characteristics:

- they are connected with real work goals and processes
- they are connected with improving performance
- they seek to balance action and reflection
- they afford increased amount of "white space," opportunities for people to think and reflect without pressure to make decisions
- intended to increase people's capacity, individually and collectively
- the focus on connecting: inquiry, experimentation, and reflection
- focus on learning about learning.

We called such efforts "learning initiatives." These go well beyond traditional educational initiatives. As Etienne Wenger offered, "a learning initiative is an initiative to improve an organization where learning is an explicit goal, 'learning' as defined by the participants themselves." Mike Beer suggested that a learning initiative is one where there exists a gap between "where we want to be" and "where we are". There are three levels at which this gap can arise:

substantive: business relevance, strategy, technology, e.g. we need to know more about "x".
context: broader questions, values, larger systems that shape interpretation and action; e.g. "fiefdoms" are internal organizational structures (in this example, covert) that are barriers to information flow.
learning how to learn; this addresses the capabilities we would need in order to reduce barriers.

Assessment for Evaluation and Assessment for Learning

An important difference concerns the intent of assessment. In one sense, when the intent is learning, assessment is inherent in any learning process. It is not possible to learn if learners cannot make sound interpretations of where they are relative to their goals. A child cannot learn to walk if she cannot tell the difference between one or two steps and several steps. Assessment and learning are inseparable, just as feedback and learning are inseparable. As Harley-Davidson's Tim Savino said, "I have difficulty telling the difference between learning and assessing," On the other hand, evaluation is not an inherent part of learning. It is not necessary that the child "evaluate" how she is doing as she learns to walk. Indeed, some researchers even argue that such evaluation can be counterproductive to learning, i.e. could cause the child to conclude that she has "failed". Such a conclusion may then become a self-fulfilling prophecy.

But the predominant colloquial use of the word "assessment" differs from assessing for learning. More often than not, "assessment" is used as a synonym for "evaluation." Indeed, frequently in our conversation, it appeared that we were not communicating, as one person talked of assessing as evaluating and someone else meant assessing as learning.

Jean Redfield of Detroit-Edison suggested that we distinguish "assessing for evaluating" from "assessing for learning." This legitimates both types of assessment, each of which should fall within the purview of this research initiative. But it also forces us to distinguish which type of assessment we are speaking of.

Peter Senge suggested that one long-term aim of this research might be to re-establish a healthy balance between assessing for learning and assessing for evaluating. If the balance has tipped to emphasis on evaluating to the extent that it actually impedes learning, learning processes will benefit from increasing emphasis on assessing for learning. "It is low leverage to complain that there is too much emphasis on evaluating, outsiders wanting to know if learners are adding value," said Senge. "It is probably much higher leverage to build our 'assessment muscle,' to help learners to get better at assessing for learning."

Different Types of Assessment and Different Audiences

Building on these ideas, Etienne Wenger offered four additional distinctions concerning the nature of different types of assessment:

inherent: assessment that is inescapable in the learning process; inherent assessment is done by the learners and may be tacit or explicit.
exportable: assessment that creates a product (insight, data, stories, ...) that can be offered to someone who has not been part of the learning process;
extractive: assessment that "extracts" information from a learning process because someone on the outside has a "need to know" for their own purposes.
systemic: assessments that reveal relationships between different practices and allow people to see how they belong within a larger community.

Further, Wenger said each of these types of assessment can be participative (negotiable) or standardized. Participative assessment engages those assessing in a social interaction to make sense of reality. Standardized assessments attempt to establish measures that require minimal interpretation and are not subject to negotiation. For example, most traditional financial measures are standardized. They are also usually extractive, although teams might also produce standardized process measures as part of their inherent work practices.

Several audiences for assessment were discussed, but a basic break down into three key audiences for improving assessment made sense:

"innovators:" those attempting to implement new work practices;
"followers:" those who would like to learn from innovators and build on their efforts -- future innovators;
"outsiders:" those, either inside the organization or outside it, who want to understand what innovators are doing and/or judge their effectiveness.

The needs of these different audiences are very different. Innovators or learners need to develop "real time" methods for better understanding the fruits of their efforts; their assessing needs to be inherent and systemic. Followers are especially interested in "exportable" assessments that give them insight that can allow them to adapt new ideas, practices, and methods to their own circumstances. Outsiders have traditionally emphasized extractive measures, and especially standardized metrics.

Next Steps

On Friday the group was invited to reflect on what had arisen over the past few days with a view to going forward. Katrin Kaeufer encapsulated our overall task aptly with " we are assessing to learn and learning to assess. ".

Four themes were identified that seemed to capture what was felt to be important:

How do we build a new tradition in assessment for learning?

How do we design research that bridges between the expectations of rigor in academia and pragmatism in corporations? How do we diffuse knowledge between the two worlds?

How do leading practitioners manage the assessment process - how do they introduce it to management? How do they develop a "health chart" for describing/reporting on progress? What style of conversation do they have?

How do you "listen for the music"⁴? That is, how do I tune myself to the subtle signals that tell me work within the organization is going well or that something is wrong?

The group self-organized into four small groups named:
New Traditions, Bridging, Health Charts, Hearing the Music.

Each of the groups met for about an hour and a person from each group has volunteered to write up a summary of the discussion - these summaries are available separately. The intention was that each group begin thinking about future steps in developing research projects.

The task now is how do we begin matching researchers with corporations around issues of assessment. The idea was put forward that "matchmaking" begin with individual corporations and researchers first. We would then follow up with a workshop in 6 months to see how projects were developing and continue to explore the deeper issues identified in this first meeting.

What will happen is that corporations and researchers will be asked to write a short proposal on what they would like to do and that we begin a process of matchmaking. The details of how this will be done will evolve through ongoing conversations via internet, telephone, and personal meetings as needed.

Interestingly, what is emerging here matches an organizing pattern emerging in other SoL initiatives: field projects linked to research forums. Projects allow testing specific ideas and methods in concrete organizational settings. Forums provide the gathering place for people to reflect, share insights and findings, and pose puzzles and deep questions. ("Forums" include electronic and face-to-face meeting spaces.) Connecting Forums and projects may emerge as a core strategy to knowledge generation and learning within the SoL community. In this light, our meeting in January can be seen as the first SoL project Forum on Assessment. The natural next step is to start developing projects.

Commitments so far are to:

a. Shape the talks by Bill O'Brien and Tom Johnson into one or two papers.
b. Compile the homework and the "reflections" at the end of the workshop into a "scrapbook" to capture the perspectives of individuals.
c. Compile the reports from each of the discussion groups and for these groups to report on progress.
d. For each individual to write a short piece on a topic of substance or interest which they want to contribute to the bank of ideas, concepts, frameworks or questions assessment of learning.
e. Invite corporations and researchers to write a short proposal on what kind of project they wish to initiate and or be involved with.
f. For Stella Humphries to continue to shepherd these activities and invite and engage the participation of others
g. For the Organizing Committee to continue to develop the project with the participation of interested parties. The Organizing Committee herewith actively invites other participants to join us in planning next steps.

⁴ The phrase arose out of a story Tom Johnson told about Marcus Wallenberg, scion of the Wallenberg family in Sweden (Scania, Saab, Asea, etc.) who, when asked what he was looking for on this regular visits to his factories and holdings, said: "I listen for the music."

The Organizing Committee

John Carroll, MIT
Stella Humphries, SoL
Bob Putnam, Action Design
George Roth, MIT
Dennis Sandow, U. of Oregon
Peter Senge, MIT

Participants

Bill Easterday	Chrysler Corporation
John Knutson	Chrysler Corporation
Jean Redfield	Detroit-Edition
Megan Clark	Ford Motor Company
Tim Savino	Harley Davidson
Jean Tully	Hewlett-Packard^*
Louann Reilly	U.S. West
Dave Berdish	Visteon Automotive Systems (Ford)
Per Bastoe	Worldbank

Bill Torbert	Boston College Carroll School of Management
Mike Beer	Harvard Business School
Linda Booth-Sweeney	Harvard Education School
Lori Breslow	MIT
John Carroll	MIT
Jody House	MIT
Stella Humphries	MIT/SoL
Katrin Kaeufer	MIT
Wanda Orlikowski^*	MIT
George Roth	MIT
Peter Senge	MIT
Joyce Fletcher^*	Northeastern University
Tom Johnson	Portland State University
David Obstfeld	University of Michigan
Dennis Sandow^*	University of Oregon
Karen Ayas	independent researcher
Etienne Wenger	independent consultant

Bob Putnam	Action Design
Joe Jaworski	Center for Generative Leadership
Bill O'Brien	Center for Generative Leadership
Rick Karash	Karash Associates
Dawna Markova	PTP
Anders Bröms	Samarbetande Konsulter AB
Juanita Brown	Whole Systems Assoc.

^* intending to participate but unable to be present at this particular workshop.

Appendix A: REFLECTIONS

Karen Ayas

We started of by engaging in a dialogue around "assessment of learning". Acknowledging the necessity to distinguish assessing for purposes of "evaluation" from assessing for purposes of "learning", resulted in a shift to "assessment for learning" which entailed "learning to assess" and "assessing to learn".

Although these subtle but important distinctions are shared by us - the participants, and helpful in understanding what is meant by "assessment", my attempts to share this with others led me to question whether assessment is the appropriate term for articulating the purpose and nature of our efforts. I would suggest that we consider "valuation", since: Valuation is setting the value of something. Value is the relative worth, merit, importance, meaning or significance; it can be the equivalent worth or return in money as well as estimated worth. Valuation of learning could then be the acknowledgment of the quality, nature and excellence, usefulness of learning, its estimated worth or perceived value. This brings a different perspective to assessment, raises a whole set of questions that needs to be expressed differently, but could be helpful in meeting the assessment challenge. For instance, the value of learning will be perceived differently at different levels in the organization (individuals, groups, projects etc.) and will also differ among entities at the same level. Then, in the valuation of learning, we will need to look for a way to reconcile those differences.

These are some preliminary thoughts to invite you to stretch them further.

Per Bastoe

Inspiring workshop - combines my two main interests - organizational learning and assessment/evaluation
Motivates to read more of what the researchers here have written
Gives a new approach to the organizational learning initiative/project in the World Bank.

- more awareness of the assessment component

I would like to try to write something on the bridging of evaluation and learning

Michael Beer

I come to the conference with a curiosity about what this was about, hoping to learn and be stimulated, but with great ambivalence about the commitment the conference and beyond might demand. I found the conference stimulated imagination & out-of-the-box thinking and that was valuable. I found a connection between my own work and the topic of assessment (though the conference triggered more then just thoughts about assessment) through the larger conversation and the sub-group on bridging.

I am still ambivalent about how much time I can commit, given other obligations. I might be willing to write an article, perhaps with someone else in the group.

David Berdish

I keep thinking of assessment as a way that I can help people rationalize or justify doing learning organizational stuff in their work. In my culture, time and resources are hard to come by. People are afraid of things that they perceive as new or incremental.

Though this was an assessment workshop, I found myself thinking more and more of leadership -- the courage of people like Bill and John of Chrysler, Tom, Joe J, Peter, Bob Womac, me, and the role we all play in assessments.

I also wish some of the academics would be more humble. I was willing to take hits on my personality, my excitability, and my listening skills. Maybe they should get off their 27 syllable high horses, get grounded, and admit they have something to learn.

Stella: hold me accountable for health charts and matching with other companies and members.

PS. Thanks! Great job! You're a great VP- I'm in love!

Anders Bröms

There is risk with measuring the "result" of a "learning process." Learning is not to be motivated or controlled by "result," it is to be motivated in its own right. Since I came in late to the "group journey," I don't know the history but see to it that you are not "misunderstood" by newcomers. I, as a "newcomer," see:

Thank you. I enjoyed the stay and am filled with impressions.

Juanita Brown

What I am taking from this workshop are the evocative thoughts shared, particularly by Bill O'Brien and Tom Johnson, with a deep appreciation for the "resonance" and "tone" which they helped to sound. I think the direction we are heading feels very "healthy" and authentic to who we are individually and collectively. I also really appreciated the way in which Stella gently but firmly helped to shepherd the process along with the other members of the Organizing Committee. I am taking new conceptual insights gained in a "field" of genuine inquiry and exploration among us all.

The focus I would like to share in my "mini-paper" - not sure quite yet - but somewhere around new tradition & playing the music.

John Carroll

There are several key concepts that emerged:

- Assessment for learning
- Etienne's categories
- Bill Torbert's categories
- Political issues
- Value & virtue issues
- David's passionate presentations
- health reports
- listen to the music
- relationship with bosses
- what the learner knows about their learning (self assessment)
- a new tradition legacy

I felt much more progress during the small group sessions then the plenary presentations and discussions. Some presentations were too vague, but they provided seeds that grew later especially in small groups and small conversations
Lack of researchers present - a continual issue for this group.
Real danger that this project won't speak to the "orthodox" in research or practitioner communities. Is this a research project??

I could write a bit on health reports as they are used in the industries I know best.

Megan Clark

This has been a stimulating process that made me think of things I hadn't thought of before... Thank you!

I caution or see some challenge in us as a group becoming "knowers" in this territory. Much of the conversation indicated freeze points to assess ~ ~ ~ I am interested in assessing during the journey (while doing, as part of work)………There was a lot of "brain power" in this room, but at times I felt being in a position where I was hearing people advocating too much.

At the end, I worried that we forced old traditions that already existed and weren't patient enough for "natural" emergence - (e.g. revert back to work we'd do anyway).

Title: Why is it important at Ford to ->Assess Learning->Learn Assessing.

Thank you so much for allowing me to participate.

Bill Easterday

My head is swimming with ideas, concepts and new opportunities. It's the result of many present with me. People have offered their heart and soul and I have grasped them and added them to my own. I leave feeling more complete and whole then when I came. I'm not sure I understand everything that's happened. And perhaps I never will. I know it will take months, maybe years for all of it to present and activate within me. Yet some of the ideas that have been generated in my mind are present now and I am going to take those ideas and act on them. It is these types of gatherings that inspire me to continue on.

Jody House

I probably followed/absorbed 20% of these past two days. My lack of expertise in industry, organizational learning, consulting, etc. pretty much limits my ability to write something that would be of any use. However, perhaps I could add something by helping to compile the bibliography.

Stella Humphries

I found very satisfying the richness of ideas, concepts, frameworks, and depth of thought. The openness and willingness to enter into unknown territory inspires the hope that we can begin to develop "new traditions" in research.

I also find exciting the possibility for this initiative to be an ongoing forum that holds the (creative) tension among folk with such different perspectives and cultures. Our conversation acknowledging the "logical-positivist academics", those oriented toward "bottom line" results ; the action oriented "pragmatists"; and the "new tradition" put us all inside the conversational space.

I am engaged by the deeper issues inherent in assessment - how do we make explicit the values we support through the very act of selecting the goals and data for assessment. Are we "only " improving current processes or are we also willing to explore ways of aligning management with alternative views of how the world works? ( e.g. mechanical vs. natural). And/or are we willing to intentionally base our learning goals on long-term legacy aspirations and the ensuing values. How do we do that in practical ways? These are deep and wonderful challenges and potential in this initiative. Tom and Bill's talks give us a strong foundation from which to begin.

Joe Jaworski

I need more time to digest and "be" before I can write useful reflections. My cup is too full at the present time. Here's what I can say now:

I loved being present in the widened arc of the SoL community. I learning so much from new people and I am so grateful to have met others, including David Berdish, Tom Johnson, Anders Bröms, Bill Torbert; I will stay connected to these people.
I am also struck by the importance of this new work.
I will try to do one page of "bullets" abut my take on our meeting of the music group.

Tom Johnson

My mission coming out of these meetings, is to engage in a more focused effort to discover and articulate "how nature works." That effort, an ongoing and never-ending activity, will create the "story" or "the score" by which I will assess how well I am doing, personally and in my organizational lines. (By that score or story I will know if I "hear the music.") That is the new understanding I hope to share with others as I go forward.

Title: "Hear the music"

Katrin Kaeufer

A) Learnings

Table: Assessment - Field

B) Conditions for learning initiatives:

gap: current reality <-> desired reality
willingness/desire to learn
safety/support

C) Process of interdependency of "assessing learning and learning to assess"

D) Title: "Listening to the Music"

Thank you to the organizing committee

John Knutson

From Bill O'Brien's "frustrations," the notion that a value-driven corporation will ultimately be successful and will have a continuing existence and develop a "legacy" is very appealing. It solves the problems of short term focus, and scorecard-driven management.

To change organizations (corporations) into operating this way requires a great deal of learning. Values are multi-dimensional and the conversations engendered are "richer." Therefore, understanding the effectiveness of learning initiatives in helping driving values into the corporate management process, is significant.

Dawna Markova

I want/ need our process to be considered by us as a learning initiative - e.g. to include exploring our preconditions, modes of reflection, etc. I am concerned about the "how" of our gathering & that it contribute to and maximize our individual and collective health and learning. There were brilliant moments without time or processes to take them deeper and wider, I also want to make sure our forms of thinking together are inclusive of all stakeholders - i.e. more opportunities for small groups, one on one, etc.

I am concerned that the next forum could become a competitive area for businesses contribution to research projects and fragment us. Also, as framed, could exclude consultants.

STELLA: Ask me about my research project developing a methodology for assessing on personal transformation.

David Obstfeld

I've enjoyed my participation in the last few days, in particular examining the links between the practitioner and academic communities. This has a afforded me an opportunity to hang out with practitioners and their concerns in a way that hasn't been available to me in some time. I am disappointed we didn't have more academics and I think this limits the potential for the most promising dialogue. I lump into the assessment challenge the kind of measurements that allows for publication in academic journals and think this should be included in our objectives even if its not SoL's focal point. The depth of my future participation will be constrained by the necessity of focusing on my dissertation. I still would like to be included in the unfolding dialogue.

I'd be happy to write a short piece on SoL's role bridging to the academics and practitioners.

Bob Putnam

The role-plays that Dave Berdish did on the last day of the conference crystallized for me a way forward. Dave played two ways that a learning leader might talk to a general manager about how a learning initiative is going. Then the general manager has to talk to a steering committee or board. These conversations are a key leverage point in shaping the assessments that actually drive decisions in organizations.

Let's get together several people whose responsibilities include the kind of conversation Dave role-played. Let's bring examples of how these conversations typically go--for better or worse. The two-column case format would be a good vehicle. Let's reflect on the examples and role-play new ways of having the conversations. I think what this could do for us is:

The people who take part could build their capabilities for having these conversations and for creating the kind of relationships that work;
From the standpoint of practical research, we could discover how people are handling these assessment conversations now. I think we'd identify generic dilemmas and paradoxes--which is the first step to developing creative ways of managing them;
We could get to work on the kind of stories that a learning leader should be able to tell--and on creating conversations in which people can distinguish a good story-teller from someone with a good story to tell.

I offer to facilitate a session like this. Are there learning leaders in companies who would like to take part? And I'd like one or two researchers to join us so that we take advantage of the opportunity to learn what people do now and what issues they face.

Jean Redfield

The pattern of my life has been as a student/learner of the world. I surprise myself by slipping between worlds, and particularly am surprised at the relative ease of my success in the corporate world - a world that is "supposed to be" hard and difficult, maybe even death to people like me. Right now, I am overcome by the possibility of being part of something that brings wholeness to the corporate world. I feel like a pioneer about to explore and build a new way of life that is about abundance, wholeness, and fulfillment.

I am thankful for the gift of meeting, and the privilege of being with, the people here who are opening my eyes to the possible patterns before us, and look forward to continuing. I look forward to crossing boundaries……

I will provide an overview of where are we at Detroit Edison, and what are the possibilities for assessment of organizational learning as a piece of the transformation underway.

Louann Reilly

I very much enjoyed participating in the assessment workshop earlier this month. I think it was a good beginning and look forward to keeping the momentum going. As I gazed around the room and listened to the different perspectives I felt a little like I was directly witnessing SoL's rebirth....

On my trip back to Denver (yes, now home of the Super Bowl Champions!), I continued to think about the "new tradition," and thought I'd share these ramblings with you before they dissipate:

Most of my thoughts have centered on trying to figure out what the crux of the difference is between what we have called "assessment for learning" and "assessment for evaluation". I looked up the definitions of "assess" and "evaluate" and found that they are considered to be synonyms, which reinforces my sense that I do not think the difference lies in the use of those two words -- that indeed, we could use them interchangeably. I am thinking that the problem is not one of the use of the concept of evaluation per se, but lies in the quality of evaluation... and the intention.

The "99%" of assessments that I would like to see us shift away from, are characterized by either poor quality (misconception, resulting from weak assessment skills/muscle, a focus on symptoms/events, lack of knowledge, etc.), or negative or misguided intention (deception, intent to conceal or misrepresent, or the intent simply to advocate vs. inquire, etc.). Or both. It may be that rationalization is what is often substituted for assessment/evaluation. I think that to better describe the behavior we were referring to as, "assessment for evaluation", we need to find a descriptive antonym for learning...?

Given that all learning involves assessment, it makes sense that both learning and assessment share the same pre-conditions (and ongoing environment) for success -- trust, openness, etc. -- and that the tools and approaches we have for building learning capacity also contribute to the assessor's capacity to assess. Much of what SoL's work is about is the building of will, of intention -- the aspiration and courage to reach beyond rationalization -- and not just the building of capacity. To bring about the "new tradition" we will need to address both will (personal mastery) and capacity. Given Bill O'Brien's comments about the role of knowledge vs. virtue in transformation, the emphasis may need to be on personal mastery -- the more controversial aspect of SoL's work for most corporate members, I believe.

As my small action step, I hope to work with a group of software developers within U.S. West to shift their project peer review processes (design reviews, project management reviews, etc.) toward emphasis on assessment for learning. I look forward to drawing upon the resources of SoL assessment project members, and hopefully contributing to those resources as well.

George Roth

The meeting was a good start on a new (yet old) area of focus. I would like to see the common understanding developed here extended beyond the smaller working groups that came to think about this topic together. There are several points which were discussed in this meeting that need to be more broadly considered, particularly by anyone or any organization designing and carrying out a programmatic learning initiative.

Learning programs in organizations are ultimately about change, and hence need to be carefully considered, planned, and supported like any other deliberate organizational change effort. While learning programs are not often thought of as organizational change efforts, they seem to inevitably have that characteristic as they become successful. Without some deliberate planning for change on the part of the leaders for learning, and their seeking support from senior management, these proponents, and the participants they engage, seem to be putting their reputations and careers at risk.
Consider the following analogy between dialogue and assessment. "Dialogue" has become an accepted technique for developing listening and learning skills in organizations. People's direct experience with dialogue, and what they observe in other groups who are able to regularly practice dialogue, provide ample evidence for its benefit and effectiveness. One way of describing dialogue, as meaning "flowing among," emphasizes its characteristic of conversations which people have that serves to develop common understanding, rather than discussion, which debate points and promote the making of decisions. A parallel similar to that of the difference between dialogue and discussion can be drawn between assessment for learning and assessment for evaluation. As organizations develop and practice skills for assessment that promotes learning, they will gain similar benefits to those of dialogue.
The process of getting a wider group of people formally involved in the assessment of an organizational improvement effort is itself an organizational learning process. A formalized and disciplined process of collecting and reviewing data from a improvement program, not solely for the purpose of "evaluating" it, but to learn from it, and develop people's judgment for what is effective, will engage them in thinking about improvement programs, and connect directly to the future actions that they can take based on their insights.
It can be seen as irresponsible, given the experiences which people shared at this workshop, to begin a learning initiative, whether at the individual skill, team, or organization-wide development program level, without having senior management support and a larger assessment/feedback program in place. Rather than considering it as a luxury to develop an assessment program after a learning initiative has been underway, assessment (for learning) needs to be planned and designed as part of the improvement program. And, through an assessment program open to others in an organization, those outside the "learning team" can be engaged in judging its effectiveness, efficacy, and broader applicability.

Peter Senge

I believe that in the combination of Bill O'Brien's frustration & Tom Johnson's "all we have to do is build the practices that are consistent with nature & unwind those which are not" lies a "new tradition" for assessing and learning.

There is, I feel, a deeper purpose emerging. The best I could articulate it now would be to say that all our efforts to build learning capabilities are incomplete until we engage with the measurement culture that dominates contemporary institutions, and the mechanical view of reality which undergirds it. This mechanical paradigm is elusive: you cannot change it by trying to change it. But, if (1) people start living more authentically and (2) we start being serious about looking at the consequences, individually and collectively, the paradigm might naturally be superseded.

Bill Torbert

These were rich, chaotic, caring, illuminating conversations that very much may represent a local, long-term flowering of a new tradition that engages in "Cooperative Ecological Alchemy," or "Transformational Action Inquiry," or more simply, listening and dancing to the music.

I loved the diversity, the dialogue, and the appreciation of the significance and challenge of committing to this -not "work"-but "play."

Appendix B: Group Reports

LISTENING FOR THE MUSIC

Bill Torbert for the "Listening" group
at the SoL Assessment Workshop, 1/14-16/98

(Dawna Markova, Joe Jaworski, Lori Breslow, Stella Humphries, Juanita Brown, Jean Redfield, Megan Clark, Katrin Kaeufer, Tom Johnson, Anders Bröms, Etienne Wenger)

Listening for the music is occurring - more or less - in the meeting between you, dear reader, and this writing. (That is, listening for the music is a form of research that occurs in the first-person present, if at all.)

I am choosing to write in Palatino italics with 1.5 spacing as a minor reminder to us of the personal, sensuous, emotional, inquiring qualities of "listening for the music." (For I too am seeking now to listen for the music - for music that will weave our past conversation at the Assessment Workshop through my present words into our future internal and shared dialogues, mutual misses and understandings, and collective actions [if any such evolve!] )

(Put differently again, this time in the obscure academese
that particularly excites Dave-Berdish-like practitioners !],
style-in-action is theory-in-use
in second-person-present research/practice
[Argyris, Putnam, Smith, 1985; Van Maanen, 1995; Torbert, 1998].

Listening for the music was first mentioned on the first evening of the Assessment Workshop by Tom Johnson (co-author of Relevance Lost) in telling us how Marcus Wallenberg, scion of the Wallenberg family in Sweden (Scania, Saab, Asea, etc.), responded when asked what he was looking for and assessing on his regular visits to his factories. Wallenberg responded, "I listen for the music."
Not "I hear the music." Not "I listen to the music." But "I listen for the music." Amidst the formality and the distractions of such visits, amidst the work and the talk and the scene, amidst the chaos and the imposed order, "I listen for the music." I listen, not to a foregrounded, preconstituted music, but for a backgrounded, implicate music that may not always be there - and that not everyone hears and attunes to when it is. In Peter Senge's words in his first summary report of our workshop, "How do I tune myself to the subtle signals that tell me work within the organization is going well or that something is wrong?"
The Wallenberg story seemed a clarifying epiphany in Tom Johnson's quasi-mystical talk (these recovering accountants!), Gradually, I interpreted Tom as advocating (among other things) that Toyota's pre-eminence as a production/organizational/marketing system is due to their turning accounting numbers from guiding tyrants into subordinate handmaidens of a real-time, trusting and trust-building partnership among organizational members. One of the pre-eminent commitments of such partnerships - whether formal or informal, implicit or explicit - is to listen for the music in all encounters and to retune one another and relevant procedures when the music is "off." Thereby does the primacy of assessment for learning and voluntary change) develop in the midst of real-time actions and relationships; while assessment based on summary numbers extracted from the situation and considered in abstraction from the context become secondary (though by no means irrelevant)? (After all, I just yesterday heard a CFO grow eloquent about how "the numbers sing.")
In the late 1980's ,Scandinavian Airlines hit the same note of real-time assessment and re-alignment, it seems to me, when they adopted as their motto "Moments of Truth." They were seeking not only to advertise their commitment to genuine responsiveness in each encounter with a customer, but also to remind themselves to "make it so" whenever and however possible in business meetings - whether with customers, with organizational colleagues, or with strategic stakeholders. Each encounter is a potential moment of truth when a real meeting and exchange of value occurs, if the proper attention is present; or is seen not to be occurring and is corrected (or is a moment when the parties altogether miss one another, if the proper attention is altogether absent).
The "Check-In" we shared on our snowy Friday morning at the Assessment Workshop itself illustrated the practice of mutual listening and attuning. (Indeed, the "Check-In" is one of the new SoL process traditions for listening for the music, is it not?). During the Check-In,

- we heard back to the origins of the "Check-In" tradition in Dawna Markova's work at Boston's Project Place in the 1970;
- we also listened together (partly with our eyes) to the serene silence of the cleansing snow outside;
- we listened into different members sense of a synchronicity among the stories and perspectives that had so far emerged;

- and alternately into a troubledness about the apparent conflict between empirical, positivistic, "extractive," outside-in, evaluative assessment and holistic, intuitive, "inherent," inside-out, learning assessment;
- and yet again into Katrin Kaeufer's momentary, fluid reconciliation of our music as she was hearing it: "we are assessing to learn and learning to assess";
- "Listening for the music in a workplace," suggested Katrin at another moment, recalling Tom's story, "means listening, in the present, for the conditions, the process, and the results (the past, present, and future) of working well." If we hear no conjoint music-in-the-making, she further suggested, we reflect, theorizing by analogy to past situations, in order to decide on new personal/ collective action for the future.

It was this comment that later led to the formation of the Listening for the Music group, as one of four subgroups, to begin to formulate directions for future SoL assessment research and practice.
In our subgroup conversation that morning, we began by explicating and storying how listening for the music implies "a conscious intent to see." A radio may blast pre-formed music so loudly that we hear it involuntarily without any conscious intent to listen, but the music-in-the-making of spiritual-social-technical-natural interactions is invisible and inaudible without a prolonged and ever-renewed conscious intent to listen synaesthetically.
Anders Bröms told us the story of being tested repeatedly in search of a diagnosis of his fainting spells, only to be diagnosed - by inspection only, no tests - by an elderly retired doctor who was a neighbor. Attending him closely, the elder observed, "You don't breathe." With this guidance and reminder, he thereafter cured himself.
Dawna Markova spoke of having learned "to see people's lights" by her Russian grandmother. Her grandmother taught her how to take a second look, through the heart. Looking in this way outside, the "poor" boy without legs on his little cart was quickly revealed to be a boy "rich" with lights who, wheeling cheerfully among others, inspired a raising of their rheostats.
Others of us mentioned:

- how plant managers will claim they can "feel a factory" within a few minutes of entering it;
- how external auditing teams from Big Six firms - in competing presentation/discussions with Boards - can either succeed or fail to convey that they make music with one another, with the internal auditors, and in the meeting with the Board itself; and how, given their otherwise comparable professionalism, these relative abilities to listen for and make music in the moment, gets or loses them long-term appointments;
- how, over several years of mentoring, "my CEO helped me grow my intuition and passion";
- how the best "bronco busters" do not in fact "break" a horse's spirit at all, but rather "gentle" and befriend it, accustoming it to the touch of saddle and hands, assessing when it becomes mountable, and then developing a trusting partnership between rider and horse through a mutual thinking/listening/ communicating process that they call "thinking riding."

Joe Jaworski modeled "listening for" far more than he did "speaking about" during the meeting, but he has since been the first to send us all a marvelously useful, well-written, and profound paper that he has co-authored with Kazimierz Gozdz and Peter Senge, called "Setting the Field: Creating the Conditions for Profound Institutional Change." The key concern of the paper is how - in a disciplined, long-term, system-wide way - to listen for, sense, and see the subtle ideational, emotional, and interaction patterns - the energy fields - of a workplace. And how can we transform the invisible and often-initially-undiscussible issues that may be inhibiting recognitions and transformations vital to an organization's future?
One story in Joe's paper is told by the 34-year-old head of a major manufacturing facility who has a heart attack, realizes on the gurney in the emergency room how he brought himself to this point by his way of living/working, and decides to transform his whole way of working. When he returns to work he sees, for the first time ever, the suffering in others' faces, a suffering that must become discussible if it is to be transformed. Can he use his own first-person recognition and transformation as a catalyst to invite second-persons to recognize and transform analogous patterns? Can Joe in his writing, or I in mine here, use stories like this as catalysts to invite third-persons to recognize and transform analogous patterns?
My own most abstract, incoherent, and longwinded contribution to our meeting, as I recall it, had something to do with octaves - musical octaves, Pythagorean octaves, the rainbow, etc. A couple of stories may do better at giving hints of what I was trying to get at. The first is about a guy who sometimes runs meetings at Motorola based explicitly on the architectonics of the octave. The second is by Michael Rossman describing his own flute playing in mathematical terms. After that, I'll share some even more abstract material on the type of mathematics and science concerned with "listening for the music." But these all go beyond our January 14-16 meeting at the SoL Assessment Workshop, so I offer them as separate reports for those who wish to delve further...

Thank you all for contributing to an inspiring and promising meeting. May we craft more such...

Coda: The Following Morning

During the night, I dreamt (among many other things) that I was put in jail on charges of being a stranger.

When I awoke this morning, I eventually associated several other stray thoughts with this dream. First, I thought about why I had left the "Listening for the Music" report unconcluded. When I had done so last night, the (non-)move seemed apt: in harmonious analogy with the inconclusive ending of the meeting itself. But this morning my stylistic creativity (and laziness, too, for I was taking every short-cut I could to get home sooner to my sick wife) seemed more like a characteristic, but nevertheless strange and stupid, failure: a likely widening of the gap between practitioner and researcher realities. (That's when I added the little parenthetical hints and citations in the early paragraphs of the report, to imbue it with just a little seasoning of academese.)

This morning it seems important to conclude by highlighting the fact that we found ourselves at the meeting talking ourselves into a vision of the kind of research/practice that will do justice to the challenge of intervening in the ways companies, consultants, and researchers assess organizational learning, as well as their overall conduct. Such visioning seems an apt first half-step-or-so; but it seems important also, in conclusion, to point out the obvious: that we scarcely mentioned specific learning/assessment theories, practices, or research methods.

Moreover, it seems important to suggest that the present-centered, participatory quality of this listening-visioning - that we (and so many others! [e.g. Abrams, 1996; Reason, 1995) are interested in cultivating - can potentially integrate the four, heretofore-usually-unintegrated-and-even-mutually-hostile types of assessment research (inherent, exportable, extractive, and systemic assessment) of which Etienne spoke, and the three, usually-equally-disconnected 'persons' of research (first-person, second-person, and third-person). The question: what does this mean and do we wish to 'make it so'?

Those not-so-concluding, concluding thoughts and questions intermingled with two other stray memories from my last look, last night, at our Reflections at the end of the Workshop. These memories illustrate, it seems to me, how intermingled first-, second-, and third-person research/practices in fact are (how intermingled passion, compassion, and dispassion actually are) and how creatively we must listen for and speak with one another if this project is to continue far and prosper us and others much.

One memory was of David Berdish's passionate cry (I was so impressed by the passion of so many of the business participants at the conference - the men a little louder, the women a little more quietly) in his Reflection

- his cry that we academics "get off (our) 27 syllable high horses, get grounded, and admit they have something to learn."

Here, as you can see, I assume that I am one of the academics who is ungrounded in David's view, and I feel the chasm he experiences between us and that we must o'erleap if we are to engage in a mythically powerful (and vulnerable) collaboration.

The other memory is of John Carroll's voice: "Lack of researchers present - a continual issue for this group... Real danger that this project won't speak to the "orthodox" in research or practitioner communities... Is this a research project??" Those words in John's Reflections reminded me in turn of Mike Beer mentioning to me the first evening that John had voiced the concern to him that he (John) was the only researcher present (of course, I understand him to mean the only senior, "orthodox" researcher). Since I had been willing to grant both Mike and me senior researcher status, I was initially lightly shocked. But Mike said that he was really viewing himself as more a consultant, so I was left to contemplate whether the distance between John and me is analogous to the distance between David and me, and what it actually consists of.

I shortly decided that there is a good chance this sort of distance can melt if there is the will to subject oneself to enough two-hour, two-scotch lunches or cocktail hours, so I have just left John a message inviting him to join me in such an occasion (beverage optional, of course). Others of you may wish to make similar or different contributions to releasing all of us strangers from our own prisons (as you already are...).

(I, for one, do not agree with the Dalai Lama (in Kundun) that

"I can only liberate myself" [although he said it at one of the most rhetorically effective moments possible, and I certainly wish his kind every possible victory].)

BRIDGING

Bob Putnam's comments on the "Bridging" Group

(David Obstfeld, Mike Beer, Rick Karash, Karen Ayas, Bill Easterday, John Knutson, Jody House)

I see two primary bridging challenges:

The Learning Organization community and "hard-nosed CEO's"--i.e., business executives who must make investment decisions;
The Learning Organization community and the academic research establishment.

I am aware that putting it this way implies that the Learning Organization community is monolithic, and of course it is not. A better framing might identify differences of perspective within our community.

The first challenge is about sustaining support in companies. The genesis of this conference, in large part, was the interest of senior decision makers in knowing whether investments in learning organization work pay off in terms of business results. Dave Berdish's two role-plays on our last day showed a learning champion telling his general manager how the learning initiative was going, so that the GM could tell the Board or Steering Committee that, presumably, controls funding.

Assessments by senior decision makers are usually distant from the learning. They are not making assessments about their own learning; they are making assessments of learning initiatives that others are involved in. And they must do this, in one way or another, because they are responsible for making wise investment decisions. The difficulty, in terms of our bridging challenge, is that this means they must be interested in assessment for evaluation or justification (i.e., from an outsider perspective). The learning organization community, as represented by most voices at our conference, finds assessment for learning (from an inside perspective) to be much more to its liking. How do we speak to the legitimate interests of senior decision makers?

The second challenge is about generating knowledge in the learning organization domain that has legitimacy in the academic research establishment. We haven't had a lot of success at this in the past several years, it seems to me; and orthodox research was barely represented at the conference. One response could be, forget the academic research establishment; let's create a knowledge generating enterprise that is integrated with practice. We, in fact, have several forms of action research with which to build, and I think we should continue to develop these newer traditions. At the same time, it would certainly help the over-all effort if there were high quality studies being published in recognized journals. And the quality of our non-traditional efforts might be enhanced by more exposure to academic discourse.

One suggestion made during the Bridging group discussion was that SoL sponsor research on organizational learning themes that would be conducted in academic research traditions. "Sponsor" might mean "provide funding," and to my knowledge SoL does not have money for this purpose. But another drawing card for academic researchers is the network of companies. Access to interesting field research sites is important to researchers. Progress, I think, will require reaching out to researchers working in academic traditions.

One idea raised in our discussion was that we imagine a process of knowing that could satisfy the interests of the several communities. In other words, what process would simultaneously support learning, enable accurate judgments and decisions from a perspective outside the learning activities, and generate knowledge?

If we can envision such a process, it would provide guidelines as we enter projects. For example, one guideline that was suggested was that all parties engage in a joint diagnosis of the situation before a decision is made about how to go forward. The executive among us, as I recall, was skeptical about how realistic that would be--which goes to show that we have more work to do on bridging.

Comments from David Obstfeld:

I thought your summary of the "Bridging" subgroup was excellent. I wanted to add two comments.

First, I see SoL playing an important role as a broker between the academic and business communities. There has been a good deal of research on brokers recently and my dissertation concerns these issues. While SoL is comprised of a diverse group of individuals, they are generally positioned between the academy and the businesses which they serve. In the process of developing effective learning interventions, SoL often imports ideas from the academic world, translates them, and applies them to business settings. We can also play a corresponding role of linking academic researchers to receptive corporations. In the process SoL's own knowledge and impact is enhanced.

Second, in order to encourage quality research in organizational learning, SoL does not have to fully fund research or give away new automobiles. An award between $500 and $2500 can generate a lot of mileage especially if it is offered by a "prestigious" panel comprised of SoL members, industry, and academia. Again such an award would present an opportunity to encourage academic research of quality and interest to SoL, and further establish SoL as an important center for work on organizational learning.

Comment from Mike Beer:

Bridging the two gaps - research - practitioner and practitioner - senior management is the riddle to be solved if it is solvable.

Commitment to co-investigation is the key to bridging the gap. Of course, the hurdle for getting involved is higher and therefore getting started is tougher, but that is the tradeoff.

A way of thinking about this is through the notion of partnership. If a partnership in pursuit of learning can be developed up front there would be no gap. The problem is that most managers are not naturally inclined to be in an inquiry and learning mode. We can achieve partnership by enrolling managers and academics in a common method which specifies the conditions for developing valid data from which the partners are not distant. Or an alternative is to start with a partnership and let the partners develop the method. This approach would suggest that we need research teams composed of management, organizational learning practitioners and academics. The action research design to which they commit would by definition meet the criteria of each community. Jointly managed and funded assessment projects is the model I am proposing.

Finally, I am uncomfortable with talking about learning without defining it. Let me suggest a broad generic definition. Organizational learning is a process of mutual adaptation between the following four forces:

The organization's context - environment, history/culture and stated purpose and strategy of management.
Organization design - structure, systems and human resources policies and practices
Leader and leadership team behavior
People - their needs, assumptions, values and beliefs

Learning involves reconceptualizing or reframing the context and design beyond the capabilities of leaders and people and learning to achieve the new aspiration. Invariably, however, it also means coming to terms with the limits of our learning capability. This sets the stage for modifying the aspiration or replacing organizational members including leaders. The tension between evaluation and learning inherent in this conceptualization is inescapable and is another way of defining the gap that exists between top managers and organizational learning practitioners and between researchers and organizational learning practitioners and I might add between researchers and top management. Normal science research is as distanced from the subject as is management from the subject (their people and themselves).

It seems to me we need a definition of the domain if we are to proceed.

HEALTH CHART

David Berdish
for the "Health Chart Group"

(John Carroll, George Roth)

This project is very, very important to me. I think it is very critical that we give some sort of assessment of learning progress to our leadership team on a regular basis and feedback should be received in a timely manner. I think that in my company (Visteon Automotive Systems, hooray!) we have to launch our new business and adopt to business practices of our customers quickly. There is a sense of urgency to learn as fast as possible, and the leadership needs to know that utilizing organizational learning is a strategic enabler.

I really want to see some work done on a "health chart". When you are in the hospital and the doctor makes the morning rounds, s/he doesn't just reel off numbers and statistics. S/he tells you what the numbers mean, stories that can help explain them, insights into why they occur, and coaching to prevent problems in the future. It is a more qualitative conversation, and requires some intuition, a little spirit, and some prayers to complement the good, hard fact. I believe I need that kind of structure to communicate to my leaders. They need more than metrics-- they need some kind of assessment to get to the deep, deep issues.

Great leaders measure differently than good managers (I said that on Friday in New Hampshire during the dialogue and was quite proud that I did).

Great leaders measure differently than good managers.

Great leaders measure differently than good managers. I believe this will be my rallying cry during this project, and the premise behind a "health chart".

I would like to see us to use some of the research and learning histories already generated by George Roth and Art Kleiner. I will also dig deeper into some of the stories in the Fieldbook. Megan Clark has already begun surveying some of the assessment requirements of learning leaders at Ford Motor Company -- I'd like to include the observations of local folks like Chrysler, Edison, Washtenaw Community College, UM-D, and colleagues that are working on this stuff at AT&T and Shell. I believe we can check our own pulses and play with some of the data. I'd like to see a roll up the sleeves approach to this research, take a turn around the learning wheel, and go out and include some more folks and check some thoughts again. The fundamental question I would ask this group is "how do you know that learning is or isn't working?"

Based on this data, I would like to see us design a basic framework that articulates the insights and desired outcomes and try them out on companies. Make the rounds, so to speak, and see what happens when we talk to them. Art Kleiner drew me a really cool matrix on change and listening -- we could test our "health" in learning against that. Right now I'm in a swirl of concepts and am looking forward to bouncing them off of the group.

Thank you very much for your patience in waiting for this document. I have thought about this every day, and the realize the importance of getting it done. I really appreciated your kindness and hospitality at New Hampshire on January 15-16 and especially the organization and coordination. This is the first MIT-OLC-SoL type meeting I've attended where I've really felt the need to get down to work and felt that I truly was accountable for the effort and the outcome. I also thought you assembled a talented group of people.

NEW TRADITIONS IN ASSESSMENT

Notes from Initial Conversation Group
(Bill Easterday, Louann Reilly, Peter Senge)
Peter M. Senge

This group began to characterize some of the attributes of what a new science of assessment may look like. Below are some first thoughts to seed a "new tradition" of assessment.

FROM	TO
assessment for evaluation	assessment for learning
auditors" seeking to evaluate	researchers seeking to understand
collecting data for someone else	collecting data for my/ ourselves
researchers define the research questions	learners together with researchers define what is wanted
detached/disembodied (knowledge exists abstracted from people)	embedded in social relationships (knowing exists in a community of actors
primary audience for research results is external to organization	primary audience for research results is internal to organization
post-hoc	here and now
presumes "objectivity"	known to be dependent on development, values, mental-models of those involved
other-oriented ("how are they doing?"	self-oriented ("how am I/are we doing?")
imposed on people	invited
(fear of) evaluation predisposes you to self-affirming data	intent of learning opens you to self-contradicting data
"culture of reports" (managing the hierarchy by controlling the information that travels upwards)	"culture of review" (bosses and subordinates reflecting together on their personal roles in producing results)

Appendix C: Homework

All the participants were asked to reflect on the context for beginning the Assessment for Learning Research Initiative and to answer four questions before coming to the Research Forum. Below is a copy of the context-setting letter:

Dear
We have only two days together and a challenging task: exploring and framing a research program into assessment of organizational learning. And -- perhaps most importantly -- engaging each other in ways that lead[s] us into ongoing inquiry and collaboration to develop such a program.
Because time together []is so short and the conceptual and practical issues are many, we ask for a little of your time beforehand to reflect on the project.
Broadly speaking, this project grew out of the recognized need to better understand and document the linkages between organizational learning and business results. This need cuts across all dimensions of the SoL community. Practical business leaders must continually address the question of "What is the return on our investments in organizational learning?" Consultants must likewise, especially those who sustain long-term relationships with key clients. Researchers are naturally (and professionally) skeptical about change strategies which are popular yet lack deeper foundations, and consequently the potential for significant business impact. Few challenges command greater interest among all parties than the challenge to develop more systematic and more usable methods to assess the consequences of organizational learning innovations.
On the other hand, the aim should not be to prove that the investment in learning caused any particular change in business performance -- that would be logically impossible. The aim should be to better understand, given appropriate time frames and conditions, the business consequences of learning initiatives -- consequences for economic performance; for workplace conditions; for relationships with customers; for communities, and other key stakeholders; for the sustainability of the natural environment; and for whatever additional outcomes the members of the enterprise consider important.
Who is the audience for this research? One is senior management and other decision makers not directly involved in learning initiatives but none[]the[]less accountable for judging their effectiveness. The second is people directly involved in such initiatives. Their need to know is somewhat different: how can better assessment lead to better learning? In distinguishing these two audiences it is also useful to distinguish two meanings of "assessment"[:] awareness versus evaluation. (Unfortunately, colloquial use of the term "assessment" is rather ambiguous on this count -- for example, in educational research, it is used as a synonym for evaluation.) For learners, awareness is essential; evaluation is optional, and sometimes counterproductive. All learning processes, whether individual or collective, depend on the awareness of the learners [I don't agree with this -- it presumes, for example, that nonhuman species that learn are "aware"; perhaps we could agree that more complex forms of learning require or are enhanced by awareness]. Feedback, in the sense of heightened awareness and understanding of the consequences of one's actions, is essential to learning. Lastly, our audience includes researchers seeking [enriched] theory based on [deeper] empirical understanding of the processes whereby people in real work settings increase knowledge and improve enterprises.
One of the challenges is how to collaboratively design projects such that the needs of all these groups are met. For example, researchers are prepared to take a long time frame to study complex organizational change processes, while practitioners, and to some degree consultants, must justify investments of people and dollars based [on] shorter term improvements. Different groups also have different emphases on what is being measured and different criteria for determining what constitutes a successful project. Our overall goal is to design and carry out this research in partnership [among] researchers, practitioners, and consultants. Moreover, we suggest that one of our principles should be that whatever assessment approach we are advocating and implementing should also serve our own efforts as a community seeking to learn together.
These are some of the central issues in our thinking. Our conversations so far forewarn us of the many different perspectives that will be brought to the table in the January workshop. Bearing this in mind, we want to create an opportunity for divergent conversation[s] as we conceptualize the project and its aims.
The tension for us is to also provide enough time for exchanging basic information on the different approaches and methodologies sin use now. We plan to lay out a modest number of alternative perspectives on assessing learning -- both what it is and how it can be observed. What methodologies [are we now using] to connect learning initiatives to observable individual and organizational consequences? How do different people operationally define enhanced organizational capacity? What methodologies do we have to connect these organizational consequences and capacit[i]es to business results?
On the basis of these discussions we need to develop some means of integrating across the perspectives. This could take the form of an integrating framework, common principles that characterize different approaches to assessment, or a common statement of purpose to which all can subscribe -- or some combination of the above.
To begin the inquiry and to help us plan the best use of our time in January we'd like you to answer three questions:
1. What could we learn by studying the assessment challenge?
2. What aspects of assessment of learning are most important to you and why?
3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?
4. What readings would you suggest?

Karen Ayas

1. What could we learn by studying the assessment challenge?
This would lead to an inquiry process which raises so many issues to be addressed: Can all consequences of organizational learning be assessed? What actually needs to be assessed? How can we evaluate the effectiveness of learning? Which approach or methodology would be useful for determining the ROI from learning? Can we measure the growth in the learning capacity of an organization? Can we develop practical tools? Will the assessment lead to more "productive" learning?

These are but a few that I can think of and I am positive that in a collaborative inquiry process, many more issues will be raised and addressed. The joint inquiry into the assessment of learning is in itself a wonderful learning process, and the different perspectives can really lead to a deeper understanding of the complexities underlying learning for all those involved.

I believe the assessment challenge is critical for developing communities of practitioners, consultants and academics committed to learning. The theory of organizational learning in itself is neither "convincing" nor "promising" for the majority of enterprises, especially those who seek to have visible benefits when they invest in learning. The assessment of learning is also crucial for identifying learning practices which are beneficial to the organization or effective for improving organizational performance or different purposes.

2. What aspects of assessment of learning are most important?
All the questions I mentioned above and probably many more. I would start with inquiry into the assessment itself. What is the deeper purpose? What do we want to achieve with it? What can organizations achieve with it? Then probe into the organizational consequences of learning and quality of learning. What would be the profile of an ideal learning organization? How can high quality learning be defined or distinguished? Which performance criteria can be related to learning or unlearning? At which integrative level should the assessment be (individual, group, whole?)

Perhaps most important of all, is putting all the knowledge created by the inquiry process into use and develop practical tools which would encourage investment in learning and also lead to more effective learning in organizations.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas into implementation?
Not much I guess, if I feel I can both contribute and learn. If I am intellectually challenged, if there is a real dialogue going on, in a trusting, non-judgmental environment, if I enjoy the process and feel that there is sufficient diversity yet common ground and understanding among the people involved and the joint inquiry leads to "knowledge" creation, what could be more appealing than being actively involved?

4. What articles etc. would you recommend?
Nyhan B., (1991) Developing people's ability to learn. Brussels: European Interuniversity Press.
Burt R. S. "The social structure of competition" in N. Nohria & R.G. Eccles (1992) Networks and Organizations. Boston, MA: Harvard Business School Press.
McKenzie, J. (1996) Paradox: the next strategic dimension. McGraw-Hill (UK).
Nonaka I., and Takeuchi (1995) The knowledge-creating company. New York: Oxford University Press.

Mike Beer

1. What could we learn by studying the assessment challenge?
While measurement and assessment tools in themselves have practical value for both practitioners and researchers, the most important thing we gain by engaging the assessment question is a clarification of what it is we mean by organizational learning. To measure something, we have to be able to be able to define it and develop the construct. By measuring it, we gain a clearer idea about what it is we are interested in and so on. In other words, we will learn what it is we are interested in through an action research process.

2. What aspects of assessment of learning are most important to you and why?
I am interested in defining a comprehensive model or theory of organizational learning and methods for assessing some or all of its components. The questions that interest me are: What do we mean by organizational learning? To what extent are changes in organizational design and processes intended to correct errors? To what extent is it the capacity of the organization and its members to engage in an inquiry process? To what extent is it a cognitive process and to what extent is it an emotional process in which the organization and its members confront their own assumptions and beliefs - how they frame problems or engage them.

It seems to me that that organizational learning involves closing the gap at three distinct levels:
1. The substantive gap - solving the task problem, whatever it is
2. The organizational gap - realigning the organization's design and behavior
3. The organization development or learning gap - learning about how to learn about the other two gaps (this involves group and individual processes.)

A problem that faces us in assessing organizational learning is the difficulty of the learning task. That is, in order for us to understand how much learning has occurred, we need to be able to state what learning needs to occur. That is what the gap was that had to be closed. I am convinced that we need to be able to find a theory that will help us understand the difficulty of the learning agenda.

Finally, I am interested in how organizational learning processes influence and produce change in the organization's leaders and their enactment of leadership.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?
Frankly I am not sure. I have my own stream of research going and so I guess I would be most interested in being involved in a project that furthers my own work in some way. Since my work is concerned with intervention intended to produce organizational change and learning, I would be interested in applying assessment methods and approaches that might emerge from our joint deliberations. And, I would be willing to contribute findings to some higher order joint effort of which I was a part. I guess I am arguing for a loose partnership in which our joint work contributed to assessing the intervention efforts of our clients/research targets (I assume most of us are doing intervention work) and which collectively added up to a body of knowledge on assessment of organizational learning.

4. What articles would you recommend we read?
I do not have a list at hand now, but I think Argyris' work on learning is relevant. Schein's work on culture and leadership is relevant. A recent conference at USC on leadership and change had a number of interesting papers - one by Heifetz and Laurie, one by Pascale, and one by Bob Quinn at Michigan. Cases like Apple Computer: Corporate Strategy and Culture (I wrote this case) offer a great way to engage the question of what is organizational learning. Cases on Hewlett Packard Santa Rosa Systems division describe organizational fitness profiling about which I write in one of the six papers you sent out.

Linda Booth-Sweeney

1. What could we learn by studying the assessment challenge?
One goal for my research is to develop better tools and approaches to help people understand the extent to which they are actually developing capabilities in systems thinking, individually and collectively. This development of systems thinking assessment tools may be an outcome of our work on the assessment challenge.

2. What aspects of assessment of learning are most important to you and why?
One aspect of assessment of learning I'm particularly interested is in assessing "feedback thought", a central notion of systems thinking. How do we develop and assess the development of an ability to think in terms of interdependence, mutual causality, self-reinforcement, balance, stability and instability, structure and behavior while harnessing some of the deepest ideas of the natural, social and behavioral sciences?

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas into implementation?
What would need to occur is for a core group of researchers, practitioners and consultants to put their best thinking into the development of systems thinking development and assessment processes.

4. What articles would you recommend we read?

Michael Chandler and Robert Boutilier (1992) "The development of dynamic system reasoning." in *Human Development, 35,*121-137

Resnick, M. (1995). Turtles, Termites and Traffic Jams. Cambridge, MIT Press.

Richardson, George P. (1991). Feedback Thought in Social Science and Systems Theory.Philadelphia, University of Pennsylvania Press.

John D. Sterman, "Modeling Managerial Behavior: Misperceptions of Feedback in a Dynamic Decision Making Experiment," Management Science, 35, 3 (1989b), 321-339

The following two articles can be found at this web address:
http://www.tiac.net/users/sustsol

Assessing System Dynamics Curricula: Past, Present,and Future by Susan L. Ganter, James K. Doyle, and Michael J.Radzicki

Measuring the Effect of Systems Thinking Interventions on Mental Models by James K. Doyle, Michael J. Radzicki, and W. Scott Trees

Lori Breslow

1. What could we learn by studying the assessment challenge?
It seems to me that simply put, the reason for assessing learning is so that we can do it better (whether we're the teacher or the learner.) The approach to assessment that makes the most sense to me is that it is part of a continuous, informative process that begins by defining goals, monitors how successful both teacher and student are in reaching those goals, and ultimately aids them both in modifying their goals and/or the techniques they use to reach them.

2. What aspects of assessment of learning are most important to you and why?
The little I know about assessment and evaluation has led me to believe the process is a methodological quagmire. Not only does each assessment technique seem to have its own drawbacks, but none seems to be able to answer the question, with any certainty "If learning has taken place, what has most contributed to it?" .

Questions of methodology are important to me for two reasons. First, as a teacher, I want to be able to guide my students' learning in the most effective way possible. Second, as director of MIT's Teaching and Learning Laboratory, I need to be able to persuade scientists, mathematicians, and engineers that the assessment and evaluation of learning is valid. This is a tough job since many of these folks are already biased (to a lesser or greater extent) against social science research for what they see as its imprecision. Convincing them that assessing whether or not their students have learned is important, and that there are ways to judge the merits of various pedagogical techniques and approaches is no small challenge!

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas for implementation?
Three things: (1) that members of the group had similar objectives, concerns, and focus; (2) that there was some expertise already present in the group so it wasn't in danger of re-inventing the wheel; and (3) that there was a commitment on the part of group members to think creatively and flexibly about the challenges associated with assessing and evaluating learning.

4. What articles and/or reports (of approaches, methods, theory, case studies, company studies, etc.) would you recommend we read?
I'm afraid I only know references that are concerned with assessment and evaluation within the setting of higher education. Within that context, one of the standards in the field is Angelo, Thomas A. and K. Patricia Cross, Classroom Assessment Techniques: A Handbook for College Teachers. 2nd edition. San Francisco: Jossey-Bass, 1993. Also, the November 1997 edition of the American Psychologist had a special section on assessing college teaching. That section generated a good deal of controversy on a listserv I belong to for university faculty developers, so I think it's a good place to see some of the intricacies associated with the topic.

Juanita Brown

1. What could we learn by studying the assessment challenge?
We might discover creative and compelling theory, tools, and methods to approach assessment in ways that are appropriate to the lived experience of organizational learning in action. By involving researchers, consultants and practitioners together in the discovery process we have a higher chance of creating interdisciplinary approaches in which the assessment process itself becomes a powerful opportunity for collaborative inquiry, learning, and action.

2. What aspects of assessment of learning are most important to you and why?
The aspect of assessment of learning that is of most interest to me is an exploration of the underlying mental models and philosophies regarding change which underpin our assessment methodologies. For example, appreciative inquiry as an action research methodology is informed by a very different set of assumptions about the nature of change in human systems than is problem-solving as an action research methodology. Both come from the action research tradition but can evoke dramatically different "realities" as they unfold within an organization.

I shall never forget a comment by Noel Tichy from Columbia University in a conversation we had about 20 years ago. He said, "what you view determines what you do." At the time he was talking about how people raised in different disciplines --i.e.. sociology, political science, organizational behavior--literally "see" different dimensions of any situation and therefore interpret "truth" from that lens. It is that "truth" that then informs intervention choices, which then influences the reality that we see.

I'm interested in challenging ourselves to explore the "truths we hold to be self-evident" as a way of looking creatively at multiple ways of assessing the effectiveness of organizational learning initiatives.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas into implementation?
Whether or not I participate in a specifically designated "core group", I see our conversations as relevant and important to my ongoing contribution to SoL through the Executive Champions Workshop which Peter and I co-facilitate and through my ongoing partnerships with organizations struggling to create results which contribute to positive futures.

4. What articles would you recommend we read?

Cooperrider, David L. and Srivastva, Suresh, "Appreciative Inquiry inOrganizational Life" in Organizational Change and Development, Vol. 1, p. 129-169, 1987, JAI Press Inc., ISBN 0-889232-749-9.

Reason, P. and Rowan, J, Human Inquiry: A Sourcebook of New Paradigm Research, 1981, Wiley

Reason, P. and Hawkins, P. Human Inquiry in Action, 1988, Sage.

Megan Clark

1. What could we learn by studying the assessment challenge?
Clear understanding of the difference between assessment and measurement within Ford, things we measure tend to lose meaning over time --- the measuring becomes the goal/objective and we forget the purpose of the measurement in the first place.

Clearer understanding and perspective of what constitutes a learning initiative --- what Organizational Learning is. In order to assess something, we need to be clear on what it is we are trying to assess. Within Ford tying learning and initiative together is dangerous. We always are under "tasks and we are constantly having initiatives to "help" us with our tasks. These are often seen as negative and top-down driven processes, not learning processes. Understanding the methods that are used today -- what works, what doesn't and why.

Understanding how assessments are really used in this context. Assess whether the use of OL tools has an impact on business results. How can we learn to assess/measure Organizational Learning without making it an initiative. Create innovative and exciting mechanisms to tie Organizational Learning and business results something none of us individually would have never have thought of.

2. What aspects of assessment of learning are most important to you and why?
Defining the "portfolio" of Organizational Learning - what do we mean when we say learning initiative? How can we catalogue learning projects simply without loosing context? What are the benefits of Organizational Learning?

How are these benefits measured/assessed in ways that is believable to the business?

What is the business asking for that they don't know to ask?

The ability to assess without loosing the magic in learning, e.g. what impact does the assessment in use change the outcome of the learning in progress?

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

Joyce Fletcher

1. What could we learn by studying the assessment challenge?
I hope it will force us to be concrete about what we are hoping to see, in behavioral and structural terms, when we look for an organization with an enhanced capacity to learn. Also, I hope it will help us articulate some specific examples of non-traditional measures of effectiveness. In addition, I hope it will focus us on process and the way in which we can help clients articulate and design organization-specific assessment measures that capture these non-traditional markers. And lastly, I hope it will shed some light on issues related to diffusion.

2. What aspects of assessment of learning are most important to you and why?
What is important to me in terms of my own learning is to further understand both the theory and practice issues of assessment. In general, I am interested in the invisible work that gets done in organizations -- work that lies outside traditional measures of effectiveness or job performance but is important to organizational success. I think there is a connection between this kind of work, which is related to organizational learning, and problem-prevention and I would be interested in discussing how to go about assessing the way that increasing the organization's capacity to learn prevents problems. In terms of aspects of theory that most intrigue me, I conceptualize organizational learning as an issue that can be better understood when its gender implications are explored (i.e. the kind of organization we envision is more stereotypically "feminine" in nature than traditional organizations) so what is most exciting/important to me is to have an assessment discussion that would help shed light on the gender related aspects of all of this. In terms of practice, the most important thing to me is to be able to hear from others about their current work with organizations, what works and does not work in assessment attempts, and come away with some practical action- oriented ideas that would help me in the work I am currently doing with organizations. I think the issue of assessment is a leverage point that can be used to counter the "anti-learning" forces that change engages, but I am not sure how to go about using it effectively.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas into implementation?
If we could make progress on 1 and 2 above, I would find continuing to work on the issues irresistible.

4. What articles and/or reports (of approaches, methods, theory, case studies, company studies, etc.) would you recommend we read?
A book I have found helpful in thinking about structural changes necessary to support organizational learning is "Sculpting the Learning Organization" by Karen Watkins and Victoria Marsick, 1993, Jossey Bass. Chapter 12 gives a nice summary of their ideas.

Joe Jaworski

1. What could we learn by studying the assessment challenge?
Given the capitalistic system we operate in, the consulting community must come up with adequate means of measuring how well organizational learning is linked to financial performance.

2. What aspects of assessment of learning are most important to you and why?
We need to know how useful our consulting approach is to clients and whether it's really worth the large fees we charge them.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?
My partner Bill O'Brien is addressing this directly in his talk.

4. What articles and/or reports would you recommend we read?
Elsa Porter is a good resource. She has worked and written extensively on government measure of performance -- there are some significant overlaps between her work and this area of assessment.

Tom Johnson

1. What could we learn by studying the assessment challenge?
If the "assessment challenge" refers to efforts to link business programs/activities with specific financial results then I see little to be gained by studying the challenge. Quantitative measurement can help us understand how a mechanical system operates, but not to understand how a natural life system operates. A mechanical system is explainable entirely in terms of its component parts. A natural system can be explained only by referring to the "patterns which connect" the relationships among the system's parts. You cannot quantify and measure patterns or relationships. Quantity is unidirectional. Relationships and the patterns that shape them are multidimensional and, therefore, not quantifiable (however, they can be expressed with number and ratio.)

How can companies "assess" what they do? In my opinion, a human organization must be treated as a life system, not as a mechanical system. Hence, organizations must assess how well they nurture patterns that resemble the patterns which connect relationships in natural systems. To nurture mechanical patterns in a life system, as the almost universal practice of "managing by results" (i.e., driving work with quantitative targets) as most organizations do, is to create unsustainable systems.

2. What aspects of assessment of learning are most important to me and why?
Most references to "learning" strike me as dealing with how individuals and organizations can act more effectively in a mechanistic setting. I am interested in exploring what "learning" might mean in natural life systems.

3. What must occur for me to find irresistible being a part of this assessment group?
I will know it when I see it, but I can't say just now what it might be. I have attended many MIT systems thinking and organizational learning conferences and workshops over the past five or six years, and have not seen it in any of those settings. Too often the emphasis in those settings is on mechanistic analysis of the status quo, quantification, and helping business find better ways to "grow" using additive processes, defined by quantitative measures, not on ways to "enrich variety" in recursive processes.

4. Articles, etc. that I would recommend.
Bateson G. 1979 "Mind and Nature" Dutton. Ch II, especially sections 9-11.
Bortoft H. 1996 "The Wholeness of Nature" Lindisfarne Press. 173-179

Rick Karash

1. What could we learn by studying the assessment challenge?

First, I think we are talking about assessment in the sense of measuring

Actions --> Learning --> Results

...that is, the effect of our actions on learning and on results.

What could we learn by studying assessment?

Increased clarity in our own minds about what we are trying to accomplish in our organizational learning efforts.
Measurement might help us fine-tune our actions.
Proof/confirmation that there are results. Such proof would bring into the fold those who have been bystanders to date.

I have (in my past life) quite a bit of experience of measurement in consumer good marketing, attitude research, and similar. It's clear to me that much is possible in measurement and my initial reaction in coming to this field was that there was under-investment in measurement; that is, if we were serious about learning, we would be measuring more than we were. At the same time, there are naive views that overlook the serious problems in making some of these measures.

What we need is a more informed view across our whole community about assessment.

3. What would need to occur at the workshop for you to be engaged to continue?

I'd like us to have a realistic appraisal of what's possible and not. Also, some clarity of next steps in creating assessment elements in our activities.

4. What articles and/or reports. would you recommend?

I think we should be aware of the four level model for assessment in learning. Author unknown to me...

Level 1: Did they like it (smile sheets, for example)

Level 2: Did they learn what they were supposed to learn (tests in course)

Level 3: Can they do the things they were supposed to be able to do.

Level 4: Does it make a difference in organizational results?

We need a vocabulary for talking about different levels of assessment. This one has helped me in thinking about assessment in corporate settings.

Katrin Kaeufer

1. What could we learn by studying the assessment challenge?

We can learn three things:

assessing the past: "reflection on action"

assessing the present: "surfacing current reality"

assessing the future: "taking the right decision"

2. What aspects of assessment of learning are most important to you and why?

Firstly, I believe assessment is often trapped in the gap between a quantitative and a qualitative approaches. These two approaches stand for two different world views, two different mental models, and paradigms. Organizational learning initiatives seem to imply a qualitative approach of assessment and because of that it is often regarded as "soft". Or, quantitative attempts to assessing learning are undertaken which do not take into account the complexity, the intangible and the long-term consequences of learning processes. I believe that the challenge here is to balance, bridge and combine the two approaches.

Secondly, assessment hurts. In the Roman Empire, the messenger of bad news was killed. Some things have changed but there is some continuity. Learning to deal with "unpopular realities" by naming them and avoiding justifications instead of initiating learning processes, implies a very personal and individual learning process.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

A big temptation would be an initiative which tries to develop a methodological approach combining qualitative and quantitative aspects.

4. Readings

Galtung, Johan. Peace by Peaceful Means. Peace and Conflict, Development and Civilization. Sage. London: 1996 (Galtung has an excellent analysis of externalities in his development theory which could be of great help for developing a qualitative method of assessing)

Hopwood, A. G.. The Archaeology of Accounting Systems. Accounting, Organization and Society, 12/3: 1987, pp. 207-234 (Hopwood describes the origin of accounting: provides great insights about the original need to assessing processes within organizations.)

John Knutson

1. What could we learn by studying the assessment challenge?

As a company struggling to improve processes and results, we need to understand the effectiveness of competing initiatives and factors in driving changes. Over the years, there have been various change initiatives that have been tried, achieved some measure of success, and then gone out of fashion or have died off for lack of a visible results. So often, such programs have been sold as a quick answer to an ill-defined question that, when results were either not forthcoming or not recognized, have been discarded, either canceled completely because of budget cutbacks or superseded by another.

Currently, organizational learning has a small group of devotees (sort of an "underground activity"). Training the skills of the five disciplines has been condensed into a three day mini-course that can be taught at a relatively low cost (no consultants) within the company. There are a number of managers that judge the organizational learning effective and have felt that their organizations are stronger because of the experience. However, there are also other techniques that are used for motivating and changing organizations. There is no one answer to the best way to drive change in the company and no clear consensus or direction that one tool or another is more or less effective. OL has become one of many tools that can be used -- certainly not an activity that is considered a key to ultimate success of the corporation.

An assessment that demonstrated examples of how OL has driven fundamental changes in organizations would be the most valuable result to be achieved. Understanding the cost/benefit of OL initiatives in a number of different settings would be very helpful. Quantitative measurement of such results may be difficult and isolating the impact of OL from other changes occurring would be a challenge. However, if there were an overwhelming number of examples which show how successful OL initiatives have lead to important organizational changes, it is possible that management might elevate and support OL as one of the more important change initiatives within the company.

2. What aspects of assessment of learning are most important to you and why?

Length of time for assessment -- results of the study should be available within a year.
Communication of assessment results should be efficient, understandable, and UN-equivocal in direction setting -- no Ph.D. thesis.
Focus should be on cost/benefit. It may be that small OL interventions yield large relative effects.
Identify which of the five disciplines (and related tools) has yielded the most positive effect. Be analytical in assessing cost/benefit.
Comparing the tools of OL with other techniques and theories about organizational change would be valuable for managers trying to decide what to do.
Systems thinking/systems dynamics has a particular appeal because so often we cannot understand the overall impact of individual actions taken within a larger context. How these disciplines might become more prevalent in companies to help our understanding of how to implement changes efforts would be very helpful.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

Practical, results-focused, intellectually stimulating, positive approach -- and available time, of which there is little.

Dawna Markova

1. What could we learn by studying the assessment challenge?

I am most interested in identifying alternative assessment methodologies including approaches to appraising new individual skills and shifts in operating norms and personal behavior. One way of thinking about these alternative methodologies* is that they focus on:

the "it"-substantive issues such as changes in work process

the "we"-evolution of social networks and knowledge diffusion

the "I"-cultivation of individual capacity to work more effectively within

the whole system.

(*inspired by Otto Scharmer)

My particular interest is in the last domain. To date, no one has come up with any evidence of what is meant by deep personal change other than personal accounts as captured in learning histories. Among the questions waiting to be addressed are: How do we know that individuals within organizations are learning? What do we mean, operationally, by "deep learning?" For example, what are the different ways we can define fundamental change in the individual's ways of thinking? The first step to such exploration would be the development of an easy-to-apply methodology that would gather data to assess change in individuals within a particular context and begin to measure the effects of particular learning processes.

2. What aspects of assessment of learning are most important to you and why?

What needs to occur at the workshop to draw me, irresistibly, into becoming part of a core group, is a true welcoming and collaboration of the diverse perspectives, capacities, and unique contributions of each participant, as well as the opportunity to be of real use.

David Obstfeld

1. What could we learn by studying the assessment challenge?

The nature of the assessment challenge is in part determined by who is doing the assessing. Managers value conciseness but risk simplistic formulations and measures. Normative constructions gloss important theoretical distinctions. Academics value complexity at the risk of abstruseness or irrelevance. As a former Training and Development Director and a current Ph.D. student, I am familiar with some of the strengths and weaknesses of both these orientations. When I returned to school, I was exposed to more complicated organizational learning issues like cognitive vs. behavioral distinctions, learning vs. adaptation, superstitious learning and competency traps, and different levels of analysis that ranged from the individual (Simon) communities of practice and routines to more macro perspectives (absorptive capacity and institutional theory). In the process of acquiring this more complex map of the field and pursuing by doctorate, I have paid less attention to more practical assessment considerations which go back to my roots as a practitioner. I often wonder about the commensurability of academic and practitioner concerns.

I believe it would be valuable to address the challenge by exploring assessment approaches that correspond with different theoretical approaches. The SoL group might begin my identifying several key domains of organizational learning theory that are often folded together under the organizational learning rubric. From there, the group might identify a handful of studies that simultaneously develop theory and assessment approaches. Individually, the different projects would push forward different domains within organizational learning and collectively the effort could provide both academia and practitioners with a rigorous overview of the field that is sorely missing.

2. What aspects of assessment of learning are most important to you and why?

I'm interested in the micro social dynamics of how people construct and occupy social space, build relationships and cross boundaries in order to change or maintain their organizational environment. My dissertation examines the different ways that people build and utilize social networks for the purpose of innovation in research and development. I'm curious about the conditions under which cohesive groups or individual entrepreneurs drive innovation within organizations. The study will also identify different approaches to establishing relationships for the purposes of knowledge creation and mobilization. My study will employ longitudinal network analysis and ethnography.

I am also interested in how organizations learn under varying levels of uncertainty. I am currently participating in an NSF study (with Kathleen Sutcliffe and Sim Sitkin) that is looking at how organizations adapt total quality efforts to the levels of uncertainty they confront. I find this research interesting and important because practices, once successful, often get systematized in a way that prevents their contingent application. The work addresses the struggle to balance methodical with more emergent forms of learning.

Finally, I am interested in mindfulness as a construct for understanding the learning process. I am working on a theory piece with Karl Weick and Kathleen Sutcliffe that explores how certain collective processes are used to develop levels of awareness that prevent errors in high reliability organizations. We are currently discussing how this might be examined empirically.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

The tension between the practitioner and academic perspectives that I alluded to above is one that I carry within myself. My personal career challenge is to develop a theoretically stimulating and rigorous path of research that also offers a practical contribution to my community. I would find a group that would engage in the struggle to balance theory development with practical assessment very attractive.

4. What articles and/or reports (of approaches, methods, theory, case studies, company studies, etc.) would you recommend we read?

The most well-known works by March and Argyris are worth noting. I've found the edited volume by Cohen and Sproull based on their earlier Organization Science issue invaluable. In particular, articles by Brown and Duguid on communities of practice, Cook and Yanow, Cohen and Bacdayan's article on routines, and the literature review by Huber are worth reading. Carol Barnett wrote an excellent literature review on organizational learning several years ago that may have never been published. I can contact her and find out. Denny Miller wrote an interesting piece, "A Preliminary Typology of Organizational Learning: Synthesizing the Literature" (Journal of Management, 1996, 22: 485-505) that offers some valuable observations about the many existing approaches to organizational learning. Lave and Wenger's work on situated learning has also been very important for me.

Bob Putnam

1. What could we learn by studying the assessment challenge?

We could get clearer about what are alternative frameworks for assessing organizational learning efforts and what the strengths and limitations are of each. Second, we could identify the distinctive contributions that we might make. Third, we might make progress toward the broader goal of developing a capability for research that links theory and practice.

2. What aspect of assessment of learning are most important to you and why?

Members of organizations already assess learning and act on their assessments. Those who are involved in learning efforts assess progress in order to make course corrections and to choose their level of commitment. Those who fund the efforts decide whether and how to continue. Others in the organization watch in order to decide if they will join in something similar.

I am interested in engaging members of all three groups in reflecting on how they are making their assessments, so that they might improve the validity of the assessments and of the choices based on them. I imagine developing an offering; call it, reflective assessment. One of the principles might be that assessments based on inferences about the work practices of others should be discussed with those others. This is likely to strike people as both sensible and problematic. It is sensible because assessments made without talking with the people whose work practices and learning are being assessed are likely to be wrong. It is problematic because talking openly about these assessments could lead to defensiveness, hurt feelings, and damaged relationships. This is why performance assessment discussions are so often avoided or smoothed over. Second, when people who have been involved in learning efforts talk with senior decision makers, the senior people may fear that they are getting only testimonials from true believers. Hence, engaging members of an organization in reflective assessment would probably require both facilitation and learning activities. The justification would be both higher quality assessments and decisions and also a likely spill-over to other difficult yet important discussions. Reflective assessment would be a learning effort that itself should be assessed.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

There are two things that I think of as necessary for me to become part of a group to develop these ideas. The first is finding some other people whom I find stimulating and who also want to pursue the ideas. The second is an organization interested in taking part. If I find that first during the workshop, perhaps we can find the second later.

Louann Reilly

2. What aspects of assessment of learning are most important to you and why?

The research questions in the introductory letter address my interests, i.e., what methodologies can be used to connect learning initiatives to observable individual and organizational consequences, and what methodologies do we have to connect these consequences and organizational learning capacities to business results. How do we assess groups vs. how do we assess individuals? Is individual change (learning) a prerequisite to, or a result of, group learning, or both -- what are the circumstances that determine this relationship?

My primary interest is in assessing systems thinking capacity and the link to business results. The two dimensions I am most curious about are 1) assessing the ability and impact of being able to see things differently (mental models changes), 2) assessing the ability to act differently. Also, what methodologies do we have for "whole system assessment" ( the inclusion of consequences distant in space or time) in measuring ROI? I am interested in the cyclical relationship between assessment and learning -- how can better assessment lead to better learning, which in turn leads to better assessment, etc.

George Roth

1. What could we learn by studying the assessment challenge?

By considering the assessment of learning, I believe we need to become more sophisticated in thinking through how the development of skills and capabilities among a group, team, or organization of people link to action (behavioral changes) and how those behavior changes link to business outcomes such as produce, service, and financial performance. The challenge which organizations face is in their quest to know how they are doing. They over specify behaviors by virtue of how they specify and use metrics, so that learning is diminished in favor of performing for the measure (see article, "When Measurement Kills Learning".) The persistent dilemma which I think we all need to consider is the tension between a need to measure or assess and the specification of performance that comes when you measure.

2. What aspects of assessment of learning are most important to you and why?

The connection between learning as a process in organizations to the overall business and financial performance of a firm. While learning is a process that develops, and perhaps liberates people, it has not been clear that these people then collectively operate so as to improve the financial performance of the firm. While I believe, and it makes intuitive sense, that developing organizational learning processes within firms will lead to improved business results, there is little evidence and understanding of other factors that are important in realizing the link between collective learning and organizational performance and profitability.

The phenomena which I am most interested in conducting research around learning processes is the abilities of organizations to diffuse best practices (where one unit in an organization has achieved significant business benefits, and the techniques, skills, or processes that are believed to have caused those results) are diffused to other units seeking to achieve similar significant improvements and business benefits.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

I would need to have funding support so as to sustain inquiry into this question and support my ability to collaborate with other researchers in working in (a) company project(s). Some of the specific information on who I would construct such a project is published on the web at http://ccs.mit.edu/lh.

4. Key References:

Experimental Learning, Experience as the Source of Learning and Development by David Kolb, Englewood Cliffs, NJ: Prentice Hall, 1984.
Organizational Learning II, Theory, Method and Practice by Argyis, and Schön, Reading, MA: Addison-Wesley, 1996.
The Clinical Perspective in Fieldwork, Ed Schein, Sage Publications Newbury, CA 1987.
Process Consultation, Vol. II, Ed Schein, Addison-Wesley: Reading, MA 1987.

Dennis Sandow

1. What could we learn by studying the assessment challenge?

If the membership of the SoL becomes cognizant of the effects of learning I believe that it will offer us these insights:

The effects of learning are best studied in vivo as employees create value added through learning.
Unlike data and information, knowledge is created by two or more people (it takes two to tango and to knowledge!), or a social network, and is a social process.
Our view of creation of capital shifts from a physical capital mental construct to a human capital construct. This will lead us to understand that capital can be optimized through the continuous creation of knowledge.
In the process of our inquiry we are navigating a new path away from the use of command and control as a means of organization to a new understanding of social systems organizational capacities and their capacity to create.

2. What aspects of assessment of learning are most important to you and why?

Our philosophical orientation is important to me. Because I have acted from a philosophy which evolved from a belief that social systems that were stable, could be reduced into a fixed set of functions and can be optimized through a external "manager(s)". If I consider the extent of my philosophical bias, I will approach assessment of learning as something I have yet to learn rather than something I have to explain or prove.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

New relations would need to evolve which result in new patterns of collaboration. This collaboration would be based on an integrated study of learning systems within member organizations.

4. What articles and/or reports (of approaches, methods, theory, case studies, company studies etc.) would you recommend we read?

From my own experience you might consider the following cases:

The performance history of a company I started in 1978. Dynatron Incorporated recruited people whose lives were at risk in state hospitals. By 1982, we had developed statistical process control manufacturing, precision training and supervision, just-in-time inventory relations, material management research and new product development with our customers. We had also constantly increased the productivity of our company.
A study of social autopoiesis documenting the employment and support of an individual who was raised in a state institution. His co-workers in a silicon wafer manufacturing plant created a learning network which 1) saved the State of Oregon $100,000/year and 2) increased the new workers productivity enabling him to leave the welfare system and live on his own.
Two studies which demonstrated that the network of those implementing performance improvement strategies dissipated in the study phase of the Deming (plan-do-study-act) spiral.

Reading list

Bailey, K. D. (1994). Sociology and the new systems theory: Toward a theoretical synthesis. Albany, NY: State University of New York Press.
Batten, D., Casti, J., & Thord, R. (1995). Networks in Action: Communication, Economics and Human Knowledge. New York, NY: Springer-Verlag.
Capra, F. (1996). The Web of Life: a new scientific understanding of living systems. (345 ed.). New York: Anchor Books.
David-Neel, A., & Yongden, L. (1967). The Secret Oral Teachings in Tibetan Buddhist Sects. San Francisco: City Lights.
Dupuy, J.-P. (1989). The autonomy of social reality: On the contribution of the theory of systems to the theory of society. World Futures, 27, 153-175.
Gutpa, M. M., & Yamakawa, T. (Eds.). (1988). Parallelism, integration, auto coordination and ambiguity in Human Support Systems.
Jung, C. (1971). The difference between Eastern and Western thinking. In J. Campbell (Ed.), The portable Jung (pp. 659). New York: The Viking Press.
Marsick, V. J. (1990). Altering the paradigm for theory building and research in human resource development. Human Resource Development Quarterly, 1(1), 5-24.
Maturana, H. R., & Varela, F. J. (1980). Autopoiesis and Cognition: Realization of the Living. Dordrecht, Holland: D. Reidel Publishing Co.
Mingers, J. (1995). Self-Producing Systems: Implications and applications of autopoiesis. New York: Plenum Press.
Nappelbaum, Z. &. (1989). Cognitive Equilibrium: On pattern stabilization and knowledge organization in decision-producing networks (pp. 7).
Nonaka, I. (1991). The knowledge-creating companies. Harvard Business Review, November-December, 96-104.
Sakaiya, T. (1991). The Knowledge Value Revolution. Tokyo: Kodansha International Ltd.
Sandow, D., & Rhodes, L. (1996). The Wizards Curtain: A reply from Oregon. Human Resource Development Quarterly, 7(2), 185-192.
Snow, R. M. (1993). Crisis as status quo: A systems approach to understanding institutional organizations during a period of societal transformation. Human Systems Management, 12, 179-191.
Thurow, L. C. (1970). Investment in Human Capital. Belmont, CA: Wadsworth Publishing Company.
Varela, F. G., Maturana, H. R., & Uribe, R. (1974). Autopoiesis: The organization of living systems, its characterization and a model. BioSystems, 5, 187-196.
Von Foerster, H. (1981). Notes on an epistemology for living things, Observing systems (pp. 258-265). Seaside, CA: Intersystems Publication.
Von Foerster, H. (1984). Principles of self-organization- In a socio-managerial context. In H. Ulrich & G. J. B. Probst (Eds.), Self-organization and management of social systems: Insights, promises, doubts and questions (Vol. 26, pp. 155). Berlin: Springer-Verlag.
Zeleny, M. (Ed.).^(Eds.). (1981). Autopoiesis: A theory of living organization. New York: North Holland.
Zeleny, M. (1987). Management support systems: Towards integrated knowledge management. Human Systems Management, 7(1), 59-70.
Zeleny, M. (1988). Knowledge as a new form of capital. Human Systems Management, 8(1), 45-58.
Zeleny, M. (1995). Human and Social Capital - Prerequisites for Sustained Prosperity. Human Systems Management, 14(4), 279-282.

Tim Savino

1. What can we learn by studying the assessment challenge?

This question of the expected ROI on organizational learning activities won't go away! However, I'm not convinced that it is a question that can be answered easily, if at all. There seem to be several problems in assessing the impact "organizational learning" activities: 1) There is no consensus on a definition of what constitutes "learning" at an individual level (though numerous definitions have been proposed); (2) This confusion is compounded when we start talking about "organizational learning". How can we assess an organization's capacity to learn if we don't have a shared understanding at the individual level? (3) When we focus on organizational results we run into the problem of trying to justify activities in terms of "cause and effect" (this is generally a losing proposition in terms of most any activity, let alone one as fuzzy as organizational learning!); (4) Many of the learning initiatives that I've seen defined are poorly framed; that is, the intention of the work is not made clear up-front, so subsequent assessment efforts are made more difficult. (To paraphrase Pogo: "If you don't know where you intend to go, any road will get you there."); (5) We don't have assessment methodologies capable of measuring the impact of this work (like trying to examine the stars without a telescope; we know they're up there but it's all kind of fuzzy).

I think a study of this subject could yield some insights/clarity on those items noted above. What is "this work" all about? I might add that a poll of my colleagues leads to one common conclusion: We have sustained our involvement because we believe this is the right thing to do...it "feels" right...we know intuitively that systems thinking leads us to a different level of decision-making.

2. What aspects of assessment of learning are most important to you and why?

I think we need a shared definition of this work. Assessment implications will follow. I am most interested in defining methodologies we can use to replicate this type of work in other projects, organizations, and communities. How do we come to understand the impact of creating "learning fields" throughout the organization?

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?

I believe there are elements of this work that are not measurable in the traditional ROI sense. It seems we can only get at these intangibles through self-reports or possibly learning histories. The "soft stuff" may be the toughest to measure, but yields the highest return. I would be intrigued if we can find a way to define, replicate and assess this dimension.

4. Recommended reading:

I think the story of Interface (carpet company) is a fascinating account of a company that has completed reinvented itself and worth study. Other related topics on field theory and some of David Klob's work over the years on Learning Styles offers a simple and useful way to think about learning in the context of action/reflection.

Bill Torbert

1. What could we learn by studying the assessment challenge?

By studying the assessment challenge we can learn:

how to differentiate among visioning, strategizing, performing, assessing;
how to differentiate among assessing: visioning, strategizing performing, and outcomes;
how to assess each of these four qualities both in real time ("during-the-act") and by use of measures and reflective analysis ("after-the-fact");
how to construct personally and organizationally potent analogs that highlight harmonies and incongruities among the four qualities;
how to correct incongruities; and
how to design learning systems that interweave attention to vision, strategizing, performing, and assessing more and more fluidly, justly, and transformationally.

2. What aspects of assessment of learning are most important to me and why?

To me, the most important part of assessment of learning is the recognition that no single specific outcome represents success in learning, in social contribution, or in the long-term sustainability of the organization. Rather, the important issue is the ongoing realigning among vision, strategy, performance, and outcomes.

3. What must occur for me to find irresistible being a part of this assessment group?

For me to find it irresistible to become part of a core group to continue this project, several things will have to happen at, or soon after, the workshop:

in terms of visioning, I will need to feel that others share my commitment to transformational mutuality in our personal, professional, and scientific relations;
in terms of strategizing, I will need to feel that the evolving "we" wishes to engage as a community of inquiry for a relatively long time (seven or more years);
in terms of performing, I will need to feel that both we and our principal clients as time goes on, are committed to cultivating timely aesthetic action that enhances moment-to-moment awareness; and
in terms of assessing, I will need to feel that we are generating real organizational learning for clients scholarly contributions that multiple that learning, and significant income that supports additional writing time for me.

4. Articles etc. that I would recommend:

Argyris C. 1994 Knowledge for Action. San Francisco: Jossey-Bass
Argyris C, Putnam R, and Smith D. 1985 Action Science: Concepts, Methods and Skills for Research and Intervention. San Francisco: Jossey-Bass
Austin J. 1997 "A method for facilitating controversial social change in organizations: Branch Rickey and the Brooklyn Dodgers" Journal of Applied Behavioral Science . 33; 1:101-118
Barrett F & CooperriderD. 1990 Generative metaphor intervention. Journal of Applied Behavioral Science . 26; 2: 219-239.
Cooperrider D and Srivasta S. 1997 "Appreciative Inquiry in Organizational Life." In R. Woodman and W. Pasmore, Research on Organizational Change and Development. Greenwich, CT: JAI Press
Harrison R. 1995 Consultant's Journey: A Dance of Work and Spirit. San Francisco: Jossey-Bass.
Heron J. 1996 Cooperative Inquiry: Research into the Human Condition. London: Sage.
Kegan R. 1994 I n Over Our Heads: The Mental Demands of Modern Life. Cambridge, MA: Harvard University Press.
Nielson R. 1996. The politics of Ethics. New York: Oxford University Press.
Porras J & Collins M. 1995 "Building a Visionary Company". California Management Review 37; 2:80-100.
Reason P 1995 Participation in Human Inquiry. London: Sage.

Etienne Wenger

1. What could we learn by studying the assessment challenge?

While it is fairly uncontroversial to assert that the performance of an organization is a function of the learning that takes place within it, we still need better causal stories that connect everyday activities, interactions, and conversations to the performance of an organization. We must develop good ways of analyzing the sets of organizational conditions and factors that allow a learning initiative to have a lasting effect on the qualities and capabilities of an organization. We need a new vocabulary to talk about the learning potential that exists in organizational structures. If studied right, the assessment challenge goes to the heart of the organizational learning question: how do organizations learn how to invest in learning and in the process learn how to learn?

2. What aspects of assessment of learning are most important to you and why?

I am most interested in the learning potential that exists in a constellation of communities of practice and in the social capital that various communities have accumulated over time, both internally and in their interconnections. Assessing this potential involves relations of core and peripheries, of sustained interactions inside communities and at their boundaries. The general questions include: How can we talk about the potential for knowledge accumulation as well as innovation, adaptability and creativity that exists within an organizational structure in ways that make sense both with respect to the goals of the organization and with respect to the experience of the various constituencies involved? How must we define success so that we can talk intelligently about this learning potential and its effects on the organization's future? How do we understand this potential in terms of the organization's evolving ability to participate successfully in broader social structures and processes, which shapes it, but which it also contributes to shaping (e.g., an industry, a knowledge ecology, the economy, society)?

3. What would need to occur at the workshop for you to find irresistible becoming part of a core group that would develop these ideas to implementation?

Everyone would probably agree that the assessment challenge is important, because assessment is part of organizational life; but the question of what to assess remains open. The situation reminds me of the story of the guy who was looking for his keys around a street light. A passerby offered to help and asked him where he had lost them. The guy showed him a spot at a certain distance. "Then why are you looking here?" asked the passerby. "Because here there is light," was the answer. Measuring often has this quality: we focus on what is easy to measure rather than on what we know matters. I guess what would excite me most would be to leave with a sense that we are collectively ready to address the assessment challenge in its actual complexity and not some watered down version of it. I would also like the workshop to go beyond an interesting exchange of ideas to define a path forward so that we can imagine how our own contributions can fit in the whole.

4. What articles and/or reports would you recommend we read?

At this point, I cannot think of many articles or reports that would be both short and specific enough to be essential reading for this particular topic. I have a short piece in the July/August 1996 issue of the HealthCare Forum, "Communities of Practice: The Social Fabric of a Learning Organization." There is also a very short piece by Thomas Stewart "The Invisible Key to Success," in Fortune, August 1996 issue. But neither piece addresses assessment very directly. The study of informal learning by EDC started out with the goal of providing assessment in terms of business results, but ended up being mostly a taxonomy of activities that support informal learning; and their report is quite long.

Appendix D and E: Guest Talks

NINE FRUSTRATIONS OF A CEO: CONNECTING VALUES AND BUSINESS PERFORMANCE

Bill O'Brien

Address given to the First Forum of the SoL Research Initiative
Assessment for Learning
University of New Hampshire
January 14, 1998

Bill O'Brien, retired CEO of Hanover Insurance Company, has given much thought to how investments in values and fundamental new capabilities on the part of the people of an enterprise connect to business performance; how the spirit and genuine commitment of the people in the enterprise underlie the success of the enterprise. In his 20 years at Hanover, he was part of a remarkably successful turnaround story, effectively remaking a troubled institution into one of the best-performing U.S. property and liability insurance companies in both growth and profitability. In the opening address of the Society for Organizational Learning's 1st Research Forum, held January 14-16, 1998, Bill shared nine frustrations he has experienced as a manager and CEO. Bill offers some interesting, intriguing, and challenging ideas for both the researcher and the practitioner. He does not tell us what we should do, but rather encourages us to think more broadly about the deepest questions and potential contributions that lie at the heart of this initiative on assessing organizational learning.

I that believe SOL's "Assessment for Learning" Research Initiative has the potential to make a significant contribution toward the long-term transformation of our institutions. But before beginning a project of this magnitude, it's important to step back and first think about the underlying principles that should inform the work. To me, a fundamental statement is that corporations are intended to create wealth, and that creating wealth is honorable work.

Now why should I bring that up? Because I see two camps of people in our organizations. One group is constantly proclaiming the bottom line and preaching about profit, while the other group sees themselves as "human development people" and focuses on things like team building and communications. The people in this second group may acknowledge that profits are necessary, but they don't really internalize the need to create wealth.

These two warring camps must perform in alignment for a corporation to be successful. Human development is crucial-in fact, I believe right to the bottom of my shoes that human capital creates financial capital, not vice-versa. A bottom-line focus is equally important to the creation of wealth, even though it oftentimes gets a bad rap. I think that what we find so offensive in what we call the bottom-line mentality is that it's so often abused by individuals who really mean, "I want you to do this because I want to look good." That is a mentality that we need to bust open. In fact, it's important to get the people in both camps to stop paying lip service to the words "profit" or "development" and to begin deeply internalizing them.

One other foundational principle: when I use the term "learning organization," I'm referring both to the acquisition of knowledge and the practice of virtue. The two cannot be separated. These are the types of things we need to be talking about in our organizations-values, vision, and moral formation.

Having laid a foundation for our discussion, I'd now like to explore some of the challenges that we face as we think about how to assess organizational learning. And I'd like to do that by describing a set of frustrations that bedeviled me for 12 years as a CEO, and many years preceding that as a manager. I lump these frustrations in packages of three.

FOG, GESTATION PERIODS, AND TIME HORIZON

Frustration #1: There is too much "fog" at any one time to differentiate clearly between an individual organization's business performance and the larger ebbs and tides of the business cycle.

To understand how this plays out, let's use the measurements of profit and growth, since it's impossible to fudge them over any long period of time. Growth means the customer is voting for you, and profit means the capitalist is voting for you. You've got to run a good company to achieve both of these measures over decades.

Now consider the type of growth and profit performance you might see from different companies over a 50-year time frame. No company grows steadily with no interruptions. Some companies have a slow, steady growth over a long period of time, though even they occasionally experience times of slowdown or drop-off in performance. Companies like Coca-Cola, 3M, Procter & Gamble, and Hewlett Packard come to mind. Then you have organizations that grow for a period of time, but eventually falter and are less successful over the long term. I put companies like Westinghouse, Sears Roebuck, and Prudential Insurance into this category.

The challenge I would like to present to the group here today is to identify when an individual company is reaching a plateau that indicates an impending decline, versus a company that is simply building momentum for further long-term growth. If this already seems difficult, go one step further and consider the larger industry context in which this occurs. For instance, if you are in the automobile business and industry demand drops from 12 million cars to 10 million, no matter who you are, your growth rate is going to suffer.

So, particularly in mature businesses, you really have to think about whether a downturn is simply part of the business cycle, or whether it is the result of management no longer putting anything in a pipeline-they're taking out investments (capital, people skills, R&D, etc.) instead of putting them into the company.

Frustration #2: We need better understanding of the lag times between cause and effect.

Everything has a gestation period. It's an old story: no matter how many people you put on the job, it still takes nine months to make a baby. Even though we might understand this idea in principle, we really don't understand gestation periods. Just as we underestimate how management decisions made a decade or more ago contribute to the results we see today, we significantly underestimate the gestation periods involved in creating wealth. For example, it took generations to build the Coca-Cola brand name. Because of that, I think you could put eight baboons at top of Coca-Cola today and with a good PR department, we wouldn't know it for 7 or 8 years! I actually have a lot of respect for Coke management, and know that it's not being run by baboons. I just want to illustrate the power of these long-term investments.

I'd like to see us develop more understanding of lag times, in part, because there's an awful lot of energy being frittered away because of unrealistic expectations. At Hanover, I used to use the expression, that "managers always want to pull up the radishes to see how they're growing." We simply don't allow enough time to see the payoffs from a program. We need to take a mature look at how long it takes to see a payoff from an investment or an action-whether it's transformation or strategy.

Let me tell you a personal story that illustrates the nature of time delays, and the tensions they create. At Hanover Insurance, we began to move into value-based governance around 1972 or '73 (although we didn't actually call it that until 1976). In the early years, people in the organization were saying, "Why are you running week-long seminars on values? Nobody else is." Despite those comments, we stuck with it, because we felt it was important for our evolution as an organization. When I succeeded Jack Adam as CEO in 1979, I inherited a very healthy company because of these foundations he had laid.

Then we ran into a fellow by the name of Peter Senge, and a man named John Beckett, who was a professor here at the Whittemore School at the University of New Hampshire. With their help, in 1981 and 1982, we began exploring how Eastern thinking and systems thinking might help our organization. Again, many people said, "This is crazy, putting hundreds of people through this kind of gobbledygook!" But because of the work we had done in 1973 with values, there was a foundation for this type of work. Building on that, we were able to begin removing the politics and the bureaucracy and the verbal gamesmanship in the organization. The result was that people began settling claims better. They were creating better relationships, and seeing better results.

As with any long-term project, we had to deal with people's impatience. It takes a long time to begin to see the payoff, and in the meantime, people question why you are investing in something that doesn't seem to be providing any obvious benefit. Without an understanding of gestation periods, you can see why managers pull up the radishes. But, if you keep pulling up the radishes every three or four days to see if they're growing, you'll destroy what you're trying to grow over the long term. We need to allow enough time to see the payoffs from a program, but even more we need to develop the understanding that will lead to this patience.

As our experience at Hanover showed, once you get the train running, you have results coming through the pipeline that provide you with the income margins to cover new investments. But if you don't continue to make these long-term investments, you'll be in the same boat as many CEO's are today: they have spent so much time down-sizing and cutting things out of the organization, that there is nothing left in the pipeline.

Frustration #3: The war between the short-term and the long-term.

There are a lot of smoke and mirrors in short-term results. I believe that you can take the same set of circumstances and get a ±120% difference in the earnings report, depending on how conservative or liberal your accounting approach is. That's a 40% variance in what are supposed to be "hard" numbers!

I faced this problem in the 1980's, when the insurance industry was grossly under-reserved because of the legal liability explosion in our country. I felt very consciously that we had to build up our reserves. But in so doing, I watched our margin of betterment relative to the industry average shrink from 8 points down to 3 points because most of my competitors were following the industry trend of under-reserving. We paid for this in the short run, in order to maintain a strong foundation for long-term performance.

The insurance business isn't the only industry that gets into these problems. When the Japanese invaded our auto markets, most auto companies ceased funding their pension plans. You might say, "Ah, so what? Times are tough so we borrow from the future." I say, "When you b.s. upward, you b.s. downward." In other words, when you snow the shareholder you create a lack of credibility within the organization. People know what you are doing. You risk fostering cynicism if people see you manipulating short-term results to make the people in power look good. Moreover, you risk creating a lack of credibility in how your organization is viewed in the financial markets.

So I think we have a challenge to stop, or at least honestly identify this practice of borrowing from today for tomorrow. What you are really doing is borrowing from your future financial health, and it catches up with you six, seven, or eight years out. That's what the short-term, bottom-line profit mentality really does. It's got nothing to do with creating long-term wealth.

I don't think it's fair to blame this short-term profit mentality solely on Wall Street. Obviously, brokerage houses are short-term oriented, because the more stocks are bought and sold, the more commission they earn, so they don't really want people holding stocks for ten or fifteen years. But I believe that is only part of the story. I think many times when chief executives tell you about the pressure they feel from the board to produce short-term results, what they're really telling saying is, "I don't know how to explain our situation so that they can see a long-term payoff." I also think those people who proclaim that they are managing for the long-term are frequently bull artists who are just trying to get a couple more years into their pension plan. We need to differentiate between the group of people out there that are really creating wealth, versus the people who are very superficially touching the buttons and only talking long-term because it's faddish.

SELF-SCORE KEEPING, VIRTUE, AND LEANNESS

Frustration #4: Self-scorekeeping seems to be more temptation than management can handle.

The obvious response to my frustration over the "smoke and mirrors" used in reporting short-term results is, "That is why we have public accounting firms." But my response to that is, "So what!" The power that organizations currently have to hire their own auditor, combined with the influence they have over those auditors and the latitude they have in reporting the quality of earnings is frightening. The biggest driving cultural force in a public accounting firm is billable hours. The last thing a young accountant wants to tell the partners is that he's lost an account because the company decided to shoot the messenger. The only thing that keeps any decency in the audit process is the threat of malpractice or professional liability suits.

I think there is a fundamental flaw in this system of self-scorekeeping. Imagine what would happen in the NBA if each team kept its own score. How would we ever know who actually performed better at the end of a game? Yet that's basically what we do in business. This is not to deny that a lot has been accomplished in the last 20 years in terms of disclosure. But the progress has been slow and painful-year-by-year, item-by-item. We disclose more now than we ever did. Maybe we do enough now. But to find out information as an investor, you've still got to be the world's greatest footnote reader. The real information isn't on page two of the annual report. For example, if you want to look at the balance sheets of the corporation, you have to know all the different corporate names which are usually in different states and countries. In point of fact, if management wants to hide something, it's practically impossible to find it.

Frustration #5: Our big problems in corporations are due to an inadequate practice of virtue, not a lack of knowledge.

We have a lot of very smart people in our companies, but the problem is that nobody believes them. Why? I grew up in an era when we were trained in "spin." The unintended consequences of this is that if you spin people two or three times, they no longer believe you. Worse yet, if you're high up in the company, not only don't they believe you, but they don't even tell you they don't believe you. They'll nod their heads and act as if they believe everything you are saying, but the truth is, they don't.

Pursuit of power, greed, manipulation of information, and the mentality of spending other people's money are rampant in our corporations. So I don't think that creating learning organizations is going to be accomplished just through knowledge or teaching new skills. Instead, I think there needs to be a deeper moral formation in our organizations. We currently pay scant attention to this aspect of leadership. Most of our leaders come up through marketing, engineering, sales, or law. But where is the moral training in those fields?

Now, combine this with some of my earlier points about gestation periods, about investing or disinvesting in long-term wealth creation, and about manipulating short-term results. My experience is that the people at the front lines usually know when a company is investing or disinvesting. They also know when management is being honest with them. So, when they hear one thing and experience another, they know that the real problem is a deficiency in leadership. It's the politics of the company, which have squeezed the courage and honesty out of people. It's the mentality that says, "Go along with me or your future will be at risk." It stems from our desire for instant gratification and immediate results.

I'm certainly not suggesting that we create some sort of device whereby one group of people measures another group's virtue. I think that would be even more dangerous than the low virtue we currently have in our organizations. I'm simply suggesting that, when thinking about learning and assessing learning, we have to be supportive of moral formation as well as business acumen.

Frustration #6: People don't seem to understand that leanness, while it reduces the comfort level in the short-term, actually increases well-being in the long-term.

At Hanover, we adopted seven values, each one focused on curing a particular corporate disease. For example, we had a value called merit, because we wanted to put a stigma on politicking. We had a value called openness, through which we wanted to reward the ability to push bad news up the organization. And we had a value called leanness, which was intended to fight corporate fat.

Why leanness? Because we noticed that, when there was a corporate turnaround or a crisis, or during the founding generation of an organization, there was leanness in a company. But as you got into two or three generations down the line, or economic times got prosperous, there was a tremendous temptation to get fat. It's unbelievable to see how, when profits go up 13, 14, or 15% in a year, everybody seems to need more people in order to expand and invest. It starts at the top. It's almost contagious, given the profligate lifestyle at the very top in corporations. And it just works its way down.

Leanness doesn't mean cheapness, or low salaries, but it simply says that for every dollar you spend, you should get a healthy return. Spend money like it's your own money. This "waste not, want not" use of resources ought to be embedded in all organizations.

I also think leanness can be very important as a foundation for how we deal with people. I really worry about why so many companies seem to be able to lay off 10,000 people, but cannot lay off one person. These are companies where when a person flunks out of job A, they get moved over to job B and then C, as long as times are good. But when times get bad, they lay the whole department off. I think these companies have missed an important notion of regularly weeding the garden for the common good.

When we took Hanover on its journey, we had to do weeding out every year of those people who just weren't meeting the standards. We simply didn't know of any way to ensure that 100% of the people we hired would meet our standards now and in the future, particularly as the company was growing and demands on them got higher. This created a dilemma, especially in an industry that is considered to have cradle-to-grave employment policies. I addressed this by saying to people, "I'd love to give you a womb-to-tomb contract, but I can't, because in the long-term, economic laws will overrule any contract I write. Instead, I'd like to make the following covenant with you: as long as you are with the company, we'll do everything we can to support your growth and development as a human being, and in turn we'll ask you to do everything you can to make us a better company. I hope you're still working here when you're 65, but if you're not, I'd like you to leave this organization far better able to move into the market than you were when you came."

QUANTIFICATION, AVOIDING FADS, AND FOSTERING A LEGACY MENTALITY

Frustration #7: Managers want to quantify everything, but there's a danger in using proxies to quantify things that can't be quantified.

When we use proxies to represent what we cannot truly quantify, everybody then rallies around that proxy, and the unintended consequences that result usually are devastating. For example, I wish I had a nickel for every time people said to me, "Hey, we're putting in these values. How the hell are we going to know if they're working?" I don't think you can quantify or measure things like "openness" or "truth." If you try to measure them within a salary system, you're going to get pseudo-compliance and less authenticity, when the goal is really to have more authenticity.

A value is simply not a value unless it's voluntarily accepted. If it's coerced or forced via a salary or a promotion program, that's dangerous. So much pressure to quantify the non-quantifiable is simply time wasted. I believe I can go in a values-based company and talk to 15 people and know whether or not things are working. I spent my whole adult life walking into branch offices of insurance companies, and it didn't take me long to get a reading on how things were going. It might take me forever to try to fix the problems, but I could get a reading by just interacting with a dozen people and having some serious discussions. The bottom line is this: there are many things you can know at a gut level that you simply can't quantify.

Frustration #8: The pressure to follow fads.

I think that some of the greatest contributions Jack Adam and I made to Hanover Insurance was to avoid dumb fads. This wasn't easy. There was no reward in it. For the two or three years that a particular fad was hot, everybody was looking at you as if you were retarded for not jumping on the bandwagon. Now whenever I'm in a search committee for a CEO, I ask myself, "Will this person be able to avoid the temptations to do the popular, dumb, fashionable, trendy things that come down the pike every four or five years?" A big part of success comes from simply not diverting resources into those kinds of things.

Frustration #9: The difficulty in fostering a legacy mentality in top management.

I would like us to find a way, through measurement devices, to encourage a legacy mentality at the top. I think most of us buy into what I call a "chopping wood" syndrome in our early life. We see ourselves as piling up wood from our direct efforts, and equate our effectiveness with how much wood we piled up today. That's a normal and healthy work approach in one's 20s and 30s. But it is a mistake when managers continue this attitude long after they rise to the top level of organizations.

A related mindset is what the Navy drilled into everyone in World War II: if it happens on your watch, it's yours. This leads people, including senior management, to obsess about the results achieved or the problems avoided while they are in the job. At the very senior level, this approach does more harm than good.

Instead, a maturing transition that senior management teams must go through is to begin seeing themselves as building something to leave for the next generation. This is a little like the transition between being a parent and being a grandparent. You love you grandchildren, but you take your responsibility a little bit differently. You worry a bit less about what they did today and look instead at their development over time. Unfortunately, there are too many senior executives who are still piling up wood, and this interferes with their ability to develop a legacy mentality. The top tier of people in our major corporations ought to spending a significant amount of time preparing the company to be handed down to the next generation.

I think we cannot just point fingers at corporate leaders on this issue -- it's getting harder and harder to find any sort of real legacy mentality in our culture, too. There's a popular bumper sticker down in Florida that says, "I'm Spending My Grandchild's Inheritance." I am not going to quibble about whether you should take a vacation or pay your grandchild's tuition, but I think we have a very powerful strain of thinking in our society that says, "The hell with my grandchild!" We see this in the difficulty of finding funding for public schools, yet we're building prisons to beat the band.

Now I don't have any illusions that a single parent struggling to make a living is going to spend a lot of time thinking about the next generation, but I believe that the people at the top of our large institutions have a responsibility to think of that corporation through the coming generations. I certainly believe that the people who handed us the current infrastructure of American corporations thought about future generations.

Ironically, I don't think that investing for the next generation has to be done at the expense of current financial performance, contrary to what many seem to think. In fact, I see the absence of a legacy mentality leading to a great deal of foolhardy spending that is also pulling down short-term performance. Let me give you one example that is close to our concerns here. If you sit at the top of a corporation and you have nine alternatives to increase wealth, the least risky, the least expensive in my judgment will often be learning and transformation. And yet companies are currently spending millions of dollars creating joint alliances around the world, and putting out huge capital outlays to "invest in the future." When I sat in the CEO's chair and people would ask me, "Can we afford this learning stuff?" I'd consider all of the alternative investments, and usually the smallest outlay of immediate dollars was transformation and learning. The people were already on the payroll; they were just under-utilized. Moreover, if you are thinking about investing in human capital in order to build financial capital, if that is a critical part of your legacy, then a legacy mindset can also be commensurate with better short-term performance.

I think the reservations senior managers have about investing in organizational learning isn't about the dollars, or the amount of time it will take. They fear the personal vulnerability that will be required to make the fundamental changes that are necessary to lead such a process.

Question & Answer

Q: You've stirred something up for me with your first point about there being too much fog at the time things are happening to know when we're plateauing. I remember being in a division at CBS after the business people had taken over. The old guard was still there, the Cronkites and Bud Benjamins, and they were talking about a time when CBS was about the public trust. These people felt keenly the "plateauing" of a great organization, but the whole terms of the business were shifting.

Q: In other cases I've seen, the causes of that downturn are really deeply embedded in the way people behave in the organization. People in the organization know, and they can tell you, when the organization is turning down. Now, the real dilemma for me is what do we do about this? I think making that knowledge public is very difficult.

BO'B: I don't think we can prevent senior management at an individual company from disinvesting in the enterprise. But I'd like to see us find ways to identify those companies and their "undertaker" behavior, so that various constituents can accurately read management's performance earlier in time. If we somehow raised our corporate expectations, and created language to describe high and low tactics, there would be stigmas attached to such behavior, and the people who are trying to truly create wealth over the long term would have more support. I would think also it would smoke out the parasitical mangers who suck out of corporations what previous generations have built.

Q: In the telecommunication business, for instance, our technicians know when we're starting to take more out of the network than we're putting back in-when we're doing practices that are actually violating the integrity of the network. But to get that information public is very problematic. Even within the company, there's a stigma about bringing up the dirty laundry. One of the issues we're faced with is, how do you find a way to support real inquiry in the organization, without putting people in the position of having to sacrifice themselves in order to blow the whistle. That's what I think of when I hear you talking abut creating a stigma around certain behaviors. If we can build enough ground support for this change in mindset over time, then people will step forward, because they won't have to sacrifice everything they've spent their whole career building. Right now, you've got a few people willing to do it, but it takes too much courage.

BO'B: Yes. That's a good example of why we cannot separate virtue from knowledge.

Q: It fascinates me, Bill, how the lack of a legacy mentality connects to the earlier frustrations you shared. For example, the challenge of long delays and gestation periods and the fact that, when you stop investing in the pipeline, sooner or later you'll see the consequences. More than anything I can imagine, it is the legacy mentality that sustains those investments. People on the front line can often tell when management is, in effect, disinvesting in the organization, even though they may be able to mask that through current financial behavior. Then, when you consider the control that management has over short-term results and the problems of self-score keeping and virtue-again, it is through our sense of responsibility beyond our own "watch" that we balance the short term and the long term and make difficult choices that are not in our own short-term interests. You illustrated this yourself with the example of building up reserves for Hanover's long-term financial health, even though it hurt short-term performance.

Yet I repeatedly encounter a particular challenge in trying to foster a legacy mentality: most CEO's in large corporations have five years or less in which to effect the organization. Sometimes they get only one year in which to do something dramatic and "earn" an additional four years. But they simply can't do anything of importance in five years. Anything that involves a significant investment in organization-wide capabilities has a gestation period, generally speaking, of much longer than five years. I think this leads, in part, to the behavior you called the temptation to follow fads. It stems from a deep need to do something dramatic during your five years in the CEO's chair.

BO'B: Yes. I think what we are talking about is really part of a larger journey to redefine leadership for the emerging corporation. In Joe Jaworski's book Synchronicity, he says that the leader's job is to identify how the world is unfolding. That contradicts this notion of a CEO needing to "leave his handprint on the organization." The world is unfolding, and the time for the authoritarian, hierarchical, command-and-control corporation is past. I think part of our challenge in the Assessment for Learning research initiative is to lay out a picture of what is emerging-for our organizations and our leaders. That work could have a very powerful influence on the way our organizations evolve.

1 One of only two companies to enter the top quartile in overall performance during the decade of the 1980's, according to a 1992 McKinsey study.

2 "The Philosophy-Performance Link, " and "The Connection between Learning and Competitiveness in Hanover," 1987 and 1988 Hanover Annual Reports, respectively.

Edited by Colleen Lannon

REFLECTIONS OF A RECOVERING MANAGEMENT ACCOUNTANT

H. Thomas Johnson

Society for Organizational Learning
Assessment for Learning Research Initiative

First Research Forum
January 14-16, 1998

A Journey Up River

I started out as an accountant in 1960. I practiced, taught, and wrote in that field from early 1960's to at least to the mid-80's. let me start by making a few comments about how I've gotten to where I am now in my thinking.

When I look back at the whole span of my career, I see myself now as starting out at the mouth of a great river. This great river is a metaphor for the economy and the business world, and the mouth of the river is where all the data about economic and business results pour out. The mouth of this river is the world that accountants and economists inhabit. That was the world I grew up in, became very comfortable with, and gained a lot of expertise in from the early 1960s to the mid-1980s.

Many of us in that world became increasingly concerned in the late 1970's and the early '80's that something was wrong with American business, and particularly American manufacturing. At the time, the numbers indicated deep problems with manufacturing productivity, and the problems were getting worse. It was getting to the point that as accountants we were virtually saying "When are we going to turn the lights out, and who was going to be the last one to turn them out?"

In that spirit I started the collaboration with Robert S. Kaplan, which led to the book, Relevance Lost: The Rise and Fall of Management Accounting (Boston: Harvard Business School Press, 1987). Although Bob and I came from quite different directions, we both had a sense that there was something wrong in business that accounting, especially management accounting, was contributing to. From our perspective as teachers and as consultants we had concluded that while management accounting probably had an important and very positive role to play in American business in the formative years of our economic system, in the 19th and early 20th centuries, this was no longer the case in the 1970s and 1980s. I had started to reach these conclusions working with Alfred D. Chandler, the business and economic historian, in the early 1970's. I worked with Al during the years he was doing research for his 1977 book The Visible Hand. On the basis of that work I had concluded that management accounting had a very positive effect on developing the great economic dynamo that existed in the American manufacturing sector in the early years of this century. But something had happened by the 1970s. By then, the major companies in manufacturing industries that had been prospering and leading the world in the early 1900's were bleeding red ink and many seemed to be dying.

The question on our minds was what had happened? And that's what provoked our interest in coming together on the book. Because I had an interest in the historical side of accounting, and Bob was very intimately connected with current practices in contemporary companies, we complemented each other. He was then living in Pittsburgh, as the Dean at Carnegie-Mellon's management school. He was meeting regularly with chairmen of companies like TRW, Westinghouse, US Steel, Alcoa, and other mainstream US manufacturers, and they were in a lot of trouble.

The upshot of our collaboration, articulated in Relevance Lost, was the argument that management accounting, despite making a positive contribution to American manufacturing growth in the 19th and early 20th centuries, was giving managers very misleading information by the 1960's and 1970's that was causing them to make poor, often disastrous, decisions. Misleading management accounting information, we believed, was causing American manufacturers to give away markets and technologies that should not have been given away. In Relevance Lost we suggested that management accountants must clean up their act. We outlined some ideas for how to do this, one of which came to be known as activity-based costing.

In retrospect, I think we made some very useful suggestions, at least from the standpoint of people who live "in the mouth of the river." From that standpoint, if management accountants implemented the reforms we proposed, then marketers and manufacturers weren't going to be misled anymore, certainly not as much as they had been, about what was profitable and what was productive, and what wasn't. Now, when they made decisions, they would make them in a clearer light of day. The specific reform we proposed that came to be known as activity-based cost was for accountants to trace the costs of objects such as products, customers, and departments to the resources that cause the spending of money, i.e., to the consumption of resources that cause costs. Those resources came to be known as "activities."

After that book came out I received many invitations to speak not only from accountants, but also from people in industrial engineering, manufacturing engineering, and operations management. The engineers were particularly enthused by the message in Relevance Lost. Some of them, after I gave a presentation, would come to me and say "you and Kaplan have done the greatest thing, because you are telling accountants what we industrial engineers have been trying to tell them for decades, but they never listened to us. Because you're coming out of accounting, they're going to listen to you. It's about time. This message is going to help us all."

But some of these engineers went on to say, "When you look at what you mean by consumption of resources, you're talking about work, aren't you?" I thought about it and said, "Well, yes, I guess we are." They said, "isn't that the cause of costs? You have people who work. If there aren't people around doing work, you probably don't have any costs, from a practical standpoint -- certainly not in the long run. So work causes costs." And then they said, "Industrial engineers and manufacturing engineers have gained a lot of insights in the last 15 years, particularly from Japan, about what it means to organize work. We've come to a whole new sense of this, and we think you should pay attention to it. We think you could push the argument you've developed even further." I said, "What do you mean?" They said, "Well, if you organize work properly, you don't have to worry about getting the lowest costs. The costs will take care of themselves. In fact, you do not need cost information to tell you how to organize work properly. If you organize work properly, the costs will take care of themselves. Look at Toyota to understand what we mean by this."

At first I thought, "these guys aren't accountants; they don't really understand." But I took up their advice. I started talking to consultants who had worked in Japan and I began to read a lot about Toyota that had been translated from Japanese. Gradually, it began to dawn on me that maybe there was a message here. It came slowly, but once it hit, it hit very hard and fast.

Along the way, other speaking invitations led me into the Deming network, to people influenced by the late W. Edwards Deming. They started to tell me another thing. They said, "This is nice, you're saying that work affects costs. We've always believed that. But, you know, there's another missing piece here in all these articles and books you write. We don't find the word customer anywhere." I scratched my head for a minute, and said, "Yes, well, maybe you're right." They said, "Well, why are we doing all the work? Isn't it to satisfy a customer?" This is what drew me in to the quality movement and gave me my first insights into systemic thinking.

As I look back, I realize now what I was doing. In terms of the metaphor of the river, I was starting to move upstream. In contemplating activities, I had starting to move above the mouth of the river, where there is this enormous amount of data about business and economic results. I was beginning to move upstream to observe conditions that gave rise to those quantitative financial data. Moving up to where I considered the nature of work, and how it is organized, was an even bigger jump than the one that took me to "activities." If you think of it in terms of moving up the Mississippi River, I was starting to see the water clear a bit. By the time I got to where you consider the way to organize work, and then to customers, I think I was almost beginning to see the bottom of the river.

All these encounters with people who worked outside accounting and outside economics was opening a whole new perspective to me. It was opening my mind to a whole new way of thinking about why we are in business and what we are doing, and about how we think about these problems that I had always studied through the medium of accounting numbers. In particular, I was beginning to be more convinced of the idea that if you do the right things -- listen to the customers, listen to the voice of the process -- then costs will take care of themselves. You didn't have to worry about having a cost system to tell you how things are going. Although at first I couldn't articulate the basis for this belief, I began to be convinced of it the more I thought about it.

Practitioners of a New Tradition

By late 1991 I had finished writing another book, Relevance Regained: From Top-Down Control to Bottom-up Empowerment (New York: The Free Press, 1992) where I expressed some of my new thinking in the form of criticisms against activity-based cost management. The new book raised the ire of my former co-author Bob Kaplan, who became outspokenly critical -- in fact, openly contemptuous and disparaging -- of what I was saying about "doing the right thing and costs will take care of themselves." To this day he continues to be the chief spokesman for the "manage by results" viewpoint that has dominated American and European management thinking since the 1960s. I sometimes wonder if the alternative that my own more recent writing offers to this viewpoint has been hidden from many people by the incessant attention that the cost management literature in the United States pays to my early-90s criticisms of activity-based costing. It's as though the American management accounting profession, led by Bob Kaplan and his minions, perceive my work only from a perspective in the mouth of the river. Perhaps it is because American management accountants have never been to the places I have traveled to upstream that I now find it difficult to communicate with them. However, I find no shortage of fascinating people who have been upstream and with whom I am communicating, and from whom I have been learning, in the past 7 or 8 years. Through these people who work in fields such as operations management, systems thinking, life sciences, design engineering, quantum physics, nonlinear mathematics, cybernetics, and quality management I hope someday to find the means to communicate again with the accountants and economists I once worked with in the mouth of the river.

Probably the most important result of my publishing the book Relevance Regained is the attention it brought from two people: Anders Bröms, a consultant in Sweden, and Kaz Mishina, a young operations management expert with a profound knowledge of Toyota who was teaching at Harvard Business School in the early 1990s. It was Mishina who contacted me first. He said "I like this new book of yours. It has a lot of good ideas that reflect what people say at Toyota. But," he said, "I detect in what you write about Toyota that you've never been in a Toyota plant. Have you?" I said, "You're right, I never have." He said, "I could tell." And he said, "What I want you to do, if you would like, is join me. I'm going to go to the new Toyota plant in Georgetown, Kentucky, to gather material for a series of Harvard Business School cases. They're going to be the first Toyota cases ever written at Harvard. Would you come and work with me?" I said, "Wow, would I come? When do we start?"

That was about six years ago. I've probably been to Georgetown 40 times since then. Although Kaz Mishina has long since returned to Japan, where I believe he now works in a government agency, he connected me with a wonderful network of Toyota people who are a continuing source of inspiration for me. It was in Georgetown that I started to learn firsthand, and finally began to understand, that "if you do things right, the costs will take care of themselves."

Soon after Mishina contacted me, I received a call from Anders Bröms in Stockholm, Sweden, who told me that he and one of his consulting clients -- the heavy truck maker Scania -- liked the message in my new book and they wanted to talk to me. Here began a relationship that has deepened enormously over the past five years. With Bröms, Scania, Toyota -- and now with an evolving relationship between people in Toyota's Georgetown plant and people in Scania that Bröms and I have helped facilitate over the past few years -- I have traveled farther upstream than I ever could have imagined possible when I wrote Relevance Regained. Indeed, thanks to Anders Bröms and his colleagues in SAM Samarbetande Konsulter, a small but fabulously exciting consulting practice in Stockholm, I have had an opportunity to explore the implications of viewing business through the lens of life systems, instead of through the usual lens of mechanistic systems that dominates American and, increasingly, most European management practice. The initial results of this exploration will be in a book that I am writing with Bröms that we hope will appear later this year or early next year.

The book will feature ideas gleaned from both Toyota, a company that most of you know a lot about already, and Scania, a company that is not well know in the United States. Whereas Toyota is expert in a unique approach to operations management, Scania is expert in a unique approach to product design that I think almost no other company in the world practices. Product design is an area I knew absolutely nothing about before I met Scania. To learn something about design engineering I came to MIT and met with Dan Whitney. I also learned a great deal from writings by his young colleague Karl Ulrich. But it's really Scania that taught me the most about design engineering, and especially about connections between Scania's approach to product design and Toyota's approach to operations management.

Somewhere during this time, in 1991 or 1992, I ran into Peter Senge and the Organizational Learning Center (OLC) and its programs. At about the same time I met Juanita Brown and I started to attend conversations with her friends in her home in Mill Valley. I had the privilege to join gatherings in her living room, in that beautiful setting on the side of Mt. Tamalpais, meeting people who were interested in things that were not different from what I was hearing from my Toyota friends, from Deming, and from Anders and the people at Scania. But it seemed to go further. We started to talk about systems, system thinking and relationships.

Somewhere along the way, about four or five years ago, somebody told me about Gregory Bateson. Now that was a remarkable turning point. I started to read Bateson perhaps four or five years ago. About three years ago, I started to finally get serious about it. (It takes a year or more to figure out what's going on when you read Bateson.) What eventually came out of reading Bateson was a sense that what we are looking at in the world of business are mechanical systems human beings have created. They're designed by minds that are good at doing mechanical things. These systems (defined essentially by lineal causality and independent relationships among parts) are then set down and put to work in a world which nature runs in quite a different way (a world of recursive feedback and interdependent relationships). Influenced by Bateson, I came to see the world as made up of two systems. There is nature's system and there is the system that self-conscious human beings create, what I would call the mechanical system. Without the human being I think there is nothing mechanical in the universe. Our problems in organizations come because they impose the mechanical onto the natural, and then operate in a necessarily parasitic way in the midst of nature's evolving system. I began to see what Bateson meant when he said that "the source of most of our problems is the difference between the way man thinks and the way nature works." With Bateson, I had advanced much further upstream. In terms of the Mississippi, I was a long way north of St. Louis.

So I set out to understand, how does nature work? This is a question I learned a lot about from reading Bateson, but also from reading Fritjof Capra and other writers talking about modern physics and evolutionary biology. I was beginning to get an appreciation of what Bateson was talking about when he said "the way nature works." I began to conclude that that's the way human beings have got to work. Work is work. Like the industrial engineers were telling me way back in the late '80's, it's all work in the end. If you get that figured out, then everything else is going to take care of itself. By now I realized that idea was even more than they knew.

When you start to study ecology, and you abstract yourself from the mess we've created in the world, and think about what nature has done for four billion years on this earth, it's a pretty remarkable accomplishment. You start with water-borne chains of carbon molecules about four billion years ago and eventually you get to dolphins dancing in the sunset and human beings. I think that's a pretty remarkable set of accomplishments. That's "results." If we can work like that system, we don't need to worry about where the results will come from.

What are the kind of messages that come out when we study how nature works? There are obvious things -- like nature is basically an inter-dependent system where everything is self-emergence, self-referential, self-identifying. Everything is working to ultimately increase diversity over time. It seems that nature's only job is to increasingly diversify the manifestations of the pattern that appears in all the matter that surrounds us. As the physicist David Bohm said, it's the constant process of the "implicate" being made "explicate," or as Aurobindo, an Indian philosopher of this century, put it, the involute becoming evolute. Before evolution there was involution. All that follows from that is involution spinning its web through the pattern we call evolution.

The Mechanical System and The Living System

The more you look at all of this, there are people who have been talking about such ideas for a long time. This is not just modern physics. It's not just evolutionary biology. These ideas don't appear for the first time with modern scientists such as Lynn Margulis and David Bohm. This is Buddha, and this is Christ, and this is Lao Tzu, and this is Pythagoras -- thinkers through the ages who have provided us profound shafts of insight. All the way down to modern times, to the 19th century with Blake and Goëthe, and to this century with Whitehead, Tillich, Bohm, and so forth. There have been thinkers expressing these ideas, and we can call them System Thinking, in a way, if we want to -- as long as we remind ourselves that there are two kinds of systems.

There's the mechanical system and the living system. Peter, you hit it in the preface to Arie's new book (Arie de Geus, The Living Company). I really want to congratulate you on saying what you did in that preface. Those few pages are his book to me. They do a nice job of making that distinction between seeing a corporation as a machine for making money and seeing it as a life system, a human community. I think this is a distinction that we've needed for a long time. It's the one I was realizing as I looked at Bateson's work. You can see, given where I was coming from, why I was troubled with System Dynamics and much of the "system thinking" stuff for a couple of years. Because I think so much of that work comes out of the mechanical side of man's thinking, and not out of an understanding of life systems.

I look at that and I say, all right, "knowledge of nature's patterns" -- that's what we've got to focus on now. Science, religion,... there are lots of places we can turn to find out more. This body of knowledge is not an empty set that we have to build from scratch. There's a lot of knowledge out there to build on. For a while, now, let's put away all the economics books and all the accounting books. Let's go back and regroup. We'll find, once we get started, that it isn't that hard, and it's good stuff when you get into it.

The other day I pulled off my shelf a book that had been sitting there for years. It was the third volume of Paul Tillich's Systematic Theology. I had the good fortune to be an undergraduate at Harvard in the last half of the 1950s. For about two years, around 1958 and 1959, I attended some courses presented by Paul Tillich. Thinking back on it now, I realize I got maybe ten percent of what he was saying. But when I looked recently at the third volume of his Systematic Theology (published a few years after I graduated from Harvard), I realized that it was the material from which the lectures I heard in the late '50s were constructed. He talked in the book about life, in terms of "actualization of potential." That was the whole subject of the introduction to the book. I remembered sitting in old Emerson Hall hearing him say the phrase "actualization of potential." There I was a 19- or 20-year old kid, sitting and listening to his thick German accent, talking about "actualization of potential." Little did I realize then how forty years later I would rediscover his words in a journey upstream where I was learning about life as "actualization of potential" in writings by Bateson, by Whitehead, from Buddha, dryer, and of course, from David Bohm. So I look at these things now, and I say, "Well, these are the messages we want to listen to."

I now think that's what "learning" really is: it is discovering and embodying nature's patterns. That's what our work is. That's what our work is as human beings. It should be the primary work of the organizations we belong to. That is what I hope we are going to discuss over the years in our meetings of SoL.

Nature and Measurement

Ed Deming used say that 97% of what matters in an organization can't be measured. Only maybe 3% can be measured. But when you go into most organizations and look at what people are doing, they're spending all their time focusing on what they can measure and none of their time on what really matters -- what they can't measure. Why would we do this? We're spending all of our time measuring what doesn't matter. In fact, its part of avoiding a lot of the really difficult and important issues, like virtue, as Bill O'Brien has pointed out (see "Nine Frustrations of a CEO," SoL Assessment Initiative Working Paper). We spend almost none of our time on what really matters.

So I asked myself, why do we measure ? Why are we doing this thing that we are talking about here today under the heading of "assessing learning?"

When I think about it historically, I realize that in the context of business the kind of measurement we're talking about really is a 20th Century phenomenon. It isn't all that old a phenomenon. Even in a broader sense that goes beyond business, to include the sciences, measurement as we know it isn't all that old. It really goes back to Galileo. I think Galileo's greatest contribution -- and it was a stunningly profound contribution -- was the notion of measurement.

You probably know the story. As I understand it, Galileo was attempting to deal with the problem Copernicus had left on the table. If the Earth moves, how the hell are we all sitting here? Why are we not tumbling all around all over the place, the building falling down and so on. In order to deal with that issue, Galileo came to the conclusion that what you have to do was somehow separate motion from the object moving. I don't think anybody had ever really thought of that before.

Aristotle certainly hadn't. For Aristotle, the object moved because it had a place it had to go, it had a purpose and a teleological end that was to be reached and that was what gave it the motion. But Galileo disagreed. He said, "separate motion from the body moving." The minute you separate them, then motion can be measured. You can put it up against a scale. Of course, the story goes from there to Descartes, who shows how to put the scale on graphs using Cartesian coordinates and all that. Then Newton finally comes up with the answer to the question that Galileo was grappling with in the first place, which was the theory of gravitation. But it all had to have that initial incredible breakthrough; it had to have Galileo say, let's separate motion from the body moving.

Let me say, measurement can be two things. You can use measurement to describe and, in that sense, Aristotle was a measurer, too. Aristotle could have said, "well, one candle is brighter than the other." No problem with that. One shade of red is brighter than another, or one person is bigger than another, or one thing's heavier than another. But with Galileo, we started down a different road, the road of quantification. When we quantify, we are separating things that in nature are inherently combined. And in so doing we take apart and change that which is natural. That's what Aristotle didn't do; that's where Galileo started taking us.

Over time, we got very good at measurement and quantification. It leads to steam engines, internal combustion, high-rise buildings with central heating, eating a good meal provided from farms many thousands of miles away, and so forth. It leads, over time, to the creation of the entire mechanical world we human beings have produced in our global economic system.

But, absent the human mind and absent the human mind's busyness with measurement, there is no such thing in nature as a mechanical thing. Nature doesn't mechanize. Nature naturalizes.

And, nature doesn't measure. This is another of the points Bateson made, 50 years ago. Nature doesn't measure, nature doesn't quantify. Nature deals only with "patterns which connect." Nature focuses on the patterns. It's patterns and relationships and connections that create the reality of nature. Galileo wanted to escape that, and so he did. He got us on the track where we are today. In escaping patterns and relationships, we mechanized.

But what we have done to achieve mechanization -- particularly in our lifetimes, in this century -- is an incredible job of destroying nature. We have destroyed nature as a result of this mechanistic thinking and the mechanistic activity that follows from it. Particularly in the engineering realm and the scientific realm -- we all know about that story. But I think we don't often appreciate the extent to which we've destroyed nature to the point where if any of us were really dumped in nature, in its raw form, we'd absolutely be at wit's end. Imagine being dumped in a setting like Lewis and Clark were in only 200 years ago, in the middle of what today is Montana or Idaho. It would scare us all to death! Most people living today wouldn't know what to do. We'd probably drop dead from the shock.

Means and Ends

Now, measurement in many ways brought us to this point where we can no longer understand or even appreciate or even want to deal with nature. We've torn apart the means to achieve our consciously-contrived ends, what Bateson called "conscious purpose." We've done that so well that now we're happy with our ends, we're comfortable with them. We've created a physical, engineered realm, destroying nature, and we don't want to talk about nature anymore. We don't even want to think about it for the most part.

Now it is only in this century that we did this to ourselves in the human organizational realm. First measurement comes to astronomy. Then it comes to physics, then to chemistry and engineering. It is not until we get down to the early 20th century when events take place that give rise to the issues we're talking about here tonight. Only then do we start to measure this thing we call the human organization. It can be any type of organization, from a hospital to a church to a business.

In the process of applying measurement to organizations, we do the same thing we did in the so-called "physical world." We destroy what's natural. The process of measuring destroys what is natural. It has to. It's an inherent characteristic of measurement. Measurement takes what's inherently interdependent, and shaped by patterns of mutual causality in nature, and turns it into something that's inherently independent and shaped by patterns of lineal causality. In creating systems whose operation can be described with quantity, we create what is mechanical. Measurement mechanizes, and mechanizing is diametrically opposed to the natural.

So, we've destroyed the natural in our organizations, and we're very comfortable that way. And we don't want to go back. We don't want to contemplate the reality that would be there if we weren't measuring, anymore than we would want to go back to the world of Lewis and Clark. We are most comfortable with the abstraction we have created with our measurement prowess.

Another way to put this is to say that in the process of destroying the natural, what we've done is pervert ends and means. Nature, in a sense, is just the endless unfolding of means. However, the unfolding occurs according to definite patterns. The outcomes, in specific detail, are random, but the patterns driving the process are not. Nature doesn't care about the ends, they just happen. But the means is very much what nature is concerned about.

But as humans we decided that we're not happy with nature's ends. We think we can do a better job; and we do it by perverting the means. Measurement is one of the principal tools through which we do it. We've gotten very good at it. We've shattered the means; we've changed them, we've manipulated them. Deming used words like 'tampering', to characterize how we try to manipulate without understanding. We do this because the ends are what are important to us. They're what we regard as the permanent thing. Whereas in nature the means are the permanent thing. We consider the means ephemeral, throw them away, forget them. The means -- do whatever you want with them. The ends are what's important, the goal itself, what you measure for. We want what we can get.

As an aside, I am coming to believe that a lot of our deepest frustrations in contemporary society come from a deep sense of what's been lost in this process. People increasingly sense anxiety and pain at having lost the natural. Most of us can't even articulate it, but we've got a gut sense of it. We've lost the joy of conversation, the quiet being in the moment, being part of a larger web, the excitement of the journey itself -- some of what "the means" mean to us as human. We've pursued the path of the mechanical to achieve our contrived ends and, although many of us are inherently uncomfortable with it, we feel the only way we can proceed is to do more of what we've been doing. I worry that the generation coming along right now hasn't even got a sense about this loss; they scare me to death, this "cyber-generation" coming along.

I come back to my question, "Why do we measure anyway?" What is it? What have we been doing to ourselves in exercising this talent, this power, this capacity? We've got a long history now to reflect upon. We can talk about what we've done in the physical realm, in science and engineering, in the technological realm. What we're doing in the business realm today is no different. We're seeing it work out there, and the inevitable consequences follow. We've mechanized everything around us by quantifying.

My feeling is, what we need (if there's an answer, and I don't pretend to give definitive answers) is that, somehow, we've got to go back and talk about what it might be like to revisit the means, as a thing in itself. As the only thing. We might say to ourselves that it's not for us to create the pattern, the pattern is already there. It's for us to put ourselves in harmony with nature's pattern, somehow. And the ends will take care of themselves. The results will be sufficient unto the day. That's a leap that we're afraid of. It scares us to death.

But it is possible. I spoke earlier of Toyota. I have had the privilege of coming as close as I think you can to seeing this way of thinking in a business, in Toyota. Finally, after years and years of walking the floors in that company and, on a few occasions, working on the line, I come to this simple idea (expressed once by Joanna Macy). The means are the ends in the making. That's all there is. Get the means right and the ends will take care of themselves.

That's why Toyota, despite having an excellent accounting system that allows them to comply with regulatory authorities and so on, actually has no standard cost accounting system in the sense of other manufacturers. They don't drive operations with the numbers. They don't measure for this purpose, for the purpose of motivating action. They follow a different logic, a deeper logic. They measure only to enhance awareness of how the work is flowing.

Science and

Measurement in the 21st Century

While nature does not quantify, nature does count. You'll see lots of things in nature in twos -- two eyes and two ears, the bilateral symmetry of our bodies, of the bodies of all mammals, and for that matter of much of life. Nature likes to work with twos, a lot: yin and yang, sexual reproduction, black and white, positive and negative. That, to me, is one of the most profound expressions of what Blake called the "twofold." An example of this is seen in Goëthe's conclusions about color, which were quite different from Newton's. UL thought Newton drew the wrong conclusions from his experiments with prisms. He showed that color was not a quantitative abstraction explained by wave lengths. He showed that the intersection of dark and light is the real origin of the colors cast on the wall by the prism. His theories are now being resurrected by Henri Bortoff, a former colleague and protégé of David Bohm's. (Bortoff, The Wholeness of Nature (Lindisfarne Press, 1996).

Goëthian science is, I think, the one that must shape our perception of reality in the 21st century. In the 21st century, Newtonian science must no longer shape our world view, if we are to survive as a species. than science is the science of than,two-foldence of twos.

So, number is there in nature. But so also is ratio, which is dividing numbers.

Working with Anders Bröms in Sweden, I found he had come up with something quite similar to analyze costs. He and his colleagues make extensive use of ratio indexes. They're very sophisticated mathematical constructions, as only Swedish engineers working 20 years on these things might come up with. He's got these beautiful pictures he can show you from clients. They tell the years when they were doing well, and the years they were doing badly, which client is doing better than another, and so forth. He does not use traditional cost information to do this. What he's developed are structural indexes that are made up of ratios. In a sense these are inequalities, based on the same principle as inequality indexes, like the Italian sociologist Vilfredo Pareto first started to use. But, with very sophisticated thinking behind these very simple ratio numbers, he can tell rich stories about a company's performance without ever resorting to numbers from the accounting system. He can also tell where their advantages are, in terms of design engineering or operational management. This has been applied with great success at Scania, and we're hoping to use it to explain better what Toyota is doing.

So when I say I'm not in favor of measuring or quantifying anymore, you understand I don't discard numbers, I don't discard ratios. There are descriptors that we can come up with that can help us see patterns. These will tell us whether one group is making it and one isn't. I think we can come up with some assessments and evaluations in answer to the issues here at this conference, in this research initiative.

We are very much at the beginning here. We're just seeing a very, very small tip of an iceberg out here. I'm not explaining it at all in detail here. I hope that the book Anders and I are writing now will help take this a bit further. It will be the first public airing of some of these views.

How Will We Know We Are Learning?

Assessment, answering the question "how do you know," is crucial. That was the question Deming wrestled with. People would talk about performance in terms of number and he would ask, "How do you know?" How can you possibly assess things with the minuscule little elements you're looking at here? How do you know?

In the end, I think the question, "How do we know we are learning?" can be addressed in terms of "Are we undoing the mechanical?" And "are we establishing and fulfilling the natural?" If we assess what we're doing in terms of how far we are progressing towards understanding and embodying how nature works, we will see that real progress is possible.

This gets to Bateson's idea -- are we undoing the way man thinks? Are we putting in its place a deeper appreciation of the way nature works? That's how we'll know we're learning. We've got to come to an understanding of what that means. We'll see it, and as we see it, we will find a reconnecting of the fragmented state of the world.

In the end, can we get an idea of what this might be like? I go back to Scania and a Swedish experience I had. Scania is a company owned by the Wallenberg family, a wealthy family that owns much of Swedish industry. The parent company, which goes by the name Investor, has held all kinds of companies, like Scania, for many years. Investor has always been run by a member of the Wallenberg family, up until this past year.

One of the great leaders of the family in the post-World War II period, Marcus Wallenberg, was very close to his companies. He regularly would go visit all the companies. He was not a person who sat above the clouds and studied his companies by looking at spreadsheets. He went down to the companies, like Scania, and when he did he invariably visited the shop floor. You can imagine, this is not the chairman of a board, this is the chairman of the chairmen of many boards. He would go down in the shop, where he would talk with workers and engineers. I believe he was trained as an engineer, so he understood what he was looking at. Someone asked him once, "How do you know, when you go into a shop, what to look for? What is it that tells you when things are right?" And he said, "I go into the shop, because that's where what matters takes place. And when I go there I listen for the music." That was his expression. "I listen for the music. And if I hear the music I know everything's all right. But if I don't hear it, then we go to work."

I think that, in a way, is what it is -- "listening for the music." Finding patterns, seeking them out. If you see them, then you know everything is all right. Maybe numbers or ratios, to some extent, can help us. But we will know, as well, when we're not hearing the music, when it's discordant, when it's disharmonious.

I think 20, 30, 40 years down the road, we must think this way and work this way. Then, in terms of my "traveling upstream" metaphor, we'll be somewhere north of Minneapolis (referring again to the Mississippi river), pretty close to the source of the river, with a very different approach to managing businesses than we've grown up with in our lifetimes.

I'm reminded that when you follow rivers upstream back home in the Cascade Mountains, to the extent that you can really find a river's source in mountains like that, you often get to the point where you realize that even when you've reached the river's source, you still haven't found the ultimate source. That realization comes when you look up, and see a cloud covering the mountain's peak. That's where the river is coming from. It's coming from the clouds, and in the clouds from the water that evaporated at the mouth of the river, where the journey began. You find, as T. S. Eliot said, "the end of all our exploring will be to arrive where we started and know the place for the first time."

Open Conversations

(What follows are some segments of the conversations that followed Tom Johnson's evening presentations at the workshop)

Open Conversation 1

Question (Lori Breslow): What worries me about what you said is that you throw out, I think, an important tool. I think there are some questions that are answered very well by measurement and there are some questions that aren't. In my view, the real problem has been in many fields in the social sciences, for example, we have tried to answer certain questions where measurement should not be used.

I think the real question is, what is it that we're trying to learn about and what methodology best helps us to learn about that?

TJ: Well, I'm not advocating that we eliminate the descriptive nature of measurement. It's probably inevitable that we will use measurement to describe features of systems. It often is very useful, just as long as we don't let measurement drive actions. Once we say: "How can I take apart the pieces of what I've just measured and make what I've measured somehow better or different, or more or less?" That's where I draw the line.

Tim Savino: I would add that the question of why are we measuring may be also thought about along the lines of, " Who is asking and why are they asking?" There are some virtues to trying to measure in order to understand, because measurement and understanding can be intimately tied together. It's when the measurement and understanding are detached, that I worry. When we measure because we've been told to measure, or in order that we can boss somebody around with our measures, when we measure and we really don't understand what any of this is for, that I think we do harm.

I think from my viewpoint, the charge to this group, to this research initiative, is, in a sense, of trying to distinguish assess or measure.

TJ: I agree, Tim, with everything you say about keeping measurement and understanding linked.

Stella Humphries (?): As an organizing committee, we felt that two words we wanted to dissociate from one another were assessment and measurement. Assessment is the larger term in the sense that it is not possible for any learning process to not involve assessing, learners making sense of their actions and the consequences of their actions. Measurement as the smaller and more contingent term, and more uncertain term.

Tim Savino: To me, what I would hope to get out of this whole effort are steps toward better understanding, rather than an ability to say, "You are a learning organization, and you're not."

Peter Senge: Tom, I don't think you said Toyota doesn't measure anything. You did say something about how they don't use the measures.

TJ: Yes, it's the way they don't use them that I find interesting, not the fact that they do not measure. Toyota measures; they just don't drive actions with quantitative targets.

Bill O'Brien: They don't use it to motivate action.

TJ: Right, they don't use measures to drive decisions about how work should be done and what work should be done and so forth. Of course, they have an excellent accounting system. They invented what we call "target costing." But that's a descriptive measurement concept, really. It's an ex-ante tool, employed before the work is even started. But once the work begins, cost targets play no role in influencing operational decisions. The things that guide the work come from a different level of abstraction than quantitative measures come from. Guiding the work are things that aren't measurable. Over time, they develop systems and patterns of behavior that are deeply ingrained in people. These are deep disciplines they have in order to know, "how is the work flowing?" Do we have a capability to detect normal from abnormal? These are the types of things they focus on, that everybody comes to know.

Bill Torbert: You're also making, it seems to me, an incredibly important point which some modern social scientists don't appreciate sufficiently at all. That concerns the distinction between ordinal measurement and interval measurement. We assume that we need to drive measurement to interval because we need to use the most sophisticated, quantitative statistics (at least what we consider the most sophisticated).

In fact, it's not clear that human psychology, except in extraordinary cases is capable of accurate interval measurement. As you say, what we measure in reality is "more or less." We can sometimes prioritize first, second and third. We can sometimes accurately distinguish nominally, sometimes. So, the more "primitive" forms of quantitative distinction, at least as we think about it in statistics, are actually the ones that are more important for the sorts of assessments we want to make in action.

But you do seem to take a very extreme position. In a sense you almost come around and oppose what Bill O'Brien first said, which was to try not to dichotomize between the human development camp and the bottom line camp. You almost seem to come around to saying, "Let's get rid of that kind of measurement I associate with the bottom line camp."

It seems to me Aristotle actually was able to combine the different types of causation you allude to. He saw that there was "efficient causation," which is, if you will, mechanical causation. But he also saw material, formal and final causation ("final" not in the sense of reaching your goal but orienting toward what your purpose is). If we come back and think about a corporation or about our personal life in those terms, we see questions like: "How do you balance final, formal, material and efficient causes in myself or my corporation? and "How do you measure those causes, especially since only the efficient cause can be measured in this quantitative way?" Perhaps, we want to understand something about the other causes.

TJ: I have to think about the connection you draw between Aristotle's concept of the efficient and my concept of the mechanical. I have never thought about it.

Peter Senge: What did you think about the point Bill made about throwing out the measurements associated with the "bottom line camp."

TJ: Throw out...? No, again, I think profit's a descriptive measure that we can use, although I think it's a very crude and ambiguous one. I have particular concerns about profit which stem from the fact that I don't think we need it. We never had it, really, until the 20th century. Before the 20th century, for all practical purposes, what businesses had, coming out of double entry bookkeeping, was the balance sheet. The balance sheet told them how much they owned and how much they owed. Knowing how much they owned and how much they owed answered all the questions that bookkeeping was capable of answering.

The question the income statement purports to answer, "how well are we doing," was answered before the 20th century by real-time information in the hands of the guy who was out in the counting house, out in the plant, out with customers, or the guy who was getting news about how the ships were faring on long voyages. Those people did not use income statements to see how well they were doing. In this century, we somehow got caught up in the idea that we must have accounting measures of performance to answer this question. The more we relied on accounting abstractions to tell us how well we are doing, the less connected we were with the concrete, natural realities that generate revenue and profit.

Peter Senge: And then problems arose when that accounting performance information was used to guide the people doing the work.

TJ: Yes. Moreover, the coming of income taxation amplified the importance of these accounting numbers and helped accelerate their use. But I think if we go back to the day when it all got started, we should have stopped and said, "Do we really need this?" "Do we really want this?" Is there, perhaps, more danger in it than help? Because we might change our minds. I would say, for example, if I could re-write all the laws, I'd go back and argue for having nothing more than standard double-entry bookkeeping with a balance sheet. Of course you also must keep track of cash flow, there's nothing wrong with that. If you do not do a good job of managing cash flow, you're out of business.

But that's not what income determination is about. I would say, get rid of income determination and the obsession with income numbers that drives so much of what accountants do today.

Bob Putnam: Could I go back for a moment? When you were saying that Toyota does not measure the work in order to drive behavior, you also said that they do attend very closely to how the work is being done, and they try to imbue everybody in being able tell the difference between when the work is flowing normally and when it is not? So I'm thinking, they are very good at assessing the work practices -- not with measurements, but assessing in the sense of knowing "Is it going well or is it going downhill?" They can build a pattern. And that's, arguably, something that makes them Toyota.

TJ: Yes, I don't know what words to use for this. Maybe words like assessment may fit. One way I've described this in what I'm writing now is to say that they have a very highly developed immune system. Through the disciplines they have imbued in their people, through the training over the years and the standardization of work and so forth, they know, as Arie de Geus put it, "what is us and what isn't us." Everybody knows this, in everything they do. It isn't through measurements that they know that. They know it in the same way your body's immune system "knows," which is probably not through measurement either. It knows it by doing it.

Bob Putnam: But it is getting information and assessing that information.

TJ: Well, the information is the work, the work is the information. It isn't coming from anywhere outside the work.

Bob Putnam: But I'm saying that, if you want to use the analogy of your immune system, your immune system is in fact getting information from the foreign substances and assessing that.

TJ: Well, it's talking to them. In a sense, the molecules in your body are talking to the molecules that shouldn't be there and they're having a conversation. That's the way the biologist Francisco Varela puts it. I don't really understand this area very well, but that's the way he talks about it: there's a conversation going on. In a sense, the information is in real time, its part of the "work." Either we kick you (the foreign element) out, we kill you, or you join us somehow and you help us.

Bob Putnam: But I think the critical point here is that it's in real time. The assessment isn't something that other people come and do on a different time cycle from the work itself, and then reintroduce their conclusions at a later time. (TJ emphasis)

TJ: Yes, it's part of the metabolic process.

Bill O'Brien: And it's done by the people who are doing the work. (TJ emphasis)

TJ: Exactly.

Bob Putnam (?): It sounds as if we're starting to develop some criteria for what we would want effective assessing might look like.

Bill Easterday (?): I was talking with somebody who was head of a car plant who said, "You know, when Toyota went into somewhere in the Southeast where they have a plant, they sent 300 managers from Japan to sort of shadow the people in the plant for a year, coaching them." My god, what an investment. But, if I then think about what those 300 Toyota coaches were doing is transferring knowledge of recognizing when the process and the practices are going right, then it starts to really make sense.

TJ: Actually, if it was Georgetown, Kentucky, I believe they sent approximately 600 people for about 3 years.

Bill Easterday: That's just mind-boggling. But from the point of what I'm thinking now, in the context of the current discussion, it starts to make sense. If you can develop the capacity of people on-site to recognize patterns in the way you have been describing -- I'll call that assessment of some kind -- if they can recognize what is going right and what is going wrong so they can make corrections, if they can transfer that sort of knowledge through a three-year process of face-to-face, one-on-one coaching...

TJ: Working together?

Bill Easterday: Right. That's a daunting challenge, but it does point to the fact that there is real knowledge for assessing what's doing well and what's not.

TJ: I think in the context of what Juanita Brown writes about it, it's a conversation. I mean, it's a conversation going on all the time that's making this work.

PS: Come back for a moment to Tim's question about who is asking the question, because it seems to be particularly appropriate. We are painting a picture of a type of learning going on within a group of people responsible for and active in doing something. But then there's somebody else over here, not involved. And they have the temerity to ask the question, "How's it going?" Now, somebody else wants to know.

There are usually two sorts of people in corporations who ask these questions: "management" and investors. Both are not directly involved in the value creating processes, they are on the outside. So it seems to me that we've come up with measures that may have been of some use as post-hoc, retrospective ways for people on the outside to know how things are doing, measures like profit.

How do we answer them? Do we answer them? Do we tell them to go away? Do we tell them, "Look, it's ultimately a matter of trust, period, end of story. If you don't understand the process and how people are learning in the process, you have no right to ask the question."

TJ: When you raise the issue of profit, again, I think that many people say, "we've got to have profit because there's a third party out here, the stock analyst." Well, when you think about it today, what accounting researchers have written about, what finance people have written about how share prices are determined, profit statistics are not as important as you think. I think good analysts for the last ten or fifteen years, who are thinking in terms of portfolio management and capital asset pricing models, don't spend a lot of time looking at accounting-based reports. They look at those a little bit, but basically the information they're putting together concerns combinations of securities, in portfolios, not the particular performance of a single company. Accounting-type measures don't really loom very large to them.

I think from a practical standpoint, this issue that many people are troubled with here, about what are we going to do about profit, can be answered if we got rid of the internal revenue code. I mean, if we didn't pay income taxes, particularly corporations, who'd give a darn? I think most of the accounting department could then be eliminated. Remember that famous picture, I think it was on the cover of Fortune, of the Chrysler controller or CFO standing next to Chrysler's annual income tax return. He was about 6' 2"" and this return loomed way over his head. If the need to prepare such a report were to go away, a whole lot of this concern about how do we measure profits, I think would also disappear. Because most of the other people in the organization and in the finance community aren't really that concerned about it anyway. I think an important political action we could all take is to write our Congressmen and press them to stop taxing income.

Peter, just to answer your point, how do we answer somebody who wants to know how it's going. My impression of what the most creative CEO's do is not throw numbers at people. They tell them, they have a conversation with analysts.

Stella Humphries (?): They tell stories.

TJ: Yes, they tell a good story about what's going on to assure analysts that the company's moving in the right path. And the worst CEO's that I see are the ones who really just rely on numbers. They don't have these conversations.

Now, I think people are getting more sophisticated, but I think there's still a huge difference in how CEO's deal with that. So I think it's possible to let people know how it's going by having a conversation.

Juanita Brown: I really want to honor the journey that Tom has been on, and his courage to continue that journey and to articulate his understandings.

TJ: Thank you. You've been an important part of it.

Peter Senge: Thank you, Tom, and everyone, for a most remarkable evening.

search
members' login (lost username/password?)
username
password