Assessing to Learn
and
Learning to Assess

Tom Johnson closed the first evening session with reflections on Bill's comments from his long career in managerial accounting. Tom is widely known as a pioneer in accounting, notably as the co-inventor, along with Robert Kaplan, of activity based costing (ABC). Their acclaimed 1987 book, Relevance Lost, showed how standard accounting practices had lost much of their usefulness to managers.

"Dr. W. Edwards Deming said that 97% of what matters cannot be measured. Today, it often seems that 97% of management attention is on measures. This means that we are spending most of our time on what doesn't matter." Why do we measure?

The history of western approaches to measurement started with Galileo, who was trying to verify Kepler's idea that the earth was moving about the sun, rather than being at the center of the solar system. Galileo, breaking from Aristotelian tradition, determined that it was necessary to separate the motion of an object from the object itself, in order to measure that motion. This led to the scientific practice of separating things which are connected, in order to measure distinct properties. In the process, we gradually began to sever the connections in our mind, to the point that we lost awareness of the connections altogether. We began to work with abstractions.

Our fondness for quantifiable measures became a foundation for the mechanical world, and for a mechanical viewpoint that gradually extended to how we see everything.

"It wasn't until the 20th century that we started to see ourselves in our organizations in this light. By this time, we had lost awareness that the process of measurement destroys what is natural. Nature does not measure. Nature recognizes patterns."

For the past ten years, Tom has been studying a small number of corporations that seem to not be caught in this measurement trap. Toyota, widely seen as the world's top car company, has no standardized cost accounting system. Scania, a leading Swedish financial services firm, has a uniquely successful approach to new product development. In both cases, it seems that people have developed the capacity to focus on the means rather than the ends in complex human processes. Toyota manufacturing people, in Japan and elsewhere, spend years learning how to assess how the work is flowing, and to learn ways to quickly detect and correct errors as work proceeds. This involves measurement but measurement in the service of learning and enhancing a rich context of tacit knowledge.

"Nature does not focus on ends. Nature knows only means." Can we understand the deeper logic of nature -- can we learn to trust that focusing on the means will suffice?

There was an interesting connection between Tom Johnson's comments and Bill O'Brien's regarding profits. O'Brien emphasized the importance of building wealth, as a primary way the corporation contributes to society. But he warned that "there is a difference between profit and creating wealth". "Wealth production" he said "is an honorable goal - profit is about looking good". He suggested that there is so much latitude in financial reporting that short-term profit figures can be greatly distorted. Tom Johnson argued similarly that, for assessing the health of an enterprise, "profit is an almost useless measure." The challenge, of course, is how to develop improved ways to assess health -- something Tom believes a few firms like Toyota and Scania are doing.

In order to develop some common language and perspectives, a simple framework was suggested for beginning to compare experiences and questions. Efforts to bring about change and enhance learning capacity can be seen in three lights: the initiative itself, "first order consequences," and longer-term business consequences. This leads to three types of questions which have a bearing on what we assess, how we assess, and why we assess (Figure 1):

1. What are the conditions present which influence receptivity and predisposition to learn? We called these "preconditions," but they might alternatively just be called "environmental conditions." If, for example, a learning initiative is counter-cultural and/or was introduced insensitively - then assessment of its effect will be distorted. This suggests that assessment of an initiative per se needs to be done within the broader context of the local environment.

2. How can we discern "first order" changes in work practices broadly construed ? (e.g. new skills, news ways of working together, new social networks, new ways of organizing tasks and distributing resources, new ways of resolving conflicts,...). What we choose to observe is colored by the answers we seek -- so we need to be aware of who is asking the question and why. Is the assessment being done (designed) to enhance the long-term benefit of the business taking into account the dual agenda of corporate and worker needs?

3. How can we link changes in work practices to business consequences, given the complexities and indirect nature of these linkages as well as influences from external (e.g. market) forces?

Investments in developing new learning capabilities take many forms. Those that are the most important have the following characteristics:

- they are connected with real work goals and processes
- they are connected with improving performance
- they seek to balance action and reflection
- they afford increased amount of "white space," opportunities for people to think and reflect without pressure to make decisions
- intended to increase people's capacity, individually and collectively
- the focus on connecting: inquiry, experimentation, and reflection
- focus on learning about learning.

We called such efforts "learning initiatives." These go well beyond traditional educational initiatives. As Etienne Wenger offered, "a learning initiative is an initiative to improve an organization where learning is an explicit goal, 'learning' as defined by the participants themselves." Mike Beer suggested that a learning initiative is one where there exists a gap between "where we want to be" and "where we are". There are three levels at which this gap can arise:

1. substantive: business relevance, strategy, technology, e.g. we need to know more about "x".

2. context: broader questions, values, larger systems that shape interpretation and action; e.g. "fiefdoms" are internal organizational structures (in this example, covert) that are barriers to information flow.

3. learning how to learn; this addresses the capabilities we would need in order to reduce barriers.

An important difference concerns the intent of assessment. In one sense, when the intent is learning, assessment is inherent in any learning process. It is not possible to learn if learners cannot make sound interpretations of where they are relative to their goals. A child cannot learn to walk if she cannot tell the difference between one or two steps and several steps. Assessment and learning are inseparable, just as feedback and learning are inseparable. As Harley-Davidson's Tim Savino said, "I have difficulty telling the difference between learning and assessing," On the other hand, evaluation is not an inherent part of learning. It is not necessary that the child "evaluate" how she is doing as she learns to walk. Indeed, some researchers even argue that such evaluation can be counterproductive to learning, i.e. could cause the child to conclude that she has "failed". Such a conclusion may then become a self-fulfilling prophecy.

But the predominant colloquial use of the word "assessment" differs from assessing for learning. More often than not, "assessment" is used as a synonym for "evaluation." Indeed, frequently in our conversation, it appeared that we were not communicating, as one person talked of assessing as evaluating and someone else meant assessing as learning.

Jean Redfield of Detroit-Edison suggested that we distinguish "assessing for evaluating" from "assessing for learning." This legitimates both types of assessment, each of which should fall within the purview of this research initiative. But it also forces us to distinguish which type of assessment we are speaking of.

Peter Senge suggested that one long-term aim of this research might be to re-establish a healthy balance between assessing for learning and assessing for evaluating. If the balance has tipped to emphasis on evaluating to the extent that it actually impedes learning, learning processes will benefit from increasing emphasis on assessing for learning. "It is low leverage to complain that there is too much emphasis on evaluating, outsiders wanting to know if learners are adding value," said Senge. "It is probably much higher leverage to build our 'assessment muscle,' to help learners to get better at assessing for learning."

Building on these ideas, Etienne Wenger offered four additional distinctions concerning the nature of different types of assessment:

1. inherent: assessment that is inescapable in the learning process; inherent assessment is done by the learners and may be tacit or explicit.

2. exportable: assessment that creates a product (insight, data, stories, ...) that can be offered to someone who has not been part of the learning process;

3. extractive: assessment that "extracts" information from a learning process because someone on the outside has a "need to know" for their own purposes.

4. systemic: assessments that reveal relationships between different practices and allow people to see how they belong within a larger community.

Further, Wenger said each of these types of assessment can be participative (negotiable) or standardized. Participative assessment engages those assessing in a social interaction to make sense of reality. Standardized assessments attempt to establish measures that require minimal interpretation and are not subject to negotiation. For example, most traditional financial measures are standardized. They are also usually extractive, although teams might also produce standardized process measures as part of their inherent work practices.

Several audiences for assessment were discussed, but a basic break down into three key audiences for improving assessment made sense:

"innovators:" those attempting to implement new work practices;

"followers:" those who would like to learn from innovators and build on their efforts -- future innovators;

"outsiders:" those, either inside the organization or outside it, who want to understand what innovators are doing and/or judge their effectiveness.

The needs of these different audiences are very different. Innovators or learners need to develop "real time" methods for better understanding the fruits of their efforts; their assessing needs to be inherent and systemic. Followers are especially interested in "exportable" assessments that give them insight that can allow them to adapt new ideas, practices, and methods to their own circumstances. Outsiders have traditionally emphasized extractive measures, and especially standardized metrics.

On Friday the group was invited to reflect on what had arisen over the past few days with a view to going forward. Katrin Kaeufer encapsulated our overall task aptly with " we are assessing to learn and learning to assess. ".

Four themes were identified that seemed to capture what was felt to be important:

• Inspiring workshop - combines my two main interests - organizational learning and assessment/evaluation
• Motivates to read more of what the researchers here have written
• Gives a new approach to the organizational learning initiative/project in the World Bank.

- more awareness of the assessment component

I would like to try to write something on the bridging of evaluation and learning

I come to the conference with a curiosity about what this was about, hoping to learn and be stimulated, but with great ambivalence about the commitment the conference and beyond might demand. I found the conference stimulated imagination & out-of-the-box thinking and that was valuable. I found a connection between my own work and the topic of assessment (though the conference triggered more then just thoughts about assessment) through the larger conversation and the sub-group on bridging.

I am still ambivalent about how much time I can commit, given other obligations. I might be willing to write an article, perhaps with someone else in the group.

I keep thinking of assessment as a way that I can help people rationalize or justify doing learning organizational stuff in their work. In my culture, time and resources are hard to come by. People are afraid of things that they perceive as new or incremental.

Though this was an assessment workshop, I found myself thinking more and more of leadership -- the courage of people like Bill and John of Chrysler, Tom, Joe J, Peter, Bob Womac, me, and the role we all play in assessments.

I also wish some of the academics would be more humble. I was willing to take hits on my personality, my excitability, and my listening skills. Maybe they should get off their 27 syllable high horses, get grounded, and admit they have something to learn.

Stella: hold me accountable for health charts and matching with other companies and members.

PS. Thanks! Great job! You're a great VP- I'm in love!

There is risk with measuring the "result" of a "learning process." Learning is not to be motivated or controlled by "result," it is to be motivated in its own right. Since I came in late to the "group journey," I don't know the history but see to it that you are not "misunderstood" by newcomers. I, as a "newcomer," see:

Thank you. I enjoyed the stay and am filled with impressions.

What I am taking from this workshop are the evocative thoughts shared, particularly by Bill O'Brien and Tom Johnson, with a deep appreciation for the "resonance" and "tone" which they helped to sound. I think the direction we are heading feels very "healthy" and authentic to who we are individually and collectively. I also really appreciated the way in which Stella gently but firmly helped to shepherd the process along with the other members of the Organizing Committee. I am taking new conceptual insights gained in a "field" of genuine inquiry and exploration among us all.

The focus I would like to share in my "mini-paper" - not sure quite yet - but somewhere around new tradition & playing the music.

There are several key concepts that emerged:

- Assessment for learning
- Etienne's categories
- Bill Torbert's categories
- Political issues
- Value & virtue issues
- David's passionate presentations
- health reports
- listen to the music
- relationship with bosses
- what the learner knows about their learning (self assessment)
- a new tradition legacy

• I felt much more progress during the small group sessions then the plenary presentations and discussions. Some presentations were too vague, but they provided seeds that grew later especially in small groups and small conversations

• Lack of researchers present - a continual issue for this group.

• Real danger that this project won't speak to the "orthodox" in research or practitioner communities. Is this a research project??

I could write a bit on health reports as they are used in the industries I know best.

This has been a stimulating process that made me think of things I hadn't thought of before... Thank you!

I caution or see some challenge in us as a group becoming "knowers" in this territory. Much of the conversation indicated freeze points to assess ~ ~ ~ I am interested in assessing during the journey (while doing, as part of work)………There was a lot of "brain power" in this room, but at times I felt being in a position where I was hearing people advocating too much.

At the end, I worried that we forced old traditions that already existed and weren't patient enough for "natural" emergence - (e.g. revert back to work we'd do anyway).

Title: Why is it important at Ford to ->Assess Learning->Learn Assessing.

Thank you so much for allowing me to participate.

My head is swimming with ideas, concepts and new opportunities. It's the result of many present with me. People have offered their heart and soul and I have grasped them and added them to my own. I leave feeling more complete and whole then when I came. I'm not sure I understand everything that's happened. And perhaps I never will. I know it will take months, maybe years for all of it to present and activate within me. Yet some of the ideas that have been generated in my mind are present now and I am going to take those ideas and act on them. It is these types of gatherings that inspire me to continue on.

I probably followed/absorbed 20% of these past two days. My lack of expertise in industry, organizational learning, consulting, etc. pretty much limits my ability to write something that would be of any use. However, perhaps I could add something by helping to compile the bibliography.

I found very satisfying the richness of ideas, concepts, frameworks, and depth of thought. The openness and willingness to enter into unknown territory inspires the hope that we can begin to develop "new traditions" in research.

I also find exciting the possibility for this initiative to be an ongoing forum that holds the (creative) tension among folk with such different perspectives and cultures. Our conversation acknowledging the "logical-positivist academics", those oriented toward "bottom line" results ; the action oriented "pragmatists"; and the "new tradition" put us all inside the conversational space.

I am engaged by the deeper issues inherent in assessment - how do we make explicit the values we support through the very act of selecting the goals and data for assessment. Are we "only " improving current processes or are we also willing to explore ways of aligning management with alternative views of how the world works? ( e.g. mechanical vs. natural). And/or are we willing to intentionally base our learning goals on long-term legacy aspirations and the ensuing values. How do we do that in practical ways? These are deep and wonderful challenges and potential in this initiative. Tom and Bill's talks give us a strong foundation from which to begin.

I need more time to digest and "be" before I can write useful reflections. My cup is too full at the present time. Here's what I can say now:

I loved being present in the widened arc of the SoL community. I learning so much from new people and I am so grateful to have met others, including David Berdish, Tom Johnson, Anders Bröms, Bill Torbert; I will stay connected to these people.

I am also struck by the importance of this new work.

I will try to do one page of "bullets" abut my take on our meeting of the music group.

My mission coming out of these meetings, is to engage in a more focused effort to discover and articulate "how nature works." That effort, an ongoing and never-ending activity, will create the "story" or "the score" by which I will assess how well I am doing, personally and in my organizational lines. (By that score or story I will know if I "hear the music.") That is the new understanding I hope to share with others as I go forward.

Title: "Hear the music"

A) Learnings

B) Conditions for learning initiatives:

• gap: current reality <-> desired reality
• willingness/desire to learn
• safety/support

C) Process of interdependency of "assessing learning and learning to assess"

D) Title: "Listening to the Music"

Thank you to the organizing committee

From Bill O'Brien's "frustrations," the notion that a value-driven corporation will ultimately be successful and will have a continuing existence and develop a "legacy" is very appealing. It solves the problems of short term focus, and scorecard-driven management.

To change organizations (corporations) into operating this way requires a great deal of learning. Values are multi-dimensional and the conversations engendered are "richer." Therefore, understanding the effectiveness of learning initiatives in helping driving values into the corporate management process, is significant.

I want/ need our process to be considered by us as a learning initiative - e.g. to include exploring our preconditions, modes of reflection, etc. I am concerned about the "how" of our gathering & that it contribute to and maximize our individual and collective health and learning. There were brilliant moments without time or processes to take them deeper and wider, I also want to make sure our forms of thinking together are inclusive of all stakeholders - i.e. more opportunities for small groups, one on one, etc.

I am concerned that the next forum could become a competitive area for businesses contribution to research projects and fragment us. Also, as framed, could exclude consultants.

STELLA: Ask me about my research project developing a methodology for assessing on personal transformation.

I've enjoyed my participation in the last few days, in particular examining the links between the practitioner and academic communities. This has a afforded me an opportunity to hang out with practitioners and their concerns in a way that hasn't been available to me in some time. I am disappointed we didn't have more academics and I think this limits the potential for the most promising dialogue. I lump into the assessment challenge the kind of measurements that allows for publication in academic journals and think this should be included in our objectives even if its not SoL's focal point. The depth of my future participation will be constrained by the necessity of focusing on my dissertation. I still would like to be included in the unfolding dialogue.

I'd be happy to write a short piece on SoL's role bridging to the academics and practitioners.

The role-plays that Dave Berdish did on the last day of the conference crystallized for me a way forward. Dave played two ways that a learning leader might talk to a general manager about how a learning initiative is going. Then the general manager has to talk to a steering committee or board. These conversations are a key leverage point in shaping the assessments that actually drive decisions in organizations.

Let's get together several people whose responsibilities include the kind of conversation Dave role-played. Let's bring examples of how these conversations typically go--for better or worse. The two-column case format would be a good vehicle. Let's reflect on the examples and role-play new ways of having the conversations. I think what this could do for us is:

• The people who take part could build their capabilities for having these conversations and for creating the kind of relationships that work;

• From the standpoint of practical research, we could discover how people are handling these assessment conversations now. I think we'd identify generic dilemmas and paradoxes--which is the first step to developing creative ways of managing them;

• We could get to work on the kind of stories that a learning leader should be able to tell--and on creating conversations in which people can distinguish a good story-teller from someone with a good story to tell.

I offer to facilitate a session like this. Are there learning leaders in companies who would like to take part? And I'd like one or two researchers to join us so that we take advantage of the opportunity to learn what people do now and what issues they face.

The pattern of my life has been as a student/learner of the world. I surprise myself by slipping between worlds, and particularly am surprised at the relative ease of my success in the corporate world - a world that is "supposed to be" hard and difficult, maybe even death to people like me. Right now, I am overcome by the possibility of being part of something that brings wholeness to the corporate world. I feel like a pioneer about to explore and build a new way of life that is about abundance, wholeness, and fulfillment.

I am thankful for the gift of meeting, and the privilege of being with, the people here who are opening my eyes to the possible patterns before us, and look forward to continuing. I look forward to crossing boundaries……

I will provide an overview of where are we at Detroit Edison, and what are the possibilities for assessment of organizational learning as a piece of the transformation underway.

I very much enjoyed participating in the assessment workshop earlier this month. I think it was a good beginning and look forward to keeping the momentum going. As I gazed around the room and listened to the different perspectives I felt a little like I was directly witnessing SoL's rebirth....

On my trip back to Denver (yes, now home of the Super Bowl Champions!), I continued to think about the "new tradition," and thought I'd share these ramblings with you before they dissipate:

Most of my thoughts have centered on trying to figure out what the crux of the difference is between what we have called "assessment for learning" and "assessment for evaluation". I looked up the definitions of "assess" and "evaluate" and found that they are considered to be synonyms, which reinforces my sense that I do not think the difference lies in the use of those two words -- that indeed, we could use them interchangeably. I am thinking that the problem is not one of the use of the concept of evaluation per se, but lies in the quality of evaluation... and the intention.

The "99%" of assessments that I would like to see us shift away from, are characterized by either poor quality (misconception, resulting from weak assessment skills/muscle, a focus on symptoms/events, lack of knowledge, etc.), or negative or misguided intention (deception, intent to conceal or misrepresent, or the intent simply to advocate vs. inquire, etc.). Or both. It may be that rationalization is what is often substituted for assessment/evaluation. I think that to better describe the behavior we were referring to as, "assessment for evaluation", we need to find a descriptive antonym for learning...?

Given that all learning involves assessment, it makes sense that both learning and assessment share the same pre-conditions (and ongoing environment) for success -- trust, openness, etc. -- and that the tools and approaches we have for building learning capacity also contribute to the assessor's capacity to assess. Much of what SoL's work is about is the building of will, of intention -- the aspiration and courage to reach beyond rationalization -- and not just the building of capacity. To bring about the "new tradition" we will need to address both will (personal mastery) and capacity. Given Bill O'Brien's comments about the role of knowledge vs. virtue in transformation, the emphasis may need to be on personal mastery -- the more controversial aspect of SoL's work for most corporate members, I believe.

As my small action step, I hope to work with a group of software developers within U.S. West to shift their project peer review processes (design reviews, project management reviews, etc.) toward emphasis on assessment for learning. I look forward to drawing upon the resources of SoL assessment project members, and hopefully contributing to those resources as well.

The meeting was a good start on a new (yet old) area of focus. I would like to see the common understanding developed here extended beyond the smaller working groups that came to think about this topic together. There are several points which were discussed in this meeting that need to be more broadly considered, particularly by anyone or any organization designing and carrying out a programmatic learning initiative.

• Learning programs in organizations are ultimately about change, and hence need to be carefully considered, planned, and supported like any other deliberate organizational change effort. While learning programs are not often thought of as organizational change efforts, they seem to inevitably have that characteristic as they become successful. Without some deliberate planning for change on the part of the leaders for learning, and their seeking support from senior management, these proponents, and the participants they engage, seem to be putting their reputations and careers at risk.

• Consider the following analogy between dialogue and assessment. "Dialogue" has become an accepted technique for developing listening and learning skills in organizations. People's direct experience with dialogue, and what they observe in other groups who are able to regularly practice dialogue, provide ample evidence for its benefit and effectiveness. One way of describing dialogue, as meaning "flowing among," emphasizes its characteristic of conversations which people have that serves to develop common understanding, rather than discussion, which debate points and promote the making of decisions. A parallel similar to that of the difference between dialogue and discussion can be drawn between assessment for learning and assessment for evaluation. As organizations develop and practice skills for assessment that promotes learning, they will gain similar benefits to those of dialogue.

• The process of getting a wider group of people formally involved in the assessment of an organizational improvement effort is itself an organizational learning process. A formalized and disciplined process of collecting and reviewing data from a improvement program, not solely for the purpose of "evaluating" it, but to learn from it, and develop people's judgment for what is effective, will engage them in thinking about improvement programs, and connect directly to the future actions that they can take based on their insights.

• It can be seen as irresponsible, given the experiences which people shared at this workshop, to begin a learning initiative, whether at the individual skill, team, or organization-wide development program level, without having senior management support and a larger assessment/feedback program in place. Rather than considering it as a luxury to develop an assessment program after a learning initiative has been underway, assessment (for learning) needs to be planned and designed as part of the improvement program. And, through an assessment program open to others in an organization, those outside the "learning team" can be engaged in judging its effectiveness, efficacy, and broader applicability.

I believe that in the combination of Bill O'Brien's frustration & Tom Johnson's "all we have to do is build the practices that are consistent with nature & unwind those which are not" lies a "new tradition" for assessing and learning.

There is, I feel, a deeper purpose emerging. The best I could articulate it now would be to say that all our efforts to build learning capabilities are incomplete until we engage with the measurement culture that dominates contemporary institutions, and the mechanical view of reality which undergirds it. This mechanical paradigm is elusive: you cannot change it by trying to change it. But, if (1) people start living more authentically and (2) we start being serious about looking at the consequences, individually and collectively, the paradigm might naturally be superseded.

These were rich, chaotic, caring, illuminating conversations that very much may represent a local, long-term flowering of a new tradition that engages in "Cooperative Ecological Alchemy," or "Transformational Action Inquiry," or more simply, listening and dancing to the music.

I loved the diversity, the dialogue, and the appreciation of the significance and challenge of committing to this -not "work"-but "play."

This group began to characterize some of the attributes of what a new science of assessment may look like. Below are some first thoughts to seed a "new tradition" of assessment.

All the participants were asked to reflect on the context for beginning the Assessment for Learning Research Initiative and to answer four questions before coming to the Research Forum. Below is a copy of the context-setting letter:

Dear

We have only two days together and a challenging task: exploring and framing a research program into assessment of organizational learning. And -- perhaps most importantly -- engaging each other in ways that lead[s] us into ongoing inquiry and collaboration to develop such a program.

Because time together []is so short and the conceptual and practical issues are many, we ask for a little of your time beforehand to reflect on the project.

Broadly speaking, this project grew out of the recognized need to better understand and document the linkages between organizational learning and business results. This need cuts across all dimensions of the SoL community. Practical business leaders must continually address the question of "What is the return on our investments in organizational learning?" Consultants must likewise, especially those who sustain long-term relationships with key clients. Researchers are naturally (and professionally) skeptical about change strategies which are popular yet lack deeper foundations, and consequently the potential for significant business impact. Few challenges command greater interest among all parties than the challenge to develop more systematic and more usable methods to assess the consequences of organizational learning innovations.

On the other hand, the aim should not be to prove that the investment in learning caused any particular change in business performance -- that would be logically impossible. The aim should be to better understand, given appropriate time frames and conditions, the business consequences of learning initiatives -- consequences for economic performance; for workplace conditions; for relationships with customers; for communities, and other key stakeholders; for the sustainability of the natural environment; and for whatever additional outcomes the members of the enterprise consider important.

Who is the audience for this research? One is senior management and other decision makers not directly involved in learning initiatives but none[]the[]less accountable for judging their effectiveness. The second is people directly involved in such initiatives. Their need to know is somewhat different: how can better assessment lead to better learning? In distinguishing these two audiences it is also useful to distinguish two meanings of "assessment"[:] awareness versus evaluation. (Unfortunately, colloquial use of the term "assessment" is rather ambiguous on this count -- for example, in educational research, it is used as a synonym for evaluation.) For learners, awareness is essential; evaluation is optional, and sometimes counterproductive. All learning processes, whether individual or collective, depend on the awareness of the learners [I don't agree with this -- it presumes, for example, that nonhuman species that learn are "aware"; perhaps we could agree that more complex forms of learning require or are enhanced by awareness]. Feedback, in the sense of heightened awareness and understanding of the consequences of one's actions, is essential to learning. Lastly, our audience includes researchers seeking [enriched] theory based on [deeper] empirical understanding of the processes whereby people in real work settings increase knowledge and improve enterprises.

One of the challenges is how to collaboratively design projects such that the needs of all these groups are met. For example, researchers are prepared to take a long time frame to study complex organizational change processes, while practitioners, and to some degree consultants, must justify investments of people and dollars based [on] shorter term improvements. Different groups also have different emphases on what is being measured and different criteria for determining what constitutes a successful project. Our overall goal is to design and carry out this research in partnership [among] researchers, practitioners, and consultants. Moreover, we suggest that one of our principles should be that whatever assessment approach we are advocating and implementing should also serve our own efforts as a community seeking to learn together.

These are some of the central issues in our thinking. Our conversations so far forewarn us of the many different perspectives that will be brought to the table in the January workshop. Bearing this in mind, we want to create an opportunity for divergent conversation[s] as we conceptualize the project and its aims.

The tension for us is to also provide enough time for exchanging basic information on the different approaches and methodologies sin use now. We plan to lay out a modest number of alternative perspectives on assessing learning -- both what it is and how it can be observed. What methodologies [are we now using] to connect learning initiatives to observable individual and organizational consequences? How do different people operationally define enhanced organizational capacity? What methodologies do we have to connect these organizational consequences and capacit[i]es to business results?

On the basis of these discussions we need to develop some means of integrating across the perspectives. This could take the form of an integrating framework, common principles that characterize different approaches to assessment, or a common statement of purpose to which all can subscribe -- or some combination of the above.

To begin the inquiry and to help us plan the best use of our time in January we'd like you to answer three questions:

1. What could we learn by studying the assessment challenge?
2. What aspects of assessment of learning are most important to you and why?
3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?
4. What readings would you suggest?

1. What could we learn by studying the assessment challenge?
This would lead to an inquiry process which raises so many issues to be addressed: Can all consequences of organizational learning be assessed? What actually needs to be assessed? How can we evaluate the effectiveness of learning? Which approach or methodology would be useful for determining the ROI from learning? Can we measure the growth in the learning capacity of an organization? Can we develop practical tools? Will the assessment lead to more "productive" learning?

These are but a few that I can think of and I am positive that in a collaborative inquiry process, many more issues will be raised and addressed. The joint inquiry into the assessment of learning is in itself a wonderful learning process, and the different perspectives can really lead to a deeper understanding of the complexities underlying learning for all those involved.

I believe the assessment challenge is critical for developing communities of practitioners, consultants and academics committed to learning. The theory of organizational learning in itself is neither "convincing" nor "promising" for the majority of enterprises, especially those who seek to have visible benefits when they invest in learning. The assessment of learning is also crucial for identifying learning practices which are beneficial to the organization or effective for improving organizational performance or different purposes.

2. What aspects of assessment of learning are most important?
All the questions I mentioned above and probably many more. I would start with inquiry into the assessment itself. What is the deeper purpose? What do we want to achieve with it? What can organizations achieve with it? Then probe into the organizational consequences of learning and quality of learning. What would be the profile of an ideal learning organization? How can high quality learning be defined or distinguished? Which performance criteria can be related to learning or unlearning? At which integrative level should the assessment be (individual, group, whole?)

Perhaps most important of all, is putting all the knowledge created by the inquiry process into use and develop practical tools which would encourage investment in learning and also lead to more effective learning in organizations.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas into implementation?
Not much I guess, if I feel I can both contribute and learn. If I am intellectually challenged, if there is a real dialogue going on, in a trusting, non-judgmental environment, if I enjoy the process and feel that there is sufficient diversity yet common ground and understanding among the people involved and the joint inquiry leads to "knowledge" creation, what could be more appealing than being actively involved?

4. What articles etc. would you recommend?
Nyhan B., (1991) Developing people's ability to learn. Brussels: European Interuniversity Press.
Burt R. S. "The social structure of competition" in N. Nohria & R.G. Eccles (1992) Networks and Organizations. Boston, MA: Harvard Business School Press.
McKenzie, J. (1996) Paradox: the next strategic dimension. McGraw-Hill (UK).
Nonaka I., and Takeuchi (1995) The knowledge-creating company. New York: Oxford University Press.

1. What could we learn by studying the assessment challenge?
While measurement and assessment tools in themselves have practical value for both practitioners and researchers, the most important thing we gain by engaging the assessment question is a clarification of what it is we mean by organizational learning. To measure something, we have to be able to be able to define it and develop the construct. By measuring it, we gain a clearer idea about what it is we are interested in and so on. In other words, we will learn what it is we are interested in through an action research process.

2. What aspects of assessment of learning are most important to you and why?
I am interested in defining a comprehensive model or theory of organizational learning and methods for assessing some or all of its components. The questions that interest me are: What do we mean by organizational learning? To what extent are changes in organizational design and processes intended to correct errors? To what extent is it the capacity of the organization and its members to engage in an inquiry process? To what extent is it a cognitive process and to what extent is it an emotional process in which the organization and its members confront their own assumptions and beliefs - how they frame problems or engage them.

It seems to me that that organizational learning involves closing the gap at three distinct levels:
1. The substantive gap - solving the task problem, whatever it is
2. The organizational gap - realigning the organization's design and behavior
3. The organization development or learning gap - learning about how to learn about the other two gaps (this involves group and individual processes.)

A problem that faces us in assessing organizational learning is the difficulty of the learning task. That is, in order for us to understand how much learning has occurred, we need to be able to state what learning needs to occur. That is what the gap was that had to be closed. I am convinced that we need to be able to find a theory that will help us understand the difficulty of the learning agenda.

Finally, I am interested in how organizational learning processes influence and produce change in the organization's leaders and their enactment of leadership.

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas to implementation?
Frankly I am not sure. I have my own stream of research going and so I guess I would be most interested in being involved in a project that furthers my own work in some way. Since my work is concerned with intervention intended to produce organizational change and learning, I would be interested in applying assessment methods and approaches that might emerge from our joint deliberations. And, I would be willing to contribute findings to some higher order joint effort of which I was a part. I guess I am arguing for a loose partnership in which our joint work contributed to assessing the intervention efforts of our clients/research targets (I assume most of us are doing intervention work) and which collectively added up to a body of knowledge on assessment of organizational learning.

4. What articles would you recommend we read?
I do not have a list at hand now, but I think Argyris' work on learning is relevant. Schein's work on culture and leadership is relevant. A recent conference at USC on leadership and change had a number of interesting papers - one by Heifetz and Laurie, one by Pascale, and one by Bob Quinn at Michigan. Cases like Apple Computer: Corporate Strategy and Culture (I wrote this case) offer a great way to engage the question of what is organizational learning. Cases on Hewlett Packard Santa Rosa Systems division describe organizational fitness profiling about which I write in one of the six papers you sent out.

1. What could we learn by studying the assessment challenge?
One goal for my research is to develop better tools and approaches to help people understand the extent to which they are actually developing capabilities in systems thinking, individually and collectively. This development of systems thinking assessment tools may be an outcome of our work on the assessment challenge.

2. What aspects of assessment of learning are most important to you and why?
One aspect of assessment of learning I'm particularly interested is in assessing "feedback thought", a central notion of systems thinking. How do we develop and assess the development of an ability to think in terms of interdependence, mutual causality, self-reinforcement, balance, stability and instability, structure and behavior while harnessing some of the deepest ideas of the natural, social and behavioral sciences?

3. What would need to occur at the workshop for you to find irresistible becoming a part of a core group that would develop these ideas into implementation?
What would need to occur is for a core group of researchers, practitioners and consultants to put their best thinking into the development of systems thinking development and assessment processes.

4. What articles would you recommend we read?