The view across: patterns of success in assessing and improving student learning

The Authors

Lynn Priddy, Director of Education and Training, Higher Learning Commission of the North Central Association, Chicago, Illinois, USA.

Abstract

Purpose – The aim of this paper is to describe how academic institutions that focus improvement of student learning do much better than those that focus on compliance and assessment.

Design/methodology/approach – A reflective observation of institutional interaction with the North Central Association Higher Learning Commission, especially the 264 colleges and universities that have participated in the Commission's assessment workshops, provides insight into the characteristics that make the most positive difference.

Findings – The paper finds that academic institutions do better when: assessment is best understood as the means and student learning itself as the end; shared responsibility and collective capacity are intentionally developed; internal leaders, of different types, are identified and developed; collaborative processes that actively engage people replace concerns about buy-in; institutions jump in and learn as they go along; program review becomes an area of shared faculty/administration interest; changed, parallel or separate core processes permit attention to enduring issues; and institutions begin wherever they chose to begin and from there develop the means to complete a full cycle of outcomes assessment. Another more recent emphasis is the need to inform the public and other stakeholders about what students are learning.

Originality/value – This paper draws on the insights of those who work at the Higher Learning Commission, who share the unusual perspective of having experience of dealing with hundreds of academic institutions.

Article Type:

Conceptual paper

Keyword(s):

Students; Learning; Assessment.

Journal:

On the Horizon

Volume:

15

Number:

2

Year:

2007

pp:

58-79

Copyright ©

Emerald Group Publishing Limited

ISSN:

1074-8121

According to Samuel Butler, “Life is like playing a violin solo in public and learning the instrument as one goes on.” Funny, he could just as easily be talking about assessment of student learning (Jonathan Keiser, The Higher Learning Commission).

If you have not yet indulged in the EDS advertisements, click on youtube.com, search for and watch the video, We Build Airplanes in the Air (EDS, n.d.a). You might view Cat Herders (EDS, n.d.b) as a sequel. Both offer instant, uncanny insight into nearly two decades of work in assessment of student learning, and how, for both accreditors and institutions, this requires re-seeing their life's work.

Both videos champion near heroic effort, activity, and dedication, not unlike the profuse effort and activity in assessment of student learning that has characterized a key aspect of the accrediting relationship for the last 18 years. Consider the following slice of history.

On November 29, 2006, 14 teams of faculty and administrators from diverse institutions representing seven states became the pioneer cohort in the launch of The Academy for Assessment of Student Learning, a new initiative of The Higher Learning Commission of NCA[1]. A further 32 institutions will follow by summer 2007, all beginning a four-year action portfolio focused on student learning, targeted at accelerating and advancing efforts to assess and improve student learning, and designed to build institution-wide capacity for and commitment to assessment of student learning. Also by summer 2007, the 500th institutional team (more than 3,000 people) will have attended one of the Commission's team-based, multi-day workshops on assessing and improving student learning, 264 different institutions just since 2004, representing all 19 states in the Commission's region, as well as every major Carnegie grouping. Again by summer 2007, the Commission will have evaluated for accreditation purposes all 1,000+ member institutions at least once, if not twice, since first positing the fundamental importance of assessing and improving student learning in 1989, requiring institutions to create a plan and to begin implementation by 1995. More than half of these institutions will have been visited yet again by summer 2007 with interim monitoring on assessment of student learning. Add to this the 3,400 individuals who attend the Commission's Annual Meeting each year, a conference which includes multiple tracks of sessions on teaching, learning, and assessment led by institutional speakers and resulting in a Collection of Papers that capture the successes and experiences in assessing to improve student learning.

The example begins the story in the middle. Indeed to start at the beginning, to inventory all the institutional innovations and reformations in curricula, pedagogy, learning environments, technology since the 1980s would merely add testimony to the claim that higher education and accreditation have committed and have sustained a stream of substantial resources and effort to assessing and improving student learning and to advancing the quality of higher education. In fact, since first positing the fundamental importance of assessment of student learning in 1989, The Higher Learning Commission (HLC) has repeatedly deepened its commitment, most recently in its newly adopted Criteria for Accreditation which make explicit the centrality of student learning to educational and organizational quality and which make assessment and improvement of student learning more fundamental than ever to the accrediting relationship (HLC, 2003).

This intense investment of effort and activity raises an obvious question: Has assessment of student learning made its way into what Ewell (1991) calls the “bloodstream” of institutional life or are we still heroically trying to herd cats and fly airplanes while still building them?

Since 2004, 264 teams of faculty and administrators have attended the Commission's assessment of student learning workshops, offering a unique opportunity to see across the efforts of these institutions. This slice of all the institutional commitment, effort, investment, and success serves to represent what the Commission has learned about the advancement of its 1,000 plus member institutions in assessing and improving what and how well its students learn.

These questions frame the Commission's learning shared in the following pages:

Reflecting on common ground: observations on what institutions are doing and discovering

Not chaos-like, together crushed and bruised,But, as the world harmoniously confused;Where order in variety we see,And where, though all things differ, all agree (Alexander Pope).

A simple exercise begins every assessment of student learning workshop. Teams of faculty and administrators are asked to discuss for a few minutes and collectively agree on two or three things that matter most about their work, why they do their work, and what they hope for the institution. You may already know the answers: students, learning, and institutional identity (read mission, type of institution, and perception of status of that institution). Research and service tend to come in later on the top ten list. In fact, of 264 institutions, 264 arrived at similar responses whether rural tribal community college or a research university. Whether simply espoused theory and “aspirational talk” or actual theory-in-use “walking the talk,” the voiced unity amidst diversity speaks to the integrity of mission and purpose these teams value. These teams care deeply about and are committed to their students, what they learn, and the distinct missions of their disciplines and institutions.

What follows is a set of observations on institutional efforts to assess and improve student learning. These observations are not specific to any one type of institution. On the contrary, the intent was to discern themes that hold true across diverse types of institutions and that address the first question above, “What catalyzes institutional efforts to assess and improve student learning and collective institutional commitment to and action for student learning?”

The “data” grounding the observations have been drawn from both structured and informal conversations and interactions across the 264 workshop teams from diverse institutions attending multi-day assessment of student learning workshops held by The HLC from 2004 to 2006. Each institution sent a team of five to ten faculty and administrators; more than 90 percent of teams included the chief academic officer. The interactions and conversations produced direct feedback and commentary from these institutions, as well as from Commission staff and workshop mentors.

Whenever possible, quotes from Commission staff, assessment of learning workshop mentors, and institutions punctuate the observations, making them more concrete, bringing in the collective voice, the stories of experience. Like data, these observations and quotes do not provide answers; they must be interpreted, discussed, and analyzed to discern what may be meaningful and useful to your institution. If you read the literature on teaching, learning, and assessment, many of these observations will resonate with much of what has been already said. They are, however, approached not from the angle of good practice in assessing, but rather from collective experience in response to this question: “What catalyzes institutional ability to integrate assessment meaningfully into organizational life and what builds collective commitment to and increased capacity for improved student learning, educational effectiveness, and organizational quality?”

Observation one

Institutions shift their focus and their conversations to asking, answering, and acting on meaningful questions about student learning, replacing compliance to an assessment mandate with commitment to student learning and to assessing so as to improve that learning. Learning becomes the ends; assessment, the means, not the reverse.

Don't establish the boundaries first, the squares, triangles, boxes of preconceived possibilities,and then pour life into them, trimming off left-over edges, ending potential (A.R. Amons).

Very early in the interactions with institutions, it becomes clear which will accomplish much in fairly short order and which will struggle to find purpose, perhaps doing much, but not accomplishing a great deal. Team dialogue, both the verbal and nonverbal, reveals which institution they are. One team huddles together (usually with a person typing madly), questioning: “It's not just writing competence is it? We say they'll be able to write maturely about the field or major, so what shows us they can do that in their papers? What does maturely mean? What do we really want to do? Do we have any examples?” Another team, a few feet away holds a very different discussion, leaned back from each other looking to the mentor, “So how do institutions do assessment after they have these outcomes? What does the Commission expect – how far along do we need to be – I mean is it enough to have our outcomes and some data? What data do we need to have and how much?” Institutions that struggle, grind themselves down in thinking that assessment is the ends versus the means. A focus on learning is lost in the need to know how to assess, the right structures, the best models to copy. On the contrary, a team that grabs onto the questions that matter most to ask and answer about their students' learning naturally link assessment as the means, not the ends, researching what will get to and reveal that learning, then figuring out how:

Many institutions have made assessment an end in itself and it's really a means. For that reason, it's been very difficult for many of them to “buy-in.’ Once they experience assessment as a tool, a very important tool that leads to improved learning over time, it's as if a bell goes off. Assessment now has value, and by implementing it in a serious manner, institutions discover that student learning overall improves (Mary Breslin, The Higher Learning Commission).

The shift defined in this observation moves the assessment focus (What is it and how do you do it?) from the ends to the means, and reasserts learning appropriately as the goal (What are students learning? Is it the right learning? What ways of assessing will give us the evidence we need to know what and how well they are learning?). Over the course of the workshop, institutions that remain focused on assessment as the ends and on compliance to “what the Commission and other accreditors expect us to do” often follow these patterns:

Of course debate about ways to assess, moving quickly to implementation as experimentation, assessment planning, or even understanding the multiple perspectives on assessment can be valuable, in fact, necessary. It is the discussions of assessment without ties to intended learning; i.e. the lack of context and purpose that present the issue for these teams.

In contrast, teams engaged in posing, talking through, and acting on conclusions to profound questions about what students should and are learning tend to:

Institutions that have made the leap to assessing student learning as a means of advancing the institution tend to be involved in dynamic, ongoing dialogues about what students should be learning and why. This dialogue helps institutions know what successful learning looks like and how it is relevant to the world beyond the institution. Asking and answering the right questions, not reproducing off-the-shelf models, is at the heart of good assessment practices. Because you are your own institutional experts, you are the right group to pose these questions and construct your models, informed perhaps by others, but yours to create (Ingrid Walker, The Higher Learning Commission).

In The Answer to How Is Yes, Block (2002, unnumbered transition page) opens by saying: “Transformation comes more from pursuing profound questions than seeking practical answers”. The observations are clear: The power of intentionally framing and striving to answer questions about student learning, in fact teaching and learning, cannot be overstated. As important, is the need to intervene with questions about learning into the conversations of institutions stuck on how to assess or focused on compliance to assessment versus commitment to student learning.

Observation two

In addition to cultivating the abilities of faculty and academic administrators to assess and improve student learning, institutions intentionally and persistently develop deep institutional commitment to, shared responsibility for, and collective capacity for improving student learning, educational effectiveness, and organizational quality.

A shared vision is not an idea … It is, rather, a force in people's hearts, a force of impressive power ℓ At its simplest level, a shared vision is the answer to the question, “What do we want to create?” (Senge, 1995).

“It's a culture of assessment we want!” declares one team member. “No, it's a learning-focused culture!” counters another. “What about a learner-focused, student development culture?” offers a third. Teams from institutions may not agree on what culture it is, but across the 264 institutions, only a few continued to describe the ultimate goal as implementing an assessment plan. The vast majority define the ultimate goal as a transformed culture. Drawing this conclusion has proven to be critically important, for implementing an assessment program, plan, or system seems to be designated as the responsibility of academic or instructional affairs. Transforming a culture requires the whole institution to own student learning as central to organizational quality and all employees to own the institution.

We have seen that where it has been done richly, assessment has the power to support a faculty transforming itself from a collection of individual subject-matter experts to an authentic, academic community of scholar-educators with a shared commitment to student learning and development. To get there, though, requires courage on the part of faculty to pursue a deeper vision of higher education and recognition on the part of administrators that faculty are not only members of a department but also members of the institution with institution-wide responsibility. And finally, it requires an accreditation process that fosters both (Monica Manning, NovaLearning, Inc.).

Over the past three years, many institutional teams have compared assessment of student learning to strategic planning, and as systems they do bear remarkable planning, evaluation, and improvement similarities. Often forgotten however, is that a good strategic plan is driven by a compelling vision of the future and the implementation of the strategic plan designed to realize that vision. At its best, a good strategic planning system engages the whole institution in the work of the institution and done well, it not only accomplishes its goals, but at the same time builds the capability of the institution to achieve even more. Student learning, the whole student experience, and the intended development of the student are the work of the whole college. It is not enough to assign assessment as a system to instructional or academic affairs; it needs a vision for what students will learn that aligns with the institution's mission, marks the distinctiveness of the institution, defines the future of learning at that institution, and thus requires the continued development of the institution's capabilities to continually improve student learning.

Institutions intentionally cultivating commitment, culture, and capacity:

What is quite noticeable is the combination of both talk and action. What is noticeably missing is a focus on assessment. The institutions repeatedly talked of starting with a focus on student learning and mission and emphatically repeated that beginning with a focus on assessment defeats efforts to build a culture that will soon embrace it naturally. In fact, these institutional groups talked from experience; most had spent several years focused on implementing assessment before stepping aside to cultivate culture or commitment. Essentially, these institutions deepened, strengthened, or initiated “a collective commitment to assessing for learning that is anchored in (1) intellectual curiosity about student learning, (2) institutional principles of commitment, and (3) meaningful beginnings” (Maki, 2004, p. 1).

So we've engaged in a paradigm shift that calls upon the cycle of introspection and consultation for purposes of accreditation to align institutional planning and improvement processes. During this time, I've become more deeply involved in the assessment of student learning, and in helping my colleagues understand the movement associated with shifting the focus to understand what we do in terms of its impact on student learning. As a result, we're working to solidify our definition of “effective as an institution” to emphasize that we are “a place where students learn”. I'm confident that we've always been such a place, but I know we're more intentional about articulating student learning as central to our purposes (Elaine Klein, Assessment Mentor, UW-Madison).

Observation three

Institutions designate leaders, intentionally and pervasively build leadership throughout the institution, and expect these leaders to call the institution into shared or split responsibility for student learning and to revamp systems, processes, and structures to ensure the centrality of student learning.

We all know the metaphor of being able to “step back” far enough to “see the forest for the trees.” But, unfortunately, for most of us when we step back we just see “lots of trees” (Senge, 1995).

How does your institution talk about assessment of student learning? Is the conversation mostly about the cost, mandated burden, and impact of assessment to your work or do you talk about it as critical to your work, an investment, a foundational process, as research into what matters most? Achieving meaningful, useful, workable, and reasonable processes for assessing and improving student learning is made possible through leadership and is near impossible without influential leaders who can call for change, align processes and structures, convene conversations, sustain priorities, and channel resources. This is perhaps one of the most striking and most reinforced pieces of learning observed across institutions; it has remained so since the early research of the Commission on what evaluators found to be critical to success (Lopez, 1998). Whether associate degree, baccalaureate, or graduate institution, private or public, for-profit or not for-profit, this observation resounds: institutions must identify a senior administrator who will consistently and persistently advocate and champion assessment of student learning throughout the institution.

“Sorry, our VPAA had to cancel last minute, but she said to come back with recommendations and ideas,” the apology spills from team after institutional team attending the assessment of student learning workshops, even entering the Academy for Assessment of Student Learning. At best, these groups tussle with “… we're not sure how things get into the budget or link with the strategic plan.” More often, the team grounds to a halt in the dilemma of “we'd really like to do __ but don't know if the institution will back us or what the Dean will think,” and in the worse case scenarios, “… well none of this makes sense; there's just too, too much to consider; I don't know why we're here; can't you just tell us what you expect us to do?”

According to Tagg (2003, p. 335), leadership is part of the “essential scaffolding for changing institutional processes and structures”, and learning paradigm colleges ensure both structural leaders, who hold the position and role in the organization, and functional leaders, who take on the role out of personal mission and interest and recruit others to participate (Tagg, 2003, p. 338). A third kind of leader, perhaps a form of Tagg's functional leader, has emerged in observations across the institutional teams: the knowledge facilitators, individuals that literally absorb all the findings and information from research, benchmarks, and good practice and delight in being the Wikipedia of student learning, assessment, and accreditation for their institutions. In The Tipping Point, Gladwell (2002) would call this third group “Mavens.” In the groups attending the workshops, they are pivotal, just-in-time knowledge generators.

What are the core attributes of these leaders? The observations bear out Tagg's (2003) and Maki's (2004) claims and read like any management or leadership textbook:

In sessions with leaders across 264 institutions, what do they say they specifically do to hold student learning central?

At institutions that have been focusing on assessing learning for several years, leaders expand involvement:

When faculty, student services professionals, and others on institutional teams are asked what matters most in leaders and leadership, three common themes appear repeatedly in the pre-work surveys and workshop affinity maps:

  1. They stay focused on student learning and “our other priorities,” respecting and giving us space, time, and resources.
  2. They ensure leadership and leaders persevere, “stay with us,” “support us,” “even through turnover.”
  3. They know and make assessment meaningful, useful, workable, and reasonable. Leaders take part in the learning, innovation, and pain of making assessment of student achievement real.

Anyone reading this list may say under their breath, “Oh and sure, just add walk on water.” However, that assumes all these abilities are in one person; in observing successful institutions, there are many leaders bringing their diverse skill sets into collective action.

Observation four

Institutions create and sustain collaborative processes that engage people in the work of assessing and improving student learning, that operate on the basis of collective agreement and responsibility, and that quiet the questions about buy-in.

Few people say they want more meetings. Many people say they miss good conversation. What does it take to create the space for a compelling conversation – one that goes beyond just talking to having meaningful consequences? Thinking together – that's more than just talking, that's going to the next step of sharing the process of how to resolve problems or address new opportunities. It's recognizing that no one has the whole answer, that it's in the collective wisdom of people who care that we're most likely to find the best ways to work together. We need to create the space for conversations that are compelling – conversations of consequence (Manning, 2004, pp. 1-2).

When asked “What did you do to get everyone at the institution committed to student learning and assessment of student learning?”, the most common responses (preceded by a pained expression) were something like these:

We didn't. We just said we needed to implement assessment, so we started.

We did that, but it didn't work.

We're still creating our assessment plan.

[Groan] Oh we talk and talk and talk about mission, culture, values, vision, outcomes. We never actually do anything.

The pursuit of “buy-in” comes in third or fourth on every institutional teams’ “to do” list. Immediately when said, individuals assume it means “faculty buy-in,” but across institutions and members of their groups, its flipside is also true, faculty talk equally of lacking administrative support, knowledge, or awareness of what is needed. In short, however, a key problem is the pursuit of “buy-in.” “Buy-in” translates into compliance; i.e. “getting you to buy into what I see as a priority.” Engagement is wholly different. A short story told by a chief academic officer may clarify the difference:

“I want to know what others are doing with their missions,” it began. The statement caught me, as I assumed as had been in the past that the conversation would be about appropriate learning outcomes and aligning them with the mission or about revisiting the mission documents to ensure learning was central or at least the “words were in there.” To my delight, the speaker, he launched into the story of his institution. “I determined to hand over the institution to its people. Our mission statement is brief, rather vague, and has no references to learning. Revising the mission means engaging the board; we wanted instead to create a grassroots definition by the institution. We simply handed the mission, as is, to the entire faculty and all others who were interested and said, “If you were to interpret the mission in terms of the most important learning you intend for our graduates, what would the definition of this mission be?” We provided six months, resources and time for small meetings anywhere and everywhere –small groups who clustered naturally and had the freedom to bring in anyone (students, community) … with only two assumptions. Whatever you create must be defined to the point of actual intended student learning that we'll hold ourselves collectively responsible for assuring. Whatever is decided will determine how we revamp the entire curricula for the baccalaureate, most pointedly our general education.” At the end of six months, mission intact, two overarching general education goals emerged with completed definitions, objectives, drafts of actual learning outcomes, and beginning outlines for overhauling the entire curricula, including general education. The two overarching goals are (1) to develop the engaged citizen, and (2) to develop the reflective practitioner. We are here as we are now creating the implementation plan that will transform the outlines into reality. You can't imagine the vigor, the excitement, the energy for this. It's our mission and we know what that means in terms of learning.

Gladwell (2002) explores social epidemics, the emergence of whole new ways of doing something or ways of being that infect and spread quickly throughout organizations. Imagine if assessment coordinators operated as if passion for student learning and assessing for learning were contagious, even infectious? Could an institution create a cultural tipping point that leads to an epidemic of insatiable curiosity about what and how well students are learning? The observations across the 264 workshop institutions that are succeeding and failing in buy-in or engagement seem to reflect Gladwell's principles. Engagement replaces buy-in:

You ask, “How do we get faculty to buy-in and do assessment?” I ask, “When do you engage faculty in talking about teaching and learning? About what they need and need to know to make learning happen?” (Robert Mundhenk, Lead Assessment Mentor, Former Senior Fellow, American Association for Higher Education (AAHE)).

In a soundbite, the message is that assessment of student learning must be meaningful, useful, workable, and reasonable. Engagement demands these.

Observation five

Despite the perceived high stakes, institutions deliberately take the leap to gathering and analyzing evidence on student learning and in the process learn what evidence to gather and how to analyze it. Institutions experiment, trip up, learn, and try again; they persevere.

It doesn't work to leap a 20-foot chasm in two ten-foot jumps (American proverb).

This observation would seem to be obvious, but in reality the most difficult struggle of the institutional teams at the workshops is multi-fold: determining what is meaningful evidence of learning; gathering evidence of student learning, particularly at the program and general education or institutional levels; finding space and time outside of hallways or episodic events to analyze, interpret, and recommend change based on the evidence; garnering the resources and acting on the recommendations; persisting to re-enter and repeat the cycle, gathering data again to see if the change made a difference. In fact, what best characterizes institutions attending the workshops is this struggle to employ actual evidence of student learning beyond the course level and beyond standardized tests.

Many issues remain constant. The majority of people that touch assessment data do not know how to interpret the information that is available. Whether it is a compilation of reports based on standardized tests or a collection of student portfolios, confusion and uncertainness clouds their work. The elaborate and disconnected plans are often too overwhelming for a department or program to manage, and difficult for people “outside” their area to understand. Historically, too much was expected in the initial stages and institutions were too busy gathering data to actually use it effectively. Many teams have not considered the research aspect of this work. Framing a hypothesis and then testing it is an exciting avenue and they respond accordingly (Karen Solomon, The Higher Learning Commission).

To say institutions have not gathered and acted on evidence of student learning to improve learning would not be true. For generations, educators have been assessing and improving student learning, but only now are those ways producing transparent evidence, permitting comparisons, and expanding to include the integrated learning and development of the student's whole experience. Invite a conversation with faculty about their courses, and you will be amazed at the rich array of assessing that is now occurring. For 20 or 30 minutes on end, they can describe with ease the integration of pedagogy with curricula, the outcomes, the use of new technology, action-based assignments and activities, and the methods of formative and summative assessing that lead to the grade. Faculty in these teams talk eloquently and consistently about teaching journals, classroom assessment techniques (CATS), scoring rubrics, authentic performance matrices, the range of tests, primary trait analyses, and other measures and methods meant both for assessing what students have learned and for engaging students in the assessing itself to further that learning. However, engage in a discussion or program or broad major, general education, or institutional learning and assessing to know what and how well it is being learned, and the conversation and action flag. Teams at the workshops are highly engaged in acting on learning results and other evidence of learning; however, most bring for the first time useful baseline results on student learning beyond the course level. Despite the solid efforts at curricular mapping of broad outcomes across courses and programs, the integration of learning across courses and other academic experiences continues to prove difficult to assess well. The issue for most institutions with course-level assessment results is not that they do not have them, but that they cannot point to them in retrievable form. Course-level assessment thrives; documentation of the learning from it still lags.

Four scenarios describe the majority of institutions attending the workshops:

  1. The data chasm. These institutions are poised, teetering on, centered on the edge, needing the push to act, to gather data and information on student learning, any data. It is the actual first leap to pursue results – any results – on what students are learning versus all the other evidence being gathered already. In ways, they are almost paralyzed, wanting either the perfect measure or wanting to stay in the comfortable realm of evidence related to defining outcomes, minutes of assessment meetings and events, curricula, learning environments, teaching strategies, course sequence, technology, and everything else related to the educational program versus the learning resulting from that program.
  2. The data deluge. These institutions are swimming and sometimes drowning in data. One institution put it this way. “It's as if we are cramming all this stuff in the closet. We just have to get more. We go out, get data, open the closet door, quick throw it in, and slam the door shut to make sure nothing tumbles out. But we have it. We either need a garage sale or a garbage truck. We have no idea how to use what we have or whether it is of any use.” Shoulders shrug, when individuals are asked, how much of the data comprise actual student learning versus retention, course completion, satisfaction, engagement, enrollment, grades, cost ratios, graduation rates and the like. “We have standardized test results” (and they rattle off names of instruments).
  3. The whoo-boy data dilemma. These institutions have made the leap to data on learning. Results have been gathered, often through cross-sectional testing at regular intervals, and the hope is that the faculty and the institution will know how effective their students are at critical thinking or computation, etc. Two common stories are told: “We bought a package of tests from that company. Lots of money. The data were useless, told us nothing about what we really care about, minimum threshold stuff at best” and “The evidence basically told us the students weren't learning anything like the level we intended; in fact, in math, we've learned exactly what the research indicates: learning deteriorates over time. Like we're going to publish that! Whoo-boy. Our efforts are DOA. You can choose if that means dead on arrival or data, outcomes, assessment. They're the same in our case.” Perceived as failure, these teams often resort back to the very beginning, needing to re-establish why assessment is anything but a mandate. Instead of failure, however, this cycle of “failed assessing” is a natural learning curve in the process experienced by most teams many times. The discovery that an assessment strategy and results produced do not work for whatever reason, is learning, is evidence of completing the cycle, evidence of having analyzed and drawn conclusions. The conclusions just happen to be that a new strategy is needed.
  4. The base is covered data. These institutions may be in their second round of learning, having already experienced the scenarios above. Emblems of breakthrough, these groups point to sets of baseline evidence on articulated program, general education, co-curricular, and/or institutional outcomes. Perhaps 10 to 15 percent of the teams talk further about having analyzed, interpreted, and made recommendations for change based on this evidence. Some have implemented the changes. Very few are yet far enough along in the cycle to have re-assessed to see if the changes have impacted the learning.

What marks the institutional teams able to persevere? Institutions achieving the breakthroughs talk consistently of doing these nine things:

  1. They focus on developing and piloting ways of assessing program and other higher level outcomes to see what produces the kind of evidence they seek.
  2. They employ standardized tests or other global measures as a complementary strategy to add broad summative results on base-level achievements or to speak generally to value-added learning.
  3. They embed program and other higher-level outcome assessment strategies into courses or integrating capstone experiences with authentic performance assessments that cover multiple learning outcomes simultaneously.
  4. They establish simple, streamlined methods for documenting evidence of student learning on a regular versus episodic basis.
  5. They implement sophisticated software that offers convenient documentation and retrieval of evidence on student learning, software ranging from course management systems to database systems to robust electronic portfolios.
  6. They institute ongoing processes that routinely engage all appropriate stakeholders (internal and external) in analyzing, interpreting, and making recommendations on data.
  7. They revise intact systems, such as program review, so that the evidence of and recommendations resulting from analysis of student learning data can be acted upon.
  8. They compare findings across programs, courses, years and sometimes benchmarking other institutions.
  9. They plan for the analysis of second-round data to determine the impact of changes made; i.e. if the changes made improved the learning.

In the long run, collecting data or reporting data is not what matters; using it is, but that means deciding what will be usable data in the first place is extremely important. For example, much of the data we collect and report, like grades, GPAs, and graduation rates is not particularly useful in helping us improve learning in specific ways, so we need to look more carefully at the utility of the information we decide to collect and remember always that assessment data have value to the degree that they can be analyzed and interpreted to lead to the improvement of student learning (Robert Mundhenk, Lead Mentor Assessment Workshops, former Senior Fellow, AAHE).

Observation six

Integration of assessment of student learning results into program review is making inroads into larger institutional evaluation, improvement, and resource allocation systems and offers shared territory for engaging faculty, administrators, and others throughout the institution.

It's not the pieces of the system rather the dynamic among them that makes the difference (Russell Ackoff).

If observations across the 264 institutions are representative of where other institutions are, then integrating recommendations based on assessment results into larger institutional systems remains a significant challenge. The majority of institutions have been able to integrate findings into the deliberations of curriculum committees, assessment committees, teaching and learning development groups, and new program development processes. However, student learning remains on the fringe of consciousness in the broad institutional systems of governance and decision-making, strategic planning, budgeting, and even academic planning. Thus, changes result, gains are made, but most are localized, occurring within programs, disciplines, courses, majors, and academic affairs generally. Although institutions have accomplished much innovative, creative, and ambitious work, Tagg (2003) explains, the impact will be limited until the institution commits to the larger, cultural transformation to a learning-focused college has occurred.

In the past year, institutions attending the workshops have found a potential avenue that not only begins to move learning from fringe to focus, but also builds institutional commitment, shared responsibility, and meaningful engagement. When institutions integrate assessment of student learning results into program review, they hitchhike on an already robust, established cycle of planning, evaluation, and improvement. This discovery has proven invaluable for institutions at the assessment workshops, as long as their program review processes are robust and focused on evaluation, development, and improvement of educational quality. The investment into effectively structured program review processes brings success, even to areas that have proven difficult to tackle, such as general education. “What we do,” states one institutional representative, “is that we make student learning data central to decisions about program quality and recommendations for program improvement. What we've found is that we can then move those recommendations into master planning and technology planning, and bingo, we're into strategic planning and budgeting. Now that we've agreed to consider general education a program that cuts across all divisions and that has distinct learning attached to it, we plan to apply the review process in support of it as well.”

A second benefit seen across institutions investing in healthy program review systems is the impact on engagement. Genuine, real engagement happens when the work is meaningful, and program review is naturally meaningful and engages diverse groups of people: faculty, academic administrators, student affairs personnel, administrative services personnel, employers, students, and other staff. One example may clarify this point. When asked, “When are you and are you not engaged?” faculty and administrations provide absolutely opposite responses. Faculty are engaged in students and their learning one-to-one and actively assessing and documenting individual learning by new means and with new technologies at in their courses. Institutional outcomes, strategic planning, institutional goals, assessment efforts, and institutional effectiveness are trumped by the passion for the learning, the student, the course, Administrators talk most frequently in terms of groups, not individuals, and about assessing and documenting what students are learning (emphasis on documenting), on the systems of analyzing and interpreting data, and on strategic and institutional goals. Where they intersect and collaborative conversation flows easily is around program review or review of major, as long as results or evidence of student learning has bearing on decisions and resource allocation. Programs are middle ground, common ground, shared ground. Further, program review is a system in which many have a stake and want a say; it is a system that spans from the front lines to the board and one that often is linked directly to resource allocation.

A third benefit to program review might very well be timing, place, and pace. Because program review processes often run on three- or four-year cycles, rotating programs in and out, the process has proven to be a “pacesetter” for gathering, analyzing, and using assessment of student learning results. For example, imagine you are a faculty member in the journalism program, a program reviewed every four years at your university. As a department, you have been assessing and documenting student learning at the course level, as well as the major or program. Until now, it was unclear what you did with your results other than discuss them with your colleagues and make changes in your program, submitting requests for additional resources that may or may not be granted. However, now that student learning results are front and center in program review, you have a natural forum for your discussions, a larger arena for discussing student learning, and a pathway for pursuing greater support for educational improvement. Further, the four-year cycle offers a built-in “feedback loop” for comparing data cycle to cycle to see how changes made impact student learning.

Observation seven

Institutions listen to, discuss, and address the real and enduring issues of assessing and improving student learning, sometimes by changing the core processes, structures, and values of the college, more often by establishing parallel or separate structures and processes.

Not everything that is faced can be changed, but nothing can be changed until it is faced (James A. Baldwin).

When asked what are the greatest challenges in implementing assessment, institutions repeatedly mentioned these, listed in order of most commonly stated:

  1. Time lags and fits and starts due to turnover of administrators and of assessment leaders.
  2. Heavy workloads and competing priorities for both faculty and administrators.
  3. Reduced budgets, lack of resources, inability to depend on resources.
  4. Integrating the work of teaching, learning, and assessment into the tenure system, union agreements, and other established processes and structures.
  5. Lack of reward systems for teaching and learning.
  6. Documentation of student learning data.
  7. Defining and assessing general education.
  8. Vague outcomes without explicit objectives or standards for performance.
  9. Few embedded processes for analyzing and using data on student learning.
  10. Confusion on balance and purposes of qualitative and quantitative approaches to assessing.
  11. Fragile links to institutional planning and resource allocation systems.
  12. Lack of consistent or agreed on language about teaching, learning, and assessment.
  13. Misfit assessment practices, including models adopted wholesale without considering appropriateness to the institution or its mission.
  14. Misfit data that do not provide the evidence needed.
  15. Lack of knowledge and awareness on the part of many in the institution, most typically administrators and staff outside of academic affairs.
  16. Minimal flow of information to the faculty on books, articles, web sites, and other resources on good practices and scholarship in assessment.

Although the list above may seem daunting, the heartening news is that awareness of assessment and issues with how to assess are falling off the bottom. In their place are bigger system and culture challenges that institutions are discussing and tackling. The ways in which institutions address the challenges are diverse and specific to their distinct contexts; however, one of the reasons for identifying the previous six observations above is simply because institutions consistently practicing them are consistently able to address if not eliminate these enduring issues.

One example will suffice to explain. Defining and assessing general education is the most commonly stated priority for attending the assessment workshops, followed closely by assessing co-curricular learning. General education also makes the top ten issues in every workshop, often deemed the area belonging to everyone and therefore the responsibility of no one. For both topics, general education and co-curricular learning, when pressed, the institutional teams say, “We just have never had the time to think it through, never intentionally established a leader to make sure it happened, got bogged down in fighting over the right ways to assess and even what it is, or never thought to get everyone together to define what it is for our institution.” Essentially, the six observations above are a mantra for time, talk, and intentional leadership that sets the context, builds collective agreement, and articulates the decisions that allow for effective assessment and improvement of learning. In reality, what the assessment workshops provide to institutions has less to do with information on good practices and more to do with three days of time for leaders and their teams to think through a challenge.

Observation eight

Institutions begin assessment of student learning in different ways, entering at different points in the cycle, taking different directions, and using diverse approaches and strategies. The process is not linear, rather reiterative and recursive with fits and starts. However, institutions eventually address and establish all points on the cycle and repeat the cycle.

I insist not on the events, but on their effect on the persons of the tale (Joseph Conrad).

Bring to mind your institution, along with its efforts to assess and improve student learning. Where did your institution begin with assessment of student learning? What did it do first?

In many ways, this final observation puts the previous ones in relationship to each other. The point is that the institutions create an ongoing cycle that includes a few key elements. However, one of the most striking aspects of this observation is the diversity in how institutions first began their assessment of student learning efforts and the diversity within institutions in how their programs, majors, and individuals first began. You might think that the order of what is done first, second, third, etc., determines the success; however, the observations do not bear that out. Instead, the order mattered less than that a few key elements became pillars of action in creating a sustainable cycle or spiral. These are not done once. Rather, they persist as concentric cycles operating at multiple levels (course, program, major, degree, general education, institution):

Again, the importance of this observation rests not in the order in which an institution began, but rather that all the fundamental elements were put in place and then set into an ongoing system. Thus, it becomes clear that it is more important to start than to deliberate over the where to begin or to ensure all areas of the institution start in the same place. These institutions start anywhere it makes sense, seems reasonable, and where there is readiness to start. Further, the institutions do not demand that every program, department, major, etc., moves in lock step fashion, approaching assessment of student learning in the same order or in the same fashion. Instead, the development becomes organic, representative of what best fits – what is meaningful, useful, workable, and reasonable – for different groups from the course to the broad institutional level.

The organic nature proves difficult to describe. These institutions spend more time defining principles and protocols for acting on student learning than dictating mechanistic processes for assessing student learning. These teams focus their process development on analysis, use, and action on assessment results. One of the assessment mentors approaches it just as organically. “I sit with each team, and I ask them: What would work for you to work on right now? Where do you need to start so that it works? What makes sense for you do first? At that point, I coach them to begin there … and return to these questions when they're ready to ask them again.”

Cultivating new ground: emerging trends in assessment of student learning

The present is the future in its most creative state (T. Irene Sanders).

Over the past three years, the workshops have not only revealed what catalyzes institutional capacity to assess and improve student learning, but in addition, they provide a rich environment for seeing the emerging trends. A number of trends in the past year mark both the progress of the past and the future in the present:

If the trends above indicate the leading edge of institutional efforts, where generally are most institutions in assessing and improving student learning? As Lopez (1998) documented, institutions display more diversity and range of progress within the institution than among institutions. The same holds true for institutions attending the workshops. Workshop teams speak routinely of areas in their institution – fewer and fewer, but still holding on – that are still defiant, resolute in never stating a learning outcome. At the same time, areas in the same institution are into their third round of data collection, are embarking on benchmarking, and are well into the trends identified above. If this is the range within an institution, where rests the bulk of the institution generally?

Questioning progress: the success story and the ongoing struggle

Just because everything is different doesn't mean anything has changed (Irene Peters).

In 2004, the Commission adopted a continuum of five fundamental questions designed to move the focus to student learning results from an insular focus on assessment as the ends versus the means. Intended in practice to evolve in line with the development of the institutions, the questions in their current form have proven a fairly good marker of where institutions are:

  1. How are your stated student learning outcomes appropriate to your mission, programs, degrees, and students? (What must your students learn and how do you know it is the right learning?)
  2. What evidence do you have that students achieve your stated learning outcomes? (What evidence do you have that they learn what you intended them to learn?)
  3. In what ways do you analyze and use evidence of student learning? (What do you do with evidence gathered on student learning?)
  4. How do you ensure collective institutional commitment to and shared responsibility for student learning and for assessment of student learning? (How do you ensure student learning is central to the institution and that your efforts to assess and improve student learning increasingly build your institution's capacity to improve student learning, educational quality, and organizational effectiveness?)
  5. How do you evaluate and improve the effectiveness of your efforts to assess and improve student learning? (How do you ensure assessment is meaningful, useful, workable, and reasonable?)

The success story

Overall, institutions are not asking what outcomes are or why assess or what to assess. Institutions have defined outcomes at multiple levels, have gathered one or two rounds of evidence of what students are learning and are now revising outcomes to more appropriately articulate the learning intended. In addition, most have had enough experience now that they are actively revising and improving the effectiveness of their assessment processes, structures, methods, and approaches. Two quotes aptly describe the successes of institutions and accreditors:

Institutional personnel – faculty, staff, and administrators – have become more aware of and committed to assessing student learning outcomes. The Commission's unwavering efforts, over the past 18 years or more to help institutions develop cultures of assessment have begun to pay off, and both students and faculty/staff are able to document the learning outcomes their efforts are producing. In large measure, institutions have passed the “what for” and “how to” stages regarding assessment, and the conversations and efforts now are more directed toward a focus on student learning and the application of assessment techniques and follow-up (John Taylor, The Higher Learning Commission).When the Commission adopted its “Statement on Assessment” in 1989, the focus was on student learning. While our strategies and processes have evolved, that focus has remained and I believe has been the strength of our initiative. Another strength is the sharing of institutional learning as a major component of our program of assistance to our members. From the earliest years when we struggled to find a handful of speakers to share good practice, our Annual Meeting has in recent years been a showcase for scores of outstanding efforts in this area. Our institutions are to be commended for their extraordinary investments of time, energy, creativity, and persistence toward the goal of improving student learning. There are many wonderful success stories to share and celebrate (Susan Van Kollenburg, The Higher Learning Commission).

The ongoing struggle

Two of the five questions above pose more issues for institutional teams: question 3, processes for analyzing and using data, and question 4, establishing institutional commitment and shared responsibility. In 2006, these topics superseded all others for faculty and administrators attending the workshops (often in reference to general education, co-curricular learning, and engagement or “buy-in”). Although processes for analyzing and using assessment results are developing, particularly in institutions integrating assessment results into program review, the processes are not yet routine or systematic. “We add them on a couple times a year,” say members of most teams. Posing even more difficulty is calling a full institution into shared responsibility for student learning results and developing broader institutional commitment to learning and capacity for assessing and improving that learning.

In many ways, institutions are like archipelagos, beautiful discipline- or program-based clusters of islands simultaneously connected and autonomous. The islands are different sizes, shapes, and have varied population densities. Distinct cultures, climates, and trade winds distinguish each island, varied natural and acquired resources, unique notions of work. There is a delicate interplay and balance between central governance across the whole and independent leadership of each island. Waterways and airways connect, but also carefully protect identity. A change of even one leader of an island can dramatically tip the balance and interactions among islands. Inhabitants can live their whole lives on a single island, connecting occasionally with those in central government, remaining focused on serving those people that come to the island. Consider how students experience institutions as archipelagos, moving from island of learning to next island of learning until all islands have been covered, all learning environments experienced. Envision how assessment of that learning is rooted to the individual islands, their cultures, and their norms, versus representative of the whole. The sense of the island overshadows a sense of the whole. With the image in mind, it is not difficult to grasp the struggle for institutional shared responsibility, collective commitment, and enhanced capacity.

Steve Spangehl of the Commission captures both the success of assessment implementation and the larger struggle with the bigger system and culture of higher education:

In the many institutions where assessment was tackled seriously, there is a palpable sense of collaborative effort, of shared interest in the common goal of defining what makes a student educated and jointly shouldering the responsibility for making sure it happens. Anecdotally, we hear that in such colleges faculty morale is higher, and that some students see the logic of the curriculum and the connections between one course and the others. New curricula, pedagogy, and technology mark the learning environments, and there is evidence that alumni see more clearly the links between education and the more fulfilling life it prepares one to lead. Some faculty, even employers, also better perceive these connections. But, wonderful as some of these changes are, assessment hasn't yet stimulated major changes in higher education. On fundamental matters, there has been movement, but hardly a tidal wave (Steve Spangehl, The Higher Learning Commission).

Both the success and struggle observations are true. Wonderful stories and case studies abound with how institutions are maturing their systems to assess and improve student learning, making changes, documenting learning, even documenting improved learning. If the evaluation of success is measured in activity, effort, investment; i.e. in implementing processes to assess and improve student learning, to reform teaching strategies, to rejuvenate curricula or to embed new technologies; then indeed, the implementation of assessment of student learning is an ongoing success. If however, the goal is not simply to implement a process, but rather to transform an instruction-driven system to a learning-results culture; i.e. to shift from an instruction to a learning paradigm (Tagg, 2003), then we are indeed in the middle, perhaps even close to the beginning, of this story.

Becoming responsive to new questions and responsible for broader conversations

New frameworks are like climbing a mountain – the larger view encompasses, rather than rejects the earlier more restricted view (Albert Einstein).

Whether near the middle or toward the beginning of the assessment of student learning story, accreditors and institutions will soon be writing a new chapter. For several decades now, the calls for quality and accountability in higher education have persisted. What is different in the most recent round is the clear demand for evidence of student learning, calls to know what and how well students are learning, calls for access to more learning, calls to know whether the learning competes favorably in a global market, calls to know whether the learning results are worth the cost, and finally calls for evidence of learning to be clear, common, consistent, and comparable. To return to the archipelago simile, higher education, to many, seems an island unto itself, in a world of its own, completely disconnected from the mainland.

Listed above are five fundamental questions created in 2004, used by the HLC to engage institutions in advancing their efforts to assess and improve student learning. A sixth question is now necessary, one focused on how an institution's external constituents know what and how well students are learning: “In what ways will we inform the public and other stakeholders about what your students are learning?”

Take a moment to read the next series of quotes from others at the HLC. It would be easy to summarize them into one voice. However, retaining the separate perspectives demonstrates the unity of purpose, the knowledge of what's ahead, and the call not only to be responsive, but also responsible to the broader public:

Walking the walk of a mission statement comes down to the ability to demonstrate results in student learning. Communicating that to various audiences may become the most important skill institutions develop (Ingrid Walker, The Higher Learning Commission).

The public's clamor for greater evidence of learning and accountability in higher education is resoundingly loud, and it is appropriate for our institutions to respond in ways that show the investments being made in education are meaningful and that they convert into dividends beneficial to society and the common good (John Taylor, The Higher Learning Commission).

Institutions continue to realize that they have some really interesting information to work from, but they many cannot figure out how to communicate their findings and recommendations. The conversations (and expectations) still focus on the selection of measures with minimal effort in developing communication streams, understanding different audiences, and building pathways to funnel recommendations for change. Moving into the future, we need to assist people to use the information that they have generated and build strategies to communicate to a range of audiences. Sharing the findings across populations is key to utilizing the information gathered and improving learning. We are on the verge of focusing on what institutions want (and need) to share with their various constituents. They need develop clear research plans, analysis strategies, and determine how the results will guide internal processes. Last, they need a plan to share the information with internal and external stakeholders (Karen Solomon, The Higher Learning Commission).

A particularly useful form of analysis for assessment and institutional effectiveness is comparison with benchmark institutions. This analysis avoids the common pitfall of ignoring vital features of an institution that distinguish it from most others. Benchmarking requires a principled rationale for identifying peers, as well as a basis for confidence in the consistent use of definitions and in the consistent character of reporting (Robert Appleson, The Higher Learning Commission).

Assessment has not yet provided either students or the public with a reliable and consistent means of documenting the outcomes of an education, even if it has made articulating educational purposes a more central concern for educators. Calls for accountability continue unabated, and assessment hasn't yet become the effective buffer it promised to be. Nor has assessment restored public trust in higher education. It has not yet helped either the public or educators themselves understand how to determine the value – the consequence of both cost and quality – of higher education. Value, in purely commercial terms, is a personal calculation, based on one's perception of factors like the cost, quality, and convenience associated with a purchase. An accurate valuation of the content of an education remains elusive – what will it do for me? how will it change me? why should I choose one provider over another? As a result, convenience and cost have pushed quality off center stage. But not for long (Steve Spangehl, The Higher Learning Commission).

Accreditors will need to contribute to the movement for comparability of performance indicators. We must engage the conversations on (1) communicating the quality of learning in ways valued by the external constituencies, (2) a standardized performance template that might include a few common measures, but allow institutional additions to make sure the uniqueness of the institution is captured, and (3) the integration into the traditional approach to accreditation of more use of benchmarking, potentially directly into the self-study process. Increasingly our systems of self-assessment and peer review will need the capacity to handle effectively the increasing complex issues of evaluating results, included student learning, rather than resources and processes and in a manner that contributes to and reflects the distinctions of institutional mission (Steve Crow, Executive Director, The Higher Learning Commission).

In their article, “Caring or uncaring assessment,” Braskamp and Schomberg (2006) propose that assessment of student learning can serve dual purposes – improvement of student learning and accountability for student learning – without either trumping the other. They begin to create an important bridge that will need crossing. Others propose that these two purposes are contradictory imperatives. Still others fear an imposed process will result in “teaching for the test,” a trivialization to minimum thresholds or a dangerous set of assumptions about value-added, all of which might lead to the abandonment of assessment as a powerful means for institutions to understand and improve student learning. Still others testify that such a system would compromise the diversity of missions and institutions, long seen as a core strength of American higher education.

All of these may be true. However, just like a focus on assessment as the ends may keep institutions from asking and answering profound questions about student learning, a focus on debating the mandate and the practical aspects and implications of how to comply may keep us from being responsive to the new, profound questions and from taking responsibility for the broader conversations. What will be our commitments to students and to the public? What responsibility do we have to inform the public about what students are learning and how well? What questions will we commit to answering and what matters most for us to tell students and the public about what is being learned and how well? What information and evidence will inform us as to whether we are achieving those commitments, answering those questions, and serving the common good? In what ways will we learn, be responsive to, and become responsible for their higher learning needs in the future? What is our vision for higher learning in the future? What will we create?

References

Block, P. (2002), The Answer to How Is Yes, Berrett-Koehler, San Francisco, CA, .

[Manual request] [Infotrieve]

Braskamp, L., Schomberg, S. (2006), "Caring or uncaring assessment", Inside Higher Education, available at: www.insidehighered.com/views/2006/07/26/braskamp (accessed November 11, 2006), .

[Manual request] [Infotrieve]

EDS (Producer) (n.d.a), We Build Airplanes in the Air, available at: www.youtube.com/watch?v=cOjMNpjjLPA (accessed November 7, 2006), .

[Manual request] [Infotrieve]

EDS (Producer) (n.d.b), Cat Herders, available at: www.youtube.com/watch?v=zHgEQ71rGwo (accessed November 7, 2006), .

[Manual request] [Infotrieve]

Ewell, P.T. (1991), “Assessing institutional effectiveness: redirecting the self-study process”, paper presented at the Annual Meeting of The Higher Learning Commission of NCA, Chicago, IL, April, .

[Manual request] [Infotrieve]

Gladwell, M. (2002), The Tipping Point: How Little Things Can Make a Big Difference, Little, Brown and Company, New York, NY, .

[Manual request] [Infotrieve]

(The) Higher Learning Commission of NCA (HLC) (2003), Handbook of Accreditation, 3rd ed., The Higher Learning Commission, a Commission of the North Central Association of Colleges and Schools (NCA), Chicago, IL, .

[Manual request] [Infotrieve]

Lopez, C. (1998), “The Commission's assessment initiative: a progress report”, paper presented at the Annual Meeting of The Higher Learning Commission of NCA. Chicago, IL, March, .

[Manual request] [Infotrieve]

Maki, P.L. (2004), Assessing for Learning: Building a Sustainable Commitment across the Institution, Stylus, Sterling, VA, .

[Manual request] [Infotrieve]

Manning, M. (2004), Creating Conversations of Consequence: An Introduction, White Paper written at the request of The Higher Learning Commission of NCA, The Higher Learning Commission of NCA, Chicago, IL, .

[Manual request] [Infotrieve]

Senge, P.M. (1995), The Fifth Discipline: The Art and Practice of the Learning Organization, Doubleday, New York, NY, .

[Manual request] [Infotrieve]

Tagg, J. (2003), The Learning Paradigm College, Anker, Bolton, MA, .

[Manual request] [Infotrieve]

Corresponding author

Lynn Priddy can be contacted at: lpriddy@hlcommission.org