Community leaders and key members can use the information gleaned from various evaluation approaches to cultivate and successfully sustain their online communities of practice in a variety of ways. Good evaluation can save community leaders time, help gain additional resources for their work, help allocate time and other resources most effectively, and, most important, enable the community to better engage members and provide resources and activities that best meet their needs. Community leaders and members are both encouraged to refer to the U.S. Department of Education draft report, Connect and Inspire (http://edcocp.org/report/), and associated online briefs (http://edcocp.org/briefs/), including Online Communities for Educators: Guidelines for Planning and Implementation and Technology for Online Communities of Practice, for guidance on important ideas and aspects associated with online communities as they begin to plan and develop their evaluation strategies. Specifically, targeted evaluation techniques allow leaders to accomplish the following:
- Gauge the health of their communities
- Determine what parts of their communities are functioning well and which are underperforming
- Learn how to meet the needs of different types of participants, and in the process, how to attract and retain more of them
- Track and report what is happening to their communities over time, including content, activities, and technical needs
- Demonstrate the impact of the community on attitudes, on practice, and on student learning
- Compare their communities to others reaching the same or similar audiences
- Make a better objective case of value for funders and sponsors
Once an online community of practice becomes active, leaders need to develop an evaluation plan to understand and document success in their community. In many contexts, evaluation approaches, which measure communities’ activities and performance, are also referred to as metric analysis—analyzing measures of success. The following are general questions for leaders to consider when selecting evaluation approaches for assessing the activities, content, and interactions in their online community. Evaluation plans should help community leaders and key members determine to what extent the following effects are evident:
- The community is serving its intended purpose and audience.
- Knowledge is shared around a domain and related practice emerging.
- Members’ interactions have continuity and depth. (Are members engaged in productive, ongoing, interactions?)
- Collaborative activities are emerging.
- Documents, tools, resources, or other artifacts are created and utilized. (How are these useful to the members?)
- The community provides value for its sponsors.
- Participants’ involvement in the community affects their professional practices and student learning. (This question goes beyond improving the community functioning to its external impact, so is generally not addressed until there is evidence that participants are actively engaged in the community’s resources, activities, and interactions.)
In many cases, a combination of evaluation methods is warranted and required to get at answers to questions like these. For example, focus groups to uncover concerns or ideas to test, online surveys to more broadly test member attitudes about those concerns or ideas, and community content analysis to see whether community members’ self-reported attitudes are reflected in their actual on-site behavior. The rest of this brief is dedicated to more fully describing and discussing these and other evaluation methodologies.