olac_127The Ohio Leadership Advisory Council (OLAC) hired Indikus to perform a 3-year program evaluation of an online professional learning system that serves more than 19,000 K-12 educators and thousands of preservice teachers.  OLAC’s leadership and sponsors needed a neutral evaluation of the learning system’s impact on school improvement and identify opportunities for improving its value to educators.

The evaluation is being conducted with a four-level evaluation design that links educators’ self-reports of learning needs, an assessment of their learning and skill development, reports of their use of the new skills, and an assessment of demonstrable effects on teaching and learning. The evaluation team designed the study to be carried out in three phases: program elaboration, module outcomes, and case studies (see figure at right for detail).

The comprehensive evaluation provides key stakeholders with an ongoing assessment of the learning system’s strengths and limitations and offers recommendations for improving modules and their delivery.  OLAC has used the feedback to refine its resources and develop new modules and messaging, leading to a much more efficacious program for its intended audience.  The evaluation team is currently preparing an impact study, using targeted outcome indicators.  Through this work, Indikus developed a new model for evaluating short-term, asynchronous, self-paced professional learning systems.

Client Comment

We highly recommend Indikus to any organization seeking to identify areas of strengths and needed growth. Indikus is responsive, communicates frequently, and delivers an outstanding report. You may contact us any time if you need more information about our experience with Indikus.

 

– Jim Gay & Don Washburn, OLAC Co-Directors

OLAC Evaluation Phases 

Program Elaboration: Combining interviews, pre-post tests, and surveys, Indikus conducted a process and retroactive outcome analysis. This led to a preliminary outcome analysis, pilot instrumentation, and user suggestions for improving the program.

Module Impacts: By linking together pre-post tests and a series of surveys, Indikus examined outcomes for a subset of five modules.  The analysis (using Repeated Measures ANOVA) followed learners from program entry to completion to their use of new behaviors to longer-term outcomes.

Case Studies: Indikus combined interviews, observations, surveys, and lead indicators to develop rich case studies of two schools and three districts that are using OLAC’s learning system in innovative and practical ways.

Five Level Evaluation for Professional Development

Readiness
Motivation
Experience

Reaction
Experience
Engagement
Value

Learning
Tested gains
Plans to Use

Behavior
Use at Work
Support

Impact
Improved system

Adapted from Kirkpatrick, D. (1996). Great ideas revisited: Techniques for evaluating training programs. Revisiting Kirkpatrick’s four-level model. Training and Development Journal, 50(1), 54-59