|
Program
Evaluation Basics
For those
of you who have had training on developing BDI
Logic Models, we believe you will find this edition of ReCAPP a natural
companion to that training. ReCAPP's July-September 2004 edition focuses
on the Basics of Program Evaluation with a particular emphasis on how
BDI Logic Models can be used to guide practitioners in developing program
evaluation plans.
ETR has
conducted various trainings on the Program Evaluation Basics throughout
the United States. Most recently (February 2004), ETR in partnership with
the National Organization on Adolescent
Pregnancy, Parenting and Prevention (NOAPPP) designed and delivered
a two-day training titled "Designing and Evaluating Program with Intent"
in Columbia, SC. We should also mention that some of the content from
this training was adapted from ETR's years of training and technical assistance
work with the California Office of Family Planning grantees. Much of what
is presented in this edition of ReCAPP has been adapted from these trainings.
We would
like to take this opportunity to thank the California Office of Family
Planning, NOAPPP, the CDC Division of Reproductive Health, and the Annie
E. Casey Foundation for their support in developing many of the materials
you will find in this edition of ReCAPP.
| What
You Will Find in this Edition of ReCAPP |
Specifically, this edition of ReCAPP includes:
- Slide Presentation
(in Flash). This presentation was adapted from a face-to-face training
on program evaluation. The slides provide information on:
- impact, outcome and process evaluation
- writing objectives
- type of evaluation designs
- qualitative and quantitative measurement methods
- constructing good evaluation tools, and
- evaluation reporting.
- "Creating Better
Surveys" Worksheet
 (pdf
file). This worksheet provides an example of a survey with several problems
in it. After working to find these problems and proposing improvements,
review the accompanying answer key to compare your answers.
- Sample Focus Group Protocols. Three focus group protocols
are provided in this section. Two were designed for teens and the other
for parents. One (Wait for Sex) was designed as an evaluation
tool and the other two as assessment tools. Specifically, the three
protocols are:
- Parent-Child Connectedness Focus
Group Protocol for Teens
- Parent-Child Connectedness Focus
Group Protocol for Parents, and
- Wait for Sex Focus Group
Protocol for Teens
- Focus Group Facilitator Guidelines.
This list of guidelines was developed as a consequence of facilitating
14 focus groups on parent-child connectedness in support of ETR's Parent-Child
Connectedness: Bridging Research and Intervention Design (PCC BRIDGE)
project funded by the Annie E. Casey Foundation.
- A Mock Board Presentation.
The Mock Board Presentation activity is designed to give an authentic
practice opportunity for presenting/reporting on program evaluation
findings.
- Evaluation Planning Matrices
(Word documents). These matrices provide the program evaluator
with a way of recording impact, outcome and process objectives and their
corresponding evaluation designs, measurement methods and questions.
- Evaluation Resources
Other useful
program evaluation resources on ReCAPP and other sites:
- Evaluation 101
Forum Summary
In September 2003, ReCAPP moderated a one-day on-line discussion forum
moderated by:
- Karin Coyle, PhD: Director of Research at ETR Associates
- Claire Brindis, DrPH: University of California at San Francisco
(UCSF)
- Valerie Sedivy PhD: Lead Evaluator at the National Organization
on Adolescent Pregnancy, Parenting & Prevention (NOAPPP)
- Lori Rolleri, MSW, MPH: ReCAPP Project Director
- Mary Martha Wilson, MA: Co-Executive Director at NOAPPP
- ReCAPP Glossary
ReCAPP's on-line glossary of research and evaluation terms
- The
National Center for Health Statistics' research-based glossary
top
Program practitioners and evaluators often elect to use a written survey
to evaluate a program. The survey may be used to obtain pre- and post-
data about a group of people receiving an intervention. The hope is that
program participants will perform better on the post-test than on the
pre-test thereby suggesting that the intervention had a positive effect
on knowledge, attitude and/or behaviors of the participants. A survey
may also be used to obtain program satisfaction data. (e.g. How well did
today's workshop meet your needs?)
There are several resources that practitioners can utilize to develop
questions for a survey. First, and in the most ideal of situations, they
can review already tested scales and questions available in books like
the Handbook
of Sexuality-Related Measures. Second, they can adapt these
questions or develop their own questions. It is always a good idea to
pilot test any survey you develop to make sure questions are understood
by participants.
Evaluation slides #29-#40
provide some additional guidance on developing questions for a survey.
ReCAPP has provided you with a short mock survey (Creating
Better Surveys Worksheet)
for your critique. After completing this exercise you may want to compare
your responses to the answer key at the end.
top
| Focus
Group Protocols and Guidelines |
In this section, you will find three examples of focus group protocols.
- Wait for Sex Focus Group Protocol
This protocol was developed as part of the evaluation strategy for a
program developed by ETR titled Wait for Sex. It was administered
at the conclusion of the 20-week program to about 12 youths (aged 11-13
years) who had participated in the program.
- Two other focus group protocols were designed for the purpose of program
assessment (information that helps inform the design of a program) rather
than program evaluation (information that tells us about the effectiveness
of a program). You will also find other materials that were needed to
support this focus group study. ETR conducted 14 focus groups in various
regions of the US with low-income parents and teens (aged 11-13 or 13-15)
to better understand how the construct of "parent-child connectedness"
is understood and what parents and teens do to feel close to each other.
This work is in support of ETR's Parent-Child Connectedness: Bridging
Research and Intervention Design (PCC BRIDGE) project funded by the
Annie E. Casey Foundation. For more information on this project, visit
ReCAPP's edition on Parent-Child
Connectedness.
The examples of focus group protocols from this project are:
Also, included are the following samples of focus group forms, which are
formatted as Microsoft Word documents for easy adaptation:
And finally, during the course of facilitating focus groups for the PCC
BRIDGE project, ETR recorded a list of tips for facilitators — Guidelines
for Focus Group Facilitation.
top
Measures
- Davis, C.M., Yarber, W.L., Bauserman, R., Schreer, G., & Davis,
S.L. (Eds). (1998). Handbook of sexuality-related measures. Thousand
Oaks: SAGE Publications, Inc.
Description available at: www.sagepub.com
Telephone: (805) 499-9774
Email:order@sagepub.com
- The Kinsey Institute for Research in Sex, Gender and Reproduction.
Measurement and Assessment Bibliography.
www.indiana.edu/~kinsey/services/bib-measures.html
- Dahlberg, L.L., Toal, S.B., Behrens, C.B. (Eds). (1998). Measuring
Violence-Related Attitudes, Beliefs, and Behaviors Among Youths: A Compendium
of Assessment Tools. Atlanta: Division of Violence Prevention, National
Center for Injury Prevention and Control, Centers for Disease Control
and Prevention.
Program Evaluation
Focus Groups
Surveys
top
Home
| Index | Topic
in Brief | Evidence-Based Programs
Skills for Educators | Skills
for Youth | Current Research
Library | Statistics
| Theories & Approaches | Links
Professional Credits | Learning Activity
| Forums |
Archives
|
|
