On September 10, 2003, Lori Rolleri from ETR was joined by four other moderators, all of whom have years of experience in evaluating a variety of adolescent reproductive health programs on the local, state and national levels, for ReCAPP's forum on Program Evaluation. Ms. Rolleri's co-moderators were Karin Coyle, PhD, Director of Research at ETR Associates; Claire Brindis, DrPH, University of California at San Francisco (UCSF); Valerie Sedivy PhD, Lead Evaluator at the National Organization on Adolescent Pregnancy, Parenting & Prevention (NOAPPP); and Mary Martha Wilson, MA, Co-Executive Director at NOAPPP.

One hundred and eighty-five participants from across the United States, Canada, Australia, Japan, and Nepal joined the forum and shared information, resources and common concerns and questions regarding program evaluation.

The forum discussion covered the basics of setting up a program evaluation for the first time as well as a number of issues associated with program evaluation. The forum's moderators fielded questions and offered advice on the following topics:

• Setting up a Program Evaluation
  
• Getting Started, and



• Putting Research to Practice



• Terms Commonly Used in Program Evaluation
  
• Impact and Outcome Evaluation



• Formative and Process Evaluation, and



• Indicators
• Types of Programs that can be Evaluated
  
• Educational and After-School Programs



• Abstinence-only Programs



• Rural Community Programs and/or those Requiring Cultural Sensitivity, and



• Theatre Programs
• Types of Evaluation
  
• Pre/Post-Test Surveys



• Focus Groups



• Institutional Review Boards (IRBs)



• Logic Models, and
• Resources

Co-moderator Mary Martha Wilson began the forum by noting that "program evaluation is a critical activity, made easier when we understand its importance and key elements in program planning, design, and implementation." She later asked her co-moderators to set forth the initial steps that must be taken in order to set up a good and solid evaluation plan. "Let's say an administrator has a program going, but no formal or solid evaluation plan in place," she said. "Where do you start in constructing an evaluation plan that is relevant and likely to produce solid data?"

"There is no easy answer," responded co-moderator Karin Coyle. She pointed out that there are many factors that impact how a program planner or administrator might proceed (e.g., funder's evaluation requirements, internal questions of interest, age and stage of program, available resources, etc.)

A crucial first step is identifying the purpose of the evaluation, which affects all subsequent steps. What questions must be answered or are of interest? There are likely to be multiple questions and the methods and steps vary for each.

Dr. Coyle also recommended meeting with a group of interested stakeholders and program staff regarding evaluation questions. If you have an evaluator on staff or use an outside evaluator, that person could assist in framing this type of meeting.

If a program is brand new, stakeholders may simply want to know how it is being implemented and what participants think about it. It may be too new to begin an impact or outcome evaluation because it's not yet stable enough. If so, the evaluation might include a process component (collecting data on who is reached, observing program implementation, collecting data on the completeness of implementation and perhaps even fidelity). Participant satisfaction data could be collected using a post-only survey.

If, on the other hand, you have a program that is more stable, the key question of interest might be "How does the program affect participants' knowledge, attitudes, and behavior?" This type of question requires extensive planning about which evaluation design to use to collect defensible data, creating or locating a good measurement tool, etc.

Once the questions have been identified, they can be refined and then program staff and their evaluators can begin to identify the appropriate methods to match the questions.

This is one of the first steps she would take if she were meeting with a group who had asked her to conduct an evaluation, Dr. Coyle concluded. Prior to the meeting and during it, she would spend time learning about the program and its goals as well as its implementation.

Co-moderator Valerie Sedivy added that she has found it helpful to sit down with program coordinators (preferably in a setting removed from the program itself) and ask them direct questions such as:

• What do you want to be able to say as a result of this evaluation?



• Who are the main groups you will report to?



• What do you see as my role? How do you anticipate working together?

She would also allow plenty of time to talk about what the evaluation could involve, including possible pitfalls, and what she will need from them.

Dr. Sedivy stressed the importance of making sure they understand that she can't create the result they are looking for, but that she can work with them to make the conditions optimal for achieving a positive result. "The bottom line," she said, "is that establishing a positive line of communication before starting any sort of evaluation activity is crucial. It's the only way you can begin to establish that you are approachable, flexible, and trustworthy."

Co-moderator Lori Rolleri asked what role the program coordinator can play in developing the evaluation plan. She has met with some program coordinators who feel frustrated by evaluation efforts because of a disconnect they feel with the individuals conducting evaluation activities. She asked forum participants how they have dealt with this disconnect and how they make sure that they get evaluation buy-in from the individuals responsible for implementing the program.

Dr. Coyle finds it helpful to include program staff and coordinators from the outset in planning the evaluation to see if there are ways to include questions of interest to program facilitators. Getting them involved in forming the evaluation enhances their buy-in.

Even if they can't be involved in the initial planning, it's really helpful to have individuals who will use the forms to review them before they are final and pilot test them. Often times, you can make simple changes that clarify the form and make it much more user-friendly for individuals who must complete it.

When you can't involve everyone, Dr. Coyle stressed the importance of clarifying the need for the information and how it will be used (e.g., invite them to lunch for a brief orientation) and she suggested sharing data with them so they can see how it's being used. She finds that program facilitators are very interested in reviewing the data and being part of a discussion about what it means and what the next steps should be.

Dr. Valerie Sedivy added that it's important to consider the purpose of the evaluation. It is also vital to determine the budget and spend time considering how this relates to the goals of the evaluation. For example, if you want to track behavior changes over time with a large group of participants (who may be difficult to reach), you will need a lot more money than if you are measuring a short-term goal with an immediate post-test.

Dr. Sedivy noted that whether or not it is necessary to bring in outside expertise should be determined in the initial discussion. One area that is useful for outsiders to evaluate is whether or not there is a logical connection between the program's activities and its goals/objectives, and whether this connection has any research to support it.

Program fidelity is another issue that is vital to consider at the outset. It's crucial that programs know whether they are actually providing the program that they think they are providing. Ideally, the program will be operating at full capacity before it is evaluated. For those participants looking for concrete ways to measure program fidelity, Dr. Sedivy recommended a guidebook she developed called "Is Your Program Ready to Evaluate its Effectiveness?" available from Sociometrics (www.socio.com/).

In response to a participant's question about putting research into practice, co-moderator Lori Rolleri suggested the following:

1. Use a Proven Program.
   If appropriate to your population, use a program that has been rigorously evaluated and found to affect behavior in a positive way. "Rigorous evaluation" usually means that the evaluation used an experimental or quasi-experimental design, had a large enough sample size, the sample was followed over time, and multivariate analysis was used.
2. Use a Logic Model to Develop a New Program.
   If one of these programs is not appropriate for your population, use a logic model to develop or adapt a program. Logic models are program planning tools that guide program developers in showing the causal pathways between a health goal (e.g., decrease in STDs among students at High School X) and the interventions selected to achieve a health goal (e.g., school-based curriculum, condom social marketing campaign, school-linked STD clinic, etc.) Logic models encourage program developers to be very purposeful about the interventions they select rather than using "intuition" to select an intervention.
3. Be Clear about which Risk and Protective Factors your Program is Going to Target.
   Adolescent pregnancy is a complicated problem which is affected by many different factors. In Emerging Answers, Doug Kirby reviews hundreds of research articles and documents over 100 risk and protective factors related to adolescent reproductive health outcomes. Intervention activities should be clearly tied to the risk and protective factors your program wishes to affect. They should be selected on the likelihood that they will affect a risk or protective factor — not because they "sound good or fun."
4. Review Health Education/Behavior Theory.
   Health Education and Behavior Theory can be very useful in designing intervention activities. ReCAPP reviews several theories in its Theories and Approaches section.
5. Evaluate your Program and Make Improvements.
   Co-moderator Claire Brindis added that one of the most important aspects of Emerging Answers was the emphasis on what the key ingredients are across so many of the programs that have been shown to be successful, as well as the literature on all the antecedent factors that appear to make such a difference in predicting poor outcomes.
   
   In using the Emerging Answers framework, it is helpful to consider how many of your new programs fit those criteria and to use that as a template for considering what strategies you are incorporating in your program.
   
   With respect to the latest in the "research into action" findings, there is much more than can be done. When providers note that a promising program has multiple sessions, many decide to utilize certain units, basically diluting what was originally found to be successful.
   
   Dr. Brindis noted that another common strategy is to develop a hybrid of multiple evidence-based programs, choosing certain units from some, while leaving others behind. This strategy is not surprising given that we often do not have the time or skills or support to fully transport one program to another setting. Also, many of the evidence-based programs may not have been tested with the group that you are serving, such as first generation immigrants.
   
   Modifying what has been shown to work while maintaining its integrity represents a challenge to the field. Dr. Brindis encouraged program developers to think carefully about what they are transporting and/or combining into a new hybrid and to maintain the intent and content of what was in the original program as much as possible.

Several participants asked for clarification on some of the key terms that are commonly used when discussing program evaluation.

Co-moderator Lori Rolleri asked her co-moderators to distinguish between the terms "impact" and "outcome" evaluation, which she said are often used synonymously. Agreeing that the two terms are sometimes used interchangeably, co-moderator Claire Brindis suggested that one way of looking at the distinction between the two is to consider the timing involved. She pointed out, though, that there are a number of different dimensions to consider as well, including research design.

Co-moderator Karin Coyle added that the fact the two terms are often used interchangeably and that there are differences in how the terms are interpreted (some individuals define "outcome" as shorter term and "impact" as longer term, and others do the reverse) makes it especially important to be very clear how you are defining the terms when you use them so that everyone is clear on your time frame.

Dr. Brindis explained that an outcome evaluation pertains to more immediate results (e.g., short term outcomes could occur as early as 30 days, longer term outcomes within the first six months), while an impact is often expected at the year and longer point.

With both an outcome and impact evaluation, one needs a comparison group — ideally, a comparison group where you have been able to randomly assign participants to groups that either receive the intervention or are not exposed to it. (Often this group may be exposed to a different kind of educational experience, perhaps in a totally different field).

While a comparison or control group may not be feasible for everyone, there are some things that can help programs assess whether they are on track. For example, you may want to explore the feasibility of being able to track participants from your program over longer periods of time to gather information on the effects of the program. This preliminary step will help you ascertain the likelihood of being able to follow up with participants and whether the intervention is strong enough to have a lasting effect.

Dr. Brindis suggested that randomization and comparison groups help to assure that the results achieved by the program are a result of the program and not because of maturational effects (teens get older and may engage in more risk-taking behaviors because of the risks that they are exposed to, not because they participated in the program).

Co-moderator Lori Rolleri asked the experts to explain the difference between formative and process evaluations. Dr. Brindis replied that formative and process can go hand-in-hand. Formative occurs in the early stages of program and evaluation development. It can help answer a series of questions — Is the program model appropriate? Is the intervention being implemented as planned? Are there active participants in each of the sessions? Is the intervention being fully implemented?

Process evaluation often occurs once you are assured that the program model is the one you are actually using and that the evaluation tool is capturing the content of the program. Process can help answer the "how many," "who," and "what" types of questions. it can also help document, for example, that the numerical goals of reaching a certain number of participants has been reached, without necessarily addressing the quality aspect of what the formative evaluation step was able to answer.

Co-moderator Karin Coyle said she likes to think of formative evaluation as a step in "forming" a program — an opportunity to collect data to shape or reshape a program. Methods such as needs assessments, focus groups, field notes, satisfaction surveys, open-ended questions on satisfaction surveys, portfolios or performance tests are some of the methods that could be used to collect formative data. She added that process evaluation, as was noted by co-moderator Brindis, allows one to document the process of the program implementation. Logs, sign in sheets, and meeting minutes are a few common ways to gather process data.

One of the forum participants asked for a clarification of the term "indicator," which he said he has often seen used with the term "measure." An indicator (as in a "measurement indicator"), explained co-moderator Karin Coyle, is a specification of how we might measure or record a broad concept. Indicators for health status, for example, could include life expectancy, birth and death rates, and many others. Often these indicators are monitored over time and may be calculated for different sub-groups.

Co-moderator Valerie Sedivy agreed, noting that to her, a measure is a more general term. A given indicator could be measured several ways. For example, an indicator related to contraceptive use might be measured by asking about use at last intercourse.

Participant Robbie Brunger, an evaluator with the Ounce of Prevention Fund of Florida, felt that the real "punch" of after-school programs that include a pregnancy prevention component is integrally related to the simple fact that they all KEEP KIDS BUSY at a time during which impregnation might otherwise occur. He asked for feedback on his conclusion that program frequency (like four times a week rather than two times a week) might be an important factor in program success; and that the actual content of "the talk" (e.g., abstinence at all costs vs. anything that works) might actually be a lot less important than the debate about it would otherwise lead one to conclude.

Co-moderator Claire Brindis agreed that the research as well as common sense have helped to inform us that the more actively engaged young people are in youth positive experiences, the greater likelihood that risk-taking behavior will be decreased. What is important to consider is what makes these programs so critically important for young people. She stressed the need to assure that after-school type programs don't just end at the elementary school years.

Dr. Brindis noted that there are many other critical ingredients — such as the types of relationships and mentoring experiences, connectivity to meaningful adults, meaningful opportunities for self-expression, affirmation, decreasing isolation that place young people at risk — that go along with the component of "keeping young people busy."

She added that providing viable alternatives to early sexual relationships makes after-school programs an important key in the portfolio of what teens need. She noted that programs such as the one described by participant Brunger that offer academic help provide another important service because school failure has long been an antecedent to early childbearing, and many young people have already dropped out of school before they have their first child during the adolescent years.

Dr. Brindis concluded that it is the composite of the protective ingredients that is offered — and not simply the fact that the students are kept busy — that gives young people the tools they need for effective decision making regarding their sexual behavior, including delaying the initiation of sex and practicing safe sex in the future.

She stressed that the "talk" should be seen as part of a constellation of conversations regarding adolescents' need for knowledge in a number of inter-related areas — from tobacco and substance abuse prevention to wise choices regarding physical activity and nutrition.

Co-moderator Karin Coyle agreed that there are a number of factors that affect decisions about risk behavior. There is a new summary of data about where and when teens first have sex (a publication by the National Campaign to Prevent Teen Pregnancy with Child Trends called "Putting What Works to Work" — Number 1, June 2003). In that summary, 42% of youth reported having first sex between 10:00 pm and 7:00 am. Far fewer (15%) report having sex the first time between 3:00 pm and 6:00 pm. She noted that these data are focused on first sex so the question about when they have sex most of the time remains unresolved.

Another participant asked for suggestions of ways to measure the efficacy of a proposed lunch program for teen mothers offered in conjunction with a pediatric clinic. She also asked for advice on possible funding for such an initiative.

Dr. Brindis said these educational groups provide a key opportunity to create strong mutual support among a group of often isolated, marginalized young people. She stressed that it is important to set up a realistic set of expectations regarding what the educational session can do. The fact that these sessions are linked to the clinical program means that contraceptives can be made readily available.

Dr. Brindis cautioned, however, that the educational effort is different. You have to decide whether you are attempting to change attitudes, knowledge, behavioral intent, etc. as well as decreasing social isolation. One approach is to do a combined assessment and baseline survey. The assessment can be part of the pediatric visit and ascertain what personal goals the mother has for herself and her child. Parenting issues, for example, knowledge of child development and what can be expected at different stages of development, can clearly be tested at baseline and further reinforced as part of the visit. Thus, beyond the one-hour intervention (or series of sessions), the assessment can be seen as an extension of the clinical visit.

As far as the availability of grants for such an effort, Dr. Brindis responded that she does not know of any in particular and pointed out that it is difficult to ascertain what costs should be anticipated for such a program apart from the health education time and the food. She suggested considering more non-traditional sources of funding, such as tapping local businesses to defray some of the food costs.

Co-moderator Lori Rolleri suggested looking at the Dollar-a-Day Program, which provides monetary incentives and peer support in an effort to prevent second pregnancies.

One participant noted that millions of dollars have been put into abstinence-only programs that, to his knowledge, have not proven to be effective. He asked for the current line of thinking in the field regarding the evaluation of abstinence-only programs and wondered what the response will be if research does indeed suggest that an abstinence-only program can be effective at preventing teen pregnancy.

Co-moderator Lori Rolleri mentioned two abstinence-only programs that have been found to be effective (as defined by some positive behavior change). One of these programs is titled Making a Difference: An Abstinence Based Approach to Prevention of STDs, HIV and Teen Pregnancy. Those wanting to learn more about this program can visit ReCAPP's section on Making a Difference under Evidence-Based Programs. The other program is Not Me, Not Now, which was evaluated for effectiveness in 2001. A journal summary of that impact evaluation can be found on ReCAPP under Current Research.

Ms. Rolleri also noted that Mathematica Policy Research, Inc. (funded by the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation) is currently evaluating 11 abstinence programs throughout the country. These programs are funded through Title V. A preliminary report that describes the evaluation approach was released in April 2002 and is available at www.mathematica-mpr.com/3rdlevel/abstinence.htm. The final study evaluation report will be completed in Summer 2005.

Co-moderator Claire Brindis thinks it will be important to review the results of abstinence-only programs with the same level of standards that we have applied to other programs. It will also be important to clearly understand what these programs entailed, their length, the population served, etc. She added that there has been a substantial body of work that shows that adolescents are capable of hearing and using a comprehensive message regarding sexual and contraceptive behavior (e.g., abstinence as well as safe sex). There is also an extensive amount of evidence that knowledge does not encourage kids to go out and have sex or that knowledge is dangerous. We will therefore have to consider how the new evidence fits within this larger context.

Many participants wrote in regarding the special constraints and challenges they face in evaluating programs with cultural and gender sensitive indicators and/or instruments. In response to a question about such indicators from a participant in the Philippines, co-moderator Claire Brindis noted that the topic is of increasing interest in the United States also as programs attempt to tailor their efforts to more diverse groups.

Along with culture and gender, we also have to be sensitive to issues of literacy and the general culture of "discomfort" often perceived in discussing topics in Adolescent Reproductive Health (ARH). Specifically in developing culturally appropriate and gender sensitive indicators, there are "universal" principles that help guide appropriate indicators. For example, in-depth knowledge of the participants is key in helping to identify the language that is being used in conceptualizing aspects of ARH that your program is attempting to impact.

Dr. Brindis suggested that the use of focus group discussions as a step in developing indicators is helpful in assessing specific language nuances that are responsive to the cultural group. These discussions can help guide the development of specific indicators. She also noted that pilot testing is helpful — especially as an opportunity to meet with representatives from the cultural and gender group you are trying to reach. Following the group's completion of the pilot test, it is helpful to go over every single item to ascertain whether the language makes sense, the reading level is appropriate to the group, and the language is culturally responsive.

A participant managing a parent education program in Jamaica, West Indies asked for suggestions on the simplest way to measure program impact for a series of 10 workshops designed to increase parent-child communication about sexuality and sexual health. The goal is to reach 7,200 parents in low-income communities in a 12-month period. The parent population are of low literacy, and many might be hard to observe in action, as programs are often conducted in the evening in rural areas. The participant also asked for suggestions on how to measure program impact at the end of the project six months later.

After noting that the project sounded interesting and presented a fascinating challenge, co-moderator Claire Brindis said that increasing communication about sexuality is a fairly broad goal, so it would be helpful to know more about the specific outcomes the program is attempting to accomplish. Is the program is attempting to increase the parents' level of comfort or their knowledge, or change their attitudes regarding opening this kind of conversation with their children, or train them to help other parents? There may be other goals as well, such as an increase in the overall level of parent and child communication, more interaction between child and parent, and changes in parenting style to actively engage the young person in problem-solving along with the parent.

That being said, Dr. Brindis suggested that the "easiest" evaluation would be to ascertain whether the numerical goals are being reached and parents are actually attending all 10 sessions. Some immediate assessment can be made at the end of the 10th session to ascertain whether these parents have any plans to speak to their children and how they intend to have the conversations. Again, Dr. Brindis pointed out that clarity on what is being attempted (such as ascertaining the frequency and topics of the conversations) will help guide the evaluation process.

As for the low literacy level of the parents, a typical paper and pencil test may not work in this setting. She suggested some alternatives, such as observing parents role play ways that they plan to bring up the conversation, interviews with a sample of those trained, and focus groups with participants after several months to assess what (if any) conversations occurred. Given the sensitive nature and the traditional lack of comfort of such discussions, this type of personal data collection requires a sensitive touch.

Parent educators can become an important resource in terms of following up with the participants as they have likely established a level of support and comfort. They may be in the best position to do follow-up interviews with a sample of parents.

Finally, children themselves (depending on their ages) can be an important source of information. This may be challenging, but it would be helpful to have their input (both baseline and follow-up) if at all possible.

Co-moderator Lori Rolleri suggested a publication by SIECUS titled: "Innovative Approaches to Increase Parent-Child Communication about Sexuality: Their Impact and Examples from the Field." 2002. This monograph reviews the impact of several programs designed to increase parent-child communication about sex. Even though specific evaluation tools are not provided, it may be helpful to contact some of the programs listed for evaluation ideas and approaches. The publication can be ordered by calling SIECUS at 202-265-2405 or emailing them at siecus@siecus.org.

Another participant who does outreach to Asian and Pacific Islander populations as well as to Mexican and Chicano communities in Colorado asked how many languages the evaluations should be translated into as well as how their content should be tweaked based on cultural issues that might effect how respondents talk about sex and sexual assault.

In response to this question and another from a participant in a rural French-speaking minority area in New Brunswick, Canada, co-moderator Mary Martha Wilson noted that rural community efforts have been her specialty and suggested that anyone wanting to discuss such issues should contact her at mmwilson@noappp.org.

top

A participant asked for research and resources related to making evaluative surveys more quantitative for an educational theatre program which presents HIV/AIDS awareness, sex awareness, and health issues education in the form of theatre.

If his theatre program is a series of productions/plays that are viewed by a target audience, responded co-moderator Mary Martha Wilson, he should think about ways in which he could develop specific pre/post test survey questions (or better still, find appropriate questions already developed) that relate to the content of the play itself. For example, if the play is focused on HIV/AIDS awareness, one could develop questions that measure levels of awareness about specific items of interest (transmission risk for example) and look at the proportion of "correct" responses per individual before and after the performance to measure how well certain information is conveyed via the performance.

Ms. Wilson has worked with pre/post test data for a breast cancer awareness outreach project in which participants were surveyed in this manner (before and after the correct information was presented), and the changes in levels of understanding were notable. This type of survey would not measure long-term recall but would hopefully assist the evaluation efforts overall.

Co-moderator Lori Rolleri offered some resources on the topic (see Resources) and suggested contacting the Nitestar Program/Star Theatre program because it has been evaluated and could be willing to share some of its instruments. The program is run out of St. Luke's Roosevelt Hospital Center in New York City and was funded through the federal Office of Adolescent Pregnancy Programs and the Division of HIV/AIDS at CDC.

Ms. Rolleri also offered to share copies of evaluations ETR did in conjunction with San Francisco Unified School District. This research evaluated Kaiser productions (SECRETS and Nightmare on Puberty Street) which involved a post-only survey for youth and a survey for teachers.

A participant from a rural county in Massachusetts noted that her teen pregnancy prevention coalition is insisting on pre/post-test surveys for all of its direct service programs. They are in the process of designing a pre/post-test that can be used throughout the state. They are hoping to develop something akin to a sexual risks scale that can assess attitudes, behaviors and intent around condom use, resisting unsafe sex, onset of first sexual experience, etc.

She is tapping into the field of substance abuse prevention, where there is a great deal of survey information in the public domain, and plans to use some of their scales for pre/post-test development since many of the risk and protective factors are the same for both. She will probably come up with 5-10 core statements (scales) that all direct service providers will use and then have them provide additional scales that are specific to their program.

Co-moderator Lori Rolleri cautioned that using the same pre/post-test throughout the state could be difficult, since not everyone is using the same program, and each program should address the same knowledge, attitudes and behaviors that the instrument measures. She suggested two resources for looking at survey questions (see Resources).

According to co-moderator Claire Brindis, it is also important to consider the content of the program, whether the providers across the programs you fund have any commonalities, and whether a tool that could be used in multiple sites might be a reality. She added that it is important to determine how long the program is — i.e., is it a one-time only event or a series of events? It is also important to have realistic expectations of what the program will be able to demonstrate and to build in accountability.

Dr. Brindis cautioned that since many of these programs are likely to be relatively small, program managers will need to consider a way of keeping track of how many sessions participants attended and who completed both pre- and post-tests.

Co-moderator Lori Rolleri asked for suggestions on how to control for the fact that all the teens do not attend all the sessions. She asked whether those teens who did not attend all the sessions are at a disadvantage in taking the post-test.

Dr. Brindis responded that it is, indeed, challenging to retain students in multi-session programs (although clearly these are preferable to "one shot" deals that have minimum impact). Some programs (especially community-based, rather than school-based) have attempted to build in incentives — e.g., you get a coupon if you attend each session and on the last session, there is a prize. The more coupons you can put in the box, the greater the likelihood of winning (the prize can be a small gift certificate for a CD, for example). Others have found it useful to have a raffle at each session, with increasingly better prizes offered along the way.

Dr. Brindis suggested thinking about ways to incorporate some "fun" into the programs so that they continue to be appealing to young people since attendance is a major problem in and out of school. She noted that while you can't control attendance, you can try to get a handle on how much exposure there was to the program.

For example, if you are running a five-part series, at the top of the post-test, you can list each of the five sessions by number or theme and ask post-test takers to circle those sessions they attended. This system can serve as a back up to sign-in sheets collected at each session.

Dr. Brindis also suggested discussing the issue of increasing young people's attendance at your educational programs with other young people.

Co-moderator Karin Coyle added her thoughts to the exposure issue. In randomized trials, the standard is to include all participants in the analyses regardless of their exposure. She agrees with co-moderator Claire Brindis that it is important to assess exposure because we often measure exposure in two ways — keeping a record of lessons attended and including a few items on the follow up surveys that measure activities or events that took place in the program. We expect to see participants in our intervention group respond affirmatively to these items at a higher rate than participants in the comparison group. It also gives us a sense of what is happening in the comparison group.

Dr. Coyle suggested that if you have a measure of exposure (e.g., a score for the number of lessons each participant received in a school-based study), you could include this in your data analyses to assess if greater exposure yielded greater outcomes.

It is very important to build on the emerging field of knowledge, Dr. Brindis added. There have been a number of evaluation tools developed in the past with "Evidence-Based Programs" that can be used as the basis for the development of the next generation of programs.

A participant added that the way her program accounts for teens who miss sessions is to have them do make-up work on the topics for the sessions they miss. They also provide an information packet to each teen on each topic, which ensures that all teens have information on all the topics included in the training. Dr. Brindis responded that not only were those good suggestions for making up for absences, but they also served to reinforce the information that the program is attempting to share and will be handy long after the training is completed.

In response to a participant's request for suggestions on meeting the challenges of convincing school principals to allow pre-tests and post-tests to be conducted with their students, Dr. Coyle agreed that it is increasingly difficult to ask for time from the schools. She works with a lot of schools in her studies and is most successful when she is very clear from the outset regarding their agreements and their benefits of participation. She also offered the following tips for working cooperatively with the schools:

1. Try to schedule data collection at times that fit in better with the school calendar (e.g., if collecting data in the spring, do it before the school testing crunch starts).
2. Use outside data collectors to minimize the burden on the teachers and use incentives for schools and/or teachers. Though this doesn't help with the time issue, it does reflect a professional courtesy that schools and teachers appreciate. Even smaller incentives like bringing two reams of paper to each teacher involved in the survey have made a big positive impression.
3. Use sampling so that you do not have to survey every student.
4. Do not grant all school requests for concessions. For example, only survey during PE if it's a last resort because this type of class is more difficult to work with because there are no desks/chairs, and most students would prefer to go to PE rather than take a survey. Other problematic times are after-school surveying and surveying at lunch. Lunch is often too short, and it's the only time youth have to decompress. After-school hours are difficult because of transportation and adding to a long day. When schools suggest these options, summarize why they are less than ideal, and ask if you can explore other options (e.g., home room).
5. Talk about the school's contribution to the project and evaluation and think about other benefits it may gain. Perhaps you can share a summary with the schools after the study, or you can provide a presentation at a teacher in-service on the issue. Schools tend to appreciate this type of support.

Co-moderator Lori Rolleri invited forum moderators and others to share some tips for running successful focus groups. In particular, she asked the respondents to discuss how conducting a focus group is different from running a workshop or discussion group. Many participants shared their own focus group experiences, challenges and questions.

Co-moderator Karin Coyle noted that there are many uses for focus groups in her work — for program development (to get opinions on issues faced by our population of interest, or to pilot test lesson activities, materials, or planned videos); for questionnaire development; and for evaluation purposes.

After encouraging participants to review the available resources on focus groups, Dr. Coyle offered the following tips:

1. Keep the group relatively small (6-8 is optimal in most cases). If you have smaller or larger groups, they can become dominated by a few individuals. Larger groups also tend to result in more side conversations, which makes them more difficult to moderate.
2. Participants should be selected based on some shared characteristic that is related to the topic of interest (e.g., they are all middle school youth of a certain age; they all have visited a certain clinic, etc.)
3. Think about characteristics that may influence the discussion and consider clustering your groups based on those characteristics. (For example, if you think gender might adversely influence the level of comfort or discussion in a group, separate participants into male and female groups.)
4. Be sure to have a protocol that includes the objectives of the group, an introduction, ground rules, and a set of open-ended questions that stem from the objectives. The questions should be limited in number and sequenced from general to specific.
5. It's good to start with an opening question that's easy and gets everyone talking, but be careful not to use a question that is status-oriented or puts some people in a position of feeling "less worthy" of being there. (For example, asking people to share their educational background or occupation in a mixed group may make some uncomfortable.)
6. Have a skilled moderator who is very familiar with the protocol and the objectives of the group. The moderator needs to be very skilled at maintaining good group dynamics — encouraging participation, probing for clarity, knowing when to move on to the next topic, making participants comfortable in sharing ideas, etc.
7. Take advantage of the richness of the focus group data when you are analyzing it. Sometimes focus groups get used to generate counts (x people said this), but really they should be used to gain insight and understanding, not to generalize to a broader population.

Participant Barbara Wilson offered her teen abstinence and pregnancy prevention program as a means of comparison between Dr. Coyle's tips on focus groups and how they actually play out in her experience. Her program provides three- to four-week workshops on improving communication with teens about pregnancy prevention and providing strategies to parents for behavior management issues. They have developed 10 open-ended questions which are moderated by the facilitator — a parent workshop trainer, teacher, or Ms. Wilson.

The following guidelines underlie these focus groups:

• Each facilitator works with a small group of parents, six to eight, depending on how many are attending the workshop series.
• Parents and family members have children attending the same school and are approximately in the same age group, 15-18 years of age.
• The moderator communicates at the parent level, not imposing his/her status, and has been trained in group dynamics.

Ms. Wilson has found that parents and family members "feel more comfortable" in discussing what they have learned with this approach as well as the techniques they are using at home with their children. This approach also helps improve on communication with peer and school personnel and provides an outlet for suggestions on how their training methods can be improved.

Dr. Coyle added the importance of providing participants with incentives, such as food, to encourage participation.

Another participant admitted that her district misused focus groups for years due to a misunderstanding about their appropriate use. She strongly recommended looking to universities and evaluation professionals for guidance prior to conducting focus groups. She described the process her district followed for conducting focus groups in all of their counties using a handful of questions which were open ended and allowed for small group discussion among a selected group of 5-7 participants. They chose people in informal leadership positions, such as neighborhood "natural" leaders and community members who had interest and experience.

Tape recorders were used to document information gathered through the discussions and compare the information to their actual data. For example, if a focus group referred to a lack of after-school programs but the data on services revealed several were available in that community, then they were able to identify a disconnect between their resources and community perceptions, which, in turn, informed their strategic planning.

Dr. Coyle agreed that it is really important to know the purpose of your focus groups and spend the time to get training. This is a resource intensive data collection approach, she stressed, and it's worth investing the time up front so you can maximize the output.

Francoise Kendall, Training Manager for the Louisiana Family Planning Program, offered some hints gleaned from her extensive experience conducting focus group discussions (FGDs). She first pointed out that focus groups are often used as a complement to individual/in-depth interviews because they stimulate interactions between participants and as a result, bring a wider range of responses and thoughts.

In addition, the participants' interactions can give great insight on how the issues that are being discussed are really perceived. Focus groups have the advantage of being less time consuming than in-depth interviews, but they require a good deal of organization.

FGDs are usually moderated by a moderator and assisted by a note taker, whose role is simply to record what is said and not to participate in the discussion. Ms. Kendall stressed the importance of choosing a trained moderator because the FGD is far more involved than a mere group discussion.

She also offered the following hints regarding FGDs:

• FGDs should be composed of participants who are homogenous in terms of social/ethnic/age/sex/religious background. For instance, you don't want to mix up 13- and 18-year-old girls and boys, or students and teachers.
• The group should have around 8-10 people. This will allow everybody to participate and also ease the work of the moderator.
• The FGD should take place in a neutral and non threatening setting, easily accessible by the participants.
• The discussion can be framed through a discussion guide meant to help the moderator with the topics that need to be covered. It is very important to ask open-ended and not leading or biasing questions, and to probe the answers to get more in-depth responses. Ms. Kendall likened probing to peeling an onion in that the discussion should move from the general to the specific. It should last about one and one-half hours.
• The moderator should preferably share the participants' ethnic/age/sex background. He/she should have great listening skills and be nonjudgmental. The moderator should also understand that his/her role is not to teach or educate but to gather information. Being in the role of listener is often a challenge since the participants often want to take advantage of having an expert there to learn something more about the topic that is being discussed.
• Several factors — such as the makeup of the population, the variables to be discussed, and the geographical areas involved — influence the number of FGDs you should conduct.

Ms. Rolleri agreed that the role the facilitator plays is crucial, and she asked her co-moderators for their thoughts on some things a focus group facilitator might do to encourage discussion and make people comfortable. Dr. Coyle shared the following insights on moderator skills:

• Listen carefully and show your respect for participants (e.g., call them by first name, listen to responses and focus on participants as they talk). Having a note taker (and taping the session) reduces distractions for the moderator so he/she can focus on running the group.
• Be comfortable with short pauses. Sometimes moderators will talk too much and not allow group members to reflect on or add to the discussion.
• Probe in nonjudgmental ways (e.g., Would you say more about that? Can you give me an example? Has anyone had a different experience?). Sometimes we use "why" as a probe, and this can put folks on the defensive because it often sounds as if a response is being questioned.
• At the outset, remind and encourage differing points of view. Focus groups are not intended to yield consensus on a topic; they are designed to elicit opinions and views from participants.
• Be cognizant of your responses (verbal and non-verbal). Try to avoid making judgmental statements in response to a comment (excellent, correct, etc.) If people start to worry that the moderator is judging their responses, they may be less comfortable sharing views. You can acknowledge participation by saying "yes" or "uh huh."
• Sometimes there may be a dominant talker, and the moderator may have to jump in and say something like "Thank you, Karin. Are there others who would like to comment on this topic?" From the beginning, you should set up the group to encourage participation and divergent views.

Co-moderator Lori Rolleri solicited advice for those groups who are required by their funders to have their program evaluation plan/protocols reviewed by an institutional review board (IRB), but do not have an IRB at their organization. She also asked forum participants to discuss the role of an IRB.

Co-moderator Karin Coyle explained that, in general, an IRB reviews and approves study protocols for planned research to ensure the protocols do not expose participants to unreasonable risks and the study includes adequate protections for participants (e.g., accurate and adequate consent procedures, protection of privacy, etc.)

IRBs also review adverse events that occur during a study, complaints expressed by participants, or suspected or alleged violations of protocols. Overall, the IRB's primary purpose is to help protect the rights of and minimize risks to study participants.

Co-moderator Claire Brindis noted that traditionally, IRBs (sometimes referred to as Committee on Human Subjects), represent a checks and balances system for assuring that research that is being conducted under the sponsorship of that organization is of the highest caliber and integrity and protects the rights of those being studied.

The whole issue of consent is an important one because we need to assure that participants have a right to participate in the program, whether or not they are willing to participate in the research as well. Funders are becoming increasingly concerned about how their funds are being used and the protection of human subjects.

If your institution does not have its own external review process, one option is to approach a local university to ascertain if they can set up a relationship with your own agency; sometimes other agencies in the community have undergone the necessary approval process to become an IRB. University groups may be able to direct you to these groups. Another tactic is to identify other groups in your community that conduct research to see if they have set up such a mechanism.

Participant Taggert Doll from Michigan said he is from an organization that does have an IRB panel. His program is available to do IRB reviews for other organizations that may not have IRBs, and he suggested checking larger organizations/universities around your area to see if they will do a courtesy review for you.

Another participant said her organization had to use a commercial IRB because the local universities wouldn't review outside proposals because of liability issues. She also pointed out that it is wise to check with your funder before proceeding because many have definite preferences on the type of IRB used.

Dr. Brindis added to the participants' suggestions by pointing out that there is also a free standing IRB, the Western Institutional Review Board at www.wirb.com. As noted on its web site, the Western Institutional Review Board (WIRB) offers a broad range of review board services. WIRB has experience reviewing federally-regulated and industry-sponsored research for both large and small institutions. WIRB can act as the sole review board or support the work of an institution's own IRB.

Co-moderator Karin Coyle suggested that a program's logic model can be a great guide for focusing its evaluation. If, for example, you have identified important determinants of the behavior of focus in your logic model, some of those determinants could become evaluation indicators (e.g., attitudes, perceived self efficacy, and connectedness to a caring adult).

ReCAPP's section on Evidence-Based Programs.


• Advocates for Youth "Science and Success" monograph at www.advocatesforyouth.org/programsthatwork/index.htm.
• Sociometrics PASHA database at www.socio.com/pasha.htm.

• Card, J.J. (Ed). (1993). Handbook of adolescent sexuality and pregnancy: Research and evaluation instruments. Newbury Park: SAGE Publications, Inc., and
  
• Davis, C.M., Yarber, W.L., Bauserman, R., Schreer, G., & Davis, S.L. (Eds). (1998). Handbook of sexuality related measures. Thousand Oaks: SAGE Publications, Inc.
  
  These two publications can be ordered from SAGE:
  SAGE Publications, Inc.
  2455 Teller Road
  Thousand Oaks, CA 91320
  (805) 499-0871
  Email: order@sagepub.com
  
  Complete descriptions of each publication are available at the SAGE web site:www.sagepub.com

• Nitestar Program/Star Theatre Program
  Cydelle Berlin, PhD, Executive Director
  Phone: (212) 523-3599
  Email: cberlin@slrhc.org
• Glik, D. et. al. (2002). Youth Performing Arts Entertainment-Education for HIV/AIDS Prevention and Health promotion: Practice and Research. Journal of Health Communication, Vol. 7, pp 39-57.
  Email: dglik@ucla.edu.
• Hillman, E. et al. Pregnancy, STDs and AIDS Prevention: Evaluation of New Image Teen Theatre. (1991). AIDS Education and Prevention, 3(4) 328-340.
  Eva Hillman is at UC San Diego.
• Spare Change Theatre Group at Six Rivers Planned Parenthood in Eureka, CA
  (707) 445-2018.
• Community Awareness Motivation Partnership (CAMP) in Alhambra, CA
  Phone: Aida Feria, (626) 369-1173
  Email: parachoices@earthlink.net

• C.A.R.T. (Compendium of Assessment and Research Tools)
  cart.rmcdenver.com
  This web site allows you to enter key words for the tools of interest, and it will provide a list of tools with sources for obtaining them.



• ReCAPP's Research Glossary.

top

• The American Statistical Association
  www.amstat.org/sections/srms/brochures/focusgroups.pdf
• The University of Arizona - Tucson
  Using Focus Groups for Evaluation
  ag.arizona.edu/fcr/fs/cyfar/focus.htm