• identify and write down the most important elements of that complex reality using the best evidence available,
  
  
• determine which parts of that complex reality are most amenable to change given the reasonable resources that you have, and
  
  
• focus your interventions on those parts.

Dr. Kirby began by admitting his bias, as a researcher, for the value and importance of conducting good research and using it to improve interventions. He noted that people can err on both sides of the equation, by conducting either too little or too much research. In the large majority of situations that he sees, people have conducted too little research or they have reviewed too little of the published research that others have already conducted. Too often, people fail to recognize that multiple forces often operate in a situation. Also, too often people are unaware of what research demonstrates can and cannot work in different situations.

Dr. Kirby consequently suggested that most communities or programs should devote greater effort both to reviewing past research and trying to assess the extent to which risk and protective factors that have been found to be important in other populations are also important in their own population.

However, Dr. Kirby pointed out that it is sometimes true that relatively little good research exists, yet the health need is great. In such situations, he encourages communities to proceed without waiting for the perfect research study. He pointed to Africa as an example, where AIDS is a truly enormous problem. We must therefore begin implementing a wide variety of interventions even though we do not yet know what the most effective approaches are.

Dr. Kirby encourages communities and programs in the U.S. to follow the following guidelines:

• Review the existing summaries of the research in your field or health topic.
  
  
• Review the studies on the most important behaviors and risk and protective factors.
  
  
• Conduct focus groups with your target population to assess perceived risk and protective factors.
  
  
• Interview selected people with knowledge about the field. Be sure to include a diversity of people who have different kinds of expertise based on their different perspectives.
  
  
• If possible, involve researchers from a nearby college or university to help you develop surveys, administer those surveys, and analyze the data.

He noted that this was a partial list and encouraged the forum to add to it.

• a radio campaign with a positive message about abstinence
• a school-wide poster contest
• classroom curriculum activities
• peer education program
• teen theater program, etc.

Ms. Rolleri, after acknowledging that these were good suggestions, noted that it takes discipline to pursue this approach. It has been her experience that some program planners often get excited about a particular intervention idea, such as teen theater, and run with it. With at least some research, and the discipline to survey other options, it is possible to discover more effective interventions that may cost even less to implement.

Another participant asked Dr. Kirby what types of strategic interventions he would utilize to target the male adolescent population when trying to reduce teenage pregnancy among this group. She also wondered how the BDI logic model would impact their developmental behaviors as it relates to engaging in risky behaviors.

Dr. Kirby responded that BDI logic models will provide a process for answering her questions. Given her program's focus on males, completing that process would entail identifying the key behaviors that males need to change in order to avoid impregnating teen females (e.g., delaying the initiation of sex, having sex less frequently, not having sex with teen females, using condoms, in a variety of ways encouraging their female partners to use contraception), then identifying the determinants of those behaviors, and finally identifying or developing strategic interventions to address those determinants. Dr. Kirby suggested she start by looking at the figures/diagrams at the end of his logic model paper which deal with teen pregnancy and then developing her own specifically for males.

Dr. Kirby also noted that part of the process of developing good BDI logic models involves looking at the evidence, and the evidence shows that some programs have reduced either teen pregnancy or the behaviors that lead to teen pregnancy (these are summarized in his paper Emerging Answers), and some programs — although not designed specifically for males — had a greater impact on males (e.g., Draw the Line and Becoming a Responsible Teen).

A participant questioned the value of applying the evaluation part of a logic model if it was unclear whether the logic model adequately represented the program being evaluated.

After clarifying the question, Dr. Kirby noted that one reasonably effective way to determine if a program is based on a logic model is to simply ask the program developers if the program is based on a logic model, and if so, to request a copy of it. If they don't have one, then your question is directly answered. If they do have one, then you should review it and determine whether it specifies the important elements of BDI logic models (i.e., the important behaviors, determinants, and intervention components specified).

Dr. Kirby cautioned that using a BDI logic model may be critical if you are helping to conduct an evaluation of the program, especially an outcome evaluation. The intervention components specified in the logic model identify the important components for which you should complete a process evaluation. For example, you might wish to assess whether the components were even implemented, if they were implemented with fidelity, how many people were reached, and the reaction of the target population. The BDI logic model also specifies what should be measured in any outcome evaluation. That is, it specifies the determinants (risk and protective factors), the behaviors and the health goal(s) that should be measured in an evaluation design if at all possible.

In conclusion, Dr. Kirby stated that BDI logic models specify what is important to measure. Without a well developed logic model, you don't really know what to measure, and an important part of any evaluation is ensuring that you are measuring the right things.

top

Co-moderator Lori Rolleri responded to a query from a participant working with a teen parent program in a school district about how best to evaluate the effectiveness of using teen parents to give talks about the reality of teen parenting to middle and high school students. Ms. Rolleri first asked what the behaviors were that the teen parent program was trying to change, noting that the teen parents and the recipients of the talks are two distinct populations and one would first have to identify the relevant group before measuring the change. She then suggested using pre-test and post-test questionnaires to measure change in the targeted group, but she added that she would have to know more about the talk's learning objectives and content in order to develop appropriate questionnaire questions.

Ms. Rolleri also suggested using a post-talk satisfaction questionnaire, which might include the following items:

• What did you like about today's talk?
  
  
• What suggestions do you have to improve today's talk?
  
  
• Identify one new thing that you learned today.
  
  
• Describe one thing from today's talk that you will apply in your day-to-day work.
  
  
• Overall, today's talk was (circle the appropriate number):
  1. Poor
  2. Fair
  3. Good
  4. Very Good
  5. Excellent
  
  
• Is there anything else you would like to share with us?

The participant elaborated that the teen parent program in question was hoping for two outcomes: to reduce the number of subsequent pregnancies of its teen parent speakers and to reduce the teen pregnancies of the students who are listening to their peers. He noted that he thought it was more likely for students to listen to a message given by teen parents.

Dr. Kirby responded with a cautionary tale about his own experience using teens as speakers which backfired. He described a program he helped develop and rigorously evaluate which used teen mothers and young HIV positive speakers to lead a 16-session course for middle school youth. The students loved the speakers but walked away with the wrong message — namely, that being teen mothers and being HIV positive must not be so bad.

Dr. Kirby pointed out that this negative experience does not necessarily mean that peer speakers are doomed to fail, but that one should be cautious in using this approach. The participant responded by saying that an informal survey evaluation of his program, asking the audience what they learned and how they felt, indicated positive results. In fact, 99% of the respondents said that they did not want to become teen parents after hearing firsthand what it's really like.

Another participant wondered about the accuracy of after-presentation surveys. As an evaluator, she has found that respondents tend to rate presenters highly. Respondents also tend to give you the feedback they think you want. For this reason, she agrees that some kind of longer term followup would be ideal, but noted that such follow-up is rarely feasible.

Another disadvantage to pre/post questionnaires, the participant pointed out, is that they often take up a significant amount of time. One has to be careful not to overburden participants and take away from the intervention time, particularly when time is limited.

More discussion followed regarding the most effective and feasible way to design pre- and post-tests and how best to obtain a comparison group. After noting that this discussion was off topic, Dr. Kirby said that if attendance is quite high, and if few youth migrate into or out of the classes between the pre-test and post-test, you may not need identifiers and can just compare the pre- and post-test means. However, if you allow more time to elapse between pre-tests and post-tests but still administer them during the same classes, you will get a better measure of slightly longer term impact.

The down side, Dr. Kirby cautioned, is that if you allow too much time between pre-test and post-test surveys, then you need a comparison group of those who do not get the intervention between the pre- and post-tests. The comparison group will enable you to control for maturation and special events that may have produced change.

Dr. Kirby then offered some final warnings about surveys that merely ask students about their experiences. He agreed with previous comments that if youth like a presenter, they are likely to rate the presenter or activities high on impact — even if they had little impact. Similarly, students have said a course had a substantial impact on them when pre- and post-tests revealed no impact.

Dr. Kirby noted, however, that sometimes programs have significant effects on important determinants and behaviors — but not on the health goals — because it is very difficult to measure the impact on the health goal. For example, studies can sometimes show that programs decrease the frequency of sex and increase condom or contraceptive use. Logically, these changes will lead to a reduction in pregnancy, but often, for statistical reasons, it is not possible to show a significant change in pregnancy rates (doing so requires a huge sample in the thousands). In these cases, the evaluator may not choose to try to measure an important behavior or health goal, even though it is in your logic model.

BDI logic models identify the intervention components that should be the focus of the process evaluation. However, the BDI logic models may not specify everything that you might wish to measure in your process evaluation. For example, you might wish to measure whether or not each intervention component was implemented, the extent to which it was implemented with fidelity, the number of people reached, and participants' reaction to the component. Not all of these elements may be spelled out by your model. Thus, you may need to provide your evaluator with more than a logic model, even though the model should be a critical part of what you provide.

A participant asked for clarification on how generating a BDI logic model for her program could assist her in designing a new local evaluation. After providing the caveat that he was making certain assumptions in order to answer her questions, Dr. Kirby advised her to first decide on the type of evaluation she is going to complete — either a process evaluation or an impact and outcome evaluation.

Having already touched on the possible elements to include in a process evaluation, Dr. Kirby elaborated on impact and outcome evaluations. If you want to attempt an outcome and have the resources to complete it, then you need a good evaluation design (e.g., pre-test/post-test surveys with intervention and comparison students), a sufficiently large sample size, and a good questionnaire that measures what you want to measure.

The BDI logic model specifies what to measure. If your logic model specifies sexual behaviors (that is, if your goal is to change specific sexual behaviors), then you need to measure those as clearly and explicitly as possible. Dr. Kirby warned that you must gather this information cautiously and carefully and be certain that you have appropriate approval and that confidentiality is maintained. He strongly encouraged the forum to involve experienced researchers if they did not have the relevant experience themselves.

Dr. Kirby also encouraged participants to ask more than one question and create multi-item scales when measuring each determinant. It is then possible to calculate the inter-item reliability of those scales. He noted that there are multiple kinds of determinants and different methods of measuring each kind.

If one of your determinants is knowledge about the risks of pregnancy or HIV and the proper way to use condoms, then questions about these topics can be asked. If one of your determinants is self-efficacy to avoid sex, then questions that describe various scenarios and ask how confident they are that they can successfully resist sex in those situations can be asked. If one of your determinants is perceived peer norms, then questions about peer norms can be asked.

Dr. Kirby noted as an aside that many questionnaires that have been used to evaluate the impact of HIV and pregnancy prevention programs on risk and protective factors and important behaviors have already been developed, such as those developed by ETR in their evaluations and by the authors of BART.

A participant asked how one can use BDI logic models most effectively to develop and assess best practice school-based and teacher training programs regarding reproductive and sexual health.

Dr. Kirby responded by saying that first, consistent with the process for creating any logic model, you need to identify the specific reproductive and sexual health goals that you have. He noted that there are many in this instance. At one time he created a list of more than 20. He then suggested selecting a few reproductive health goals since neither training programs nor BDI logic models can handle 20 or more goals at one time.

Once you have identified the goals, you must then identify the important youth behaviors that you want to change and perhaps the youth determinants of those behaviors. It is at this point, Dr. Kirby cautioned, that it begins to get complicated. You are not developing a logic model primarily for the youth, he noted, but rather a logic model for a training program to change the behaviors of teachers so that they, in turn, will change the behavior of youth. This requires adding additional columns or using "nested models" (discussed, along with examples, on pages 17-19 of his logic model paper [pdf file]).

After identifying the youth behaviors and the determinants of the youth behaviors, Dr. Kirby said, the next step is to identify the teachers' behaviors that should be changed (in order to change the youths' determinants). Then you need to identify the determinants of those behaviors. Finally you need to design interventions to change those teachers' determinants.

Dr. Kirby also noted that when evaluating teacher trainings, you should try to measure whether or not the training affected the determinants of the teachers' behavior (e.g., their knowledge of best practices, their motivation, their skill, etc.) and also their actual teaching behaviors. In an ideal world, you would also measure the actual impact of the training on the students involved, but he noted that there are rarely resources available to do this.

In response to a question about how logic models can portray the theories of change used in developing effective interventions, Dr. Kirby pointed out that "theories of change" means different things to different people. Some theories of change are identical to BDI logic models in that they clearly lay out how interventions are going to produce various changes, which, in turn, will change important behaviors and, in turn, will achieve a health goal. For example, Dr. Kirby saw a theory of change presented in the Plain Talk intervention that he considered to be quite a good logic model.

Other people use "theory of change" more specifically to refer to particular theories of change, e.g., Prochaska's theory of change. Some of these theories recognize that people may be at various stages in a process of changing behavior (e.g., precontemplation, contemplation, preparation, action and maintenance). Dr. Kirby suggested that this is a useful, but complementary, way of looking at change in human behavior.

Typically, various factors move people from one stage to the next. For example, becoming more aware of the immediate health risks of smoking might cause some people to contemplate not smoking, and peers' norms may help people move toward quitting smoking. Thus, logic models can be used to identify the important factors that move people from one stage to the next.

Co-moderator Lori Rolleri recommended ReCAPP's Theories and Approaches section, which has very practical descriptions of two health education theories: social learning theory and health belief model. She also noted that ReCAPP's February 2003 edition will focus on Stages of Change Theory.

In his paper Emerging Answers, Dr. Kirby identified literally hundreds of determinants that previous research has found to be related to sexual behaviors that lead to pregnancy or STD including HIV. These determinants characterized the communities, families, available health services, schools, and peers with whom youth might be involved and, of course, the youth themselves — their biological characteristics, their values, their personality traits, their other risk behaviors, their hopes for the future, and their beliefs, attitudes, motivations, and skills involving various sexual behaviors and pregnancy and STD.

Given such an array of determinants, it is essential to narrow the list. Dr. Kirby noted that there are several ways of doing so. One way is to identify the most important determinants by reviewing the major studies that have measured multiple determinants, conducted the appropriate statistical analysis, and demonstrated which determinants end up being most important. This is being done with the ADD Health data. For example, Bob Blum just released an important analysis of the ADD Health data showing that parent-child connection was more important in delaying the initiation of sex than was parent-child communication about sexuality.

Another way of determining which determinants are important in your community is to conduct focus groups with your target population and interview experts in your community.

And finally, often determinants that are more proximal to specific behaviors are more important than those that are more distal. For example, perceived peer norms about initiating sex, personal values about sex at an early age, and opportunity to have sex are more proximal to initiating sex at an early age and more highly related to early initiation of sex than are participating in religious services, communication with parents about a variety of topics, or peer school performance. These determinants are more distal; they are somewhat related to early initiation of sex but not as highly related to initiation of sex as proximal determinants.

Dr. Kirby concluded this part of the discussion by reminding the forum that there are two important criteria, both of which must be met, when selecting determinants. Pick determinants which are the most important and which you can change given your resources.

She added that common responses — like addressing factors such as higher income level, more appropriate parent supervision, or higher educational performance — are mostly ineffectual when you are working with a child who has been living and working on the street for the past three years. Instead, the most effective interventions seem to center around building a connection with young persons (even if you only see them for an hour) and increasing their internal locus of control by respecting their right to make choices and treating them as the experts about what they need. Such antecedents, however, are harder to measure and define.

The participant added that she runs into a great deal of criticism when laying out a logic model connecting these kinds of interventions to behaviors like increased condom use because it's difficult to prove and define the connection they've established or demonstrate how they've increased the youths' "internal locus of control." She asked Dr. Kirby for suggestions on how to build a convincing logic model on such loosely defined antecedents.

In response, Dr. Kirby noted that the participant was actually asking three separate questions: how to define "connection" and "locus of control," how to establish they are related to teen pregnancy, and how to change them.

Without trying to define them, Dr. Kirby commented that he thinks connections are one of the most important determinants of behaviors. In a viewpoint article published in the Nov/Dec 2001 issue of Family Planning Perspectives, he argued that connection was one of two very important determinants. He added that it is not the connection per se that is important, but rather connection to pro-social people or groups that is important. He suggested using the article and the material reviewed in it to make a case to critics that connection to the right groups is an important determinant of sexual risk-taking.

Dr. Kirby did not, however, find evidence (in his review for Emerging Answers) that locus-of-control was important. He directed the participant to a study on locus-of-control for further information while noting that it is a very broad concept. He suggested shifting to the concept of "self-efficacy," which is much more behavior specific. Also, research does suggest that self-efficacy to say no to unwanted sex, to insist on condom or contraceptive use, etc. are related to their associated behaviors. However, he cautioned that any shift could mean that her current intervention may need to be changed.

Dr. Kirby's paper on BDI logic models discusses the importance of organizing a group of key stakeholders to develop a logic model. A participant asked what his experience has been in convincing grassroots stakeholders about the benefit of the process and solicited feedback from the rest of the forum on what their experience has been with this part of the process.

Another participant expanded on the first question about stakeholders, asking the forum to share their experiences with involving the key stakeholders — youth — in developing the logic model.

Julie Blackwood, from the CAMC Health Education and Research Institute, agreed that it is essential to get buy-in from stakeholders, even though it is tempting to skip this part of the process in the interest of moving forward more quickly. She stated that she has not had the opportunity to implement the process with a group of key stakeholders, but that she has worked through a couple of "trial" exercises with a group of six to eight people. She started with something "simple" like applying the logic model process to "planning a vacation" and "painting a house," which allowed them to get familiar with the process so that they could then apply it to health-related topics. She added that getting people to agree on definitions/terms of success, such as what the ultimate goal is, are key to the success of the process.

One reason that a wide variety of stakeholders like the logic model process, Dr. Kirby noted, is that an important step in logic models is to think broadly and to see the wide variety of risk and protective factors that affect behaviors. If the stakeholders are diverse, then the risk and protective factors that they believe to be important are also diverse. Logic models provide the opportunity for each of the stakeholders to identify factors they believe are important and to see them on paper (or on the board) in one of the steps. At a minimum, they feel heard when they see their thoughts recognized in writing.

Then, in the next step, when they review evidence for the relative importance of different risk and protective factors, and when they review the resources that are available to address different risk and protective factors, they hopefully understand why their favorite risk and protective factors are not kept in the model (if in fact they are deleted).

Sometimes BDI logic models are used to show how different organizations or different sectors in the government can have an impact on a health problem. Once again, different stakeholders like these models because they can then see how they and others can contribute to a health goal.

The reactions were mixed, however, as far as youth being a part of this process. Some youth understand logic models and think they are useful, but others do not. Dr. Kirby thinks most youth have the potential for understanding logic models, but they may need to be taken through the process at a slower pace, they may need more individualized attention, and they probably need a slightly different training model directed towards their specific needs. He added that it would be an interesting project to develop such a model.

Co-moderator Lori Rolleri pointed out that one of the things that she particularly likes about the BDI logic model is its relative simplicity. She asked the forum what their experience has been in communicating the BDI logic model process to people who are completely unfamiliar with the whole idea of a "logic model" and whether it needed to be simplified even further.

Dr. Kirby agreed that involving a diverse group of people with diverse views and interests and getting them to agree on what are the most important behaviors, determinants and interventions is not necessarily an easy process and could backfire — it could simply clarify and sharpen some of the ideological differences of those involved.

Dr. Kirby cautioned the forum that even when people share many values and beliefs about effective approaches, they may still disagree on a variety of matters. Thus, creating a good, evidence-based logic model is not something that can be completed in a few hours or even in a few days. It takes considerable thought, considerable discussion, and considerable attention to the relevant evidence over a more lengthy period of time. However, once you have gotten a diverse group of important stakeholders to agree on a model that will form the basis for an intervention, then you have a much stronger foundation for program development, implementation and maintenance.

Responding to a participant's query about involving various organizations in the obesity battle, Dr. Kirby noted that the BDI logic models work especially well with obesity due to the well-defined behaviors that contribute to obesity and the fact that multiple groups can play a role.

In response to a request for specific examples of how he has involved stakeholders in BDI logic models, Dr. Kirby stated that most of his work with logic models in this country has involved teen pregnancy and STDs, including HIV. Because some of the effective approaches for reducing teen pregnancy have involved programs that are broad youth development programs and not sexuality focused programs, people from a variety of sectors have participated in meetings to reduce teen pregnancy.

For example, they have involved educators from schools (some of whom were going to implement after-school programs), clergy (some of whom wanted to implement youth development or sex education programs in their churches), businessmen (who were concerned about providing greater employment opportunity to youth and thereby reducing teen pregnancy), clinicians or health educators (who were concerned about the availability of reproductive health services and sex education) and, of course, representatives of many youth-serving agencies (who were implementing youth developing programs).

Recognizing the wide variety of risk and protective factors associated with teen sexual risk-taking and the fact that different groups could address different risk and protective factors brought people together and helped them realize that each was making a different, but important, contribution to the problem.

Dr. Kirby also discussed some examples of how he used logic models in other countries. In Thailand, logic models were used to design interventions to prevent HIV transmission; in Sri Lanka, to develop models for better nutrition; in Malaysia, to develop programs to delay the onset of sexual intercourse among youth; in India, to increase family well-being; and in yet another country, to improve education. Currently, Dr. Kirby added, WHO (the World Health Organization) is developing logic models for youth development as well as health goals.

Co-moderator Lori Rolleri asked Dr. Kirby to comment on his experiences observing numerous groups develop BDI logic models throughout the United States in terms of what they have done well and what they have done poorly.

Dr. Kirby began by stating that he has been very impressed with people's ability to understand and apply the basic concepts of BDI logic models. They do identify appropriate behaviors and put them in the behaviors' column; they do identify determinants and put them in the determinants column, etc. And commonly when they are finished, they say, "This really makes sense."

However, he hastened to add that he has also seen at least five common problems. First, groups often do not think of the wide array of risk and protective factors that affect most behaviors, and they sometimes naively assume that changing just one or two risk or protective factors will markedly change the desired behavior, when, in fact, this is typically not the case. Dr. Kirby encourages program developers to think really seriously about whether or not improving their selected risk and protective factors will actually change the desired behavior.

Second, he sometimes sees people keeping on their "blinders" and simply justifying their existing programs rather than thinking systematically about the behaviors, then the determinants, and finally the interventions. That is, they identify the risk and protective factors that their existing program may impact rather than the risk and protective factors that need to be changed and which they might be able to change if they improved their program with their existing resources.

Third, he sees groups struggle while they grapple with the complexities of reality. Reality is complex, and BDI logic models are simplifications of that reality. Sometimes more complex models are needed; sometimes additional columns or nested models need to be incorporated. Knowing when and how to add complexity requires considerable experience.

Fourth, he sees many groups quickly creating initial drafts of models during his workshops. These are good first drafts but are not always followed up with better, more fully developed drafts. Often, participants do not return to their communities, review the research, conduct focus groups, involve other people, and continually improve and update their models, which is critical.

Finally, for some, but not all purposes, the final logic models need to be quite specific. Some of the examples in Dr. Kirby's paper provide that specificity, but too often groups don't get to that level of detail. He noted, however, that they simply may not have time to do so in his workshops. Despite these concerns, Dr. Kirby commented that he is very pleased with the programs that many groups have made in short periods of time.