A Guide for Education Personnel:
Selecting an Intervention or Program
"You have to drill through mud and water to get oil, you have to sift
through sand and silt to get gold, you have to chop and hack through stone to
get diamonds -
so why do so many people feel that the treasure of ideas
should come with little or no effort?"
~ Sydney J. Harris ~
Authors
Judy Shanley, Senior Research Analyst
Amanda Schwartz, Former Research Associate
Jessica Saunders, Research Assistant
The Elementary and Middle Schools Technical Assistance Center
The American Institutes for Research
1000 Thomas Jefferson Street, NW
Washington, DC 20007-3835
www.emstac.org
This publication is available in alternate format upon request.
We
would like to offer our appreciation to the individuals who reviewed the information
related to Selecting an Intervention or Program. Thank you to Laurie Baker,
a principal in Arlington, VA, and Kathy Harvey, a special educator in Seattle,
WA.
To download this document in its entirety, as a PDF file, click
here.
Choose the PDF option if you plan to print this webpage.
To download this document as a Word file, click here.
Why is this Information Important
Where to Find Information about Programs and Interventions
A Process for Deciding the Appropriateness of a Program
Questions to Guide the Selection of an Intervention
Questions related to Research
Questions related to Implementing the Intervention
Questions related to Resources
Choosing Technical Assistance Interventions and Programs Scale
Choosing Technical Assistance Interventions and Programs
Goal: To help school
professionals identify criteria through which they can 
determine
the appropriateness of an intervention for implementation within their school.
Why is this Information Important?
School professionals often hear claims from product salespeople that their products or programs will positively affect student outcomes. Product developers tout that their products are the classroom panacea, and flood education professionals with written materials and brochures. These interventions or programs may offer education professionals the hope that the program will enhance student learning, and will lead to increased student success. As you, the educator, review the vast amount of products, materials, and programs that cross your desk, it is helpful to have the information and tools that enable you to make an informed decision regarding the appropriateness, or "best fit" between an intervention and the needs of your school and setting. In an education environment, in which personnel and fiscal resources are scarce, it is important to find cost-effective, yet quality-driven solutions to enhance student learning outcomes. The inappropriateness or "lack of fit" between an intervention and the needs of a school district are costly in terms of personnel and fiscal resources, and most importantly, costly for the students for whom the intervention or program was directed.
A structured and systematic procedure can help district personnel assess the viability of a particular intervention or program and determine its appropriateness for their district. The evaluation criteria indicated in this product are not intended to be exhaustive of all of the factors that district personnel should consider when evaluating the viability of a specific intervention/program. However, these variables can serve as a template -- a foundation -- to facilitate thinking regarding the advantages or disadvantages of a particular school-based or district-wide strategy.
Before you can confidently make a decision about whether a specific intervention or program is appropriate for your setting, you must have a thorough understanding of the needs, resources, and capacity of your setting. You may want to use an established model (such as Havelock’s Change Process Model) to help you structure your efforts to implement and sustain an innovation.
| Havelock Change Process 1. Care - develop a sense of concern. 2. Relate - build relationships. 3. Examine - understand problem - establish objectives. 4. Acquire - access resources. 5. Try - move from knowledge to action. 6. Extend - gain wider acceptance of change. 7. Renew - continuous improvement. (Based upon the change agent process, Havelock, 1995) |
As indicated by Havelock's Model, the steps of caring, relating, and examining, should precede any activities related to acquiring and trying resources (Havelock, 1995). Going through the steps of developing concern, building relationships, and examining the problem will facilitate your ability to identify intervention features and characteristics that are important for your setting. Information about the needs, resources, and capacity within your setting can help you make an informed decision about the interventions and programs that you need to acquire.
Where to Find Information about Programs and Interventions
|
Informational Sources
|
There
are many resources through which you can find information about specific programs
or interventions. Some resources such as discussing programs with colleagues,
subscribing to educational Listservs, and searching the Internet, may be more
accessible than others, such as attending vendor fairs and conferences. Even
though you may not be able to attend events, you may be able to obtain a conference
program or session abstracts to learn more about the content of the meeting.
Visit the website of the conference host to learn whether resource information
is available. If you are unable to leave your site, you can communicate with
others by telephone, email and through the Internet. Some school districts,
nearby associations, and local higher education organizations may be able to
assist by offering video-conferencing capabilities. Gathering information can
be time consuming - but it can also be fun. The time you invest in gathering
information, and selecting the appropriate program or intervention for your
site will save you time once the program is implemented. Also, by evaluating
multiple programs or interventions, you get a broader sense of what is available
to you and what will best suit your needs.
A Process for Deciding the Appropriateness of a Program
Deciding
whether a particular program or intervention is appropriate for your school
or district can be a daunting task. However, the consequence of implementing
a school-wide or district-wide program or intervention that is not consistent
with your needs, culture or philosophy is even more daunting. A systematic procedure
can make this evaluation phase easier and can result in high quality outcomes
for your students:
Acquire
Information: Acquire an overview of the program/intervention in writing
from the publisher, developer, or other external resources. Ask the developer
if evaluation data are included in this overview material and whether case
studies that exemplify the use of the program/intervention are available.
If possible, acquire the names and contact information for districts that
are currently using the intervention to serve as a reference.2. Obtain References: Contact school districts
that have experience using this intervention. Use the personnel resource network
that you have developed to learn about districts that have had positive and
negative experiences with the 
intervention.
3. Observe Program: Observe the program/intervention first-hand through a site visit or through video. Keep in mind that some programs may use materials or resources that are not necessary for the characteristics of your program.
4. Analyze Information: Analyze the quantitative and qualitative evaluative data available through the program developer. Complement these data with data you obtain through independent resources such as school districts that are using the intervention, results of independent studies cited in literature, and federally funded projects and studies.
Questions to Guide the Selection of an Intervention
As you collect information about a specific program or intervention, it
is helpful to use questions to organize and prioritize the information you collect.
These questions can relate to a variety of domains including the research and
studies; implementing the intervention; and resources. We offer the following
framework to help guide your effort to organize the information that you obtain
regarding a program or intervention.
It is helpful to conduct a literature review of current research related to the intervention or program you are considering. Through an extensive search of the Educational Resources Information Center (ERIC) database, you can acquire articles and publications that reflect research conducted in the field. When reading and reviewing these materials, it may be helpful to ask yourself the following questions:
o Do I have research and data from studies that focus on evaluating the impact of the intervention or program?
o Do the findings from studies provide descriptive information and outcome data from studies that were conducted in similar settings as mine?
o Do the studies clearly define the independent variables (what is being manipulated in the study - or what the treatment is - such as differing reading programs or different kinds of instructional grouping techniques)? Are these the same variables that describe the intervention I want to implement?
o Do the studies clearly define dependent variables (outcomes or what you want to measure such as reading levels, or behavioral incidents)? Are these dependent variables the same variables that I want impacted by the intervention under consideration?
o Do the studies use large sample sizes - or was the intervention evaluated using a small number of subjects?
o Were conditions in the study similar to the ones in my setting? For instance, if a study evaluated the effectiveness of a specific reading intervention and the study was conducted in classrooms that utilized a collaborative teaching model, do I have these same conditions and resources available to me?
o Do the studies contain outcome data for students with disabilities who are similar to the students I serve? For example, did the studies include students with disabilities who are culturally or linguistically diverse or English language learners with disabilities?
Questions Related to Implementing the Intervention
Information
about the origin of a program can be useful in determining whether the intervention
has a successful history. Information about the development of the intervention
can help you determine whether the settings in which the program was developed
were similar to your setting. For instance, if a for-profit corporation developed
a character education program in a small, private school, this program may not
be a good fit for a large, diverse urban school that would require purchasing
several hundred copies of the curriculum. Primarily, the program was developed
in a setting in which the students were probably very different from the students
in your setting. Secondly, if the developer is interested in earning money (and
this is common within the education community), you should ask if you are getting
the most return for your investment? The following questions will help you think
about issues related to implementing the program:
o What are the historical origins of the intervention/program – how long has the program been around?
o Who developed the intervention or program? Do you have information regarding what circumstances surrounded its development?
o How pervasive is this intervention or program? How widely is the intervention used – in what kinds of settings does the intervention appear most suited? Are the settings in which the program is widely used similar to mine? Factors to consider include: demographics of students to which the intervention is targeted, size of school or district, resources needed to successfully implement the program.
o What kind of training and inservice is required to implement and maintain the intervention? Does the developer provide external support? What are the costs associated with this support?
o How much flexibility do we have in implementing the intervention?
Questions Related to Resources
Professional
development specialists and those persons who select interventions and programs
are concerned about the resources necessary to acquire, implement, maintain,
and evaluate the program. These questions can be considered when you are comparing
the resource needs of the program and the resource capacity of your school or
district:
o How much does the intervention or program cost? Are there costs associated with both implementation and maintenance?
o What are the personnel and staffing resource needs of the intervention/program?
o How much time (contracted or volunteers) will staff be required to give?
o Are additional materials needed to implement the program?
o If we implement this program, will resources be taken from another program or area - or will another area or school need be compromised because of the resources devoted to this program?
We
hope this product has helped you think about factors, issues and concerns related
to the selection of school or district-wide interventions and programs. With
a careful selection process in place, you and your district can balance the
needs of your site with the characteristics of a specific program or intervention.
The Choosing Technical Assistance Interventions and Programs Scale included
in this packet will help you systematically organize the information you obtain
regarding the characteristics of an intervention and can help you make judgements
about the features of programs that are important to you. The scale can also
be used to facilitate your ability to rank each intervention and product based
on its features, capabilities, and characteristics. We have also included references
in this product and resources where you can go to obtain additional information
regarding selecting technical assistance interventions and programs.
Every setting is unique and the criteria that a professional will use to determine the appropriateness of an intervention will vary. This product is not an exhaustive list of the factors that may impact upon your decision to select a program. As is the case in almost any sector, or any industry, caveat emptor or buyer beware - be an informed consumer.
Choosing Technical Assistance Interventions and Programs Scale
Name of Program/Intervention:___________________________________________________________
This scale is designed to help you organize the research you have conducted regarding a specific program or intervention and can help you use your findings to rate programs and interventions according to the characteristics that you deem important in your setting. The scale will help ensure that you have considered the elements necessary to determine whether an intervention or program fits the needs of your setting. Fill out the scale for each new intervention or program you are considering implementing within your site.
Directions: Using a scale from 1 to 5 (1 = least true, 5 = most true), rank your answers to each of the statements contained in the scale. When you have answered all of the statements, total your responses and compare the total score from each intervention or program. Interventions or programs that yield the highest score may be the best fit for your school or district.
|
Evidence of Effectiveness Based on Practice |
1 |
2 |
3 |
4 |
5 |
N/A |
|
1. The other schools or districts currently using this intervention or program are similar to my school or district. |
||||||
| 2. The other schools or districts currently using this intervention or program cited benefits that will help me solve some of the target issues I have identified. | ||||||
|
3. When I visited or viewed a video of a site using this intervention or program, I could see how it was effective in targeting the issue I am trying to solve. |
||||||
|
4. The intervention is used and recommended by professionals in the field. |
||||||
| 5. There is sufficient evidence to show that this intervention or program will be an effective means of serving the diverse population of students in my school or district. |
|
Evidence of Effectiveness based on Practice (continued) |
1 |
2 |
3 |
4 |
5 |
N/A |
| 6. The teachers and administrators I am targeting are prepared to implement this intervention or program. | ||||||
|
7. The design of this intervention or program will not alter the structure of the schools or district I am targeting in negative ways. |
||||||
|
8. The changes facilitated by this intervention or program are manageable and/or gradual.
|
||||||
|
Evidence of Effectiveness based on Research |
1 |
2 |
3 |
4 |
5 |
N/A |
|
9. The research data I used to learn more about this intervention or program included well defined independent and dependent variables, sample size, experimental controls, statistical process and methodology, and qualitative information. |
||||||
|
10. The qualitative and quantitative data concerning this project show high rates of success with populations similar to those I am targeting. |
||||||
|
11. The qualitative data concerning this project emphasized programmatic success with students similar to those in the school or district I serve. |
||||||
|
12. The methodology used provides statistically valid information.
|
||||||
|
13. The intervention or program has shown consistent success over a period of time. |
||||||
|
|
15. The developers of this intervention or program are experts in the field and have added to the knowledge base which teachers in the schools I serve have used or supported. |
||||||
|
16. The developers of this intervention or program have similar philosophies to the teachers who will be implementing it. |
||||||
|
17. Data show that this intervention or program positively affects teaching and learning in the classroom. |
||||||
|
18. This intervention or program is designed to improve student success as shown through tangible results (informal and formal assessments). |
||||||
|
Indicators related to Intervention Characteristics |
1 |
2 |
3 |
4 |
5 |
N/A |
|
19. The fiscal costs of implementing and maintaining this intervention or program are within my school’s or district’s budgetary means. |
||||||
|
20. The schools or districts I serve have enough staff/personnel to implement and maintain this intervention or program. |
||||||
|
21. My school or district will have flexibility when implementing this intervention. |
||||||
|
22. The intervention or program includes initial training for the school or district personnel who will be implementing it. |
|
23. This intervention or program has a support system I can refer to if I ever have questions or need help problem-solving issues. |
||||||
|
24. The activities and events that need to be put in place to implement the selected design are accessible and manageable. |
Beswick, Richard (1990). Evaluating Educational Programs. ERIC Digest Series Number EA 54. Available online: http://ericae.net/edo/ED324766.htm
Crawford, David C. (1995). Suggestions to Assess Nonformal Education Programs. Nonformal Education Institute. Available online: http://www.ag.ohio-state.edu/~brick/nfeeval.htm
Havelock, R. (1995). The Change Agents Guide. Englewood Cliffs, NJ: Educational Technology Publications
Scriven, Michael (1999). The Nature of Evaluation Part I: Relation to Psychology. Practical Assessment, Research, & Evaluation, 6(11). Available online: http://ericae.net/pare/getvn.asp?v=6&n=11
Scriven, Michael (1999). The Nature of Evaluation Part II: Training. Practical Assessment, Research, & Evaluation, 6(12). Available online: http://ericae.net/pare/getvn.asp?v=6&n=12
Stake, Bob (1999). Representing Quality in Evaluation. Presentation for the Annual Meeting of the American Educational Research Association in Montreal, Canada. ERIC document: ED437422.
Standish, William (1998). Some Evaluation Questions. Practical Assessment, Research & Evaluation, 6(3). Available online: http://ericae.net/pare/getvn.asp?v=6&n=3.
Web Based Resources
ERIC Digests
http://www.ed.gov/databases/ERIC_Digests/index
Guide to the United States Department of Education Programs and Resources
http://web99.ed.gov/GTEP/Program2.nsf
OSEP Sponsored Web Sites
http://www.ed.gov/offices/OSERS/OSEP/links.html
Searching the OSEP Discretionary Grant Projects Database