Program Evaluation for Organizations under CAPC · 2012-12-10 · program evaluatio fon r organizations under the community action program for children (capc) presentation of evaluation

HV 745 .Q44 T373 1998 V. 3

Program Evaluation for Organizations under CAPC

(Community Action Program for Children)

©

PRESENTATION OF

EVALUATION GUIDES

Centre de recherche sur les services communautaires

Faculté des sciences sociales

DIVERSITÉ WAL

Association des centres jeunesse du Québec

SANTÉCOM

(

PROGRAM EVALUATION FOR ORGANIZATIONS UNDER THE COMMUNITY ACTION PROGRAM FOR

CHILDREN (CAPC)

PRESENTATION OF EVALUATION GUIDES

MNM M\MI N SANTÉ PUBLIQUE DU QUÉBEC CENTRE DE DOCUMENTATION

MONTRÉAL

Patricia Dumas Hector Ouellet

André Beaudoin

with Nicole Caron

MARCH 1998

Secretarial Services: Jocelyne Gallant

English Translation: Translation Bureau

Public Works and Government Services Canada

The masculine used herein refers to both genders.

ISBN: 2-89497-017-X

These documents were produced as part of a contract with CLSC Les Blés d'Or evaluating Health Canada's Community Action Program for Children (CAPC).

The viewpoints herein are those of the authors and do not necessarily reflect the official policy of Health Canada or the Province of Quebec.

i Table of contents

Foreword i i

Introduction 1

Chapter 1 Understanding the Evaluation Process 4 Section 1: Synopsis 4 Section 2: Contents 6 Section 3: Highlights 20 Summary 25

Chapter 2 Participant-focused Evaluation 26 Section 1: Synopsis 26 Section 2: Contents 28 Section 3: Highlights 46 Summary... : 50

Chapter 3 A Guide to Evaluation for Community Groups 51 Section 1: Synopsis 51 Section 2: Contents 53 Section 3: Highlights 63 Summary 64

Conclusion 65

ii Foreword

Aspeaker who was to talk on the topic of p lanning and

who wanted to lead the listeners to have realistic

expectations about the potential of the planned action

began his presentation with the following:

"When Christopher Columbus discovered America, he

didn't know where he was going. He didn't know how to

get there. He didn't know how long the trip would take or

what resources would be needed along the way. Once he

landed, he didn't know where he was. And yet he

discovered America. "

To which someone interested in the field of eva lua t ion

could have responded:

"At least Columbus had an objective: finding India.

Having an objective enabled Columbus tô decide to

attempt the trip and to pull together the required

resources. The few navigational instruments (in

evaluation terms, these are referred to as evaluation tools)

Program Evaluation for Organizations Presentation of Evaluation Guides

I | iii

he had indicated that he was heading westward and o n

course. Despite their primitiveness, his instruments

allowed him to calculate the time lapsed since departure

and provided a rough estimation of the distance traveled.

| Furthermore, these instruments (in this case, his own

observations of the results, combined with a comparison

between what he expected to discover to what he actually

saw) indicated to him that either he hadn't reached India

or that his description of India was incorrect."

This is what evaluation is all about: a means for

stakeholders and practitioners to guide their actions.

The Community Action Program for Children (CAPC)

evaluation team set a number of objectives for itself,

including the publication of manuals to provide groups

working under the CAPC framework with support that

would correspond as closely as possible to their needs and

real potential for carrying out evaluation activities. Three


iv manuals have been produced.1 The first presents and

clarifies the main ideas, topics, and notions relating to the

concept of evaluation. This is the Introductory Manual.

The second manual aims at providing tools to

organizations that want to evaluate their performance in

order to enhance the quality of the programs that they

deliver. It provides concrete examples that illustrate

different ways of using evaluation methods and tech-

niques. It also shows how to handle a certain number of

ethical and methodological challenges often encountered

in carrying out evaluations. The title of the second manual

is Evaluation Tools for Enhancing Program Quality.

The third manual, entitled Presentation of Evaluation

Guides, offers a fairly detailed presentation of the contents

of certain "evaluation guides" that have been developed

These three manuals (in English and French versions) may be obtained from the Centre de recherche sur les services communautaires (community services research centre), whose address is provided on the back cover.


and are available. We have focused on highlighting the

aspects of these guides that seem the most relevant or

useful to organizations.

We would like to express our appreciation to the

individuals who, in one way or another, have helped in

preparing these manuals. We are particularly grateful to

the members of the CAPC advisory committee for program

evaluation, whose names and affiliations are listed below.

André Beaudoin Lyne Champoux Martine Cinq-Mars Richard Cloutier Danielle Couture/ Lucie Lafrance Anne Dubé Richard Foy Michel Gaussiran Florence Isabelle

Gisèle Laramée Hector Ouellet

CRSC, Université Laval (Québec) CRSC, Université Laval (Québec) Regroupements des projets PACE des Laurentides (Montreal) CRSC, Université Laval (Québec) Mauricie/Bois-Francs Regional Board (Trois-Rivières) La Débrouille (Rimouski) Le Pignon Bleu (Québec) CAPC (Montreal) CLSC Seigneurie Beauharnois (Valleyfield) ENAP (Québec) CRSC, Université Laval (Québec)


vi We sincerely hope that these manuals will be of use and that the organizations for which they have been published will find them of value in their efforts to develop services offered to children and parents in their communities.


1 Introduction

his manual aims specifically at providing a relatively

detailed look at the contents of three evaluation

guides that have been developed by authors with a

purpose similar to ours: helping acquaint community

groups with the evaluation process.

Each of these guides is presented separately in its own

chapter comprising three sections: synopsis, content

description, and highlights. Section 1 (synopsis) offers a

brief outline of the guide, including ordering information.

Section 2 (contents) provides a short but as accurate as

possible summary of the guide's contents. Section 3

(highlights) points out what we consider to be the most

important elements in the guide and offers a commentary

on the author(s) positions. We also draw attention to what

we deem significant shortcomings, when appropriate.

Our purpose in producing this manual is to provide

descriptions that are detailed enough to give a clear picture

Program Evaluation for Organizations t Presentation of Evaluation Guides

2 of the contents of these guides so that the reader can decide

whether or not to purchase them.

Chapter 1, entitled "Understanding the Evaluation

Process," presents How about...Evaluation!, a 1993 manual

written by Jacqueline D. Holt for First Nations and Inuit

communities.

Chapter 2 focuses on participant-focused evaluation based

on Keeping On Track - An Evaluation Guide for

Community Groups, produced by Diana Ellis, Gayla Reid,

and Jan Barseley for the Women's Research Centre. This

guide was written for community groups providing

services to victims of AIDS.

Chapter 3 presents a guide that we consider particularly

useful in carrying out an evaluation audit of an

organization. It was written by Christine Daniel in 1992 for

the Centre de formation populaire de Montréal


3 To summarize, each of these guides is presented as follows:

- a brief outline of the guide, including ordering

information;

- a description of the guide's contents;

- the most important elements in the guide.


4

Chapter 1 Understanding the Evaluation Process1

Section 1 : Synopsis

For Whom?

This manual was written for First Nations and Inuit

communities in Canada who are responsible for

implementing mental-health and child-development

programs and projects in their communities.

By Whom?

The guide was sponsored by Mental Health Advisory

Services, under the Department of National Health and

Welfare's Medical Services Branch and written by

Jacqueline D. Holt.

1 This chapter deals with How about... Evaluation! A Handbook about Project Self Evaluation for First Nations and Inuit Communities by Jacqueline D. Holt (January 1993).


5 For What Purpose?

This document aims at describing the program

evaluation process and at explaining its uses and

significance.

Structure

This 56-page document contains 11 short chapters and a

bibliography. The first four chapters introduce the topic

of evaluation; the remainder deal with selecting the

person to head up the evaluation and with questions

relating to gathering and processing information.

Where to Order Health Canada Health Promotion and Social Development Office Complexe Guy-Favreau 200 René-Lévesque Boulevard West Montréal, QUÉBEC H2Z1X4 Telephone: (514) 283-6533 Fax: (514) 283-3309


6

Section 2 : Contents

Some Definitions

This guide starts off with a glossary of some frequently

used evaluation terms (Chapter 1; pages 1 to 3). Four

terms of special interest to us in this glossary are "hard

data/' "soft data," "open questions/' and "closed

questions."

"Hard data" are described as being "measurable and

tangible data, usually numbers," while "soft data" are

defined as "opinion and attitude about something."

They are said to be qualitative. An "open question" is

defined as one "that can be answered with description

and opinion, using any words you want." A "closed

question" is described as one "that can be answered

with only one response, like yes or no."

The guide continues with an overview of evaluation

(Chapter 2; pages 5 to 10), starting with the author's


7 perspective of evaluation, which she describes as "as

way of measuring if a project is doing what it says it will

do."

Why We Do Evaluations

This chapter discusses four reasons for carrying out

evaluation. These reasons are described in the

following terms:

1- to help you understand how your project is working; 2- to show your sponsors what you are doing; 3- to show other communities what has worked for

you; 4- to show your staff what they are doing.

The Role of Evaluation

Holt then describes where evaluation fits in the

operating cycle of a program, stating that "evaluation,

ideally, is an ongoing activity." In other words, it

should be initiated when the program is being planned,

continue throughout the life of the program, and then

shed light on how to make improvements with respect


8 to planning and implementation or to be a source of

ideas for new programs and projects.

Criteria for Quality in Evaluation

This chapter closes by listing the required criteria to

ensure the quality of any evaluation:

- The evaluation must be useful to the people who request the information and to the people in the program.

- It must be practical to implement. - It must be conducted in an ethical manner. - It must be technically accurate.

The Question of Objectives

The author next tackles the topic of program objectives

(Chapter 3; pages 11 to 14). Objectives are presented as

the main component of an evaluation since they

describe what the program is supposed to accomplish.

Objectives should be formulated according to the

following criteria.


- They must specifically state the expected results. - They must state the expected results in measurable

terms. - They must identify when the results will happen.

This chapter also gives three examples of how to

formulate objectives as outlined below.

- A program entitled "Life Skills for Teenagers" could

have an objective formulated as "the involvement

of teenager girls between the age of 15 and 19 in the

life skills program will increase by 50% in the next

two years."

- A program entitled "Aftercare for Psychiatric

Patients" could have an objective formulated as "re-

admissions to psychiatric institutions will drop by

30% in the next year."

- A program entitled "Alcohol Abuse" could have an

objective formulated as "establish an alcohol abuse

counselling program to handle 50 people annually

in the next three years."


10 Types of Evaluation

In Chapter 4 (pages 15 to 20), the author discusses three

types of program evaluation that can be used:

- needs assessment;

- process evaluation; and

- impact evaluation2.

This chapter points out that selecting one type of

evaluation over another depends on the stage of life of

the program. For example, needs assessment would be

appropriate if the program were in planning, whereas

process evaluation would apply to an ongoing program.

Impact evaluation is appropriate for focussing on the

products or results of the program.

Holt briefly describes how each of these types of

evaluation is useful. For example, she indicates that

needs assessment helps get an idea of the kinds of

problems in the community; which segments of the

2 This notion seems to cover what other authors call "impact" or "effects".


11 population have the most urgent needs; and how the

needs can be met. With respect to process evaluation,

Holt describes it as the type of evaluation that shows

how the program could be improved; how the program

could help a greater number of people; what kind of

human resources are needed; and how program costs

can be reduced. Impact evaluation is considered as the

one of the three that can be used to determine if the

program was effectively implemented; what impact it

had on participants; if it should be continued; how

much it cost; and how it compares with similar

programs.

Evaluation Approaches

The last part of this chapter discusses two evaluation

approaches: quantitative and qualitative.

Holt defines the quantitative approach as trying "to

determine the cause and effect relationships in a pro-

gram." In her opinion, this approach uses "measu-

rements, numbers, and statistics to compare program


12 results." The information obtained can therefore be

considered hard data.

As for the qualitative approach, Holt views it as

examining "the qualities of a program." It serves "to try

to understand the meaning of a program and its

outcomes from the view of people." The information

that is found is considered soft data.

Holt claims that the qualitative approach is most

appropriate for evaluating mental-health and child-

development programs because:

- it uses a number of different methods to gather information;

- it looks at the program as a whole; - it calls on program participants, staff, and commu-

nity members as sources of information; - it is dynamic, interactive, and exciting; - it leaves room for creativity; and - it doesn't require a rigid design.


13 Planning an Evaluation

• Clarify program objectives

• Select the type of evaluation

• Select an evaluator

Chapter 4 winds up with a reference to a checklist to

help in evaluation planning. The list starts off with the

notions dealt with above, indicating that the planning

phase must begin by clarifying program objectives and

selecting the appropriate type of evaluation.

In Chapter 5 (pages 21 to 24), Holt gives her viewpoint

on how to choose an evaluator. Although the

concerned group or organization is responsible for the

evaluation, they may delegate responsibility to others,

in particular, peers and consultants. The choice depends

on the stage of the program and the type of evaluation.

To illustrate, needs assessment, which is carried out

during program planning, could be performed by

someone who is thoroughly familiar with the

community.


14 Process evaluation, which occurs while the program is

ongoing, could be conducted by someone thoroughly

familiar with the program, such as a program manager,

a staff member, a participant, a community elder, or

even an outside consultant acting as an advisor.

Holt states that impact evaluation should be carried out

by an individual that has no involvement whatsoever

in the program. The evaluator could be from another

program, a member of another community, and so on.

In Chapter 6 (pages 27 to 36), Holt describes the

information collection plan, which she suggests should

be based on the type of information sought and the

resources available.

• Determine the Type of Information Required

Holt points to three types of information:

- input (resources);

- processes (operations);

- outcomes (impact).


15 • Determine the Sources of Information Available

Holt describes information sources as belonging to

three main types:

- files, documents, and information systems; - interviews with participants, their families, and

staff; - observations made during the program.

This chapter ends with a table (information sources

chart) designed to help in developing the information

collection plan. The table is divided into three columns

for the sources of information; the rows are for the

types of information. Holt provides questions in each

table cell that a group should ask itself when carrying

out an evaluation. She also suggests using the table to

organize and analyze data.

Carrying Out the Evaluation

• Collecting Information

Holt deals with collecting information in Chapter 7

(pages 37 to 44), where she describes how to gather


16 information from each of the sources and raises other

important considerations. For example, in collecting

information from information sources and files, Holt

recommends finding out what kind of information has

been recorded and how often, then refer to the type of

information required in order to determine if the data

might be pertinent.

As for interview material, Holt indicates that it can be

obtained in a variety of ways, such as face-to-face

interviews, telephone interviews, or self-reporting

questionnaires. She points out that the questions

should be simple and touch on a single subject at a

time. Holt also states that a test should be carried out

before actually using a written questionnaire.

Holt's views on observations is that they should be

used once the files have been examined and the

interviews conducted because the information can be

used as a foundation for observations. She feels that it

is not necessary to draw up a checklist or interview


guide. Holt thinks that observing, feeling, and listening

to what is going on is more important. On the other

hand, she emphasizes that notes should be taken as

soon as the observations have been made.

• Interpreting the Information

In Chapter 8 (pages 45 to 50), Holt deals with

interpreting information. She suggests basing the

interpretation on the information sources chart o n

pages 32-33 and working along step-by-step. The process

begins by examining the information collected from

each of the sources and asking what it reveals about the

inputs, processes, or outcomes. She recommends

preparing a description of what was discovered for each

type of information. For example, with respect to

inputs, you might ask if the program is being run by

experienced, qualified staff in a suitable setting. If so,

why? If not, why not? Analysis of outcomes could

respond to questions such as: Do participants benefit

from the program? If so, why? If not, why not?


18 According to Holt, the most important step is giving

meaning to all the information gathered, which means

drawing conclusions. She recommends using the

information sources chart to formulate conclusions for

each of the sections by asking yourself if the numbers

point to a trend, if there is a pattern or theme in

activities, and so on. Holt emphasizes looking for a

logical chain of evidence. For example, do the

documents support what is happening in the program?

• Writing the Report

Holt then proposes writing down your ideas and

discussing them with the manager and program staff.

• Discussing the Results

Chapter 9 (pages 51 and 52) provides an overview of the

process with a checklist itemizing the main tasks to be

carried out during the evaluation.


19 A Word on Confidentiality

The guide ends with two notes: one is an evaluation

checklist; the other deals briefly with the topic of

confidentiality.

Other References

In Chapter 11, Holt refers the reader to The Evaluator's

Handbook by Joan Herman, Lynn Lyons Morris, and

Carol Fitz-Gibbon for further reading and reminds us

that Sage Publications puts out a number of books and

handbooks on the topic.


20

Section 3 : Highlights

Originality

This guide familiarizes the reader with the overall

evaluation process. Its most original feature, however,

doesn't lie with its description of the process itself, but

rather in its approach to accountability for the

evaluation. Although her design is not very elaborate,

it does have a certain number of features that sets it

apart from conventional designs.

To begin with, Holt clearly states that the organization

is responsible for initiating the evaluation of its

programs and has the freedom to designate whoever it

wishes to implement the evaluation. From this

viewpoint, the evaluation vests fully in the

organization. It is not imposed by an outside agency, but

emerges as an activity that promotes "food for thought"

for the organization regarding the orientation and

improvement of its programs.


21 • Selecting an Evaluator

Unlike what you might expect, the choice of an

evaluator is not limited to experts or specialists in the

field. Holt states that the candidate could come from

individuals involved in the program, including

participants, staff members, and managers as well as

peers from other programs. In this regard, a peer is

someone that carries out similar functions in a similar

program.

The idea of turning to peers can be very attractive and

should be explored. As Holt claims, it provides an

opportunity for someone from outside the program but

who is very familiar with its problems to critically

examine it.

Recourse to peers or to individuals directly involved in

the program could also be paired with using experts,

referred to as consultants, because they play the relative

discrete role of advisor. While the use of the term


22 consultant isn't new, it seems particularly apt to us in

this context.

We are aware that, for reasons of reliability, validity,

and objectivity, experts usually tend to control the

evaluation, often nearly absolutely. In Holt's context,

having the expert as advisor to the individual actually

carrying out the evaluation indicates that the expert's

role is not controlling the evaluation, but rather acting

as a partner who helps maintain the conditions

required for structuring a valid, productive evaluation.

• On the other hand, we should point out some points

that we consider debatable. First of all, unlike Holt, we

feel that the qualitative and quantitative approaches

must be seen as complementary.

• We also feel that her planning and implementation

phases are inadequately documented to really be of help

to a novicè. So, while he may see that objectives are the

main component of an evaluation, he won't find much


23 help on how they should be formulated. Furthermore,

there's scant information on how to use objectives in

subsequent steps, such as data interpretation.

• Along the same lines, there is little information

regarding the choosing and designing data collection

techniques.

Moreover, there seems to be some confusion

concerning the terms "sources of information" and

"gathering tools." It would be useful to remind the

reader that there are three main sources of information:

documents, observation by the researcher, and

information provided by the subjects. There are,

however, many information gathering tools. To

illustrate, information from subjects can be obtained by

informal interview, guided interview, questionnaire,

and so on.


24 • In concluding these remarks, we should point out that

the explanations on how to analyse the information are

too sparse to actually guide anyone through the process.

Our advice to the reader, then, is to turn to other

resources to help in this phase of the evaluation

process.


25

Summary

Despite Holt's limited treatment of certain evaluation

phases, this handbook offers a new way of looking at the

process that puts the organization at the centre of the

evaluation process, which is certainly an original

contribution.


26 Chapter 2

Participant-focused Evaluation3


For Whom?

This manual was written for non-profit community

groups with a social-change orientation.

By Whom?

The guide was supported by funding from Health and

Welfare Canada under the National Welfare Grants

Program and the Federal Centre for AIDS.

It's an adaptation of an older handbook by Jan Barnsley,

Diana Ellis, and Helga Jacobson (The Women's

Research Centre, Vancouver, BC).

3 This chapter presents Keeping On Track - An Evaluation Guide for Community Groups by Diana Ellis, Gayla Reid, and Jan Barnsley (The Women's Research Centre, 1990).


27 For What Purpose?

The purpose of this guide is to present an approach to

evaluation that will make it possible for groups to face

evaluation with information and confidence, as

respected participants in the evaluation process. The

authors refer to their approach as being participant

focused, meaning that it's the group that decides what

questions to ask, what criteria to choose, and so on.

Structure

The guide breaks down into six chapters and three

appendices, covering a total of 79 pages. It discusses how

evaluation can be useful, how to design an evaluation,

and how to select an approach, in addition to

presenting data collection, analysis, and use.

Where to Order

Women's Research Centre 101-2245 West Broadway Vancouver, BC Canada V6K2E4


28


The Usefulness of Evaluation

This guide starts off by presenting why the authors feel

that evaluation is useful.

- It helps to see what's really involved in the work the

organization does.

- It provides a chance to step back from the day-to-day

work and think about what you're doing and why.

- It helps you figure out how well you're doing what

you set out to do.

- It helps you understand why you're having a

problem and how to solve it or live with it.

- It shows how you might improve your program.

- It can give you ideas and information for planning

future work.

The authors state that evaluation must be adapted to

the group's needs and interests if it is to be useful. This

is why they propose an approach referred to as

participant-focused evaluation.


29 Participant-focused Evaluation

This approach is based on the belief that evaluation

must meet the conditions listed below in order to suit

the needs and interests of the community group.

- It must take into account the nature of the group,

which means an organization that generally

operates on a shoestring budget, has flexible

structures, and an orientation that must constantly

shift to meet the needs of the target population.

- Evaluation must be based on questions that the

group asks itself about its work.

- It must use criteria that the group decides on with

respect to the work to be evaluated.

- It must involve the group in analysing the

information collected throughout the process so that

the group can develop the means to improve its

work during the evaluation.


30 After setting the stage in the first chapter of the guide,

the authors move on to illustrating how to carry out

the various phases involved in participant-focused

evaluation.

Preparing for Evaluation

Chapter 2 (pages 9 to 13) describes how to apply this

approach to the first phase of evaluation, which is

preparation. This is the phase that allows the group to

determine its evaluation needs and expectations, and to

make sure the evaluation is useful to the group.

• Make a List of Questions

The first step is for the group to come up with

questions it wants answered about its own work, such

as whether you are really doing what you set out to do

or you may want to know which parts of your program

are the most useful and why.


31 • Getting Answers to Your Questions

The next step in preparation is to determine how the

group is going to get the answers to its questions. This

begins by examining the ways that the group keeps

track of information (logbooks, minutes, forms, and so

on) and then determining how it can go about getting

the additional information needed.

• Dealing with the Information

The third step is to decide if you need help in collecting

and analysing the information required for the

evaluation. You may have the necessary resources,

want to add someone from inside or outside the group,

or decide to collect less data.

• Decide Who Should Be Involved

The fourth step in preparing for evaluation is to

determine w h o should be involved. In this regard, the

authors insist that there should be a feedback

mechanism to ensure that anyone who should have

input does have input. The authors suggest that one


32 solution can be forming an evaluation committee

mandated to meet regularly with the evaluator:

to raise and discuss issues related to the preparation

and use of evaluation tools for program activities;

- to monitor the entire evaluation process and make any

necessary modifications;

- to review the evaluation data and conclusions, and

discuss what they mean for the program's work;

- to act as liaison between the group and the evaluator.

• Draft a Preliminary Budget

The last step in preparation consists in estimating how

much the evaluation will cost. The authors stress that

you should take pains to estimate the costs for all the

tasks that might be involved in the evaluation,

including staff time, overtime related to data collection

and analysis, photocopying, stationery, resource

materials, and the like. The budget estimate can be used

to apply for funding and in negotiating with outside

evaluators, if necessary.


33 Chapter 2 ends with a table that summarizes means for

organizing the preparation phase.

Negotiating with the Funder

Chapter 3 (pages 14 to 19) looks at issues pertaining to

negotiating the evaluation method with the funder

under the title "Negotiating for an Evaluation

Method."

• Identify What the Funder Wants and Needs

The authors start off the chapter with a reminder that

most funding agencies expect to receive a report that

provides an accounting of what happened to the funds.

Since funders and groups don't necessarily have the

same needs and expectations of the evaluation and

therefore won't necessarily view the evaluation

procedures in the same way, it is very important that

the group identify what the funder expects from the

evaluation. To simplify, the authors provide a list of

questions concerning why the funder wants the


34 evaluation, what the funder wants to evaluate, the

criteria or indicators that the evaluator deems

important, the use the funder will make of the

evaluation results, and so on.

• Comparing the Funder's Needs and Expectations to

those of the Group

Once you have a clear idea of the funder's needs and

expectations, you should compare them with your own

to make sure that they are compatible. If there are

differences, the authors suggest that you write down

how you see things and send it to the funder's

representative, asking for clarification, if necessary.

• . Get It in Writing

The authors emphasize that the best way to protect

everyone's interests is to put all agreements into

writing for reference during the evaluation.


35 • Planning the Evaluation

In Chapter 4 (pages 20 to 40), the authors offer a fairly

detailed presentation of the second phase in the

evaluation process: developing an evaluation design.

Their approach has seven steps.

- Clarify the Evaluation Focus and Assumptions of

Success

The first step in developing an evaluation design is

to clarify the evaluation focus and assumptions of

success. Evaluation focus refers to the program

targets and objectives that the evaluation will focus

on. For example, is the aim to change existing

legislation or policy, create awareness in a segment

of the population, change attitudes or values, or

improve skills?

The authors recommend considering questions such

as those below in determining the assumptions of

success for evaluating your program. What does

success look like to you? What would different


36 degrees of success look like? Is it success only if the

desired changes are brought about?

- Review the Context

The second step consists in reviewing the

geographical, social, and economic context or

environment of the work. Given that the context

plays a major role in the program's success or

failure, the authors feel it important to draw up a

list of the conditions specific to each situation. To

make this process easier, they ask you to consider a

certain number of general factors, which include

accessibility issues, cost of existing services, media

access, and the like.

- Finalize Evaluation Goals and Objectives

Once the group has clarified the focus, assumptions

for success, and the context of the work, it has all the

information needed to finalize the evaluation goals

and objectives. This is the third step in developing

the evaluation design.


37 The authors stress that program goals and objectives

differ from evaluation goals and objectives. They

define program goals and objectives as specific,

measurable targets of what is to be accomplished by a

given point in time, while evaluation goals and

objectives are presented as being directly related to

the ends targeted by the evaluation, that is,

determining, describing, and measuring the

program and its outcomes. The authors illustrate

this with a series of examples.

One of the examples provided in this regard deals

with an advocacy group working on discrimination

against people with AIDS/HIV. One of the goals of

this program could be to develop advocacy skills of

group members. The corresponding objective could

be to produce a how-to kit on advocacy skills. The

evaluation goal could be to examine the

effectiveness and impact of the program i n

developing advocacy skills in your community,

while the objectives could be to provide feedback to


38 program workers about the program's effectiveness

and to assess how well the advocacy kit suited

members' needs.

- Evaluation Instruments

Choosing the evaluation instruments is the fourth

step. The authors remind us that the selection of

evaluation instruments is influenced by the type of

information needed for evaluation, the skills

required for collecting and analysing the

information, and the time and resources available to

the group. They then proceed to identify some tools

(logbooks, journals, files, questionnaires, interviews,

and the like) and how to use them.

- Developing a Work Plan

- Deciding What to Do with the Evaluation Results

- Reaching Agreement on Evaluation Results

The work continues with developing a work plan, a

discussion on what to do with the evaluation


39 results, and the need to get the group's approval on

the design. The authors suggest using flip-chart

paper to develop a work plan table. The table would

have four columns with the following titles:

objectives, tasks, how long will it take?, and dates for

accomplishment.

Once you've gotten through these steps, you can

move on to using the evaluation plan.

Carrying Out the Evaluation

• Properly Using Evaluation Instruments

Chapter 5 (pages 41 to 47) discusses using the

evaluation plan, starting with a few of the evaluator's

responsibilities during data collection. They point to

providing any necessary training on how to use the

instruments and ensuring that the tools are being used

properly.

• Compiling and Analysing Data


40 • Discussing Conclusions

The authors provide some direction on how to go

about compiling and analysing the data once it has been

collected. They suggest photocopying and coding (or

numbering) all the originals in order to be able to trace

who said what. Then, read the data, looking for

patterns in the responses, separating data by themes or

groups of questions, and thinking about any possible

conclusions. This should wind up with a discussion of

the conclusions involving everyone concerned.

This chapter also emphasizes the importance of

analysing and discussing the data throughout the entire

evaluation process -not just at the end. The authors

also state that the people doing the work must be

involved in the discussion as well. Discussion

throughout the process will ensure that the evaluation

is as useful and meaningful as it can be for the

program.


41 • Writing the Report

The authors next touch on reporting the evaluation's

results or findings, particularly the written report. They

offer an outline that can be used as a model in writing

the evaluation report, pointing out that the report

contents will depend on:

- who the report is for;

- what you want to tell them;

- what they need to know.

Using the Evaluation Findings

At the close of the chapter, the authors point out that

evaluation doesn't end with writing the report, since

the findings are supposed to be useful and used.

Accordingly, they present a number of ways in which

the results can be used. For example, the results can

help the group's planning on issues and actions or to

prove the group's worth to funders and the

community.


42 Self-evaluation

Chapter 6 (pages 48 to 56) walks you through the steps

in a self-evaluation, which is presented as a dress

rehearsal for a possible outside evaluation. The means

suggested by the authors is a workshop in which eight

topics are reviewed:

1- group history;

2- goals and objectives;

3- the context in which the group operates;

4- description of the work;

5- external relations;

6- achievements;

7- internal organization and structure;

8- implications for future work.

The authors map out how to work through each of

these topics, beginning with group history. The purpose

is to sketch the group's history based on participant

experience and perspectives. Individuals are asked to

reflect on how they got involved initially, on the work


43 they wanted to do in the group, and what was most

important to them about what the group was doing.

Then they are asked to consider how the group was

created, what were the first actions/activities the group

undertook, and what were the most significant

influences on the group.

The authors suggest pulling together all pertinent

records and documents when discussing goals and

objectives. After writing the goals and objectives on a

flip chart, you can ask each individual what they mean

for him.

The context can be examined by describing the factors or

aspects of the environment that most directly affect the

group's activities (e.g., allies in the community,

funding sources, cultural situation, etc.) and taking

note of the positive/negative impact that each has on

the group's work.


44 As for the description of the work, the authors suggest

initiating a group discussion on the different areas of

activity and asking participants to list the jobs that they

do in each area. In reviewing the lists, the group will be

able to see what gets done and what doesn't. This

process brings out both successes and shortcomings.

To examine external relations, the authors recommend

drawing up a list of the individuals and organizations

the group has already worked or dealt with. Then

analyse their influence on your group's work.

In looking at achievements, you should list what

you've achieved and what was accomplished with each

achievement. Then look back over your original

objectives to see. any changes in direction or gaps that

have become apparent.

The discussion on internal organization and structure

should bring out how the group is making decisions

and how communication within the group is actually

taking place.


45 Implications for future work is the step in which you

pull everything together that you've learned from the

self-evaluation process. You need to ask yourself if

you've done what you wanted to do, what the self-

evaluation says about future work, and what direction

should the group take from where you are.

The authors suggest setting aside an evening and a full

day to carry out the self-evaluation. In addition, they

state that, in most cases, pulling in an outside facilitator

is a good idea, especially one who is familiar with and

supports the group, although not an active member of

it.

Background Materials

Three appendices round out the guide, including a

short glossary, examples of data-gathering tools, and a

sample evaluation form for collecting numerical

information from a questionnaire or interview.


46

Section 3 : Highlights

Summary

This guide represents a major effort to make

evaluation a more democratic process. It provides a

wealth of information that can help groups come to

grips with their own evaluation, which is presented as

a process that is for the individuals directly involved in

designing and carrying out the work as well as those

the work is aimed at.

• Noteworthy Features

Two items pertaining to involving the main people

concerned by the process particularly attracted our

attention: information about preparing for an

evaluation and information about negotiating with

funders. Recognizing them as phases with high

potential for influencing the remainder of the process,

the authors provide a number of guidelines that can be

very useful in focussing an evaluation on group needs

and expectations.


47 With respect to preparation, the five steps suggested

help the group to clarify its and organise its ideas about

the questions that it is asking itself about the work, the

resources needed to collect and analyse data, and so on.

These steps can be quite valuable in helping the group

design an evaluation that responds to the group's

context and in helping the group get ready for

negotiations with funders.

We feel that the main contribution of this guide,

however, is the chapter on negotiating. What's more,

we find it rather exceptional that an evaluation guide

discusses the issue. At a time when funding agencies

are increasingly requiring community groups to

evaluate their work, this section should be welcomed

by groups.

The issues that the authors recommend bringing up

with funders are clear, specific, and undoubtedly

founded on concrete experiences. These questions focus

mainly on the funder's perspective and expectations,


48 such as the kind of data sought and how the funder

gauges success. We believe that these questions can

serve as significant references points for the group in

obtaining the information that it needs to compare the

funder's needs and expectations with its own.

There are also suggestions for carrying out effective

negotiations with the funder and for protecting the

rights of all parties, especially the group's. It is a shame,

however, that the authors didn't see their way clear to

provide a sample or model protocol of agreement,

which would have been extremely useful and would

have made the guide that much more complete.

We should also draw attention to the importance the

authors place on having everyone involved in the

group's work participate in the discussions on results

analysis. In their opinion, this is how to maximize

usefulness to the group.


49 The authors feel strongly about using the results, a

point that is often ignored in the evaluation process.

The suggestions they make are practical and highlight

the various uses for evaluation results.

This guide does have its share of shortcomings,

however. The chapter dealing with developing the

evaluation design sketches a fairly good picture of the

various steps normally involved in this phase, yet

some of the elements linked to their functioning need

to be more clear and specific. The concept of objective is

not very well-stated, either. Lasdy, if simplicity is a

concern in selecting instruments for measuring the

achievement of objectives, as the authors seem to feel,

then they need to pay more attention to the steps

involved in preparing the evaluation and using the

recommended tools. The guidelines and examples they

provide are useful but inadequate.


50 Along similar lines, we think that more attention

should have been given to considering requirements in

establishing the linkages between the work carried out

and the observed outcomes.

Summary

Overall, this guide offers a clear view of the concerns,

constraints, and problems facing groups who want to carry

out an evaluation or who are required to do so. Keeping

On Track is a major tool for helping groups develop an

evaluation design that really suits them. It is rich in ideas,

means, techniques, and approaches that can help groups

make evaluation a constructive exercise that adheres to the

perspectives and interests of the group. In this regard, it

succeeds quite well.


51

Chapter 3 A Guide to Evaluation for Community Groups'


For Whom?

This manual was written as a teaching aid to help

groups produce an evaluation audit of their operations,

activities, objectives, target clientele, and financial

situation.

By Whom?

The guide was produced by the Centre de formation

populaire de Montréal.

Structure

The guide is 36 pages long, divided into four chapters,

five appendices, and a short bibliography. The first

4 This chapter presents L'évaluation d'un organisme communautaire et social by Christine Daniel (Centre de formation populaire, 1990), which is available only in French.


52 three chapters focus on preparing for and conducting a

team meeting to carry out an evaluation audit. The

approach involves creating group dynamics that would

make it possible to adjust group objectives and

activities on an ongoing basis. The fourth chapter deals

with presenting an annual report to the general

assembly and with defining new orientations arising

from the report and the related operational planning.

Where to Order

Centre de formation populaire 3575, boulevard St-Laurent, bureau 406 Montréal (Québec) H2X2T7


53


The Role of Evaluation

The author starts by reminding us that evaluation plays

a very important role in the group. Although it's not

easy to evaluate a community group, evaluation

provides the group with an excellent opportunity to

discuss and ask itself questions about what it has done

and what its implications for future work are.

A Collective Process

The author sees evaluation as being a collective process

that must be properly prepared and led. In this light,

she outlines a process that can guide a group through

evaluation. Daniel identifies and expands on various

activities that can be used to cany out an evaluation

audit during a team meeting.

Setting Up the Evaluation Meeting

To set up the evaluation meeting, the author suggests

answering the following questions:


54 - Why evaluate?

- What do you want to evaluate?

- What approach should be used?

- What information sources can be used for

background?

- Who will lead (facilitate) the evaluation meeting?

- How much time will be required for the meeting?

- When and where should the meeting be held?

Why Evaluate?

The first question deals with evaluation usefulness.

The author states that evaluation shouldn't be

undertaken without determining, from the outset, why

you want to evaluate. Daniel points out that evaluation

must first and foremost allow the group to improve its

work, measure its achievements, better define its

objectives, and examine its shortcomings. In addition,

it must serve to identify and promote the need for the

organization, its relevance, and the quality of its work.


55 Specifying the Evaluation Focus

The second question refers to choosing the focus of the

evaluation. In addition to the reasons that naturally

come to mind, the author points to the organization's

objectives, its democratic functioning, financial

situation, target clientele, management of human

resources, work organization, teamwork, and so on.

Since there are many aspects that could be evaluated,

Daniel thinks that the group should determine, from

the outset, which it wants to look at.

Specifying the Evaluation Approach

With respect to the evaluation approach, the author

mentions that, regardless of the activity evaluated, you

need to pay particular attention to several main

considerations: program effectiveness, efficiency,

impact, and relevance. She provides a number of

questions relating to each topic that help define them

in her terms. To illustrate, Daniel suggests examining

the relevance of the activities by comparing them to the

group's mission and accounting for any discrepancies.


56 Identifying Instruments

The author lists five types of data-collection tools;

descriptions and examples of each are provided. Daniel

groups them into categories: quantitative instruments,

qualitative instruments, documents relating to group

objectives, documents relating to organization

operation, and synthesis tools. Quantitative

instruments include statistics, questionnaires, surveys,

financial statements, and so on. Qualitative

instruments take in activity reports, press clippings,

questionnaires, surveys, participant evaluation of

activities, and the like.

Extracting Pertinent Information

Daniel states that these instruments will be of little

value if they are just left on the table during the

evaluation meeting. Consequently, she proposes that

you start by pulling out just the information relevant

to the annual report using her synthesis tool, which

allows you to organize the main information required

for the evaluation. This tool consists of a table with


57 four columns —objectives, activities carried out,

deadlines, and individual responsible— that lets you

rapidly see the program's main elements.

Choosing a Facilitator

Daniel feels that the individual responsible for heading

up the evaluation should be very familiar with what

the role entails and have the following personal

characteristics: have a thorough understanding of the

evaluation process, be capable of interpreting

information, and be accepted by all participants.

Setting the Schedule

Concerning the amount of time required for the

meeting, the author notes that you should allow at

least a minimum amount of time, which, of course,

depends on the group's particular situation. In many

cases, two days appears to be ideal to carry out a fairly

comprehensive evaluation.


58 When and Where

The author then draws attention to the fact that,

although the remaining questions relate less to content,

they are none the less important. Daniel points out that

you should wind up the preparation by selecting a

location and time that will be let the participants focus

as freely as possible on the topics under discussion.

Evaluation: An Exercise related to Defining Objectives and Planning

In broaching the second phase of the evaluation process

(implementation), the author reminds us that

evaluation is a dynamic, ongoing process in the life of a

program or organization. It can be used to periodically

re-evaluate the organization's mission and adjust its

objectives and the means used to achieve them.

Daniel provides an example of how a group advocating

for the rights of the unemployed operates. She describes

their general objectives (e.g., improve the living

conditions of the unemployed; inform the unemployed


59 of all amendments to legislation affecting their status);

its specific objectives (e.g., study and analyse

unemployment reform in Canada; highlight the stakes

under the reform); the means used (e.g., information

meetings; newsletter); and the outcomes (e.g.,

significant participation in activities; MP who is very

friendly but not very supportive). She then gives a

series of questions to ask yourselves during evaluation

(e.g., What does the group want to know? Has the

group achieved its objectives?) and examples of how to

redefine objectives (e.g., call attention to new reforms

affecting the unemployed; denouncing the reforms as

regressive and highlighting positive points).

Conducting the Evaluation Meeting

• Remind Everyone of the Objectives

The author then moves on (Chapter 3, pages 21 to 26) to

the evaluation meeting and provides an outline of

how to conduct it. The agenda is fairly simple: it

consists of discussing the objectives already established


60 one by one. Of course, the information yielded by the

various instruments go into the discussion. Daniel

suggests closing the discussion with a summary of the

main points retained for each objective.

• Discussing Operations

The next step in Daniel's process is to examine team

functioning and life by discussing the relations between

the various of participants in the group, the organi-

zation and climate of work, the motivation and

satisfaction of group participants, and so on.

Evaluating the Meeting

Daniel suggests that you end the meeting with the

highlights of what you have learned from the eva-

luation experience, both individually and collectively.

The Evaluation Report

The author emphasizes that meeting notes or minutes

are essential. She recommends using a table divided

into two columns. The first column is for comments,


61 opinions, statistics, and the like relating to objective

evaluation. The second column is for anything

pertaining to perspectives.

Future Implications

Once the report has been written, you go on to the third

phase in the process: orientation and planning.

Daniel reminds us (Chapter 4) that planning is carried

out during the first phase of a new cycle for the

organization because the evaluation conclusions are

needed to decide whether to formulate new

orientations or stay the course with existing ones. She

points out that most groups handle this in a general

assembly. Consequently, she recommends that the

evaluation report should be present to the general

assembly, starting with each objective, then indicating

what was retained during the evaluation meeting, and

finally reporting on the suggestions for future work.


62 Once this has been completed, you need to determine

more specific objectives and plan how they will be

achieved. The author indicates that planning is usually

carried out by a working group, the board of directors,

or both together.


63

| Section 3 : Highlights |

We think that this guide is important for two reasons: it

reminds us that an evaluation audit can be used for the

ongoing adjustment of an organization's objectives and

activities; and it sheds light on the major phases in

conducting this type of evaluation.

By fitting evaluation into the life cycle of a program or

project, Daniel reminds us that evaluation is not an end in

itself, but rather a means for examining the group's work

and for improving it. She presents the concept of

evaluation in a much larger perspective than a simple

routine review and sets it apart from management control,

which makes organizations give an accounting to funders.

We feel that these are exciting and practical aspects for

organizations.

The evaluation process itself is presented in a clear but

rather sparse manner. The main phases for carrying out a

good evaluation are discussed and practical

implementation guidelines provided.


64

Summary

The process presented in this third guide aims at helping

groups carry out a regular, systematic audit of how their

operations, activities, and objectives: the process proposed

by Daniel focuses on the internal operations of the

organization and targets the regular adjustment of

objectives and activities that promote objective

achievement.


65

Conclusion

Our purpose in writing this manual has been to

present a relatively detailed critique of the contents

of three evaluation guides designed to help community

groups get better acquainted with evaluation. The first

guide describes the evaluation process. The second

presents the concept of participant-focused evaluation. The

third describes a method for organizations that we have

termed evaluation audit.

In dealing with each of these guides, we have focused on

highlighting the elements that we feel are most interesting

or useful for organizations.

The first guide provides readers with an opportunity to

develop a clearer understanding of the overall evaluation

process. Its originality lies with its view of who takes

charge of the evaluation process.


66 The second guide describes participant-focused evaluation,

whose objective is to provide groups with an approach that

empowers them in conducting their evaluation with all

the confidence and information required.

We were especially drawn to the importance given to

negotiations with sponsors with respect to evaluation: it

should be carried out so as to ensure that the evaluation

remains an exercise focused on group needs and

expectations.

We feel that the third guide is particularly useful in

carrying out an evaluation audit. This guide presents

evaluation as a collective process and offers an exceptional

opportunity to reflect on the group's work, to examine the

work that has been done, and to retarget future action.

The main idea that sets these guides apart is focus on

adopting approaches that empower groups in carrying out

evaluation. This empowerment includes defining the

goals and criteria that will serve as reference points in


67 evaluating the program, project, activity, or organization,

and then selecting the design best suited to the stated

needs. Empowerment also refers to analysing and

interpreting the results, even if outside expertise is

required, and using the results to improve the quality of

the work.


0 13,067 Vol .3

E-1974

Tard, Caroline Ouel le t , Hector e t a l .

Program Evaluation for Organizatior under CAPC (Community Action Pro-gram for hilf lrpn) * Vol, P r r ^ r tattura of Eva 1 uati onN<hn des

0 13,067 Vo l . 3

To o r d e r ï

C e n t r e d e r e c h e r c h e s u r les s e r v i c e s c o m m u n a u t a i r e s , U n i v e r s i t é Lava l , B u r e a u 2 4 4 6 , P a v i l l o n G i a r l e s - D e K o n i n c k , Q u é b e c ( Q u é b e c ) G 1 K 71M

Thi s ser ies «Program EVal u a i i o n f o r Organizations u n d e r C A P C ( C o m m u n i t y Ac t i on P r o g r a m to r C h i l d r e n ) » c o m p r i s e s t h r e e v o l u m e s :

O Introductory Manual

© Evaluation Tools for Enhancing Program Quality

Ô Presentation of Evaluation Guides

T h e p r o d u c t i o n of t he se t h r e e d o c u m e n t s h a s b e e n s u b s i d i z e d b y d ie C o m m u n i t y Ac t i on P r o g r a m tor C h i l d r e n , H e a l t h C a n a d a , w i t h t he a g r e e m e n t of d ie P r o v i n c e of Q u e b e c .

T e l e p h o n e : ( 4 1 3 ) 6 5 6 - 2 6 7 4 F a x : E M a i l : W3:

(418) 656-7787 c r sc@crsc ,u la v a l.ca h t t p f / / w w w . c rsc .u lava l ,ca

Documents

Program Evaluation for Organizations under CAPC · 2012-12-10 · program evaluatio fon r organizations under the community action program for children (capc) presentation of evaluation