Assessing Your Impact

Learning from Experience: Hamilton and Winnipeg

Hamilton Community Foundation: Tackling Poverty Together

Background

Beginning in 2004, Hamilton Community Foundation made a four-year, $3 million commitment to support its Tackling Poverty Together program. The goal of the program is to identify and support strategies that prevent poverty and to help create and sustain pathways out of poverty for low-income people in Hamilton. In line with the various roles of a community foundation, the program has several elements:

Grantmaking: Grants from the Community Fund continue to provide funding in all sectors but grants within and across those sectors are focused on work that addresses the conditions that lead to poverty that contributes to the reduction of poverty.
Leadership: HCF uses its own capacity to engage policy makers in dialogue and to influence decision making to address poverty issues and convenes diverse stakeholders such as policy decision-makers, philanthropists, community activists, academics, consumers, and service providers to add perspective to each other's work, to inform the Foundation's work and to jointly problem-solve.
Communication: HCF uses its own capacity to increase public compassion for those experiencing poverty and to shine a light on needs, strategies and opportunities for involvement.
Knowledge Development: HCF closely monitors community trends, develops analysis to assist in their own work and that of others and measures the success of its own interventions, and learns and shares its knowledge with others.

Evaluation Approach

Hamilton Community Foundation evaluated its Tackling Poverty Together (TPT) grantmaking program, at its approximate mid-point in 2005, using a framework developed by Dr. Arnold Love that examines impact at three levels:

Level 1: Outcomes achieved for the intended participants (beneficiaries) of each TPT project, such as homeless people who now have secure shelter.
Level 2: Intermediate effects attained in terms of changes in the grantee organizations (e.g., greater organizational effectiveness) and the community (eg., improved citizen engagement, capacity building)
Level 3: Measures of foundation performance such as improved governance, more effective grantmaking, and increased knowledge.

The specific evaluation components, administered at the mid-point of the Tackling Poverty Together initial phase, are linked to the three levels. Taken together, these components supply the following evaluative information:

Achievement of participant outcomes by grantees
Information from grantees about changes in attitudes, behaviour, responsibilities, and outcomes they observe
Feedback about organizations and community capacity building
Feedback about TPT policy influence and strategic systems change
Feedback about the HCF TPT grantmaking process and strategic directions

In terms of Level 1 (Outcomes), grantees are accountable for the achievement of participant outcomes and the specified project goals/objectives. Grantees conduct their own evaluations and report back. The Foundation has helped build the evaluation capacity of those agencies. More information on this is provided in the section called Grantee Evaluation Support.

For Level 2 (Capacity-Building and Knowledge Creation), HCF convened meetings that encouraged grantees and other relevant parties to reflect together on the evaluation of their projects and the lessons they are learning about poverty reduction. These meetings helped assess how TPT grants are improving grantee effectiveness. Key informant interviews helped assess whether TPT has influenced others to support poverty reduction and how well TPT has influenced the thinking of policy makers, funders, thought leaders and the public in Hamilton about poverty.

Looking at Level 3 (Foundation Operations and Strategy), HCF evaluated the grantmaking process and how this work influences the broader poverty strategy in Hamilton and the overall grantmaking strategy of the foundation. To elicit this information, three surveys were designed and conducted- for TPT grant recipients, all organizations that applied to TPT and were declined, and a select group of agencies that could have applied but didn't.

Grantee Evaluation Support

Hamilton Community Foundation determined that it would be reasonable to adjust its expectations of the evaluation activities that grantees would undertake according to the size and length of the grant and the capacity of the agencies.

Grantees were divided into two tiers:

Tier 1 included those who were receiving multiple year grants
Tier 2 included those who were receiving grants for one year only and those that received small amounts.

For Tier 1 grantees, the expectation was that they would evaluate their outcomes in terms of qualitative and quantitative changes using valid and reliable evaluation methods. In order support this, HCF set aside funds to build the evaluation capacity of those grantees and used it to provide specific evaluation training to these agencies as a group. In mid-2005 the Foundation held a half-day evaluation workshop at which Dr. Arnold Love presented the basic approach to outcome evaluation and HCF outlined the TPT reporting requirements.

A second meeting in early 2006 allowed those agencies to learn from each other's experiences and build their own capacity to monitor and evaluate their programs. Dr. Love presented evaluation challenges and themes arising from their reports to HCF and posters for each agency, describing their programs and progress to date, were displayed. Continuing to build evaluation capacity, HCF held a third workshop with Sherri Torjman, exploring the policy impact of the TPT grantees' work and ways to assess the impact of their work collectively.

For Tier 2 grantees the expectation has been that they will provide HCF with interim and final reports about the achievement of their project outcomes and lessons learned, but not that they will conduct in-depth evaluations of their projects.

HCF reports that the (ongoing) focus on building the evaluation capacity of TPT grant recipients has forged a much closer, flexible relationship with them. It has also strengthened the ties between recipient organizations and their understanding of the collective impact of their work.