Measuring Transformation Through Houses

Final Evaluation Report for PVC Program/USAID

Donna Anderton, external evaluator

Issue Studied and Relevance to Habitat for Humanity

This was the final report by an external evaluator of a three-year USAID matching grant with Habitat for Humanity International (Cooperative Agreement No. FAO-A-00-99-00057-00). The grant was designed to develop expertise within HFH to evaluate the impact of HFH’s work; learn how to be more efficient and effective in the work; and to institutionalize planning, monitoring and evaluation throughout HFHI. Three countries took part in this PME grant: Guatemala, Sri Lanka and Nepal.

Methods

The evaluator reviewed and analyzed the detailed implementation plan (DIP), revised performance matrix, HFHI’s reports to USAID, and the reports and evaluations of mid-term evaluators. Aspects of the program were assessed through the review of HFH documents such as evaluation reports from each country, evaluation instruments, summaries of trainings for PME staff, an evaluation process manual, reports on PME conferences and evaluation trips. The evaluator also met with key staff and leadership of HFHI; worked closely with the evaluation team members; and met with board members, staff and homeowners at the affiliate and national level in participatory meetings and interviews. The Appreciative Inquiry method was the main method for collecting data, utilizing the same planned questions in all countries and groupings.

Key Findings

A major outcome of this grant was the development of evaluation instruments to identify and operationalize indicators of community impact. Currently, HFHI recognizes the following seven international indicators: 1) number of families served; 2) funds raised through local initiative; 3) mortgage repayment; 4) number of new affiliates formed; 5) number of new countries added; 6) expenses ratios; and 7) tithes to other affiliates. In addition, each of the pilot countries developed a number of national and affiliate indicators that measured the impact of Habitat for Humanity in the following areas: homeowner education, homeowner health, family economic situation, community empowerment and ability to effect change, individual empowerment, peace and reconciliation, and partnerships.

At the time of the evaluation, the three pilot programs had qualitative results but had not yet generated quantitative results. The qualitative results show that affiliates which were most active in their evaluation work demonstrated the most community impact, because they acted on their observations. Evaluation results were communicated to the area offices and led to changes in Habitat for Humanity’s programs at the area level.

The PME work conducted in all three pilot countries yielded important improvements in each program such as changes in house design and cost; strengthening of homeowner selection and education processes; increased repayment rates; development of board membership; and development or strengthening of partnerships with external groups for issues such as health and hygiene training, assistance with income generation, and improvement in construction materials and techniques. Habitat for Humanity’s programs also benefited from an increase in local fund raising and increased community awareness of the program, resulting in a greater number of houses built and families helped.

Systematic evaluation enabled HFH to better communicate the known impact of its program, which could potentially lead to improved institutional capacities to strengthen HFHI/national organization partnership processes, grant management and PME processes, results and applications globally for HFHI. Processes are underway to enable HFHI to quantitatively measure the impact of its work, to identify which characteristics of the organization have the most impact, and to use this information for planning future work to maximize program capacity.

In addition to the universal qualitative results indicated above, each of the pilot countries had country specific successes. Guatemala saw their arrears rate drop from 29 percent in 1998 to a sustained 10 percent over the past four years, while local fund raising was more than tripled. House construction increased to 2,400 houses in 2002, up from 880 in 1998. At least five new strategic partnerships were formed as a result of the evaluation, and six or more partnerships for funding were developed. The PME results were reported to the national board and were used by the board in developing their new five-year participatory strategic plan.

Similarly, in Nepal the national board has a strong commitment to PME. They held a PME retreat at the end of the grant period to understand and utilize PME in helping to prepare, plan, review and focus Nepal HFH in its short- and long-term planning.

The work of the PME team in Sri Lanka has been very effective. They have been involved with several important evaluations including the Sri Lanka Tea Estate Impact Study, and the Save and Build program. Their work has been used by the national director and board to develop Sri Lanka’s three-year strategic plan and for developing best practices for financial management. They are currently testing health, economic impact, and peace and reconciliation indicators rigorously.

Implications for HFH

The use of PME strategy for various purposes has important implications for sustainability. The challenge will be to find ways to communicate qualitative and quantitative findings widely and effectively and to use the results, both positive and negative, to capitalize and build on findings. The results to date indicate that acting on PME findings leads to increased house production.

Several lessons can be gleaned from the PME work done in all three countries. Quality assurance beforehand and attention to homeowners afterward yielded the best outcomes, regardless of the urgency for housing. PME is an effective way to assess needs and gather information. Homeowners are a crucial source of information to increase the efficiency, effectiveness and overall impact of HFH’s work. Participation in the data gathering/survey process enables partner families to better understand the benefits received and the moral obligation to contribute, and leads to empowerment. Solving identified problems builds strong loyalty and satisfaction, leading to improved repayment rates and an increased ability to build more houses.

At the organizational level, PME reports to leadership should be short and concise. This enables the leadership team to effectively see that the PME work helps HFH build more houses, and to utilize the information gathered in developing and measuring strategic goals.

Evaluator Recommendations

• Maintain PME teams in all three pilot countries to continue and simplify PME underway, to expand data and utilize evaluation instruments and impact indicators, and to complete input and analysis of data in databases.

• Utilize data and reports to determine lessons and their implications for affiliate and national office activity. Maximize impacts and systematically incorporate lessons learned from evaluations into planning and implementation.

• Strengthen communications and communications systems for ongoing PME programs and for sharing lessons with leadership at all levels and the broader Habitat for Humanity community.

• Appoint a new global PME assignee to oversee and strengthen aspects of PME functions in the field, and to insure that information is transferred to other regions as appropriate.

• Utilize evidence of Habitat’s impacts not only for internal changes but also for developing partnerships with other entities that meet the non-housing needs of homeowners (i.e., income generation) or that fund work that can be verified as impacted by the Habitat for Humanity process (i.e., health, education).

Questions for Reflection

• What evaluations have we done on the work in our affiliate? Do we encourage homeowner participation in conducting evaluations? How have we used the information gathered to improve our program?

• What steps can we take to incorporate planning, monitoring and evaluation into our program?

• What information do we need to develop a strategic plan for our affiliate?

• What other indicators have emerged from the work presented in this report?

Sources Used in Study

• Audited financial statements – for affiliate mortgage values, ReStore sales

• IRS Form 990 – salary/benefits for employees, taxes, operating expenses

• Property Valuation Administrator office – for assessed value and current tax of properties

• Kentucky Housing Affordability Index – for average mortgage rate

• Consumer Price Index

The entire paper is available at the following address: http://elearning.hfhu.org/hfhu/documents/research/mtthReport.pdf

Additionally, an annex to this paper covering the Statement of Work is available at the following address: http://elearning.hfhu.org/hfhu/documents/research/mtthSow.pdf