The Evaluability Assessment Project

The Hewlett Foundation awarded Mathematica Policy Research a grant to conduct the Evaluability Assessment during the pilot phase of Fursa kwa Watoto, from 2014-2015, as the project was designed and implemented in 60 schools. Both Dubai Cares and the Hewlett Foundation agreed that employing a rigorous, evidenced-based, and collaborative approach to developing the FkW initiative would yield a stronger program than moving quickly to evaluation without a fully developed package of interventions. The EA provided the Government of Tanzania, and other stakeholders with important information on the fidelity of the intervention package as piloted and for stakeholders to assess whether to conduct an impact evaluation of FkW as the initiative was extended to 120 schools during the rollout phase (2016-2017).

The EA involved active participation in Steering Committee and Monitoring Evaluation and Learning Working Group (MELWG) meetings and phone discussions; working in partnership to develop the program theory of change and the monitoring, learning and evaluation framework; reviewing reports and documentation; and helping to develop, implement, and learn from the MEL activities conducted throughout the pilot. We observed the partners’ approach, implementation, and response to program problems and weaknesses throughout the pilot phase. The Steering Committee, technical working group (TWG), and MELWG worked together to modify training components to reduce or remove program weaknesses.

Methods to assess FkW

In order to assess the components of Outcome 1, partners in the learning collaborative implemented monitoring, evaluation, and learning activities including the 1) the Teacher Observation tool, 2) the Classroom Observation Tool, 3) qualitative interviews with teachers, head teachers, deputy head teachers, and focus group discussions with parents, and 4) classroom observations. Once analyzed, the MELWG received all data and reviewed findings, which were later presented to and discussed by the FkW Steering Committee and used to inform programmatic improvements.

FIRST CONCLUSION

The FkW preprimary package was developed over two years in a consultative and iterative process. The intervention is theory driven, based on the latest research on preprimary education, and pilot-tested. Throughout the course of the pilot phase, all programmatic concerns were raised by members of the learning collaborative and discussed in detail among the Steering Committee until solutions were identified. To ensure continuous quality improvement, the program has had ongoing monitoring, evaluation, and learning activities.

In the final EA discussion and data collection, the learning collaborative and other stakeholders endorsed the FkW intervention, despite some uncertainty about future scalability and sustainability. Stakeholders agreed that the program rollout should be rigorously evaluated and costed given its potential to yield strong positive impacts on children’s learning outcomes and social development.

Thus, we recommend a rigorous RCT to measure the impacts of the intervention on student learning outcomes. This study will be an important contribution to the evidence on what works in preprimary education that will guide policy and practice in Tanzania. We expect that the evaluation results will contribute to decision making in Tanzania and will also inform global efforts to identify and test effective low-cost interventions.

REVISED RECOMMENDATIONS

The FkW collaborative conducted two years of monitoring, evaluation, and learning (MEL) activities and an evaluability assessment to understand the strengths and weaknesses of Component 1, Model 1 at each stage of implementation. Grounded in the range of MEL data and information collected and analyzed over two years, the pilot initiative yielded evidence of positive changes in teaching practices and learning environments. The MEL activities generated quantitative and qualitative evidence that teachers gained and applied skills, and classrooms were transformed into engaging and stimulating learning environments. Furthermore, respondents perceived important changes in children’s learning outcomes, including enhanced literacy, numeracy, and social-emotional development. Teachers and parents thought these impacts were related to better instructional practices and learning environments. National, district, ward, and local education officers; school inspectors; and school management committees were particularly receptive to the FkW approach once they were trained (toward the end of the pilot phase). Mathematica worked with partners in the collaborative to provide technical assistance on design instruments, develop an analytic plan, and analyze and present data. Once analyzed, the FkW MEL working group received and reviewed findings, which were later presented to and discussed by the FkW Steering Committee.

Changing context in schools: As the FkW pilot concluded activities at the end of 2015, the situation in pre-primary classrooms became even more difficult, when the Tanzanian government issued Circular 5 to instruct schools to implement fee-free education, essentially removing the requirement for parents to pay fees and make contributions and thus allowing children to attend school for free. This new policy had serious implications. Although the pre-primary enrollment rate was only 40 percent of pre-primary-age children (ages 5 and 6) in 2012, by the end of 2016, the net intake/enrollment rate for five-year-olds was 46.7 percent and the gross intake/enrollment rate was 102.6, demonstrating that schools were enrolling students under and over the ages of 5 and 6. In fact, from 2015 to 2016, pre-primary enrollment increased by 46.6 percent in government schools (GoT: PO-RALG 2017). When fees were removed, student enrollment rose rapidly and overwhelmed many classes, grades, and schools. Tanzanian classrooms, already characterized by the poor quality of their learning environments and the ineffectiveness of their teaching practices, were suddenly severely overcrowded as parents flocked to register their children for school.

Government partners, our FkW colleagues in Moshi and Mwanza, as well as many journalists, have documented the strain on schools, lack of materials and books, overcrowded classrooms, and use of unsafe learning spaces.4 Our partners, particularly within schools in Mwanza, have observed drastic overcrowding in classrooms, as well as a severe shortage of teachers and paraprofessional teachers in the classrooms. As of July of 2016, the average teacher-to-student ratio in pre-primary classrooms in Mwanza was one teacher to 85 students (with a range of 32 to 156 students per class). The ratio was lower in Moshi at one teacher to 44 students (with a range of 20 to 80 students per class).

The FkW Steering Committee acknowledged that the dramatic enrollment increases in the majority of schools would likely prevent teachers from fully adopting the improved instructional practices, and that overcrowding would keep children from realizing the maximum benefit of the new practices. Consequently, the planned impact evaluation should not be implemented, because it is unlikely that FkW was effective as it actually was designed.

We determined it would be prudent to instead develop approaches to help teachers effectively manage the large class sizes and continue to conduct monitoring, evaluation, and learning (MEL) activities designed to answer persisting questions about the implementation questions. This would be the best way to help the fidelity, scalability, and sustainability of FkW.

The Learning Agenda was designed to assess the FkW model, including teacher training and instructional practices, classroom management, and school leadership, in the changing context in schools based on whether they participate in FkW. The Learning Agenda was also designed to measure enrollment changes, track decisions and adaptations needed in the challenging context, and crystallize key policy and programmatic issues to provide action-oriented, almost real-time recommendations to schools and education officials. Further, the Learning Agenda is designed to provide the information needed to achieve the goals of the ESDP given that Tanzania policymakers at all levels of government, the donor partners, and school administrators and staff do not necessarily have the information needed to improve quality in pre-primary. The country requires evidence and guidance to increase the speed at which the policy is operationalized at scale. A research and evaluation agenda focused on these issues is an excellent opportunity to directly contribute to the realization of this plan over the next decade. Building from the FkW pilot, we will continue to provide credible, timely, and rigorous evidence that guides stakeholders in the realization of the ESDP, the Education Training Policy 2014, the Primary Education Development Plan II, and the Literacy and Numeracy Education Support, and ultimately improves student learning across Tanzania.