Scalable Early Warning Systems for School Dropout Prevention Evidence from a 4.000-School Randomized Controlled Trial

The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.


Introduction
Across many low-and middle-income countries, young people's educational attainment continues to fall short of aspirations, as high rates of enrollment in early grades quickly decline due to students dropping out before completing a full course of basic education.In Guatemala, the country of focus in this paper, education is de jure compulsory through ninth grade, but nearly 40% of sixth graders drop out before getting there.For youth who drop out prematurely, global evidence suggests that, on average, they will earn less and experience more social and economic challenges than their peers with more years of completed education (Patrinos and Psacharopoulos 2004;Oreopoulos and Salvanes 2011;Bentaouet-Kattan and Székely 2015;Cardenas, De Hoyos, and Székely 2015).
Dropping out of school is often not a discrete decision, but the culmination of an ongoing process of disengagement.A growing body of research in education, psychology, and behavioral economics points to the importance of understanding the early origins of dropout and young people's underlying decision-making processes (O'Donoghue and Rabin 2001;Hammond et al 2007;Mullainathan and Shafir 2013;Frazelle and Nagel 2015).These features suggest that the most effective way to prevent dropout from occurring in the long-term is to build human capital from early childhood onwards.At the same time, there is a pressing need to directly intervene among youth already at high risk of dropping out.Many interventions have shown some promise, depending on the country context and level of education targeted -including information provision on the value of education, remedial tutoring, socioemotional skills coaching, and scholarships (Dearden et al 2009;Di et al 2013;Chappell et al 2015;and Avitabile et al 2019 among many others).These interventions vary significantly in their targeting approaches and level of intensity, resulting in important differences in the resources (human, financial, and otherwise) required to implement them successfully.In countries like Guatemala with high rates of dropout and constrained resources, there is strong demand for accurate targeting and low-cost, contextappropriate interventions.
In this paper, we report the outcomes of an early warning system pilot program in Guatemala aimed at reducing dropout in the transition from primary to lower secondary school.Early warning systems -which vary in their structure but usually include monitoring routinely collected indicators of student risk and taking preventative actions tailored to students' needs -are in place in at least 30 U.S. states and the majority of European countries (European Commission 2013;O'Cummings and Therriault 2015).As more low and middle-income countries invest in developing reliable education management information systems, student data that forms the foundation of an early warning system is becoming increasingly available.However, little evidence exists on whether and how such systems can effectively reduce dropout in countries with limited capacity to respond to students' needs.In Guatemala, the pilot program, named ENTRE (Estrategia Nacional para la Transición Exitosa -National Strategy for Successful Transition), was designed to utilize recently developed data systems to identify students at risk and activate preventative, evidence-based measures at the school level.ENTRE focuses on three main factors identified as both particularly important for causing dropout in the primary to secondary transition in Guatemala and addressable through relatively low-cost interventions: a lack of knowledge among key primary school actors about effective measures to help students stay in school; a lack of knowledge among these same actors about which students are most at risk of dropping out; and a lack of prioritization of dropout as a problem to be addressed, as school actors have myriad responsibilities and dropout in the transition to lower secondary occurs after students have left their primary schools.
To assess the effectiveness of this early warning system approach, implementation of ENTRE was designed as a four-arm randomized experiment (three treatment arms and one control arm) among one-fifth of all the public primary schools in the country.In the first treatment arm, school principals and sixth grade teachers receive a user-friendly guidance manual and half-day training on specific, evidence-based methods to help students make the transition to lower secondary school -in essence, providing them more information on how to help students stay in school.A letter signed by the Minister of Education accompanied the guidance manual, emphasizing the importance of every Guatemalan child completing at least nine years of basic education (primary and lower secondary) and the responsibility of schools to support them in doing so.The second treatment arm adds to the first a list of sixth-grade students at high risk of dropping out based on a prediction model using administrative data-providing them more accurate information on who needs the most help.The third treatment arm adds to the first two small behavioral nudges to try and keep dropout prevention top-of-mind for principals and teachers.A fourth, control arm included schools that received no intervention or information about ENTRE.
We find that assignment to the program significantly reduces dropout by 1.3 percentage points (3.6 percent of the baseline dropout rate) in the subsequent school year.The effect size is similar across the three treatment arms, ranging from 1.1 to 1.5 percentage points, and we cannot reject the hypothesis that the impacts are equal across treatment arms.This suggests that the core intervention of providing information on how dropout can be prevented is driving the results.
Impacts are heterogeneous, with dropout reduction concentrated among higher-risk students, students attending larger primary schools (and therefore likely facing fewer supply constraints), and male students.Furthermore, imperfect compliance, both in terms of incomplete take-up of treatments and contamination of the control group, suggest that intent-to-treat (ITT) estimates are likely a lower bound of the effect of the program on the treated.We therefore also estimate impacts on program compliers and find that the local average treatment effect is a 3.1 percentage point reduction in dropout (about 8 percent of the baseline dropout rate), roughly double the ITT estimate but much less precisely estimated.
To further understand the mechanisms driving the results, we collected data on several intermediate outcomes and conducted a series of focus groups with program actors.Across treatment arms, school principals report an increase in their own expectations of the returns to education for their students, in their beliefs about their own ability to change students' outcomes, their self-reported prioritization of dropout prevention among their responsibilities, and a large increase in targeting their efforts to at-risk students.
While the magnitude of the estimated impacts is modest, the program leverages the Ministry of Education's existing administrative data structures, management systems, and personnel, with limited additional costs.Specifically, total costs per student are estimated at USD 2.91, 36% of which corresponds to the design of the program-i.e., the cost of implementing the program amounts to USD 1.85 per student-and 45% to the opportunity cost of principals and teachers.
Estimates of the monetary benefits accruing to students who do not drop out from higher labor market earnings suggest an internal rate of return of 28% and a benefit-cost ratio of almost 19-to-1, suggesting that ENTRE is a highly cost-effective, sustainable, and scalable approach to addressing short-run drivers of dropout.Moreover, ENTRE may be a platform program on which additional interventions to reduce dropout can be developed and rolled out, presenting a promising opportunity for Guatemala and other countries interested in leveraging the basic preconditions to identify and support students at risk of dropout.
The remainder of the paper is organized as follows.Section 2 describes the education context in Guatemala.Section 3 describes the ENTRE program, while Section 4 describes the experimental design and data.Section 5 presents the impact evaluation results and evidence on likely mechanisms.Section 6 concludes.

Context
Over the past two decades, Guatemala has continued to make important progress in expanding access to primary and secondary education, as well as in improving student achievement.Primary and lower secondary school completion rates have increased by 43% and 112%, respectively, since 2000.At the same time, average learning outcomes have also improved -Guatemalan sixth graders' performance in both reading and math improved much more between 2006 and 2013 than the LAC region overall (World Bank, 2019).Increasing average learning outcomes while expanding access is a significant accomplishment, yet much remains to be done.Despite this progress, early school dropout remains an important challenge to increasing educational attainment in Guatemala.School dropout occurs at all ages, but becomes widespread starting in early adolescence.Whereas net enrollment is over 85 percent in primary school, it falls to only 45 percent in lower secondary and 25 percent in upper secondary (UNESCO, 2020).
Dropout is particularly concentrated in the transitions from primary to lower secondary (6th to 7th grade) and from lower to upper secondary (9th to 10th grade), where approximately one out of three students drops out in each of these transitions (Adelman et al, 2018).These high dropout rates reflect systemic issues that require medium to long term approaches and significant investment to comprehensively address.In Guatemala, children's needs are immense.
Over half the population is under the national poverty line, and the country has the world's sixth highest rate of chronic malnutrition (World Bank, 2020).As a result, many children come to the school system suffering from multiple dimensions of deprivation, yet public education services are inadequate.The Government of Guatemala spends less than 3% of GDP on education, well below regional peers, primarily due to low revenues overall, as the government does devote over 20% of its budget to education.As a result, both the quantity and quality of education supply are insufficient to meet existing needs.For example, preschool starts relatively late, typically around age 6, only lasts for one year, and covers less than half of the population.In addition, the supply of secondary education is also insufficient and skewed towards urban and more prosperous areas of the country.In the departments with the highest poverty rates, there is only one secondary school for every 400-500 secondary-school age youth, while in lower poverty departments, this ratio is closer to one school for every 100 youth.
Yet there is a pressing need for targeted and scalable action in the short term among the large pool of youth already at high risk of dropping out.Evidence from across Latin America shows that greater shares of youth who are out of school and out of work contribute to a range of negative social and economic outcomes, including higher rates of crime, long-term reductions in labor productivity and growth, and increases in inequality (De Hoyos, Rogers, and Székely 2016).

The ENTRE program
In 2017, motivated by the challenges and constraints described above, the Government of Guatemala, with support from the World Bank, designed the ENTRE program as a scalable and low-cost early warning system.The program is aimed at reducing dropout in the transition from primary to lower secondary school, where almost a third of students stop their formal education.
Like most early warning systems, the key goals of ENTRE are to provide targeted support to students at-risk and to take timely, preventive actions.At a minimum, this requires that school principals and teachers have a basic knowledge of effective dropout prevention strategies, are able to identify students at risk, and implement these strategies in a timely manner.However, quantitative and qualitative data-including focus groups with teachers and school principals, as well as interviews with public officials conducted to inform the design of ENTRE -showed important gaps in all these dimensions.
First, school actors have limited knowledge of, or support on, dropout prevention strategies, and school principals-who play a leading role in these initiatives-are usually regular teachers without any specific leadership or management preparation.Second, school actors have limited information on who is at risk of dropping out.Data on mandatory "dropout risk flags" reported by school principals and teachers show that they identify on average fewer than 5% of the students who end up dropping out in the transition from primary to lower secondary.1Third, there are several parallel demands and multiple tasks that compete for the attention of principals and teachers, which may prevent dropout prevention from being top-of-mind, particularly since dropout in the transition from primary to lower secondary occurs after students have left their primary schools and are no longer under their official responsibility.For instance, surveys of school principals show that reducing dropout is not a top priority for them, as they rank it third on average out of five options (below teaching/curriculum and internal administrative tasks).
Considering these gaps, the government designed ENTRE to combine three key components: a brief training on actionable, evidence-based strategies to support students to stay in school; information on sixth-grade students at high risk of dropping out; and small behavioral nudges to try and keep dropout prevention top-of-mind for principals and teachers.
Component 1: Short training on actionable strategies to prevent dropout The first component consists of a half-day training for school principals and sixth-grade teachers on specific, evidence-based methods to help students make the transition from primary to lower secondary -in essence, providing them more information on how to help students stay in school.
The training was complemented by a short and user-friendly guidance manual (13 pages of text and informative illustrations in the main manual, plus 13 pages of additional resources in annex) that summarizes the training content, a copy of which was given to each participant at the beginning of the half-day training.Along with the guide, the school principals received a personalized letter from the Minister of Education that emphasized the importance of every Guatemalan child completing at least nine years of basic education and urged the school principal to lead her or his staff in implementing what they learned through the ENTRE program.
The training covered the objectives of ENTRE, some of the key barriers to staying in school (lack of motivation, financial constraints, lack of family support, poor academic performance, and inertia), and several methods and resources to address these barriers.For instance, to address the lack of motivation to continue studying, in line with international evidence on cost-effective strategies for this barrier, the training presented information on economic returns to education in Guatemala (which are among the largest in Latin America) and recommended a set of activities to effectively communicate these returns to students.The participants, divided into small groups, then had the opportunity to practice the approach with role-play exercises, and to brainstorm about the logistics of implementation.The sixth-grade teacher and the principal of the same school were intentionally placed together so they could start discussing implementation in their own school.
To address financial constraints, the participants were informed about means and merit-based scholarships available through the Ministry of Education (MINEDUC) and provided with instructions on how to apply to them (including where to locate the required forms).As highlighted by empirical evidence (see Bettinger et al, 2018), the participants were also informed about the importance of helping disadvantaged families to fill out these forms, since many times despite qualifying for this type of benefit, low-income families do not apply to them.
The training also included a section on the importance of focusing efforts on students at-risk of dropout.Participants were encouraged to use their own judgement to assess this risk, but they were also informed that some schools would receive complementary information on students at-risk as part of MINEDUC's efforts to identify the most effective content for ENTRE (component 2).
Component 2: Information on students at high risk of dropping out The second component of ENTRE is a list of sixth-grade students at high risk of dropping out in the transition to lower secondary, providing school actors with more information on who needs the most help.The list was estimated using routinely collected administrative panel data on students and combining linear regression models with a simple algorithm.These models are able to correctly identify 82% of the sixth-grade students who will drop out within the next year, performing better than other commonly used targeting approaches, and as well as models used in the U.S. (For more details on the methodology, please see Adelman et al., 2018).
The list was specific for each school, providing the name and identification number of each student, and was addressed to the school principal.To reduce any risk of negative labeling, both the letter from the Minister and the content of the training emphasized the fundamental importance of avoiding any type of stigmatization of students at risk of dropping out.The main objective of this component was to help school actors target their support to students most in need of it, given limited time and resources. 2omponent 3: Behavioral nudges The third component consisted of five monthly reminders sent to school principals in order to keep dropout top-of-mind and motivate them to act.The messages were sent through SIRE, an online portal that school principals use regularly to conduct different administrative tasks and exchange information with MINEDUC. 3 Each month, between June 20 and October 1, these messages appeared on the homepage of the portal as soon as school principals logged in.The reminder messages utilize various behavioral insights to motivate action, namely: • Social recognition (message 1): Social incentives can exert a powerful effect on behavior given the human desire for status and recognition.Social rewards, such as status and recognition, can motivate people to exert effort and can even substitute for monetary rewards in some situations.The first message of SIRE was inspired by this theory, informing school principals about an official certificate to recognize the schools that made the most progress in terms of reducing dropout and the publication of the list of recognized schools in a popular newspaper in Guatemala.This information can act as an extrinsic incentive for school teams to act.
• Salience (messages 2-4): Research shows that people are more likely to respond to stimuli that are timely and accessible, and are more likely to do something that their attention is drawn towards.Even though school principals are being provided with concrete ideas, dropout prevention may not be their foremost priority, especially given the number of other tasks they are also responsible for.Salient reminders about their goal (helping at-risk students), and the tools and resources they have at their disposal can draw their attention to the task and be an impetus for action.Messages 2-4 are therefore structured to increase the salience of dropout prevention, including motivational phrases, reminders of key actions, and references to the guidance manual.
• Loss frame (message 5): People sometimes put more weight on potential losses than on potential gains.This tendency can also affect people's level of effort in response to various incentives.Message framing is based on prospect theory (Tversky & Kahneman, 1981), that posits that framing a behavior in terms of its prospective costs or benefits can have significantly different effects on individuals' decision making.The final reminder message utilizes loss frame to communicate the urgency of acting now since failure to do so could result in losing something, namely the chance for social recognition.received an envelope containing the guidance manual, the letter from the minister (if they were school principals), and the list of students at-risk (if they were school principals and were in the corresponding treatment arm).
While the materials in the first and second components of ENTRE were distributed during the trainings, the third component of the program -the reminders-was delivered exclusively through SIRE.The first message announcing social recognition was activated about six weeks after the training, on June 20.From that date on, reminders were updated in SIRE on a monthly basis, on July 16, August 6, September 3 and October 1. School principals were informed by their supervisors about the presence of a new reminder in SIRE and were invited to enter the online portal each time.The platform was also used to capture baseline and follow-up information through online questionnaires, which complemented baseline administrative information available at the beginning of the school year and a follow-up survey implemented on paper after the school year came to an end.

Eligibility
As mentioned above, there are areas in Guatemala where children just do not have reasonable access to secondary schools.Given its features, ENTRE is targeted at schools and students where the supply-side constraints are less likely to be binding.Therefore, to minimize the role of these constraints, the universe of eligible schools was restricted to public schools in the top 30% of municipalities in terms of supply of secondary schools per student.In addition, within those municipalities, the very small schools likely to be located in remote rural areas (the schools belonging to the bottom 15% in terms of school size) were excluded from eligibility, as these schools tend to face binding local secondary school supply limitations.Finally, to target schools that could benefit more from the type of information provided by the program, we dropped schools that-according to our predictive model-had zero students at-risk of dropping out.As a result, the universe of eligible schools totals 6,037 public primary schools.6

Experimental design and research questions
The program was designed as a multi-arm randomized experiment.Randomization was used to create four experimental groups of 1,000 schools each (three treatment arms and one control arm).
Namely, from the 6,037 eligible schools, a simple random sample of 4,000 was drawn and then each was randomly assigned to one of the four experimental groups.The schools in the first treatment group receive component 1 (the half-day training and the user-friendly guidance manual, as well as a letter from the Minister of Education) which is aimed at improving knowledge on simple evidence-based methods to help students make the transition to lower secondary school -in essence, providing teachers and principals with more information on how to help students stay in school.The second treatment arm adds to the first arm the list of sixth-grade students at high risk of dropping out (component 2), providing school actors more accurate information on who needs the most help.The third treatment arm adds to the first two arms small behavioral nudges We also collected data on the quality of implementation of the program and several intermediate (self-reported) outcomes through a series of paper-based and online questionnaires integrated into the SIRE platform.All schools participating in the experiment were requested to complete an baseline questionnaire online, two follow-up online surveys, and a final, paper-based follow-up survey.In addition, the 3,000 treatment schools were also requested to complete an online

Program Participation
To assess the take-up of the program we utilize multiple data sources.For treatments 1 and 2, we use school principals' self-reports on the reception of the dropout prevention guide (which is also a proxy of participating in the training) and the list of students at risk, respectively.This information was collected in the last follow-up questionnaire, which was the only paper-based follow-up questionnaire and the one with the highest response rate.The proportion of schools with data on the reception of the guide was about 85 %, with no significant differences across control and treatment arms.For reception of the list of students at risk, the response rate was lower, with significant differences between the treatment and the control groups (the proportion of schools with data on this question was 77% and 55%, respectively).The difference may be driven by school principals in the control group not understanding the question (as they likely never heard about the list of students at-risk).The data on the take-up of treatment 3 comes directly from SIRE administrative records and is proxied by a dummy capturing whether the school principals logged into SIRE at least once in the month that followed the activation of the reminders on the platform.

Outcome data
The main outcome indicator is a dummy variable capturing whether individual students who were enrolled in primary school in April 2018 had dropped out by April 2019.To construct this variable, we exploit the panel structure of our data-using the students' identification numbers to track their academic trajectories through different years-with a variable reporting the enrollment status of the students during any given academic year (i.e., currently enrolled or dropped out).If a student is not found in the next year's panel, we consider that the student has dropped out.In addition, potential errors in the linkage of students' administrative records across years would also be interpreted as dropouts in our dataset.We do not expect these challenges would affect our identification strategy for two main reasons.First, even with these limitations, the SIRE data replicates well the dropout patterns observed in Guatemalan household surveys (Adelman et al, 2018).Second, there is no reason to believe that this potential measurement error would be different across randomly selected control and treatment groups.In short, we expect relatively small and random measurement error in dropout, creating (if anything) a small downward bias to our estimates.so we consider analysis of this data as exploratory.In particular, 26% (22%) of the treatment (control) schools missed the baseline questionnaire, 13% (11%) the final paper-based questionnaire, and 59% (63%) the short-term online follow-up.However, the key results are robust to controlling for baseline responses and to dropping T3 schools, which had higher response rates. 7  Finally, the attrition rate is not correlated with the baseline responses, nor with student dropout rates at baseline or follow-up.

Qualitative analysis
We also performed a series of focus groups (12 in total) to gather qualitative information on the challenges of program implementation.We interviewed almost 100 people involved in the program, including 63 teachers and principals from across treatment arms and 34 trainers in 4 different locations. 8Short paper-based questionnaires were also completed during these focus group sessions.

Descriptive statistics and balancing tests
Table 2 reports the average baseline characteristics for the students and their schools according to the administrative data available. 9The table reports the averages for the control arm (column 1), the pooled treatment arms (column 2), and each treatment arm separately (columns 3-5).The administrative data includes detailed academic grades by subject (math, language, natural sciences, etc.), age, gender, school dropout rates, the average dropout probability for the cohort, and regional variables.
Overall treatment and control groups are relatively balanced.Most of the differences in the variables considered are statistically insignificant.The chi squared test after running seemingly unrelated regressions of each of these baseline variables on treatment assignment shows that the 7 These schools had higher response rates in online questionnaires likely because the reminders made schools more prone to see the survey link; furthermore, they explain all the difference on the response rates between treatment and control schools in the online survey used. 8The teachers and principals come from various departments, including Quetzaltenango, Quiché, Totonicapán, Chimaltenango, Sololá, Sacatepéquez, Jalapa, and Jutiapa. 9We did not include in this analysis the complementary online data given that the take-up was roughly 70%.
differences in all the pre-treatment administrative variables are jointly insignificant (the lowest pvalue is 0.22).
However, the treatment groups perform worse than the control group in two key dimensions, namely dropout (36% for the pooled treatment vs. 34%), and school size (268 vs. 296, respectively), suggesting that the treatment schools are relatively more vulnerable than the control schools.In particular, the 2-percentage points gap in dropout rates is an issue because it creates a significant downward bias in the impact of the program (particularly taking into account that we are analyzing a very low-cost intervention).
To correct for this bias, as will be discussed in the next section, we will include school-fixed effects in our estimations (exploiting the availability of repeated cross-sections from the administrative data).

Results
Program effects (Intent-to-Treat Estimates) We start by analyzing the intent-to-treat estimates on dropout.We first estimate the following "pooled" model -which maximizes our power to assess the impact-using a linear probability model (estimated with OLS): Where   is the dropout status of student s in school j between year t and year t+1, Entre is a dummy capturing whether the school was (randomly) assigned to the program,   is a dummy indicating whether the program is already being implemented (i.e., whether t=school year 2018-19),  represent the students characteristics, and   and   are year and school fixed-effects. 10  Standard errors are clustered by school and year.
In addition, we estimate the following model to capture the impact of the (random) assignment to the three alternative treatment arms: Table 3 reports the estimated models.In this table, columns 1-3 show the results for three different specifications: (i) no control variables, (ii) basic student-level characteristics, and (iii) studentlevel characteristics and school-fixed effects (the last one is our preferred specification and corresponds to equation ( 1)).
The results for the pooled model are reported in the first panel of Table 3.The magnitude of the effect is quite similar in all three specifications, but we gain more precise estimates as we add more 10 Alternative specifications are considered in the robustness section.
controls. 11According to our preferred specification (column 3), attending a school that was assigned to the program significantly reduces dropout by 1.3 percentage points (about 4 percent of the baseline dropout rate) and this effect is significant at the 1 percent level.In the second panel, we report the estimated ITT estimates for each arm.The results for our preferred specification (column 3) indicate that the effect of the treatment is similar across the different arms, ranging from 1.1 to 1.5 percentage points, with all the coefficients statically significant at the 5 or 10 percent level (depending on the arm considered).In fact, we cannot reject the hypothesis that the impact is the same across the three treatment arms, suggesting that the basic intervention on how dropout can be prevented is mostly driving the results. 11This is indeed a general pattern that we observe throughout the analysis conducted in this paper.For simplification purposes, in the rest of the paper we mostly report the equivalent to column (iii) -our preferred specification -but more detailed tables are available upon request.We also conduct additional checks in the robustness and placebo sections.
( positive, but small and not statistically significant).For the other two treatments (both of which received the list of students at-risk) there is no difference between the point estimates for boys and girls, indicating that the benefits of the programs were evenly distributed by gender.Since the only common difference between T1 and T2-T3 is the provision of information on who is at risk, this provides suggestive evidence that the program might have corrected gender-biased perception on who is at risk of dropping out.Indeed, pre-program administrative data does suggest that these biases exist.While observed data shows that boys are 13% less likely to drop out, principals' perceptions data on who is at-risk of dropout and who needs more support is biased towards boys, and the bias is quantitatively substantial (between 45% and 50% gender gaps in the risk perceptions).This is illustrated in Figure 2.
Figure 2. Risk ratios: data vs. perceptions (boys/girls) Notes: The observed risk ratio measures the actual dropout rate of boys divided by the actual dropout rate of girls, based on observed pre-treatment data (2017).Perception risk ratios are based on the (pre-treatment) self-reported data by school teams on who is atrisk of dropout (or need support), diving the percentage of boys who are marked as at-risk of dropping out (or in need of support) by school principals by the corresponding percentage of girls marked as such by them.

Placebo tests
Since our identification strategy combines random treatment assignment of schools with school fixed effects, our main identification assumption is that there are no different pre-existing trends for treatment and control schools.To assess if this is a valid assumption, we conduct several placebo tests.While in our main experiment we compare treatment and control schools between these trends should be also captured by our placebo experiments.Tables 5 and 6 show that the coefficients of the placebo regressions are all quite small in magnitude and statistically insignificant, providing strong evidence against pre-existing trends.This is further illustrated in figures 3 and 4.  Table 6.Placebo test of the heterogeneity in the impact of ENTRE on students' dropout rate Notes: 1) Placebo: estimates using data from 2016 and 2017, when the ENTRE program did not exist.Experiment: estimates using data from 2017 and 2018 (year of the intervention) and the preferred specification in Table 4. 2) Standard errors clustered at schoolyear level in parenthesis; 3) *** p<0.01, ** p<0.05, * p<0.1.4) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1.5) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year 2017 in Placebo; = year 2018 in Experiment; 0 otherwise); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group.6) Interaction j in columns (i) is given by the variable Risk sjt (Probability of dropout for student s that attends school j in year t estimated in the Early Warning System); in columns (ii) by Large school sjt: (1=Large school: size in 2017 > median), and in columns (iii) by Man sjt: (1=Man).7) All specifications in panel (i) include the corresponding Postt , Entrejt and Interaction j dummies, while those in panel (ii) include the Postt and Entre i jt dummies; 8) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System).

Mechanisms
Our multi-arm design sheds some light on the potential mechanisms driving the results.As discussed above, the bulk of the impact seems to be explained by the component providing information on how to prevent dropout.In addition, the benefits of the program were mostly captured by students at risk of dropping out.We also find strong suggestive evidence that the information on at-risk students might have contributed to correct school principals' gender biasedperceptions of who is more likely to drop out.
In this section we further explore the potential mechanisms by analyzing data self-reported data from school principals on relevant channels.The analysis suggests that the program had a shortterm impact on two relevant factors that might be linked to the knowledge and motivation of school principals: the perceptions of returns to schooling and the dropout expectations for the current academic year.In addition, the program appears to have large and more sustained effects on both the provision of targeted support to students at-risk and the prioritization of dropout as an issue.
In the first panel of Table 7 we show the coefficient capturing the (pooled) impact of ENTRE on different self-reported indicators (each coefficient corresponds to a different regression).The second panel reports the number of observations (schools) corresponding to each regression.The outcome variables are expressed as changes between the baseline and follow-up surveys.We measure the impact at two different moments of follow-up: short-term (about 2 months after the training was implemented and about a month before the enrollment for the next academic year begins) and mid-term (in the middle of the next academic year, about 8 months after the training was conducted).(4) All regressions control for the average characteristics of the students in the school (proportion of men, average risk of dropout and proportion of students of age 11, 12, 13, 14, 15, 16, 17, 18, and 19 & more).( 5 The analysis shows some interesting findings.First, being assigned to ENTRE reduced principals' dropout expectations in the short-term by 0.22 standard deviations and increased the perceived returns to education (one of the topics most highlighted in the ENTRE training) by 0.25 standard deviations.However, both effects vanish by the next academic year.This may be capturing a shortterm (but timely) impact on the knowledge/motivation of school principals in the few months that followed the training, right before the enrollment process for the next academic year started.
We also analyze the impact of the program on a set of variables capturing principals' perceptions of their role in preventing dropout and of the growth mindset of students.We find some evidence of a positive impact in these dimensions (e.g. the analysis suggests that the program increases the growth mindset mentality of students -as reported by the principal-by 3 percentage points), but the impacts are rather small and only significant at the 10 percent level.
In addition, we do not find any impact on principals' self-reports related to students' families.The relevance of regular communication with families was one of the aspects highlighted in both the training and the guide.However, according to self-reported data, neither the frequency of meetings nor principals' perceptions of knowledge of families was affected by ENTRE.Overall, the coefficients (mostly) have the expected signs, but they are statistically insignificant.
Next, we look at the impact of the program on targeted support to students at-risk of dropping out.This is at the heart of the ENTRE program and highlighted in a special section of the training for all schools across the three treatment arms.The results show a strong effect in this dimension.
According to our estimation, the program improved the provision of targeted support by more than 60% of a standard deviation.While the impact decreased with time, it remained statistically significant in the next academic year.
Finally, we looked at the impact of ENTRE on principals' self-reported priorities.One of the channels through which ENTRE was expected to work was through the higher prioritization of dropout prevention among competing tasks (a challenge identified in the focus groups that informed the design of the program).School principals were asked to rank priorities from 1 to 5, with 1 being the least important, and 5 the most important.Our estimation indicates that being assigned to ENTRE significantly improved the ranking of dropout prevention among competing tasks in both the short and medium-term.Two months after the training was conducted, the ranking of dropout prevention improved by almost 0.5 (about 34% of a standard deviation).In the next academic year, the effect was cut by almost half (but it was still statistically significant).

Qualitative results
Results from the focus groups with teachers, principals and trainers are mostly in line with the main quantitative outcomes described above.In particular, the dropout prevention guide was the element in the intervention most frequently exalted, with an excellent reception among teachers and principals, which highlights the importance of the core intervention on how dropout can be prevented.School actors also declared an increase in their targeted efforts regardless of the treatment arm to which they belonged.Moreover, principals in the third treatment arm reported that the automated reminders were too frequent and confusing, possibly explaining why the third treatment arm was indistinguishable in its impact.
The pattern of heterogeneous effects by dropout probability, school size, and gender is further confirmed by the anecdotical evidence provided in the interviews.Besides the above-mentioned general claim of a greater emphasis on those students who needed more support, principals and teachers in multi-grade schools13 reported having no time for ENTRE activities, which is consistent with the program's lack of effect on small schools.In addition, when asked about proposals to improve the program, many school actors suggested that the guide should include differentiated strategies for boys and girls.
Self-reported data from open questions in the focus groups also provide information on mechanisms and compliance.Teachers and principals claimed to have focused their strategies on motivating students to pursue secondary.At the time the interviews were made (about 4 months after the training was implemented), school actors showed a strong confidence on the positive results from the program, consistent with the short-term impact on dropout expectations described in the subsection above.Finally, trainers reported some problems with the logistic during the delivery of the guide and the list of students at risk, which suggests that compliance was not perfect and justifies the analysis in the next subsection.
list (T2 and T3) report that they actually received it.On the other hand, a fraction of the schools in the control group (11%) and the T1 group (25%) also report receiving the list. 14  14 Since we have perfect control on the elaboration of the lists (and these lists were only produced for T2 and T3), we know that this is mostly capturing either measurement error, social desirability bias in the answers of school principals, or schools getting the list of other schools (it is unlikely than the list of students at-risk from another school would be useful in these schools).
( Finally, we assess the take-up of the reminders sent through SIRE.These reminders were activated in a sample of schools by an experienced technical team of the government using a specific software used to send alerts to subsamples of schools.The team had access to three pieces of information regarding the implementation of this component.First, using a guest id, the team verified that the reminders were posted according to the planned calendar.Second, the team had access to the list of schools that according to the SIRE team received the reminders.This list perfectly matched the treatment assignment.Third, the team had access to schools' SIRE login activity for the month that followed the activation of the reminders.We combined these last two data sets to proxy the (timely) take-up of the component.Specifically, we approximate the takeup of the reminders with a dummy that equals one if the school logged into SIRE within that month and the SIRE team reports that the school received the reminders.Based on this, almost 80 percent of the schools in T3 were exposed to the reminders in a timely manner, and there was no contamination to other arms. 15

LATE
As shown in the previous section, the take-up and contamination rates differ across treatment arms.
In this context, it is not straightforward to compute the LATE (Behaghel et al, 2014).To address this, we do a rough estimation of the LATE for the pooled model using participation in the training as a proxy for take-up (as described above).There are three main reasons to select this variable to capture overall participation in the program.First, the training/guide was probably the most relevant activity in terms of both budget and substance (as suggested also by the ITT estimates).
Second, other key elements of the program were distributed in the training, namely the Minister's letter and the list of students at-risk.Third, the quality of the data capturing participation in the training is more reliable than the data capturing participation in other components.16

Scalability and cost-benefit analysis
As the size of the RCT suggests, the program was designed to be implemented at scale by relying on existing MINEDUC staff, including trainers, teachers, and school principals, and at a very low cost per student benefited by the program (roughly USD 2 dollars).The training and overall implementation of the program was chiefly in the hands of MINEDUC, with minor support from the World Bank.Expanding ENTRE to additional grades would reduce the costs per student by approximately 76%, given that the training and materials for principals (which represent roughly half of the budget for an average school) are a "fixed cost" for the school, there are many multiplegrade schools and the number of teachers per student is substantially larger in the sixth grade than in the first five years of primary education.
The cost-benefit analysis was conducted using the J-PAL tool and following its cost guidelines. 17 As previously mentioned, the intervention is low-cost, with a total cost per student of USD 2.91, 36% of which corresponds to the design of the program and 45% to the opportunity cost of The impact of ENTRE is heterogeneous, with significantly larger effects for students at higher risk of dropout, students in large schools, and boys.In particular, ITT (LATE) estimates show that the program reduced dropout by about 2.9 (6.1) percentage points for the students with the highest risk, 2.1 (4.6) percentage points for students in large schools, and approximately 1.9 (4.4) percentage points for boys.
Our multi-arm design sheds some light on the potential mechanisms driving the results.The results indicate that the bulk of the impact seems to be explained by the component providing information on how to prevent dropout (as opposed to providing information on who is at-risk of dropping out or small behavioral nudges to try and keep dropout prevention top-of-mind for principals).In a setting like Guatemala, where school actors have limited knowledge of and support on dropout prevention strategies, and school principals-who play a leading role in these initiatives-are usually regular teachers without any specific leadership or management preparation, the training and guidelines on simple strategies to prevent dropout appear to be highly impactful.This component was also the most intensive in terms of time and resources (representing approximately 92% of the cost of the program).
Self-reported data from school principals also helps to further unpack the mechanisms driving the results.This data shows that about two months after the training, principals in treated schools perceived higher returns to schooling, ranked dropout as a higher priority, and provided more targeted support to at-risk students.The impact on targeted support and prioritization are not only large in quantitative terms but also persist in the next academic year (with a smaller but still economically and statistically significant impact).
Finally, the results suggest that school actors have gender-biased perceptions of who is at risk of dropout (significantly underestimating the relative likelihood of dropout for girls) and that providing objective information on these probabilities seems to have corrected these biases (and distributed the benefits of the program more evenly across boys and girls).This is an important suggestive result for the dropout prevention literature that calls for further research.

Figure
Figure 1: message example (first message)

(
component 3) to try and keep dropout prevention top-of-mind for principals and teachers.A fourth, control arm included schools that received no intervention.This multi-arm design, summarized in Diagram 1, allows us to disentangle the impact of each component on dropout.Diagram 1: four-experimental groups Data Our main data source is the administrative school data set compiled by SIRE.This data set includes unique student identifiers that has enabled longitudinal tracking of student academic records since 2011.The academic records include basic demographic information and detailed grades earned bysubject and year, as well as information on enrollment status, including whether the student dropped out and whether they passed the grade.This data was the main input for estimating the dropout prediction model, as well as the data source for the main outcome of interest (student level dropout).
test for the guidance manual and paper-based questionnaires at the end of the halfday training.

Figure 3 .
Figure 3. Impact of ENTRE on students' dropout rate in the experiment and the placebo

Figure 4 .
Figure 4. Heterogeneous impact of ENTRE on students' dropout rate in the experiment and the placebo

Notes: ( 1 )
Each coefficient in panel (1) corresponds to a different regression of treatment assignment on the change between baseline and follow-up in the outcome variable mentioned in the name of each raw.(2) Short-term refers to the change between the baseline and the first follow-up and Medium-term corresponds to the change between the baseline and the third follow-up.(3) Observations in panel (2) corresponds to the number of schools without missing data who were therefore used in the regressions.

Notes: ( 1 )
Standard errors in parenthesis.(2) *** p<0.01, ** p<0.05, * p<0.1.(3) GUIDEj: 1 = School j declared having received the guide in any format; LISTj: 1 = School j declared having received the list of students at-risk of dropout; REMINDERSj : 1 = School j entered the system to see the first reminder.(4) Assigned Treatment i j: (1= School j was in the treatment group i).(5) Controls at school level: Average characteristics of the students in the school (proportion of men, average risk of dropout and proportion of students of age11, 12, 13, 14, 15, 16, 17, 18, and 19 & more).(6) Regressions are without intercept and controls are centered at the mean.(7) Observations are weighted by school size.

Figure 5 .
Figure 5. Take-up of ENTRE components by the different treatment arms and the control group

O
Figure A1: Intervention timeline and activities

Table 1 .
Table 1 summarizes the main questionnaires used to collect data.Summary of data sources

Table 2 .
Baseline average characteristics of the students in control and treatment groups.Statistically significant difference with respect to the control group at 10%.** Statistically significant difference with respect to the control group at 5%. *** Statistically significant difference with respect to the control group at 1%. (2) Standard errors are clustered at school level.

Table 3 .
Impact of the program on the probability of student dropout ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group.5) All specifications in panel (i) include the corresponding Postt and Entrejt dummies, while those in panel (ii) include the Postt and Entre ijt dummies; 6) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System).
Notes: 1) Standard errors clustered at school-year level in parenthesis.2) *** p<0.01, ** p<0.05, * p<0.1.3) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1; 4) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year of the intervention; 0 = year before the intervention);

Table 4 .
In Table4, we report the same models but include interactions with variables to capture the presence of heterogeneous effects, namely: (i) the ex-ante probability of dropout of the students (based on the predictive model described above); (ii) the school size; and (iii) the gender of the student.As in the previous table, the first panel reports the pooled model, and the second panel the corresponding coefficients for the three arms. 12Heterogeneity in the impact of the program on the probability of student dropout.The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1.4) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year of the intervention; 0 = year before the intervention); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group.5) Interaction j in column (i) is given by the variable Risk sjt (Probability of dropout for student s that attends school j in year t estimated in the Early Warning System); in column (ii) by Large school sjt: (1=Large school: size in 2017 > median), and in column (iii) by Man sjt: (1=Man).6) All specifications in panel (i) include the corresponding Postt , Entrejt and Interaction j dummies, while those in panel (ii) include the Postt and Entre i jt dummies; 6) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System).
April 2018 and April 2019 (i.e., before and after the pilot was implemented), in our placebo experiments we compare them between April 2017 and April 2018 (i.e., one year before the implementation of the ENTRE pilot).If our main experiment is just picking up pre-existing trends, Observed: dropout risk Perception: dropout risk Perception: needs support

Table 5 .
Placebo test of the impact of the program on the probability of student dropout ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year 2017 in Placebo; = year 2018 in Experiment; 0 otherwise); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group.6) All specifications in panel (i) include the corresponding Postt and Entrejt dummies, while those in panel (ii) include the Postt and Entre ijt dummies; 7) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System).

Table 7 .
Impact of the program on different self-reported indicators at school level

Table 8 .
Regressions of take-up on treatment and control assignment

Table 9
reports the LATE estimates for the pooled model.The results indicate that ENTRE reduces dropout by 3.1 percentage points for students in complier schools (column 3 of Table9), roughly doubling the ITT estimates.As expected, the LATE estimations are significantly more imprecise, with the standard errors three times larger than those corresponding to the ITT.Table10shows a similar pattern of heterogeneous effects by dropout probability, school size, and gender.The results show that participating in the program reduces dropout by about 6 percentage points for the students with the highest risk, 4.5 percentage points for students in large schools, and approximately 4 percentage points for boys.

Table 9 .
Impact of the program on the probability of student dropout: LATE estimates.The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1.(5)ENTREj represents the interaction Entrejt x Postt where in this case Entrejt: (1= Student s attends a school j in year t that was treated) and Postt: (1 = year of the intervention; 0 = year before the intervention).(6)All specifications include the corresponding Postt and Entrejt dummies.(7) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System).

Table 10 .
Heterogeneity in the impact of the program on the probability of student dropout: LATE estimates.Notes: (1) Local Average Treatment effects were estimated by instrumenting treatment (proxied as the effective reception of the ENTRE guide according to self-reported data in follow-up 3) with treatment assignment.(2)Standarderrorsclustered at schoolyear level in parenthesis.(3)***p<0.01,**p<0.05,*p<0.1.(4)Thedependentvariableisadummyvariable that is equal to 1 if student s in school j dropped out school between year t and year t+1.(5)ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that was treated) and Postt: (1 = year of the intervention; 0 = year before the intervention).(6)Interactionj in column (i) is given by the variable Risk sjt (Probability of dropout for student s that attends school j in year t estimated in the Early Warning System); in column (ii) by Large school sjt: (1=Large school: size in 2017 > median), and in column (iii) by Man sjt: (1=Man).(7)All specifications include the corresponding Postt , Entrejt and Interaction j dummies.(8) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System).