Opinion14th March 2017
James O’Donoghue works for the Ark Education Partnerships Group, leading on work in Uganda with the Ugandan government and UK Department for International Development on school accountability and public-private partnership research and reforms.
On a recent work trip to Uganda for the Ark Education Partnerships Group, my arrival coincided with the release of the results of the national, public secondary examinations – the Uganda Certificate of Education (UCE). Each year this event is heralded with quite a lot of pageantry; the chairman of the national examinations board presents a hefty dossier to the Minister for Education amidst much media coverage. The newspapers then release dedicated supplements of full league tables with the performance of every school. So far, so reasonable and transparent.
And, evidently, so interesting. Returning to my Kampala hotel one evening, I found the hotel night askari (guard) poring over a newspaper league table supplement, trying to find his alma mater (in Tororo District, in eastern Uganda right on the border with Kenya). He wanted to know how his old school had performed in relation to the others in his district. This may have had more to do with re-visiting old school rivalries than informing where he might send his children. No matter how he’d use the information, however, the key point was that he wanted to know.
Unfortunately for our askari – taking an interest in school performance during the warmth of a long Kampala evening – what the newspaper told him was not particularly useful. League tables are only as good as the information they present. This is where Uganda’s transparent league tables fall down; they’re not so transparent after all.
Traditionally in the usual Ugandan league tables, schools are ranked by the number of students who achieve a ‘division one’ in UCE – the highest grade out of four (each being an average aggregated score across eight subjects). Ranking only by how many children achieve a division one may be simple, but is so crude as to be nearly useless. A school where a single student achieved a division one but the rest got division fours or failed would be ranked higher than a school where all children achieved a division two and none failed. In fact, with a division one being genuinely quite difficult to achieve, it means many schools appear on the league tables with simply a ‘zero’, meaning there is nothing else to differentiate between their performance.
Our night guard’s school was one of these. Being so disproportionately skewed toward the top grade makes it nearly impossible for him to develop a true understanding of how well his former school actually performed. More difficult still for parents trying to choose schools for their children; for the national inspectorate to decide how to deploy their limited resources to visit and monitor schools; or for government to direct constrained funding and support in the most effective and equitable way.
As well as not being helpful, it’s not fair. Schools which take students with better grades at entry are more likely to be recognised and lauded, thereby encouraging schools to select on ability. Further, schools are then encouraged to put their effort and resources into those children perceived as capable of achieving a division one, to the detriment of other learners. On the other hand, schools which take students with poorer performance on entry, but which help them to make good progress (even if they don’t reach the hallowed division one) are ignored.
It’s worth noting here that in Uganda, the newspapers and not the government produce the league tables. Therefore, the league tables aren’t strictly part of an official accountability framework for schools. However, that they are produced using publicly available government data and are the main source of information to the public on school performance, means that they are de facto an essential part of the current Ugandan school accountability framework; and a part which parents, the government and schools themselves pay a lot of heed to.
Therefore, there is an excellent opportunity to improve this system to make it more useful for all those with a stake in it. The Ark Education Partnerships Group is working with the Ugandan National Examinations Board to address the weaknesses in how school performance is measured and presented. We’ve helped them to develop a metric whereby schools’ performance in the national secondary exams is measured by the amount of progress students make between entering and leaving secondary school – the amount of value added to a student by a school. Schools get credit when a student performs better than expected, given their prior attainment.
Under this method, a student’s success in school is not defined by meeting a simplistic – and often unobtainable – threshold of performance, but by how well they progress in their learning between arriving at and leaving secondary school. Further, the development of every child matters for a school, because the exam results of each child impact the school’s overall progress score. Consequently, schools are measured by whether they help all students make progress or not.
This is helpful. Askaris – and parents – can get a better idea of whether their child would be likely to make good progress in their learning, should they attend a given school. Schools themselves can analyse the results data to see which groups of students make the most or least progress in which subjects, and then act accordingly. Lastly, governments can use this analysis as part of a ‘top-down’ accountability system to see which schools are adding value to students and which are not. This can then help them to inform where to deploy finite school inspection, monitoring and improvement resources to create maximum impact in the most equitable way, while also getting value for money.
This ‘school effectiveness measure’, as it is called in Uganda, is similar to the recently launched ‘Progress 8’ measure in the UK and is in the vanguard of school accountability reforms globally. Last year the Ugandan exam board released the top 100 schools as measured by this metric and it plans to release national rankings based on the latest public exam results in the near future. As the role of the newspapers and media is crucial in the dissemination of this information, we’re also working closely with them to make sure this new measurement is communicated clearly so the public understand and can use it.
League tables are not uncontroversial, certainly, and there have been accusations that they create perverse incentives for schools to try to ‘game’ the system. This is all the more reason to make sure they present accurate and fair information on robust metrics, such as student progress.
Moreover, there is evidence to show league tables work. The UK has been publishing school league tables since 1992 when John Major’s government launched them as part of the Citizen’s Charter; ‘Progress 8’ is the latest iteration of these. The initial rationale was that they would help create ‘bottom-up’ accountability; better-informed parents making better-informed decisions about where to send their children to school, thereby exposing schools to some market pressures to increase performance (as funding follows students in the UK, parental decisions to withdraw their children can have real consequences for financing to schools).
In 2001, this hypothesis was put to the test. The newly devolved Welsh government decided to abolish public league tables. This created a ‘natural’ control group, against which to measure the impact of league tables on English schools still included in them. What was found was fascinating. Abolishing league tables had a significant negative impact on school performance; the effect was equivalent to adding an extra eight students to every class. Beyond this, the lowest performing schools were the worst affected, thereby suggesting the reform increased inequality. And if we are to take the (albeit not unquestionable) hypothesis that the lowest-performing schools serve the poorest, most challenged areas, then it could be argued the reform also increased inequity.
Evidence from the rest of the world is less clear-cut. However, it does suggest that the active dissemination of performance information to parents combined with some form of follow-up (such as encouragement to parents to use the information, or government sanctions and rewards) can result in school improvement. Research in Pakistan showed how provision of performance data to parents resulted in improved enrolment and test scores – as well as lower fees, an interesting case for league tables driving greater equity. A study in Mexico found that a low-stakes publication of performance data was itself sufficient to prompt improvement among low-performing schools.
It’s early days for this approach to measuring school effectiveness in Uganda and we’re yet to see if it will work. It will need to be evaluated properly to see whether this form of presenting school performance information prompts the positive behaviour change from parents, schools and government officials, which are the real drivers of school improvement. None of this is easy or guaranteed. There will also undoubtedly be some bumps in the road; not least opposition from stakeholders with vested interests in the apple cart not being upset, such as schools which benefit from being able to cherry-pick the highest performing students on entry.
Transport and fruit-based clichés aside, when these reforms are proven and fully enacted, our hotel askari – and others like him all over Uganda – will have more accurate information to make a better-informed decision about where to send his children to school. He will have an expectation that those schools will value the progress that all his children make in their learning, regardless of their starting point. He’ll have the information he needs to be able to hold his children’s schools accountable. Lastly, he’ll know his government has better information to do the same. The endgame is better schools and improved student learning. If these accountability reforms help to achieve this, it’ll definitely be worth a few spilt apples and bumps in the road.