Reports  Clients  Testimonials  Resources  Order  FAQ  About



Is Your School Comparing Apples to Oranges?

Many schools take their average diploma score, as reported on their IBIS School Statistics report, and compare it to the worldwide average presented in the IBO Statistical Bulletin. That's not surprising, given that those are the only averages readily available to schools.[1]

However, such a comparison is often "apples to oranges" and can be misleading. This article will explain the issue and how we resolve it.


The Issue

Let's look at an actual example involving May results from Acadamigo International School.
(Although the name of the school is fictitious, the data presented are real.)

As the table shows, it appears that Acadamigo International School has been outperforming the world by more than 3 diploma points each year.[2]

However...

The school's average diploma score, as presented on their IBIS School Statistics report, is qualified. Specifically, the average is clearly stated to be the...

"Average points obtained by candidates who passed the diploma." [emphasis added]

In contrast, the worldwide average diploma score, as presented in the IBO Statistical Bulletin, includes all candidates, even those who did not pass the diploma.

Thus, the comparison shown in the table above -- the comparison made by many schools -- is actually between:

Successful candidates at the school (first row of data), and
Successful and unsuccessful candidates worldwide (second row of data).

Given that Acadamigo International School had candidates who do not earn the diploma because they scored too few points, that's obviously not a fair comparison.

There's nothing intrinsically wrong with a school looking at the average diploma score of successful candidates only. But that average shouldn't be compared to a worldwide average that includes unsuccessful candidates. That's comparing apples and oranges, and doing so artificially inflates the school's performance relative to the world.


The Resolution

A correct comparison would use either of these pairs:

School average -- all candidates, successful and unsuccessful
World average -- all candidates, successful and unsuccessful

Or

School average -- successful candidates only
World average -- successful candidates only

Our own IB Score Reports have always used the first pair of averages, those that include all candidates, both successful and successful, rather than just successful.

The primary reason is simply that it makes intuitive sense to us. When I report the average of an exam I give in my math class, I don't exclude those who failed. (And students, parents, and administrators would be surprised were I to exclude those who failed when reporting the average.) Similarly, the IBO doesn't remove students who earned 1's or 2's or 3's when calculating the average for a subject. So our thinking from the start was, why do anything different for diploma scores?

Thus, our IB Score Reports would present Acadamigo International School's results as follows.

The new table shows that Acadamigo International School has been outperforming the world by 2.5 to 3.1 points each year, which is a bit lower than before, although still quite impressive.

Most importantly, the comparison is now fair: both the school average and the worldwide average are based on all candidates, both successful and unsuccessful.


Successful Candidates Only

The other way to construct a fair comparison would be to use the school average and worldwide average of successful candidates only.[3] Like so:

Now, when the fair comparison is made between successful candidates only, the data show that the school has typically underperformed against the world!

The explanation for this surprising result is simple:

Schools generally have just a few candidates who are not awarded the diploma. Removing those candidates raises the school's average, but only a small amount. For Acadamigo International School, the difference was always less than 1 point, as we can see in the following table:

But where the worldwide average is concerned, removing unsuccessful candidates tends to raise the average by about 4 points!

To review:

Generally speaking, the average diploma score for all May candidates worldwide has been about 30. And generally speaking, the average diploma score worldwide for May candidates who earned the diploma has been about 34.

Excluding unsuccessful candidates from a school's average tends to raise their average a small bit, while excluding unsuccessful candidates from the world average tends to raise the world average quite a bit.

Thus, our client schools fare better when comparing their average of all candidates against the world average of all candidates, rather than comparing their average of successful candidates against the world average of successful candidates.

Specifically, the difference between a school's average and the worldwide average will be greater if we compare all candidates in each case, rather than just successful candidates.

(In the case where a school doesn't have any unsuccessful candidates, "removing" them will obviously leave their average unchanged. Even then it will be best for the school to have the comparison be between all candidates rather than just successful candidates.)


November Results

So far, the discussion has only addressed May results. What about November results?

Essentially, everything said about May averages holds for November averages as well. As the following table shows, the worldwide average of successful November candidates is about 4 points higher than the worldwide average of all November candidates.

[Note: The November 2015 Statistical Bulletin had not been published at the time this article was written.]


Summary

In summary, many schools are making the following comparison:

School's average from IBIS report
World average from Statistical Bulletin

Which is actually this apples-to-oranges comparison:

School average -- successful candidates only
World average -- all candidates, both successful and unsuccessful

Instead, one of the following comparisons should be used:

School average -- All candidates
World average -- All candidates

School average -- successful candidates only
World average -- successful candidates only

We use the first in our IB Score Reports for two simple reasons: it makes sense and schools fare better in their comparison against the world.


One Drawback

One unavoidable consequence of our approach is that "all candidates" includes diploma candidates who receive scores in some, but not all subjects.

Admittedly, a better approach would be to include all and only those diploma candidates who have scores for all subjects, plus TOK and EE. Unfortunately, the IBO doesn't disaggregate worldwide diploma scores in that way, and so our choices for the worldwide comparison norm are just "All Candidates" (where that really does include candidates who earn 0, 1, 2, etc. total points) or "Successful Candidates Only." As we've explained, we prefer the former rather than the latter.


Notes

1. The IBO Statistical Bulletin is a publicly available document of aggregate data and results published approximately six months after each exam session. In contrast, the IBIS School Statistics report is a private report of data and results pertaining to an individual school. (Return to text)

2. The worldwide average diploma scores presented in the May IBO Statistical Bulletins in 2011 through 2014 are 1/10th of a point higher than the averages that we calculate using the distribution of diploma scores presented in the same Statistical Bulletins. In 2015, the reported average matched our calculated average. Traditionally, there has been no such discrepancy in the November Statistical Bulletins. The reported diploma averages match the calculated averages for 2011, 2012, and 2013. However, in 2014, the reported diploma average was 31.2. In contrast, our calculated average (again, based on the distribution of scores presented in the same Statistical Bulletin) is 30.8. Whenever there is a discrepancy between the reported and calculated average, our IB Score Reports use the lower value, because that's more favorable to our client schools. (Return to text)

3. The worldwide average of diploma scores of successful candidates only is not explicitly presented in the IBO Statistical Bulletins. We calculated it for each year and session using the distribution of scores presented in the relevant Statistical Bulletin. The calculated value for May 2014 (32.2), while strangely low, is correct for the distribution of scores presented in the IBO Statistical Bulletin for that session. (Return to text)


Would You Like To Have IB Score Reports For Your School?

Getting started is easy. Just click here to send us an email: support@acadamigo.com