Weightings & Normalization
Otherwise referred to as standardization or z-score aggregation, this page explains in detail how raw scores are converted into the final scores presented in our tables
Designing a ranking on a single index - such as a rich list - is relatively straightforward. Compiling a multi-index ranking is a little more complex. In combining the indices for the Times Higher - QS World University Rankings the following guiding principles have been followed.
- Fair and even application of weighting across the whole range for each indicator
- Intuitive and comparable scores for each of the six indicators
- Great strengths in particular broad subject fields should contribute to the overall position of an institution
The information on this page explains in details the steps taken to get from our original scores to the final socres presented in our tables.
Weightings
The allocation of weightings for the THE - QS World University rankings, remains the responsibility of the team at Times Higher Education. The current weightings - assigned by indicator - are presented in Figure 1 below: Academic Peer Review 40%, Employer/Recruiter Review 10%, Student Faculty Ratio 20%, Citations per Faculty 20% and International Factors 5% each.
It might be fair to suggest, then, that Times Higher Education consider Research Quality to be be 3 times more important than Teaching Quality in the mission, and therefore evaluation, of world-class universities. The appropriate allocation of weightings has to take into account a little more than simply the importance of the criterion being measured, it also has to consider the appropriateness of the indicators to evaluate that criterion.
- Figure 1: Weightings by Indicator
- Figure 2: Weightings by Criteria
In the future, should additional effective indicators for Teaching Quality or Graduate Employability be identified, it seems likely that they will be included at the cost of the Research Quality indicators - further differentiating this evaluation from others.
Normalisation
Once the data is collected and the weightings are decided upon, the next thing to do is to calculate standard scores for each column of data so that thy are compatible wth each other andalow us to combine the data reliably and apply the weigtings fairly in the calculation of the overall score. Before 2007, the approach taken here was an over-simplistic one - find the top scoring institution, award them 100 notional points and scale the remaining entries proportionally to that top performer. This approach had some disadvantages:
- Anomalous application of weightings
- Lack of control for "outliers"
- The smallest of errors in the assessment of the top performing institution in any indicator could have dramatic effects
From 2007, a more complicated, but widely used standarization or normalization method has been adopted involving z-scores. There are numerous online sources explaining how this works: -
Wikipedia - http://en.wikipedia.org/wiki/Standard_score
UCLA - http://www.gseis.ucla.edu/courses/ed230a2/notes/z1.html
There are even a range of video podcasts available on YouTube: -
Z Scores Explained - http://uk.youtube.com/watch?v=1xhCL5m4nI0
Calculating Z Scores - http://uk.youtube.com/watch?v=s0lLBcARxL4
In order to calculate z-scores the mean and the standard deviation of the sample are required. These are as follows...
| Indicator | Mean | Standard Deviation |
|---|---|---|
| Academic Peer Review | 102.73 | 71.75 |
| Employer Review | 90.02 | 15.57 |
| Student Faculty | 0.09 | 0.05 |
| Citations per Faculty | 24.93 | 26.48 |
| International Faculty | 0.14 | 0.14 |
| International Students | 0.11 | 0.09 |
To prepare for the application of z-scores the a natural log is applied to the raw data for the given indicator, be it a weighted total from the peer review or an (inverted) student faculty ratio, to draw in the outliers, and once the scores are calculated, their position on the normal curve is plotted resulting in their score for each indicator.
Compiling the Final Scores
Compiling the final score is relatively straightforward. We simply multiply each indicator score by its weighting factor, sum the size resulting figures together, round to one decimal place and then scale to the top performing institution, resulting in a final score out of 100.
