This post in a nutshell: Are rankings of universities within Japanese society similar to rankings of universities used internationally? What components of international ranking scores track the most commonly used rankings in Japan?
The best-known means by which universities in Japan are ranked is called a hensachi score, which tells how far from the statistical mean a typical student admitted to a given institution scores on a test. A score of 50 is at the mean, 60 is one standard deviation above the mean, 70 is two SDs above the mean, etc. A high hensachi, say 65, for a given department at a given institution means students admitted do better than at least 95% of the test-taking population on that test (one SD turns out to be quite a leg up on one’s peers). Hensachi scores vary by department; a medical university like Juntendo will have a high hensachi for its medical department (72) and a much lower one for its nursing department (55). The private university I used to teach at has hensachi ranging from 35 to 65 for its various departments.
Source. Looks more complicated than it is.
On the other hand, colleges in the US are usually evaluated according to a range of measures. It’s not difficult to find out the acceptance rate at a given school, but more commonly seen are suggested minimum SAT and ACT scores as well as high school GPAs. Placing these on a normal curve is possible but not widely done.
More commonly, people rely on national and international rankings like that of US News and World Report or the Times Higher Education World University Rankings, the latter of which are the subject of this post. The THE Rankings caused a bit of a stir here recently as Japan’s universities underperformed in the list of the world’s top 980 universities, with only 2 institutions in the first 100 and many schools highly regarded in Japan falling rather high (as we shall see, the prestigious private MARCH schools – Meiji, Aoyama, Rikkyō, Chūō, and Hōsei all come in above [EDIT: meaning worse than] #800, or off the list altogether).
A common interpretation of the discrepancy between the ranking and prestige that Japanese universities enjoy locally and those they receive internationally is that Japanese universities serve some local societal need that is not addressed, or at least is underrepresented, in the THE Rankings. The most obvious of these is simply sorting: Japanese universities are a final stamp on a long process of streaming by test scores in order to serve the efficient hiring by companies after college, which justifies the use of a single number (hensachi) to rank universities and explains some other mysterious practices like much of 4th year attendance being superceded by job hunting. Of course, this thinking would justify any allotment of hensachi rankings; a more prestigious school is more selective and therefore more prestigious, and a selective school is more prestigious and therefore more selective; the hensachi of a school telling employers all they need to know about its graduates. Other local needs that have been proposed as explanations of this gap include the necessarily limited local market for research in the vernacular (including in-house university publications; I haven’t been able to find any of these in the database that the THE Rankings use to compile their Citations and Research scores), the lack of direct industry ties among Japanese universities, lack of weighting for patents (my ex-employer was particularly well-known for this in Japan), and a lack of outreach apparati for recruiting foreign students and staff. This last one seems to be a result of misplaced priorities than institutional disadvantage.
The components do not all contribute equally to the THE Ranking. Notably, Teaching and Research are less weighted this year than in previous years, a fact which this article blames for Tokyo U’s drop in this year’s rankings. As we shall see, these two areas are relatively strong for Japanese universities.
In the interests of finding out just how much the THE Rankings correlated to Japan’s internal rankings of universities, I went through the THE Rankings for Japanese universities and recorded the component scores that contributed to their final rankings. I then figured out the average (median unless there were 3 or fewer departments, then mean) hensachis across all departments for those universities and entered them in a Numbers file, using this site to find hensachis for university departments. Then I computed the correlations between the component scores, the THE Rankings, and the hensachis for those universities. I only included schools ranked 800 or lower [EDIT: meaning better than 800]; including the final group of 801-980 would have doubled the amount of work. Sorry.
Not exactly a creative use of statistics, but it involved a lot of copying and pasting and you know, it’s the kind of thing you expect someone to do. So I did it. You can see the raw numbers in the following graph.
Means: Teaching 32.2, Research 23.4, Citations 33.4, Int’l Outlook 23.9, Industry Income 47.9. THE Rankings are given as averages of the ranges given, for example 601-800 is entered as 700.
The overall correlation between the THE Rankings and hensachis was -0.5778. For only schools in the THE top 500, the correlation was -0.6128. These numbers are negative because, as should be expected, rank falls as hensachi rises. The closer a university is to #1 ranking, the harder it is to get into. For schools ranked above [EDIT: i.e. worse than] #500, there is almost no correlation between ranking and hensachi: -0.1339. This should not be a surprise, as almost all of those universities have identical ranking scores.
Immediately we can see that hensachis do not track the THE Rankings exactly, and sometimes are curiously divergent. Schools that are very difficult to get into, for example Waseda University with its median hensachi of 69 (rejecting nearly 98% of all applicants) [EDIT: not really rejecting of course, just out of the reach of. Students at -2 SDs presumably don’t apply to Waseda.] is ranked by THE as just as strong as Toyohashi University of Technology, which with its hensachi of 50 gives students even odds of getting in. On the other hand, another school with the same hensachi of 69, Osaka University, is ranked just out of the top 200. A student with much better-than-average scores, according to THE, would be much better off choosing Osaka. On the other hand, a completely average student could get into a school which the THE considers just as good as Waseda, which in Japan has a level of prestige similar to Princeton or Stanford in the USA.
Correlations, positive and negative
So what about the components of the THE Rankings? How do those compare with universities’ hensachis?
This next graph has the correlations of each component score with hensachi:
Teaching: 0.7368, Research: 0.6833, Citations: 0.2492, Int’l Outlook: 0.4683, Industry Income:0.4145
And only for universities within the best 500:
Teaching: 0.8527, Research: 0.7785, Citations: -0.4863, Int’l Outlook: 0.3218, Industry Income: 0.3576
And above [EDIT: meaning not in the top] 500:
Teaching: 0.3738, Research: 0.2289, Citations: 0.3258, Int’l Outlook: 0.1926, Industry Income: -0.2048
To reiterate, a hensachi tells us how difficult it is to get into a university; it is not a direct measure of quality. It tells us how far above average a student has to be to have a chance of getting in, which depends on how many students try to get into a university and how many seats the university has available. A high hensachi is a measure of selectivity, which is at best an indirect indicator of quality.
Out of the components that make up the THE Rankings, Teaching and Research are by far the most correlated with hensachi, across all spectra of Japanese universities in the THE list. Among the top-ranked 500 schools only, Teaching and Research have near-perfect correlations with hensachi. This supports the conclusion that Teaching and Research are responsible for much of any university’s competitiveness in Japan, as represented in its hensachi.
Interestingly, Citations are negatively correlated with hensachi among the Japanese universities in the THE top 500. Universities whose research is cited more often tend to be less selective of incoming students, or students are dissuaded by the type of academic strength shown in numbers of citations when deciding which schools to try to gain admission to. This is not the same as Citations being irrelevant; that would show up in a correlation of near 0. I have no idea why Citations would correlate negatively with hensachi. Hopefully someone in the comments can suggest a reason.
Industry Income is negatively correlated with hensachi in universities outside the top 500. Correlations are all weaker outside the top 500, suggesting that objective (at least as far as the THE Rankings’ component scores reflect objective evaluations) evaluations matter less as those evaluations drop. Still, I await explanation for why Industry Income could be negatively correlated with hensachi.
Among the cream of the crop, hensachi can generally be a heuristic for quality of undergraduate experience. The main part of a 4-year student’s experience of a university is in its teaching, and hensachi correlates very strongly with the Teaching component of the THE Rankings. However, among weaker schools by THE standards, hensachi fails to predict quality as strongly in any domain. Importantly, this applies to weak schools as determined by THE, not by hensachi: many schools with high hensachi are ranked above [EDIT: worse than] #600 by THE, including Waseda, Keio, and the aforementioned MARCH schools. For Japanese high schoolers looking at prospective undergraduate universities, my advice is to choose the lowest hensachi school from the THE top 500, like Tokyo Metropolitan University or University of Tsukuba.
For graduate students, hensachi is a much less useful tool for assessing a university’s strength, as Research, Citations, and Industry Income are likely to be much more important to students conducting original research. Of course, I have no idea if prospective graduate students in Japan look at their universities’ hensachi at all; hensachi are computed for the difficulty of gaining admission to undergraduate departments.
Of course this assumes that quality matters. If students are looking at university is mostly a stepping stone toward a prestigious job in Japan, one may be inclined to ignore THE Rankings altogether and regard only Japan-specific rankings as relevant. If one looks at the hensachis of the aforementioned MARCH schools along with Waseda and Keio not as reflections of the numbers of students wanting to be educated at those institutions but instead as reflections of the numbers of students wanting the career opportunities having been admitted to one of those institutions grants you, they make sense regardless of the objective academic strength of those schools. High-hensachi but low-THE-ranked universities can be seen as expensive and elaborate filters of workers for Japanese companies, just as Brian McVeigh suggests of Japanese universities as a whole.
There is an element of self-fulfilling prophecy to “prestige” in Japanese universities which can be seen especially strongly in high-hensachi private universities. Bearing in mind what I said about Japanese high schools a while ago, it may be worth attending one of these universities if only for the opportunity to spend 4 years with like-minded peers. However, if most of these peers are at that school for the career boost and not for the education, you will essentially be placing yourself in a Community of Practice for careerism and not for cultivation of the mind. This may account for the low THE Rankings of some schools whose student bodies are clearly very devoted to some goal; that goal may just not be higher education as THE understands it.
For those of you who skipped to the bottom:
The data suggests that hensachi is not completely unmoored from educational quality, but that some forces besides educational quality also drive hensachi upwards or downwards.