This post in a nutshell: Are rankings of universities within Japanese society similar to rankings of universities used internationally? What components of international ranking scores track the most commonly used rankings in Japan?
Introduction
The best-known means by which universities in Japan are ranked is called a hensachi score, which tells how far from the statistical mean a typical student admitted to a given institution scores on a test. A score of 50 is at the mean, 60 is one standard deviation above the mean, 70 is two SDs above the mean, etc. A high hensachi, say 65, for a given department at a given institution means students admitted do better than at least 95% of the test-taking population on that test (one SD turns out to be quite a leg up on one’s peers). Hensachi scores vary by department; a medical university like Juntendo will have a high hensachi for its medical department (72) and a much lower one for its nursing department (55). The private university I used to teach at has hensachi ranging from 35 to 65 for its various departments.
On the other hand, colleges in the US are usually evaluated according to a range of measures. It’s not difficult to find out the acceptance rate at a given school, but more commonly seen are suggested minimum SAT and ACT scores as well as high school GPAs. Placing these on a normal curve is possible but not widely done.
More commonly, people rely on national and international rankings like that of US News and World Report or the Times Higher Education World University Rankings, the latter of which are the subject of this post. The THE Rankings caused a bit of a stir here recently as Japan’s universities underperformed in the list of the world’s top 980 universities, with only 2 institutions in the first 100 and many schools highly regarded in Japan falling rather high (as we shall see, the prestigious private MARCH schools – Meiji, Aoyama, Rikkyō, Chūō, and Hōsei all come in above [EDIT: meaning worse than] #800, or off the list altogether).
A common interpretation of the discrepancy between the ranking and prestige that Japanese universities enjoy locally and those they receive internationally is that Japanese universities serve some local societal need that is not addressed, or at least is underrepresented, in the THE Rankings. The most obvious of these is simply sorting: Japanese universities are a final stamp on a long process of streaming by test scores in order to serve the efficient hiring by companies after college, which justifies the use of a single number (hensachi) to rank universities and explains some other mysterious practices like much of 4th year attendance being superceded by job hunting. Of course, this thinking would justify any allotment of hensachi rankings; a more prestigious school is more selective and therefore more prestigious, and a selective school is more prestigious and therefore more selective; the hensachi of a school telling employers all they need to know about its graduates. Other local needs that have been proposed as explanations of this gap include the necessarily limited local market for research in the vernacular (including in-house university publications; I haven’t been able to find any of these in the database that the THE Rankings use to compile their Citations and Research scores), the lack of direct industry ties among Japanese universities, lack of weighting for patents (my ex-employer was particularly well-known for this in Japan), and a lack of outreach apparati for recruiting foreign students and staff. This last one seems to be a result of misplaced priorities than institutional disadvantage.
The components do not all contribute equally to the THE Ranking. Notably, Teaching and Research are less weighted this year than in previous years, a fact which this article blames for Tokyo U’s drop in this year’s rankings. As we shall see, these two areas are relatively strong for Japanese universities.
In the interests of finding out just how much the THE Rankings correlated to Japan’s internal rankings of universities, I went through the THE Rankings for Japanese universities and recorded the component scores that contributed to their final rankings. I then figured out the average (median unless there were 3 or fewer departments, then mean) hensachis across all departments for those universities and entered them in a Numbers file, using this site to find hensachis for university departments. Then I computed the correlations between the component scores, the THE Rankings, and the hensachis for those universities. I only included schools ranked 800 or lower [EDIT: meaning better than 800]; including the final group of 801-980 would have doubled the amount of work. Sorry.
Numbers
Not exactly a creative use of statistics, but it involved a lot of copying and pasting and you know, it’s the kind of thing you expect someone to do. So I did it. You can see the raw numbers in the following graph.
The overall correlation between the THE Rankings and hensachis was -0.5778. For only schools in the THE top 500, the correlation was -0.6128. These numbers are negative because, as should be expected, rank falls as hensachi rises. The closer a university is to #1 ranking, the harder it is to get into. For schools ranked above [EDIT: i.e. worse than] #500, there is almost no correlation between ranking and hensachi: -0.1339. This should not be a surprise, as almost all of those universities have identical ranking scores.
Immediately we can see that hensachis do not track the THE Rankings exactly, and sometimes are curiously divergent. Schools that are very difficult to get into, for example Waseda University with its median hensachi of 69 (rejecting nearly 98% of all applicants) [EDIT: not really rejecting of course, just out of the reach of. Students at -2 SDs presumably don’t apply to Waseda.] is ranked by THE as just as strong as Toyohashi University of Technology, which with its hensachi of 50 gives students even odds of getting in. On the other hand, another school with the same hensachi of 69, Osaka University, is ranked just out of the top 200. A student with much better-than-average scores, according to THE, would be much better off choosing Osaka. On the other hand, a completely average student could get into a school which the THE considers just as good as Waseda, which in Japan has a level of prestige similar to Princeton or Stanford in the USA.
Correlations, positive and negative
So what about the components of the THE Rankings? How do those compare with universities’ hensachis?
This next graph has the correlations of each component score with hensachi:
And only for universities within the best 500:
And above [EDIT: meaning not in the top] 500:
To reiterate, a hensachi tells us how difficult it is to get into a university; it is not a direct measure of quality. It tells us how far above average a student has to be to have a chance of getting in, which depends on how many students try to get into a university and how many seats the university has available. A high hensachi is a measure of selectivity, which is at best an indirect indicator of quality.
Out of the components that make up the THE Rankings, Teaching and Research are by far the most correlated with hensachi, across all spectra of Japanese universities in the THE list. Among the top-ranked 500 schools only, Teaching and Research have near-perfect correlations with hensachi. This supports the conclusion that Teaching and Research are responsible for much of any university’s competitiveness in Japan, as represented in its hensachi.
Interestingly, Citations are negatively correlated with hensachi among the Japanese universities in the THE top 500. Universities whose research is cited more often tend to be less selective of incoming students, or students are dissuaded by the type of academic strength shown in numbers of citations when deciding which schools to try to gain admission to. This is not the same as Citations being irrelevant; that would show up in a correlation of near 0. I have no idea why Citations would correlate negatively with hensachi. Hopefully someone in the comments can suggest a reason.
Industry Income is negatively correlated with hensachi in universities outside the top 500. Correlations are all weaker outside the top 500, suggesting that objective (at least as far as the THE Rankings’ component scores reflect objective evaluations) evaluations matter less as those evaluations drop. Still, I await explanation for why Industry Income could be negatively correlated with hensachi.
Discussion
Among the cream of the crop, hensachi can generally be a heuristic for quality of undergraduate experience. The main part of a 4-year student’s experience of a university is in its teaching, and hensachi correlates very strongly with the Teaching component of the THE Rankings. However, among weaker schools by THE standards, hensachi fails to predict quality as strongly in any domain. Importantly, this applies to weak schools as determined by THE, not by hensachi: many schools with high hensachi are ranked above [EDIT: worse than] #600 by THE, including Waseda, Keio, and the aforementioned MARCH schools. For Japanese high schoolers looking at prospective undergraduate universities, my advice is to choose the lowest hensachi school from the THE top 500, like Tokyo Metropolitan University or University of Tsukuba.
For graduate students, hensachi is a much less useful tool for assessing a university’s strength, as Research, Citations, and Industry Income are likely to be much more important to students conducting original research. Of course, I have no idea if prospective graduate students in Japan look at their universities’ hensachi at all; hensachi are computed for the difficulty of gaining admission to undergraduate departments.
Of course this assumes that quality matters. If students are looking at university is mostly a stepping stone toward a prestigious job in Japan, one may be inclined to ignore THE Rankings altogether and regard only Japan-specific rankings as relevant. If one looks at the hensachis of the aforementioned MARCH schools along with Waseda and Keio not as reflections of the numbers of students wanting to be educated at those institutions but instead as reflections of the numbers of students wanting the career opportunities having been admitted to one of those institutions grants you, they make sense regardless of the objective academic strength of those schools. High-hensachi but low-THE-ranked universities can be seen as expensive and elaborate filters of workers for Japanese companies, just as Brian McVeigh suggests of Japanese universities as a whole.
There is an element of self-fulfilling prophecy to “prestige” in Japanese universities which can be seen especially strongly in high-hensachi private universities. Bearing in mind what I said about Japanese high schools a while ago, it may be worth attending one of these universities if only for the opportunity to spend 4 years with like-minded peers. However, if most of these peers are at that school for the career boost and not for the education, you will essentially be placing yourself in a Community of Practice for careerism and not for cultivation of the mind. This may account for the low THE Rankings of some schools whose student bodies are clearly very devoted to some goal; that goal may just not be higher education as THE understands it.
For those of you who skipped to the bottom:
The data suggests that hensachi is not completely unmoored from educational quality, but that some forces besides educational quality also drive hensachi upwards or downwards.
Mark: My words 
I do not think that Brian McVeigh has been fair in his criticisms of Japanese universities. I should like to see more rigorous academic analysis on local universities’ different strengths and weaknesses, and rankings of quality on those terms. I believe that foreign critiques usually come from foreign academics who have been working in language and humanities or social studies or possibly communications departments, which may not be Japanese higher-ed strengths as a whole, and the insights and experiences of such academics or writers tend to be skewed as a result. As for higher hensachi being tied to expensive schools, not quite true, since highest hensachis are attached to the public and imperial universities. I think intellectual rigour and educational quality generally does tie in with the reputation and standing of the ranking school… But again, it depends on the department and discipline. Humanities students are especially notorious for being lax, absenteeism or wild fun pursuits. However , this distinction between humanities students, is in fact, not much different from the scenario in UK higher education. As for criticisms of careerism, yes and no, it’s like two sides of the same coin, perhaps Japanese universities would see better THE rankings if they placed more weight on universities’ rankings matching job placements with the marketplace. Afterall, there are often criticisms of too high rates of top ranking US and UK colleges’ grads not being employed while aaddled with a huge debt. Perhaps, you could say the local situation is doing something right here. I know that the Science and Engineering courses of MARCH universities, for example, are extremely rigorous, the classes are actually advanced, tough and there is no room for slackness, and a high non-attendance rate guarantees failing the year and non-advancement to the next year, and consequently failure to graduate. Completion of of assignments is also an important component, and the assignments have to be handed in on time, with no excuses, at least from my experience. Again, students report that rigour and slackness varies for different faculties. English literature students are more notorious for no-shows or partying through schoollife. Certain courses like literature, psychology or criminology etc., are text oriented or the research may be involve more autonomy or the studies somewhat self-study-dependent, so that student life tends to be more fluid and autonomous, and such students can seem to be less accountable or academically involved. On another but related tangent, school club life, the social life and social skills aspects of college club activity-life, while not strictly related to the students’ course of study are considered important, in a sense, like British schools, this is the place to network and build your “old boys'” ties for the future information exchanges. Whether this qualifies as careerism I don’t know, but these networking skills and networks are acknowledged as an important aspect and asset that enhances the value of one’s education, traditionally even more so for Japanese society, although there have been concerns that this “strength” or “value” of highered could be in the process of being eroded. I think that up to the ’70s, before the THE paper-arms race, it could be argued that the idea of higher institutions as a collegial clearinghouse of networked intellectuals was perhaps closer to the Japanese model, than the current one of marketplace specialist students chasing THE ranked universities. Just because it’s a filtered system doesn’t necessarily mean that networked system is less valuable or that the educational quality is lower, the latter allegation by McVeigh, imho, hasn’t been proven convincingly, but more in his cherry-picked anecdotal fashion, especially since he diid not name names nor mention specifics of types of disciplines.
LikeLike
Thanks for the comment. I agree that McVeigh’s book should be read mostly as an interesting angle on the meaning of higher education supported by mostly personal experience. I also agree that much criticism of Japanese universities by English-speakers who work in them comes from the humanities. I don’t suppose that there is much mystery as to why – I found myself wondering how much of the THE International Outlook rankings for universities in Japan were inflated by the presence of native-speaking foreign language teachers, which can’t be taken at face value as evidence of internationalism.
I was unaware of the weakness of humanities departments in Japan, although I am of course aware that some people consider them expendable.
I don’t believe that hensachi is correlated positively or negatively with expense. I just say that low-THE-ranked but high-hensachi schools may justify their price and exclusivity through other means.
LikeLike
This seems way too defensive. (Scapegoating the humanities?)
LikeLike
To the extent that research and citations are measures of school quality, then I imagine Japanese schools are going to face a difficult path as few publications are available online, and many are in school journals that are not peer reviewed. To much of the research world, a univ dept journal that’s not peer reviewed is basically a faculty newsletter, and not a real journal article. Publishing in Japanese makes it more difficult for results to be widely read but I imagine scholars in other languages face a similar barrier. But if it can’t be found via a Google search then it’s the tree falling in a forest. If no one hears it, did it make a sound? And if no one heard it, then no one can cite it.
You’re right to point out that the rankings students look at depend on students’ goals. If you want a good job in Japan, you need to look at students’ job placement. You might also visit campuses and see where you might be happy. The happiness index isn’t part of these rankings but is very important.
LikeLike
Thanks for the comment. I used to read the blog you and your students maintained.
At writing and publishing seminars many people describe kiyo and SIG newsletters as the easiest step to get something on your CV. Unfortunately (as described in Diane Hawley Nagatomo’s last book) many universities seem to prioritize “uchimuki” kiyo publishing and committee work over “sotomuki” international and peer-reviewed publishing. If this is the case a lot of cultural factors will have to be changed for J universities to start getting more citations.
LikeLiked by 1 person
At the end of Paragraph 7, shouldn’t it be
“ranked 800 or higher”?
LikeLike
I suppose so… I was struggling with how to avoid referring to the same school as having a high rank (and thus a lower number) in the THE rankings and a high hensachi (and a higher number) and high scores on the THE ranking components. I settled on referring to all numbers as simply high or low and sometimes rankings as within or without. Sorry, I know this is confusing.
LikeLike
I enjoyed reading your recent post analyzing the correlation between the THE rankings and hensachi. To complement this piece, I would love to see an analysis of the data (i.e., test scores) used to calculate hensachi, and particularly a critical review of whether that data can be considered to be reliable.
As for this article, let me suggest a correction to the below sentence:
“Schools that are very difficult to get into, for example Waseda University with its median hensachi of 69 (rejecting nearly 98% of all applicants) is ranked by THE as just as strong as Toyohashi University of Technology, which with its hensachi of 50 gives students even odds of getting in.”
As I’m sure you realize, Waseda doesn’t reject nearly 98 percent of applicants — just as Harvard, Yale, Stanford, etc. (i.e., schools having median SAT/ACT scores in the 99th percentile) — don’t reject 99 percent of applicants to their schools.
On a tangent, I have never given much credence to the THE rankings for undergraduates because of the heavy weight accorded research and citations, when the majority of universities (i.e., those not near the top of the rankings) don’t offer many research opportunities to undergrads, and most data shows no significant correlation between faculty research and effective teaching. Now, many of the top overseas universities (i.e., Harvard, Yale, Stanford, etc.), and even the top liberal arts colleges, do offer considerable research and funded internship opportunities to undergrads. However, at Japan’s top universities I would expect undergrad research participation and funding to *pale greatly* when compared to participation and funding at the undergraduate level at top universities and colleges overseas.
LikeLike
Thanks for the comment. Of course you are right; I’ve edited to text to clarify. What data are you citing on the lack of correlation between research and teaching quality? I think it would be pretty salient for people using the THE to choose undergrad universities. I agree with your last comment. I did research as an undergraduate but nothing close to what I hear goes on at the Ivy League schools.
LikeLike
Here’s a link to a piece that does a very good job of examining the data on the correlation (or lack thereof) between faculty research and undergraduate teaching.
Click to access prince-jee07.pdf
LikeLiked by 1 person
Thank you for writing this interesting article.
-The language barrier and a disconnect/Galapagos syndrome of course is something that affects Japanese academia (or even the country as a whole), especially I imagine in the middle tier area. Could be less so in the top tier? And this affects all components. I would imagine the two Reputation Surveys and the Citations components are the largest disadvantages to academia rankings of Japanese universities in the THE (the international component is relatively smaller at 7.5%).
-“International outlook (staff, students, research): 7.5%”: is obviously a huge advantage to European universities and Anglophone universities. Someone in academia from California moves to Texas and it does nothing, but someone in academia from Spain moves to France and it raises the French university’s THE points. The Anglophone effect is probably larger, of course, giving an advantage in the rankings to all American (including Canadian), European, and former-European-colony area universities (e.g. Australia, New Zealand, Singapore, Hong Kong). Japanese universities are hugely disadvantaged here. Not just the barrier between Japan and English, but the language barrier between Japanese and the languages of its nearest geographical neighbors are also much larger than between European neighbors/across the Atlantic. Plus, those geographical neighbors of Japan are more likely to experience brain drain out to Anglophone countries rather than Japan anyway.
-“Industry income (knowledge transfer): 2.5%”: this obviously advantages engineering universities over others. Just looking at the top rankings, University of Oxford at number 1 scores 62.5, CalTech at number 2 scores 90.8. But this component is worth only 2.5%.
-“Research (volume, income and reputation): 30%” and “Doctorate-to-bachelor’s ratio: 2.25%”: give an advantage to universities with large graduate departments. Undergraduates don’t produce research. They experience and learn research under a professor and maybe write a thesis with that, so it improves the teaching quality for undergrads, but it’s really not going to contribute to raising a university’s research output. That’s all about graduate students. Don’t know how this affects Japanese universities, but I imagine if job placement is the primary motivation for students, there is less motivation for the university to have a large graduate department.
-Something like a “Standardized component score”: [(school’s component score – mean or median observed component score)/standard deviation of all schools’ component scores] or simply a “Relative component score”: [(school’s component score – mean or median observed component score)/(range of component scores/2)] would show in what components Japanese universities fare worse than other universities. And ideally the Teaching, Research, and Citation components would be broken down even further to see which exact categories are showing differences (because it’s easy to see here filtering to show only Japanese universities that their research and teaching pull up their rankings and their citations and international outlook drag down their rankings), though I don’t know if this data is available.
-Following your link, the highest hensachi departments were medical, law/government, and economics/business departments. Medical students probably go straight to practicing medicine after graduation (as opposed to getting an MD/PhD – the ratio of MD’s doing graduate research to MD’s seeing patients is probably smaller than the ratio of science/engineering PhD’s in academia to science/engineering PhD’s working in industry), and law/government, and economics/business go to work instead of grad school. So high hensachi –> go to work after graduation –> no contribution to future research or citations output. Perhaps this is why Citations correlate negatively with hensachi? A Japanese university that consists mainly of science and engineering departments would do relatively well on the THE rankings compared to hensachi (their hensachi loses out to universities with strong medical, law/government, and economics/business departments). A Japanese university that consists mainly of “job creating” departments (e.g. medical, law/government, and economics/business) would do relatively poorly on the THE rankings compared to their hensachi. Perhaps hensachi is an indicator of desire to find a high-paying job in addition to obviously an indicator of the hardworking-ness of a high school student rather than an indicator of a student interested in a graduate academic career – which is what the THE rankings would reward instead.
LikeLiked by 1 person
Addition: If hensachi is purely from admissions test scores, then it’s basically Japan’s version of SAT/ACT scores. And SAT/ACT scores for science/engineering schools in the US are definitely higher than SAT/ACT scores for liberal arts schools on average.
The THE rankings heavily reward research output (which is by professors and graduate students) and some international flavor (i.e. an anglophone bonus). Because of it’s international scope, it feels at first like a very general and comprehensive ranking, but it is rewarding specific things – it’s still a specific type of ranking.
For example, if you look at business school rankings, both US and Europe I believe, the rankings reward job placement and starting salaries and almost nothing else.
Hensachi measures the quality of incoming students, but not outgoing. Companies hiring graduates from schools using admissions test hensachi would be mistaken (and I think such a practice is detrimental to the motivation of Japanese universities to improve their teaching quality), but companies would not be mistaken to use a “business school ranking-style” method to hiring graduates from school instead. The former measures incoming and the latter measures outgoing. The former is a school’s “privilege” but the latter is a school’s actual product. Companies, I would guess both in Japan and the west, don’t use THE-style rankings that reward academic research when choosing their graduates (which is why business school rankings are done completely separately).
LikeLiked by 1 person
Thanks for the in-depth comment. I hadn’t thought of the fact that medical (and other) schools in Japan might have lower citation scores than those in many other countries due to the fact that they accept students right out of high school and send them thereafter straight into the workforce. I don’t know enough about the type of research that is done in the 6 years 18-24 year olds are at Juntendo to say confidently whether it should be less than at an American or British medical school. Its citations score (23.5) is certainly low. I know anecdotally that some doctors who graduate from that particular institution do post-graduate research at other universities, which would count toward the other university’s citations score but not Juntendo’s. (Incidentally, Toyota Tech’s citations score is 78.6, which also fits the pattern you describe).
LikeLiked by 1 person
Who would expect Juntendo to have high citation scores as it is not research-oriented like, for example, its next door neighbor TMDU, which is a national university?
btw, I like the article
LikeLike
“I have no idea why Citations would correlate negatively with hensachi. Hopefully someone in the comments can suggest a reason.”
My father was a professor of provincial national university so I may add anecdotal guess.
Firstly, private university is solely geared toward undergraduate, where professors have to teach more classes and class size are so much larger. My father had an offer from such institution with much higher pay but he declined because he didn’t want to be a glorified high-school teacher. Academic in private universities do not have time to do research. So while private universities in Kansai and Kanto urban area are more “popular” for students and have higher hensachi than provincial national universities, they may not have any research output, especially scientific one which match their domestic prestige.
On flip side, provincial national university, which often hover around 50 in hensachi, often have surprisingly strong in research, especially in science. Professors from most provincial national universities are usually PhD graduate of more prestigious universities who didn’t get the job in their own university. While their post is less prestigious, provincial national university provide somewhat ideal environment if you want to dedicate your time just doing research. Not only there are less class and less students to teach, there are less admin and other activities which you are forced to do if you are professor in prestigious university. Also, commute takes 15 mins cycling from/to your work, and everything including house is cheap so you could easily buy a house. Also, your minions (aka graduate students) also live 10-15 mins from your lab so you can run the lab 24-7, Mon-Sun.
A colleague of my father racked up +300 papers published before his retirement. In his field most would produce one or two papers a year but he was cranking out about 10. He was more of a head of research factory managing his minions (aka PhD students). In Japan, prestige of universities are solely about difficulty of entrance exams for undergraduate and it says very little about what happen at postgraduate level.
LikeLiked by 1 person
Thank you! That is extremely, supremely interesting.
LikeLike