Times Higher Education Rankings and Hensachi in Japanese universities

This post in a nutshell: Are rankings of universities within Japanese society similar to rankings of universities used internationally?  What components of international ranking scores track the most commonly used rankings in Japan?

Introduction

The best-known means by which universities in Japan are ranked is called a hensachi score, which tells how far from the statistical mean a typical student admitted to a given institution scores on a test.  A score of 50 is at the mean, 60 is one standard deviation above the mean, 70 is two SDs above the mean, etc.  A high hensachi, say 65, for a given department at a given institution means students admitted do better than at least 95% of the test-taking population on that test (one SD turns out to be quite a leg up on one’s peers).  Hensachi scores vary by department; a medical university like Juntendo will have a high hensachi for its medical department (72) and a much lower one for its nursing department (55).  The private university I used to teach at has hensachi ranging from 35 to 65 for its various departments.

6_8_1
Source.  Looks more complicated than it is.

On the other hand, colleges in the US are usually evaluated according to a range of measures.  It’s not difficult to find out the acceptance rate at a given school, but more commonly seen are suggested minimum SAT and ACT scores as well as high school GPAs.  Placing these on a normal curve is possible but not widely done.

More commonly, people rely on national and international rankings like that of US News and World Report or the Times Higher Education World University Rankings, the latter of which are the subject of this post.  The THE Rankings caused a bit of a stir here recently as Japan’s universities underperformed in the list of the world’s top 980 universities, with only 2 institutions in the first 100 and many schools highly regarded in Japan falling rather high (as we shall see, the prestigious private MARCH schools – Meiji, Aoyama, Rikkyō, Chūō, and Hōsei all come in above #800, or off the list altogether).

A common interpretation of the discrepancy between the ranking and prestige that Japanese universities enjoy locally and those they receive internationally is that Japanese universities serve some local societal need that is not addressed, or at least is underrepresented, in the THE Rankings.  The most obvious of these is simply sorting: Japanese universities are a final stamp on a long process of streaming by test scores in order to serve the efficient hiring by companies after college, which justifies the use of a single number (hensachi) to rank universities and explains some other mysterious practices like much of 4th year attendance being superceded by job hunting.  Of course, this thinking would justify any allotment of hensachi rankings; a more prestigious school is more selective and therefore more prestigious, and a selective school is more prestigious and therefore more selective; the hensachi of a school telling employers all they need to know about its graduates.  Other local needs that have been proposed as explanations of this gap include the necessarily limited local market for research in the vernacular (including in-house university publications; I haven’t been able to find any of these in the database that the THE Rankings use to compile their Citations and Research scores), the lack of direct industry ties among Japanese universities, lack of weighting for patents (my ex-employer was particularly well-known for this in Japan), and a lack of outreach apparati for recruiting foreign students and staff.  This last one seems to be a result of misplaced priorities than institutional disadvantage.

The components do not all contribute equally to the THE Ranking.  Notably, Teaching and Research are less weighted this year than in previous years, a fact which this article blames for Tokyo U’s drop in this year’s rankings.  As we shall see, these two areas are relatively strong for Japanese universities.

the-world-university-rankings-2016-2017-methodology-small
Source.

In the interests of finding out just how much the THE Rankings correlated to Japan’s internal rankings of universities, I went through the THE Rankings for Japanese universities and recorded the component scores that contributed to their final rankings.  I then figured out the average (median unless there were 3 or fewer departments, then mean) hensachis across all departments for those universities and entered them in a Numbers file, using this site to find hensachifor university departments.  Then I computed the correlations between the component scores, the THE Rankings, and the hensachis for those universities.  I only included schools ranked 800 or lower [EDIT: meaning better than 800]; including the final group of 801-980 would have doubled the amount of work.  Sorry.

Numbers

Not exactly a creative use of statistics, but it involved a lot of copying and pasting and you know, it’s the kind of thing you expect someone to do.  So I did it. You can see the raw numbers in the following graph.

THE rankings and hensachi.jpg
Means: Teaching 32.2, Research 23.4, Citations 33.4, Int’l Outlook 23.9, Industry Income 47.9. THE Rankings are given as averages of the ranges given, for example 601-800 is entered as 700.

The overall correlation between the THE Rankings and hensachis was -0.5778.  For only schools in the THE top 500, the correlation was -0.6128.  These numbers are negative because, as should be expected, rank falls as hensachi rises.  The closer a university is to #1 ranking, the harder it is to get into.  For schools ranked above #500, there is almost no correlation between ranking and hensachi: -0.1339.  This should not be a surprise, as almost all of those universities have identical ranking scores.

Immediately we can see that hensachis do not track the THE Rankings exactly, and sometimes are curiously divergent.  Schools that are very difficult to get into, for example Waseda University with its median hensachi of 69 (rejecting nearly 98% of all applicants) [EDIT: not really rejecting of course, just out of the reach of.  Students at -2 SDs presumably don’t apply to Waseda.] is ranked by THE as just as strong as Toyohashi University of Technology, which with its hensachi of 50 gives students even odds of getting in.  On the other hand, another school with the same hensachi of 69, Osaka University, is ranked just out of the top 200.  A student with much better-than-average scores, according to THE, would be much better off choosing Osaka.  On the other hand, a completely average student could get into a school which the THE considers just as good as Waseda, which in Japan has a level of prestige similar to Princeton or Stanford in the USA.

Correlations, positive and negative

So what about the components of the THE Rankings?  How do those compare with universities’ hensachis?

This next graph has the correlations of each component score with hensachi:

Screen Shot 2016-10-08 at 12.10.21.png
Teaching: 0.7368, Research: 0.6833, Citations: 0.2492, Int’l Outlook: 0.4683, Industry Income:0.4145

And only for universities within the best 500:

Screen Shot 2016-10-08 at 12.10.21 copy.png
Teaching: 0.8527, Research: 0.7785, Citations: -0.4863, Int’l Outlook: 0.3218, Industry Income: 0.3576

And above 500:

Screen Shot 2016-10-08 at 12.10.23.png
Teaching: 0.3738, Research: 0.2289, Citations: 0.3258, Int’l Outlook: 0.1926, Industry Income: -0.2048

To reiterate, a hensachi tells us how difficult it is to get into a university; it is not a direct measure of quality.  It tells us how far above average a student has to be to have a chance of getting in, which depends on how many students try to get into a university and how many seats the university has available.  A high hensachi is a measure of selectivity, which is at best an indirect indicator of quality.

Out of the components that make up the THE Rankings, Teaching and Research are by far the most correlated with hensachi, across all spectra of Japanese universities in the THE list.  Among the top 500 schools only, Teaching and Research have near-perfect correlations with hensachi.  This supports the conclusion that Teaching and Research are responsible for much of any university’s competitiveness in Japan, as represented in its hensachi.

Interestingly, Citations are negatively correlated with hensachi among the Japanese universities in the THE top 500.  Universities whose research is cited more often tend to be less selective of incoming students, or students are dissuaded by the type of academic strength shown in numbers of citations when deciding which schools to try to gain admission to.  This is not the same as Citations being irrelevant; that would show up in a correlation of near 0.  I have no idea why Citations would correlate negatively with hensachi.  Hopefully someone in the comments can suggest a reason.

Industry Income is negatively correlated with hensachi in universities outside the top 500.  Correlations are all weaker outside the top 500, suggesting that objective (at least as far as the THE Rankings’ component scores reflect objective evaluations) evaluations matter less as those evaluations drop.  Still, I await explanation for why Industry Income could be negatively correlated with hensachi.

Discussion

Among the cream of the crop, hensachi can generally be a heuristic for quality of undergraduate experience.  The main part of a 4-year student’s experience of a university is in its teaching, and hensachi correlates very strongly with the Teaching component of the THE Rankings.  However, among weaker schools by THE standards, hensachi fails to predict quality as strongly in any domain.  Importantly, this applies to weak schools as determined by THE, not by hensachi: many schools with high hensachi are ranked above #600 by THE, including Waseda, Keio, and the aforementioned MARCH schools.  For Japanese high schoolers looking at prospective undergraduate universities, my advice is to choose the lowest hensachi school from the THE top 500, like Tokyo Metropolitan University or University of Tsukuba.

For graduate students, hensachi is a much less useful tool for assessing a university’s strength, as Research, Citations, and Industry Income are likely to be much more important to students conducting original research.  Of course, I have no idea if prospective graduate students in Japan look at their universities’ hensachi at all; hensachi are computed for the difficulty of gaining admission to undergraduate departments.

Of course this assumes that quality matters.  If students are looking at university is mostly a stepping stone toward a prestigious job in Japan, one may be inclined to ignore THE Rankings altogether and regard only Japan-specific rankings as relevant.  If one looks at the hensachis of the aforementioned MARCH schools along with Waseda and Keio not as reflections of the numbers of students wanting to be educated at those institutions but instead as reflections of the numbers of students wanting the career opportunities having been admitted to one of those institutions grants you, they make sense regardless of the objective academic strength of those schools.  High-hensachi but low-THE-ranked universities can be seen as expensive and elaborate filters of workers for Japanese companies, just as Brian McVeigh suggests of Japanese universities as a whole.

There is an element of self-fulfilling prophecy to “prestige” in Japanese universities which can be seen especially strongly in high-hensachi private universities.  Bearing in mind what I said about Japanese high schools a while ago, it may be worth attending one of these universities if only for the opportunity to spend 4 years with like-minded peers.  However, if most of these peers are at that school for the career boost and not for the education, you will essentially be placing yourself in a Community of Practice for careerism and not for cultivation of the mind.  This may account for the low THE Rankings of some schools whose student bodies are clearly very devoted to some goal; that goal may just not be higher education as THE understands it.

For those of you who skipped to the bottom:

The data suggests that hensachi is not completely unmoored from educational quality, but that some forces besides educational quality also drive hensachi upwards or downwards.

Advertisements

14 thoughts on “Times Higher Education Rankings and Hensachi in Japanese universities

  1. I do not think that Brian McVeigh has been fair in his criticisms of Japanese universities. I should like to see more rigorous academic analysis on local universities’ different strengths and weaknesses, and rankings of quality on those terms. I believe that foreign critiques usually come from foreign academics who have been working in language and humanities or social studies or possibly communications departments, which may not be Japanese higher-ed strengths as a whole, and the insights and experiences of such academics or writers tend to be skewed as a result. As for higher hensachi being tied to expensive schools, not quite true, since highest hensachis are attached to the public and imperial universities. I think intellectual rigour and educational quality generally does tie in with the reputation and standing of the ranking school… But again, it depends on the department and discipline. Humanities students are especially notorious for being lax, absenteeism or wild fun pursuits. However , this distinction between humanities students, is in fact, not much different from the scenario in UK higher education. As for criticisms of careerism, yes and no, it’s like two sides of the same coin, perhaps Japanese universities would see better THE rankings if they placed more weight on universities’ rankings matching job placements with the marketplace. Afterall, there are often criticisms of too high rates of top ranking US and UK colleges’ grads not being employed while aaddled with a huge debt. Perhaps, you could say the local situation is doing something right here. I know that the Science and Engineering courses of MARCH universities, for example, are extremely rigorous, the classes are actually advanced, tough and there is no room for slackness, and a high non-attendance rate guarantees failing the year and non-advancement to the next year, and consequently failure to graduate. Completion of of assignments is also an important component, and the assignments have to be handed in on time, with no excuses, at least from my experience. Again, students report that rigour and slackness varies for different faculties. English literature students are more notorious for no-shows or partying through schoollife. Certain courses like literature, psychology or criminology etc., are text oriented or the research may be involve more autonomy or the studies somewhat self-study-dependent, so that student life tends to be more fluid and autonomous, and such students can seem to be less accountable or academically involved. On another but related tangent, school club life, the social life and social skills aspects of college club activity-life, while not strictly related to the students’ course of study are considered important, in a sense, like British schools, this is the place to network and build your “old boys'” ties for the future information exchanges. Whether this qualifies as careerism I don’t know, but these networking skills and networks are acknowledged as an important aspect and asset that enhances the value of one’s education, traditionally even more so for Japanese society, although there have been concerns that this “strength” or “value” of highered could be in the process of being eroded. I think that up to the ’70s, before the THE paper-arms race, it could be argued that the idea of higher institutions as a collegial clearinghouse of networked intellectuals was perhaps closer to the Japanese model, than the current one of marketplace specialist students chasing THE ranked universities. Just because it’s a filtered system doesn’t necessarily mean that networked system is less valuable or that the educational quality is lower, the latter allegation by McVeigh, imho, hasn’t been proven convincingly, but more in his cherry-picked anecdotal fashion, especially since he diid not name names nor mention specifics of types of disciplines.

    Like

    • Thanks for the comment. I agree that McVeigh’s book should be read mostly as an interesting angle on the meaning of higher education supported by mostly personal experience. I also agree that much criticism of Japanese universities by English-speakers who work in them comes from the humanities. I don’t suppose that there is much mystery as to why – I found myself wondering how much of the THE International Outlook rankings for universities in Japan were inflated by the presence of native-speaking foreign language teachers, which can’t be taken at face value as evidence of internationalism.
      I was unaware of the weakness of humanities departments in Japan, although I am of course aware that some people consider them expendable.
      I don’t believe that hensachi is correlated positively or negatively with expense. I just say that low-THE-ranked but high-hensachi schools may justify their price and exclusivity through other means.

      Like

  2. To the extent that research and citations are measures of school quality, then I imagine Japanese schools are going to face a difficult path as few publications are available online, and many are in school journals that are not peer reviewed. To much of the research world, a univ dept journal that’s not peer reviewed is basically a faculty newsletter, and not a real journal article. Publishing in Japanese makes it more difficult for results to be widely read but I imagine scholars in other languages face a similar barrier. But if it can’t be found via a Google search then it’s the tree falling in a forest. If no one hears it, did it make a sound? And if no one heard it, then no one can cite it.

    You’re right to point out that the rankings students look at depend on students’ goals. If you want a good job in Japan, you need to look at students’ job placement. You might also visit campuses and see where you might be happy. The happiness index isn’t part of these rankings but is very important.

    Like

    • Thanks for the comment. I used to read the blog you and your students maintained.
      At writing and publishing seminars many people describe kiyo and SIG newsletters as the easiest step to get something on your CV. Unfortunately (as described in Diane Hawley Nagatomo’s last book) many universities seem to prioritize “uchimuki” kiyo publishing and committee work over “sotomuki” international and peer-reviewed publishing. If this is the case a lot of cultural factors will have to be changed for J universities to start getting more citations.

      Liked by 1 person

    • I suppose so… I was struggling with how to avoid referring to the same school as having a high rank (and thus a lower number) in the THE rankings and a high hensachi (and a higher number) and high scores on the THE ranking components. I settled on referring to all numbers as simply high or low and sometimes rankings as within or without. Sorry, I know this is confusing.

      Like

  3. I enjoyed reading your recent post analyzing the correlation between the THE rankings and hensachi. To complement this piece, I would love to see an analysis of the data (i.e., test scores) used to calculate hensachi, and particularly a critical review of whether that data can be considered to be reliable.

    As for this article, let me suggest a correction to the below sentence:

    “Schools that are very difficult to get into, for example Waseda University with its median hensachi of 69 (rejecting nearly 98% of all applicants) is ranked by THE as just as strong as Toyohashi University of Technology, which with its hensachi of 50 gives students even odds of getting in.”

    As I’m sure you realize, Waseda doesn’t reject nearly 98 percent of applicants — just as Harvard, Yale, Stanford, etc. (i.e., schools having median SAT/ACT scores in the 99th percentile) — don’t reject 99 percent of applicants to their schools.

    On a tangent, I have never given much credence to the THE rankings for undergraduates because of the heavy weight accorded research and citations, when the majority of universities (i.e., those not near the top of the rankings) don’t offer many research opportunities to undergrads, and most data shows no significant correlation between faculty research and effective teaching. Now, many of the top overseas universities (i.e., Harvard, Yale, Stanford, etc.), and even the top liberal arts colleges, do offer considerable research and funded internship opportunities to undergrads. However, at Japan’s top universities I would expect undergrad research participation and funding to *pale greatly* when compared to participation and funding at the undergraduate level at top universities and colleges overseas.

    Like

  4. Thank you for writing this interesting article.

    -The language barrier and a disconnect/Galapagos syndrome of course is something that affects Japanese academia (or even the country as a whole), especially I imagine in the middle tier area. Could be less so in the top tier? And this affects all components. I would imagine the two Reputation Surveys and the Citations components are the largest disadvantages to academia rankings of Japanese universities in the THE (the international component is relatively smaller at 7.5%).

    -“International outlook (staff, students, research): 7.5%”: is obviously a huge advantage to European universities and Anglophone universities. Someone in academia from California moves to Texas and it does nothing, but someone in academia from Spain moves to France and it raises the French university’s THE points. The Anglophone effect is probably larger, of course, giving an advantage in the rankings to all American (including Canadian), European, and former-European-colony area universities (e.g. Australia, New Zealand, Singapore, Hong Kong). Japanese universities are hugely disadvantaged here. Not just the barrier between Japan and English, but the language barrier between Japanese and the languages of its nearest geographical neighbors are also much larger than between European neighbors/across the Atlantic. Plus, those geographical neighbors of Japan are more likely to experience brain drain out to Anglophone countries rather than Japan anyway.

    -“Industry income (knowledge transfer): 2.5%”: this obviously advantages engineering universities over others. Just looking at the top rankings, University of Oxford at number 1 scores 62.5, CalTech at number 2 scores 90.8. But this component is worth only 2.5%.

    -“Research (volume, income and reputation): 30%” and “Doctorate-to-bachelor’s ratio: 2.25%”: give an advantage to universities with large graduate departments. Undergraduates don’t produce research. They experience and learn research under a professor and maybe write a thesis with that, so it improves the teaching quality for undergrads, but it’s really not going to contribute to raising a university’s research output. That’s all about graduate students. Don’t know how this affects Japanese universities, but I imagine if job placement is the primary motivation for students, there is less motivation for the university to have a large graduate department.

    -Something like a “Standardized component score”: [(school’s component score – mean or median observed component score)/standard deviation of all schools’ component scores] or simply a “Relative component score”: [(school’s component score – mean or median observed component score)/(range of component scores/2)] would show in what components Japanese universities fare worse than other universities. And ideally the Teaching, Research, and Citation components would be broken down even further to see which exact categories are showing differences (because it’s easy to see here filtering to show only Japanese universities that their research and teaching pull up their rankings and their citations and international outlook drag down their rankings), though I don’t know if this data is available.

    -Following your link, the highest hensachi departments were medical, law/government, and economics/business departments. Medical students probably go straight to practicing medicine after graduation (as opposed to getting an MD/PhD – the ratio of MD’s doing graduate research to MD’s seeing patients is probably smaller than the ratio of science/engineering PhD’s in academia to science/engineering PhD’s working in industry), and law/government, and economics/business go to work instead of grad school. So high hensachi –> go to work after graduation –> no contribution to future research or citations output. Perhaps this is why Citations correlate negatively with hensachi? A Japanese university that consists mainly of science and engineering departments would do relatively well on the THE rankings compared to hensachi (their hensachi loses out to universities with strong medical, law/government, and economics/business departments). A Japanese university that consists mainly of “job creating” departments (e.g. medical, law/government, and economics/business) would do relatively poorly on the THE rankings compared to their hensachi. Perhaps hensachi is an indicator of desire to find a high-paying job in addition to obviously an indicator of the hardworking-ness of a high school student rather than an indicator of a student interested in a graduate academic career – which is what the THE rankings would reward instead.

    Liked by 1 person

    • Addition: If hensachi is purely from admissions test scores, then it’s basically Japan’s version of SAT/ACT scores. And SAT/ACT scores for science/engineering schools in the US are definitely higher than SAT/ACT scores for liberal arts schools on average.

      The THE rankings heavily reward research output (which is by professors and graduate students) and some international flavor (i.e. an anglophone bonus). Because of it’s international scope, it feels at first like a very general and comprehensive ranking, but it is rewarding specific things – it’s still a specific type of ranking.

      For example, if you look at business school rankings, both US and Europe I believe, the rankings reward job placement and starting salaries and almost nothing else.

      Hensachi measures the quality of incoming students, but not outgoing. Companies hiring graduates from schools using admissions test hensachi would be mistaken (and I think such a practice is detrimental to the motivation of Japanese universities to improve their teaching quality), but companies would not be mistaken to use a “business school ranking-style” method to hiring graduates from school instead. The former measures incoming and the latter measures outgoing. The former is a school’s “privilege” but the latter is a school’s actual product. Companies, I would guess both in Japan and the west, don’t use THE-style rankings that reward academic research when choosing their graduates (which is why business school rankings are done completely separately).

      Liked by 1 person

    • Thanks for the in-depth comment. I hadn’t thought of the fact that medical (and other) schools in Japan might have lower citation scores than those in many other countries due to the fact that they accept students right out of high school and send them thereafter straight into the workforce. I don’t know enough about the type of research that is done in the 6 years 18-24 year olds are at Juntendo to say confidently whether it should be less than at an American or British medical school. Its citations score (23.5) is certainly low. I know anecdotally that some doctors who graduate from that particular institution do post-graduate research at other universities, which would count toward the other university’s citations score but not Juntendo’s. (Incidentally, Toyota Tech’s citations score is 78.6, which also fits the pattern you describe).

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s