Best class ever, except for gathering statistics

I just turned in grades for the summer intermediate skills class I had over the summer, and set out to compile some useful statistics just as in previous semesters. Unfortunately, the data doesn’t* say much… because the scores were too high. Every assignment category, from attendance to final exams, was higher than the same class in spring semester, sometimes by ridiculous amounts. For example:

Attendance

Spring 2018: 90.09%, standard deviation 12.9

Summer 2018: 95.85%, stdev 7.5
(including one student who was out of the country for 2 weeks in a row – otherwise it’d be 97.17% and stdev 3.97)

Homework

Spring 2018: 84.36%, stdev 14.0

Summer 2018: 96.19%, stdev 7.9

Grammar quizzes

Spring 2018: 83.9% stdev 15.5

Summer 2018: 87.95%, stdev 9.7

As the standard deviations imply, there wasn’t much spread between the highest- and lowest-performing students, and even less between the many varieties of average-performing students. This was basically a good thing – there is no upside to a large spread of homework scores for pedagogy or validity. It’s not as if my homework scores failed to validly** track some educational construct because everyone was doing uniformily well.

Summer classes have a lot of perks. They meet twice as often, 4 days instead of 2, letting you take 2 days out of the week for something like student presentations without creating a yawning 2-week gap between instructional days. The students are more dedicated – only 20% of my students in summer were taking any other classes. The class meetings are shorter too, which probably helped my students, about half of which worked. Of those who worked, 19% had morning shifts, 73% had afternoon shifts, 64% evening, and 30% night (between 10 PM and 5 AM). Despite these fairly high numbers, almost everyone did almost all the homework and did about equally well on projects, quizzes, and tests.

It’s a bit of a shame for data collection, because although I haven’t cracked the statistics textbook I was convinced to buy, I did start the term with a much more complete questionnaire on my students’ jobs, as you can see. In the end, presumably because of the narrow spread in grades overall, this yielded some correlations (evening shifts were most negatively correlated with final grades) but no significant differences between working and non-working students, even at p<0.05. Scores were too similar to yield differences among different types of students.

This didn’t confirm my big hypothesis, that working students are at an unfair disadvantage given that community colleges exist specifically to serve non-traditional college students. I have, however, narrowed my hypotheses for future work surveys a bit because “hours spent using English at work” was about as negatively correlated with final grades as “total weekly working hours” (-0.46 vs. -0.39). Next semester, I will have to compare hours of English use at work to overall hours of English use to see if working students have more opportunity for input and output, and if this is so, ask why this doesn’t yield significantly higher performance on at least some types of assignments. I can anecdotally see that students who use English at work benefit from doing so. I need to plan my classes so that this is reflected in their grades, or at least not reflected negatively.

If future classes continue to find a difference between working and non-working students irrespective of whether they use English at work, it may be that the type of competence fostered by having a service industry job where you use your L2 doesn’t outweigh the necessity of somewhat narrow means of assessment in an academic ESL class. For example, it’s inevitably my working students who have the most natural grasp of which modals can be used for formal and casual requests, offers, or requests for permission, but unless they can carve out time between the end of their shift and taking care of an elderly parent to show that grasp in an assignment, their homework scores won’t be commensurate with their abilities. The lens of assessment is only focused on students when they do assignments, not when they practice modals for hours at a time every day at work.

It may help make my classes more equitable in this regard if I minimize the amount of “assignment” they have to do to prove they’ve been getting input, while still being hard enough to fake to prevent cheating. I have a type of assignment that is aimed at dragging along as much real-world practice as possible for a minimum of “assignment”, which is sometimes very close to “go get some input, then check a box when you’re done”. An example is a book report where the students choose any graded reader from our library and then turn in a pretty perfunctory worksheet that they could probably do in 5 minutes. To me, this type of assignment is justified by 1) the high ratio of interlanguage-developing work to product, 2) the promotion of available outside resources, and 3) the high motivation levels of my intermediate students, which reduce the odds of cheating (also, the low grading time). If a similar assignment said “start 3 conversations and fill out a perfunctory report afterwards”, this could reward the time my working students spend talking without pandering specifically to them.

Maybe the future of all ESL homework is “get input, and prove you got it”. At least at the intermediate (i.e., not academic writing) level, this probably maximizes opportunities for interlanguage development while minimizing what are in my view the less valid aspects of the grading process.

*”Data” is an uncountable noun, unless you are writing for an academic journal or have a mobile datum plan like Titus Andromedon’s that just comes with the one.

**Now that we’ve split the infinitive, the only question is whether we’ll be able to fuse it in a stable way and provide unlimited, grammatical energy for the entire world.

Stereotype threat and ELT

When they speak their L2, our language students are undertaking something mentally taxing while monitoring themselves for mistakes and in the presence of people who expect them to struggle. This is almost a perfect recipe for stereotype threat.

What is stereotype threat?

In case you’re behind on your liberal intelligentsia required reading, stereotype threat (ST) is “subtle reminders of stereotypes that presume the incompetence of certain groups. This ‘threat in the air’ can cue a concern with confirming these stereotypes that can impair the ability to perform up to one’s potential” (Schmader, 2010, p. 14). In short, fear of confirming negative stereotypes about one’s group takes up mental overhead and reliably and demonstrably hurts performance, and triggering this effect is as simple as reminding people of the stereotype before giving the test. This effect is real and has been replicated many times with many different groups – men and women most often (Johns et al, 2005) , but also White men and Asian men (Aronson et al, 1999) – even tracking implicit bias scores on a national scale in a study with hundreds of thousands of subjects (citation too long – click here).

The precise psychological mechanism behind this is apparently under dispute, but general anxiety along the lines of an affective filter (I don’t think I need a citation for this) seems not to be it. Rather, mental resources seem to be taken up imagining ways to fail. Working memory available for the task is reduced in favor of monitoring oneself for mistakes and spontaneous, intrusive negative thoughts (Cadinu et al., 2005; Schmader, 2010).

What’s it got to do with ELT?

I think it should be clear that our students, to varying degrees, are under ST almost all the time. Less obvious is the fact that many teachers are, too. Learners and teachers alike may be facing a penalty to their language use that has a cause besides incomplete knowledge or acquisition.

If placed in a context where stereotypes are known and especially when ELLs are implicitly being compared to NSs, we can expect ELLs to perform worse than otherwise at language-mediated tasks (I’m reminded of this article in which the author recounts having found solace in the relatively language-free world of math in her teenage ESL years – the Asian math stereotype probably didn’t hurt either). We can expect NNS teachers also to make more errors when they know they are being evaluated by NS teachers. Performance is likely to be worse both for input and for output in both cases. As Rydell et al (2010) write, “At least in the present task setting, we see that overt emphasis on the existence of the stereotype both prevents learning … and, to a significant degree, prevents expression of learning that has already occurred” (p. 14046) (Yes, that is the real page number). ST is likely to affect students in the ELT classroom as well – an ESL class in the USA where everyone thinks “Asian students don’t talk” is probably worse for Asian students, all other things held equal, than an EFL class in Asia taught by a NNS.

These conditions follow NNSs outside the classroom, too. Even well-known ELLs like Arnold Schwarzenegger and Melania Trump have jokes made at the expense of their intelligence – mostly based on accent, the hardest part of NS speech to adopt. It doesn’t seem to have discouraged Arnold, but whenever he speaks in public he is one error away from confirming everyone’s perception of him. I have certainly experienced this feeling myself, and I didn’t have Arnold’s fortitude. Our students’ lives are replete with conditions in which they will be judged on their language use and stereotypes about their national group or ESL students in general are known.

The mechanisms of ST appear especially designed to vitiate SLA. Working memory is probably as relevant as a danger to language acquisition as it is to math, but hyperconsciousness of mistakes is clearly more relevant to language use than many other subjects. Teachers may be instructing students to do exactly that as an effort to encourage noticing (Schmidt, 1993), usually thought of as a good thing, while ST holds self-monitoring to be an inhibitor of performance (Schmader, 2010). It is possible that while noticing facilitates acquisition in the long run, it distracts from other essential processes (e.g. understanding the intentions of one’s conversation partner) in the short run.

In fact, one effect of ST has been described as reduced ability to sort relevant information from noise, which would clearly hurt students’ ability to notice and turn input to intake. One such experiment used Chinese characters to test women’s “visual processing”, and found a ST effect of clear relevance for language teachers (Rydell et al, 2010).

Questions for study

If you haven’t noticed yet, I haven’t done any research to back up my suspicions that ST is an extremely important future topic for SLA. I do have a few ideas for research questions:

• Assuming ST for SLA is real, how will we know? Grammaticality judgment tests seem the most analogous to the mathematics-based research on ST that has been the most common so far, but wouldn’t real-time processing skills (like participating in a conversation) show a larger effect?
• What constitutes a “trigger” for ST? Is the presence of NSs enough, or the possibility that NSs will read/see the students’ output, or just a box for “nationality” at the top of the test?
• For that matter, how would you avoid triggering ST or creating a control group? ST-inducing instructions often look something like Candinu et al’s: “recent research has shown that there are clear differences in the scores obtained by men and women in logical-mathematical tasks” (2005 p. 574) (Interestingly, they left it to the test-takers to infer that women did worse, not just different, on these tests.) Non-ST instructions either simply leave that part out or explicitly negate it, along the lines of “… that there are no differences in the scores…”. How would this condition be accomplished plausibly on a language test of NNSs? Would it be believable to preface a test with, “This grammar topc shows no measurable differences between American and Chinese test-takers”?
• What groups have relevant stereotypes that could trigger ST? Is “ESL student” enough of a stigma? (Many students act as if it were.)
• Are different ELT classes more threatening than others? Can interventions by the teacher mitigate ST, for example by making explicit the fact that students will not be judged by NS norms?

References

Aronson, J., Lustina, M. J., Good, C., Keough, K., Steele, C. M., & Brown, J. (1999). When white men can’t do math: Necessary and sufficient factors in stereotype threat. Journal of experimental social psychology 35/1, pp. 29-46.

Cadinu, M., Maass, A., Rosabianca, A., and Kiesner, J. (2005). Why Do Women Underperform under Stereotype Threat? Evidence for the Role of Negative Thinking. Psychological Science 16/7 pp. 572-578. Available at: http://www.jstor.org/stable/40064271

Johns, M., Schmader, T., & Martens, A. (2005). Knowing is half the battle: Teaching stereotype threat as a means of improving women’s math performance. Psychological Science 16/3, pp. 175-179.

Rydell, R. J., Shiffrin, R. M., Boucher, K. L., Van Loo, K., Rydell, M. T., & Steele, C. M. (2010). Stereotype threat prevents perceptual learning. Proceedings of the National Academy of Sciences of the United States of America, 107/32 pp. 14042-14047. Available at: http://www.jstor.org/stable/25708852

Schmader, T. (2010). Stereotype Threat Deconstructed. Current Directions in Psychological Science 19/1 pp. 14-18. Available at: http://www.jstor.org/stable/41038531

Schmidt, R. (1993). Awareness and Second Language Acquisition. Annual Review of Applied Linguistics 13, pp. 206-26.

Tokyo Medical University and anticipatory childcare penalties

As you may have heard, in a scandal that incorporates almost everything toxic about Japan’s educational, workplace, and oyaji cultures, Tokyo Medical University, a top medical school in Japan, was discovered to have had a secret policy of discriminating against female applicants to their medical program for almost the last decade. Specifically, they reduced female applicants’ entrance exam scores to 0.9 or 0.8 of their actual levels so as to keep the female population of incoming classes down to 30%. College entrance exams being pretty much the single most important determiner of a young person’s career prospects, lots of people are livid in Japan, and the international press has picked up the story. It’s quite a blood-boiler.

Here are a few random thoughts that squeak out between the anger:

The shortest physically productive pre-activity

It’s common knowledge among ESL teachers that any activity should be prefaced with a pre-activity. Not only textbooks, but handouts, powerpoint presentations, and even off-the-cuff improvisations by the teacher are prefaced by some schemata-activating questions, discussion points or pictures, the theory being that students are better able to engage with the main activity after their brains have all the context-appropriate neurons firing.

I have never seen any evidence that the principle of preparing students for any activity should only apply to main activities, however, so it stands to reason if we are right about the importance of schema activation that these pre-activities could use pre-activities of their own. If you follow my logic, responsible pedagogy should involve a preⁿ⁺¹-activity before any preⁿ-activity.

This presents a philosophical and practical pedagogical problem, as responsible language teaching now seems to entail an infinite series of increasingly small preⁿ-activities, which in an echo of Zeno’s arrow, mean that we can never actually physically reach the start of our main activity.

With an eye toward helping my fellow language teachers out of this conundrum, I would like to propose a pragmatic (no pun intended) solution to the preⁿ-activity dilemma, which is this:

Teachers should not have pre-activities whose length would be shorter than the time it takes for light to travel from the teacher to the nearest student. The last pre-activity whose length is longer than this time will be called the shortest physically productive pre-activity.

I will illustrate this principle by assuming a few values:

• The speed of light is 3.08 * 10⁸ m/s
• Our main activity is planned for 20 minutes (1200 seconds).
• Our pre¹-activity is 5 minutes, or 1/4 the main activity, and schemata-activating pre-activities for other pre-activities will also last 1/4 as long as the activity that they prepare for.
• For the sake of simplicity, the nearest student is seated 3.08 m from the teacher.

 

Given our values for the distance between the teacher and the nearest student, it takes 1/10⁸ seconds for light to travel from the teacher to the nearest student. Any preⁿ-activity that takes less time than that will be over before the last one can be physically sensed by the students.

It’s worth pointing out that light is the fastest known physical phenomenon in the universe; no cognitive activity (or any activity with a physical substrate) can outpace it, no matter how “quick” the student. The speed of light is, therefore, a crucial property to consider when planning preⁿ-activities whose length is measured in millionths of seconds.

The question then is what value of n in a preⁿ-activity yields an activity whose length is less than 1/10⁸ seconds. I solved for n by plugging in the values above:

1200/4ⁿ = 1/10⁸

1200=4ⁿ/10⁸

120000000000=4ⁿ

120000000000^1/n=4

Doing these calculations the old-fashioned way, I come up with a value of 18 for n as the last preⁿ-activity whose length is longer than the time it takes for light to travel from the teacher to the student. Therefore, with the assumptions above, a main activity should be preceded by exactly 18 pre-activities, the 18th pre-activity being the shortest physically productive pre-activity.

It is to be hoped that teachers integrate this knowledge into their lesson planning thoughtfully and responsibly.

Clichés of polyrhythm/meter in metal

Here is my fourth-ever metal post. The others continue to get a few views every day from people who are probably disappointed by the rest of the content of this blog (and vice versa for English teachers who collide with metal posts). Obviously, my music theory metalanguage is quite shabby, but I’m going to try here to describe a few things I’ve heard over and over again in metal bands.

Circular thinking in deterrence

Back in my undergrad criminology classes, the professors would often raise interesting case studies of crimes where the benefits to society of punishment were unclear. Often, these were cases where giving “just deserts” to one party harmed another innocent party, for example the accused’s dependents. Inevitably, one classmate would respond to such case with “they knew what they were getting into” or simply “fry ’em!”

The punitive impulse is strong. It effects an attitude toward crime that one sees everywhere, and especially these days toward illegal border crossing. The reactions to people being punished for crossing the border show this clearly in part because of the inhumanity of the punishment being delivered – in this case, as in many others, not just to the accused.

Here’s exhibit A:

Fox News' Rachel Campos-Duffy: "I spoke to some African-Americans who say, 'Gosh, the conditions of the detention centers are better than some of the projects that I grew up in.'" pic.twitter.com/nQ1Lx5qMwl

— Contemptor (@TheContemptor) June 22, 2018

As hilariously disingenuous as the first part of that is, it’s the last part that I want to focus on today – the part where she says that children were brought to the US under “irresponsible” conditions. In her mind, and in many others’, the irresponsibility of bringing kids across the border under threat of separation justifies the harsh punishment of separation, because only irresponsible parents who deserve to lose their kids would cross the border. (Left out of the discussion, as always, are whether children of such parents “deserve” institutionalization and lifelong trauma). That one word, “irresponsible” encapsulates a lot of the circular thinking of deterrence.

That thinking always follows this path: the more severe a punishment, the more deserving of it a rational person will be for committing a crime that carries that punishment. If the punishment for jaywalking were squassation, only a truly irresponsible person would jaywalk, ergo that person would deserve squassation.

The weaknesses of this logic are first, that people aren’t rational in avoiding punishment or in any other domain. Criminology, like economics, has undergone a reevaluation since its purely rationalist days (I want to say it started with Beccaria…), but this post-rationalism hasn’t permeated the collective consciousness.  Just as people aren’t solely motivated by marginal dollars, they aren’t solely motivated by the sticks and carrots of criminal justice. Also, people tend to apply this philosophy of punishment unequally – for populations with which they have little empathy, they see only sticks and carrots, but for their own community, they want trust, norms, and the restitution of dignity to victim and transgressor.

Criminology (at least at the undergraduate level) divides rationales for punishment of criminal behavior into 4 categories: retribution, rehabilitation, incapacitation, and deterrence (specific and general). My feeling is that people feel highly retributive towards people that they feel little connection too, and that this feeling is often recast for public discourse in deterrent terms – terms that are self-justifying.

It would be nice if politicians who care so much about illegal alien children and their irresponsible parents would care about the way our U.S. Veterans live in the streets without shelter and food. #MAGA Website: https://t.co/2oje01cofB pic.twitter.com/aB1uYiPZDd

— Dr. Maria Peiro (@Peiro4Congress) June 23, 2018

As a bone to throw my remaining TEFL readership, let me say that I think this idea of self-justifying punishment has some importance in syllabus design as well – we could design a pop quiz on an assigned reading that is 30% of final grade, and it would be justifiable in the same sense that public flogging for vandalism is.

The Ramadan penalty?

Spring semester ended ten days into Ramadan, and a quarter of my evening advanced academic reading & writing class was fasting. This means that they took their finals after not eating or drinking anything for about 12 hours. Did this affect their scores? Should we care, since this is a nominally optional practice?

The short answer: it depends/possibly/needs further study, but we should definitely care.

This class makes a decent model to study this question for 3 reasons: a relatively high percentage of the class was observant Muslims (at least as far as fasting and refusing candy that contains gelatin goes), there are 6 big tests (3 reading and 3 writing) during the semester which are similiar in format to one another, and 2 of those tests (1 each of reading and writing) are in finals week – that is, during Ramadan. If fasting students were hurt academically, it should show up on those last 2 tests.

Without posting my students’ actual test scores here, I will just say that on the reading test, fasting students faced a 12% drop compared to the previous reading test, while the rest of the class had only a 2% drop. That is, the final reading test was harder for everyone, but much harder for fasting students. On the other hand, fasting students did 1% better on the final writing test than on the previous writing test while the rest of the class did 5% worse. All of this should be taken with a heaping tablespoon of salt, since the sample size for the while class was 22, producing a hilariously high t (significant at p<1).

So before any conclusions can be drawn, someone with access to administrative-level amounts of data should take a look at whether students who fast during Ramadan suffer disadvantages on tests or assignments offered during that time (the mid-May to mid-June, prime exam time at many schools).

And yes, we should care about this, first since a demographically specific disadvantage during a certain time of the year reduces the validity of the test. We don’t want to assess how well our students know a subject at 7:00 pm during Ramadan for the same reason we don’t want to assess the average temperature of an area at 3:00 am during January. Also, as I pointed out in my last end-of-the-semester statistics post, community colleges exist specifically to allow access to higher education to nontraditional student populations, including working people, older students, and immigrants (who are often older, working, and with family obligations as well). Obviously, this applies even moreso to ESL. If a quarter of the students our departments specifically exists to serve have a reliable disadvantage on certain tests because of factors that we can change, we should probably change those factors.

From Etsy

N-identities in Manzanar and Love Wagon (あいのり)

The last day of class, instead of having the potluck that my students were probably hoping for, we did a very quick analysis of the book we had just finished reading (Farewell to Manzanar) using Gee’s NIDA identities.

To briefly summarize what those are:

N-identity (nature identity) is the part of identity which is supposed to come from nature. It often includes visible traits like gender and race and the palette of traits and abilities that are thought to stem from them. As the Rachel Dolezal controversy shows, what is N for some people is I or A (see below) for others, and people can be quite unforgiving when they think an N characteristic is being wrongly taken on or rejected. My students were astute in noticing that even N identities change when the people around to perceive and interpret them change – the main character in Farewell to Manzanar has different N-identities when surrounded by other Japanese-Americans than with other Americans..

I-identity (institutional identity) comes from institutions of which one is part. For example, my ability to pass as a teacher comes mostly from my employment by schools, and not many people would accept the legitimacy of a “teacher” identity without it. It can be fun to imagine which kinds of jobs require institutional recognition to be considered a legitimate claim to identity – to me, “artist” is not an I-identity, but “animator” is. “Philosopher” is not an I-identity, but “researcher” is. My students said many characters in FtM lost their I-identities (in most cases, fishermen who worked together) when they were forced to move into the camps.

D-identity (discursive identity) comes from interactions with other people wherein one comes to be known as a certain “type” of person. This tracks what most people call a “personality”, but unlike “personality” has no implication of permanence. That is, one can have different D-identities among different groups of people. The Papa character in FtM is a bit of a stereotypical alpha in the way he interacts with others, which shifts from comforting to ironic as his life circumstances change from independent businessman to unemployed drunk.

A-identity (affinity identity) is similar to I-identity in that it relates to larger social groups of which we consider ourselves part. Unlike I-identity, A-identity doesn’t require any kind of actual membership in a group, only affinity for it. One can have an A-identity as a Premier League fan without any formal affiliation in the form of membership in a team or fan club. Notably, and as some of my very clever writing students mentioned, A-identity can be almost entirely imaginary – Papa from FtM imagines himself to be the inheritor of a samurai legacy, although the samurai ceased to exist before he was born and are well on their way to being more a cultural trope than a social class at the time the story takes place. One student mentioned this aspect of A-identity in a presentation, which was a great example of critical thinking.

What I like about these categories of identity is that they make clear both that identity is a multifaceted and context-dependent phenomenon and that it depends on other people and society. That is, you can have multiple identities, and none of them are purely a result of you choosing the type of person you want to be after doing some deep thinking alone or “finding yourself”.

My students did a very good job applying these on short notice to a book they’d probably grown quite sick of on a day when many people were already mentally on vacation. What they said reminded me of some things I’d been seeing on Netflix recently.

Correlations with final grades, spring 2018 edition

Every semester I throw a bunch of survey data, biographical data, and assignment scores from my classes into an Excel sheet and see what pops up.  This semester, like the last one, yielded some interesting information.

The tl;dr version is:

1. Work is a huge predictor of low grades
2. I should continue to push the importance of drafts in writing
3. I need to be careful not to evaluate students too much on their familiarity with my style of class
4. Perhaps I need to design better questionnaires

Read on for the details.

Job hurdles in ESL and EFL #3 – Foreign degrees

Here’s something I bet you hadn’t thought of: a foreign degree, even from a country whose degrees the US recognizes, may disadvantage you in the hiring process simply because of the extra step it takes for employers to process your application. You will probably not know this is happening, because it results, like every other failed application, in simply not hearing back from the hiring board.

(A bit of background: I got my MA while living and working in Japan from the University of Leicester, and now live and work in California. Most of my colleagues have MAs from public universities in California, something I didn’t realize the significance of until after the episode described here.)