Do the timing and number of edits on a draft predict improvement?

Since I started using Google Classroom for writing classes a few years back, I’ve noticed a pattern in the emails Google sends you whenever a student clears a comment you left. A few times, I’ve been able to tell when a student was still working on a paper past the deadline or if they got enough sleep the night before (emails at 3:20 AM are a bad sign). Most often though, you just find that a lot of students are making edits the morning that a paper is due, as your first email check of the morning features 30+ emails all saying “Bob resolved a comment in Final Essay”.

There exists a tool called Draftback (introduced to me, as with many edtech tools, by Brent Warner), a browser extension for Chrome, that lets you replay the history, letter by letter, of any Google Doc that you have edit access on. Its most obvious utility is as a tool for detecting academic dishonesty that plagiarism checkers like Turnitin miss (like copy/pasted translations, which show up in the editing history as whole sentences or paragraphs appearing all at once as opposed to letter by letter). It also has the benefit of showing you the exact times that edits were made in a document, which you can use to track how quickly students started responding to feedback, how many revisions they made (grouped helpfully into sessions of edits made less than 5 minutes apart), and whether these revisions were all made in the 10 minutes the student said he was just running to the library to print it. Draftback is the kind of tool that you hope not to need most of the time, but is hard to imagine life without when you need it.

This video gives a good introduction to Draftback.

With the pattern in my email inbox fresh in my mind (a term just having ended here), I thought I’d use Draftback to see whether this flurry of last-minute editing had some bearing on grades. To be specific, I used Draftback to help me answer these questions:

• Do numbers of edits correlate with scores on final drafts (FD) on papers?
• Does the timing of edits correlate with FD scores?
• Do either of these correlate with any other numbers of interest?

This required quite a bit of work. First, I copied and pasted rough draft (RD) and FD scores for each one of my students’ essays for the past 3 terms, totalling 6 essays, into a big Google Sheet, adding one more column for change in grade from the RD to the FD (for example, 56% on the RD and 92.5% on the FD yields a change of 65.18%). Then, I generated a replay of the history of each essay separately. Because each essay is typed into the same Google Doc, this gives me the entire history of the essay, from outline to final product. After each replay was generated (they take a few minutes each), I hit the “document graphs and statistics” button in the top right to see times and numbers of edits in easier-to-read form. I manually added up and typed the timing and number of the edits into the Google Sheet above. Last, I thought of some values culled from that data I might like to see correlated with other values. Extra last, I performed a few t-tests to see if the patterns I was seeing were meaningful.

(The luxury of a paragraph about how annoying the data was to compile is part of the reason I put these on my blog instead of writing them up for journals.)

Example “document graphs and statistics” page. From this, I would have copied 1468 edits for the due date (assuming the due date was Monday the 30th), 79 edits 4 days before the due date, and 1911 edits for 5 days before the due date, with 0 edits for every other day.

The values that I thought might say something interesting were:

• % of edits (out of all edits) that occurred on a class day
  • I’m curious whether students who edit on days when they don’t actually see my face do better – i.e., if students who edit on the weekends write better. Eliminating class days also helpfully eliminates lab days, the two class days a week when all students are basically forced to make edits. Incidentally, our classes meet Mon-Thu and final drafts are always due on the first day of the week. The average across all the essays was 63%, with a standard deviation of 38%.
• % of edits that occurred on the due date
  • Specifically, before 1 PM – all my final drafts are due at the beginning of class, and all my classes have started at 1 PM this year. My assumption is that a high % of edits on the due date is a sign of poor work habits. The average was 21% with a standard deviation of 31%.
• total # of edits
  • One would hope that the essay gets better with each edit. This number ranged from near 0 to more than 6000, with both an average and standard deviation of about 1700. Obviously, if you calculate this number yourself, it will depend on the length of the essay – mine were all between 3 and 5 pages.
• maximum # of edits per day
  • I’m interested in whether a high number of edits per day predicts final grades more than a high number of edits total. That is, I want to know if cram-editing benefits more than slow-and-steady editing. The average and standard deviation for this were both about 1200.
• # of days with at least 1 edit
  • Same as the above – I want to know if students who edit more often do better than ones who edit in marathon sessions on 1 or 2 days. The average was 3.25 days with a standard deviation of about 1 day.

All of the above were computed from the due date of the last RD to the due date of the FD, up to a maximum of 1 week (my classes last for 6 weeks, and there is very little time between drafts – read more about the writing process in my classes here). When I was done, after several hours of just copying numbers and then making giant correlation tables, I had hints of what to look into more deeply:

2 essays from each student, each taken separately.

As you can see in cells C9-H14 (or duplicated in I3-N8), students didn’t necessarily use the same revision strategies from essay to essay. A student who had a ton of edits on one day for essay 1 might have fewer edits spread out over more days for essay 2, as evidenced by the not-terribly-strong correlations in the statistics between essay 1 and essay 2. To take one example, “days with > 0 edits” on essay 1 was correlated with “days with > 0 edits” on essay 2 at just 0.21 (cell M7). Some of these differences were still statistically significant at p=0.05 (a good enough p for a blog, imo):

• Students who did > 2000 total edits on essay 1 had an average of 3428 total edits on essay 2. Students who did <= 2000 total edits on essay 1 had an average of 1650 total edits on essay 2.
• Students who did > 50% of their edits for essay 1 on the due date did an average of 45% of their edits for essay 2 on the due date. Students who did <= 50% of edits on essay 1 on the due date did an average of 17% of their edits for essay 2 on the due date.

Anyway, because it seemed prudent to consider the strategies used on each essay rather than the strategies used by each student, I made a second spreadsheet where the individual essays rather than the students (who each wrote 2 essays) are the subject of comparison, resulting in this much-easier-to-read correlations table:

Here I treat each essay as a unique data point rather than 2 products of the same student.

Columns I and J (or rows 9 and 10) are probably the most interesting to other writing teachers: those hold the correlations between statistics derived from Draftback data and I) final draft scores and J) change in score between the rough draft and final draft. In plain English, the correlations here suggest:

• As expected, % of edits on class days and % of edits on the due date are negatively correlated with the final grade for the essay. That is, people who did a lot of their edits in class or right before turning in the essay seemed to do worse (but not by much-neither produces statistically significant differences in FD grades or in improvement between RD and FD).
• Total # of edits and max edits per day are both positively correlated with final grades (and with each other). Editing more tends to produce better essays.
• Everything that is true for the final scores is also true for the change in scores between RD and FD. The fact that RDs were even more negatively correlated with % edits on class days and % edits on the due date than those values were with FDs mean that the changes appear to be positively correlated, but I take it as meaning that those strategies with an improvement from very bad RD scores to mildly bad FD scores.

To give a bit more detail, these were some statistically significant differences (p=0.05):

• Students who did > 2000 total edits had an average grade of 86.8% on the FD. Students who did <= 2000 total edits had an average grade of 78.7% on the FD.
• Students who did > 3000 total edits had an average grade improvement of 17.8% between the two drafts. Students who did <= 3000 total edits had an average grade improvement of 4.9%.
• Students who did edits on > 3 days had an average grade of 84.8% on the FD. Students who did edits on <= 3 days had an average grade of 78.9%.
• Students who did edits on > 5 days (that is, almost every day) had an average grade improvement of 33.6% between the two drafts. Students who did edits on <= 5 days had an average grade improvement of 5.8%.

The data suggests a fairly uncontroversial model of a good writing student – one who edits often, both in terms of sheer numbers of changes and in terms of frequency of editing sessions. In fact, “model student” rather than “model essay” may be what the data is really pointing at – the amount and timing of the work that went into a particular essay seems sometimes to show more about the student’s other work than it does about the quality of that essay.

For example, it’s not clear why data derived from the time period between RD and FD would be correlated with RD scores (in fact, you would expect some of the correlations to be negative, as high RD scores might tell a student that there is less need for editing), but perhaps the fact that the same data points that are correlated with FD scores are correlated in the same ways with RD and final course grades indicates that the data shows something durable about the students who display them (my caveat earlier notwithstanding). It is feasible that the poor work habits evidenced by editing a major paper a few hours before turning it in might affect students’ other grades more than that paper itself.

In fact, this seems to be the major lesson of this little research project. One t-test on % edits on due date was statistically significant – one that compared students’ final course grades. To be precise, students who did > 20% of their total edits on the due date had average course grades of 84.5%. Those who did <= 20% of their total edits on the due date had average course grades of 88.8%.

Just to pursue a hint where it appeared, I went back into my stat sheets for each class for the last year and copied the # of assignments with grade 0 (found on the “other stats” sheet) for each student into my big Google Sheet. Indeed, there was a statistically significant difference. That is, students who made > 20% of edits made on the day an essay was due got a score of 0 on 5% of assignments across the term, and students who made <= 20% of edits made on the day an essay was due got a score of 0 on 3.2% of assignments across the term.

Like many characteristics of “good students”, from growth mindset to integrative motivation, whether a pattern of behavior correlates with success and whether it is teachable are two almost unrelated questions. It doesn’t necessarily follow from this research that I should require evidence of editing every day or that I should move due dates forward or back. It does suggest that successful students are successful in many ways, and that editing essays often is one of those ways.

I might just want to tell my students that I really love the Google Docs “cleared comment” emails that I get on Monday morning and I wish I got them all weekend, too.

The Academic Support Catch-22

There is a pattern among formerly-known-as-remedial “academic support” classes that I’ve noticed that may work against their intended purpose.

The pattern is a result of the assumption that the subtext of planning and preparation in most assignments in college needs to be made text. That is, the assumptions of what needs to happen for a college student to be successful need to be made explicit and accounted for. For example, here is a representation of creative writing that I think gives a pretty accurate representation of the work that has to be done vs. what ends up on the page:

Source.

Academic support often seems to work by taking all of those hidden parts of the writing process out in the open and making them graded assignments themselves. An assignment that in another class might look like this:

Write a research paper on a topic covered in this class. (100 pts)

might turn into a weeks-long writing unit like this:

• Brainstorming discussion notes (classwork)
• Research goal discussion: 5 pts
• Mind map: 2 pts
• Library scavenger hunt (classwork)
• Works Cited and Plagiarism worksheet: 5 pts
• Outline w/ annotated Works Cited page: 10 pts
• Outline pair feedback (classwork)
• Introduction in-class writing (not graded)
• Rough draft 1: 10 pts
• RD1 peer feedback (classwork)
• RD1 tutoring visit reflection discussion: 5 pts
• RD2: 20 pts
• RD2 professor feedback reflection Flipgrid: 5 pts
• RD2 office hours appointment: 2 pts
• FD: 70 pts
• FD writing process reflection discussion: 5 pts
• Optional FD re-submission for makeup points
• Optional FD re-submission for makeup points reflection
• Optional FD re-submission for makeup points reflection²

Ok, the last two are jokes, but otherwise this writing process, where every step is explained, given its own rubric, shared, and reflected upon, is quite normal for a writing class that is coded “for English learners”, “academic support”, or just has a professor trying a more workshoppy approach.

This can be invaluable unless it sets too strong a precedent for explicit requirements of the writing process in students’ minds. Some students, particularly in ESL, may have no idea at all what the writing process is supposed to entail or how to use the resources like libraries, tutoring, etc. It’s better that at least one class during a college student’s first year puts this all on the record, but it might be counterproductive if too many do. It shouldn’t be lost on us that each step made explicit in the “academic support” writing process makes it resemble a typical college writing assignment less and less. If students expect these steps always to be explicitly outlined, they may neglect them or delay them on assignments where they are not.

The contrast between two types of assignments in my classes crystallize these concerns for me. The first type resembles the detailed, all-steps-accounted-for work flow above. I have 2 papers in a term whose writing processes basically fill all of the 2 or 3 weeks ahead of their final due dates with discussions, peer review, presentations, and pre-writing. The second type is an “all-term” assignment given the first week of class and due the last week, usually worth a significant amount of points but doable in a few hours with the right preparation. Examples of this type of assignment are “go to an on-campus event and take detailed notes” or “email a professor in the department you plan to major in and ask 3 questions”. Students tend to do the first type of assignment with the appropriate level of dedication, preparing them well for the big essays that come at the end of the two- or three-week unit. At the same time, they tend to leave the second type of assignment until the weekend before the last week of class, days before they are due, and often run into problems like not having campus events to go to on Presidents’ Day weekend (this post is a topical one). This tells me that, in my classes at least, the precedent of having all the “underwater part of the iceberg” work outlined in detail for some assignments results in the underwater part being ignored for others.

Another factor may be that, for the first type of assignment, students are all doing the same thing at the same time and know that avoiding embarrassment during a week’s worth of discussions and presentations depends on their doing their work. For the second, on the other hand, students may all go to different events, email different professors, etc. all at different times and never have to show their work to their classmates. Again though, it is not unusual for major assignments in other classes to be solitary affairs. The many reasons that students seem to neglect solitary assignments with implicit requirements on time and preparation only highlight the problems that that neglect causes.

I don’t really have a solution for the skewing of expectations that academic support seems to produce – I just verbally warn students that most of the steps in our writing process will need to be taken of their own volition in their History, Psychology or Accounting classes. Maybe I need to give points for reflecting on that warning.

COCA for translationists

(Corpus of Contemporary American English, alongside the other BYU corpora from Mark Davies)

For basically all my career, from my eikaiwa days Japanese university to community college to the IEP I teach at now, I’ve been trying to get my students to see vocabulary as more than lists of words with accompanying translations.

Source

Sure, knowing one translation of “marry” is probably better than not knowing anything about “marry”, but it really just gets your foot in the door of knowing that word (and leaves you less able to enjoy semantically ambiguous sentences like “The judge married his son”). You still don’t have much of an idea of what kind of person uses that word, in what kind of situation, and (of special concern for fluency) what other words usually surround that word.

Part of what cramming for tests does to language learners (and really learners of anything) is convince them that the minimum amount of knowledge to be able to fill in the right bubble is efficient and expedient. One of the longest-running efforts of my career is trying to disabuse my students of the notion that when vocabulary is concerned, this kind of efficiency leads to anything worthwhile. To the contrary, the more seemingly extraneous information you have about any given word, the better you will remember it and the more fluently and accurately you will be able to use it.

(Naturally, the site where I first encountered this phenomenon was in Japan, where the question “What does that mean?” is almost incomprehensible except as a synonym for “Translate this into Japanese according to the translation list provided by your instructor”. But knowing a word and being able to use it (a dichotomy which collapses with any scrutiny) demands (again, a collapsed dichotomy being treated as a single subject) quite a lot more than an abstract token in a foreign language being linked to a more familiar token in one’s first language in memory. One can know that “regardless” “means” とにかく or 関係なくin Japanese without knowing what preposition usually follows it, which noun from “outcome”, “result”, or “upshot” most commonly follows that preposition, or that it has an even more academic ring than near-synonym “nonetheless” (which doesn’t have an accompanying preposition at all). Interestingly, overreliance on translation seems to be something of a vestigial trait of language education in Japan – people justify it for its utility on tests, but the tests themselves haven’t required translation in many years.)

Even when my students understand this, however, they still aren’t sure how to implement it. I get a lot of positive reactions to comparisons between chunks in English and in their first language (asking how many words a child hears in phrases like in “Idowanna”, やだ, 我不想 or je veux pas) or between words and animals (a lion can technically eat roast turkey, but what do lions usually eat?). Students readily identify chunks and idiomatic expressions that they hear outside of class (“Would you like to” and “got it” are some of the most-noticed). In the run-up to a vocabulary quiz though, where I want students to show all that they know about vocabulary, what I see most often on students’ desks is the familiar lists of translated pairs:

regardless 而不管 however 然而 nonetheless 尽管如此 nevertheless 但是 notwithstanding 虽然

It seems that students, when they “study”, tend to default to the strategies that they think got them through high school. Usually, students who have this tendency also have familiar patterns of scoring on quizzes: fine-to-high scores on the cloze (fill-in-the-blank) questions and low scores on anything outside of the narrow range where translation is applicable. I see this as a result of not being able to see how to use this knowledge of other features of vocabulary in their customary mode of studying.

I started using COCA in class as a way to plug the fuzzy, often-neglected dimensions of vocabulary learning – in particular register, genre, colligation and collocation – into a behavioral pattern that students have completely mastered. That is, COCA is a way to make a more complete picture of vocabulary compatible my students’ most familiar way of studying – sitting at a desk and looking up discrete words.

With that long preamble over, let’s have a look at the specific activities I use over the course of a term.

First glance at COCA

Starting on the first day, words of particular interest are added to a class web site – either my own, Vocabulary.com, or Quizlet (I’ve tried quite a few) – and drawn on for review, activities, and quizzes. Starting in week two, I introduce the idea of chunks (which they need in order to complete the reading circles sheets from that week on), either with a presentation or less formally, for example with a quiz game.

In a shorter term, I’ll introduce COCA the same week, or in a longer semester, around week 4 (my IEP has lightning-quick 6-week terms). The introduction usually has to be done in the lab – it’s much better if each student can do his or her own searches. I alternate between a worksheet and a presentation for the first introduction. This takes about an hour.

From experience, students never fail to see the utility of COCA at this stage and never seem to have trouble with the idea of another online resource. The issues that typically arise on the first day are:

1. COCA locks out searches from IP addresses if there are too many in one day (as in a class of 20 or so all using COCA for the first time in a lab). This usually starts to afflict my classes after the first 20 minutes or so of searches.
2. At minimum, students have to create accounts after the first few searches, which used to require a .edu email address, but doesn’t seem to now.
3. The use of spaces on COCA is idiosyncratic. A search for ban_nn* (without a space) will find intances of “ban” used as a noun, while ban _nn* (with a space) will find “ban” plus any noun, for example “ban treaty”, or hilariously, “ban ki-moon”. ban* (without space) will find any word starting with “ban”, and ban * (with space) will find “ban” plus any word or punctuation mark. Punctuation needs to be separated with spaces as well. These rules trip up students fairly early on, as they search for, for example due to the fact that* and don’t find what they expect.

Weekly activities

After the first introduction, COCA will be in at least one homework or classwork assignment every week.

Classwork

From time to time, but especially before quizzes, students do a jigsaw-style group activity I call vocabulary circles. As you can see, a good half of it is COCA-derived. If you don’t know how these usually work, students with different jobs are assigned one word per group, share them with “experts” who had the same job from other groups, reconvene and share them with their own group, and then have to take turns presenting all their group’s work to their classmates.

Reading

COCA searches are a part of many of the reading circles sheets I use (reading circles are the only way I do any intensive reading in class). Vocabulary specialists (or whatever you call them) are always responsible for chunks as a category of vocabulary as well as collocations for other words.

Discussions

Starting the week that COCA is introduced, weekly “Vocabulary Logs” on Canvas include COCA work like that reproduced below:

This week, you must use COCA to find something interesting about a word from our class vocabulary list. You must find these 3 things:

What other words usually come before and after that word?
Who usually uses that word? (For example, lawyers, academic writers, news anchors, etc.)
Which forms of the word are the most common? (For example, “present simple”, “plural”, “adverb”, etc.)

You get 6 points for answering all of these questions.
Then, in a reply, use a classmate’s word in a new example sentence that you make. This section will be graded on correctness, so read your classmate’s post carefully. (2 pts)

Or this option to take translationism head-on:

This week, you will compare a word from another language (for example, your first language) to a word in English. The words should be translations of each other.
You will point out how the two words are similar or different in these areas:

Collocation: Do the same or similar kinds of words come before or after the words?
Grammar: Are the words the same part of speech? Are the rules for the parts of speech different in the two languages?
Register: Do the words appear in the same kinds of situations? Are they similar in formality?
Meaning: Do the words have second or third meanings that are different?

This post is worth 6 points. Reply to a classmate for 1 more point.

Quizzes

The quizzes in my classes after COCA has been introduced all have some explicitly COCA-derived questions and some questions that are graded on COCA-relevant considerations.

In questions like the one below, “grammar” includes part of speech and colligation.

Use the word in a sentence that makes the meaning clear. (1 pt for grammar and 1 pt for clear meaning)
(sustainable) _____________________________________________________________

Some questions target collocations specifically (ones that have been discussed specifically in class):

Circle the most common collocation. (1 pt each)
A difficult environment can precipitate ( fights / conflict / argument ).
Adaptation ( onto / to / with )  a new culture takes time.

Other questions target the colligations of vocabulary that should be familiar for other reasons:

Fill in the blank with one of the following. (1 pt each)
Regardless of Owing to Because Also
_______________________ the waiter made a mistake with our order, our meal was free. _______________________, the chef sent us a free dessert. Lucky us!

Students cannot have COCA open during the quiz, but they can (and are advised to) get to know the words inside and out beforehand. As you may have seen, our vocabulary lists can grow fairly long by the end of the term, but words often appear on more than one quiz.

Essays

See my last post on the subject.

I am getting on board the “reflection as revision” train – grading reflection on grammar instead of grammatical accuracy on all drafts besides the first. COCA is the vehicle I use for this.

Conclusions

I presented this to you as a way to get students with an unhealthy focus on one-to-one translation to think about vocabulary in a way that better facilitates real-world use. Actually, it works even better with students predisposed to think of vocabulary in more holistic terms – but those students would often be fairly good learners just with enough input. The advantage of using COCA is that it can easily piggyback on habits that certain students may overuse – many of my students have browser extensions on their computers that translate any word the mouse hovers over. Adding one more dictionary-like tool that includes what dictionaries miss is a way to swim with that tendency rather than against it.

Teach a man to find correlations, he posts them for a lifetime

Aphorism showing its age aside, this post is designed for both men and women who use Canvas and are curious about statistics that may be hiding in their classes’ grades.

I have my own data to share about this semester’s classes, but first, here is a tool that you can use to do the same:

Stat sheet for grades 1.1

And an explanation of how to use it:

On to what I found.

I had 4 classes this semester – 2 Oral Language classes and 2 Written Language classes, both in the 2nd to last term of my university’s IEP. My university’s IEP works a bit unusually – my 4 classes were just 2 groups of people meeting for 4.5 hours a day 4 days a week, about half of which was “Oral Language” and half of which was “Written Language”. The first group of people were my students for the first “term” (=half of a semester), and the second group were mine for the second term. All told, I still had 4 gradebooks on Canvas to export and fiddle with. Between the 4 of them, I found these interesting statistical tidbits:

Scores of 0 are more predictive of final grades than full scores are

One would expect the number of 0s on assignments to negatively correlate with final grades, and the number of full scores to do the opposite. That is, thankfully, true. However, they correlate at different rates – across all my classes, on average, 0s are more (negatively) correlated with final grades than full scores are (positively) correlated. The reason for this is that full scores were more evenly distributed among all students than 0 scores, which were concentrated among a few students. The one class for which this was not true was the one that I changed my late work policy and started giving 1/2 credit for certain late assignments.

This would not be a cause for any particular change except for 2 reasons: 1) as shown by the last class, many of the 0s that students were getting were from late work rather than unsubmitted work, and 2) we have a fairly strict policy about grading by SLOs (student learning outcomes, one of the first abbreviations I had to learn upon my return to the USA after years in Japan), and nowhere in our SLOs does it say that students should learn the sometimes-merciless grading policies that one may encounter at university.

Therefore, I should really make the “late work gets partial credit” policy permanent. I should also probably give fewer full scores.

5% 0s is a line in the sand

I enjoy running t-tests to see what values in what grade categories produce statistically significant differences (p=0.01) in my students’ final grades. One t-test I ran (on the “other stats” sheet in the file linked above) was seeing if students who missed 5% of assignments were different in statistically significant ways from those who didn’t. It turns out that they are, in all 4 of my classes this semester. On the other hand, those who missed 2% of assignments weren’t. Perhaps I should give an opportunity to make up homework on about 2% of assignments (as I already do for classwork).

I’m hoping that my future classes have grades that reflect the average quality of their work, which in turn reflects their ability to do academic work in English, rather than their tendency to check due dates and read rubrics thoroughly on Canvas. These are important skills, but I won’t want to make them a bottleneck through which every grade must pass.

RDs need a bump, FDs need a nerf

Across 4 essays in both Written Language classes, the average correlation of rough draft scores with final grades was 0.70. The average correlation of final draft scores with final grades was 0.76. Since final drafts are worth at least twice as many points as rough drafts, this is rather surprising – even moreso because for 3 of the 4 essays, the rough drafts’ correlations are actually higher than the final drafts’ (the last had a very low correlation for the rough drafts).

I’ve been making changes to my writing process over the last few semesters, and it seems I need to make a few more. I think part of the comparitively low correlations that final drafts have is due to my grading practices – I think I take it easier on final drafts precisely because they’re so many points. My average scores for final drafts are higher than for rough drafts, and the standard deviations as lower – roughly 62%-95% with an average of 78% for rough drafts and 65%-95% with an average of 80% for final drafts. It’s not a huge difference, but looking back at the scores now they don’t seem to reflect the range in quality of the essays. Part of the high correlations for the rough drafts is also due to the skills that are involved in producing a first draft – planning, reading, responding to a prompt, and a bit of grammar – that are assessed in a lot of other assignments as well. Final drafts, meanwhile, assess (in addition to the same things that first drafts assess, but less directly) responding to criticism and editing, which don’t figure largely in many other assignments. Seeing how first drafts track more of the skills that I care about, and I seem to grade them with less of a high-stakes mentality, I should probably weight them more. On the other hand, since final drafts have a somewhat narrow range of skills that they assess, I should weight them less, or even separate my grades for final drafts into smaller sub-assignments like the COCA assignments I currently use, but also a written response to criticism and proof of visiting tutors instead of trying to indirectly read those things into the final draft.

I need to keep in mind too that I’m not necessarily serving my students well if I introduce them into a writing process that none of their psychology, history, or any other professors will use – I hear that most papers turned in for any class other than English are just the final drafts, already assumed to be revised and polished to a sheen. Maybe having one paper like this per term is also justifiable just in terms of preparing students for being taught by PhDs who know more than anyone else in the world about the behavior of certain species of field mice under certain conditions but have never studied pedagogy.

Look forward to more like this same time next semester, and let me know if you find the sheets useful for your own classes.

Review: Interchange (13th ed.)

The Interchange (formerly New Interchange) series is a mainstay of ELT worldwide, used in contexts as diverse as “cottage industry” (Nagatomo, 2013) private language academies to institutions of higher education. The series has undergone significant changes with its 13th edition that warrant fresh review.

A former edition

To begin with, significant revisions have been made to the content and layout of every chapter, in the words of the publisher, to “bring our content and delivery into alignment with the norms of the 2050s”. To this end, many chapters have been struck entirely or completely rewritten. The section on ethnic foods from book 2, unit 4, for example, is not only gone (a welcome change) but replaced with a pre-activity on the meaning of “tradition” that is more postmodern than many will find in their own Zones of Proximal Development.

None of this would be significant, however, without the accompanying revisions to grammar presentation and newfound focus on project-based learning. The new edition of Interchange changes its fundamental teaching strategy so much as to be unrecognizable compared to earlier versions, both in method and in geopolitical consequences.

Indeed, the methods are so innovative and the learning so efficient that within one semester students display 4-skills competence indistinguishable from native English speakers – at least the somewhat more stilted types of native speakers that populate English textbooks. The new student-centered activities sections have also made learners egocentric, hedonistic devotees of an urbane, bourgeois lifestyle often completely at odds with those of their surrounding cultures. In fact, students seem so transformed by their exposure to this textbook that their former linguistic and cultural identities completely disappear. Students leave classes having conversations about “their hobbies” or “their weekend plans” apparently never reverting to their former Spanish, Chinese, or Qatari selves, becoming strangers to their families and neighbors. Putting aside the decimation of local communities, the new presentation of language items is much improved. At least in municipialities where the new Interchange books have been used, few among the educated classes speak any language than Standard American English.

As one can imagine, the local and national governments of these areas have taken steps to discourage (to put it delicately) the use of the new, unprecedentedly effective Interchange books. Indonesia has taken an early lead in this regard, suspending visas for foreign English teachers and confiscating all (even previous editions of) Interchange. Police are being trained to conduct interrogations in Standard American English (“Excuse me, / I was wondering if / you would mind / telling me your political affiliations”). There are stories of language store owners being detained by paramilitary groups, although not with an official government mandate as yet (Liong, in publication). As if in anticipation of these events, Cambridge University Press made the electronic edition of Interchange purchasable with a variety of virtual currencies and viewable from within a browser window on any phone. The spread of Interchange 13th ed. and its devastating research-based methodology has therefore been impossible to stem.

An explosive rise in vigilantism has been another effect of the pedagogic success of Interchange. With the mitigation of distinct linguistic and cultural identities, societies have seen rising racism and other quasi-biological ideologies of difference that seek to reify formerly “obvious” national and ethnic borders. Informal communities of practice, usually composed of young men (although posses might be a better word) roam city blocks like the home of this publication in Tokyo, seeking to enforce ethnic unity on a purely physical basis – length of nasal bridge, eye color, hair texture, attached earlobe, etc. and interrogating those who deviate in impeccable textbook English (“Oh, my! Just have a look at his nose, will you? It seems quite wide for a Han, doesn’t it?”). Pre-existing ideologies of racial difference, present if marginal in many societies, have been used as historical justification for what amounts to racial terror. Because language use, especially in the age of Interchange 13th ed., does not reliably correlate with racial characteristics, this phenomenon has not directly victimized English speakers, but rather visible minorities of any language background. As such, it is better seen as a side effect of the extreme English fluency brought about by Interchange than as a countervailing force.

Foreign English teachers like this writer find themselves trapped between governments’ anti-English programs on one hand and paramilitary groups’ informal efforts at racial homogenization on the other. One hopes that this review, and further revisions to the Interchange series, help to reverse current deleterious trends in geopolitics even at the expense of the rapid and effortless English mastery present in its current edition.

References

Liong, W. (2058). Governmental and non-governmental revanchist efforts in linguistically flattened societies. The Language Teacher, 82(3), 45-59.

Nagatomo, D. H. (2013). The advantages and disadvantages faced by housewife English teachers in the cottage industry Eikaiwa business. The Language Teacher, 37(1), 3-7.

A taxonomy of comma splices

Working in academic ESL, I see all manner of comma splices – ones that immediately strike me as non-native, ones that seem to reflect English acquired via Facebook, and ones that could be made by professional writers. I’m 90% sure I saw some in the David Sedaris book I just finished, a COCA search finds a bunch under every genre (try the search terms “, _nn* [be] .”), and as a few survey respondents have pointed out, Dickens’ A Tale of Two Cities begins with a Guinness-worthy streak of them.

I set out trying to separate the types of comma splices I was seeing, assuming that they had some kind of logic to them. After all, all of the people I’ve seen splicing with commas know that periods (full stops) exist and are presumably using commas for some reason. What follows below is a list of the categories of comma splice that I saw with examples written by me, followed by the results of a survey I did in an attempt to figure out which type of comma splice is the least offensive. My Twitter followers are mostly language teachers (as are the LinkedIn groups that I posted the survey in), so their reactions ought to be interesting.

For the record, and just to show that I’m not just asking my Twitter followers to figure out what a comma splice is, a comma splice is two independent clauses separated by a comma and only a comma. Take out the comma and it’s a run-on, an error that probably deserves its own survey. You can fix comma splices in three ways:

1. Add a conjunction (preferably and, so, or but – many teachers use the acronym FANBOYS, but honestly, how often do you see “for” as a conjunction?)
2. Change the comma to a semicolon, or
3. Change the comma to a period.

See? Here are the categories.

Elaborative comma splices

The price does not reflect the cost of production, it reflects supply and demand.

Similar to the corrective comma splices below, this type of comma splice seeks to add information to the previous independent clause, in this case by adding information. I flubbed the writing of this one a bit by leaving out the word “just”; feel free to ignore this item. Run your own survey if you like.

Explanatory comma splices

The chain went bankrupt, its sales had been falling for years.

Here, the independent clause after the comma explains the one before; it gives a reason. Anecdotally, these are very common among ELLs.

Juxtapositional comma splices

Dogs are descended from wolves, house cats came from feral cats.

Here, the comma separates two independent clauses that are meant as opposites of each other. In style manuals, cases like these would call for a coordinating conjunction like “but” or “yet”, or a subordinating conjunction like “while” or “although”. Many readers, not to mention writers, seem to find them unobjectionable with no conjunction at all.

Corrective comma splices

Richards was not the director, Denton was.

Here, the second clause supplies the correct information whose existence is implied in the first.

Incidentally, the range of relationships between clauses that one sees in comma splices makes one feel the poverty of meanings in the English palette of conjunctions, and particularly the coordinating conjunctions.

Chronological comma splices

The new phone model came out, planning for the next model began soon after.

In this comma splice, the connection between the clauses is chronological; the first precedes the second.

I may have tripped an additional wire in readers’ minds here by making the subject of the second clause a gerund, which could be interpreted as the start of a participial phrase, triggering a garden path (see below). This wouldn’t be a graceful participial phrase, but of course people don’t always respect “the rules” as those go either.

Non-sequitur comma splices

The air is thick with moisture, the office has a meeting at noon.

Not much to say except that this sentence was as unpopular as you might imagine.

Appositive comma spices

The article was written by Malcolm Gladwell, he is a Canadian journalist.

Here, the second clause also supplies more information about something in the first clause by defining a noun. I’m calling them appositive comma splices because without a subject or verb they would simply be appositive phrases.

Conjunctive adverb comma splices

There is a water shortage, therefore citizens are being asked to take shorter showers.

Here, the comma comes before a word with a similar meaning to a conjunction but in a different grammatical class. Called “linking words”, “transitions”, or “conjunctive adverbs” by writing teachers, these words most commonly come at the start of a complete sentence or between commas, as with some other adverbials, in the middle of an independent clause set apart by commas. To illustrate what I understand is the conventional usage of these words:

• There is a water shortage. Therefore, citizens are being asked to take shorter showers.
• There is a water shortage; therefore, citizens are being asked to take shorter showers.
• There is a water shortage. Citizens, therefore, are being asked to take shorter showers.

As an academic writing teacher, my radar is always up for comma splices with these words (which we usually teach explicitly as a counterpart to coordinating and subordinating conjunctions), and to be honest I’m a bit disappointed that so many people disagree with me. Being a responsible writing teacher, though, I need to respect the vox populii. My job is not to make my ELLs the most comma splice-averse people on campus.

Quasi-participial phrase comma splices

The customers are waiting in line, they are holding their baskets.

Like the appositive comma splice above, the second independent clause could easily be reduced down to a familiar and well-accepted construction, the participial phrase.

(side note: I absolutely hate the term “reduced adjective clause” for participial phrases. I don’t think the term bootstraps learners’ understanding enough to justify the additional metalanguage and I don’t think it captures at all how fluent speakers think of them.)

Rephrasing comma splices

It is wasteful, it is inefficient.

Here, the second phrase is semantically very similar to the first. Anecdotally, I don’t see these very much in academic writing classes.

Garden path comma splices

The manager adjourned the meeting, switching off the projector caused the room to go dark.

This is the least comfortable for me to read, and is the reason that, if I remember correctly, Steven Pinker rejects any emails written with comma splices. Here, the comma splice encourages a misprediction by the reader as to how the sentence will proceed – the reader imagines that the manager adjourned the meeting by or while switching off the projector, only to trip over another finite verb in “caused”, which forces them to reappraise “switching” as a gerund instead of as a present participle, leading them up the garden path, so to speak.

Results

The above were presented to anyone who followed my link (mostly Twitter followers, who are mostly language teachers) with a 1-5 Likert-like scale of acceptability, 1=totally unacceptable, 3=don’t know/can’t decide, 5=totally acceptable. The following blurb was presented as well:

The following are all examples of comma splices. Please rate their acceptability as if you were a college writing teacher and they were produced by freshman writing students for a take-home writing assignment that has a grading rubric that includes grammatical accuracy.
For “totally unacceptable” answers, there would be a penalty in points for the error in question. For “totally acceptable” answers, the sentence in question could stand as an example of correct usage and could be recommended to other students.

46 people answered the survey, which again makes me wish I’d had Twitter when I was doing my MA research.

First major result

People vary widely in how they view the comma splices above. The mean score across all items was 2.33, slightly disapproving, and the mean standard deviation was 1.26. The median across all answers was 2 and the mode (most common answer) across all answers was 1.

Second major result

People who accept one of the sentences above tend to accept others as well; people who don’t don’t. All items are positively correlated with all other items, although some more than others. The average correlation of every item with every item (including itself) was 0.407.

Other results

Means

The mean scores were:

1. The price does not reflect the cost of production, it reflects supply and demand. (elaborative, but reads like corrective) Mean: 3.20, Median 4, Stdev 1.55; notably correlated with #2 (explanatory) and #3 (juxtapositional) at about 0.51 each, and #4 (corrective) at 0.59. This is one of 3 of my sentences that scored above 3, i.e. more accepted than not. 2 of them, this one and #4, have the same “not A, B” structure (although I didn’t intend for this one to), making that an early candidate for a characteristic of “acceptable” comma splices. It is probably good news that these two are more correlated with each other than most.
2. The chain went bankrupt, its sales had been falling for years. (explanatory) Mean 1.84, Median 2, Stdev 1.09; notably correlated with #1. Early on, when I only had about 20 respondents, this one was correlated with #5 (chronological), #7 (appositive), #9 (quasi-participial phrase), and #11 (garden path). I guessed that people who accept (or not) one type of comma splice tend to accept (or not) others as well, and these hills in the correlations might even out with more survey participants. It turns out that is what happened.
3. Dogs are descended from wolves, house cats came from feral cats. (juxtapositional) Mean 2.40, Median 2, Stdev 1.48; notably correlated with #1. This one fell slightly under mean acceptability. I suppose it screams a bit too strongly for a semicolon.
4. Richards was not the director, Denton was. (corrective) Mean 3.71, Median 4, Stdev 1.41; notably correlated with only #1, which shares its “not A, B” structure. This one had the highest score overall and was one of 3 (#1, this one, and #8) that had a mode of 5, “totally acceptable”.
5. The new phone model came out, planning for the next model began soon after. (chronological) Mean 1.96, Median 2, Stdev 1.17; notably correlated with #6 at 0.654 (see below).
6. The air is thick with moisture, the office has a meeting at noon. (non-sequitur) Mean 1.38, Median 1, Stdev 0.89; notably correlated with #5. This one had the lowest score, as one would expect. It seems that chronological comma splices are disapproved of by the same people who disapprove of comma splices whose clauses are completely unrelated.
7. The article was written by Malcolm Gladwell, he is a Canadian journalist. (appositive) Mean 1.66, Median 1, Stdev 1.16; not notably correlated with any other item.
8. There is a water shortage, therefore citizens are being asked to take shorter showers. (conjunctive adverb) Mean 3.25, Median 4, Stdev 1.60; notably correlated with #9 at 0.557. I’m not sure what makes people who accept this sentence also accept the next (and vice versa), as their clauses have very different relationships with each other. To me, this item is unique in that the relationship between the two clauses is actually transparent and given literally by a conjunction-like word (therefore). I don’t see a connection between this item and any other, but it seems that some people do.
9. The customers are waiting in line, they are holding their baskets. (quasi-participial phrase) Mean 1.93, Median 2, Stdev 1.08; notably correlated with #8 (0.557) and #10 (0.516). Even more mysterious than the lack of relationship (at least to me) with these two items is the fact that this item is much less accepted than either of them.
10. It is wasteful, it is inefficient. (rephrasing) Mean 2.68, Median 2, Stdev 1.55; notably correlated with #9. This is one where respondents might also be reacting to the redundancy in the two clauses. I said in my blurb to treat the assignment as a first-year college English assignment, and respodents might have struggled with thinking of a college writing assignment where saying the same thing twice, even with orthodox correct grammar, would be acceptable.
11. The manager adjourned the meeting, switching off the projector caused the room to go dark. (garden path) Mean 1.57, Median 1, Stdev 0.93. I would actually be worried if this one were approved of by most people.

Last, the item with the highest average correlations with the other items overall is #3 (chronological) at 0.469, followed closely by #9 (quasi-partipial phrase) at 0.462. In a nutshell, if someone accepts #3 or #9, he or she will probably be willing to accept other comma splices too, and vice versa – someone who rejects #3 or #9 probably rejects others.

Unsurprisingly, a comma splice in which the two clauses appear to have nothing to do with one another (#6) strikes almost everyone as wrong. The fact that this one scored lower when all of them are “technically” wrong is interesting, showing that semantic or discursive rather than grammatical congruence determines acceptability for most people. As is the premise of this entire project, all of these are equally wrong for exactly the same reason, at least according to the rules of grammar most of us use when giving feedback on student writing. If my Twitter followers are a sign of the attitudes that most grammatically-minded people take toward comma splices, then corrective comma splices may be bordering on safe for beginning writing students, and at least some conjunctive adverbs like therefore might be heading toward acceptability in the grammatical role usually reserved for conjunctions.

Comments

There was a space at the end for respondents to leave qualitative comments on what makes a comma splice acceptable or not. Here are some of them:

It seems that if the topics of the two parts are more closely aligned the error doesn’t seem quite as wrong. [That’s probably exactly what my students think]

It seems to me like if the comma could be replaced by a semi-colon or a colon it “sounds” more acceptable.

Something about rhetorical effect and whether the subject has changed. Dickens and the beginning of A Tale of Two Cities set some kind of precedent of acceptability.

Most of them should just be periods; some could be semicolons. [When you do a survey of different sentences which make the same language-related mistake in different ways, you always get a few respondents who refuse to look past the issue that is the baseline for the entire exploration. Ah well, uncovering this attitude is part of the purpose of a survey like this]

I love them all

I wish there was a little more info on the type of assignment. Going by the statement that there’s a grammar rubric, I’m thinking it’s straight up conventional rules, and they’re all “wrong” in that case. If it was at all a creative writing assignment, I’d give some of the ones that imply a series more leeway, like the wasteful/inefficient one. I think even more so if there was a third clause: It is wasteful, it is inefficient, it is wrong. [A very useful point!]

Super interesting survey! They’re more acceptable when the second clause describes the first clause of adds more information, e.g., Gladwell question. [Interestingly, most respondents disagreed.]

These are all independent clauses [See above]

There were all unacceptable …

I guess it might sound more acceptable if the second sentence seemed to be tacked on as an afterthought, like in the Denton example.

Continuation of a idea- more like an a positive. Or listing.

I think if one of the clauses is made unequal either by shortening it or using a pronoun, a comma splice might be acceptable. For e.g. “dogs are descended from wolves, house cats from feral cats.” (But in this case would it still be a comma splice?) [Not according to my sense of grammar orthodoxy since the second phrase lacks a verb]

As with anything in language matters, conventions are merely an agreement of that languages speakers. They are subject to change with each generation. If a new generation agrees that they should change conventions, then the convention will be changed. It appears that normally 75% agreement is a threshold for introducing the new convention. It would seem to me that young writers at the threshold percentage have changed the convention of the use of the comma, especially in the case of what traditionalists now call comma splices. [Curious where the 75% number comes from]

Thanks for reading! Feel free to share these results over Thanksgiving dinner.

Participial adjectives and the nouns that love them

At the request of Mr. Mark Brierley after the last post on participial adjectives, I’ve tried to come up with a few tests of my ambiguously defined “association” between participial adjectives and the verbs that they bear some family relation to. These tests probably have very little validity (bearing in mind that I never really defined “close” to begin with), but they do come in legitimate-looking tables.

All of them here are variations on what I call the “shared subject/object” test. Basically, if a participial adjective like “interesting” often has the same subject that a verb like “interest” does, or a participial adjective like “interested” often has the same subject that a verb like “interest” takes as an object, we can call them “close”.

I only did these searches for 5 verb/adjective sets, because until my institution pays for BYU corpora membership, I can only do so many searches a day. Also, this many searches on the iWeb corpus is already quite unhealthy.

So first, some raw numbers.

	verb subj			pres adj subj
	#pron	#n	%pron	#pron	#n	%pron
excite	1551	938	62.31%	17773	3405	83.92%
annoy	3069	476	86.57%	8885	476	94.92%
overwhelm	704	742	48.69%	3517	3513	50.03%
inspire	10137	5347	65.47%	955	901	51.45%
amaze	4744	362	92.91%	95559	35325	73.01%

These are the numbers of subjects of these verbs and their present participial adjectives which are pronouns and nouns respectively, followed by the percent of the total number of subjects that are pronouns. As you can see, the verb “annoy” comes with a pronoun subject (usually “it”) 86.57% of the time, while the adjective “annoying” has a pronoun subject even more often (also usually “it”). I didn’t keep track of how many of these are the dummy “it” seen in sentences like “it annoys me that robo-callers always spoof numbers from your own area code”, but I suspect it is a lot.

Next we have the differences between those rates of pronoun subjects:

excite	21.61%
annoy	8.34%
overwhelm	1.34%
inspire	14.01%
amaze	19.90%

According to this probably unscientific measure, the verb “overwhelm” is “closer” to its present participial adjective “overwhelming” than “excite” is to “exciting”.

Let’s make this a little easier to visualize. On this test of “closeness”, the rankings are:

	Test #1 ranking
excite	5
annoy	2
overwhelm	1
inspire	3
amaze	4

Here is the same data for past participial adjectives. Because past participial adjectives are related to the passive voice of transitive verbs (e.g. “it excites me” is similar in meaning to “I am excited” rather than “it is excited”), I compared the rates of pronouns as the objects of verbs rather than subjects.

	verb obj			past adj subj
	#pron	#n	%pron	#pron	#n	%pron
excite	8407	760	91.71%	68728	5913	92.08%
annoy	8910	569	94.00%	3205	483	86.90%
overwhelm	1684	241	87.48%	629	23	96.47%
inspire	24886	10713	69.91%	835	124	87.07%
amaze	11698	237	98.01%	39188	3182	92.49%

excite	0.37%
annoy	7.09%
overwhelm	8.99%
inspire	17.16%
amaze	5.52%

This time, “excite” seems “closer” to “excited” than “overwhelm” is to “overwhelmed”, but there is a problem that invalidates most if not all of what I think I can learn from this particular group of searches. The searches that I used to get these results were overwhelms _nn*, overwhelms _p*, _nn* [be] overwhelmed_j* and _p* [be] overwhelmed_j*. Do you see the problem? If you do, you probably use the BYU corpora as much as I do (seek help!). When you search for _nn* [be] overwhelmed_j*, you will get back a lot of noun phrases that happen to end with a noun, for example “the man in the back row was overwhelmed“, but not so for overwhelms _nn*, which requires the first word after “overwhelms” to be a noun to return anything. English noun phrases being what they are, a lot of perfectly good noun phrases that start with articles or adjectives are not being detected by my search. That is, “overwhelms people” comes back as a hit, but “overwhelms a person” does not. In short, my searches for the relative frequencies of subjects and objects are not very valid – at best they tell me that pronouns are fairly common (pronoun searches do not suffer from the problem above since they don’t follow articles or adjectives). The same search problem vitiates all of my verb/past participial adjective searches, so take them all with grains of salt. I’m not going to give rankings to these due to the Sochi-like untrustworthiness of the results.

Now for something a bit more concrete: What specifically are the nouns that come with these verbs and adjectives?

	verb subj #1	#2	#3	pres adj subj #1	#2	#3
excite	light	work	energy	future	life	home
annoy	thing	noise	voice	thing	noise	voice
overwhelm	world	love	life	evidence	response	odds
inspire	story	work	music	story	passion	work
amaze	love	place	thing	food	product	stuff

Here we see some interesting things – the top 3 subjects for both verbs and present participial adjectives are the same for “annoy”/”annoying”, and 2 of the three are the same for “inspire”/”inspiring”. Here is an update ranking of “closeness”:

	Test #1 ranking	Test #2 ranking
excite	5	3 (tie)
annoy	2	1
overwhelm	1	3 (tie)
inspire	3	2
amaze	4	3 (tie)

	verb obj #1	#2	#3	past adj subj #1	#2	#3
excite	people	electrons	students	team	people	company
annoy	people	users	others	people	fans	others
overwhelm	people	students	visitors	heart	system	clinics
inspire	people	confidence	others	post	pork	owl-ways
amaze	people	visitors	audiences	people	doctors	kids

First of all, the #1 object for all of the verbs is “people”, as in “it excites people” or “it inspires people”. There is more variety in the subjects of past participial adjectives, but 2 of the top hits are still “people” – “people are annoyed” and “people are amazed”. The objects suffer from the same validity problem outlined earlier, so take this with even more salt, but we can say at least that “people” is a common object for verbs whose past participial adjective counterparts have the subject “people”.

Last but not least, the #1 hits as a percent of the total hits. The first column is the #1 subject for the verb as a % of the hits for that verb, and the second column is the #1 subject for that adjective as % of the hits for that verb. The 3rd column is the 1st divided by the 2nd – the closer to 1 that number is, the more similar the proportion of those two words in the hits.

	verb subj #1 as % of total	pres part adj #1 as % of total	ratio	pres adj subj	verb subj as % of total	ratio
excite	0.046	0.014	3.295	0.037	0.001	62.500
annoy	0.103	0.103	1.000	0.088	0.088	1.000
overwhelm	0.042	0.004	9.667	0.219	0.006	35.000
inspire	0.057	0.059	0.968	0.159	0.159	1.000
amaze	0.030	0.008	3.667	0.071	0.003	27.648

Based on the above, “annoy” and “annoying” are quite “close”, but “overwhelm” and “overwhelming” are not. This is the last test to count toward official rankings:

	Test #1 ranking	Test #2 ranking	Test #3 ranking
excite	5	3 (tie)	4
annoy	2	1	1
overwhelm	1	3 (tie)	5
inspire	3	2	2
amaze	4	3 (tie)	3

That gives us an idea of which of the verb/present participial adjective pairs is “closest” according to all of the tests I ran. But first, the same results for verbs/past participial adjectives:

	verb obj #1 as % of total	past part adj #1 as % of total		#1 as % of total past adj subj	verb obj #1 as % of total	total nouns
excite	0.160	0.003	60.377	0.160	0.063	2.534
annoy	0.465	0.465	1.000	0.296	0.296	1.000
overwhelm	0.286	0.000	Div 0	0.130	0.043	3.000
inspire	0.149	0.000	Div 0	0.041	0.000	Div 0
amaze	0.257	0.257	1.000	0.226	0.226	1.000

Likewise for verbs and past participial adjectives – “annoy” and “inspire” are closer to “annoyed” and “inspired” than “overwhelm” is to “overwhelmed”.

So to close this post, here are the final overall rankings for “closeness” of verbs and present participial adjectives:

	Test #1 ranking	Test #2 ranking	Test #3 ranking	Overall
excite	5	3 (tie)	4	5
annoy	2	1	1	1
overwhelm	1	3 (tie)	5	3
inspire	3	2	2	2
amaze	4	3 (tie)	3	4

“Annoy” is the clear winner, followed not too distantly by “inspire”.

Since I still have the data that I used in the last post, I can tell you that these rankings correlate with the % of uses that were verbs at -0.51 (that is, numerically higher rankings, i.e. 4 and 5, tended to be used as verbs less than as adjectives). That says to me that the words that have a fully established independent life as adjectives tend to have less in common collocation-wise with their verb parents. This makes sense – the average child hears the adjective “exciting” applied to all kinds of activities before he ever learns that light “excites” electrons and has to form the relationship between those two words post hoc. I guess, although I don’t know, that “inspire” and “inspiring” might tend to be acquired much closer to the same time in a child’s life and in more similar situations.

As always, further research is required! Who has a paid BYU corpora membership?