Every semester I throw a bunch of survey data, biographical data, and assignment scores from my classes into an Excel sheet and see what pops up. This semester, like the last one, yielded some interesting information.
The tl;dr version is:
- Work is a huge predictor of low grades
- I should continue to push the importance of drafts in writing
- I need to be careful not to evaluate students too much on their familiarity with my style of class
- Perhaps I need to design better questionnaires
Read on for the details.
Rough drafts and final drafts both predict final grades well
This semester, I taught 2 academic writing classes intended to prepare ESL students for transfer-level writing courses. Each class featured at least 2 take-home essays among various other writing assignments. These often make up a large % of the final grade for the class. It should not be surprising then that the scores for final drafts correlated rather strongly with final grades for the course: 0.63, 0.81, 0.64, 0.67, and 0.87 (mean: 0.72) for all the take-home essays in my 2 writing classes. On the other hand, rough drafts were just 1 or 5 points of the 50 or 100 points that the whole essay was worth, which made them a supremely tiny fraction of final grades. But surprisingly, the correlations between scores for these rough drafts and final grades were not that much lower than for the final drafts: 0.82, 0.66, 0.66, 0.44, 0.12, 0.03, 0.53, and 0.74 (mean: 0.50. Those two low scores in the middle were both drafts of the same essay, oddly enough). Rough draft scores correlated with final draft scores at 0.57 and 0.84. What this says to me is that I should continue to give points, perhaps more, for draft 1 of every essay and emphasize its importance. Incidentally, my students didn’t like the activities I used to review draft 1 – mostly reading circles-style peer review – but I view that as another incentive for them to actually write it and put some thought into it more than as a corrective tool. I believe I will continue to do this, making some changes to my reading circles peer review sheets to make them more useful or just faster.
The college bonus and the work penalty
My Excel sheet has some data from a demographic survey at the beginning of the semester. There were very small correlations between final grades and years living in the US (same finding as last semester), number of countries lived in, and whether the student is repeating the course. There was a somewhat small correlation between having a college degree already and final grades (0.23). A bit less than 36% of my students already have degrees in their countries of origin, and these students did somewhat better in my classes. There was also a medium-sized negative correlation between currently having a job and final grades: -0.40. Not surprisingly, a large part of this seems to be attendance; those scores were correlated with having a job at -0.56.
This is a problem in community college in a way that it wouldn’t necessarily be in other contexts. If I taught at a 4-year university, I might be able to rationalize this as the cost of not prioritizing one’s education enough. As it is, in community college, classes exist specifically to serve people who don’t or can’t participate in the more demanding 4-year system. On one hand, it won’t get any easier for them after my class. On the other hand, I (and we) should probably be doing more to accommodate people that our milieu exists specifically to accommodate.
Attendance and take-home writing vs. attendance and quizzes
Attendance, thankfully, was correlated positively with every other component of final grades. However, it was much more strongly correlated with writing (0.69) than with grammar quiz scores (0.40). This tells me I’m probably a better writing teacher than a grammar teacher – or that grammar is just harder to teach in 4 months. Or maybe my belief in the latter causes the former.
Activities they were comfortable doing and final scores
The above is the survey that I gave out at the beginning of class. Out of the “5 activities that you feel the most comfortable doing”, some notable correlations with final scores were:
- “shopping”: 0.22
- “working”: -0.31 (-0.45 with grammar quiz scores)
- “solving grammar problems”: 0.22 (0.29 with grammar quiz scores)
- “writing in college”: -0.14
Again, there is a sizeable penalty for working that seems to more than offset any gains in fluency that come from work.
I’m a bit puzzled as to why “solving grammar problems” would yield a bonus in scores, while “writing in college” would not. Perhaps it is related to the types of writing students are thinking of when answering that question: students who checked this box were often skilled writers but not in the academic style that we use. It is possible that these students may have gotten positive feedback on purely grammar-focused writing assignments in previous semesters.
Activities they had used before and final scores
Out of the “5 activities that you have used the most in English classes”, some notable correlations with final scores were:
- “translation”: -0.13
- “copying words many times”: -0.11
- “reading short articles: 0.37
- “group problem-solving”: -0.12
- “listening to audio”: 0.20
Looking at these scores together, they really seem to be asking students “Have you taken an ESL class before?” (aside from the “group problem-solving” question, which is anomalous). It raises the question whether our ESL classes are valid in that they really test and reward English skill rather than the know-how of getting by in ESL classes, none of which have much translation or rote learning and most of which have short readings and audio materials.
Study methods from a random week and final scores
At a random point during the semester, each class got a survey of how many hours during the previous week they had spend doing various study or study-related activities, including reviewing class notes, using social media. These also ended up correlating with final grades in strange ways. To cut to the chase, only one activity was positively correlated with final scores:
- Using social media in English: 0.11
The rest were almost all negative:
- Reviewing vocabulary on websites: -0.19
- Reviewing grammar on websites: -0.19 (-0.50 with grammar quizzes!)
- Reading other books (besides textbooks): -0.23
- Using learning apps: -0.15
- Talking to teachers: -0.15
To be honest, I’m not sure what to make of all this. I’ll probably change the format (many students were clearly guessing on social media use – one student wrote “300 hours”) and try again next semester. In the meantime, I will be more careful to design assignments – in particular grammar quizzes – that track something most of us would agree resembles real-world language use.
UPDATE: The work penalty
I went through the list again, filling in missing info from what I remember students saying (some didn’t fill out the surveys but talked about their jobs to me or in class discussions) and ended up with an even larger penalty: -0.51. I also went through the grade sheets in detail and pulled different types of homework assignments for comparison.
It turns out that the work penalty, as I’m calling it, is more associated with certain types of evaluation than with others:
You can see above that working students tended to get lower grades overall on every component of final grades besides extra credit (which tends to be done sporadically within the class but usually by the same people).
Also, not every type of homework was equally affected; language logs (weekly online logs of incidental language use or input) and reading circles (full-page worksheets on one aspect of a book or article that the entire class read) were more penalized than other types of homework. It’s hard to draw a pattern from this, but one possible explanation is that types of homework that I used more than once and/or homework that took a long time to complete were more likely to be skipped by busy students.
The question I find myself facing now is how much I can modify these assignments to be more work-friendly without undermining the main goals of the course.
FURTHER UPDATE: Stuff I neglected to mention
The difference between working and non-working students in final grades was significant at p<0.01.