Correlations with final grades, spring 2018 edition

Every semester I throw a bunch of survey data, biographical data, and assignment scores from my classes into an Excel sheet and see what pops up.  This semester, like the last one, yielded some interesting information.

The tl;dr version is:

  1. Work is a huge predictor of low grades
  2. I should continue to push the importance of drafts in writing
  3. I need to be careful not to evaluate students too much on their familiarity with my style of class
  4. Perhaps I need to design better questionnaires

Read on for the details.

Rough drafts and final drafts both predict final grades well

This semester, I taught 2 academic writing classes intended to prepare ESL students for transfer-level writing courses. Each class featured at least 2 take-home essays among various other writing assignments. These often make up a large % of the final grade for the class. It should not be surprising then that the scores for final drafts correlated rather strongly with final grades for the course: 0.63, 0.81, 0.64, 0.67, and 0.87 (mean: 0.72) for all the take-home essays in my 2 writing classes. On the other hand, rough drafts were just 1 or 5 points of the 50 or 100 points that the whole essay was worth, which made them a supremely tiny fraction of final grades. But surprisingly, the correlations between scores for these rough drafts and final grades were not that much lower than for the final drafts: 0.82, 0.66, 0.66, 0.44, 0.12, 0.03, 0.53, and 0.74 (mean: 0.50. Those two low scores in the middle were both drafts of the same essay, oddly enough). Rough draft scores correlated with final draft scores at 0.57 and 0.84. What this says to me is that I should continue to give points, perhaps more, for draft 1 of every essay and emphasize its importance. Incidentally, my students didn’t like the activities I used to review draft 1 – mostly reading circles-style peer review – but I view that as another incentive for them to actually write it and put some thought into it more than as a corrective tool. I believe I will continue to do this, making some changes to my reading circles peer review sheets to make them more useful or just faster.

The college bonus and the work penalty

My Excel sheet has some data from a demographic survey at the beginning of the semester. There were very small correlations between final grades and years living in the US (same finding as last semester), number of countries lived in, and whether the student is repeating the course. There was a somewhat small correlation between having a college degree already and final grades (0.23). A bit less than 36% of my students already have degrees in their countries of origin, and these students did somewhat better in my classes. There was also a medium-sized negative correlation between currently having a job and final grades: -0.40. Not surprisingly, a large part of this seems to be attendance; those scores were correlated with having a job at -0.56.

This is a problem in community college in a way that it wouldn’t necessarily be in other contexts.  If I taught at a 4-year university, I might be able to rationalize this as the cost of not prioritizing one’s education enough. As it is, in community college, classes exist specifically to serve people who don’t or can’t participate in the more demanding 4-year system. On one hand, it won’t get any easier for them after my class. On the other hand, I (and we) should probably be doing more to accommodate people that our milieu exists specifically to accommodate.

Attendance and take-home writing vs. attendance and quizzes

Attendance, thankfully, was correlated positively with every other component of final grades. However, it was much more strongly correlated with writing (0.69) than with grammar quiz scores (0.40). This tells me I’m probably a better writing teacher than a grammar teacher – or that grammar is just harder to teach in 4 months. Or maybe my belief in the latter causes the former.

Screen Shot 2018-05-26 at 22.00.49.png

Activities they were comfortable doing and final scores

The above is the survey that I gave out at the beginning of class. Out of the “5 activities that you feel the most comfortable doing”, some notable correlations with final scores were:

  • “shopping”: 0.22
  • “working”: -0.31 (-0.45 with grammar quiz scores)
  • “solving grammar problems”: 0.22 (0.29 with grammar quiz scores)
  • “writing in college”: -0.14

Again, there is a sizeable penalty for working that seems to more than offset any gains in fluency that come from work.

I’m a bit puzzled as to why “solving grammar problems” would yield a bonus in scores, while “writing in college” would not. Perhaps it is related to the types of writing students are thinking of when answering that question: students who checked this box were often skilled writers but not in the academic style that we use. It is possible that these students may have gotten positive feedback on purely grammar-focused writing assignments in previous semesters.

Activities they had used before and final scores

Out of the “5 activities that you have used the most in English classes”, some notable correlations with final scores were:

  • “translation”: -0.13
  • “copying words many times”: -0.11
  • “reading short articles: 0.37
  • “group problem-solving”: -0.12
  • “listening to audio”: 0.20

Looking at these scores together, they really seem to be asking students “Have you taken an ESL class before?” (aside from the “group problem-solving” question, which is anomalous). It raises the question whether our ESL classes are valid in that they really test and reward English skill rather than the know-how of getting by in ESL classes, none of which have much translation or rote learning and most of which have short readings and audio materials.

Study methods from a random week and final scores

At a random point during the semester, each class got a survey of how many hours during the previous week they had spend doing various study or study-related activities, including reviewing class notes, using social media. These also ended up correlating with final grades in strange ways. To cut to the chase, only one activity was positively correlated with final scores:

  • Using social media in English: 0.11

The rest were almost all negative:

  • Reviewing vocabulary on websites: -0.19
  • Reviewing grammar on websites: -0.19 (-0.50 with grammar quizzes!)
  • Reading other books (besides textbooks): -0.23
  • Using learning apps: -0.15
  • Talking to teachers: -0.15

To be honest, I’m not sure what to make of all this. I’ll probably change the format (many students were clearly guessing on social media use – one student wrote “300 hours”) and try again next semester. In the meantime, I will be more careful to design assignments – in particular grammar quizzes – that track something most of us would agree resembles real-world language use.

UPDATE: The work penalty

I went through the list again, filling in missing info from what I remember students saying (some didn’t fill out the surveys but talked about their jobs to me or in class discussions) and ended up with an even larger penalty: -0.51. I also went through the grade sheets in detail and pulled different types of homework assignments for comparison.

It turns out that the work penalty, as I’m calling it, is more associated with certain types of evaluation than with others:

Screen Shot 2018-05-28 at 11.30.32.png
Left to right: Final grades (blue), attendance (red), homework (yellow), take-home writing (green), grammar quizzes (purple), extra credit (turquoise).  Among specific types of homework: Language logs (pink), reading circles (green), misc. others (red).

You can see above that working students tended to get lower grades overall on every component of final grades besides extra credit (which tends to be done sporadically within the class but usually by the same people).

Also, not every type of homework was equally affected; language logs (weekly online logs of incidental language use or input) and reading circles (full-page worksheets on one aspect of a book or article that the entire class read) were more penalized than other types of homework. It’s hard to draw a pattern from this, but one possible explanation is that types of homework that I used more than once and/or homework that took a long time to complete were more likely to be skipped by busy students.

The question I find myself facing now is how much I can modify these assignments to be more work-friendly without undermining the main goals of the course.

FURTHER UPDATE: Stuff I neglected to mention

n=74

The difference between working and non-working students in final grades was significant at p<0.01.

Advertisements

5 thoughts on “Correlations with final grades, spring 2018 edition

  1. You’re absolutely correct in concluding that work hours have a negative affect on course success, probably mediated by the effect on attendance. So any steps you can take to address this issue are going to benefit students.

    You’d find the same work – attendance – grade relationships at most 4-year institutions as well, and this is particularly true of regional state universities (e.g. the University of Wisconsin – Parkside) and branch campuses (e.g. University of CT at Stamford). Few students at any college or university get a full ride, and those who don’t get a full ride from their parents (i.e. most people) are going to be working as well as taking classes.

    On the other hand, your sample size probably doesn’t have enough statistical power to allow you to interpret correlations < .50. However, pooling the responses from several classes would increase the n, which could solve this problem and provide more accurate information regarding how different course components affect student course success.

    Liked by 1 person

    • Funny how I neglected to mention those very important numbers. I update the post with n=74 (3 classes) and at least one t. I try to do the same surveys across all my classes every term.

      Like

      • Good to know the sample size. A quick look at the appropriate power table (e.g. http://www.unc.edu/~nielsen/soci708/cdocs/cohen.pdf) indicates that a sample size of 74 only provides sufficient power to detect large effect sizes for p-values of .05 or .01. Of course, strictly speaking, the above holds when the analysis is restricted to a single independent variable and a single dependent variable – which is not the case here, since there are quite a few independent variables that are related to each other. But that’s not a deal breaker (a) because this is an exploratory analysis and (b) because, as I commented earlier, the sample size doesn’t provide enough power to make claims one way or the other for variables with smaller r-values, which means you can simply ignore them and focus on the factors that are strongly associated with outcomes.

        So you’re now in a good place, research-wise, because your exploratory analyses have identified a few key factors (i.e. variables with r > .40) that you can start to look at in more detail immediately. In the meantime, you don’t have to discard variables that might be associated with more modest effects – they can be analyzed at a later date (i.e., when you’ve accumulated larger samples that have the power to detect the moderate to modest effects that may or may not be associated with them).

        Now you should be able to dig deeper into your data to determine the relative effects of work/attendance, previous knowledge, and key class assignments (grammar quizzes, essays) on course outcomes. My personal preference would be to perform a step-wise regression examining the effects of key class assignments after controlling for prior level of student knowledge (step 1) and work/attendance (step 2). You’ve got a possible measure of prior level of student knowledge in rough draft scores, and you could use work hours as a proxy for external factors affecting student performance. If you didn’t want to do a regression analysis, you could simply examine partial correlations. That is, what does the relationship between grammar quizzes or essays and course outcome look like after you’ve partialed out prior student knowledge or # of work hours/attendance? Your current sample should be large enough to start to address these questions. I think the results might be very interesting.

        Liked by 1 person

      • I’m looking more into the work question over summer. Are you still doing research? You seem to remember terms that I forgot within 6 months after finishing my MA.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s