Teach a man to find correlations, he posts them for a lifetime

Aphorism showing its age aside, this post is designed for both men and women who use Canvas and are curious about statistics that may be hiding in their classes’ grades.

I have my own data to share about this semester’s classes, but first, here is a tool that you can use to do the same:

Stat sheet for grades 1.1

And an explanation of how to use it:

On to what I found.

I had 4 classes this semester – 2 Oral Language classes and 2 Written Language classes, both in the 2nd to last term of my university’s IEP. My university’s IEP works a bit unusually – my 4 classes were just 2 groups of people meeting for 4.5 hours a day 4 days a week, about half of which was “Oral Language” and half of which was “Written Language”. The first group of people were my students for the first “term” (=half of a semester), and the second group were mine for the second term. All told, I still had 4 gradebooks on Canvas to export and fiddle with. Between the 4 of them, I found these interesting statistical tidbits:

Scores of 0 are more predictive of final grades than full scores are

One would expect the number of 0s on assignments to negatively correlate with final grades, and the number of full scores to do the opposite. That is, thankfully, true. However, they correlate at different rates – across all my classes, on average, 0s are more (negatively) correlated with final grades than full scores are (positively) correlated. The reason for this is that full scores were more evenly distributed among all students than 0 scores, which were concentrated among a few students. The one class for which this was not true was the one that I changed my late work policy and started giving 1/2 credit for certain late assignments.

This would not be a cause for any particular change except for 2 reasons: 1) as shown by the last class, many of the 0s that students were getting were from late work rather than unsubmitted work, and 2) we have a fairly strict policy about grading by SLOs (student learning outcomes, one of the first abbreviations I had to learn upon my return to the USA after years in Japan), and nowhere in our SLOs does it say that students should learn the sometimes-merciless grading policies that one may encounter at university.

Therefore, I should really make the “late work gets partial credit” policy permanent. I should also probably give fewer full scores.

5% 0s is a line in the sand

I enjoy running t-tests to see what values in what grade categories produce statistically significant differences (p=0.01) in my students’ final grades. One t-test I ran (on the “other stats” sheet in the file linked above) was seeing if students who missed 5% of assignments were different in statistically significant ways from those who didn’t. It turns out that they are, in all 4 of my classes this semester. On the other hand, those who missed 2% of assignments weren’t. Perhaps I should give an opportunity to make up homework on about 2% of assignments (as I already do for classwork).

I’m hoping that my future classes have grades that reflect the average quality of their work, which in turn reflects their ability to do academic work in English, rather than their tendency to check due dates and read rubrics thoroughly on Canvas. These are important skills, but I won’t want to make them a bottleneck through which every grade must pass.

RDs need a bump, FDs need a nerf

Across 4 essays in both Written Language classes, the average correlation of rough draft scores with final grades was 0.70. The average correlation of final draft scores with final grades was 0.76. Since final drafts are worth at least twice as many points as rough drafts, this is rather surprising – even moreso because for 3 of the 4 essays, the rough drafts’ correlations are actually higher than the final drafts’ (the last had a very low correlation for the rough drafts).

I’ve been making changes to my writing process over the last few semesters, and it seems I need to make a few more. I think part of the comparitively low correlations that final drafts have is due to my grading practices – I think I take it easier on final drafts precisely because they’re so many points. My average scores for final drafts are higher than for rough drafts, and the standard deviations as lower – roughly 62%-95% with an average of 78% for rough drafts and 65%-95% with an average of 80% for final drafts. It’s not a huge difference, but looking back at the scores now they don’t seem to reflect the range in quality of the essays. Part of the high correlations for the rough drafts is also due to the skills that are involved in producing a first draft – planning, reading, responding to a prompt, and a bit of grammar – that are assessed in a lot of other assignments as well. Final drafts, meanwhile, assess (in addition to the same things that first drafts assess, but less directly) responding to criticism and editing, which don’t figure largely in many other assignments. Seeing how first drafts track more of the skills that I care about, and I seem to grade them with less of a high-stakes mentality, I should probably weight them more. On the other hand, since final drafts have a somewhat narrow range of skills that they assess, I should weight them less, or even separate my grades for final drafts into smaller sub-assignments like the COCA assignments I currently use, but also a written response to criticism and proof of visiting tutors instead of trying to indirectly read those things into the final draft.

I need to keep in mind too that I’m not necessarily serving my students well if I introduce them into a writing process that none of their psychology, history, or any other professors will use – I hear that most papers turned in for any class other than English are just the final drafts, already assumed to be revised and polished to a sheen. Maybe having one paper like this per term is also justifiable just in terms of preparing students for being taught by PhDs who know more than anyone else in the world about the behavior of certain species of field mice under certain conditions but have never studied pedagogy.

Look forward to more like this same time next semester, and let me know if you find the sheets useful for your own classes.

6 thoughts on “Teach a man to find correlations, he posts them for a lifetime

  1. “Across 4 essays in both Written Language classes, the average correlation of rough draft scores with final grades was 0.70. The average correlation of final draft scores with final grades was 0.76.”

    How strong is the correlation between initial draft score and final draft score? That is an important piece of the puzzle.

    Since both first draft and final draft scores are related to the course grade, you need to find out the relationship of final draft scores with final grades after controlling for rough draft scores. Why? Because it’s possible that what the correlational analysis is actually measuring is not the effect of the writing project (operationalized as the final draft score) on student knowledge (operationalized as the final course grade), but the effect of the student’s prior knowledge of the language before enrolling in the class (which is likely to vary among students, which is likely to contribute to the final course grade, and which is likely to be exhibited by their initial drafts).

    This type of analysis might find that after controlling for prior knowledge (operationalized as the initial draft score), revision and submission of a final draft IS NOT related to the final course grade. Or it might find that even after controlling for prior knowledge (as exhibited by first drafts), there IS a strong relationship between final draft scores and final course grades. A simple correlational analysis will not provide the information needed to decide between these two very different possibilities.

    Long story short, without an analysis that incorporates the appropriate controls, it’s not possible to conclude what effect this aspect of the course has on student learning. That’s the problem with relying only on simple correlational analyses, unfortunately. 😦

    Also, doing many simple pair-wise comparisons with an alpha level of .05 or .01 increases the possibility of Type I error, and you should find some way to correct for this. There are lots of ways to do this. See for example: https://www.researchgate.net/post/Is_it_necessary_to_do_correction_to_significance_level_while_conducting_Pearson_correlation_coefficients_between_different_continuous_variables

    Safe travels, if you’re traveling during the school break. Hope you have a great holiday! πŸ™‚

    Liked by 1 person

      1. Well, I enjoy learning about statistics and performing statistical analyses – it can be fun πŸ™‚ But beyond that, as an experienced practitioner, you can trust your own perceptions and observations as reliable sources of information. So I’m curious to hear your thoughts regarding the strengths and weaknesses of the format you described (2 8-week sessions?), what you think worked and what went over like a lead balloon this term, differences and similarities between the students you’ve taught in California and students where you are now, etc. The end of the term provides a time for reflection, and gives us an opportunity to reinvent ourselves (or at least our courses) – so statistical analyses aside, what do you see as the next step? πŸ™‚

        Liked by 1 person

      2. It’s going to be more like 6 weeks soon (4 days a week, 4.5 hours a day). The advantages are mostly in throughput and providing more entry points to students moving to Utah from other countries (and I did figure out that the 6 terms function more as entry points than exit points). One downside is that you end up unable to give long-term projects to be completed at home, but many writing classes seem to be leaning towards a more workshoppy approach anyway, and 4.5 hours a day is great for that.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s