End-of-semester Quantitative Feedback

(Written to give my mind a rest from grading)

Introduction and Pedantry

At the end of the semester I like to use a survey to gauge what students found valuable in my class.  The survey is just a list of class activities from the semester and then two columns with spaces for scores – an “I like it” column and an “It helps me learn” column.  There is a Likert-style scale of 1-5 to be used with both columns across the top of the sheet.  So for example, a student who really enjoyed our reading textbook but doesn’t feel like it was useful for learning would give it scores of 5 (in answer to “I like it”) and 2 or 1 (in answer to “It helps me learn”).

Screen Shot 2017-12-15 at 12.32.42.png
More or less like this.  Screenshot of MS Word with fun, colorful red and blue underlines kept intact.  In the future I will probably make this more intuitive-looking – some students opted to fill in the blanks with full written critiques.

I plan on standardizing this survey across my classes in the future, but this semester everyone had a different list of activities.  I know from their survey, for example, that my intermediate integrated skills class enjoyed their grammar book more than their reading book, but because I referred to these by name in the survey, I can’t really compare their answers to those for the reading and grammar books in my academic writing classes (naturally, the books themselves are different in structure and approach as well, which limits how comparable they are).  All of my classes did have a few items which were worded the same and were similar enough in practice to warrant comparison.

Those items were reading circles, Kahoot!, at-home writing, language logs, and teacher-fronted grammar lessons.  Before we get to the meat of this post, let me just make sure everyone knows what those are and explain how I do them.

Reading circles, in a nutshell, consist of reading reflection groups where each member of the group has a different “job”.  A group of 5 people in most of my classes might have had a Summarizer, a Vocabulary Enricher, a Grammarian, a Connector (who had to, for example, find articles on similar topics to the reading on the Internet), and an Artist.  On a reading circles day, everyone would have read a section of a book or an article over the weekend, and completed half of the sheet at home.  They would then gather in “expert groups”, consisting of people with the same job, and compare answers.  Some versions of my reading circles worksheets had a part of the worksheet that had to be completed during this time.  After a decent amount of time, they would meet with their reading circles group members, all of whom had different “jobs”.  All of my reading circles sheets had short sections that had to be completed during this time by listening to the other group members.  Much of this is standard for this kind of activity.  In practice, not everyone would do their homework, and 3 or 4 people out of a class of 25 would be hurriedly filling in the parts that they were supposed to have done over the weekend in their “expert groups”.  I noted who didn’t do the assignment for grades but let them do this so that they would have at least something to show their reading circles afterward.  Students always seemed much more engaged during reading circles than any teacher-fronted activities, but as we shall see, that isn’t necessarily reflected in the answers they gave to the survey.

Kahoot! is an online game-show-like platform that seems pretty well-known, although I’d never heard of it before my first CATESOL meeting last December.  I mostly used it to review readings (the same ones as the reading circles), about 3 times a semester, with Jolly Ranchers candy as prizes for the winning teams.

At-home writing comprises paragraphs and essays, any of which had at least 3 drafts.  Students turned these in on paper, Canvas, Turnitin.com or all of these at once.  As I found out a bit late, the steps for submitting work electronically or viewing feedback are not obvious for many students, and as in my time in Japan, there is a strong pro-handwriting bias among ESL students – some students view typing it out as the very last step in completing a paper.  Anyway, this was one of the few times in the semester that students would get individual feedback on their writing from me.

Language logs are simple scaffolds for out-of-class input.  They look more or less like schedules organized by weeks with spaces for students to write what they read and what they noticed (for writing classes – “noticed” here could mean content or form) or what they read, what they watched, and who they talked to (for integrated skills classes).  The spaces are intentionally kept small to keep the focus on input rather than rigorous and thorough reporting.  For me, these have a lot of room for improvement – I personally kept forgetting that students had them (I had planned to check them every 2 weeks, but it ended up being more like every 4 weeks), and the students reciprocated.  I also had to remind students quite a few times that the purpose of the logs was to record their extra input, not to record the homework that I had assigned them, and that conversations with their spouses in their first language didn’t count as language log material.  Also, the “I noticed…” sections were often filled with verbatim quotes rather than reflections.  Still, a number of students rose to the occasion and read, watched or talked voluminously.  I remember seeing written on language logs entries like “I talked to a woman at the supermarket about expensive eggplants” “CNN – California wildfire – scary!!” “Breitbart – Anti-Trump conspiracy” and plenty of other windows into my students’ intellectual lives.  Yes, I’m proud of the student who reads Breitbart – I suppose in terms of acculturation it’s somewhat analogous to Americans in Japan who become ardent supporters of the Imperial system and all of its apologia.  A sure sign of language learning progress, albeit also a phase I hope they grow out of.

A confounding factor for measuring how much students liked/valued the logs themselves is that I also had them share them with classmates before turning them in.  The discussions that arose from this were almost always lively and engaging, and it is certainly possible that some students answered positively for Language Logs while mostly thinking of the enjoyable conversations around them rather than the input that is their main purpose (at least from my perspective).

Teacher-fronted grammar lessons are probably familiar to most readers of this blog.  Mine are not particularly unusual, I think, except that I tend to give absurd examples and lots of analogies to food (an independent clause is a burger, a dependent clause is fries, and adverbials are drinks and toppings).

Numbers and stuff

On to the data.

Screen Shot 2017-12-13 at 9.10.38.png

For all 3 classes, the most popular class activity was Kahoot! followed closely by grammar lessons.  The one most viewed as helpful was at-home writing, followed again by grammar lessons.  That’s a bit interesting.  The other values on the table are a bit more interesting.

The second column, consisting of t-values, shows basically how meaningful the differences between the “I like it” and “It helps me learn” are.  t-values, if I recall correctly from the last time I googled them, are roughly the odds that a difference between 2 populations (or a change in 1 population) could have been coincidence even if the populations themselves are actually not different with regards to the value you are testing.  Generally, the null hypothesis (that there is no significant difference between the populations tested) is rejected if t is below 0.05 or 0.01.  The computed value of t depends on the differences between the populations’ answers and on the size of the population.  I only computed t for “I like it” and “It helps me learn” scores for the same activity, and the numbers in the center column are those t-values.  As you can see, the only one that would pass a conventional test for significance is at-home writing, although grammar on the whiteboard is close.  This tells us that the different values for “I like it” and “It helps me learn” for at-home writing are probably large enough for us to assume that a difference would be found even if I taught thousands of students instead of about a hundred.  I find this interesting mostly because it shows how large the gap is between enjoyment and valuation of paragraphs and essays – a gap which might generally be found among students who feel that some things that aren’t enjoyable are nonetheless good for your brain, which might call the eating your vegetables effect.  (I would be tempted to conclude that the relative lack of enjoyment causes the feeling that it must be useful except that Language Logs have an even lower enjoyment score and a correspondingly low usefulness score.)

The last column is standard deviation, or how widely answers are dispersed.  As you can see, “at-home writing helps me learn”‘s answers are the least dispersed of any item, meaning that there was higher consensus around the usefulness of at-home writing than, say, Kahoot!.  This means that not only was the mean higher, showing that on average more people found it useful, but people agreed more on how useful it was.  Language logs, on the other hand, had wide disagreement on their usefulness (and enjoyability).  It seems that students are much more unanimous on some questions than others.

Last, I have the correlations.  Not too much to say about this, except that liking/valuing Kahoot! is negatively correlated with almost everything else.  The positive correlations between reading circles and Language Logs could be explained by the social nature of both (see the confounding factor of the Language Logs above).  I have no idea what could be behind valuing reading circles and valuing grammar on the whiteboard/projector.

Screen Shot 2017-12-13 at 9.05.52.png

Discussion and hedging

One must keep in mind that students are likely judging the usefulness of activities based on changes in their abilities that they can detect; a very long-term effect or a subconscious one will be mostly invisible and may feel useless, while one that gives the rush of endorphine that comes from solving a puzzle may not be as effective in long-term acquisition but will seem to have led to some understanding.  This is a circuitous way of saying that we can’t trust students at the end of a 4-month course to know what actually helped them learn.  I tend to regard the Language Logs as the most beneficial, because they 1) facilitate large amounts of input, 2) are student-directed and therefore more likely to keep their interest, and 3) are the most likely to be continued outside of class.  Of course, stuff that seems pedagogically useless to students is not likely to lead to re-registration in the spring, and if students don’t sign up for classes, it’s hard to say I served them well by insisting on nutritious but unrewarding educational broccoli.  Activities like Kahoot! may be worth the time and effort if only to provide the hit of pure enjoyment that keeps people looking forward to the next serving of ice cream when they’ve finished their vegetables.

Advertisements

5 thoughts on “End-of-semester Quantitative Feedback

  1. This is great, and has made me think that the mess around setting up Kahoot might not be worth the effort (with evidence supported by my Quizlet experiment).

    I am going to massively redesign a couple of courses this year and you’ve given me ideas about evaluation of course quality.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s