Beck Wise's blog

How to Outsource Your Grading and Look (and Feel) Good Doing It

Beck Wise — Wed, 28 Jan 2015 22:09:43 +0000

Author:

Image Credit:

"Crowdsurfing at the Tocotronic show". Photo: daspunkt.

Or, The Power of Crowdsourcing Assessment.

Like a lot of instructors at UT, I have required presentations in my classes and over the years, these presentations have taken a lot of different forms, from three solid days of argumentative presentations to close out the semester in my first-year writing class, to having students introduce a critical section of the text and lead discussion in my current literature class. One thing that hasn't changed, though, is the way I assess presentations. Which is to say: I don't.

This isn't to say that presentations aren't assessed, though. Whether course grades are determined by the instructor, in the traditional mode, or argued for by students in the Learning Record, I consider it critical that students receive concrete feedback on their various achievements within their presentations -- and further, that this feedback come from more than just the instructor.

Peer review for writing assignments gives students the opportunity to receive criticism and guidance from others in the class as they move through their writing process towards complete drafts, but it's tough to think of an equivalent opportunity for presentations. While you do write and rewrite in preparation, the nature of a classroom speech is essentially one and done. It's rare for students to present more than once in any single time-strapped college course, and there's certainly no way to revise and resubmit! The best a student can hope for is substantive feedback on their presentation that's specific to that class but generic enough to apply to the presentations they'll have to make later in their careers.

However, you can bring in more voices through peer assessment. Whenever I require presentations of my students, I also require that they provide feedback to each of their peers and that they record the grade the speaker has earnt. The grade isn't the important bit, so I won't spend time here on the various ways I've handled those in the past. But receiving written feedback from 20 or so peers on a single piece of work is the single most effective way I've found of letting students self-identify things to work on. Instead of yet another professor insisting that they speak slower, they get 15 classmates noting that the speed of the presentation made it tough to understand: they see the pattern and can act accordingly. In addition, students gain practice listening critically and assessing performance--and they tend to like the idea that their comments and assessments matter.

I can't deal with reams of paper anymore, so instead of handing out paper comment cards or the like, I use Google Forms to administer these presentation assessments, using the computers in my current networked classroom or, in non-digital spaces, having students bring their own devices and having a few paper worksheets on hand for those who don't have or prefer not to use web-capable tech in class. Google Forms--online questionaires whose responses automatically populate a spreadsheet on your Google Drive--have several affordances that make them well-suited to this kind of exercise:

Students can fill them in anonymously, with no sign-in required (or the possibility that their handwriting will be recognised by their peer), but I can still ask them to self-identify alongside their feedback; this lets me monitor the process to ensure that everyone is participating fully and seriously, and helps make sure the process doesn't further prove the Greater Internet Fuckwad Theory
You can set up questions with a variety of answer types, from multiple choice (with one or more possible answers) to checkboxes to short open responses to paragraphs to drop-downs. I use short answer questions to collect the presenter's and reviewer's names, multiple choice questions to assess the specific goals of the presentation ("Did the presenter clearly identify important features of their excerpt?" "Yes" - "Sometimes / kind of" - "No"), an open paragraph box for detailed feedback, and a final multiple choice question for the assessed grade. You can also select which questions are compulsory
You can copy and paste the detailed feedback straight from that column in the spreadsheet into another document to be provided to the student--it's quick and painless
You can embed the form into your class website for easy access; if you don't have one, a URL shortener will be your friend

I was tempted, in composing this blog post, to title it 'Many Hands'--but as we all know, many hands can as easily make a big damn mess as they can make light work. And having students offer peer feedback doesn't abnegate your responsibility as an instructor to offer feedback--the document I give students after presentations includes an average two pages of peer feedback plus a paragraph or two of my own comments, of the length I would give even without peer feedback. (This semester I'm also including graphs of the responses to the multiple-choice questions.) This process in no way saves me time--but by automating the process in this way, it adds maybe a minute to the time I spend writing feedback, while ensuring my students get far more information about their performance. That's a trade-off I can get behind.

Tell me where it hurts: Designing mid-semester course evaluations

Beck Wise — Mon, 15 Sep 2014 14:38:32 +0000

Author:

Beck Wise

Image Credit:

Christiaan Triebert, 'Teddy Bear Hospital'

For as long as I’ve been teaching, I’ve made a point of checking in anonymously with my students at mid-semester, asking them what’s working and what’s not to ensure we can all get the most out of the time available to us.

This is something that a lot of instructors do, of course, but the implementations vary, from formats which mimic university-mandated end-of-semester evaluations to formal or informal conversations with students. I administer mine as a one-page paper questionnaire in my otherwise-paperless class, leaving students to write anonymously while I wait outside at the beginning of the classroom.

I’ve tweaked the questions over time, trying to come up with phrasing that is clear and concise, but my questions have generally set up pairs of binaries: What do you like? What do you not like? What do you want to do more of? What do you want to do less of? What should we stop doing? What should we start doing? The survey always ends with a few yes/no questions – which, despite requiring zero effort, strangely tend to go unanswered more than the open-ended ones: Are you getting the support you need to meet your goals in this class? Are our discussions helpful? Do we address a range of viewpoints?

But the single best question I have asks students for a gestalt rating of the class. Our university-wide surveys ask for this too: ‘Overall, the instructor/class was: very unsatisfactory; unsatisfactory; satisfactory; very good; excellent’ – and looking through the department and university averages, results for this tend to hover between ‘very good’ and ‘excellent’. Good news for the department, of course, but I can’t help wondering what ‘very good’ and ‘excellent’ even mean in this scenario. I’m also not sure that ‘satisfactory’ reads as a genuinely neutral middle option – and if averages are that high across the university, I can’t help but wonder whether responses reflect students’ real assessments of their instructors.

OK, maybe I’m an over-thinker, but I decided to go with a more intuitive ‘overall’ rating: a children’s pain scale. For those who are unfamiliar with them, children’s pain scales ask kids to compare their pain to a series of six expressive faces, ranging from a big grin (‘0 / no hurt’) to a sobbing grimace (’10 / hurts worst’). They’re designed to get answers out of kids who can’t conceptualise their feelings on a numerical scale or who don’t have the language skills to accurately describe how they feel – and they’ve proven effective in my course surveys.

Part of this, I think, is that they serve as an icebreaker of sorts – the surveys don’t look institutional with the row of faces across the top (although, of course, those are taken from another kind of institution). It seems, though, that they give students a more accessible (visual) vocabulary for assessing their sense of the class. And it offers up results that differ markedly from my end-of-semester results on that numerical scale: my mid-semester check-ins reveal that my class ‘hurts a little more’ (just on the good side of the middle of the scale), but my end-of-semester results still hover in the ‘very good’ to ‘excellent’ range, leaving me totally convinced that survey design does skew results.

(It’s worth noting that I do make adjustments based on the mid-semester questionnaires that could result in improved ratings – but the changes are minimal and my most-frequent response is ‘More people complained about people not talking than actually talk, so that one is on you guys’, which I rather doubt contributes to better feelings about me and the course in the latter half of the semester.)

Not convinced? It’s also really fun to look through surveys that come back with students’ chosen faces adorned with a dazzling array of top hats, monocles, eye patches and mustaches.

Hacking (Our) Community

Beck Wise — Mon, 24 Feb 2014 06:00:00 +0000

Author:

Beck Wise

Image Credit:

Beck Wise

This semester, I'm teaching 'Rhetoric of Hacking', an intermediate writing/composition class. The course title is something of a vexed topic; it was chosen to comply with the usual pattern of writing course names at UT, but it started off as 'Hacking Rhetoric', a name designed to imply that we would not just be discussing rhetoric about hacking, but also hacking rhetoric itself, transforming our own work and that of other people. I ask students to engage with the usual range of public discourse and create some of their own but, on the basis that you can't really learn without doing, I also ask them to engage in various hacking practices over the course of the semester. Not only does this give them practical insight into the idea that hacking is itself a rhetorical practice, it offers students a way to understand and experience hacking from a position of some power -- instead of viewing themselves as passive (potential) victims of hacking, they can live out some of the risks and rewards that drive hackers.

We start small and safe: use a technique from Lifehacker for a week; play around with Popcorn Maker, a tool for remixing web videos, and Hackasaurus, a tool that allows you to mirror and hack websites; complete a few lessons on Hack This Site; be a Wikipedia editor for the day (I think the most compelling takeaway lesson from that was 'Don't sign up with your real, full name if your plan is just to deface pages').

And then we go big: hack your peers.

Every student in the class has an account on the course blog, which they've been using all semester to post weekly reflections or compositions, as well as artist statements for their more out-there compositions--and when mid-October rolled around, each student was randomly assigned to hack one other student's account, then compose some kind of rhetorical intervention and replace an existing blog post with their new text. Everyone followed up with a Hacker Artist Statement, discussing their hacking process and the rhetorical effect attempted by their hack, and then a shorter Reflection, discussing the experience of being hacked; both pieces were to be posted publicly on the course blog.

When I dreamt up this assignment, it simultaneously thrilled and terrified me. On the one hand, it guaranteed everyone in the class would be able to think about hacking from a perspective of 'real' 'victimhood'; it also meant that everyone would get to undertake the kind of anonymous, public hacking that exists in the real world, where most of our previous exercises had been personal and private.

On the other ... it was tough to think of a better or faster way to destroy a classroom community and create student paranoia than by setting them all against each other, tasked with anonymously interfering with the work they've spent the semester producing.

The results were more than a little surprising. With my own paranoia about creating paranoia in full swing, I'd scheduled this month-long project quite late in the semester, hoping to establish a strong sense of trust amongst my students before beginning the blog hack. In line with my commitment to student-centred learning, I also offered the students control over the assignment parameters, while sketching out broad strokes ('everyone will hack someone, everyone will write an artist statement and a reflection, nobody will set out to hurt anyone, this is the timeline') and establishing the expectation that this assignment should not be comfortable, but neither should it be terrifying; rather, we should all feel productively uncomfortable. Everything beyond that was up for grabs and the students spent a class session establishing password guidelines, rules for preserving content, adding a deadline to offer their anonymous hackers hints. The results of that discussion -- the final assignment prompt -- are publicly available on the course website.

That session marked a turning point in the classroom environment -- and, surprisingly for me, the start of a very positive shift in the class dynamic. Students leapt to debate the pros and cons of various ideas, including students I hadn't heard a peep out of all semester. They left the class chattering keenly. When I walked into the room for the next class -- and, indeed, when I have walked in for every class since -- people have been talking, where once they were sitting quietly, absorbed in their phones or laptops. After requesting that we eliminate required comments from the course blog because they felt 'forced' and unnecessary, students started having conversations in the comments section. And classroom discussion has been amped up considerably.

Some of this, I'm sure, comes from trying to tease out the information they need to complete their hacks; one of my students wrote precisely that in his artist statement, saying:

As we were leaving class, I struck up a conversation with him regarding the absurdity of some of the password requirements we were assigned. Specifically, I noted that “there are at most 5 unique high school mascots in the entire state of Texas, those would be crazy easy to guess”, an observation with a bit of truth (it made football games in my highschool awkward when it was the bulldogs vs. the bulldogs for the 5th time in a season). As I had anticipated, Sean responded with (to paraphrase): “Yeah, there are like 10 schools with the same mascot as mine, the Wildcats”. Paydirt!

But most of the conversation I witnessed didn't turn on those very specific pieces of information that would allow the hacker access to their target's blog. Instead, students had conversations about campus events, political developments, favourite foods, their looming assignments ... community wasn't destroyed. It was created.

As my students are posting their artist statements and starting to finish their own reflections, I find myself reading and re-reading them, and thinking about my class design for next semester. Do I trust this assignment to build community again, rather than undermine it? To what extent was community there to start with? (I'm honestly aghast at the number of artist statements that lead with 'I had no idea who ... was', in a class of 18 that meets twice a week, in which I go out of my way to address people by name.) How can I administer this project differently? Can I make it MORE student-centred? MORE experiential? How can I extend that (very) productive discomfort to the semester as a whole? The only thing I know right now is -- hell yes, I am keeping this terrifying and terrific assignment.