Wrapping Things Up

A UK city street lined with stores. On the sidewalk running down the left side of the image stands a bike stand with a bike wrapped in blue paper with the words The Electric repeated all over it. A similar bike is on the right side of the street as well.

Wrapped by morebyless on Flickr, used under a CC-BY 2.0 license

After writing yesterday’s post, I read up on group assessment, reviewed some materials on contract grading, and revisited some information on ungrading. I’ve been collecting references in Zotero for years now, so it was relatively easy to pull up the relevant information.

All this reading and rereading resulted in resetting the effort expectations table. I’m essentially using the same chart as in the past, though the assignments and the feedback system are changed slightly. Here’s an excerpt from the chart as it is now:

Grade Level Effort Expectations
A Earn a Complete on the following:

  • Five Major Project Submissions
  • 95% of the Project Wrappers
  • 95% of the Check-In Surveys
  • 95% of the Weekly Activities (including Try-Its & Self-Checks)
B Earn a Complete on the following:

  • Four Major Project Submissions
  • 85% of the Project Wrappers
  • 85% of the Check-In Surveys
  • 85% of the Weekly Activities (including Try-Its & Self-Checks)

The numbers now follow the grade level numbers: A=95%, B=85%, C=75%, D=65%, and F=64% or less. The number of major projects is also restored to the previous version: A=5, B=4, C=3, D=2, F=less than 2.

Dealing with Ghosting Students

The one nagging problem with the setup has been the ability of student to ghost their group and still get credit. That scenario could potentially give a student a C or better even though they have done little to no work. That bugs me. I want to trust students and believe that no one would do this. As much as I want to, I just can’t do it. This phenomenon is frequently called “the free-rider effect” in the articles I have been reading (e.g., Chang & Kang, 2016; Cherney et al., 2018).

It’s occurred to me in the last hour that the solution isn’t in changing the effort expectations chart. Rather it’s to change the way I manage the groups. A number of the articles and chapters that I reviewed focused on group sizes from three to five students. Some folks say more; some say less. I’ve done some work on how to form groups online (in a project that I was never able to maintain, which is a long story for another day).

I’m thinking that I should aim for four to five students in a group. With that number, even if a student does disappear, there should still be enough folks leftover to complete the projects. Groups of three would be problematic in that scenario. I’m also thinking of using the method of forming groups that I tried in the past: groups are composed of students who like to work in similar ways and (as possible) at similar times. For instance, students who like to work ahead of time and be finished in advance are grouped together. Students who like to wait until the last minute are in groups together, and so forth.

I’m grading for completion, not quality, so it doesn’t worry me that (in my experience anyway) the students who like to work ahead usually do better on their work. The other groups aren’t at any disadvantage. They just have to complete their work. They don’t have to knock my socks off.

What I’m adding is a practice that I found mentioned online and among some of my colleagues on Facebook. To put it bluntly, a free-riding student who disappears from their group or does not contribute can be kicked out of the group by the other group members. That student can then either work on the group projects independently or join a group of other students who have been removed from their groups. If I adopt this system, no student should pass the course while doing none of the work.

Assessing the Project Management Work

With all that sorted out, I just need to figure out how to focus assessment on the process of managing the project, not so solidly on the project itself. I’m worried about the workload, so I don’t want to add onerous reflections.

I’m also not a fan of asking students to use on of the strategies where they grade each other by assigning points or percentages based on the work each person does. See the “Student Assessment of Group Product” section of the Grading Methods for Group Work page from Carnegie Mellon’s Eberly Center for examples. Such systems work, but they can put students at odds and can be unfair to students who do not fit the demographics of other group members (whether because of race, class, gender, LGBTQ status, and so forth.

I think the answer is to add project wrappers to the major projects. Somehow as I was reading yesterday, I happened upon exam wrappers. I’ve heard the term many times before, but had always thought they were a way of adding information when you turned in an exam. I assumed that they were similar to the draft letters and memos that I have used frequently, where students tell me about their project and what they want me to comment on as I add feedback.

In other words, I missed their full potential. Exam wrappers are also used after an exam, when students receive the feedback on their work. They typically ask students to look at the errors, figure out the correct answers, and reflect on what went awry. That process is similar to the Revision Plan assignment that I have used in the past. In that assignment, I ask students to return to a project from earlier in the term and write a description or instructions for revising the project. Depending upon the course, I might also ask for storyboards, outlines, or other kinds of mockups. I liked that assignment as a final exam because it allowed students to explain what they’d do if they had more time without asking them to do all that work while they were swamped with other finals.

Somehow, despite knowing about exam wrappers, I didn’t see the connection of using them both before AND after the exam. You’d think that the work wrapper would have tipped me off—it is an active learning strategy that wraps around another assessment. Yes, I’m feeling a bit daft right now.

I realized that adding something when a major project is submitted and then after the feedback is returned allowed me to structure reflection in a meaningful way that avoided asking students to write long reflections and to come up with deep thoughts about their work. I can ask some basic survey questions about the time spent on the project, which portions they focused on, how group members coordinated, and so on in the wrapper they complete when turning in their work. After I return the work with feedback, I will have similar survey style questions but also some short answer questions that will ask them to reflect on what they did and how them might change in the future to improve their writing and collaboration.

Asking multiple-choice and short-answer questions, I can limit the amount of effort that students have to expend. Coordinating the pre- and post-wrappers should trigger reflections on cause and effect (or perhaps impact) of the strategies they have used. Asking for a revision or revision plan focuses the work on the project (the product). Asking students to think about why their work habits resulted in the performance and how they might change it in the future focuses the task on the process. It feels like the perfect solution for paying attention to the project management aspect of the work that they will do in the course.

FINALLY, I think I have things figured out. None of it is written up or published in Canvas, mind you, but I think I know what I’m doing. I see a lot of long work days between now and Monday, when classes start. On the chance that you aren’t aware of the practice of exam wrappers, I’m including three links to basic information on the strategy:

Temporary Signs of Progress

A cream colored billboard interrupts a cloudy blue sky. The billboard reads, Temporary Sign.

Another beautifully random temporary sign from TFL by WithAssociates on Flickr, used under a CC-BY-SA license

On Friday, I ordered Ellen Carillo’s The Hidden Inequities in Labor-Based Contract Grading (2021). It arrived today, and I read it quickly. It’s a short but very useful book. I was especially pleased to have some of the questions that I have about contract grading as Inoue (2023) defines it validated in Carillo’s argument.

I’ve been worried that I’m asking for too much work to earn the various grade levels. Admittedly, I’m at a point I typically reach when I’m working on assessment and assignments: Everything feels completely random. Why 95% for a B? Why all five projects for a C or higher? Why any of this? The only thing I can say is that I know why there are five projects–because that is set (more or less) in the department’s new course template. Everything else is just a well-intentioned guess.

I mentioned last week that from my perspective, some of the measures of labor in Inoue’s system have the potential for ableism. Carillo amplifies my concern, bringing in discussion of the inequality of using the number of hours of labor to measure assessment. I’ve been worried about students who have less time to commit to coursework because of other obligations (such as work, parenting responsibilities, and requirements of scholarships). Carillo addresses those concerns and further argues that different people need varying amounts of time to complete a task. What takes me one hour may take someone else two hours. She points out how this system is particularly troubling for students with disabilities, who may face “access fatigue” (Konrad 2021) and, therefore, may need more time to complete their tasks. See the concept of “crip time” (Price 2011) for more information.

Where does all that leave me? The randomness of assessment systems and the inequality of focusing on the amount of work students are able to complete land me back where I began. I’m no longer feeling all that confident about the grade expectations I posted yesterday. Asking students to do everything for a B feels wrong now. Asking them to do even more work for an A feels even worse.

Now I think these are the plans and questions for the Fall:

  • I’m rearranging the effort chart from yesterday to set most of the numbers back to the previous version, from the Spring 2023 course. 95% of the work is required for an A. 85% is required for a B, and so forth.
  • I’m still unsure on the number of major projects that are required for each level. I am (currently anyway) keeping the changes to the assessment of a piece of work to reduce some of the specifications grading I was using (Nilson 2015). Previously the system was all five required for an A, four for a B, and so on. What confuses me are the group projects.
    • Are all projects the same (individual and group)? So the scale from the Spring works as is?
    • Should there be some analysis of how much work each group member puts in to determine when a group project “counts”?
    • Is there a way to count the group projects AND require some additional project management assessment that counts separately?
  • Should I add some projects (like those previously required to move from a B to an A) for students to use to increase their grades as needed? Honestly that feels like extra credit, and extra credit is a move back to grading practices. Though if I’m honest, ALL of this is about counting things. Maybe they aren’t grade points, but they’re still numbers being counted and compared. I guess I’m leaning toward no on this question.
  • I feel as if I’m not assessing project management at all. As it’s set up, I’m assessing the product created by a group, but that’s not the same thing as project management. Even an unhinged and dysfunctional group can turn in a project that meets criteria. I hate to add more work to the course, but there needs to be some reflection of the group and report on its accomplishments.

It would really be nice to settle on all these aspects of the course. I think I’m off to research the assessment of groups and assessment of project management. If I can figure out those issues, maybe the rest of things will fall into place. A girl can hope anyway.

A Tentative Effort Expectations Contract Chart

A white sign with black text labeling the path as Effort Street (SW17)

Effort by secretlondon123 on Flickr, used under a CC-BY-SA 2.0 license

I’ve spent the day reading (and rereading) on contract grading, including all of these resources:

  • Inoue, Asao B. (2019). Labor-Based Grading Contracts: Building Equity and Inclusion in the Compassionate Writing Classroom. The WAC Clearinghouse and University Press of Colorado. https://wac.colostate.edu/books/perspectives/labor/
  • Stommel, Jesse. (2017, October 26). Why I Don’t Grade. Jesse Stommel. http://www.jessestommel.com/why-i-dont-grade/
  • Stommel, Jesse. (2018, March 11). How to Ungrade. Jesse Stommel. http://www.jessestommel.com/how-to-ungrade/
  • Craig, Sherri. (2021). Your Contract Grading Ain’t It. WPA: Writing Program Administration, 44(3), 145–146.
  • Blum, Susan D. (Ed.). (2020). Ungrading: Why rating students undermines learning (and what to do instead) (First edition). West Virginia University Press.

I stand by the short list of goals that I shared in my last post. If anything, I feel stronger about them. I am set with focusing on Completion meaning that “you do all that is asked of you in the manner and spirit it is asked” (Inoue, 2023, p. 327). Less of a specifications grading system (Nilson, 2015), the system checks to ensure that the basic things that are asked for are there. I’m leaning on the explanations from Inoue’s work here. There’s much more explanation in the book (which is available free from the WAC Clearinghouse), but I particularly liked the three questions he included for assessing the work students did during a week in FYC:

Each labor assignment is complete and counts if it meets in the affirmative the following questions:

  • Is the labor product(s) posted on time and in the correct place?
  • Does the labor product(s) include everything I asked for and meet the minimum word count?
  • Is there a labor tweet/Slack(s) posted as instructed (if applicable)?

(Inoue, 2023, p. 196)

My class will have different framing questions, but the idea will be the same:

  • Did the student turn in the work before the end of the grace period?
  • Does the work include everything listed in the criteria for the activity?
    • Is it the right assignment? (e.g., Is it a memo?)
    • Is it on the right topic? (e.g., Is it a good news memo about employee holidays?)
    • Is it a full draft? (e.g., Does it include memo headings and a memo body?)

I may be addressing some specific issues that arose in the past with my questions 😉 You know, like that one time a student responded to the assignment for a pitch presentation on their proposed report topic with a presentation that argued Lionel Messi is the GOAT. Um, no. That is not a Complete. Nor were the Recommendation Reports that were only outlines. Yes, that was a submission, but not a Complete.

I feel reasonably comfortable that this kind of framing questions will address to check student work to ensure that students are on track and doing what they are supposed to. Further, I see what I’ve come up with as more rigorous than what I was doing in the past. Students have to do all of the work with attention to its expectations to do well in the course. It wasn’t really a goal to increase the rigor, but I’m happy that it has turned out that way.

I’m also going with the essential idea of Inoue’s grade levels, though I am phrasing my system in terms of work completed rather than work that is missing to avoid the deficit focus. Like Inoue’s system, my tentative plan allows students to earn a B by doing everything in a way that meets basic criteria. To earn more than a B, I will follow Inoue again and provide a list of additional tasks that students can complete. Each task earns 1/3 of a grade. For instance, if a student with a B does one task on the list, their grade will increase to a B+. This extra work for grades higher than a B also increases the rigor in the course.

The table below shows the tentative contract for effort in the course. Naturally I will need to add a lot more information. In particular I need to include an explanation of what it means to earn a Complete as well as sketch out all of the tasks that are available for grades above a B. I want to adopt Inoue’s practice of allowing students at any level to improve grades by completing these extra tasks as well (2023, p. 330–331). So a student with a C+ could complete a couple of extra tasks and move up to a B. That gives students more options so that one bad week doesn’t ruin their chances.

Here’s that setup so far:

Grade Level Effort Expectations
A Meets all of the expectations for a B, and earns a Complete on up to three of the following (each raises your grade 1/3 step):

  • Earns a Complete on 100% of the Check-In Surveys and Weekly Activities
  • Substantially revises two Try-Its (?)
  • Writes a Midterm Assessment of your work in the course.
  • Substantially revise the 1st or 2nd major project
  • peer reviews?
  • what else?
B Earn a Complete on the following:

  • Five Major Project Submissions
  • 95% of the Check-In Surveys
  • 95% of the Weekly Activities (including Try-Its & Self-Checks)
C Earn a Complete on the following:

  • Five Major Project Submissions
  • 85% of the Check-In Surveys
  • 85% of the Weekly Activities (including Try-Its & Self-Checks)
D Earn a Complete on the following:

  • Four Major Project Submissions
  • 75% of the Check-In Surveys
  • 75% of the Weekly Activities (including Try-Its & Self-Checks)
F Earn a Complete on the following:

  • Fewer than four of the Major Project Submissions
  • 64% or less of the Check-In Surveys
  • 64% or less of the Weekly Activities (including Try-Its & Self-Checks)

Weighing Student Work

Gold colored scales of justice sitting on a wooden table

Scales of Justice by Michelle Grewe on Flickr, used under public domain

I’m definitely doing too much thinking about contracts and how to set up the expectations fairly. So much so that I’ve realized that I’m stuck on the very essence of grading. I’m no where near ungrading. I’ve lost all track of where I should be.

Fooling Myself

To start, what are my beliefs about assessment? I would be easier to point to the people whose pedagogy I try to work within: Jesse Stommel, Alfie Kohn, and Susan Blum.

Grades don’t work. They cause students to focus on the wrong things. In my contract, students may still be focused on the wrong thing. They are counting (and obsessing) over writing tasks. I’m fooling myself to think that the counting of Completes is immune from the curse of focusing on the elusive grade.

My great confusion on how to count (or not) the group work should have tipped me off sooner. Trying to figure out what would count as effort in a group is the wrong thing. I’m trying to invent a system where I decide how the quality and quantity of work determines the ultimate grade.

I’m still acting as the arbiter of course knowledge and assessment since I am the judge of Complete or Incomplete. I decide whether the effort that students put in counts. If I were truly faithful to the ideals of ungrading, I would be powerless. Students would determine what and when they have learned.

So what if I scrapped it all? What if I counted only submissions? Did the student turn the task in? Yes? Then it counts toward the expectations.

A Model Contract

So what if I go back to Inoue’s Labor-Based Grading Contracts system? I don’t like it for reasons I’ll get into, but it is a starting place. Here are his contract expectations (p. 127):

Grade Level # of Non-Participation Days # of Late Assignments # of Missed Assignments # of Ignored Assignments
A (4.0) 3 3 1 0
B (3.1) 3 3 1 0
C (2.1) 4 4 2 0
D (1.1) 5 5 3 1
E (0.0) 6 6 4 2

First, let me get my problems with these guidelines out of the way. I don’t like counting attendance, which is what the # of Non-Participation Days essentially does. I believe it’s not only ableist, but also inhumane to students stretched thin with other obligations. The system does have exceptions for excused absences, which could reduce some of the ableism. Still in a time when COVID still exists, I do not want to encourage any system that would ask a potentially sick student to come to class to avoid slipping down the expectations chart. Since my courses are all asynchronous online classes, it’s also irrelevant. There is not such thing as an attendance day to count at all.

I’m not a big fan of punishing students for late assignments either, so that column of the chart is problematic for me too. I believe in a more humane system that trusts students to put in their best effort. If they need an extra day, they should have it. I do recognize that Inoue’s course is highly interactive, and if students do not bring their work to class, they miss getting feedback from others. Peer review is a tough problem to negotiate in a due date system. I may not like this portion of the chart, but I understand why it’s there and critical to the way his classes work.

What counts in this contract model bugs me the most though. This is a deficit model. It counts failures and devalues the idea of failing as a critical part of learning. I’ve read enough of Failing Sideways to know that it’s more valuable (to me anyway) for students to be allowed to fail, to try again, to take risks without being punished.

Toward My Own Contract Model

I need to articulate my own model, and this time, I want something that is more in line with the spirit of ungrading. I haven’t worked out the details yet. Honestly, I haven’t even begun the work of determining what I expect students to do to demonstrate their learning. I do know that I have these goals in mind:

  • I want a positive model that counts things students do rather than things that they don’t. Such a thing should be more Canvas-friendly anyway. As far as I know, Canvas cannot use a system that counts failures.
  • I’m going to have to work in more self-reflection and group reflection. The challenge will be to add that reflection work without over-burdening students. They already have a lot of writing to do in this course.
  • I need that reflection to be authentic, which can be a challenge. In my experience, students are so over reflection. They have been writing reflections on their work their entire academic career. As a result, reflections can look more like an exercise in busywork instead of showing any deep recognition of who and when their writing has changed.
  • I still have to find a way to check student work to ensure that students are on track. I know that I should just trust students to do the work, and not feel compelled to check that they are not turning in blank files (yes, that has happened in the past). I’m just not there though, and I’m relatively sure that anything else would be frowned upon.
  • I‘m going to need a good explanation of the difference between feedback and completing/submitting work. A document can count as a submission and still have extensive feedback on how to improve it.
  • I’m going to have to figure out how to manage the extra work that moves students from a B to an A. I left that part out of Inoue’s system above. He offers several possibilities, including revising projects and creating more in-depth work (p. 132&endash;133). I will surely include revisions, but I need to think that system through.
  • I need to invent a TARDIS because I need to squeeze a lot more time out of this next week. Classes start Monday the 21st.

How I Almost Undermined My Ungrading System

It seemed like a simple explanation at the time. By the time I was almost done, I realized that I had almost undermined my entire ungrading system.

This fall, I have a few hard deadlines. Most work has a grace period that does not end until the last full week of classes. For example, peer review activities have a firm deadline for technical and logistical reasons. I explain the system in my Short Guide to the course:

Firm Due Dates explanation, transcript at the link

I further explain the two rounds of peer review in a course FAQ:

How do the two rounds of peer feedback work? Do I have to do both of them?

The idea of two rounds is more to help people who are working more slowly. Nearly all students I’ve had either choose the first round or the second-chance round. I don’t remember anyone trying both, but you can. You do provide feedback to 2 other people whenever you participate (but you get feedback from 2 people too). The activity is essentially the same. It’s just the time frame that changes.

Despite the fact that information is out there, some students remained confused. I wasn’t worried. Students cannot possibly remember everything I tell them or recall where are the details are in the course documents. What did worry me was that their email messages communicated a bit of panic through comments like this one, “I missed the feedback post. Can I still achieve the grade that I want?”

My first impulse was to find a way to clarify in a way that would calm the panic (and end the extra emails I was receiving). I decided that perhaps students needed a better understanding of the expectations for the work. For the peer review work, they needed something to clarify that I only expected them to complete half of the posted activities (either the first or second opportunity).

I immediately thought, “Infographic!” and I worked up the image below. Pie charts seemed like the perfect way to express the expectations. I started with the “Complete Half” section of the graphic and then added the other sections to represent all the work in the course.

I was quite pleased with the emerging look of the graphic: very polished, and very Virginia Tech (If I do say so myself). It was that “Complete Most” section that started my teacher-senses worrying.

Activity Expectations, with transcript at link

I couldn’t decide how much of the pie chart properly represented “most” of the work in that section. Obviously more than half. And the image needed to be visually distinguishable from the Half Pie. Alternately, shading 3/4 of the pie seemed wrong too. I didn’t want a pie chart that suggested a firm number. I wanted a visual representation that was less specific. I settled on approximately 60%, as shown in the image.

I felt that the infographic was about ready to go and turned my attention to writing up the transcript. As I wrote it up, I realized that it was likely to raise more questions:

  • Exactly how many is most?
  • How many activities are there? I need to figure out if I have most of them done.
  • What if I completely only 51% of the activities? Is that most?
  • If I have 60% of the work done, can I skip the rest?
  • What happens if I don’t complete all of the activities in the “Try to Complete All” section?

As these questions bubbled up, I realized that I had almost undermined the entire system. The moment I started to put numbers on the work, I turned on the traditional grading system. Students were going to shift immediately from a focus on improvement and their best effort to an obsession with reaching the right number. And so this infographic will never be used in my courses. Instead it prompts me that I have to be careful to return students to details on the ungrading system when questions come up. I have to remind them that they are in charge of their grades. I have to turn the question to them. When a student asks, “Can I still achieve the grade that I want?” I need to remember to reply, “Are you putting in the effort that grade would require? Does your work represent your best effort?”

Transparent Feedback Strategies

A Guide to Feedback on Your Writing, with transcript at linkMy assessment this term focuses on feedback, as a part of the #ungrading approach I have adopted. Since I do not put A’s and B’s on student work, feedback is critical.

Understandably, some students are a bit nervous about this approach. In particular, they worry that they will not be able to tell what the feedback tells them about their work. Their uneasiness is a strong indication of how they have used grades to determine their accomplishment in the past. Combine their apprehension about #ungrading with the fact that every teacher provides feedback differently, and it’s little wonder they aren’t quite sure how all this feedback is going to work.

As I began adding feedback to the work they have submitted, I realized that I could tell students more about the logistics of my feedback system and that it was in my benefit to do so for several reasons:

  • I do not comment on every piece of writing students do, but they have no way of knowing whether the lack of a comment means I haven’t seen their text or I saw it and chose to say nothing.
  • I use boilerplate for those frequently-used comments, but students do not know why I might repeat exactly the same feedback.
  • I automate some feedback in Canvas on activities that count like check-ins. If students have not done those activities, Canvas can assign them an Incomplete (or zero). They need to understand how to use that feedback as a reminder to get the activity done.

To explain all these practices to students, I created the simple infographic on the right, with an online transcript of the text. I aimed at short and clear explanations, without expansive details on pedagogical or practical reasons for the different strategies.

I plan to share it on Tuesday, after our Labor Day holiday. I’m hoping to hear that the details help make the logistics more transparent.