How I changed teaching after AI managed to do all my homework assignments
Around 2021, well before ChatGPT launched, Vincent Hellendoorn suggested I try GPT-3 on the reading quizzes in my course. It produced convincing answers passing our rubric without actually seeing the assigned paper. At the time, I changed nothing. Five years later, AI agents could do all my assignments and I have redesigned most assessments in that course, even though what I want students to learn has barely changed. The strategy is always the same: No longer test understanding with anything that is done at home and instead focus on interactions with a TA, on exam, and on a video demo. Some of these changes violate evidence-based best pedagogy practices, and I made them anyway.
For the last couple of years, I have mostly taught the course Machine Learning in Production, an upper-level course on building production-ready software around ML models with a heavy focus on MLOps, usually with 100 to 170 students. These days one common question when talking to other educators is how we have changed teaching in the age of generative AI and coding agents, so let me outline what we did.
We have shifted covered topics with changing AI innovations and tools, but I barely touched the overall learning goals. I am fortunate that this is not an intro course and that the learning goals are not about writing code or using specific tools; they are about engineering tradeoffs, anticipating and mitigating risks, and teamwork. I think these are skills still worth acquiring, even if some can be simulated and offloaded to a model. (Revising an intro course or a traditional software engineering course likely would shift learning goals much more.)
Also possibly important: We give students permission to use AI in all settings, in any form, without attribution, except for written and oral exams. We even encourage the use of AI tools in many places. I do not think policing AI is feasible even if we wanted, and more importantly I do think that students need to learn responsible use of these technologies anyway.
AI is forcing me to abandon evidence-based best practices
Let’s start with this point upfront, since it is more important than what we actually do: Unfortunately, AI is actively undermining several evidence-based teaching practices (e.g., see How Learning Works and The ABCs of How We Learn). For example, the evidence favors frequent low-stakes assessments with feedback (e.g., homework, quizzes) over few high-stakes ones (e.g., exams) – but AI is undermining practice in low-stakes settings and pushing us more toward exams.
Similarly, I always provided a safety net where students can make mistakes and resubmit a limited number of assignments to regain lost points (a core recommendation of specifications grading and grading for equity to focus on learning outcomes, not the process), but we felt that this process was abused with AI: first submit a generated assignment solution without thinking and only look at the issues raised in grading for a resubmission (the typical story of externalizing the cost of AI use). In response, we have since taxed resubmissions with a 10% penalty.
Also in-class interactions allow engaging with materials in an early low-stakes setting, but with AI I have seen many student groups offload the discussion questions to a model. Pen-and-paper submissions could fix this, but aside from a higher grading workload, it would also raise stress for students, take away from the low-stakes environment, and delay feedback.
In general, this is a balancing act and I tend to err on the side of keeping low-stakes repeated interactions even though it can be abused. Yes, some students will get through the class without much deep learning, but it provides a better environment for those students who want to learn. I don’t want to get back to the model I’ve experienced during my own studies in Germany with mostly optional homework and a single exam at the end of the semester that was responsible for 100% of the grade in the course. This was nice for students who were self-motivated and good at learning for an exam (like me, I guess), but had failure and drop-out rates of 50 to 80%.
Written reflections → 15-minute conversations
Now for actual changes in the course: I have given up all parts of assignments that required a written text answer. I still ask for reports that describe a solution and link to the relevant code fragments, but that’s just for navigating their solution and I’m fine with receiving AI generated documents for that. In contrast, reflection documents, like “What were challenging parts?”, “How would you improve teamwork?” or in reading quizzes “For scenario X, identify one plausible data quality problem you might expect that relates to one of the four data cascades discussed in the paper…” have become pointless and can be entirely delegated. Short of hiding the evaluation rubric, I can see no way of stating what I expect in a good answer that cannot be completely offloaded to an LLM.
For written reflections, which I used to have as a part of pretty much every assignment, I now shifted to in-person interactions with a TA. After every assignment, each student needs to schedule a 15-minute meeting with a TA to answer a couple of questions in a live conversation ( apparently Stanford cs221 is evaluating the same kind of approach in a controlled experiment this semester). I still share the reflection prompts in the assignment as examples of the kind of questions we ask. Students can still generate an initial answer with an LLM, but they may need to memorize parts of it, and we try to challenge them with follow up questions. The check-in meetings are part of the assignment and currently worth 20% of the assignment points, graded pass/fail. Students can try again if they fail and I encourage my TAs to have fairly high standards – we usually fail quite a few students on their first attempt.
There are drawbacks to this design, but overall I am happy with the tradeoffs: Penalty-free retries reduce fairness concerns about TA grading of oral interactions; a more strict TA costs a student time, but not points. Oral check-ins demand more from students with anxiety, but so do written exams, and formal disability accommodations can provide a path in both cases. In fact, professional communication about technical work is a learning goal and oral check-ins train this more than written reflections. Regarding scale: We run the course at a 20:1 student-TA ratio with about 10h of work per TAs per week (fortunate, I know), so the check-ins amount to roughly 300 minutes per TA every two weeks, which is workable.
For reading quizzes, I just gave up. I did not think doing in-person pen-and-paper quizzes in class would be worth the stress and the needless memorization work that those would be causing. I actually kept online reading quizzes around for a long time just to signal that I wanted students to look at the paper, fully understanding that most would just ask an LLM. These days, I still assign readings, but only half as many and without any points attached. Instead, I try to integrate lessons from the readings into in-class discussions. Still most students do not do the readings and just ask an LLM when we get to that point in the class (so nothing changed on that front), but those that do might get more out of it.
Minor note: We observed that some students used AI during live discussions over Zoom (e.g. Cluely) and we will likely only offer in-person checkins in the future in response.
For coding tasks: code + videos + in-person knowledge checks
We have weekly labs that are low-stakes small tasks to explore new tools (e.g., Kafka, Grafana, Docker, Weights and Biases). These tasks are necessarily scoped small and need to provide some help to students starting out – so they are obviously easily automateable by coding agents. We again rely on in-person check-ins – show the TA evidence that you completed the task and be able to answer a few questions, graded pass/fail. Again, we frequently send students back to read more documentation (or let their chatbot summarize the relevant part) and let them try again without a penalty until the time of the lab session runs out.
For assignments, we use the same check-in with a TA discussed for reflections to let them explain part of their technical solution. In teamwork, we have longer debriefing sessions after each milestone (30-60 min per team). We award “beyond-the-comfort-zone” bonus points to a team if the TA can ask any team member to explain any part of the implementation. (Yes, I know, bonus points are a scam. I use them anyway. Sue me.)
Also having students produce a short video demoing a feature they implemented worked really well for an assignment to extend a web application, because it required the feature to actually work with a user interface in a real workflow. I think producing videos for other parts could also work, as long as it’s not just reading an AI-produced script, but actually grounded in some technical work.
More of the grade now depends on what happens in the classroom
As I see in many other courses, we also shift more points from activities done at home (e.g., homework) to activities done in the classroom (e.g., exams, participation). Exams are now worth 25% instead of 15%, and I suspect I will raise this further in the future. The debriefing is 10 to 20% of the homework and group work grade. Still the majority of points are associated with homework and group work done at home and most students get full or nearly full credit, but the main grade differentiation now comes from exam grades.
I have not yet introduced graded in-class pen-and-paper quizzes that many other instructors now use, but it is an option. I prefer debriefing with a TA for now.
Freeing up TA time with AI grading of unsupervised work
As students can more easily produce large amounts of code and lengthy text documents, traditional manual grading has become more tedious (the typical asymmetry of lower production costs without lowering manual review costs). The turning point for me was a year ago, when a TA shared how he felt silly grading a solution where the commit message included “Authored by Claude Code.”
We have since built infrastructure to autograde code and written reports of homework with LLMs (institutionally approved LLMs). Autograding follows a relatively straightforward LLM-as-a-judge approach, where an LLM is prompted with a specific rubric item and instructions and parts of the solution (e.g., code diff, reports). To make autograding easier, I now ask for solutions across multiple markdown files in the solution repositories rather than a single PDF. The autograder makes a judgment as either “pass” or “needs review” with comments for the TAs. We usually spot-check a few “pass” grades (almost never finding issues), but TAs spend most of their time just with the “needs review” answers (many of which actually pass). The LLM-generated notes also speeds up the manual review process as it can provide meaningful context. In the end, TAs spend 50 to 80% less time grading (grading 80% less content) and spend more time interacting with students face to face – which they also prefer. At the same time, we still never deduct points without a human having reviewed that answer. Note that we grade against clear pass/fail criteria based on the concept of specifications grading, which works well for this kind of assessment, but also gives the students’ AI agents very clear instructions on how to do their homework for them.
We now use the same LLM-as-a-judge approach to provide feedback to in-class discussions. We have breakout sessions (think-pair-share style) in every lecture and ask students to post their answers to a shared Slack channel before we talk about them. At the scale of the class with 100 to 170 students, I rarely had time to give individual feedback, but now I’ve automated that too. A custom Slack bot takes their answers, runs them through an LLM-as-a-judge check against a number of criteria. We usually covered the criteria underlying the checks in the lecture content, though we do not share the specific checks with students. With a second prompt, we then turn the check results into hopefully constructive feedback that is posted as a Slack response, encouraging students to revise their answer, after which we provide another round of feedback the same way. The prompts and one example of how this looks can be found in this gist. This feedback generator is also available to students later through a web or Slack interface to try different answers as they prepare for the exams. I’m also thinking about making the feedback available already during their breakout discussions to push them to think about their answers more deeply before they post them.
Usually, both autograding and automated discussion feedback requires some calibration, often tweaking the prompts to make the model less sensitive or to look for specific problems. Usually an AI agent is helpful in creating the context and prompts for the checks from slides in the first place and to come up with common problems and corresponding checks given a couple of solutions.
Bigger assignments, as agents made the old, scoped ones trivial
Given that students spend less time manually reading code, learning libraries, and writing code, we can scale the scope of the work. Usually, we try to scale it to the point where current AI agents cannot solve a task without more hands-on guidance and feedback.
For example, our first homework assignment that was intended to screen for existing coding skills was to extend an Instagram-clone with two AI-powered features. The provided starter code albumy was a relatively small (12k LOC), clean implementation for a textbook example. While the correct solution could be implemented in approximately 20 lines of code, the challenge used to be understanding the code, finding the right libraries, solving a dependency incompatibility issue, and integrating everything. The assignment was intentionally scoped to make it feasible to extend a web application even with little prior knowledge of HTML and Flask. This became trivial: In the fall of 2025, Claude Code could solve the entire assignment, including the writeup and reflection, without any interaction, simply by pointing it to the assignment text.
We replaced that assignment with a similar task to implement two AI-powered features, but now in Zulip, a production-quality team chat application with a large code base (> 500k LOC). Current coding agents can work with this code base, but require interaction to solve the task correctly. The required features are also bigger, require more backend and front-end changes, usually with several hundred lines of code. Without coding agents, this task would likely only be feasible for experienced web developers in the allocated time. The revised assignment has a similar function of demonstrating coding (or code generation) proficiency and to shift the student’s attention from model benchmarking in traditional ML courses to building product features in my course, but at a much larger and more realistic scale.
Similarly, we raised requirements in other assignments. Where previously we asked students to perform hazard analysis manually, we now ask them to build automation for the process, run it at some scale, and then curate relevant results. We add additional requirements to the group project too.
Note that we have to assume that students are using AI coding agents for their work and that they are somewhat competent at it. How to use AI coding agents effectively is not a learning goal and not something we cover in this course (deliberate decision), though we help students on demand in office hours. This mirrors how I treated programming skills before AI: I never taught Python or HTML in this course either, but I assumed that students have basic programming skills and can pick up new languages, libraries, and tools on their own; the first assignment screened exactly for that. At this point, proficiency with AI coding tools is simply the new baseline, and other courses like 17-214 can teach it properly (and pretty much everybody is exposed to these tools or actively using them already anyway). I am aware that students may need to subscribe to an AI provider and that this can create equity issues. I do not believe that suggesting using low-cost open-weights models would be a fair alternative. Instead, I usually argue that subscription costs are actually lower than textbook costs: A textbook easily costs $100 these days (and publishers are busy destroying the second-hand market), compared to $20/month for 3 months of a semester. For a while AI companies have been generous with student credits, but that time is mostly over. The textbook analogy is not perfect, but AI costs are nearly negligible compared to the cost of attending CMU in the first place. In the end, I think it is the responsibility of the university or other institutions to provide the support infrastructure (e.g., through scholarships) to provide students with equitable access, just as they do for textbooks. We now mention the expectation to use a commercial subscription in our syllabus.
Letting students get burned by confident-wrong AI hopefully calibrates trust
In an ideal world, I would like to always design parts of an assignment that AI agents get confidently wrong, just to teach students to not over-rely on these tools. Finding that 80% of the class lost points on a simple question because they didn’t check their generated answers is a great learning experience – similar to phishing-attack training (i.e., sending fake phishing emails to see whether employees fall for them). The point is to raise awareness of automation bias and calibrate trust down to a more appropriate level.
In practice, I have not succeeded designing assignments explicitly for likely AI mistakes, but we have now found multiple such cases by accident:
-
ChatGPT was confidently wrong about where hardware fits in Jackson’s distinction of the World vs the Machine (even when providing the full paper). About 20% of students used to make that mistake before ChatGPT, then it was 80% after, and even telling students explicitly upfront about ChatGPT making this mistake reduced it only to 50%.
-
Claude Code is remarkably insistent on really bad solutions to an agent security problem (how to make sure that the agent always confirms an MCP action with a financial transaction before executing it), producing a fancy-sounding solution (“two-phase confirmation protocol”) that is completely insecure when thinking about it adversarially. Again 80% of students fell for this mistake initially – including most of my TAs who I had to ask to regrade the entire assignment. More recent models make this mistake less, so unfortunately students no longer fall for it reliably.
Debriefing after such a failure is a good learning opportunity to talk about automation bias and different forms of human oversight (and how this is really hard in practice). The course has a safety net with resubmission opportunities, where having a majority of students initially fail an assignment does not impact their grades if they learn from it.
Generally, pushing assignments to a scope and complexity that exceeds what an agent can comfortably do in a single step helps with experiencing this occasionally. I wish I could design for it explicitly.
Students seem to accept these changes, though I lack evidence on learning outcomes
Beyond what we grade (which is mostly AI generated), I cannot really measure whether students learn more or less. Grade distributions are not a meaningful signal; average grades have drifted down a bit over the years, but that started before most of these changes and might be explained by offloading work to AI or simply by changing demographics in the course. What I can observe is that students readily accept the new formats, and many students who I speak to appreciate these changes, including the flexibility, the in-person interactions, and even the pushback from TAs that forces them to actually understand their solutions. Admittedly, most students I speak to about this in more depth are prospective TAs, not quite a representative sample. Student meetings are revealing in another way: Students regularly complain about classmates offloading their thinking to AI, and within teams we regularly see both conflicts about AI use and self-policing behavior, where teams work out many of these issues on their own.
Continuous adoption seems unavoidable
The pace with which I make changes is increasing. This course was fairly stable for many semesters, with the occasional new example or updated tool; now I make larger changes semester after semester on most assessments (except for exams). Many actions that we take decay as models improve: Cluely has been used to cheat on Zoom check-ins, questions that reliably tripped up coding agents stopped working with more recent models, and the Zulip assignment may eventually become too easy too. We now need to check which assignments and teaching strategies survive the latest generation of tools each semester. Maybe I will have to give in to pen-and-paper quizzes at some point. On the positive side, AI also helps me to adapt and change assignments and build autograders, so adaptation becomes cheaper too.
Back in 2021, I could watch GPT-3 answer my reading quizzes, note that things will change but comfortably do nothing. Now in 2026, I do not get to do that anymore.
