The fall term is almost upon us, and law profs are all trying to figure out what to do differently with their classes in the Age of AI. Given that I’ve been writing a little bit about AI and its impact on legal education, readers might assume that I have some sort of experience with AI. Nope. None at all. Not until last week.
What brought me to ChatGPT (and Claude, though I haven’t really tinkered with that yet) was, ironically, the need to go device-free for my Evidence class this coming term. Having decided that my students should have at least one dedicated site for slow, attentive, un-AI-mediated learning, it didn’t make obvious sense for me to spend all that time just lecturing. There will still be lecturing, and perhaps there will be more than I anticipate (and more than there should be). I’m having trouble, though, with the idea that my students - who can now invent their own bespoke tutors and AI-generated summaries of cases - should have all or even most of their class time occupied by a talking head. (Don’t get me wrong: I am a great talking head. But even so.)
That means spending much more time on problems; fact patterns. Now, I have been teaching for a long time, but I do not have enough fact patterns for every single module in the course. Other law profs, I think, will understand why. As I have said before, the decision to cover a given topic on an exam is driven (in part) by the amount of time spent on that topic in class. (If I spend three classes on a topic, and expect students to do three classes’ worth of reading on that topic, I can’t help but think that I really ought to test students on that topic in the exam.) That logic will apply every year. So I have lots of fact patterns for some topics, and few to none for others.
I may also need different fact patterns. It’s one thing to lecture on a topic, and then give students a super-complicated fact pattern to test them on. It’s another thing to introduce students in class - the students, say, who didn’t do the reading, and won’t have had the benefit of a lecture - to a topic with a super-complicated fact pattern. Not so good. Better to warm them up with a relatively simple fact pattern, and then move on to the brown- or black-belt level stuff.
So I need lots of new fact patterns. But, People of Substack, I do not have time to write the equivalent of ten final exams by Labour Day. Certainly, it is not a good use of my time. So I have come, hat in hand, to ChatGPT to help me produce these fact patterns.
And, in all honesty, it’s very impressive. I can direct - sorry, prompt - ChatGPT to go to a link to any case reported online, and tell it to turn it into a law school fact pattern. (Yes, that is literally the prompt I used. Yes, one can construct longer, better prompts that will produce better results.) And, by golly, it will do it - and in only a few minutes. Sure, it needs refining, but there is no getting around the fact that the first cut is not a bad cut. Just knowing, moreover, that I can do that with any case, on ten minutes’ notice, seems like a kind of pedagogical miracle. I wouldn’t necessarily trust it to give me (or my students) the answers (yet), but come on - the questions generated are comparable to or better than those one would find in the leading Canadian casebooks on Evidence. (No disrespect intended to anyone.)
Having seen what the ChatGPT can produce, virtually instantaneously, on demand, by way of law school fact patterns, I was primed this morning to listen to the new episode of Advisory Opinions - in which, Sarah Isgur, David French, and Christopher Scalia read a one-act play produced by ChatGPT (I think?) based on the United States Supreme Court’s decision in Heller. It was pretty good. We were told, furthermore, that Isgur only had the idea of doing this the afternoon before the podcast aired. Now, it had never before occurred to me to have students produce one-act plays about landmark cases. (Holy hell, what a terrific way to have them learn and demonstrate knowledge of a case!) It would, however, take them weeks (at least in the olden times). I’m pretty sure it would take me a few days to produce a decent first draft. Again, this is remarkable stuff.
There’s more. A couple of weeks ago, Peter Sankoff was chatting with me (over email) about a couple of recent cases. We had a short but frank exchange of ideas. And then, to my surprise, Pete told me that he had taken our exchange, run it through ChatGPT, and had it produce a dialogue between ‘him’ and ‘me’. He sent it to me. It’s not perfect. It doesn’t quite capture my irrepressible charm, humour, and erudition. But, even in its raw form, it was (again) pretty good.
As I said, this is all, pedagogically speaking, super-duper-intriguing. I can, in minutes or (at most) an hour or two, invent not just brand-new fact patterns, but Socratic dialogues and one-act plays? I attended an interesting lecture a couple of weeks ago in which I was shown how to produce visual images of classic cases. I presume I could, with a little extra work, make a mini-movie. (Peter Lorre playing Therrien in the movie adaptation of R v Bradshaw is self-recommending.)1
There are other pedagogical benefits. Importantly, students no longer have to wait for me to produce new fact patterns. Traditionally, one of the big bottlenecks for keen students looking to try their hand at new practice problems has been… me. Now, though, a student that wants to generate infinite fact patterns can do so without my assistance. I have warned my students not to uncritically look to AI for answers to those problems. If, however, students can use chatbots to challenge themselves, and get themselves asking deeper questions, then that is all to the good.
There are other, more ambitious possibilities for those who are more tech-savvy than me. Exciting times.
For the record, I wouldn’t use (and haven’t used) AI for my writing - online or scholarly. I see no value in that, since I use my writing to help me think through ideas. For me, writing and thinking are inseparable, and I would never want to outsource that; it would be outsourcing some of the most rewarding, fulfilling parts of my life. Yes, I’m happy enough to let ChatGPT ‘help’ me put a fact pattern together, but I’m not asking it for answers to the fact pattern, or asking it what cases to turn into fact patterns, or even what issues arise in the fact pattern - since those are matters of academic judgment. Indeed, what I want is precisely something to which I and my students can apply our minds. To my way of thinking, this is just a way of creating the sorts of problems that generations of law professors have gleaned from casebooks written by other law professors. (That said, I would be curious to know how other legal academics feel about the distinction I’m drawing.)
*
As I said, I have been nudged into using AI because I had already decided to go device-free for Evidence Law - and because, having reached that decision, I wanted to make the most of the learning opportunities that would generate. I admit that, when I made that initial decision, it was with some nervousness.
Now, though, the University of Toronto’s Faculty of Law has adopted a default policy of no laptops in the classroom. This, I don’t mind saying, eases my mind a great deal. The University of Toronto quite clearly has one of the leading law schools in the country. (Most expensive too, it must be said, but never mind.) I feel like I’m in good company.
The decision is not, of course, universally popular. Some U of T law students seem to be unhappy about the new policy, arguing that it requires them to alter study habits that have been ingrained over many years. No doubt that’s true, and I don’t want to dismiss that. But it’s also true that the pedagogical benefits of writing notes by hand have been known for quite some time - and that academics and students face novel challenges that warrant new approaches to teaching and learning. We all need to adapt.
More irksome (to me anyway) is the suggestion that the U of T policy is sheer Ludditism. I have seen, in a few corners of social media, individuals suggesting that this is a ‘laptop ban’ and that, rather than ban technology, U of T should be teaching its students how to use AI properly. To my mind, these criticisms are either mistakes about what the policy is, or mistakes about what law schools ought to do in order to ‘teach AI’.
On the first point, though others are obviously better placed than I am to explain U of T’s policy, my understanding is that the ‘laptop ban’ is only a default rule from which individual instructors are entitled to depart. It is no more a 'laptop ban’ than the pre-existing policy was a ‘laptop requirement’. Individual instructors are entitled to decide how best to teach their own students. There are, moreover, a number of ‘safeguards’ for students who are uncomfortable with writing by hand. Per Janet Hurley of the Toronto Star:
Faculty that opt to make a course laptop-free are expected to deliver a record of in-class discussion in at least one of the following ways: AI-generated notes; a designated student note-taker who would be allowed to use a laptop; or class slides, speaker notes and other materials provided by the instructor.
“The aim here is to take some pressure off, by providing notes so that students don’t need to take comprehensive notes by hand,” Stacey wrote, adding that the school recognizes students have varying ability and comfort with handwriting.
Many students who responded to the survey said they “haven’t handwritten notes since high school and for some of them, that’s been the better part of a decade,” said McLachlan.
Students will be permitted to use a tablet and stylus, and the law school’s exam policy, which allows the use of laptops, also still stands.
This strikes me as quite reasonable.
On the second point - what should law schools be doing to teach responsible use of AI? - I think we should be relatively humble and candid about what law schools are in a position to do relatively well, and what they’re not.2 The American law professor and legal ethicist, W Bradley Wendel, wrote an excellent post a few weeks ago in which he observed:
[S]pecifically with regard to AI, legal educators should not assume that we are in the best position to provide the practical training that recent law school graduates need. The reason is threefold, with each branch related to professional competence: (1) the vast majority of us law school faculty, even those with relatively recent practice experience or those in clinical or skills-training positions, are not up to speed on what law firms and other legal practice organizations are doing right now with AI; (2) even if a faculty member is pretty sharp on today’s applications, the technology is changing so rapidly that we have no idea what the students will be doing with AI tools 2+ years down the road when they graduate; and (3) we need all the time we can get to train students in the fundamentals of legal reasoning, critical thinking, and the exercise of sound judgment, which continue to be essential in a world in which human lawyers work extensively with AI tools to deliver legal services to clients that satisfy standards of competence, ethics, and professionalism.
I recommend the full piece, where he develops each of these points at length. Wendel’s second point certainly deserves more attention than it is getting. This is not like the situation we confronted a few decades ago, when commercial legal databases arrived on the scene. There were only a few of them (Quicklaw; Westlaw; Lexis-Nexis), and each one was particularly good at some things, and less good at others. No law firm was inventing its own bespoke legal database (because why on earth would you do that?). With a limited range of commercial database options, it made sense to teach students how to use them. Moreover, the capacities and characteristics of the various databases did not fundamentally change.
The situation with AI is just not like that. As Wendel says, the technology is likely to change in profound ways even in the next year or two. More importantly, many larger law firms are not using off-the-shelf AI, but are having bespoke AI programs made just for them - precisely because AI allows for the sort of ‘vibe-coding’ that makes such a thing cost-effective for firms, and because firms want to train their AI in particular areas of legal specialization and context.
More generally, I am somewhat skeptical that a student who is able to come to grips with legal reasoning and analysis, and is equipped with sound legal judgment, is going to need law schools to help them with the tech. I’m sorry, but a lot of the young people who are coming into law school have lived and breathed AI for much of their adult lives. They have an intuitive feel for this stuff that (I feel confident saying) the vast majority of law professors do not. (We didn’t have to teach law students how to use desktops, laptops, word processing software, or email either.)
What law students need to understand is what makes a legal analysis or argument good or bad, better or worse - and not in the abstract, but in the context of a living, concrete legal dispute between human beings with real human interests and values. That is where we come in. We can teach that - or at least bring most of them up to a basic, middling level of competence - because we have always been teaching that. (Or we purported to anyway.)
But to teach those essential legal skills, we need to take the AI crutch away from students for periods of time. We need students to actively try to make sense of arguments and analyses and interpretations, make mistakes, figure out where they went wrong and course-correct. If students do that enough, watching their instructors and peers think through, justify, and critique legal arguments and analyses and interpretations, then doing it themselves, they will acquire legal judgment.
The kicker is that, by doing all of this, and acquiring legal judgment, law students will become better users of AI in their legal work. This seems to be an idea that lots of critics of the University of Chicago or University of Toronto approach find elusive - i.e., that law schools may be better able to make their graduates effective users of AI by not making AI available in the classroom.
That does not mean - or, anyway, does not have to mean - restricting access to AI for any and all purposes outside the classroom. On the contrary, we probably want law students to use the tech in some ways and for some purposes. I have lately been musing about the possibility of students using AI as a tutor. Imagine students posing questions to a suitably ‘trained’ chatbot, soliciting criticism and feedback on legal analyses and arguments, sharpening their understanding of concepts and doctrines, and then coming into class prepared to deal with fact patterns on their own. How freakin’ cool would that be? If, for the past hundred years, law schools could have provided a tutor for every single law student, of course we would have done it. And now, in principle, we can! Magic.
To make that work, though, students also need to be forming independent legal judgment without the tech. Otherwise, they won’t really grasp the limits of the tools they’re using - and they won’t be any use to all of the lawyers and law firms wanting people skilled in using them.
*
None of the above tells us how we should teach legal writing. That remains a critical problem. As I have said on several occasions, it isn’t good enough just to abolish all of our take-home research assignments and replace them with in-class tests and exercises. The ability to do independent research, synthesize primary and secondary sources, critically assess them, and construct an elegant and persuasive legal argument is core to what we expect law graduates to be able to do. (Hell, it’s core to what we ought to expect any recipient of a liberal education to be able to do, but hooboy that feels like a distant dream at this point.) To give up on independent research is to give up on our students.
Anthropic has announced that, in order to comply with the European Union’s AI legislation, writing produced with Claude will have a ‘watermark’. Anthropic will soon release software capable of detecting the watermark. Other major AI players will soon add their own watermarks. I had been cautiously optimistic about this development. Even without any effective detection software right now, I mused, universities and law schools might credibly hold out the possibility that such software might come into being. Maybe this would just be a Boogeyman. But Boogeymen can serve a useful purpose in the ethical ecosystem.
Anyway, I am no longer so optimistic. Given Anthropic’s explainer about how the watermark works and what its limitations will be, I am not so confident that any watermark-detection program would be useful in catching most of what we would conceivably want to catch. It might be useful in telling us, in probabilistic terms, whether Claude was used to write the first draft, but the reliability of that probability assessment will depend on the length of the document. It also appears unable to tell us whether Claude was used to proofread or edit a document initially written by a human being. These are significant limitations. Pangram may be the best detection option for the foreseeable future, and that is not nearly good enough.
A sober note on which to end a pretty upbeat post. I'll let you know how it goes with Peter Lorre.
To say nothing of the fact that I can finally live out my dream project of making a movie based on the facts of the Australian High Court ruling in Weissensteiner v The Queen.
To be clear, I do not claim to know what U of T Law is doing, and I do know that there are legal academics at U of T who are world-leaders in thinking about artificial intelligence and the law.
Great post Michael! So much to think about here.