"Demonstrating" Competency
On the National Requirement and Canadian legal education
On 12 March 2024, the Federation of Law Societies of Canada [FLSC] approved a new National Requirement - establishing a condition for the ongoing accreditation of Canadian law schools. Once the requirement comes into effect, in 2029, law schools must meet the standards set out in the National Requirement in order for their JD programs to be recognized by the various law societies across the country. For my purposes, I am especially interested in requirement 1.3:
C. APPROVED CANADIAN LAW DEGREE The Federation will accept an LL.B. or J.D. degree from a Canadian law school as meeting the competency requirements if the law school offers an academic and professional legal education that will prepare the student for entry to a bar admission program and the law school meets the following criteria:
….
1.3 Holders of the degree have met the competency requirements.
On the face of it, 1.3 makes accreditation conditional on a law school producing graduates who actually meet the competency requirements. Moreover, to meet the competency requirements, an “applicant [to a bar admission program] must have demonstrated” competency in a wide range of skills and subject-matters. These competencies are set out in Part B of the National Requirement. It is a lengthy list. I bring that up, not because I wish to quibble with any particular skill or subject listed, but because I wish to draw attention to the challenges of creating and administering a three-year law degree in which every graduate has demonstrated competency in all of those skills and subjects. For, taking that language seriously, it does not seem to be enough for the law school simply to provide instruction in the relevant subject areas; it must be able to attest that graduates have demonstrated - presumably, through some sort of assessment - competency in those areas.
To be clear, this language of “demonstrated competency” is not new. It dates back to the 2011 version of the National Requirement. The competencies themselves have changed since then, but not the stated expectation that they be “demonstrated” by law graduates. Each year, the Dean of a law school must submit a form, along with supporting documents, to the FLSC, attesting to the school's compliance with the National Requirement - including that all of its graduates have “demonstrated competency”.
(Actually, the wording in the form varies: sometimes, the Dean must explain how students will achieve understanding of competencies; sometimes, how students will have demonstrated competencies. I am not sure how much is meant to turn on this, given that the Dean is surely expected to say what he or she reasonably believes about the law program and all of its graduates.)
To me, the language of “demonstrated competency” is striking. As I said, this sounds much more demanding than mere instruction in the skills and subjects set out in Part B. What, though, would it mean for law schools - and the FLSC - to take such a requirement truly seriously?
What does it mean to demonstrate competency?
Assessment in law school courses is still dominated by in-class written examinations of two- to three-hours. (If anything, law professors will lean more and more heavily on written examinations as concerns grow about the use by students of artificial intelligence for take-home assignments (e.g., research papers; memoranda; reflective journals; etc. More on this later.) It has not traditionally been expected that a written examination for a course will assess students on all of the topics and materials assigned in that course. On the contrary, the professor will devise questions - typically, at least for survey courses, fact patterns (“hypos”) - that raise a number of legal issues which the student is expected to ‘spot’. (I note that ‘issue-spotting’ is in fact one of the competencies a law student is expected to demonstrate - so good news there.) Large swaths of the course content may not be assessed in the exam - precisely because there is only so much that a single, three-hour written examination can cover. And that has generally been thought to be fine, inasmuch as the possibility that anything could be covered in the exam would (one hopes anyway) incentivize students to study everything.
In light of the National Requirement, however, surely there is a problem if students are not actually examined (or otherwise assessed) on certain subjects. Take, for example, because it is near and dear to my heart, the “process of statutory construction and analysis”. Now, this is a topic that may arise in a number of courses in one way or another - e.g., criminal law; administrative law; to some extent constitutional law; perhaps in the context of a course on sale of goods. It would not, however, be typical to require students to engage in any significant statutory interpretation or analysis in, say, a final examination. (Students would often be expected to know something about how a court has already interpreted a statutory provision or scheme, but not to actually undertake an interpretation “from scratch” themselves.) Statutory interpretation is, by its nature, a time-consuming affair, requiring close, subtle reading. And a suitable statutory interpretation question must give students more than just a statutory provision in isolation - ideally, there must be some statutory context (so: surrounding provisions; possibly the legislative scheme as a whole); some guidance as to what animated the creation of the statute, so that students can wrestle with legislative intent and possibly level-of-abstraction problems; perhaps dictionary definitions and more.
(To give you an idea of the challenge, here is an earlier post in which I reproduce a statutory interpretation problem I have used for my course in Criminal Law.)
It can be done. I’ve done it many times. But giving that sort of question does impose structural constraints on the rest of the exam. If I assign a proper stat interp question, I must proceed on the basis that students will need a great deal of time to do the sort of close reading and nuanced thinking needed to answer it. It follows that the question must be worth enough marks to warrant the time expenditure: I cannot really have a question that takes students one hour (out of three) to answer, that is worth less than 33% of the overall grade. This leaves less time (and fewer marks) available for other topics. Yes, the fact pattern can raise more than just stat interp issues - it may engage issues concerning fault or defences. Even so, there is much that will get squeezed out. Which is a good, practical reason not to assign a stat interp problem in an examination - leaving more room for the other substantive topics, and getting more overall coverage. I still like to have a stat interp problem in my Crim exams, because I believe it is important. But not every criminal law professor does this.
I raise all this in order to illustrate the practical challenges for assessing law students on something like statutory interpretation in an examination. If we don’t, can we really say that students have demonstrated competency in statutory interpretation and analysis? It is hard to see how. They might know it, but they haven’t demonstrated that knowledge to us.
We can press this even further. Suppose I examine my students on stat interp, but some of them fail that question - not the whole exam; just that question. Surely, they have not demonstrated competency in stat interp, even if they demonstrated competency in “criminal law” (more or less). Indeed, it is not inconceivable that a student who did very well on the rest of the exam might, overall, get significantly more than a bare pass - again, without demonstrating competency in statutory interpretation. What does that mean for the student and for the law school’s JD program as a whole? Must the law school ‘hold back’ students from graduating unless and until they have demonstrated competency in the specific stat interp module of the course? That would be a far more granular investigation into a student’s understanding of ‘the law’ than law schools have traditionally done.
And while I’m at it, what does it mean to demonstrate competency? Is a D- grade on a stat interp problem really a demonstration of competency? A C-? I know the old joke about what you call a law student who got a D - answer: “a lawyer” - but it seems as though the National Requirement at least purports to hold students (and law schools) to a higher standard. Does the mere fact that the student has overcome the minimum standards for academic progression mean that the student has demonstrated sufficient competency for professional purposes? I think this question is particularly urgent given the ubiquitous grade inflation in law schools. A passing grade, after all, is not what it used to be.
Whose responsibility is it?
Many of the skills and subjects set out in Part B of the National Requirement are not ‘local’ to any one course in law school. One could, in principle, take a dedicated, upper-year course in ‘legal reasoning’, ‘the common law method’, ‘equity’, or ‘statutory interpretation’. (Perhaps some jurisprudence, legal theory, or legal history courses would serve.) But most students get exposure to these skills and ideas by taking other subjects in which they are applied. They learn the common law method, we hope, by taking courses in torts, contracts, property, and more. They learn statutory interpretation, we hope, by taking courses in criminal law, administrative law, and other courses. (There is also legal research and writing.) These skills and subjects, in other words, are often taught in a diffused, refracted manner - through, again, exposure to how those skills and ideas are used and manifested in discrete sub-disciplines, rather than through dedicated, highly abstract courses on analogical reasoning, stare decisis, or linguistic canons of interpretation.
There is much to speak for this way of conveying legal reasoning skills. But it may mean that no individual professor has primary responsibility for ensuring that students understand, say, the law of precedent. Students may, in any given course, receive little more than superficial instruction on the underlying legal techniques - the focus being on the particular ‘takeaways’ from this case or that case; how this or that statutory provision has been interpreted, without much attention to how those results were actually reached. Certainly, there is no guarantee that any given course will assess students on their understanding of the legal method - rather than on a set of substantive doctrines or ‘rules’.
Returning to statutory interpretation for a moment, it seems to me - if I am to judge from the leading casebooks and textbooks on criminal law; as well as from assorted syllabi and scattered conversations - lots of criminal law professors simply don’t take much interest in statutory interpretation, or think it is especially important. They recognize that the Criminal Code is a statute, of course, and that a mysterious case called Rizzo Shoes seems to come up with peculiar regularity, but discussion of statutory interpretation may occupy no more than a single lecture over the term - a lecture that dwells overlong on the ‘rule of strict construction’ - and then barely gets mentioned again for the rest of the course. (I wrote a book complaining about this.) I take it as obvious that a student who was asked a question about statutory interpretation, having received such perfunctory instruction, would not know much about the subject. But I also take it as obvious that the professor who dedicated such little time to the subject would not make the question very demanding and would not expect much. Would even a high grade given under those circumstances really demonstrate competency? Surely not.
Now, if criminal law is meant to convey the techniques of statutory interpretation, construction and analysis, this will not do. But is it? Should it? Criminal law deals with a very special kind of statute: a Code. A Criminal Code is not like other statutes. And criminal offences that define public wrongs are not really like statutory provisions that, say, create and empower administrative bodies; or that determine how one may create a valid will. Realistically, we do not approach these statutes in quite the same way. To get a clear sense of whether a student really understands statutory interpretation in the round, one would need to see how he or she deals with a fair sample of different statutes. And my point is that this will never happen - barring the unusual student who takes a dedicated, elective course on statutory interpretation. Even if professors across different subjects all meaningfully examine the student on statutory interpretation - which is already an enormous ‘if’ - the fundamental focus in each of the exams will be on the student’s mastery of the specific subject-matter in the course, and not ‘general’ skills. That will always be someone else’s problem.
Of course, one could imagine a law school administration actively and aggressively re-organizing the curriculum so that every skill and subject listed in Part B had a clear ‘point-person’ - someone directly accountable for ensuring that the skill or subject was taught to the requisite (though nebulous) standard, and rigorously assessed students on it. That would be a pretty big deal in principle. By and large, for reasons of academic freedom, law schools do not generally micro-manage what and how professors teach and assess in ‘their’ courses. There would be resistance.
How do we deal with the AI problem?
Earlier, I parenthetically observed that law schools will need to start taking seriously the threat posed by AI to the integrity of written assignments. I already get the sense that professors are less willing to base grades on papers, memos, etc. It is simply too difficult - and, frankly, too dispiriting - to conclusively determine whether (and how) a student used AI to produce (or “polish”) written work. One can check for plagiarism, look for hallucinations, and cross-examine students on this or that suspicious-looking passage. But good grief. This just does not scale. And who wants that kind of suspicious relationship with their own students anyway?
This is potentially a significant problem for law schools if we are to take seriously the idea that students must demonstrate competency in legal research; and written legal communication:
1.2. Legal Research The applicant must have demonstrated the ability to: a. identify legal issues; b. select sources and methods and conduct legal research relevant to Canadian law; c. use techniques of legal reasoning and argument, such as case analysis and statutory interpretation, to analyze legal issues; d. identify, interpret and apply results of research; and e. effectively communicate the results of research
1.3. Oral and Written Legal Communication The applicant must have demonstrated the ability to: a. communicate clearly, professionally and effectively in the English or French language; b. identify the purpose of the proposed communication; 4 c. use correct grammar, spelling and language suitable to the purpose of the communication and for its intended audience; and d. effectively formulate and present well-reasoned and accurate legal argument, analysis, advice or submissions.
The problem is threefold. First, fewer written assignments means fewer opportunities for students to demonstrate competency. Second, even supposing that every student had adequate opportunities, how could we possibly know - now, never mind in 2029 - how many used AI to write those assignments? How could any law school, in confidence and good faith, declare that its students had demonstrated competency without having some reliable way to enforce academic misconduct regulations? One must keep in mind that it is not only bad people who cheat - students may cheat simply because they know/believe that other students are cheating and don’t want to be chumps. (This is especially true insofar as law school grades are officially or unofficially ‘curved’.)
The third problem is more indirect. As students increasingly lean on AI to do their written work - and indeed their thinking - for them, their research and analytical skills will grow increasingly flabby. (This is by no means a new concern; but AI will accelerate the deterioration.) Will the threshold for ‘demonstrating competency’ stay put or will it drop? I have been in academia too long to believe that grades will necessarily decline as student performance declines. Again, though, the fact that students’ academic progression might be unaffected does not have to mean that they will have demonstrated competency for professional purposes.
Concluding Thoughts
I have emphasized some of the issues that would arise for law schools if they took seriously the requirement that graduates have ‘demonstrated competency’ in the skills and subjects set out in Part B. (Not all of them, I hasten to add. Given the added skills and subjects in the 2024 National Requirement, there will also be an urgent need at many law schools to hire professors in certain areas; to create new mandatory courses or program requirements; and more. But since my focus here is on the language of “demonstrated competency” itself, I set these other challenges aside.)
How to overcome these issues? Well, the easiest, least disruptive way would simply be to proceed on the basis that the language of ‘demonstrating competency’ should not be taken literally; that merely an assurance that students have received some instruction in the listed skills and subjects is sufficient. In other words, the easiest path is not to take seriously the “demonstrated competency” language. I expect that this is what law schools, by and large, have done; and that the FLSC has, thus far, accepted that. Law societies do not really want to get into the business of managing law schools - even if they had the resources to do so. Nor do they really want to scrutinize every single graduate. To some extent, then, we likely remain in a situation where (as Harry Arthurs put it, describing the pre-2009 “dispensation”) “law societies pretend to regulate, and law schools pretend to comply.”
Yet the very fact that the FLSC demands that law schools attest to their compliance with the National Requirement, and the law schools purport to do so, makes the situation more unstable. For the FLSC can always decide that it will cease to accept the attestations provided by law schools; demand more than mere assurances that graduates meet the Requirement on the basis of the instruction supposedly provided. Law schools, in America and Canada, are increasingly under a political microscope. In an age of outspoken law professors on social media, the public has grown skeptical of the legal academy and legal education. Even if the law societies do not want to get deeply involved in curricular matters, they may decide that they have no choice. And law professors themselves can hardly complain. When the first National Requirement was imposed, some mild protests were raised. But the Trinity Western affair, in which a law society squelched a proposed law school arguably for ideological reasons while much of the legal academy cheered it on for doing so, no doubt emboldened the professional regulators to see the law schools as basically their turf. In light of recent history, law professors should not blithely assume that they will never be held to the plain language of the National Requirement.
Comments welcome! I hope everyone is enjoying the weekend.
(Many thanks to Ian Holloway for offering feedback on a draft of this post.)
