The LLT Lab: Scientific Research at Hofstra Law School
Hofstra Law School has created a new kind of research institution: a research laboratory for law modeled on research laboratories in the sciences. The Research Laboratory for Law, Logic and Technology (LLT Lab) conducts empirical research on the reasoning in legal decisions that connects the evidence in the case to the findings of fact (usually called “fact-finding”). In conducting this research, the LLT Lab operates out of a theoretical framework, formulates and tests hypotheses, and disseminates its work products for replication and use by others. This innovative and unique program employs a team approach to data generation and analysis, and integrates research with legal education.
The goal is not only improving legal research and education, but also having an impact on legal decision making in society. Many important aspects of life depend upon accuracy and fairness in decision making – such as legal decisions about employment, housing, education, immigration, disability, and health care benefits. Decisions in these areas by courts or administrative agencies have two components: deciding what the legal rules are (conclusions of law), and deciding whether those rules apply in a particular case (fact-finding). Factfinding is critical but under-studied. Justice and the rule of law require that findings of fact be based reasonably and transparently on the evidence, that similar cases be decided similarly, and that outcomes be reasonably predictable. At the same time, an increased complexity in legal rules and evidence (including expert and scientific evidence) has increased societal costs and has limited access to justice for many Americans. By making its work available to all participants in legal decision-making processes, the LLT Lab aims to increase the transparency, accuracy, efficiency, and accessibility of such decision making.
Legal Reasoning and the Need for Empirical Research
Law is a pragmatic profession. Judges and regulators always balance two different types of objectives: the epistemic objective of producing findings of fact that are as accurate as possible and warranted by the evidence available, and non-epistemic objectives such as procedural fairness to parties, administrative efficiency, and specific substantive objectives (such as protecting public health from unsafe food). In addition, judges and regulators must make important decisions in real time, based on incomplete evidence. The reasoning structures they employ have evolved to serve this pragmatic orientation. Legal reasoning tends to be dynamic and probabilistic in nature, efficiently arriving at plausible conclusions, but those conclusions are subject to revision if new evidence arises or old evidence needs reanalysis. These characteristics make legal reasoning a leading example of what logicians call “default reasoning.”
The pragmatic nature of legal reasoning requires empirical research into how such competing values are being balanced in different legal contexts. Trying to solve legal problems under the rule of law creates reasoning patterns that are effective in solving such problems. Each particular area of law evolves new concepts and modes of reasoning tailored to achieving its own balance of objectives. Only empirical research into the reasoning of actual decisions can discover what factfinders in different areas find plausible, and how those factfinders evaluate nonexpert and expert evidence to reach their conclusions or findings.
Figure 1. Part of the vaccine rule tree, showing three sub-issues for proving causation and the logical connectives aND, or and UNLEss.
Aspects of a new Research Paradigm
Just as science laboratories generate data by classifying and measuring real-world objects or events, the LLT Lab generates data by modeling the logical structure of the reasoning recorded in legal decisions. Such “logic models,” which capture the essential inference structure of the factfinder’s reasoning, have two major components: the legal rules applicable to all similar cases, and the evidentiary reasoning applying those rules to the particular case. First, lab researchers create “rule trees” constructed out of propositions and logical connectives, as models of the legal rules governing the decision-making process. These rule trees are inverted, with the (root) proposition to be proved at the top, and branches extending downward containing the propositions needed to prove the immediately higher proposition. A complete rule tree identifies all the issues of fact in the case, and all the acceptable lines of proof for the ultimate issue.
For example, a major research project in the LLT Lab studies proof of causation in vaccine cases – that is, how to prove whether or not a vaccination caused a patient’s later injury or medical condition. Such difficult issues are decided by “special masters” within the United States Court of Federal Claims in Washington, D.C. Figure 1 shows part of the lab’s rule tree for compensation claims in vaccine cases. The top proposition of the entire tree is the ultimate issue the petitioner must prove – namely, that the petitioner is entitled to compensation. At the bottom of the diagram is a three-part test for proving causation. The petitioner filing the claim must prove: (1) that a “medical theory causally connect[s]” the vaccination and the injury; (2) that a “logical sequence of cause and effect” shows that the vaccination “was the reason for” the injury; and (3) that a “proximate temporal relationship” exists between the vaccination and the injury. (The quotations are from the lead case of Althen v. Secretary of Health and Human Services, 418 F.3d 1274, 1278 (Fed. Cir. 2005).) Figure 1 also shows three logical connectives used in constructing rule trees: “AND” (all connected conditions must be true in order to prove the conclusion); “OR” (at least one connected condition must be true); and “UNLESS” (if the defeating condition is true, then the conclusion is false, even if the other conditions are true).
Second, in modeling the evidentiary reasoning in a particular case, LLT Lab researchers attach the findings of fact to the issues identified by the rule tree, and then create logic models of the reasoning supporting those findings. Thus, the logic model for an entire case includes the generic rule tree with the reasoning of the particular factfinder attached. For example, Figure 2 is a picture of a computer screen showing some of the modeled reasoning from the vaccine decision Casey v. Secretary of Health and Human Services, Case No. 97-612V (December 12, 2005).
The special master found that there was indeed an adequate medical theory of causation, and supported that conclusion by two alternative lines of reasoning based on two causal pathways (direct viral infection and immunemediated inflammatory response). In modeling this reasoning, LLT Lab researchers used the plausibility connective “MAX.” This connective assigns to the conclusion the highest degree of plausibility assigned to any one of the supporting lines of reasoning. On a color computer display or a page printed in color, the round icon before each sentence in the model has a color that indicates the plausibility value assigned to the assertion. In the complete case model, each of these two alternative lines of reasoning also contains further reasoning, which proves each of these two conclusions.
Illustration of a
portion of the
logic model for
the Casey decision
using the Legal
The LLT Lab uses special software called Legal Apprentice™ (a product of Apprentice Systems, Inc.) to create its logic models. The software keeps track of the logic, and propagates plausibility values and truth values up the tree, from individual items of evidence to the ultimate conclusion. The software also creates HTML documents of the logic models, as well as files of the models formatted in XML (a standard format used in Internet-based programs).
As with any scientific research, the next phase in the LLT Lab is to analyze patterns and trends within the data collected. After a lab project (such as the Vaccine-Injury Project, illustrated in Figure 2) selects a sample of decisions to study and generates models for the reasoning in those decisions, lab researchers identify, abstract and formalize the inference patterns that re-occur within those decisions. The LLT Lab is especially interested in discovering “plausibility schemas,” which are patterns of reasoning that warrant default inferences to presumptively true conclusions. The research tries to identify which patterns the factfinders consider persuasive or not, and why. Because complete evidence is almost never available, this usually means developing “theories of uncertainty” – explanations about what evidence is missing, what uncertainty (potential for error) is inherent in drawing the conclusion, and how it could be reasonable to draw the conclusion even without the missing evidence.
The mission of the LLT Lab is not merely to study fact-finding using scientific methods, but also to improve actual decision making in society. The lab uses its website to make publicly available its database of logic models of decisions. Lab researchers also post commentary on those decisions in the form of blogs, as well as articles about patterns and trends they discover across multiple cases, and about broad aspects of the reasoning they study. A priority is developing and providing useful tools that will assist parties, attorneys and decision makers in reaching accurate decisions more efficiently.
The LLT Lab’s systematic focus on description and critique of reasoning and its mission to improve actual decision making in society, together with its organizational structure, enable an integration of research, education and practice. Faculty and students work in teams – reviewing each other’s logic models for decisions, orienting and training new researchers in the LLT Lab’s methodology, writing commentary on cases and topics through blog entries and articles, and brainstorming about hypotheses to test and the patterns discovered in decisions. Research, education and practice are three dimensions of the same core activity. Conducting the research is simultaneously training in logic skills and education in reasoning, while the research products are useful tools in legal practice.
Finally, the LLT Lab’s research methodology is designed to be collaborative not only within the lab itself, but also with other research laboratories. Because the methodology is logic-based, it is possible to compare rule trees and evidentiary reasoning across different areas of law, across different legal systems, and across time. And because the methodology is standardized, it can be used to produce comparable data (models) in multiple labs. For example, the lab currently has a joint research project with the International and Comparative Law Research Laboratory (Lider-Lab) of the Scuola Superiore Sant’Anna in Pisa, Italy. Together, the two labs are conducting comparative investigations of medical malpractice decisions in the United States and Italy, looking for similarities and dissimilarities in the rule systems and proof patterns. Using a single modeling framework allows the two labs to create logic models that can be compared directly to each other.
Hypotheses at the Cutting Edge
True to its roots in scientific method, the LLT Lab formulates and tests hypotheses about both its legal subject matter and its own methodology. For example, one objective of the lab is to refine its protocols for generating the logic models for legal decisions, and to test the reliability of those protocols and the validity of the resulting models. Scientific “reliability” here means the degree of variability in modeling when different researchers model the same decision, and scientific “validity” means the degree to which a model accurately captures the reasoning reported by the factfinder. It is a working hypothesis of the lab that it can develop protocols that will reliably produce acceptably accurate models for legal decisions written by a variety of authors in a natural language such as English. Such protocols provide orientation materials for training new lab researchers, as well as general educational materials for training students in logic skills. Such protocols may also make it possible to automate parts of the modeling process by developing computer software.
An example of a substantive hypothesis about the law involves the influence of legal policy on fact-finding. The hypothesis being tested in the LLT Lab’s vaccine project is that the special masters who act as factfinders have developed default inference patterns peculiar to this area of law, in which the presumptive warrant is furnished in critical part by social policies. The lab is investigating the extent to which those policies guide decisions about how much evidence is sufficient to establish an issue of fact, when residual uncertainty is acceptable, and when burdens of proof shift among the parties. Gathering data about whether and how this actually occurs may lead to a normative critique of the extent to which it should occur.
A third example of a testable hypothesis involves the dynamics within factfinding processes. The hypothesis is that certain fact-finding structures are more likely to develop “soft rules” of inference. Soft rules are general patterns of default reasoning that have become “safe havens” of inference because a reviewing authority (such as an appellate court) has decided that a particular finding is a reasonable inference from particular evidence. The hypothesis is that in an area of complex cases (such as the vaccine compensation cases), with a small number of repeat factfinders (the special masters), and documentation of the supervisory decision once it occurs (the court judgments), at least some patterns determined by authority to be reasonable would become “safe havens” for factfinders who do not wish to be reversed and who have an incentive to be efficient in deciding cases. Such patterns might become de facto default rules of inference in evidence assessment, and carry over from case to case. They are not rules of law, but “soft rules” of practice. The extent of such a phenomenon might have implications not only for increased efficiency in factfinding, but also for decreased fairness to parties in later cases.
Expected Impact of the LLT Lab
The LLT Lab’s approach to research and education has considerable potential as a paradigm. With respect to benefits to society generally, the goal is to produce databases of logic models for legal decisions in important social areas (such as vaccine-injury compensation), together with libraries of reasoning patterns that may be useful across many areas of law. By making this research publicly available to all participants in the legal process, the LLT Lab’s work should increase the transparency and predictability of future decisions, and help ensure that similar cases will be decided similarly. Accuracy should increase as fact-finding reasoning is scrutinized. Moreover, decision-making processes should become more efficient because all participants will be able to better organize their evidence and better assess the settlement value of their cases. Finally, justice should increase because information and insights generated by the LLT Lab will be accessible to parties that could not otherwise afford such expensive and challenging research. These benefits to society (increased transparency, predictability, accuracy, efficiency, and access to justice) should be achievable in many areas of the law, as work at the LLT Lab and other legal research labs progresses.
With respect to impact on research, the LLT Lab demonstrates how to apply scientific methods of modeling and measurement to legal reasoning, and especially to the reasoning of factfinders in actual cases. The research develops libraries of plausibility schemas, or normative patterns of default reasoning, and tests important hypotheses about the structure and dynamics of fact-finding. Moreover, the LLT Lab shows how the model of a research laboratory in the sciences can be applied in a legal setting, so that teams of students and faculty, employing tested methods of data gathering and analysis, can produce research that is valuable to society. This work can also provide a paradigm for research in non-legal areas where documented decision making is available.
The LLT Lab’s databases and pattern libraries should also provide valuable resources for research in related fields outside the law. The lab’s modeling protocols and databases of analyzed legal decisions should provide resources for formal and informal logic theory, as well as for natural-language research in linguistics (especially semantics). Moreover, the LLT Lab’s work should expand the empirical basis for research on artificial intelligence and law, particularly in the area of evidentiary reasoning, and the lab’s modeling protocols should assist artificial-intelligence researchers in automating the extraction of reasoning from natural-language documents. The subtleties of legal reasoning are difficult for non-lawyers to study, but the LLT Lab’s methodology makes legal logic more accessible to non-lawyers.
With regard to the impact on education, the LLT Lab provides a unique paradigm for legal education and for higher education generally. The same techniques developed for analyzing the reasoning of a factfinder will be useful in training students in logic and argumentation skills. The database of modeled cases provides numerous examples of evidentiary reasoning for students to study. Through the use of a team approach to research, the LLT Lab demonstrates how students can acquire logic skills in a research laboratory, while simultaneously producing important databases and tools for society. As a result, the education process, in both law and elsewhere, might become more effective pedagogically, more engaging to students, and more productive for society.