Full Breakdown
Evaluating AI in Legal Tech: The Launch of Lake Merritt
9/3/2025, 1:43:15 PM
Introduction to Lake Merritt
Lake Merritt is an open-source AI evaluation system designed to assist legal, risk, and product leaders in assessing the effectiveness of AI tools within their organizations. Developed by Dazza Greenwood, a prominent figure in legal technology, Lake Merritt aims to provide a framework for evaluating AI performance tailored specifically for legal applications. The Alpha version of the platform allows users to define success metrics, create "golden datasets," and test AI behavior against real-world legal and business priorities. Greenwood emphasized that the platform is not merely a technical tool but a strategic function essential for ensuring compliance and trust in AI systems.
Features and Functionality
The platform offers a user-friendly interface that allows legal professionals to conduct evaluations ranging from simple CSV-based assessments to complex multi-step agent testing. The "Eval Pack" architecture enhances transparency and version control, making it adaptable to evolving needs. Greenwood noted that Lake Merritt is particularly valuable in a legal landscape where few evaluation systems exist, enabling non-engineering experts to gain insights into the performance of AI tools they are integrating.
The Concept of Agentic AI
In parallel discussions about AI in the legal field, the concept of agentic AI has emerged. True agentic AI is characterized by its ability to autonomously make decisions and adapt plans based on user intent. However, many tools currently marketed as agentic AI do not meet this definition. Critics argue that these tools often operate through linear orchestrations, executing a series of predetermined API calls without genuine decision-making capabilities. This misclassification complicates the understanding of what constitutes effective AI in legal contexts.
Criticism of Current AI Tools
Critics, including Richard Tromans and Jake from Flank, have pointed out that many so-called agentic AI tools merely automate processes without the ability to reflect on their actions or adjust based on outcomes. They argue that true agentic AI should be able to handle unexpected requests and interact directly with non-experts, functioning as a member of a legal team rather than just a tool for lawyers. The current market is flooded with products that fail to deliver on this promise, leading to confusion among legal professionals regarding the capabilities of these systems.
Official Statements & Responses
Dazza Greenwood stated, "Lake Merritt is designed to answer the most important AI governance question: how do we know this system is working the way we need it to?" This highlights the platform's focus on empowering legal professionals to take charge of AI evaluations. In contrast, critics emphasize the need for clarity in the definitions and capabilities of AI tools, warning against the proliferation of products that do not genuinely embody agentic qualities.
What's Next for Lake Merritt and AI in Legal Tech
As Lake Merritt continues to develop, it is expected to introduce new workflows and features that enhance its evaluation capabilities. The ongoing discourse around agentic AI will likely influence future innovations in legal technology, pushing for more sophisticated systems that can genuinely replace human expertise in specific legal tasks while maintaining necessary oversight.
In summary, Lake Merritt represents a significant step forward in the evaluation of AI tools in the legal sector, while the conversation around agentic AI underscores the need for clarity and genuine innovation in the field.
