Percipient develops specialized software and AI systems for legal applications, where statutory accuracy, evaluation, and zero tolerance for hallucination are essential

I was brought on as an AI consultant to help their team develop in-house capabilities around fine-tuning/post-training and evaluating open-weights foundational models. The engagement focused on fine-tuning models for agentic legal tasks (with a particular emphasis on employment law) and establishing rigorous evaluation standards so the team could iterate independently

Some highlights

• Domain-specific evaluation harness: General-purpose LLM benchmarks (like MMLU) fail to measure legal competence. Inspired by domain-specific legal evaluation frameworks (such as Harvey's Legal Agent Benchmark), I designed a custom evaluation harness tailored for our tasks. The framework combined deterministic programmatic checks (citation validity, schema compliance, etc) with LLM-as-a-judge rubrics to reliably benchmark candidate checkpoints
• Fine-tuning on agent trajectories: We designed and implemented end-to-end data preparation and supervised fine-tuning (SFT) pipelines using multi-turn agent trajectories. By training open-weights foundational models on trajectories containing intermediate tool calls, document retrieval steps, and other details, we aim to improve model reliability on multi-step legal agentic tasks, especially for more compact/smaller models
• Preference alignment with DPO: To understand how to better align model outputs with high legal standards, I explored direct preference optimization (DPO) workflows in the post-training pipeline
• Our stack: Configured an efficient infrastructure setup on RunPod using Unsloth for parameter-efficient fine-tuning (PEFT / LoRA), also cutting memory overhead with fused kernels (such as fla and causal-conv1d) . For batch evaluation and model deployment, we aim at vLLM inference instances
• Knowledge transfer & team enablement: To support Percipient's internal team long-term, I delivered persistent technical documentation (like step-by-step experiment runbooks) and hands-on demos/handoffs

And the nicest thing? We plan to make our fine-tuned model available at Percipient's Certera AI! Here's how it works: