LegalMind-Gemma4-31B-BillAnalyst

A LoRA-adapted Gemma 4 31B model fine-tuned for multi-step US Congressional bill analysis — built end-to-end using the Adaption Labs AutoScientist platform for the 2026 Adaption AutoScientist Challenge (Legal Category).

---

Model Details

Model Description

LegalMind-Gemma4-31B-BillAnalyst is a parameter-efficient LoRA adapter that transforms Google's Gemma 4 31B instruction-tuned model into a specialized legal reasoning engine for US Congressional legislation. Unlike simple bill summarization models, it performs 8 distinct multi-step legal analysis tasks — from statutory interpretation with step-by-step reasoning traces to structured compliance checklist generation and explicit abstention when bill text lacks requested information.

The model was trained entirely through the Adaption Labs AutoScientist pipeline, which automated hyperparameter optimization, data enhancement, and the training loop. The result is an 82% training win rate against the unmodified baseline (up from just 18%), representing a 355% relative improvement.

• Developed by: lohithaaa
• Funded by: Self-funded; compute credits provided by Adaption Labs (1,000 platform credits)
• Shared by: lohithaaa
• Model type: Causal Language Model (LoRA Adapter)
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model: google/gemma-4-31B-it

Model Sources

• Repository: lohithaaa/adaption-us-bill-analysis-tasks
• Dataset: lohithaaa/adaption-us-bill-analysis-tasks
• Training Platform: Adaption Labs AutoScientist
• Competition: 2026 Adaption AutoScientist Challenge — HackIndia

---

Results at a Glance

Metric	Base Model	Adapted Model	Delta
Training Win Rate	18%	82%	+64 points
Relative Improvement	—	—	355%
Final Training Loss	—	0.761	—
Final Eval Loss	—	0.751	—

Training Loss Progression

Step	Training Loss	Eval Loss	Learning Rate
1	1.5427	—	0.0 (warmup)
7	1.4932	—	1.0e-4 (peak)
14	1.0220	1.0489	9.6e-5
25	0.9026	0.8542	7.6e-5
36	0.7837	0.7780	4.7e-5
47	0.7528	0.7589	2.1e-5
58	0.7615	0.7507	1.0e-5

Key observation: Train loss and eval loss converge closely throughout training (final gap: 0.011), confirming no overfitting. Eval loss monotonically decreased from 1.049 → 0.751 across all 5 checkpoints.

---

Uses

Direct Use

This model is designed for automated legal analysis of US Congressional legislation. It can be used directly for:

• Statutory interpretation — Analyzing the legislative mechanism, instruments used, and enforcement pathways of a bill
• Impact analysis — Assessing stakeholders, fiscal implications, and unintended consequences
• Compliance checklist generation — Extracting structured obligations, timelines, and penalties into table format
• Legal risk assessment — Evaluating constitutional challenges, litigation risk, and implementation complexity
• Amendment analysis — Identifying and analyzing existing laws being modified by a bill
• Comparative bill analysis — Side-by-side comparison of two competing legislative approaches
• Key provisions extraction — Parsing provisions into structured tables with type classification (Obligation/Authorization/Definition)
• Abstention — Correctly refusing to fabricate information when the bill text doesn't contain the requested data

Downstream Use

• Legal technology platforms for automated legislative monitoring
• Congressional research assistants for policy analysts
• Regulatory compliance workflows for organizations tracking federal legislation
• Legal education tools for law students studying statutory interpretation
• Government affairs teams analyzing pending legislation

Out-of-Scope Use

• Not legal advice: This model is a research and analysis tool. Its outputs should never be treated as professional legal counsel or relied upon for legal decisions.
• Non-US legislation: Trained exclusively on US Congressional bills; not designed for UK, EU, or other jurisdictions.
• Case law analysis: Not trained on judicial opinions, court rulings, or case precedents.
• Contract review: Not designed for contract clause analysis or commercial agreement review.
• Real-time legal research: Does not have access to current legislation or legal databases.

---

Bias, Risks, and Limitations

• Jurisdictional bias: Trained exclusively on US federal legislation. Performance on state-level bills, foreign legislation, or international law is untested and likely degraded.
• Temporal bias: Training data consists of historical Congressional bills. The model may not reflect current legislative drafting conventions or recently enacted legal frameworks.
• Hallucination risk: While the model includes abstention training (~11% of dataset), it may still occasionally fabricate statutory references, dollar amounts, or section numbers not present in the input text.
• Base model limitations: Inherits the biases, factual limitations, and failure modes of the underlying Gemma 4 31B architecture.
• Structured output consistency: The model is trained to produce tables, checklists, and structured formats, but may occasionally produce malformed markdown or inconsistent formatting.
• Training data scope: The 6,648 training examples are derived from 1,000 unique bills. Analysis patterns may be less robust for bill topics underrepresented in the BillSum sample.

Recommendations

• Always verify model outputs against the original bill text before use in any professional context.
• Use the model's reasoning traces (<reasoning_start> / </reasoning_start>) to audit the analytical process and identify potential errors.
• Cross-reference extracted dollar amounts, section numbers, and dates with the source document.
• Treat the model as an analytical assistant, not an authoritative legal source.
• Be especially cautious with abstention — if the model says information is "Not specified in bill text," verify this claim manually.

---

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-31B-it",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-31B-it")

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "lohithaaa/adaption-us-bill-analysis-tasks"
)

# Example: Statutory Interpretation
prompt = """Analyze the following US Congressional bill and identify its primary
legislative mechanism. Explain step-by-step: (1) what existing law or framework
it modifies or creates, (2) the specific legal instruments used (amendments, new
sections, appropriations, mandates), and (3) the enforcement or implementation
pathway.

SECTION 1. SHORT TITLE.
    This Act may be cited as the 'Fair Lending for All Act'.

SEC. 2. PROHIBITION ON DISCRIMINATORY LENDING.
    (a) IN GENERAL.—No covered lender shall deny or vary the terms of a
    residential mortgage loan based on the race, color, religion, national
    origin, sex, or age of the applicant.
    (b) ENFORCEMENT.—The Bureau of Consumer Financial Protection shall
    enforce this section under the authorities granted by the Consumer
    Financial Protection Act of 2010.
    (c) PENALTIES.—Any lender found in violation of subsection (a) shall
    be subject to a civil penalty of not more than $100,000 per violation."""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Details

Training Data

Dataset: lohithaaa/adaption-us-bill-analysis-tasks

Source: 1,000 US Congressional bills from the BillSum dataset (CC0 Public Domain), programmatically transformed into 6,648 multi-task legal reasoning examples.

Task Distribution:

Task Type	Count	%	What It Teaches
Statutory Interpretation	1,000	15.0%	Legislative mechanism analysis with reasoning traces
Impact Analysis	1,000	15.0%	Multi-factor stakeholder and fiscal assessment
Compliance Checklist	1,000	15.0%	Structured obligation extraction into tables
Risk Assessment	1,000	15.0%	Constitutional & litigation risk scoring
Key Provisions	1,000	15.0%	Table-format provision extraction with type classification
Abstention/Safety	723	10.9%	Explicit refusal when information is missing
Amendment Analysis	675	10.2%	Cross-referencing existing laws being modified
Comparative Analysis	250	3.8%	Side-by-side bill comparison

Key dataset design decisions:

• Every output includes <reasoning_start> / </reasoning_start> step-by-step reasoning traces
• Deterministic metadata (dollar amounts, section counts, "shall"/"must" frequency) extracted directly from bill text
• SHA-256 hash-based deduplication on output text
• ~11% abstention tasks teaching the model when NOT to answer
• Capped at 6,648 rows (within the empirically optimal 5,000–8,500 range)

Training Procedure

Adaption Labs AutoScientist Pipeline

This model was trained entirely through the Adaption Labs AutoScientist platform, which automated data enhancement, hyperparameter search, and the training loop. Below are the exact pipeline configurations used:

Enhancement Recipe Configuration:

Recipe Toggle	Setting	Strategic Rationale
Prompt Rephrase	❌ OFF	Deliberately disabled to preserve deterministic numeric data. Our dataset contains exact dollar amounts ($100,000), section references (SEC. 2), and word-frequency counts ('shall' appears 14 times). Automatic rephrasing would corrupt these strict numeric constants into vague approximations, degrading training signal quality.
Prompt Deduplication	✅ ON	Second-layer deduplication on top of our pre-processing SHA-256 hash dedup, ensuring maximum dataset cleanliness and preventing the model from memorizing repeated output patterns.
Prompt Metadata Injection	✅ ON	Enriches each prompt with additional contextual metadata, improving the model's understanding of legal domain nuances and bill structure.
House Special	✅ ON	Adaption's most powerful proprietary combination of optimization recipes, applied for maximum quality uplift across all training examples.
Reasoning Traces	✅ ON	Critical toggle for our dataset — every output contains <reasoning_start> step-by-step deductive logic. This ensures the training loop specifically optimizes for interpretable, traceable reasoning pathways.
Hallucination Mitigation	✅ ON	Reinforces the ~11% abstention tasks where the model must refuse to fabricate information. Works in concert with our "Not specified in bill text" training examples to minimize hallucination of legal facts.

Blueprint (System Preamble):

A custom system preamble was configured in the Blueprint stage to align the enhancement pipeline with the dataset's behavioral objectives:

You are an expert legal analyst specializing in US Congressional legislation.

Rules:
1. Always use <reasoning_start> and </reasoning_start> tags for step-by-step analysis
2. Present outputs in structured formats (tables, numbered steps, checklists)
3. Reproduce numeric data exactly as stated — never approximate
4. If information is missing from bill text, state "Not specified in bill text"
5. Systematically evaluate: legislative mechanism, stakeholders, fiscal impact,
   implementation challenges, and constitutional considerations
6. Never fabricate case citations, statutory references, or legal advice

This ensures the platform's enhancement algorithms optimize in the same direction as the training data — reinforcing reasoning traces, structured output, deterministic numeric accuracy, and explicit abstention behavior.

Training Hyperparameters

All hyperparameters were automatically selected by the Adaption Labs AutoScientist — no manual tuning was performed:

• Training regime: bf16 mixed precision
• Training method: Supervised Fine-Tuning (SFT)
• Adapter: LoRA (Low-Rank Adaptation)
• LoRA rank (r): 8
• LoRA alpha: 8
• LoRA dropout: 0.0
• Target modules: q_proj, v_proj
• Learning rate: 1e-4
• LR scheduler: Cosine (0.5 cycles)
• Min LR ratio: 0.1
• Warmup ratio: 0.1
• Weight decay: 0.0
• Max gradient norm: 2.0
• Epochs: 1
• Batch size: 1 (max for memory)
• Total training steps: 58
• Train on inputs: false (output tokens only)
• Number of evaluations: 5

Speeds, Sizes, Times

• Total training steps: 58
• Total floating point operations: 3.96 × 10¹⁸
• Eval runtime per checkpoint: ~17.2 seconds
• Eval throughput: 0.87 samples/second
• Adapter size: 41.8 MB (adapter_model.safetensors)
• Framework: PEFT 0.15.1, Transformers 5.10.1

---

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluation was performed using the Adaption Labs platform's internal held-out evaluation methodology. The platform generates test prompts from the legal domain and uses an LLM-as-a-judge framework to compare the adapted model's responses head-to-head against the unmodified baseline.

Additionally, 5 validation checkpoints were evaluated during training using a held-out split from the training data (15 samples, 2 eval steps per checkpoint).

Factors

• Task diversity: Performance evaluated across 8 distinct legal reasoning task types
• Reasoning quality: Assessed via step-by-step reasoning trace coherence
• Numeric accuracy: Precision of extracted dollar amounts, section references, and dates
• Abstention behavior: Correct refusal rate when bill text lacks requested information
• Structured output quality: Consistency and completeness of tables, checklists, and formatted outputs

Metrics

• Win Rate (primary): Percentage of head-to-head comparisons where the adapted model's response is judged superior to the base model's response by an LLM judge
• Training Loss: Cross-entropy loss on training examples
• Eval Loss: Cross-entropy loss on held-out validation split

Results

Metric	Value
Training Win Rate (Adapted)	82%
Training Win Rate (Base)	18%
Win Rate Delta	+64 points (355% relative)
Final Training Loss	0.761
Final Eval Loss	0.751
Train/Eval Loss Gap	0.011 (no overfitting)

Eval loss progression across 5 checkpoints:

Checkpoint	Step	Epoch	Eval Loss
1	14	0.24	1.0489
2	25	0.43	0.8542
3	36	0.62	0.7780
4	47	0.81	0.7589
5	58	1.00	0.7507

Eval loss decreased monotonically across all checkpoints, confirming stable convergence without overfitting.

Summary

The model achieves an 82% training win rate against the unmodified Gemma 4 31B baseline, a massive improvement from the 18% base performance. The 64-point gap demonstrates that the multi-task legal reasoning dataset successfully exposed and addressed fundamental weaknesses in the base model's ability to perform structured legal analysis, step-by-step reasoning, and principled abstention.

This result validates the data-centric approach: rather than scaling model parameters, switching to complex reasoning tasks that exploit the baseline deficit yielded dramatically superior results compared to simple summarization (which achieved only a ~4% gap in prior experiments).

---

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Managed by Adaption Labs (GPU cluster)
• Hours used: < 1 hour (58 training steps + 5 eval checkpoints)
• Cloud Provider: Adaption Labs infrastructure
• Compute Region: Unknown (managed by platform)
• Carbon Emitted: Estimated minimal due to short training duration and LoRA parameter efficiency (only q_proj and v_proj updated)

Technical Specifications

Model Architecture and Objective

• Architecture: Gemma4ForConditionalGeneration (60 transformer layers, mixed sliding/full attention)
• Hidden size: 5,376
• Attention heads: 32 (with 16 key-value heads for sliding, 4 for global/full attention)
• Head dim: 256 (sliding), 512 (global)
• Intermediate size: 21,504
• Vocabulary: 262,144 tokens
• Context window: 262,144 tokens (max position embeddings)
• Sliding window: 1,024 tokens
• Precision: bfloat16
• Adapter: LoRA on q_proj and v_proj (rank 8, alpha 8)
• Total adapter parameters: ~41.8 MB
• Objective: Causal language modeling (next-token prediction on output tokens only)

Compute Infrastructure

Hardware

Managed by Adaption Labs AutoScientist platform. Training utilized their GPU infrastructure with batch size 1 and max memory utilization.

Software

• PEFT: 0.15.1
• Transformers: 5.10.1
• Training Platform: Adaption Labs AutoScientist

---

Citation

BibTeX:

@misc{legalmind-gemma4-billanalyst-2026,
  title={LegalMind-Gemma4-31B-BillAnalyst: Multi-Task Legal Reasoning LoRA for Gemma 4 31B},
  author={lohithaaa},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/lohithaaa/adaption-us-bill-analysis-tasks},
  note={Trained using Adaption Labs AutoScientist platform for the 2026 AutoScientist Challenge (Legal Category)}
}

APA:

lohithaaa. (2026). LegalMind-Gemma4-31B-BillAnalyst: Multi-Task Legal Reasoning LoRA for Gemma 4 31B. Hugging Face. https://huggingface.co/lohithaaa/adaption-us-bill-analysis-tasks

---

Glossary

• LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning method that injects trainable low-rank matrices into specific layers of a pre-trained model, enabling adaptation without modifying the full parameter set.
• SFT (Supervised Fine-Tuning): Training a model on input-output pairs where the correct output is provided.
• Win Rate: The percentage of head-to-head comparisons where the adapted model's response is judged better than the base model's response by an LLM evaluator.
• AutoScientist: Adaption Labs' automated ML pipeline that co-optimizes data quality and training hyperparameters.
• Reasoning Traces: Explicit step-by-step deductive logic enclosed in <reasoning_start> / </reasoning_start> tags.
• Abstention: The model's trained ability to refuse to answer when the input text lacks the information needed to respond accurately.

Acknowledgments

• Adaption Labs — AutoScientist platform for automated data enhancement and model training
• HackIndia — Competition hosting and organization
• BillSum Dataset — Source data (CC0 Public Domain)
• Google DeepMind — Gemma 4 31B base model

Model Card Authors

lohithaaa

Model Card Contact

Via Hugging Face: lohithaaa

Framework versions

• PEFT 0.15.1
• Transformers 5.10.1