Pablo-Flores-Mollinedo commited on
Commit
f5c3a1b
·
verified ·
1 Parent(s): ff71332

Add external SOTA context charts

Browse files
Files changed (1) hide show
  1. README.md +60 -0
README.md CHANGED
@@ -123,3 +123,63 @@ Typical reporting:
123
  - category breakdown for combinational, arithmetic, sequential, FSM, memory, and larger RTL designs
124
 
125
  This adapter has been evaluated on the full internal 30-task suite with both pass@1 and pass@5. Official external SOTA comparison should next run VerilogEval and RTLLM directly with the same decoding settings.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
123
  - category breakdown for combinational, arithmetic, sequential, FSM, memory, and larger RTL designs
124
 
125
  This adapter has been evaluated on the full internal 30-task suite with both pass@1 and pass@5. Official external SOTA comparison should next run VerilogEval and RTLLM directly with the same decoding settings.
126
+
127
+
128
+ ## External SOTA context
129
+
130
+ > Important: the following charts are **context only**, not official leaderboard placement. External results come from the public Chip Design LLM Zoo table, while this adapter's score is from this repository's internal 30-task paper-style suite. Benchmarks, prompts, decoding settings, and contamination controls differ.
131
+
132
+ Sources:
133
+
134
+ - Chip Design LLM Zoo reports VerilogEval, VerilogEval v2, RTLLM, and other RTL benchmark results and ranks models by VerilogEval pass@1 / RTLLM correct rate: https://iprc-dip.github.io/Chip-Design-LLM-Zoo/
135
+ - NVIDIA's VerilogEval repository describes the VerilogEval harness and dataset: https://github.com/NVlabs/verilog-eval
136
+ - The VerilogEval paper page describes a 156-problem Verilog generation benchmark: https://research.nvidia.com/publication/2023-09_verilogeval-evaluating-large-language-models-verilog-code-generation
137
+ - RTL-Coder is an RTL-code-generation fine-tuning project with RTLCoder-Mistral inference scripts and synthetic RTL data flow: https://github.com/hkust-zhiyao/RTL-Coder
138
+
139
+ ### Reported top performers vs this adapter
140
+
141
+ ![External SOTA context](model_card_assets/external_sota_context.svg)
142
+
143
+ ### Reported fine-tuned/open Verilog models
144
+
145
+ ![Top fine-tuned context](model_card_assets/top_finetuned_context.svg)
146
+
147
+ Representative reported models found in the public tables/search results include:
148
+
149
+ | Model / method | Notes |
150
+ |---|---|
151
+ | ScaleRTL-32B / ScaleRTL†-32B | fine-tuned RTL model, strong VerilogEval reported scores |
152
+ | ChipSeek-R1 | fine-tuned 7B RTL model, strong VerilogEval-Machine reported score |
153
+ | CodeV-CodeLlama / CodeV-DeepSeek / CodeV-CodeQwen | fine-tuned 6.7B/7B RTL/code models |
154
+ | DecoRTL-CodeV | RTL-specific model reported on VerilogEval-Human |
155
+ | VeriReason-Qwen2.5-7B | fine-tuned Qwen2.5 7B RTL model |
156
+ | RTL++ @ 200K Trained | fine-tuned 7B RTL model |
157
+ | RTLCoder-Mistral | RTL-Coder project model family |
158
+ | qwen3-32b-verilog-lora | Hugging Face Verilog LoRA adapter |
159
+
160
+ ### Why this comparison is cautious
161
+
162
+ The current v4.1 adapter has been tested thoroughly on this repository's full internal suite:
163
+
164
+ ```text
165
+ 30 tasks total
166
+ 22 functional simulation tasks
167
+ 8 compile-only / RTL-heavy tasks
168
+ pass@1 task pass: 83.33%
169
+ pass@5 task pass: 86.67%
170
+ ```
171
+
172
+ But it has **not yet** been run directly on official VerilogEval or RTLLM. Fair SOTA comparison requires running all candidate models with the same:
173
+
174
+ ```text
175
+ benchmark version
176
+ prompt formatting
177
+ sampling temperature / k
178
+ max tokens
179
+ code extraction rules
180
+ iverilog/vvp version
181
+ timeout
182
+ contamination policy
183
+ ```
184
+
185
+ Next recommended step: run this adapter on official VerilogEval and RTLLM harnesses, then replace the context charts with direct apples-to-apples results.