Instructions to use Pablo-Flores-Mollinedo/verilog-qwen2.5-coder-7b-v4-1-seq-bit-fsm-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Pablo-Flores-Mollinedo/verilog-qwen2.5-coder-7b-v4-1-seq-bit-fsm-lora with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct") model = PeftModel.from_pretrained(base_model, "Pablo-Flores-Mollinedo/verilog-qwen2.5-coder-7b-v4-1-seq-bit-fsm-lora") - Notebooks
- Google Colab
- Kaggle
Add external SOTA context charts
Browse files
README.md
CHANGED
|
@@ -123,3 +123,63 @@ Typical reporting:
|
|
| 123 |
- category breakdown for combinational, arithmetic, sequential, FSM, memory, and larger RTL designs
|
| 124 |
|
| 125 |
This adapter has been evaluated on the full internal 30-task suite with both pass@1 and pass@5. Official external SOTA comparison should next run VerilogEval and RTLLM directly with the same decoding settings.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
- category breakdown for combinational, arithmetic, sequential, FSM, memory, and larger RTL designs
|
| 124 |
|
| 125 |
This adapter has been evaluated on the full internal 30-task suite with both pass@1 and pass@5. Official external SOTA comparison should next run VerilogEval and RTLLM directly with the same decoding settings.
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
## External SOTA context
|
| 129 |
+
|
| 130 |
+
> Important: the following charts are **context only**, not official leaderboard placement. External results come from the public Chip Design LLM Zoo table, while this adapter's score is from this repository's internal 30-task paper-style suite. Benchmarks, prompts, decoding settings, and contamination controls differ.
|
| 131 |
+
|
| 132 |
+
Sources:
|
| 133 |
+
|
| 134 |
+
- Chip Design LLM Zoo reports VerilogEval, VerilogEval v2, RTLLM, and other RTL benchmark results and ranks models by VerilogEval pass@1 / RTLLM correct rate: https://iprc-dip.github.io/Chip-Design-LLM-Zoo/
|
| 135 |
+
- NVIDIA's VerilogEval repository describes the VerilogEval harness and dataset: https://github.com/NVlabs/verilog-eval
|
| 136 |
+
- The VerilogEval paper page describes a 156-problem Verilog generation benchmark: https://research.nvidia.com/publication/2023-09_verilogeval-evaluating-large-language-models-verilog-code-generation
|
| 137 |
+
- RTL-Coder is an RTL-code-generation fine-tuning project with RTLCoder-Mistral inference scripts and synthetic RTL data flow: https://github.com/hkust-zhiyao/RTL-Coder
|
| 138 |
+
|
| 139 |
+
### Reported top performers vs this adapter
|
| 140 |
+
|
| 141 |
+

|
| 142 |
+
|
| 143 |
+
### Reported fine-tuned/open Verilog models
|
| 144 |
+
|
| 145 |
+

|
| 146 |
+
|
| 147 |
+
Representative reported models found in the public tables/search results include:
|
| 148 |
+
|
| 149 |
+
| Model / method | Notes |
|
| 150 |
+
|---|---|
|
| 151 |
+
| ScaleRTL-32B / ScaleRTL†-32B | fine-tuned RTL model, strong VerilogEval reported scores |
|
| 152 |
+
| ChipSeek-R1 | fine-tuned 7B RTL model, strong VerilogEval-Machine reported score |
|
| 153 |
+
| CodeV-CodeLlama / CodeV-DeepSeek / CodeV-CodeQwen | fine-tuned 6.7B/7B RTL/code models |
|
| 154 |
+
| DecoRTL-CodeV | RTL-specific model reported on VerilogEval-Human |
|
| 155 |
+
| VeriReason-Qwen2.5-7B | fine-tuned Qwen2.5 7B RTL model |
|
| 156 |
+
| RTL++ @ 200K Trained | fine-tuned 7B RTL model |
|
| 157 |
+
| RTLCoder-Mistral | RTL-Coder project model family |
|
| 158 |
+
| qwen3-32b-verilog-lora | Hugging Face Verilog LoRA adapter |
|
| 159 |
+
|
| 160 |
+
### Why this comparison is cautious
|
| 161 |
+
|
| 162 |
+
The current v4.1 adapter has been tested thoroughly on this repository's full internal suite:
|
| 163 |
+
|
| 164 |
+
```text
|
| 165 |
+
30 tasks total
|
| 166 |
+
22 functional simulation tasks
|
| 167 |
+
8 compile-only / RTL-heavy tasks
|
| 168 |
+
pass@1 task pass: 83.33%
|
| 169 |
+
pass@5 task pass: 86.67%
|
| 170 |
+
```
|
| 171 |
+
|
| 172 |
+
But it has **not yet** been run directly on official VerilogEval or RTLLM. Fair SOTA comparison requires running all candidate models with the same:
|
| 173 |
+
|
| 174 |
+
```text
|
| 175 |
+
benchmark version
|
| 176 |
+
prompt formatting
|
| 177 |
+
sampling temperature / k
|
| 178 |
+
max tokens
|
| 179 |
+
code extraction rules
|
| 180 |
+
iverilog/vvp version
|
| 181 |
+
timeout
|
| 182 |
+
contamination policy
|
| 183 |
+
```
|
| 184 |
+
|
| 185 |
+
Next recommended step: run this adapter on official VerilogEval and RTLLM harnesses, then replace the context charts with direct apples-to-apples results.
|