metadata
language:
- en
license: cc-by-4.0
library_name: transformers
tags:
- mergekit
- merge
- mistral
- text-generation
- roleplay
base_model:
- SanjiWatsuki/Kunoichi-7B
- SanjiWatsuki/Silicon-Maid-7B
- KatyTheCutie/LemonadeRP-4.5.3
- Sao10K/Fimbulvetr-11B-v2
model-index:
- name: Chewy-Lemon-Cookie-11B
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 48.75
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=FallenMerick/Chewy-Lemon-Cookie-11B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 33.01
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=FallenMerick/Chewy-Lemon-Cookie-11B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 4.61
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=FallenMerick/Chewy-Lemon-Cookie-11B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 3.91
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=FallenMerick/Chewy-Lemon-Cookie-11B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 15.95
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=FallenMerick/Chewy-Lemon-Cookie-11B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 25.19
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=FallenMerick/Chewy-Lemon-Cookie-11B
name: Open LLM Leaderboard
Chewy-Lemon-Cookie-11B
This is a merge of pre-trained language models created using mergekit.
GGUF quants:
- https://huggingface.co/backyardai/Chewy-Lemon-Cookie-11B-GGUF
- https://huggingface.co/mradermacher/Chewy-Lemon-Cookie-11B-GGUF
- https://huggingface.co/mradermacher/Chewy-Lemon-Cookie-11B-i1-GGUF
Merge Details
Merge Method
This model was merged using the following methods:
- passthrough
- task arithmetic
Models Merged
The following models were included in the merge:
- SanjiWatsuki/Kunoichi-7B
- SanjiWatsuki/Silicon-Maid-7B
- KatyTheCutie/LemonadeRP-4.5.3
- Sao10K/Fimbulvetr-11B-v2
Configuration
The following YAML configurations were used to produce this model:
slices:
- sources:
- model: SanjiWatsuki/Kunoichi-7B
layer_range: [0, 24]
- sources:
- model: SanjiWatsuki/Silicon-Maid-7B
layer_range: [8, 24]
- sources:
- model: KatyTheCutie/LemonadeRP-4.5.3
layer_range: [24, 32]
merge_method: passthrough
dtype: bfloat16
name: Big-Lemon-Cookie-11B-BF16
---
models:
- model: Big-Lemon-Cookie-11B-BF16
parameters:
weight: 0.85
- model: Sao10K/Fimbulvetr-11B-v2
parameters:
weight: 0.15
merge_method: task_arithmetic
base_model: Big-Lemon-Cookie-11B-BF16
dtype: bfloat16
name: Chewy-Lemon-Cookie-11B
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 21.91 |
| IFEval (0-Shot) | 48.75 |
| BBH (3-Shot) | 33.01 |
| MATH Lvl 5 (4-Shot) | 4.61 |
| GPQA (0-shot) | 3.91 |
| MuSR (0-shot) | 15.95 |
| MMLU-PRO (5-shot) | 25.19 |