Mistral-7B-DadJokes-GGUF

This is a Mistral-7B-v0.1 model fine-tuned on Reddit-Dad-Jokes dataset in a GGUF format. It can generate cringe and toxic jokes on any short prompt:

[INST] What horse says when she sees an old lady? [/INST] Move over grandma, I'm going to trot!
[INST] What is a lawyer's favorite drink? [/INST] Whine.
[INST] My obese pet parrot died over the weekend. [/INST] Now I just have a big fat birdcage.
[INST] My wife asked why there was a blanket on top of me and my laptop. [/INST] I told her that I was just covering all my bases.
[INST] My girlfriend changed after she became a vegetarian. [/INST] She finally lost her meat.

Used data

We use a Kaggle Reddit Dad Jokes dataset formatted in a base+punchline tuples. The model task was to predict the punchline given the base. Prompt format is the same as for original Mistral-7B-0.1 model:

[INST] base [/INST] punchline

Fine-tuning process

The model was fine-tuned with QLORA using the LLM_QLORA trainer script with the following configuration:

base_model: mistralai/Mistral-7B-v0.1
model_family: llama  # if unspecified will use AutoModelForCausalLM/AutoTokenizer
model_context_window: 256  # if unspecified will use tokenizer.model_max_length
data:
  type: dadjoke
  train: "dadjokes/dataset/train.csv"
  eval: "dadjokes/dataset/test.csv" 
lora:
  r: 8
  lora_alpha: 32
  target_modules:  # modules for which to train lora adapters
  - q_proj
  - k_proj
  - v_proj
  lora_dropout: 0.05
  bias: none
  task_type: CAUSAL_LM
trainer:
  batch_size: 8
  gradient_accumulation_steps: 1
  warmup_steps: 100
  num_train_epochs: 1
  learning_rate: 0.0002  # 2e-4
  logging_steps: 20
trainer_output_dir: trainer_outputs/
model_output_dir: models/

Fine-tuning took ~70 minutes on a single RTX 4090.

Running the model locally

This model can be run with a llama-cpp on a CPU using the following command:

./main -n 64 -m models/ggml-model-q4_0.gguf -p "[INST] My girlfriend changed after she became a vegetarian. [/INST]"

system_info: n_threads = 8 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | 
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 64, n_keep = 0


 [INST] My girlfriend changed after she became a vegetarian. [/INST] She finally lost her meat [end of text]

llama_print_timings:        load time =   439.38 ms
llama_print_timings:      sample time =     4.62 ms /     6 runs   (    0.77 ms per token,  1298.98 tokens per second)
llama_print_timings: prompt eval time =  1786.76 ms /    18 tokens (   99.26 ms per token,    10.07 tokens per second)
llama_print_timings:        eval time =   833.66 ms /     5 runs   (  166.73 ms per token,     6.00 tokens per second)
llama_print_timings:       total time =  2627.55 ms
Log end

License

Apache 2.0