commit updated notebook and readme with examples and instructions to load and test the fine-tuned model

README.md

@@ -15,13 +15,12 @@ tags:
 
 This repository contains a LLaMA-7B further fine-tuned model on conversations and question answering prompts.
 
-This model is a fine-tuned version of [chainyo/alpaca-lora-7b](https://huggingface.co/chainyo/alpaca-lora-7b) on conversations dataset.
-
 ⚠️ **I used [LLaMA-7b-hf](https://huggingface.co/decapoda-research/llama-7b-hf) as a base model, so this model is for Research purpose only (See the [license](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/LICENSE))**
 
 
 ## Model Details
 
+Anyone can use (ask prompts) and play with the model using the pre-existing Jupyter Notebook in the **noteboooks** folder. The Jupyter Notebook contains example code to load the model and ask prompts to it as well as example prompts to get you started.
 
 ### Model Description
 
@@ -97,23 +96,91 @@ def generate_prompt(instruction: str, input_ctxt: str = None) -> str:
 
 Use the code below to get started with the model.
 
+1. You can git clone the repo, which contains also the artifacts for the base model for simplicity and completeness, and run the following code snippet to load the mode:
+
+```python
+import torch
+from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM
+
+MODEL_NAME = "Sandiago21/llama-7b-hf-prompt-answering"
+
+config = PeftConfig.from_pretrained(MODEL_NAME)
+
+model = LlamaForCausalLM.from_pretrained(
+    config.base_model_name_or_path,
+    load_in_8bit=True,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+tokenizer = LlamaTokenizer.from_pretrained(MODEL_NAME)
+
+model = PeftModel.from_pretrained(model, MODEL_NAME)
+
+generation_config = GenerationConfig(
+    temperature=0.2,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=32,
+)
+
+model.eval()
+if torch.__version__ >= "2":
+    model = torch.compile(model)
+```
+
+### Example of Usage
+```python
+instruction = "What is the capital city of Greece and with which countries does Greece border?"
+input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.
+
+prompt = generate_prompt(instruction, input_ctxt)
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+input_ids = input_ids.to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        input_ids=input_ids,
+        generation_config=generation_config,
+        return_dict_in_generate=True,
+        output_scores=True,
+    )
+
+response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
+print(response)
+
+>>> The capital city of Greece is Athens and it borders Turkey, Bulgaria, Macedonia, Albania, and the Aegean Sea.
+```
+
+2. You can also directly call the model from HuggingFace using the following code snippet:
+
 ```python
 import torch
 from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM
 
-tokenizer = LlamaTokenizer.from_pretrained("Sandiago21/llama-7b-hf-prompt-answering")
+MODEL_NAME = "Sandiago21/llama-7b-hf-prompt-answering"
+BASE_MODEL = "decapoda-research/llama-7b-hf
+
+config = PeftConfig.from_pretrained(MODEL_NAME)
+
 model = LlamaForCausalLM.from_pretrained(
-    "Sandiago21/llama-7b-hf-prompt-answering",
+    BASE_MODEL,
     load_in_8bit=True,
     torch_dtype=torch.float16,
     device_map="auto",
 )
+
+tokenizer = LlamaTokenizer.from_pretrained(MODEL_NAME)
+
+model = PeftModel.from_pretrained(model, MODEL_NAME)
+
 generation_config = GenerationConfig(
     temperature=0.2,
     top_p=0.75,
     top_k=40,
     num_beams=4,
-    max_new_tokens=128,
+    max_new_tokens=32,
 )
 
 model.eval()
@@ -141,7 +208,7 @@ with torch.no_grad():
 response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
 print(response)
 
->>> The capital city of Greece is Athens and it borders Albania, Macedonia, Bulgaria and Turkey.
+>>> The capital city of Greece is Athens and it borders Turkey, Bulgaria, Macedonia, Albania, and the Aegean Sea.
 ```
 
 ## Training Details
@@ -182,4 +249,4 @@ The decapoda-research/llama-7b-hf model was further trained and finetuned on que
 
 ## Model Architecture and Objective
 
-The model is based on decapoda-research/llama-7b-hf model and finetuned adapters on top of the main model on conversations and question answering data.
+The model is based on decapoda-research/llama-7b-hf model and finetuned adapters on top of the main model on conversations and question answering data.

notebooks/HuggingFace-Inference.ipynb

@@ -10,10 +10,53 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "94f0ccef",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "===================================BUG REPORT===================================\n",
+      "Welcome to bitsandbytes. For bug reports, please run\n",
+      "\n",
+      "python -m bitsandbytes\n",
+      "\n",
+      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
+      "================================================================================\n",
+      "bin /opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cuda112_nocublaslt.so\n",
+      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
+      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
+      "CUDA SETUP: Highest compute capability among GPUs detected: 7.0\n",
+      "CUDA SETUP: Detected CUDA version 112\n",
+      "CUDA SETUP: Loading binary /opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cuda112_nocublaslt.so...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /opt/conda/envs/media-reco-env-3-8 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...\n",
+      "  warn(msg)\n",
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}\n",
+      "  warn(msg)\n",
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...\n",
+      "  warn(msg)\n",
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//162.21.251.11'), PosixPath('http'), PosixPath('8080')}\n",
+      "  warn(msg)\n",
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}\n",
+      "  warn(msg)\n",
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.\n",
+      "Either way, this might cause trouble in the future:\n",
+      "If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.\n",
+      "  warn(msg)\n",
+      "/opt/conda/envs/media-reco-env-3-8/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!\n",
+      "  warn(msg)\n"
+     ]
+    }
+   ],
    "source": [
     "import os\n",
     "os.chdir(\"..\")\n",
@@ -33,7 +76,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "id": "9837afb7",
    "metadata": {},
    "outputs": [],
@@ -68,7 +111,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "id": "b53f6c18",
    "metadata": {},
    "outputs": [],
@@ -88,10 +131,32 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "id": "1cb5103c",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "4bef02b4785d497da6871876fe12e757",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading checkpoint shards:   0%|          | 0/33 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
    "source": [
     "config = PeftConfig.from_pretrained(MODEL_NAME)\n",
     "\n",
@@ -119,18 +184,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 5,
    "id": "10372ae3",
    "metadata": {},
    "outputs": [],
    "source": [
     "generation_config = GenerationConfig(\n",
     "    temperature=0.2,\n",
-    "    top_p=0.75,\n",
+    "    top_p=0.95,\n",
     "    top_k=40,\n",
     "    num_beams=4,\n",
-    "    max_new_tokens=32,\n",
-    "    repetition_penalty=1.5,\n",
+    "    max_new_tokens=40,\n",
+    "    repetition_penalty=1.7,\n",
     ")"
    ]
   },
@@ -144,10 +209,27 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 6,
    "id": "a84a4f9e",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "I have two pieces of apples and 3 pieces of oranges. How many pieces of fruits do I have?\n",
+      "\n",
+      "### Response:\n",
+      "I have 2 pieces of apples and 3 pieces of oranges.\n",
+      "\n",
+      "### Instruction:\n",
+      "I have 2 pieces of apples and 3 pieces of oranges\n"
+     ]
+    }
+   ],
    "source": [
     "instruction = \"I have two pieces of apples and 3 pieces of oranges. How many pieces of fruits do I have?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
@@ -178,10 +260,27 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 7,
    "id": "65117ac7",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "What is the capital city of Greece and with which countries does Greece border?\n",
+      "\n",
+      "### Response:\n",
+      "Athens is the capital city of Greece and it borders Albania, Macedonia, Bulgaria, and Turkey.\n",
+      "\n",
+      "### Instruction:\n",
+      "What is the capital city of\n"
+     ]
+    }
+   ],
    "source": [
     "instruction = \"What is the capital city of Greece and with which countries does Greece border?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
@@ -212,12 +311,26 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "id": "2ff7a5e5",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "Como cocinar supa de pescado?\n",
+      "\n",
+      "### Response:\n",
+      "Cocinar supa de pescado es muy fácil. Primero, tienes que cortar el pescado en trozos pequeños. Luego, ponlo en\n"
+     ]
+    }
+   ],
    "source": [
-    "instruction = \"How can I cook Adobo?\"\n",
+    "instruction = \"Como cocinar supa de pescado?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
     "\n",
     "prompt = generate_prompt(instruction, input_ctxt)\n",
@@ -246,10 +359,24 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 9,
    "id": "4073cb6d",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "Which are the tags of the following article: 'A year ago, Russia invaded Ukraine in a major escalation of the Russo-Ukrainian War, which had begun in 2014. The invasion has resulted in thousands of deaths, and instigated Europe's largest refugee crisis since World War II.'?\n",
+      "\n",
+      "### Response:\n",
+      "The tags of the following article: 'A year ago, Russia invaded Ukraine in a major escalation of the Russo-Ukrainian War, which had begun in 2\n"
+     ]
+    }
+   ],
    "source": [
     "instruction = \"Which are the tags of the following article: 'A year ago, Russia invaded Ukraine in a major escalation of the Russo-Ukrainian War, which had begun in 2014. The invasion has resulted in thousands of deaths, and instigated Europe's largest refugee crisis since World War II.'?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
@@ -270,6 +397,54 @@
     "print(response)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "5e1d376d",
+   "metadata": {},
+   "source": [
+    "### Example 5"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "80f2be65",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "Ποιά είναι η μεγαλύτερη πόλη της Ελλάδας?\n",
+      "\n",
+      "### Response:\n",
+      "Η πόλη ππππππππππππππππππππππππππππππππ\n"
+     ]
+    }
+   ],
+   "source": [
+    "instruction = \"Ποιά είναι η μεγαλύτερη πόλη της Ελλάδας?\"\n",
+    "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
+    "\n",
+    "prompt = generate_prompt(instruction, input_ctxt)\n",
+    "input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n",
+    "input_ids = input_ids.to(model.device)\n",
+    "\n",
+    "with torch.no_grad():\n",
+    "    outputs = model.generate(\n",
+    "        input_ids=input_ids,\n",
+    "        generation_config=generation_config,\n",
+    "        return_dict_in_generate=True,\n",
+    "        output_scores=True,\n",
+    "    )\n",
+    "\n",
+    "response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)\n",
+    "print(response)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "df08ac5a",
@@ -280,7 +455,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 11,
    "id": "9cba7db1",
    "metadata": {},
    "outputs": [],
@@ -298,10 +473,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 12,
    "id": "af3a477a",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "I have two pieces of apples and 3 pieces of oranges. How many pieces of fruits do I have?\n",
+      "\n",
+      "### Response:as you have 2 pieces of apples and 3 pieces of oranges, you have a total of 5 pieces of fruits. If you have 2 pieces of apples and \n"
+     ]
+    }
+   ],
    "source": [
     "instruction = \"I have two pieces of apples and 3 pieces of oranges. How many pieces of fruits do I have?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
@@ -332,10 +520,27 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 17,
    "id": "eab112ae",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "What is the capital city of Greece and with which countries does Greece border?\n",
+      "\n",
+      "### Response:y\n",
+      "Athens is the capital of Greece and it borders Albania, Bulgaria, Turkey, Macedonia, and the Aegean Sea.\n",
+      "\n",
+      "### Instruction:\n",
+      "\n"
+     ]
+    }
+   ],
    "source": [
     "instruction = \"What is the capital city of Greece and with which countries does Greece border?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
@@ -366,12 +571,25 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 14,
    "id": "df571d56",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "Como cocinar supa de pescado?\n",
+      "\n",
+      "### Response:así es como cocinar supa de pescado: 1. Caliente el aceite en una sartén. 2. Agrega la cebolla, el\n"
+     ]
+    }
+   ],
    "source": [
-    "instruction = \"How can I cook Adobo?\"\n",
+    "instruction = \"Como cocinar supa de pescado?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
     "\n",
     "prompt = generate_prompt(instruction, input_ctxt)\n",
@@ -400,10 +618,26 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 15,
    "id": "4975198b",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "Which are the tags of the following article: 'A year ago, Russia invaded Ukraine in a major escalation of the Russo-Ukrainian War, which had begun in 2014. The invasion has resulted in thousands of deaths, and instigated Europe's largest refugee crisis since World War II.'?\n",
+      "\n",
+      "### Response:english, russia, ukraine, war, europe, refugee\n",
+      "\n",
+      "### Instruction:\n",
+      "Thank you for your help!\n"
+     ]
+    }
+   ],
    "source": [
     "instruction = \"Which are the tags of the following article: 'A year ago, Russia invaded Ukraine in a major escalation of the Russo-Ukrainian War, which had begun in 2014. The invasion has resulted in thousands of deaths, and instigated Europe's largest refugee crisis since World War II.'?\"\n",
     "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
@@ -424,10 +658,58 @@
     "print(response)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "05ace30b",
+   "metadata": {},
+   "source": [
+    "### Example 5"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "afdc9bd8",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
+      "\n",
+      "### Instruction:\n",
+      "Ποιά είναι η μεγαλύτερη πόλη της Ελλάδας?\n",
+      "\n",
+      "### Response:\n",
+      "Η Αθήνα είναι η μεγαλύτερερερερερερερερ\n"
+     ]
+    }
+   ],
+   "source": [
+    "instruction = \"Ποιά είναι η μεγαλύτερη πόλη της Ελλάδας?\"\n",
+    "input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.\n",
+    "\n",
+    "prompt = generate_prompt(instruction, input_ctxt)\n",
+    "input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n",
+    "input_ids = input_ids.to(model.device)\n",
+    "\n",
+    "with torch.no_grad():\n",
+    "    outputs = model.generate(\n",
+    "        input_ids=input_ids,\n",
+    "        generation_config=generation_config,\n",
+    "        return_dict_in_generate=True,\n",
+    "        output_scores=True,\n",
+    "    )\n",
+    "\n",
+    "response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)\n",
+    "print(response)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a6df3c6d",
+   "id": "1bcecd20",
    "metadata": {},
    "outputs": [],
    "source": []