Instructions to use Sandiago21/llama-7b-hf-prompt-answering with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Sandiago21/llama-7b-hf-prompt-answering with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Sandiago21/llama-7b-hf-prompt-answering")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Sandiago21/llama-7b-hf-prompt-answering")
model = AutoModelForCausalLM.from_pretrained("Sandiago21/llama-7b-hf-prompt-answering", device_map="auto")

PEFT
How to use Sandiago21/llama-7b-hf-prompt-answering with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Sandiago21/llama-7b-hf-prompt-answering with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Sandiago21/llama-7b-hf-prompt-answering"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Sandiago21/llama-7b-hf-prompt-answering",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Sandiago21/llama-7b-hf-prompt-answering

SGLang

How to use Sandiago21/llama-7b-hf-prompt-answering with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Sandiago21/llama-7b-hf-prompt-answering" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Sandiago21/llama-7b-hf-prompt-answering",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Sandiago21/llama-7b-hf-prompt-answering" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Sandiago21/llama-7b-hf-prompt-answering",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Sandiago21/llama-7b-hf-prompt-answering with Docker Model Runner:
```
docker model run hf.co/Sandiago21/llama-7b-hf-prompt-answering
```

Sandiago21 commited on Jun 5, 2023

Commit

2f7732f

1 Parent(s): c709ea2

Update README.md with improved way to load and use the model

Browse files

Files changed (1) hide show

README.md +35 -34

README.md CHANGED Viewed

@@ -98,49 +98,50 @@ def generate_prompt(instruction: str, input_ctxt: str = None) -> str:
 Use the code below to get started with the model.
 ```python
-from transformers import LlamaTokenizer, LlamaForCausalLM
-from peft import PeftModel
-MODEL_NAME = "decapoda-research/llama-7b-hf"
-tokenizer = LlamaTokenizer.from_pretrained(MODEL_NAME, add_eos_token=True)
-tokenizer.pad_token_id = 0
-model = LlamaForCausalLM.from_pretrained(MODEL_NAME, load_in_8bit=True, device_map="auto")
-model = PeftModel.from_pretrained(model, "Sandiago21/llama-7b-hf")
 ```
 ### Example of Usage
 ```python
-from transformers import GenerationConfig
-PROMPT = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nWhich is the capital city of Greece and with which countries does Greece border?\n\n### Input:\nQuestion answering\n\n### Response:\n"""
-DEVICE = "cuda"
-inputs = tokenizer(
-    PROMPT,
-    return_tensors="pt",
-)
-input_ids = inputs["input_ids"].to(DEVICE)
-generation_config = GenerationConfig(
-    temperature=0.1,
-    top_p=0.95,
-    repetition_penalty=1.2,
-)
-print("Generating Response ... ")
-with torch.no_grad():
-  generation_output = model.generate(
-      input_ids=input_ids,
-      generation_config=generation_config,
-      return_dict_in_generate=True,
-      output_scores=True,
-      max_new_tokens=256,
-  )
-for s in generation_output.sequences:
-    print(tokenizer.decode(s))
 ```
 ### Example Output

 Use the code below to get started with the model.
 ```python
+import torch
+from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM
+tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")
+model = LlamaForCausalLM.from_pretrained(
+    "chainyo/alpaca-lora-7b",
+    load_in_8bit=True,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+generation_config = GenerationConfig(
+    temperature=0.2,
+    top_p=0.75,
+    top_k=40,
+    num_beams=4,
+    max_new_tokens=128,
+)
+model.eval()
+if torch.__version__ >= "2":
+    model = torch.compile(model)
 ```
 ### Example of Usage
 ```python
+instruction = "What is the capital city of Greece and with which countries does Greece border?"
+input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.
+prompt = generate_prompt(instruction, input_ctxt)
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+input_ids = input_ids.to(model.device)
+with torch.no_grad():
+    outputs = model.generate(
+        input_ids=input_ids,
+        generation_config=generation_config,
+        return_dict_in_generate=True,
+        output_scores=True,
+    )
+response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
+print(response)
+>>> The capital city of Greece is Athens and it borders Albania, Macedonia, Bulgaria and Turkey.
 ```
 ### Example Output