Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,21 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-fp16.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q2_k.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q3_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q5_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q6_k.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-fp16.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q2_k.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q3_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q5_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q6_k.gguf filter=lfs diff=lfs merge=lfs -text
+internlm2_5-7b-chat-1m-q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,106 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- chat
+---
+# InternLM2.5-7B-Chat-1M GGUF Model
+
+## Introduction
+
+The `internlm2_5-7b-chat-1m` model in GGUF format can be utilized by [llama.cpp](https://github.com/ggerganov/llama.cpp), a highly popular open-source framework for Large Language Model (LLM) inference, across a variety of hardware platforms, both locally and in the cloud.
+This repository offers `internlm2_5-7b-chat-1m` models in GGUF format in both half precision and various low-bit quantized versions, including `q5_0`, `q5_k_m`, `q6_k`, and `q8_0`.
+
+In the subsequent sections, we will first present the installation procedure, followed by an explanation of the model download process. 
+And finally we will illustrate the methods for model inference and service deployment through specific examples.
+
+## Installation
+
+We recommend building `llama.cpp` from source. The following code snippet provides an example for the Linux CUDA platform. For instructions on other platforms, please refer to the [official guide](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#build).
+
+- Step 1: create a conda environment and install cmake
+
+```shell
+conda create --name internlm2 python=3.10 -y
+conda activate internlm2
+pip install cmake
+```
+
+- Step 2: clone the source code and build the project 
+
+```shell
+git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
+cd llama.cpp
+cmake -B build -DGGML_CUDA=ON
+cmake --build build --config Release -j
+```
+
+All the built targets can be found in the sub directory `build/bin`
+
+In the following sections, we assume that the working directory is at the root directory of `llama.cpp`.
+
+## Download models
+
+In the [introduction section](#introduction), we mentioned that this repository includes several models with varying levels of computational precision. You can download the appropriate model based on your requirements.
+For instance, `internlm2_5-7b-chat-1m-fp16.gguf` can be downloaded as below：
+
+```shell
+pip install huggingface-hub
+huggingface-cli download internlm/internlm2_5-7b-chat-1m-gguf internlm2_5-7b-chat-1m-fp16.gguf --local-dir . --local-dir-use-symlinks False
+```
+
+## Inference
+
+You can use `llama-cli` for conducting inference. For a detailed explanation of `llama-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
+
+```shell
+build/bin/llama-cli \
+    --model internlm2_5-7b-chat-1m-fp16.gguf  \
+    --predict 512 \
+    --ctx-size 4096 \
+    --gpu-layers 32 \
+    --temp 0.8 \
+    --top-p 0.8 \
+    --top-k 50 \
+    --seed 1024 \
+    --color \
+    --prompt "<|im_start|>system\nYou are an AI assistant whose name is InternLM (书生·浦语).\n- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory (上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n- InternLM (书生·浦语) can understand and communicate fluently in the language chosen by the user such as English and 中文.<|im_end|>\n" \
+    --interactive \
+    --multiline-input \
+    --conversation \
+    --verbose \
+    --logdir workdir/logdir \
+    --in-prefix "<|im_start|>user\n" \
+    --in-suffix "<|im_end|>\n<|im_start|>assistant\n"
+```
+
+## Serving
+
+`llama.cpp` provides an OpenAI API compatible server - `llama-server`. You can deploy `internlm2_5-7b-chat-1m-fp16.gguf` into a service like this:
+
+```shell
+./build/bin/llama-server -m ./internlm2_5-7b-chat-1m-fp16.gguf -ngl 32
+```
+
+At the client side, you can access the service through OpenAI API:
+
+```python
+from openai import OpenAI
+client = OpenAI(
+    api_key='YOUR_API_KEY',
+    base_url='http://localhost:8080/v1'
+)
+model_name = client.models.list().data[0].id
+response = client.chat.completions.create(
+  model=model_name,
+  messages=[
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": " provide three suggestions about time management"},
+  ],
+  temperature=0.8,
+  top_p=0.8
+)
+print(response)
+```

internlm2_5-7b-chat-1m-fp16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07c1eb5406028d04d3bbf93d5bf53b13ff054a8832b272d0a00323c5fffac833
+size 15478092608

internlm2_5-7b-chat-1m-q2_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d898a1dd4810e506e8d05f312e0281ed33929c503230e1e4a35ee1e5fe24e655
+size 3005449024

internlm2_5-7b-chat-1m-q3_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1f05c3a442e4f03187136b4e235247280b4be70848ba53e4b88348a67e6c754
+size 3830379328

internlm2_5-7b-chat-1m-q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2c6912a6cd6e37faec1d333fa95e5272b06e3f911c315a1c588b46cc93362bf
+size 4453245760

internlm2_5-7b-chat-1m-q4_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5373bd21a01a76777d52b7929dc9440d6e7ac66d8f221d78117cfb8e9e871e2
+size 4712768320

internlm2_5-7b-chat-1m-q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cd9cf7f4768d0330ade772dd0dc819bc422a35ad39f21348e1c125078ff2895
+size 5373043520

internlm2_5-7b-chat-1m-q5_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce53338af367ef322e654f2c447a4855b9abe6d77ac9b0d65dee419ae68d6294
+size 5506736960

internlm2_5-7b-chat-1m-q6_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d5e92b27fb41bd7168d236379fc66589d93bc4bf237881950b65740f550c8c3
+size 6350328640

internlm2_5-7b-chat-1m-q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3be6827347870aaf46aec26ed8e0c5c533304ac0a00ea1d406cddad0ec151242
+size 8224240448