How to load a downloaded hugging face model in the docker script?

[saakethch]

the code seems to be referencing hf_hub but i do not wish to download the model each time. right now my code looks like this
def load(self):
access_token = "hf_rdS...."

    self._model = AutoModelForCausalLM.from_pretrained(
        "./mistralaiprivate/Mistral-7B-Instruct-v0.1",
        torch_dtype=torch.float16,
        device_map="auto",
        token=access_token
    )
    using this I'm deploying a docker image for this model

Could you please help me out here?

[AndiMajore]

Hey,
I intend to update the setup in the future in a way, that the model is 'saved' in a volume, so does not need re-downloading with each restart.
What you can do for now, and this is what I also do with this image, is 1. download the model from hugging-face into some directory on your system, 2. mounting this directory as a volume in the docker-compose.ym file, 3. creating a config.json file according to Xinference custom-model specs with this path as URL="file:///..." and 4. also mounting this file with the docker-compose.yml and then adjusting the "register" command in the register_and_launch.sh

You can see step 2. and 4. in the docker-config.yml of this project:

 volumes:
      - /cosybio/project/LLM/xinference_configs/:/cosybio/project/LLM/xinference_configs/
      - /cosybio/archive/old_projects/LLM/models/:/cosybio/archive/old_projects/LLM/models/

The config.json should look as follows:

{
  "version": 1,
  "context_length": 4096,
  "model_name": "llama2-13b-chat-hf",
  "model_lang": [
    "en"
  ],
  "model_ability": [
    "embed", "generate", "chat"
  ],
  "model_specs": [
    {
      "model_format": "pytorch",
      "model_size_in_billions": 13,
      "quantizations": [
        "none",
        "4-bit",
        "8-bit"
      ],
      "model_id": "",
# set this to your model directory
      "model_uri": "file:///$path_to_your_model_folder/llama2-13b-chat-hf"
    }
  ],
# adjust this part to your model
 "prompt_style": {
    "style_name": "LLAMA2",
      "system_prompt": "<s>[INST] <<SYS>># some system prompt in the style you need<</SYS>>\n\n" ,
      "roles": [ "[INST]",
        "[/INST]"
      ],
      "intra_message_sep": " ",
      "inter_message_sep": " </s><s>",
      "stop_token_ids": [
        2
      ],
      "stop": [
        "</s>"
      ]
   }
}

I hope that helps you setting up your Mistral docker container :)