{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "SageMaker Deploy Mixtral",
  "keywords": "Mixtral,SageMaker,MoE",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/sagemaker-deploy-mixtral",
  "publish_date": "2023-12",
  "corpus_stage": "推理部署",
  "notes": "在SageMaker部署Mixtral",
  "id": "997fa9190a3c",
  "collected_at": "2026-10-02T14:29:19",
  "title": "Deploy Mixtral 8x7B on Amazon SageMaker",
  "body_markdown": "Deploy Mixtral 8x7B on Amazon SageMaker\nDecember 12, 2023\n9\nminute read\nView Code\nMixtral 8x7B\nis the open LLM from\nMistral AI\n. The Mixtral-8x7B is a Sparse Mixture of Experts it has a similar architecture to Mistral 7B, but comes with a twist: it’s actually 8 “expert” models in one. If you want to learn more about MoEs check out\nMixture of Experts Explained\n.\nIn this blog you will learn how to deploy\nmistralai/Mixtral-8x7B-Instruct-v0.1\nmodel to Amazon SageMaker. We are going to use the Hugging Face LLM DLC is a new purpose-built Inference Container to easily deploy LLMs in a secure and managed environment. The DLC is powered by\nText Generation Inference (TGI)\na scalelable, optimized solution for deploying and serving Large Language Models (LLMs). The Blog post also includes Hardware requirements for the different model sizes.\nIn the blog will cover how to:\nSetup development environment\nRetrieve the new Hugging Face LLM DLC\nHardware requirements\nDeploy Mixtral 8x7B to Amazon SageMaker\nRun inference and chat with the model\nClean up\nLets get started!\n1. Setup development environment\nWe are going to use the\nsagemaker\npython SDK to deploy Mixtral to Amazon SageMaker. We need to make sure to have an AWS account configured and the\nsagemaker\npython SDK installed.\nPython\n#\nTODO\n: update when container is added to sagemaker sdk\n!\npip install\n\"sagemaker>=2.199.0\"\n--\nupgrade\n--\nquiet\nIf you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find\nhere\nmore about it.\nPython\nimport\nsagemaker\nimport\nboto3\nsess = sagemaker.\nSession\n()\n# sagemaker session bucket -> used for uploading data, models and logs\n# sagemaker will automatically create this bucket if it not exists\nsagemaker_session_bucket=\nNone\nif\nsagemaker_session_bucket\nis\nNone\nand\nsess\nis\nnot\nNone\n:\n# set to default bucket if a bucket name is not given\nsagemaker_session_bucket = sess.\ndefault_bucket\n()\ntry\n:\nrole = sagemaker.\nget_execution_role\n()\nexcept\nValueError:\niam = boto3.\nclient\n(\n'iam'\n)\nrole = iam.\nget_role\n(\nRoleName\n=\n'sagemaker_execution_role'\n)[\n'Role'\n][\n'Arn'\n]\nsess = sagemaker.\nSession\n(\ndefault_bucket\n=sagemaker_session_bucket)\nprint(\nf\n\"sagemaker role arn:\n{\nrole\n}\n\"\n)\nprint(\nf\n\"sagemaker session region:\n{\nsess.boto_region_name\n}\n\"\n)\n2. Retrieve the new Hugging Face LLM DLC\nCompared to deploying regular Hugging Face models we first need to retrieve the container uri and provide it to our\nHuggingFaceModel\nmodel class with a\nimage_uri\npointing to the image. To retrieve the new Hugging Face LLM DLC in Amazon SageMaker, we can use the\nget_huggingface_llm_image_uri\nmethod provided by the\nsagemaker\nSDK. This method allows us to retrieve the URI for the desired Hugging Face LLM DLC based on the specified\nbackend\n,\nsession\n,\nregion\n, and\nversion\n. You can find the available versions\nhere\nNote: At the time of writing this blog post the latest version of the Hugging Face LLM DLC is not yet available via the\nget_huggingface_llm_image_uri\nmethod. We are going to use the raw container uri instead.\nPython\n# COMMENT IN WHEN PR (https://github.com/aws/sagemaker-python-sdk/pull/4314) IS MERGED\n# from sagemaker.huggingface import get_huggingface_llm_image_uri\n# # retrieve the llm image uri\n# llm_image = get_huggingface_llm_image_uri(\n#   \"huggingface\",\n#   version=\"1.3.1\"\n# )\nregion_mapping = {\n\"af-south-1\"\n:\n\"626614931356\"\n,\n\"il-central-1\"\n:\n\"780543022126\"\n,\n\"ap-east-1\"\n:\n\"871362719292\"\n,\n\"ap-northeast-1\"\n:\n\"763104351884\"\n,\n\"ap-northeast-2\"\n:\n\"763104351884\"\n,\n\"ap-northeast-3\"\n:\n\"364406365360\"\n,\n\"ap-south-1\"\n:\n\"763104351884\"\n,\n\"ap-south-2\"\n:\n\"772153158452\"\n,\n\"ap-southeast-1\"\n:\n\"763104351884\"\n,\n\"ap-southeast-2\"\n:\n\"763104351884\"\n,\n\"ap-southeast-3\"\n:\n\"907027046896\"\n,\n\"ap-southeast-4\"\n:\n\"457447274322\"\n,\n\"ca-central-1\"\n:\n\"763104351884\"\n,\n\"cn-north-1\"\n:\n\"727897471807\"\n,\n\"cn-northwest-1\"\n:\n\"727897471807\"\n,\n\"eu-central-1\"\n:\n\"763104351884\"\n,\n\"eu-central-2\"\n:\n\"380420809688\"\n,\n\"eu-north-1\"\n:\n\"763104351884\"\n,\n\"eu-west-1\"\n:\n\"763104351884\"\n,\n\"eu-west-2\"\n:\n\"763104351884\"\n,\n\"eu-west-3\"\n:\n\"763104351884\"\n,\n\"eu-south-1\"\n:\n\"692866216735\"\n,\n\"eu-south-2\"\n:\n\"503227376785\"\n,\n\"me-south-1\"\n:\n\"217643126080\"\n,\n\"me-central-1\"\n:\n\"914824155844\"\n,\n\"sa-east-1\"\n:\n\"763104351884\"\n,\n\"us-east-1\"\n:\n\"763104351884\"\n,\n\"us-east-2\"\n:\n\"763104351884\"\n,\n\"us-gov-east-1\"\n:\n\"446045086412\"\n,\n\"us-gov-west-1\"\n:\n\"442386744353\"\n,\n\"us-iso-east-1\"\n:\n\"886529160074\"\n,\n\"us-isob-east-1\"\n:\n\"094389454867\"\n,\n\"us-west-1\"\n:\n\"763104351884\"\n,\n\"us-west-2\"\n:\n\"763104351884\"\n,\n}\nllm_image =\nf\n\"\n{\nregion_mapping[sess.boto_region_name]\n}\n.dkr.ecr.\n{\nsess.boto_region_name\n}\n.amazonaws.com/huggingface-pytorch-tgi-inference:2.1.1-tgi1.3.1-gpu-py310-cu121-ubuntu20.04-v1.0\"\n# print ecr image uri\nprint(\nf\n\"llm image uri:\n{\nllm_image\n}\n\"\n)\n3. Hardware requirements\nMixtral 8x7B\nis 45B parameter Mixture-of-Experts (MoE) model. This means we’ll need to have enough VRAM to hold a dense 47B parameter model. Why 47B parameters and not 8 x 7B = 56B? That’s because in MoE models, only the FFN layers are treated as individual experts. This allows Mixtral 8x7B to achieve the latency of ~12B models by being 45B parameters.\nNote: The Hugging Face LLM DLC version 1.3.1 does not yet support quantization techniques with AWQ and GPTQ. We expect to support these techniques in the next releases. We will update this blog post once the new version is available.\nModel\nInstance Type\nQuantization\nNUM_GPUS\nMixtral 8x7B\n(ml.)p4d.24xlarge\n-\n8\nMixtral 8x7B\n(ml.)g5.48xlarge\n-\n8\nThe parameter tested for\nMAX_INPUT_LENGTH\n,\nMAX_BATCH_PREFILL_TOKENS\n,\nMAX_TOTAL_TOKENS\nand\nMAX_BATCH_TOTAL_TOKENS\nare the same for both instances. You can tweak them for your needs.\n4. Deploy Mixtral 8x7B to Amazon SageMaker\nTo deploy\nMixtral 8x7B\nto Amazon SageMaker we create a\nHuggingFaceModel\nmodel class and define our endpoint configuration including the\nhf_model_id\n,\ninstance_type\netc. We will use a\ng5.45xlarge\ninstance type, which has 8 NVIDIA A10G GPUs and 192GB of GPU memory. You need atleast > 100GB of GPU memory to run Mixtral 8x7B in float16 with decent input length.\nPython\nimport\njson\nfrom\nsagemaker.huggingface\nimport\nHuggingFaceModel\n# sagemaker config\ninstance_type =\n\"ml.g5.48xlarge\"\nnumber_of_gpu =\n8\nhealth_check_timeout =\n300\n# Define Model and Endpoint configuration parameter\nconfig = {\n'HF_MODEL_ID'\n:\n\"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n,\n# model_id from hf.co/models\n'SM_NUM_GPUS'\n: json.\ndumps\n(number_of_gpu),\n# Number of GPU used per replica\n'MAX_INPUT_LENGTH'\n: json.\ndumps\n(\n24000\n),\n# Max length of input text\n'MAX_BATCH_PREFILL_TOKENS'\n: json.\ndumps\n(\n32000\n),\n# Number of tokens for the prefill operation.\n'MAX_TOTAL_TOKENS'\n: json.\ndumps\n(\n32000\n),\n# Max length of the generation (including input text)\n'MAX_BATCH_TOTAL_TOKENS'\n: json.\ndumps\n(\n512000\n),\n# Limits the number of tokens that can be processed in parallel during the generation\n# ,'HF_MODEL_QUANTIZE': \"awq\", # comment in to quantize not supported yet\n}\n# create HuggingFaceModel with the image uri\nllm_model =\nHuggingFaceModel\n(\nrole\n=role,\nimage_uri\n=llm_image,\nenv\n=config\n)\nAfter we have created the\nHuggingFaceModel\nwe can deploy it to Amazon SageMaker using the\ndeploy\nmethod. We will deploy the model with the\nml.g5.48xlarge\ninstance type. TGI will automatically distribute and shard the model across all GPUs.\nPython\n# Deploy model to an endpoint\n# https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy\nllm = llm_model.\ndeploy\n(\ninitial_instance_count\n=\n1\n,\ninstance_type\n=instance_type,\ncontainer_startup_health_check_timeout\n=health_check_timeout,\n# 10 minutes to be able to load the model\n)\nSageMaker will now create our endpoint and deploy the model to it. This can takes a 10-15 minutes.\n5. Run inference and chat with the model\nAfter our endpoint is deployed we can run inference on it. We will use the\npredict\nmethod from the\npredictor\nto run inference on our endpoint. We can inference with different parameters to impact the generation. Parameters can be defined as in the\nparameters\nattribute of the payload. You can find supported parameters in the\nhere\nor in the open api specification of the TGI in the\nswagger documentation\nThe\nmistralai/Mixtral-8x7B-Instruct-v0.1\nis a conversational chat model meaning we can chat with it using the following prompt:\n<s> [INST] User Instruction 1 [/INST] Model answer 1</s> [INST] User instruction 2 [/INST]\nLets see, if Mixtral can come up with some cool ideas for the summer.\nPython\n# Prompt to generate\nprompt=\nf\n\"<s> [INST] What are cool ideas for the summer to do? List 5. [/INST] \"\n# Generation arguments\npayload = {\n\"do_sample\"\n:\nTrue\n,\n\"top_p\"\n:\n0.6\n,\n\"temperature\"\n:\n0.9\n,\n\"top_k\"\n:\n50\n,\n\"max_new_tokens\"\n:\n1024\n,\n\"repetition_penalty\"\n:\n1.03\n,\n\"return_full_text\"\n:\nFalse\n,\n\"stop\"\n: [\n\"</s>\"\n]\n}\nOkay lets test it.\nPython\nchat = llm.\npredict\n({\n\"inputs\"\n:prompt,\n\"parameters\"\n:payload})\nprint(chat[\n0\n][\n\"generated_text\"\n])\nAwesome, we tested infernece now lets build a cool demo which support streaming responses.\nAmazon SageMaker supports streaming responses\nfrom your model. We can use this to stream responses, we can leverage this to create a streaming gradio application with a better user experience.\nWe created a sample application that you can use to test your model. You can find the code in\ngradio-app.py\n. The application will stream the responses from the model and display them in the UI. You can also use the application to test your model with your own inputs.\nPython\n# add apps directory to path ../apps/\nimport\nsys\nsys.path.\nappend\n(\n\"../demo\"\n)\nfrom\nsagemaker_chat\nimport\ncreate_gradio_app\n# Generation arguments\nparameters = {\n\"do_sample\"\n:\nTrue\n,\n\"top_p\"\n:\n0.6\n,\n\"temperature\"\n:\n0.9\n,\n\"top_k\"\n:\n50\n,\n\"max_new_tokens\"\n:\n1024\n,\n\"repetition_penalty\"\n:\n1.03\n,\n\"return_full_text\"\n:\nFalse\n,\n\"stop\"\n: [\n\"</s>\"\n]\n}\n# define format function for our input\ndef\nformat_prompt\n(\nmessage\n,\nhistory\n,\nsystem_prompt\n):\nprompt =\n\"\"\nfor\nuser_prompt, bot_response\nin\nhistory:\nprompt =\nf\n\"<s> [INST]\n{\nuser_prompt\n}\n[/INST]\n{\nbot_response\n}\n</s>\"\nprompt +=\nf\n\"### Instruction\n\\n\n{\nuser_prompt\n}\n\\n\\n\n\"\nprompt +=\nf\n\"### Answer\n\\n\n{\nbot_response\n}\n\\n\\n\n\"\n# Response already contains \"Falcon: \"\n# add new user prompt if history is not empty\nif\nlen(history) >\n0\n:\nprompt +=\nf\n\" [INST]\n{\nmessage\n}\n[/INST] \"\nelse\n:\nprompt +=\nf\n\"<s> [INST]\n{\nmessage\n}\n[/INST] \"\nreturn\nprompt\n# create gradio app\ncreate_gradio_app\n(\nllm.endpoint_name,\n# Sagemaker endpoint name\nsession\n=sess.boto_session,\n# boto3 session used to send request\nparameters\n=parameters,\n# Request parameters\nsystem_prompt\n=\nNone\n,\n# System prompt to use -> Mistral does not support system prompts\nformat_prompt\n=format_prompt,\n# Function to format prompt\nconcurrency_count\n=\n4\n,\n# Number of concurrent requests\nshare\n=\nTrue\n,\n# Share app publicly\n)\n6. Clean up\nTo clean up, we can delete the model and endpoint.\nPython\nllm.\ndelete_model\n()\nllm.\ndelete_endpoint\n()\nThanks for reading! If you have any questions, feel free to contact me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Deploy Mixtral 8x7B on Amazon SageMaker",
    "description": "In this blog post you will learn how to deploy Mixtral 8x7B to Amazon SageMaker.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Deploy Mixtral 8x7B on Amazon SageMaker",
    "og:description": "In this blog post you will learn how to deploy Mixtral 8x7B to Amazon SageMaker.",
    "og:url": "https://www.philschmid.de/sagemaker-deploy-mixtral",
    "og:image": "https://www.philschmid.de/static/blog/sagemaker-deploy-mixtral/thumbnail.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Deploy Mixtral 8x7B on Amazon SageMaker",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Deploy Mixtral 8x7B on Amazon SageMaker",
    "twitter:description": "In this blog post you will learn how to deploy Mixtral 8x7B to Amazon SageMaker.",
    "twitter:image": "https://www.philschmid.de/static/blog/sagemaker-deploy-mixtral/thumbnail.jpg",
    "article:published_time": "2023-12-12"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/997fa9190a3c/code/body-000.txt",
      "chars": 107,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-001.txt",
      "chars": 736,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-002.txt",
      "chars": 1744,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-003.txt",
      "chars": 991,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-004.txt",
      "chars": 320,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-005.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-006.txt",
      "chars": 333,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-007.txt",
      "chars": 94,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-008.txt",
      "chars": 1438,
      "language": "text"
    },
    {
      "file": "attachments/997fa9190a3c/code/body-009.txt",
      "chars": 40,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/sagemaker-deploy-mixtral/gradio.png",
      "bytes": 40807,
      "reused": false,
      "local_path": "attachments/997fa9190a3c/images/body-05b6cfb807e255f9bd75.png",
      "src": "https://www.philschmid.de/static/blog/sagemaker-deploy-mixtral/gradio.png",
      "alt": "gradio",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/llm-sagemaker-sample/blob/main/notebooks/deploy-mixtral.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1",
      "label": "Mixtral 8x7B",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mistralai",
      "label": "Mistral AI",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/moe",
      "label": "Mixture of Experts Explained",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/text-generation-inference",
      "label": "Text Generation Inference (TGI)",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/aws/deep-learning-containers/blob/master/available_images.md",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/sagemaker-llama-llm",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://huggingface.github.io/text-generation-inference/",
      "label": "swagger documentation",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/de/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/",
      "label": "Amazon SageMaker supports streaming responses",
      "type": "a"
    },
    {
      "href": "https://github.com/philschmid/llm-sagemaker-sample/blob/main/demo/sagemaker_chat.py",
      "label": "gradio-app.py",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:54Z",
  "source_url": "https://www.philschmid.de/sagemaker-deploy-mixtral",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11033,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/997fa9190a3c/pages/source.html",
  "local_html": "attachments/997fa9190a3c/pages/article.html",
  "body_base_path": "attachments/997fa9190a3c/pages/",
  "body_markdown_path": "attachments/997fa9190a3c/pages/article.md"
}