{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Inferentia2 Llama 3 70B",
  "keywords": "Inferentia2,Llama 3,推理",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/inferentia2-llama3-70b",
  "publish_date": "2024-05",
  "corpus_stage": "推理部署",
  "notes": "在Inferentia2上推理Llama 3 70B",
  "id": "671a584b4fb2",
  "collected_at": "2026-10-02T14:29:16",
  "title": "Deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum",
  "body_markdown": "Deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum\nMay 23, 2024\n11\nminute read\nView Code\nLlama 3\nis the latest is the open LLM from Meta, released in April 2024. It is trained on more data - 15T tokens and supports context length window up to 8K tokens and is one of the best open available LLMs. Meta fine-tuned conversational models with Reinforcement Learning from Human Feedback on over 10 million human annotations.\nIn this blog you will learn how to deploy\n/meta-llama/Meta-Llama-3-70B-Instruct\nmodel on AWS Inferentia2 with Hugging Face Optimum on Amazon SageMaker. We are going to use the Hugging Face LLM Inf2 Container a new purpose-built Inference Container to easily deploy LLMs on AWS Inferentia2 powered by\nText Generation Inference\nand\nOptimum Neuron\n.\nIn the blog will cover how to:\nSetup development environment\nRetrieve the new Hugging Face LLM Inf2 DLC\nDeploy Llama 3 70B to inferentia2\nRun inference and chat with the model\nBenchmark llama 3 70B with llmperf on AWS Inferentia2\nClean up\nLets get started! 🚀\nQuick intro: AWS Inferentia 2\nAWS inferentia (Inf2)\nare purpose-built EC2 for deep learning (DL) inference workloads. Inferentia 2 is the successor of\nAWS Inferentia\n, which promises to deliver up to 4x higher throughput and up to 10x lower latency.\ninstance size\naccelerators\nNeuron Cores\naccelerator memory\nvCPU\nCPU Memory\non-demand price ($/h)\ninf2.xlarge\n1\n2\n32\n4\n16\n0.76\ninf2.8xlarge\n1\n2\n32\n32\n128\n1.97\ninf2.24xlarge\n6\n12\n192\n96\n384\n6.49\ninf2.48xlarge\n12\n24\n384\n192\n768\n12.98\nAdditionally, inferentia 2 will support the writing of custom operators in c++ and new datatypes, including\nFP8\n(cFP8).\n1. Setup development environment\nWe are going to use the\nsagemaker\npython SDK to deploy Llama 3 to Amazon SageMaker. We need to make sure to have an AWS account configured and the\nsagemaker\npython SDK installed.\nPython\n!\npip install\n\"sagemaker>=2.199.0\"\n\"gradio<4\"\ntransformers\n--\nupgrade\n--\nquiet\nIf you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find\nhere\nmore about it.\nPython\nimport\nsagemaker\nimport\nboto3\nsess = sagemaker.\nSession\n()\n# sagemaker session bucket -> used for uploading data, models and logs\n# sagemaker will automatically create this bucket if it not exists\nsagemaker_session_bucket=\nNone\nif\nsagemaker_session_bucket\nis\nNone\nand\nsess\nis\nnot\nNone\n:\n# set to default bucket if a bucket name is not given\nsagemaker_session_bucket = sess.\ndefault_bucket\n()\ntry\n:\nrole = sagemaker.\nget_execution_role\n()\nexcept\nValueError:\niam = boto3.\nclient\n(\n'iam'\n)\nrole = iam.\nget_role\n(\nRoleName\n=\n'sagemaker_execution_role'\n)[\n'Role'\n][\n'Arn'\n]\nsess = sagemaker.\nSession\n(\ndefault_bucket\n=sagemaker_session_bucket)\nprint(\nf\n\"sagemaker role arn:\n{\nrole\n}\n\"\n)\nprint(\nf\n\"sagemaker session region:\n{\nsess.boto_region_name\n}\n\"\n)\n2. Retrieve the new Hugging Face LLM Inf2 DLC\nThe new Hugging Face TGI Neuronx DLCs can be used to run inference on AWS Inferentia2. You can use the\nget_huggingface_llm_image_uri\nmethod of the\nsagemaker\nSDK to retrieve the appropriate Hugging Face TGI Neuronx DLC URI based on your desired\nbackend\n,\nsession\n,\nregion\n, and\nversion\n. You can find all the available versions\nhere\n.\nPython\n#\nTODO\n: Comment in when released\nfrom\nsagemaker.huggingface\nimport\nget_huggingface_llm_image_uri\n# retrieve the llm image uri\nllm_image =\nget_huggingface_llm_image_uri\n(\n\"huggingface-neuronx\"\n,\nversion\n=\n\"0.0.22\"\n)\nprint(\nf\n\"llm image uri:\n{\nllm_image\n}\n\"\n)\n3. Deploy Llama 3 70B to inferentia2\nAt the time of writing,\nAWS Inferentia2 does not support dynamic shapes for inference\n, which means that we need to specify our sequence length and batch size ahead of time.\nTo make it easier for customers to utilize the full power of Inferentia2, we created a\nneuron model cache\n, which contains pre-compiled configurations for the most popular LLMs, including Llama 3 70B.\nThis means we don't need to compile the model ourselves, but we can use the pre-compiled model from the cache. You can find compiled/cached configurations on the\nHugging Face Hub\n. If your desired configuration is not yet cached, you can compile it yourself using the\nOptimum CLI\nor open a request at the\nCache repository\nBelow is an example on how to compile Llama 3 70B with Optimum CLI, thats not needed in this case as we are using the pre-compiled model from the cache.\nExample: Compile Llama 3 70B with Optimum CLI\nBash\n# login into the huggingface hub to access gated models, like llama\nhuggingface-cli\nlogin\n--token\n[API_TOKEN]\n# compile model with optimum for batch size 4 and sequence length 2048\noptimum-cli\nexport\nneuron\n-m\nmeta-llama/Meta-Llama-3-70B-Instruct\n--batch_size\n4\n--sequence_length\n2048\n--num_cores\n24\n--auto_cast_type\nfp16\n./llama-70b-chat-neuron\n# push model to hub [repo_id] [local_path] [path_in_repo]\nhuggingface-cli\nupload\naws-neuron/Llama-3-70b-chat-seqlen-2048-bs-4\n./llama-70b-chat-neuron\n./\n--exclude\n\"checkpoint/**\"\n# Move tokenizer to neuron model repository\npython\n-c\n\"from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-70B-Instruct').push_to_hub('aws-neuron/Llama-3-70b-chat-seqlen-2048-bs-4')\"\nNote: You need to compile models on an AWS EC2 instance with Inferentia2 support. Compilation can take up to 45 minutes.\nDeploying Llama 3 70B as Endpoint\nBefore deploying the model to Amazon SageMaker, we must define the TGI Neuronx endpoint configuration. We need to make sure the following additional parameters are defined:\nHF_NUM_CORES\n: Number of Neuron Cores used for the compilation.\nHF_BATCH_SIZE\n: The batch size that was used to compile the model.\nHF_SEQUENCE_LENGTH\n: The sequence length that was used to compile the model.\nHF_AUTO_CAST_TYPE\n: The auto cast type that was used to compile the model.\nWe still need to define traditional TGI parameters with:\nHF_MODEL_ID\n: The Hugging Face model ID.\nHF_TOKEN\n: The Hugging Face API token to access gated models.\nMAX_BATCH_SIZE\n: The maximum batch size that the model can handle, equal to the batch size used for compilation.\nMAX_INPUT_LENGTH\n: The maximum input length that the model can handle.\nMAX_TOTAL_TOKENS\n: The maximum total tokens the model can generate, equal to the sequence length used for compilation.\nSelect the right instance type\nLlama 3 70B is a large model and requires a lot of memory. We are going to use the\ninf2.48xlarge\ninstance type, which has 192 vCPUs and 384 GB of accelerator memory. The\ninf2.48xlarge\ninstance comes with 12 Inferentia2 accelerators that include 24 Neuron Cores. If you want to find the cached configurations for Llama 3 70B, you can find them\nhere\n. In our case we will use a batch size of 4 and a sequence length of 4096.\nBefore we can deploy Llama 3 70B to Inferentia2, we need to make sure we are logged in to the Hugging Face Hub and have the necessary permissions to access the model. You can request access to the model\nhere\n.\nBash\nhuggingface-cli\nlogin\n--token\n[API_TOKEN]\nAfter that we can create our endpoint configuration and deploy the model to Amazon SageMaker.\nPython\nfrom\nhuggingface_hub\nimport\nHfFolder\nfrom\nsagemaker.huggingface\nimport\nHuggingFaceModel\n# sagemaker config\ninstance_type =\n\"ml.inf2.48xlarge\"\nhealth_check_timeout=\n2400\n# additional time to load the model\nvolume_size=\n512\n# size in GB of the EBS volume\n# Define Model and Endpoint configuration parameter\nconfig = {\n\"HF_MODEL_ID\"\n:\n\"meta-llama/Meta-Llama-3-70B-Instruct\"\n,\n\"HF_NUM_CORES\"\n:\n\"24\"\n,\n# number of neuron cores\n\"HF_BATCH_SIZE\"\n:\n\"4\"\n,\n# batch size used to compile the model\n\"HF_SEQUENCE_LENGTH\"\n:\n\"4096\"\n,\n# length used to compile the model\n# \"HF_AUTO_CAST_TYPE\": \"bf16\",  # dtype of the model\n\"HF_AUTO_CAST_TYPE\"\n:\n\"fp16\"\n,\n# dtype of the model\n\"MAX_BATCH_SIZE\"\n:\n\"4\"\n,\n# max batch size for the model\n\"MAX_INPUT_LENGTH\"\n:\n\"4000\"\n,\n# max length of input text\n\"MAX_TOTAL_TOKENS\"\n:\n\"4096\"\n,\n# max length of generated text\n\"MESSAGES_API_ENABLED\"\n:\n\"true\"\n,\n# Enable the messages API\n\"HF_TOKEN\"\n: HfFolder.\nget_token\n(),\n# pass the huggingface token\n}\n# create HuggingFaceModel with the image uri\nllm_model =\nHuggingFaceModel\n(\nrole\n=role,\nimage_uri\n=llm_image,\nenv\n=config\n)\nAfter we have created the\nHuggingFaceModel\nwe can deploy it to Amazon SageMaker using the\ndeploy\nmethod. We will deploy the model with the\nml.inf2.48xlarge\ninstance type. TGI will automatically distribute and shard the model across all Inferentia devices.\nPython\n# deactivate warning since model is compiled\nllm_model._is_compiled_model =\nTrue\nllm = llm_model.\ndeploy\n(\ninitial_instance_count\n=\n1\n,\ninstance_type\n=instance_type,\ncontainer_startup_health_check_timeout\n=health_check_timeout,\nvolume_size\n=volume_size\n)\nSageMaker will now create our endpoint and deploy the model to it. This can takes a 30-40 minutes, we are working on improving the deployment time.\n4. Run inference and chat with the model\nAfter our endpoint is deployed we can run inference on it. We will use the\npredict\nmethod from the\npredictor\nto run inference on our endpoint. We can inference with different parameters to impact the generation. Parameters can be defined as in the\nparameters\nattribute of the payload. You can find supported parameters in the\nhere\n.\nThe Messages API allows us to interact with the model in a conversational way. We can define the role of the message and the content. The role can be either\nsystem\n,\nassistant\nor\nuser\n. The\nsystem\nrole is used to provide context to the model and the\nuser\nrole is used to ask questions or provide input to the model.\nJSON\n{\n\"messages\"\n: [\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n:\n\"You are a helpful assistant.\"\n},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n:\n\"What is deep learning?\"\n}\n]\n}\nPython\n# Prompt to generate\nmessages=[\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n:\n\"You are a helpful assistant.\"\n},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n:\n\"What is deep learning?\"\n}\n]\n# Generation arguments\nparameters = {\n\"model\"\n:\n\"meta-llama/Meta-Llama-3-70B-Instruct\"\n,\n# placholder, needed\n\"top_p\"\n:\n0.6\n,\n\"temperature\"\n:\n0.9\n,\n\"max_tokens\"\n:\n50\n,\n\"stop\"\n: [\n\"<|eot_id|>\"\n],\n}\nOkay lets test it.\nPython\nchat = llm.\npredict\n({\n\"messages\"\n:messages, **parameters,\n\"steam\"\n:\nTrue\n})\nprint(chat[\n\"choices\"\n][\n0\n][\n\"message\"\n][\n\"content\"\n].\nstrip\n())\nAwesome, we tested infernece now lets build a cool demo which support streaming responses.\nAmazon SageMaker supports streaming responses\nfrom your model. We can use this to stream responses, we can leverage this to create a streaming gradio application with a better user experience.\nWe created a sample application that you can use to test your model. You can find the code in\ngradio-app.py\n. The application will stream the responses from the model and display them in the UI. You can also use the application to test your model with your own inputs. With\nshare=True\nyou can share the application with others, since gradio with create a public link for you valid for 72 hours.\nPython\n# add apps directory to path ../apps/\nimport\nsys\nsys.path.\nappend\n(\n\"../demo\"\n)\nfrom\nllama3_chat\nimport\ncreate_gradio_app\n# create gradio app\ncreate_gradio_app\n(\nllm.endpoint_name,\n# Sagemaker endpoint name\nsession\n=sess.boto_session,\n# boto3 session used to send request\nsystem_prompt\n=\n\"You are an helpful Assistant, called Llama 3. Knowing everyting about AWS.\"\n,\nconcurrency_count\n=\n4\n,\n# Number of concurrent requests\nshare\n=\nTrue\n,\n# Share app publicly\n)\n5. Benchmark llama 3 70B with llmperf on AWS Inferentia2\nWe successfully deployed Llama 3 70B to Amazon SageMaker and tested it. Now we want to benchmark the model to see how it performs. We will use a\nllmperf\nfork with support for\nsagemaker\n.\nFirst lets install the\nllmperf\npackage.\nPython\n!\ngit clone https://github.com/philschmid/llmperf.git\n!\npip install -e llmperf/\nNow we can run the benchmark with the following command. We are going to benchmark using\n5\nconcurrent users and max\n50\nrequests. The benchmark will measure\nfirst-time-to-token\n,\nlatency (ms/token)\nand\nthroughput (tokens/s)\nfull details can be found in the\nresults\nfolder\n🚨Important🚨: This benchmark was initiatied from Europe, while the endpoint runs in us-east-1. This has significant impact on the\nfirst-time-to-token\nmetric, since it includes the network communication. If you want to measure the\nfirst-time-to-token\ncorrectly, you need to run the benchmark on the same host or your production region.\nPython\n# tell llmperf that we are using the messages api\n!\nMESSAGES_API\n=true python llmperf/token_benchmark_ray.py \\\n--\nmodel {llm.endpoint_name} \\\n--\nllm-api\n\"sagemaker\"\n\\\n--\nmax-num-completed-requests\n50\n\\\n--\ntimeout\n600\n\\\n--\nnum-concurrent-requests\n5\n\\\n--\nresults-dir\n\"results\"\nLets parse the results and display them nicely.\nPython\nimport\nglob\nimport\njson\n# Reads the summary.json file and prints the results\nwith\nopen(glob.\nglob\n(\nf\n'results/*summary.json'\n)[\n0\n],\n'r'\n)\nas\nfile\n:\ndata = json.\nload\n(\nfile\n)\nprint(\n\"Concurrent requests: 5\"\n)\nprint(\nf\n\"Avg. Input token length:\n{\ndata[\n'mean_input_tokens'\n]\n}\n\"\n)\nprint(\nf\n\"Avg. Output token length:\n{\ndata[\n'mean_output_tokens'\n]\n}\n\"\n)\nprint(\nf\n\"Avg. First-Time-To-Token:\n{\ndata[\n'results_ttft_s_mean'\n]*\n1000\n:.2f\n}\nms\"\n)\nprint(\nf\n\"Avg. Thorughput:\n{\ndata[\n'results_mean_output_throughput_token_per_s'\n]\n:.2f\n}\ntokens/sec\"\n)\nprint(\nf\n\"Avg. Latency:\n{\ndata[\n'results_inter_token_latency_s_mean'\n]*\n1000\n:.2f\n}\nms/token\"\n)\nResults with 150 token generation:\nBash\nConcurrent\nrequests:\n5\nAvg.\nInput\ntoken\nlength:\n550\nAvg.\nOutput\ntoken\nlength:\n150\nAvg.\nFirst-Time-To-Token:\n5505.33ms\nAvg.\nThorughput:\n132.8\ntokens/sec\nAvg.\nLatency:\n23.46ms/token\nThats it! We successfully deployed, tested and benchmarked Llama 3 70B on AWS Inferentia2. The benchmark is not a full representation of the model performance, but it gives you a first good indication. If you plan to use the model in production, we recommend to run a longer and closer to your production benchmark, modify the number of replicas see (\nScale LLM Inference on Amazon SageMaker with Multi-Replica Endpoints\n) and most importantly test the model with your own data.\n6. Clean up\nTo clean up, we can delete the model and endpoint.\nPython\nllm.\ndelete_model\n()\nllm.\ndelete_endpoint\n()\nThanks for reading! If you have any questions or feedback, please let me know on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum",
    "description": "Learn how to deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum on Amazon SageMaker.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum",
    "og:description": "Learn how to deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum on Amazon SageMaker.",
    "og:url": "https://www.philschmid.de/inferentia2-llama3-70b",
    "og:image": "https://www.philschmid.de/static/blog/inferentia2-llama-3-70b/thumbnail.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum",
    "twitter:description": "Learn how to deploy Llama 3 70B on AWS Inferentia2 with Hugging Face Optimum on Amazon SageMaker.",
    "twitter:image": "https://www.philschmid.de/static/blog/inferentia2-llama-3-70b/thumbnail.jpg",
    "article:published_time": "2024-05-23"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/671a584b4fb2/code/body-000.txt",
      "chars": 75,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-001.txt",
      "chars": 736,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-002.txt",
      "chars": 255,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-003.txt",
      "chars": 751,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-004.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-005.txt",
      "chars": 1114,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-006.txt",
      "chars": 256,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-007.txt",
      "chars": 152,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-008.txt",
      "chars": 372,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-009.txt",
      "chars": 127,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-010.txt",
      "chars": 506,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-011.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-012.txt",
      "chars": 266,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-013.txt",
      "chars": 589,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-014.txt",
      "chars": 179,
      "language": "text"
    },
    {
      "file": "attachments/671a584b4fb2/code/body-015.txt",
      "chars": 40,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/inferentia2-llama-3-70b/llama3.png",
      "bytes": 106926,
      "reused": false,
      "local_path": "attachments/671a584b4fb2/images/body-95ad753bc839f4bb67b9.png",
      "src": "https://www.philschmid.de/static/blog/inferentia2-llama-3-70b/llama3.png",
      "alt": "custom container",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/huggingface-inferentia2-samples/blob/main/llama3-70b/deploy-llama-3-70b-inferentia2.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/llama3",
      "label": "Llama 3",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct",
      "label": "/meta-llama/Meta-Llama-3-70B-Instruct",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/text-generation-inference/index",
      "label": "Text Generation Inference",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum-neuron/index",
      "label": "Optimum Neuron",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/de/ec2/instance-types/inf2/",
      "label": "AWS inferentia (Inf2)",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/ec2/instance-types/inf1/?nc1=h_ls",
      "label": "AWS Inferentia",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/aws/deep-learning-containers/releases?q=tgi+AND+neuronx&expanded=true",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://awsdocs-neuron.readthedocs-hosted.com/en/v2.6.0/general/arch/neuron-features/dynamic-shapes.html",
      "label": "AWS Inferentia2 does not support dynamic shapes for inference",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum-neuron/guides/cache_system",
      "label": "neuron model cache",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/aws-neuron/optimum-neuron-cache/tree/main/inference-cache-config",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/optimum-neuron/cli/compile",
      "label": "Optimum CLI",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/aws-neuron/optimum-neuron-cache/discussions",
      "label": "Cache repository",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/aws-neuron/optimum-neuron-cache/blob/6392789bb1f2ab2af2867f0ab564058b781054ca/inference-cache-config/llama.json",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/text-generation-inference/messages_api",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/de/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/",
      "label": "Amazon SageMaker supports streaming responses",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/demo/sagemaker_chat.py",
      "label": "gradio-app.py",
      "type": "a"
    },
    {
      "href": "https://github.com/philschmid/llmperf",
      "label": "llmperf",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/sagemaker-multi-replicahttps://www.philschmid.de/sagemaker-multi-replica",
      "label": "Scale LLM Inference on Amazon SageMaker with Multi-Replica Endpoints",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:51Z",
  "source_url": "https://www.philschmid.de/inferentia2-llama3-70b",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 14225,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/671a584b4fb2/pages/source.html",
  "local_html": "attachments/671a584b4fb2/pages/article.html",
  "body_base_path": "attachments/671a584b4fb2/pages/",
  "body_markdown_path": "attachments/671a584b4fb2/pages/article.md"
}