{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Deploy FLAN-T5 on SageMaker",
  "keywords": "SageMaker,FLAN-T5,部署",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/deploy-flan-t5-sagemaker",
  "publish_date": "2023-02",
  "corpus_stage": "推理部署",
  "notes": "在SageMaker部署FLAN-T5",
  "id": "97061a6686c9",
  "collected_at": "2026-10-02T14:29:15",
  "title": "Deploy FLAN-T5 XXL on Amazon SageMaker",
  "body_markdown": "Deploy FLAN-T5 XXL on Amazon SageMaker\nFebruary 8, 2023\n9\nminute read\nView Code\nWelcome to this Amazon SageMaker guide on how to deploy the\nFLAN-T5-XXL\non Amazon SageMaker for inference. We will deploy\nphilschmid/flan-t5-xxl-sharded-fp16\nto Amazon SageMake for real-time inference using Hugging Face Inference Deep Learning Container.\nWhat we are going to do\nCreate FLAN-T5 XXL inference script with bnb quantization\nCreate SageMaker\nmodel.tar.gz\nartifact\nDeploy the model to Amazon SageMaker\nRun inference using the deployed model\nQuick intro: FLAN-T5, just a better T5\nFLAN-T5 released with the\nScaling Instruction-Finetuned Language Models\npaper is an enhanced version of T5 that has been finetuned in a mixture of tasks. The paper explores instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. The paper discovers that overall instruction finetuning is a general method for improving the performance and usability of pretrained language models.\nPaper:\nhttps://arxiv.org/abs/2210.11416\nOfficial repo:\nhttps://github.com/google-research/t5x\nBefore we can get started we have to install the missing dependencies to be able to create our\nmodel.tar.gz\nartifact and create our Amazon SageMaker endpoint.\nWe also have to make sure we have the permission to create our SageMaker Endpoint.\nPython\n!\npip install\n\"sagemaker==2.116.0\"\n\"huggingface_hub==0.12.0\"\n--\nupgrade\n--\nquiet\nIf you are going to use Sagemaker in a local environment (not SageMaker Studio or Notebook Instances). You need access to an IAM Role with the required permissions for Sagemaker. You can find\nhere\nmore about it.\nPython\nimport\nsagemaker\nimport\nboto3\nsess = sagemaker.\nSession\n()\n# sagemaker session bucket -> used for uploading data, models and logs\n# sagemaker will automatically create this bucket if it not exists\nsagemaker_session_bucket=\nNone\nif\nsagemaker_session_bucket\nis\nNone\nand\nsess\nis\nnot\nNone\n:\n# set to default bucket if a bucket name is not given\nsagemaker_session_bucket = sess.\ndefault_bucket\n()\ntry\n:\nrole = sagemaker.\nget_execution_role\n()\nexcept\nValueError:\niam = boto3.\nclient\n(\n'iam'\n)\nrole = iam.\nget_role\n(\nRoleName\n=\n'sagemaker_execution_role'\n)[\n'Role'\n][\n'Arn'\n]\nsess = sagemaker.\nSession\n(\ndefault_bucket\n=sagemaker_session_bucket)\nprint(\nf\n\"sagemaker role arn:\n{\nrole\n}\n\"\n)\nprint(\nf\n\"sagemaker bucket:\n{\nsess.\ndefault_bucket\n()\n}\n\"\n)\nprint(\nf\n\"sagemaker session region:\n{\nsess.boto_region_name\n}\n\"\n)\n1. Create FLAN-T5 XXL inference script with bnb quantization\nAmazon SageMaker allows us to customize the inference script by providing a\ninference.py\nfile. The\ninference.py\nfile is the entry point to our model. It is responsible for loading the model and handling the inference request. If you are used to deploying Hugging Face Transformers that might be new to you. Usually, we just provide the\nHF_MODEL_ID\nand\nHF_TASK\nand the Hugging Face DLC takes care of the rest. For\nFLAN-T5-XXL\nthats not yet possible. We have to provide the\ninference.py\nfile and implement the\nmodel_fn\nand\npredict_fn\nfunctions to efficiently load the 11B large model.\nIf you want to learn more about creating a custom inference script you can check out\nCreating document embeddings with Hugging Face's Transformers & Amazon SageMaker\nIn addition to the\ninference.py\nfile we also have to provide a\nrequirements.txt\nfile. The\nrequirements.txt\nfile is used to install the dependencies for our\ninference.py\nfile.\nThe first step is to create a\ncode/\ndirectory.\nPython\n!\nmkdir code\nAs next we create a\nrequirements.txt\nfile and add the\naccelerate\nand\nbitsandbytes\nlibrary to it. The\naccelerate\nlibrary is used efficiently to load the model on the GPU. The\nbitsandbytes\nlibrary is used to quantize the model to 8bit using LLM.int8(). LLM.int8 introduces a new quantization technique for Int8 matrix multiplication, which cuts the memory needed for inference by half while. To learn more about check out this\nblog post\nor the\npaper\n.\nPython\n%%writefile code/requirements.txt\naccelerate==\n0.16\n.0\ntransformers==\n4.26\n.0\nbitsandbytes==\n0.37\n.0\nThe last step for our inference handler is to create the\ninference.py\nfile. The\ninference.py\nfile is responsible for loading the model and handling the inference request. The\nmodel_fn\nfunction is called when the model is loaded. The\npredict_fn\nfunction is called when we want to do inference.\nWe are using the\nAutoModelForSeq2SeqLM\nclass from transformers load the model from the local directory (\nmodel_dir\n) in the\nmodel_fn\n. In the\npredict_fn\nfunction we are using the\ngenerate\nfunction from transformers to generate the text for a given input prompt.\nPython\n%%writefile code/inference.py\nfrom\ntyping\nimport\nDict, List, Any\nfrom\ntransformers\nimport\nAutoModelForSeq2SeqLM, AutoTokenizer\nimport\ntorch\ndef\nmodel_fn\n(\nmodel_dir\n):\n# load model and processor from model_dir\nmodel =  AutoModelForSeq2SeqLM.\nfrom_pretrained\n(model_dir,\ndevice_map\n=\n\"auto\"\n,\nload_in_8bit\n=\nTrue\n)\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_dir)\nreturn\nmodel, tokenizer\ndef\npredict_fn\n(\ndata\n,\nmodel_and_tokenizer\n):\n# unpack model and tokenizer\nmodel, tokenizer = model_and_tokenizer\n# process input\ninputs = data.\npop\n(\n\"inputs\"\n, data)\nparameters = data.\npop\n(\n\"parameters\"\n,\nNone\n)\n# preprocess\ninput_ids =\ntokenizer\n(inputs,\nreturn_tensors\n=\n\"pt\"\n).input_ids\n# pass inputs with all kwargs in data\nif\nparameters\nis\nnot\nNone\n:\noutputs = model.\ngenerate\n(input_ids, **parameters)\nelse\n:\noutputs = model.\ngenerate\n(input_ids)\n# postprocess the prediction\nprediction = tokenizer.\ndecode\n(outputs[\n0\n],\nskip_special_tokens\n=\nTrue\n)\nreturn\n[{\n\"generated_text\"\n: prediction}]\n2. Create SageMaker\nmodel.tar.gz\nartifact\nTo use our\ninference.py\nwe need to bundle it together with our model weights into a\nmodel.tar.gz\n. The archive includes all our model-artifcats to run inference. The\ninference.py\nscript will be placed into a\ncode/\nfolder. We will use the\nhuggingface_hub\nSDK to easily download\nphilschmid/flan-t5-xxl-sharded-fp16\nfrom\nHugging Face\nand then upload it to Amazon S3 with the\nsagemaker\nSDK. The model\nphilschmid/flan-t5-xxl-sharded-fp16\nis a sharded fp16 version of the\ngoogle/flan-t5-xxl\nMake sure the enviornment has enough diskspace to store the model, ~30GB should be enough.\nPython\nfrom\ndistutils.dir_util\nimport\ncopy_tree\nfrom\npathlib\nimport\nPath\nfrom\ntempfile\nimport\nTemporaryDirectory\nfrom\nhuggingface_hub\nimport\nsnapshot_download\nHF_MODEL_ID\n=\n\"philschmid/flan-t5-xxl-sharded-fp16\"\n# create model dir\nmodel_tar_dir =\nPath\n(\nHF_MODEL_ID\n.\nsplit\n(\n\"/\"\n)[-\n1\n])\nmodel_tar_dir.\nmkdir\n()\n# setup temporary directory\nwith\nTemporaryDirectory\n()\nas\ntmpdir:\n# download snapshot\nsnapshot_dir =\nsnapshot_download\n(\nrepo_id\n=\nHF_MODEL_ID\n,\ncache_dir\n=tmpdir)\n# copy snapshot to model dir\ncopy_tree\n(snapshot_dir, str(model_tar_dir))\nThe next step is to copy the\ncode/\ndirectory into the\nmodel/\ndirectory.\nPython\n# copy code/ to model dir\ncopy_tree\n(\n\"code/\"\n, str(model_tar_dir.\njoinpath\n(\n\"code\"\n)))\nBefore we can upload the model to Amazon S3 we have to create a\nmodel.tar.gz\narchive. Important is that the archive should directly contain all files and not a folder with the files. For example, your file should look like this:\nmodel.tar.gz/\n|- config.json\n|- pytorch_model-00001-of-00012.bin\n|- tokenizer.json\n|- ...\nPython\nimport\ntarfile\nimport\nos\n# helper to create the model.tar.gz\ndef\ncompress\n(\ntar_dir\n=\nNone\n,\noutput_file\n=\n\"model.tar.gz\"\n):\nparent_dir=os.\ngetcwd\n()\nos.\nchdir\n(tar_dir)\nwith\ntarfile.\nopen\n(os.path.\njoin\n(parent_dir, output_file),\n\"w:gz\"\n)\nas\ntar:\nfor\nitem\nin\nos.\nlistdir\n(\n'.'\n):\nprint(item)\ntar.\nadd\n(item,\narcname\n=item)\nos.\nchdir\n(parent_dir)\ncompress\n(str(model_tar_dir))\nAfter we created the\nmodel.tar.gz\narchive we can upload it to Amazon S3. We will use the\nsagemaker\nSDK to upload the model to our sagemaker session bucket.\nPython\nfrom\nsagemaker.s3\nimport\nS3Uploader\n# upload model.tar.gz to s3\ns3_model_uri = S3Uploader.\nupload\n(\nlocal_path\n=\n\"model.tar.gz\"\n,\ndesired_s3_uri\n=\nf\n\"s3://\n{\nsess.\ndefault_bucket\n()\n}\n/flan-t5-xxl\"\n)\nprint(\nf\n\"model uploaded to:\n{\ns3_model_uri\n}\n\"\n)\n3. Deploy the model to Amazon SageMaker\nAfter we have uploaded our model archive we can deploy our model to Amazon SageMaker. We will use\nHuggingfaceModel\nto create our real-time inference endpoint.\nWe are going to deploy the model to an\ng5.xlarge\ninstance. The\ng5.xlarge\ninstance is a GPU instance with 1 NVIDIA A10G GPU. If you are interested in how you could add autoscaling to your endpoint you can check out\nGoing Production: Auto-scaling Hugging Face Transformers with Amazon SageMaker\n.\nPython\nfrom\nsagemaker.huggingface.model\nimport\nHuggingFaceModel\n# create Hugging Face Model Class\nhuggingface_model =\nHuggingFaceModel\n(\nmodel_data\n=s3_model_uri,\n# path to your model and script\nrole\n=role,\n# iam role with permissions to create an Endpoint\ntransformers_version\n=\n\"4.17\"\n,\n# transformers version used\npytorch_version\n=\n\"1.10\"\n,\n# pytorch version used\npy_version\n=\n'py38'\n,\n# python version used\n)\n# deploy the endpoint endpoint\npredictor = huggingface_model.\ndeploy\n(\ninitial_instance_count\n=\n1\n,\ninstance_type\n=\n\"ml.g5.xlarge\"\n)\n4. Run inference using the deployed model\nThe\n.deploy()\nreturns an\nHuggingFacePredictor\nobject which can be used to request inference using the\n.predict()\nmethod. Our endpoint expects a\njson\nwith at least\ninputs\nkey.\nWhen using generative models, most of the time you want to configure or customize your prediction to fit your needs, for example by using beam search, configuring the max or min length of the generated sequence, or adjusting the temperature to reduce repetition.\nThe Transformers library provides different strategies and kwargs to do this, the Hugging Face Inference toolkit offers the same functionality using the parameters attribute of your request payload. Below you can find examples on how to generate text without parameters, with beam search, and using custom configurations. If you want to learn about different decoding strategies check out this\nblog post\n.\nPython\npayload =\n\"\"\"Summarize the following text:\nPeter and Elizabeth took a taxi to attend the night party in the city. While in the party, Elizabeth collapsed and was rushed to the hospital.\nSince she was diagnosed with a brain injury, the doctor told Peter to stay besides her until she gets well.\nTherefore, Peter stayed with her at the hospital for 3 days without leaving.\n\"\"\"\nparameters = {\n\"early_stopping\"\n:\nTrue\n,\n\"length_penalty\"\n:\n2.0\n,\n\"max_new_tokens\"\n:\n50\n,\n\"temperature\"\n:\n0\n,\n\"min_length\"\n:\n10\n,\n\"no_repeat_ngram_size\"\n:\n3\n,\n}\n# Run prediction\npredictor.\npredict\n({\n\"inputs\"\n: payload,\n\"parameters\"\n:parameters\n})\n# [{'generated_text': 'Peter stayed with Elizabeth at the hospital for 3 days.'}]\nLets try another examples! This time we focus ond questions answering with a step by step approach including some simple math.\nPython\npayload =\n\"\"\"Answer the following question step by step:\nRoger has 5 tennis balls. He buys 2 more cans of tennis balls.\nEach can has 3 tennis balls. How many tennis balls does he have now?\n\"\"\"\nparameters = {\n\"early_stopping\"\n:\nTrue\n,\n\"length_penalty\"\n:\n2.0\n,\n\"max_new_tokens\"\n:\n50\n,\n\"temperature\"\n:\n0\n,\n}\n# Run prediction\npredictor.\npredict\n({\n\"inputs\"\n: payload,\n\"parameters\"\n:parameters\n})\n# [{'generated_text': 'He buys 2 cans of tennis balls, so he has 2 * 3 = 6 tennis balls. He has 5 + 6 = 11 tennis balls now.'}]\nDelete model and endpoint\nTo clean up, we can delete the model and endpoint.\nPython\npredictor.\ndelete_model\n()\npredictor.\ndelete_endpoint\n()\nThanks for reading! If you have any questions, feel free to contact me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Deploy FLAN-T5 XXL on Amazon SageMaker",
    "description": "Learn how to deploy Google's FLAN-T5 XXL on Amazon SageMaker for inference.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Deploy FLAN-T5 XXL on Amazon SageMaker",
    "og:description": "Learn how to deploy Google's FLAN-T5 XXL on Amazon SageMaker for inference.",
    "og:url": "https://www.philschmid.de/deploy-flan-t5-sagemaker",
    "og:image": "https://www.philschmid.de/static/blog/deploy-flan-t5-sagemaker/thumbnail.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Deploy FLAN-T5 XXL on Amazon SageMaker",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Deploy FLAN-T5 XXL on Amazon SageMaker",
    "twitter:description": "Learn how to deploy Google's FLAN-T5 XXL on Amazon SageMaker for inference.",
    "twitter:image": "https://www.philschmid.de/static/blog/deploy-flan-t5-sagemaker/thumbnail.png",
    "article:published_time": "2023-02-08"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/97061a6686c9/code/body-000.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-001.txt",
      "chars": 786,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-002.txt",
      "chars": 11,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-003.txt",
      "chars": 94,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-004.txt",
      "chars": 1056,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-005.txt",
      "chars": 541,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-006.txt",
      "chars": 81,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-007.txt",
      "chars": 89,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-008.txt",
      "chars": 396,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-009.txt",
      "chars": 231,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-010.txt",
      "chars": 594,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-011.txt",
      "chars": 708,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-012.txt",
      "chars": 523,
      "language": "text"
    },
    {
      "file": "attachments/97061a6686c9/code/body-013.txt",
      "chars": 52,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deploy-flan-t5-sagemaker/sagemaker-endpoint.png",
      "bytes": 69558,
      "reused": false,
      "local_path": "attachments/97061a6686c9/images/body-e291b9f6ca41545549a9.png",
      "src": "https://www.philschmid.de/static/blog/deploy-flan-t5-sagemaker/sagemaker-endpoint.png",
      "alt": "flan-t5-on-amazon-sagemaker",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deploy-flan-t5-sagemaker/flan-t5.webp",
      "bytes": 115792,
      "reused": false,
      "local_path": "attachments/97061a6686c9/images/body-0e6ad465f6da4438e122.webp",
      "src": "https://www.philschmid.de/static/blog/deploy-flan-t5-sagemaker/flan-t5.webp",
      "alt": "flan-t5",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/amazon-sagemaker-flan-t5-xxl",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/google/flan-t5-xxl",
      "label": "FLAN-T5-XXL",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16",
      "label": "philschmid/flan-t5-xxl-sharded-fp16",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2210.11416.pdf",
      "label": "Scaling Instruction-Finetuned Language Models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.11416",
      "label": "https://arxiv.org/abs/2210.11416",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/t5x",
      "label": "https://github.com/google-research/t5x",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/custom-inference-huggingface-sagemaker",
      "label": "Creating document embeddings with Hugging Face's Transformers & Amazon SageMaker",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/hf-bitsandbytes-integration",
      "label": "blog post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2208.07339",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/auto-scaling-sagemaker-huggingface",
      "label": "Going Production: Auto-scaling Hugging Face Transformers with Amazon SageMaker",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/how-to-generate",
      "label": "blog post",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:50Z",
  "source_url": "https://www.philschmid.de/deploy-flan-t5-sagemaker",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.11416.pdf",
      "bytes": 1557309,
      "reused": false,
      "local_path": "papers/repair/06f84833c3ff7d851862.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.11416",
      "bytes": 1557309,
      "reused": true,
      "local_path": "papers/repair/face74216dc244661745.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2208.07339",
      "bytes": 719021,
      "reused": true,
      "local_path": "papers/repair/30fa6cd91bee75957c34.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 11650,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/97061a6686c9/pages/source.html",
  "local_html": "attachments/97061a6686c9/pages/article.html",
  "body_base_path": "attachments/97061a6686c9/pages/",
  "body_markdown_path": "attachments/97061a6686c9/pages/article.md"
}