{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Deploy GPT-J on SageMaker",
  "keywords": "SageMaker,GPT-J,部署",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/deploy-gptj-sagemaker",
  "publish_date": "2022-01",
  "corpus_stage": "推理部署",
  "notes": "在SageMaker部署GPT-J",
  "id": "6401068a381b",
  "collected_at": "2026-10-02T14:29:15",
  "title": "Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker",
  "body_markdown": "Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker\nJanuary 11, 2022\n10\nminute read\nView Code\nAlmost 6 months ago to the day,\nEleutherAI\nreleased\nGPT-J 6B\n, an open-source alternative to\nOpenAIs GPT-3\n.\nGPT-J 6B\nis the 6 billion parameter successor to\nEleutherAIs\nGPT-NEO family, a family of transformer-based language models based on the GPT architecture for text generation.\nEleutherAI\n's primary goal is to train a model that is equivalent in size to GPT⁠-⁠3 and make it available to the public under an open license.\nOver the last 6 months,\nGPT-J\ngained a lot of interest from Researchers, Data Scientists, and even Software Developers, but it remained very challenging to deploy\nGPT-J\ninto production for real-world use cases and products.\nThere are some hosted solutions to use\nGPT-J\nfor production workloads, like the\nHugging Face Inference API\n, or for experimenting using\nEleutherAIs 6b playground\n, but fewer examples on how to easily deploy it into your own environment.\nIn this blog post, you will learn how to easily deploy\nGPT-J\nusing\nAmazon SageMaker\nand the\nHugging Face Inference Toolkit\nwith a few lines of code for scalable, reliable, and secure real-time inference using a regular size GPU instance with NVIDIA T4 (~500$/m).\nBut before we get into it, I want to explain why deploying\nGPT-J\ninto production is challenging.\nBackground\nThe weights of the 6 billion parameter model represent a ~24GB memory footprint. To load it in float32, one would need at least 2x model size CPU RAM: 1x for initial weights and another 1x to load the checkpoint. So for\nGPT-J\nit would require at least 48GB of CPU RAM to just load the model.\nTo make the model more accessible,\nEleutherAI\nalso provides float16 weights, and\ntransformers\nhas new options to reduce the memory footprint when loading large language models. Combining all this it should take roughly 12.1GB of CPU RAM to load the model.\nPython\nfrom\ntransformers\nimport\nGPTJForCausalLM\nimport\ntorch\nmodel = GPTJForCausalLM.\nfrom_pretrained\n(\n\"EleutherAI/gpt-j-6B\"\n,\nrevision\n=\n\"float16\"\n,\ntorch_dtype\n=torch.float16,\nlow_cpu_mem_usage\n=\nTrue\n)\nThe caveat of this example is that it takes a very long time until the model is loaded into memory and ready for use. In my experiments, it took\n3 minutes and 32 seconds\nto load the model with the code snippet above on a\nP3.2xlarge\nAWS EC2 instance (the model was not stored on disk). This duration can be reduced by storing the model already on disk, which reduces the load time to\n1 minute and 23 seconds\n, which is still very long for production workloads where you need to consider scaling and reliability.\nFor example, Amazon SageMaker has a\n60s limit for requests to respond\n, meaning the model needs to be loaded and the predictions to run within 60s, which in my opinion makes a lot of sense to keep the model/endpoint scalable and reliable for your workload. If you have longer predictions, you could use\nbatch-transform\n.\nIn\nTransformers\nthe models loaded with the\nfrom_pretrained\nmethod are following PyTorch's\nrecommended practice\n, which takes around\n1.97 seconds\nfor BERT\n[REF]\n. PyTorch offers an\nadditional alternative way of saving and loading models\nusing\ntorch.save(model, PATH)\nand\ntorch.load(PATH)\n.\n“Saving a model in this way will save the entire module using Python’s\npickle\nmodule. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved.”\nThis means that when we save a model with\ntransformers==4.13.2\nit could be potentially incompatible when trying to load with\ntransformers==4.15.0\n. However, loading models this way reduces the loading time by\n~12x,\ndown to\n0.166s\nfor BERT.\nApplying this to\nGPT-J\nmeans that we can reduce the loading time from\n1 minute and 23 seconds\ndown to\n7.7 seconds\n, which is ~10.5x faster.\nTutorial\nWith this method of saving and loading models, we achieved model loading performance for\nGPT-J\ncompatible with production scenarios. But we need to keep in mind that we need to align:\nAlign PyTorch and Transformers version when saving the model with\ntorch.save(model,PATH)\nand loading the model with\ntorch.load(PATH)\nto avoid incompatibility.\nSave\nGPT-J\nusing\ntorch.save\nTo create our\ntorch.load()\ncompatible model file we load\nGPT-J\nusing Transformers and the\nfrom_pretrained\nmethod, and then save it with\ntorch.save()\n.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer,GPTJForCausalLM\nimport\ntorch\n# load fp 16 model\nmodel = GPTJForCausalLM.\nfrom_pretrained\n(\n\"EleutherAI/gpt-j-6B\"\n,\nrevision\n=\n\"float16\"\n,\ntorch_dtype\n=torch.float16)\n# save model with torch.save\ntorch.\nsave\n(model,\n\"gptj.pt\"\n)\nNow we are able to load our\nGPT-J\nmodel with\ntorch.load()\nto run predictions.\nPython\nfrom\ntransformers\nimport\npipeline\nimport\ntorch\n# load model\nmodel = torch.\nload\n(\n\"gptj.pt\"\n)\n# load tokenizer\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(\n\"EleutherAI/gpt-j-6B\"\n)\n# create pipeline\ngen =\npipeline\n(\n\"text-generation\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer,\ndevice\n=\n0\n)\n# run prediction\ngen\n(\n\"My Name is philipp\"\n)\n#[{'generated_text': 'My Name is philipp k. and I live just outside of Detroit....\nCreate\nmodel.tar.gz\nfor the Amazon SageMaker real-time endpoint\nSince we can load our model quickly and run inference on it let’s deploy it to Amazon SageMaker.\nThere are two ways you can deploy transformers to Amazon SageMaker. You can either\n“Deploy a model from the Hugging Face Hub”\ndirectly or\n“Deploy a model with\nmodel_data\nstored on S3”\n. Since we are not using the default Transformers method we need to go with the second option and deploy our endpoint with the model stored on S3.\nFor this, we need to create a\nmodel.tar.gz\nartifact containing our model weights and additional files we need for inference, e.g.\ntokenizer.json\n.\nWe provide uploaded and publicly accessible\nmodel.tar.gz\nartifacts, which can be used with the\nHuggingFaceModel\nto deploy\nGPT-J\nto Amazon SageMaker.\nIf you still want or need to create your own\nmodel.tar.gz\n, e.g. because of compliance guidelines, you can use the helper script\nconvert_gpt.py\nfor this purpose, which creates the\nmodel.tar.gz\nand uploads it to S3.\nBash\n# clone directory\ngit\nclone\nhttps://github.com/philschmid/amazon-sagemaker-gpt-j-sample.git\n# change directory to amazon-sagemaker-gpt-j-sample\ncd\namazon-sagemaker-gpt-j-sample\n# create and upload model.tar.gz\npip3\ninstall\n-r\nrequirements.txt\npython3\nconvert_gptj.py\n--bucket_name\n{model_storage}\nThe\nconvert_gpt.py\nshould print out an S3 URI similar to this.\ns3://hf-sagemaker-inference/gpt-j/model.tar.gz\n.\nDeploy\nGPT-J\nas Amazon SageMaker Endpoint\nTo deploy our Amazon SageMaker Endpoint we are going to use the\nAmazon SageMaker Python SDK\nand the\nHuggingFaceModel\nclass.\nThe snippet below uses the\nget_execution_role\nwhich is only available inside Amazon SageMaker Notebook Instances or Studio. If you want to deploy a model outside of it check\nthe documentation\n.\nThe\nmodel_uri\ndefines the location of our\nGPT-J\nmodel artifact. We are going to use the publicly available one provided by us.\nPython\nfrom\nsagemaker.huggingface\nimport\nHuggingFaceModel\nimport\nsagemaker\n# IAM role with permissions to create endpoint\nrole = sagemaker.\nget_execution_role\n()\n# public S3 URI to gpt-j artifact\nmodel_uri=\n\"s3://huggingface-sagemaker-models/transformers/4.12.3/pytorch/1.9.1/gpt-j/model.tar.gz\"\n# create Hugging Face Model Class\nhuggingface_model =\nHuggingFaceModel\n(\nmodel_data\n=model_uri,\ntransformers_version\n=\n'4.12.3'\n,\npytorch_version\n=\n'1.9.1'\n,\npy_version\n=\n'py38'\n,\nrole\n=role,\n)\n# deploy model to SageMaker Inference\npredictor = huggingface_model.\ndeploy\n(\ninitial_instance_count\n=\n1\n,\n# number of instances\ninstance_type\n=\n'ml.g4dn.xlarge'\n#'ml.p3.2xlarge' # ec2 instance type\n)\nIf you want to use your own\nmodel.tar.gz\njust replace the\nmodel_uri\nwith your S3 Uri.\nThe deployment should take around 3-5 minutes.\nRun predictions\nWe can run predictions using the\npredictor\ninstances created by our\n.deploy\nmethod. To send a request to our endpoint we use the\npredictor.predict\nwith our\ninputs\n.\nPython\npredictor.\npredict\n({\n\"inputs\"\n:\n\"Can you please let us know more details about your \"\n})\nIf you want to customize your predictions using additional\nkwargs\nlike\nmin_length\n, check out “Usage best practices” below.\nUsage best practices\nWhen using generative models, most of the time you want to configure or customize your prediction to fit your needs, for example by using beam search, configuring the max or min length of the generated sequence, or adjust the temperature to reduce repetition. The Transformers library provides different strategies and\nkwargs\nto do this, the Hugging Face Inference toolkit offers the same functionality using the\nparameters\nattribute of your request payload. Below you can find examples on how to generate text without parameters, with beam search, and using custom configurations. If you want to learn about different decoding strategies check out this\nblog post\n.\nDefault request\nThis is an example of a default request using\ngreedy\nsearch.\nInference-time after the first request:\n3s\nPython\npredictor.\npredict\n({\n\"inputs\"\n:\n\"Can you please let us know more details about your \"\n})\nBeam search request\nThis is an example of a request using\nbeam\nsearch with 5 beams.\nInference-time after the first request:\n3.3s\nPython\npredictor.\npredict\n({\n\"inputs\"\n:\n\"Can you please let us know more details about your \"\n,\n\"parameters\"\n: {\n\"num_beams\"\n:\n5\n,\n}\n})\nParameterized request\nThis is an example of a request using a custom parameter, e.g.\nmin_length\nfor generating at least 512 tokens.\nInference-time after the first request:\n38s\nPython\npredictor.\npredict\n({\n\"inputs\"\n:\n\"Can you please let us know more details about your \"\n,\n\"parameters\"\n: {\n\"max_length\"\n:\n512\n,\n\"temperature\"\n:\n0.9\n,\n}\n})\nFew-Shot example (advanced)\nThis is an example of how you could\neos_token_id\nto stop the generation on a certain token, e.g.\n\\n\n,\n.\nor\n###\nfor few-shot predictions. Below is a few-shot example for generating tweets for keywords.\nInference-time after the first request:\n15-45s\nPython\nfrom\ntransformers\nimport\nAutoTokenizer\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(\n\"EleutherAI/gpt-j-6B\"\n)\nend_sequence=\n\"###\"\ntemperature=\n4\nmax_generated_token_length=\n25\nprompt=\n\"\"\"key: markets\ntweet: Take feedback from nature and markets, not from people.\n###\nkey: children\ntweet: Maybe we die so we can come back as children.\n###\nkey: startups\ntweet: Startups shouldn’t worry about how to put out fires, they should worry about how to start them.\n###\nkey: hugging face\ntweet:\"\"\"\npredictor.\npredict\n({\n'inputs'\n: prompt,\n\"parameters\"\n: {\n\"max_length\"\n: int(len(prompt) + max_generated_token_length),\n\"temperature\"\n: float(temperature),\n\"eos_token_id\"\n: int(tokenizer.\nconvert_tokens_to_ids\n(end_sequence)),\n\"return_full_text\"\n:\nFalse\n}\n})\nTo delete your endpoint you can run.\nPython\npredictor.\ndelete_endpoint\n()\npredictor.\ndelete_model\n()\nConclusion\nWe successfully managed to deploy\nGPT-J\n, a 6 billion parameter language model created by\nEleutherAI\n, using Amazon SageMaker. We reduced the model load time from 3.5 minutes down to 8 seconds to be able to run scalable, reliable inference.\nRemember that using\ntorch.save()\nand\ntorch.load()\ncan create incompatibility issues. If you want to learn more about scaling out your Amazon SageMaker Endpoints check out my other blog post:\n“MLOps: End-to-End Hugging Face Transformers with the Hub & SageMaker Pipelines”\n.\nThanks for reading! If you have any question, feel free to contact me, through\nGithub\n, or on the\nforum\n. You can also connect with me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker",
    "description": "Learn how to deploy EleutherAIs GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker",
    "og:description": "Learn how to deploy EleutherAIs GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker.",
    "og:url": "https://www.philschmid.de/deploy-gptj-sagemaker",
    "og:image": "https://www.philschmid.de/static/blog/deploy-gptj-sagemaker/cover.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker",
    "twitter:description": "Learn how to deploy EleutherAIs GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker.",
    "twitter:image": "https://www.philschmid.de/static/blog/deploy-gptj-sagemaker/cover.jpg",
    "article:published_time": "2022-01-11"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/6401068a381b/code/body-000.txt",
      "chars": 201,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-001.txt",
      "chars": 256,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-002.txt",
      "chars": 396,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-003.txt",
      "chars": 300,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-004.txt",
      "chars": 676,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-005.txt",
      "chars": 87,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-006.txt",
      "chars": 87,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-007.txt",
      "chars": 131,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-008.txt",
      "chars": 158,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-009.txt",
      "chars": 751,
      "language": "text"
    },
    {
      "file": "attachments/6401068a381b/code/body-010.txt",
      "chars": 52,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deploy-gptj-sagemaker/model_load_time.png",
      "bytes": 46127,
      "reused": false,
      "local_path": "attachments/6401068a381b/images/body-acf7c457b012418ce44c.png",
      "src": "https://www.philschmid.de/static/blog/deploy-gptj-sagemaker/model_load_time.png",
      "alt": "model load time",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/amazon-sagemaker-gpt-j-sample",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://www.eleuther.ai/",
      "label": "EleutherAI",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/EleutherAI/gpt-j-6B",
      "label": "GPT-J 6B",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/gpt-3-apps/",
      "label": "OpenAIs GPT-3",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/inference-api",
      "label": "Hugging Face Inference API",
      "type": "a"
    },
    {
      "href": "https://6b.eleuther.ai/",
      "label": "EleutherAIs 6b playground",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/de/sagemaker/",
      "label": "Amazon SageMaker",
      "type": "a"
    },
    {
      "href": "https://github.com/aws/sagemaker-huggingface-inference-toolkit",
      "label": "Hugging Face Inference Toolkit",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/general/latest/gr/sagemaker.html",
      "label": "60s limit for requests to respond",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html",
      "label": "batch-transform",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers",
      "label": "Transformers",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/tutorials/beginner/saving_loading_models.html",
      "label": "recommended practice",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1-Y5f8PWS8ksoaf1A2qI94jq0GxF2pqQ6?usp=sharing",
      "label": "[REF]",
      "type": "a"
    },
    {
      "href": "https://docs.python.org/3/library/pickle.html",
      "label": "pickle",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/sagemaker/inference",
      "label": "“Deploy a model from the Hugging Face Hub”",
      "type": "a"
    },
    {
      "href": "https://github.com/philschmid/amazon-sagemaker-gpt-j-sample/blob/main/convert_gptj.py",
      "label": "convert_gpt.py",
      "type": "a"
    },
    {
      "href": "https://sagemaker.readthedocs.io/en/stable/",
      "label": "Amazon SageMaker Python SDK",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/sagemaker/train",
      "label": "the documentation",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/how-to-generate",
      "label": "blog post",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/mlops-sagemaker-huggingface-transformers",
      "label": "“MLOps: End-to-End Hugging Face Transformers with the Hub & SageMaker Pipelines”",
      "type": "a"
    },
    {
      "href": "https://discuss.huggingface.co/c/sagemaker/17",
      "label": "forum",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:51Z",
  "source_url": "https://www.philschmid.de/deploy-gptj-sagemaker",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11637,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6401068a381b/pages/source.html",
  "local_html": "attachments/6401068a381b/pages/article.html",
  "body_base_path": "attachments/6401068a381b/pages/",
  "body_markdown_path": "attachments/6401068a381b/pages/article.md"
}