{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Getting Started PyTorch 2.0 Transformers",
  "keywords": "PyTorch 2.0,Transformers,编译",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/getting-started-pytorch-2-0-transformers",
  "publish_date": "2023-03",
  "corpus_stage": "预训练/SFT",
  "notes": "PyTorch 2.0 Transformers入门",
  "id": "2090d1c338e0",
  "collected_at": "2026-10-02T14:29:17",
  "title": "Getting started with Pytorch 2.0 and Hugging Face Transformers",
  "body_markdown": "Getting started with Pytorch 2.0 and Hugging Face Transformers\nMarch 16, 2023\n9\nminute read\nView Code\nOn December 2, 2022, the PyTorch Team announced\nPyTorch 2.0\nat the PyTorch Conference, focused on better performance, being faster, more pythonic, and staying as dynamic as before.\nThis blog post explains how to get started with PyTorch 2.0 and Hugging Face Transformers today. It will cover how to fine-tune a BERT model for Text Classification using the newest PyTorch 2.0 features.\nYou will learn how to:\nSetup environment & install Pytorch 2.0\nLoad and prepare the dataset\nFine-tune & evaluate BERT model with the Hugging Face\nTrainer\nRun Inference & test model\nBefore we can start, make sure you have a\nHugging Face Account\nto save artifacts and experiments.\nQuick intro: Pytorch 2.0\nPyTorch 2.0 or, better, 1.14 is entirely backward compatible. Pytorch 2.0 will not require any modification to existing PyTorch code but can optimize your code by adding a single line of code with\nmodel = torch.compile(model)\n.\nIf you ask yourself, why is there a new major version and no breaking changes? The PyTorch team answered this question in their\nFAQ\n:\n“We were releasing substantial new features that we believe change how you meaningfully use PyTorch, so we are calling it 2.0 instead.”\nThose new features include top-level support for TorchDynamo, AOTAutograd, PrimTorch, and TorchInductor.\nThis allows PyTorch 2.0 to achieve a 1.3x-2x training time speedups supporting\ntoday's 46 model architectures\nfrom\nHuggingFace Transformers\nIf you want to learn more about PyTorch 2.0, check out the official\n“GET STARTED”\n.\nNow we know how PyTorch 2.0 works, let's get started. 🚀\nNote: This tutorial was created and run on a g5.xlarge AWS EC2 Instance, including an NVIDIA A10G GPU.\n1. Setup environment & install Pytorch 2.0\nOur first step is to install PyTorch 2.0 and the Hugging Face Libraries, including\ntransformers\nand\ndatasets\n.\nPython\n# Install PyTorch 2.0 with cuda 11.7\n!\npip install\n\"torch>=2.0\"\n--\nextra-index-url https://download.pytorch.org/whl/cu117\n--\nupgrade\n--\nquiet\nAdditionally, we are installing the latest version of\ntransformers\nfrom the\nmain\ngit branch, which includes the native integration of PyTorch 2.0 into the\nTrainer\n.\nPython\n# Install transformers and dataset\n!\npip install\n\"transformers==4.27.1\"\n\"datasets==2.9.0\"\n\"accelerate==0.17.1\"\n\"evaluate==0.4.0\"\ntensorboard scikit-learn\n# Install git-fls for pushing model and logs to the hugging face hub\n!\nsudo apt-get install git-lfs\n--\nyes\nThis example will use the\nHugging Face Hub\nas a remote model versioning service. To push our model to the Hub, you must register on the\nHugging Face\n. If you already have an account, you can skip this step. After you have an account, we will use the\nlogin\nutil from the\nhuggingface_hub\npackage to log into our account and store our token (access key) on the disk.\nPython\nfrom\nhuggingface_hub\nimport\nlogin\nlogin\n(\ntoken\n=\n\"\"\n,\n# ADD YOUR TOKEN HERE\nadd_to_git_credential\n=\nTrue\n)\n2. Load and prepare the dataset\nTo keep the example straightforward, we are training a Text Classification model on the\nBANKING77\ndataset. The BANKING77 dataset provides a fine-grained set of intents (classes) in a banking/finance domain. It comprises 13,083 customer service queries labeled with 77 intents. It focuses on fine-grained single-domain intent detection.\nWe will use the\nload_dataset()\nmethod from the\n🤗 Datasets\nlibrary to load the\nbanking77\nPython\nfrom\ndatasets\nimport\nload_dataset\n# Dataset id from huggingface.co/dataset\ndataset_id =\n\"banking77\"\n# Load raw dataset\nraw_dataset =\nload_dataset\n(dataset_id)\nprint(\nf\n\"Train dataset size:\n{\nlen(raw_dataset[\n'train'\n])\n}\n\"\n)\nprint(\nf\n\"Test dataset size:\n{\nlen(raw_dataset[\n'test'\n])\n}\n\"\n)\nLet’s check out an example of the dataset.\nPython\nfrom\nrandom\nimport\nrandrange\nrandom_id =\nrandrange\n(len(raw_dataset[\n'train'\n]))\nraw_dataset[\n'train'\n][random_id]\n# {'text': \"I can't get google pay to work right.\", 'label': 2}\nTo train our model, we need to convert our \"Natural Language\" to token IDs. This is done by a Tokenizer, which tokenizes the inputs (including converting the tokens to their corresponding IDs in the pre-trained vocabulary) if you want to learn more about this, out\nchapter 6\nof the\nHugging Face Course\n.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer\n# Model id to load the tokenizer\nmodel_id =\n\"bert-base-uncased\"\n# Load Tokenizer\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\n# Tokenize helper function\ndef\ntokenize\n(\nbatch\n):\nreturn\ntokenizer\n(batch[\n'text'\n],\npadding\n=\n'max_length'\n,\ntruncation\n=\nTrue\n,\nreturn_tensors\n=\n\"pt\"\n)\n# Tokenize dataset\nraw_dataset =  raw_dataset.\nrename_column\n(\n\"label\"\n,\n\"labels\"\n)\n# to match Trainer\ntokenized_dataset = raw_dataset.\nmap\n(tokenize,\nbatched\n=\nTrue\n,\nremove_columns\n=[\n\"text\"\n])\nprint(tokenized_dataset[\n\"train\"\n].features.\nkeys\n())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask','lable'])\n3. Fine-tune & evaluate BERT model with the Hugging Face\nTrainer\nAfter we have processed our dataset, we can start training our model. We will use the bert-base-uncased model. The first step is to load our model with\nAutoModelForSequenceClassification\nclass from the\nHugging Face Hub\n. This will initialize the pre-trained BERT weights with a classification head on top. Here we pass the number of classes (77) from our dataset and the label names to have readable outputs for inference.\nPython\nfrom\ntransformers\nimport\nAutoModelForSequenceClassification\n# Model id to load the tokenizer\nmodel_id =\n\"bert-base-uncased\"\n# Prepare model labels - useful for inference\nlabels = tokenized_dataset[\n\"train\"\n].features[\n\"labels\"\n].names\nnum_labels = len(labels)\nlabel2id, id2label = dict(), dict()\nfor\ni, label\nin\nenumerate(labels):\nlabel2id[label] = str(i)\nid2label[str(i)] = label\n# Download the model from huggingface.co/models\nmodel = AutoModelForSequenceClassification.\nfrom_pretrained\n(\nmodel_id,\nnum_labels\n=num_labels,\nlabel2id\n=label2id,\nid2label\n=id2label\n)\nWe evaluate our model during training. The\nTrainer\nsupports evaluation during training by providing a\ncompute_metrics\nmethod. We use the\nevaluate\nlibrary to calculate the\nf1 metric\nduring training on our test split.\nPython\nimport\nevaluate\nimport\nnumpy\nas\nnp\n# Metric Id\nmetric = evaluate.\nload\n(\n\"f1\"\n)\n# Metric helper method\ndef\ncompute_metrics\n(\neval_pred\n):\npredictions, labels = eval_pred\npredictions = np.\nargmax\n(predictions,\naxis\n=\n1\n)\nreturn\nmetric.\ncompute\n(\npredictions\n=predictions,\nreferences\n=labels,\naverage\n=\n\"weighted\"\n)\nThe last step is to define the hyperparameters (\nTrainingArguments\n) we use for our training. Here we are adding the PyTorch 2.0 introduced features for fast training times. To use the latest improvements of PyTorch 2.0, we only need to pass the\ntorch_compile\noption in the\nTrainingArguments\n.\nWe also leverage the\nHugging Face Hub\nintegration of the\nTrainer\nto push our checkpoints, logs, and metrics during training into a repository.\nPython\nfrom\nhuggingface_hub\nimport\nHfFolder\nfrom\ntransformers\nimport\nTrainer, TrainingArguments\n# Id for remote repository\nrepository_id =\n\"bert-base-banking77-pt2\"\n# Define training args\ntraining_args =\nTrainingArguments\n(\noutput_dir\n=repository_id,\nper_device_train_batch_size\n=\n16\n,\nper_device_eval_batch_size\n=\n8\n,\nlearning_rate\n=\n5e-5\n,\nnum_train_epochs\n=\n3\n,\n# PyTorch 2.0 specifics\nbf16\n=\nTrue\n,\n# bfloat16 training\ntorch_compile\n=\nTrue\n,\n# optimizations\noptim\n=\n\"adamw_torch_fused\"\n,\n# improved optimizer\n# logging & evaluation strategies\nlogging_dir\n=\nf\n\"\n{\nrepository_id\n}\n/logs\"\n,\nlogging_strategy\n=\n\"steps\"\n,\nlogging_steps\n=\n200\n,\nevaluation_strategy\n=\n\"epoch\"\n,\nsave_strategy\n=\n\"epoch\"\n,\nsave_total_limit\n=\n2\n,\nload_best_model_at_end\n=\nTrue\n,\nmetric_for_best_model\n=\n\"f1\"\n,\n# push to hub parameters\nreport_to\n=\n\"tensorboard\"\n,\npush_to_hub\n=\nTrue\n,\nhub_strategy\n=\n\"every_save\"\n,\nhub_model_id\n=repository_id,\nhub_token\n=HfFolder.\nget_token\n(),\n)\n# Create a Trainer instance\ntrainer =\nTrainer\n(\nmodel\n=model,\nargs\n=training_args,\ntrain_dataset\n=tokenized_dataset[\n\"train\"\n],\neval_dataset\n=tokenized_dataset[\n\"test\"\n],\ncompute_metrics\n=compute_metrics,\n)\nWe can start our training by using the\ntrain\nmethod of the\nTrainer\n.\nPython\n# Start training\ntrainer.\ntrain\n()\nUsing Pytorch 2.0 and supported features in\ntransformers\nallows us train our BERT model on\n10_000\nsamples within\n457.7964\nseconds.\nWe also ran the training without the\ntorch_compile\noption to compare the training times. The training without\ntorch_compile\ntook 457 seconds, had a\ntrain_samples_per_second\nvalue of 65.55 and an\nf1\nscore of\n0.931\n.\nBash\n{\n'train_runtime'\n:\n696.2701,\n'train_samples_per_second':\n43.1,\n'eval_f1':\n0.928788}\nBy using the\ntorch_compile\noption and the\nadamw_torch_fused\noptimized , we can see that the training time is reduced by 52.5% compared to the training without PyTorch 2.0.\nBash\n{\n'train_runtime'\n:\n457.7964,\n'train_samples_per_second':\n65.55,\n'eval_f1':\n0.931773}\nOur absoulte training time went down from 696s to 457. The\ntrain_samples_per_second\nvalue increased from 43 to 65. The\nf1\nscore is the same/slighty better than the training without\ntorch_compile\n.\nPytorch 2.0 is incredible powerful! 🚀\nLets save our results and tokenizer to the Hugging Face Hub and create a model card.\nPython\n# Save processor and create model card\ntokenizer.\nsave_pretrained\n(repository_id)\ntrainer.\ncreate_model_card\n()\ntrainer.\npush_to_hub\n()\n4. Run Inference & test model\nTo wrap up this tutorial, we will run inference on a few examples and test our model. We will use the\npipeline\nmethod from the\ntransformers\nlibrary to run inference on our model.\nPython\nfrom\ntransformers\nimport\npipeline\n# load model from huggingface.co/models using our repository id\nclassifier =\npipeline\n(\n\"sentiment-analysis\"\n,\nmodel\n=repository_id,\ntokenizer\n=repository_id,\ndevice\n=\n0\n)\nsample =\n\"I have been waiting longer than expected for my bank card, could you provide information on when it will arrive?\"\npred =\nclassifier\n(sample)\nprint(pred)\n# [{'label': 'card_arrival', 'score': 0.9903606176376343}]\nConclusion\nIn this tutorial, we learned how to use PyTorch 2.0 to train a text classification model on the BANKING77 dataset. We saw that PyTorch 2.0 is a powerful tool to speed up your training times. In our example running on a NVIDIA A10G we managed to achieve 52.5% better performance. The Hugging Face Trainer allows you to easily integrate PyTorch 2.0 into your training pipeline by simply adding the\ntorch_compile\noption to the\nTrainingArguments\n. We can further benefit from PyTorch 2.0 by using the new fused AdamW optimizer when bf16 is available.\nAdditionally, I want to mentioned that we reduced the training time by 52%, which could be interpreted in a cost saving of 52% for the training or in 52% faster iterations cycles and time to production. You should be able to see even better improvements by using A100 GPUs or by reducing the \"Trainer\" overhead, e.g. removing evaluation and logging.\nPyTorch 2.0 is now officially launched and we are excited to see what the future brings. 🚀\nThanks for reading! If you have any questions, feel free to contact me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Getting started with Pytorch 2.0 and Hugging Face Transformers",
    "description": "Learn how to get started with Pytorch 2.0 and Hugging Face Transformers and reduce your training time up to 2x.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Getting started with Pytorch 2.0 and Hugging Face Transformers",
    "og:description": "Learn how to get started with Pytorch 2.0 and Hugging Face Transformers and reduce your training time up to 2x.",
    "og:url": "https://www.philschmid.de/getting-started-pytorch-2-0-transformers",
    "og:image": "https://www.philschmid.de/static/blog/getting-started-pytorch-2-0-transformers/thumbnail.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Getting started with Pytorch 2.0 and Hugging Face Transformers",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Getting started with Pytorch 2.0 and Hugging Face Transformers",
    "twitter:description": "Learn how to get started with Pytorch 2.0 and Hugging Face Transformers and reduce your training time up to 2x.",
    "twitter:image": "https://www.philschmid.de/static/blog/getting-started-pytorch-2-0-transformers/thumbnail.png",
    "article:published_time": "2023-03-16"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2090d1c338e0/code/body-000.txt",
      "chars": 137,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-001.txt",
      "chars": 257,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-002.txt",
      "chars": 109,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-003.txt",
      "chars": 277,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-004.txt",
      "chars": 175,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-005.txt",
      "chars": 626,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-006.txt",
      "chars": 574,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-007.txt",
      "chars": 309,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-008.txt",
      "chars": 1196,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-009.txt",
      "chars": 32,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-010.txt",
      "chars": 82,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-011.txt",
      "chars": 83,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-012.txt",
      "chars": 129,
      "language": "text"
    },
    {
      "file": "attachments/2090d1c338e0/code/body-013.txt",
      "chars": 426,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/getting-started-pytorch-2-0-transformers/tensorboard.png",
      "bytes": 115834,
      "reused": false,
      "local_path": "attachments/2090d1c338e0/images/body-c15f6b8dcf118e011159.png",
      "src": "https://www.philschmid.de/static/blog/getting-started-pytorch-2-0-transformers/tensorboard.png",
      "alt": "tensorboard",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/pytorch-2-0-bert-text-classification.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/get-started/pytorch-2.0/",
      "label": "PyTorch 2.0",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join",
      "label": "Hugging Face Account",
      "type": "a"
    },
    {
      "href": "https://github.com/pytorch/torchdynamo/issues/681",
      "label": "today's 46 model architectures",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers",
      "label": "HuggingFace Transformers",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/models",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/banking77",
      "label": "BANKING77",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/datasets/index",
      "label": "🤗 Datasets",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/course/chapter6/1?fw=pt",
      "label": "chapter 6",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/course/chapter1/1",
      "label": "Hugging Face Course",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/bert-base-uncased",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/spaces/evaluate-metric/f1",
      "label": "f1 metric",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:52Z",
  "source_url": "https://www.philschmid.de/getting-started-pytorch-2-0-transformers",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11144,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2090d1c338e0/pages/source.html",
  "local_html": "attachments/2090d1c338e0/pages/article.html",
  "body_base_path": "attachments/2090d1c338e0/pages/",
  "body_markdown_path": "attachments/2090d1c338e0/pages/article.md"
}