{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Knowledge Distillation BERT Transformers",
  "keywords": "知识蒸馏,BERT,压缩",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/knowledge-distillation-bert-transformers",
  "publish_date": "2022-02",
  "corpus_stage": "推理部署",
  "notes": "BERT知识蒸馏教程",
  "id": "2b67b4a16a3e",
  "collected_at": "2026-10-02T14:29:14",
  "title": "Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker",
  "body_markdown": "Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker\nFebruary 1, 2022\n12\nminute read\nView Code\nWelcome to this end-to-end task-specific knowledge distillation Text-Classification example using Transformers, PyTorch & Amazon SageMaker. Distillation is the process of training a small \"student\" to mimic a larger \"teacher\". In this example, we will use a\nBERT-base\nas Teacher and\nBERT-Tiny\nas Student. We will use\nText-Classification\nas our task-specific knowledge distillation task and the\nStanford Sentiment Treebank v2 (SST-2)\ndataset for training.\nThey are two different types of knowledge distillation, the Task-agnostic knowledge distillation (right) and the Task-specific knowledge distillation (left). In this example we are going to use the Task-specific knowledge distillation.\nTask-specific distillation (left) versus task-agnostic distillation (right). Figure from FastFormers by Y. Kim and H. Awadalla [arXiv:2010.13382].\nIn Task-specific knowledge distillation a \"second step of distillation\" is used to \"fine-tune\" the model on a given dataset. This idea comes from the\nDistilBERT paper\nwhere it was shown that a student performed better than simply finetuning the distilled language model:\nWe also studied whether we could add another step of distillation during the adaptation phase by fine-tuning DistilBERT on SQuAD using a BERT model previously fine-tuned on SQuAD as a teacher for an additional term in the loss (knowledge distillation). In this setting, there are thus two successive steps of distillation, one during the pre-training phase and one during the adaptation phase. In this case, we were able to reach interesting performances given the size of the model:79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.\nIf you are more interested in those topics you should defintely read:\nDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\nFastFormers: Highly Efficient Transformer Models for Natural Language Understanding\nEspecially the\nFastFormers paper\ncontains great research on what works and doesn't work when using knowledge distillation.\nHuge thanks to\nLewis Tunstall\nand his great\nWeeknotes: Distilling distilled transformers\nInstallation\nPython\n#%pip install \"pytorch==1.10.1\"\n%pip install transformers datasets tensorboard\n--\nupgrade\n!\nsudo apt-get install git-lfs\nThis example will use the\nHugging Face Hub\nas remote model versioning service. To be able to push our model to the Hub, you need to register on the\nHugging Face\n.\nIf you already have an account you can skip this step.\nAfter you have an account, we will use the\nnotebook_login\nutil from the\nhuggingface_hub\npackage to log into our account and store our token (access key) on the disk.\nPython\nfrom\nhuggingface_hub\nimport\nnotebook_login\nnotebook_login\n()\nSetup & Configuration\nIn this step we will define global configurations and paramters, which are used across the whole end-to-end fine-tuning proccess, e.g.\nteacher\nand\nstuden\nwe will use.\nIn this example, we will use\nBERT-base\nas Teacher and\nBERT-Tiny\nas Student. Our Teacher is already fine-tuned on our dataset, which makes it easy for us to directly start the distillation training job rather than fine-tuning the teacher first to then distill it afterwards.\nIMPORTANT\n: This example will only work with a\nTeacher\n&\nStudent\ncombination where the Tokenizer is creating the same output.\nAdditionally, describes the\nFastFormers: Highly Efficient Transformer Models for Natural Language Understanding\npaper an additional phenomenon.\nIn our experiments, we have observed that dis-\ntilled models do not work well when distilled to a\ndifferent model type. Therefore, we restricted our\nsetup to avoid distilling RoBERTa model to BERT\nor vice versa. The major difference between the\ntwo model groups is the input token (sub-word) em-\nbedding. We think that different input embedding\nspaces result in different output embedding spaces,\nand knowledge transfer with different spaces does\nnot work well\nPython\nstudent_id =\n\"google/bert_uncased_L-2_H-128_A-2\"\nteacher_id =\n\"textattack/bert-base-uncased-SST-2\"\n# name for our repository on the hub\nrepo_name =\n\"tiny-bert-sst2-distilled\"\nBelow are some checks to make sure the\nTeacher\n&\nStudent\nare creating the same output.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer\n# init tokenizer\nteacher_tokenizer = AutoTokenizer.\nfrom_pretrained\n(teacher_id)\nstudent_tokenizer = AutoTokenizer.\nfrom_pretrained\n(student_id)\n# sample input\nsample =\n\"This is a basic example, with different words to test.\"\n# assert results\nassert\nteacher_tokenizer\n(sample) ==\nstudent_tokenizer\n(sample),\n\"Tokenizers haven't created the same output\"\nDataset & Pre-processing\nAs Dataset we will use the\nStanford Sentiment Treebank v2 (SST-2)\na text-classification for\nsentiment-analysis\n, which is included in the\nGLUE benchmark\n. The dataset is based on the dataset introduced by Pang and Lee (2005) and consists of 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes a total of 215,154 unique phrases from those parse trees, each annotated by 3 human judges. It uses the two-way (positive/negative) class split, with only sentence-level labels.\nPython\ndataset_id=\n\"glue\"\ndataset_config=\n\"sst2\"\nTo load the\nsst2\ndataset, we use the\nload_dataset()\nmethod from the 🤗 Datasets library.\nPython\nfrom\ndatasets\nimport\nload_dataset\ndataset =\nload_dataset\n(dataset_id,dataset_config)\nPre-processing & Tokenization\nTo distill our model we need to convert our \"Natural Language\" to token IDs. This is done by a 🤗 Transformers Tokenizer which will tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary). If you are not sure what this means check out\nchapter 6\nof the Hugging Face Course.\nWe are going to use the tokenizer of the\nTeacher\n, but since both are creating same output you could also go with the\nStudent\ntokenizer.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(teacher_id)\nAdditionally we add the\ntruncation=True\nand\nmax_length=512\nto align the length and truncate texts that are bigger than the maximum size allowed by the model.\nPython\ndef\nprocess\n(\nexamples\n):\ntokenized_inputs =\ntokenizer\n(\nexamples[\n\"sentence\"\n],\ntruncation\n=\nTrue\n,\nmax_length\n=\n512\n)\nreturn\ntokenized_inputs\ntokenized_datasets = dataset.\nmap\n(process,\nbatched\n=\nTrue\n)\ntokenized_datasets = tokenized_datasets.\nrename_column\n(\n\"label\"\n,\n\"labels\"\n)\ntokenized_datasets[\n\"test\"\n].features\nDistilling the model using\nPyTorch\nand\nDistillationTrainer\nNow that our\ndataset\nis processed, we can distill it. Normally, when fine-tuning a transformer model using PyTorch you should go with the\nTrainer-API\n. The\nTrainer\nclass provides an API for feature-complete training in PyTorch for most standard use cases.\nIn our example we cannot use the\nTrainer\nout-of-the-box, since we need to pass in two models, the\nTeacher\nand the\nStudent\nand compute the loss for both. But we can subclass the\nTrainer\nto create a\nDistillationTrainer\nwhich will take care of it and only overwrite the\ncompute_loss\nmethod as well as the\ninit\nmethod. In addition to this we also need to subclass the\nTrainingArguments\nto include the our distillation hyperparameters.\nPython\nfrom\ntransformers\nimport\nTrainingArguments, Trainer\nimport\ntorch\nimport\ntorch.nn\nas\nnn\nimport\ntorch.nn.functional\nas\nF\nclass\nDistillationTrainingArguments\n(\nTrainingArguments\n):\ndef\n__init__(\nself\n, *\nargs\n,\nalpha\n=\n0.5\n,\ntemperature\n=\n2.0\n, **\nkwargs\n):\nsuper().__init__(*args, **kwargs)\nself\n.alpha = alpha\nself\n.temperature = temperature\nclass\nDistillationTrainer\n(\nTrainer\n):\ndef\n__init__(\nself\n, *\nargs\n,\nteacher_model\n=\nNone\n, **\nkwargs\n):\nsuper().__init__(*args, **kwargs)\nself\n.teacher = teacher_model\n# place teacher on same device as student\nself\n.\n_move_model_to_device\n(\nself\n.teacher,\nself\n.model.device)\nself\n.teacher.\neval\n()\ndef\ncompute_loss\n(\nself\n,\nmodel\n,\ninputs\n,\nreturn_outputs\n=\nFalse\n):\n# compute student output\noutputs_student =\nmodel\n(**inputs)\nstudent_loss=outputs_student.loss\n# compute teacher output\nwith\ntorch.\nno_grad\n():\noutputs_teacher =\nself\n.\nteacher\n(**inputs)\n# assert size\nassert\noutputs_student.logits.\nsize\n() == outputs_teacher.logits.\nsize\n()\n# Soften probabilities and compute distillation loss\nloss_function = nn.\nKLDivLoss\n(\nreduction\n=\n\"batchmean\"\n)\nloss_logits = (\nloss_function\n(\nF.\nlog_softmax\n(outputs_student.logits /\nself\n.args.temperature,\ndim\n=-\n1\n),\nF.\nsoftmax\n(outputs_teacher.logits /\nself\n.args.temperature,\ndim\n=-\n1\n)) * (\nself\n.args.temperature **\n2\n))\n# Return weighted student loss\nloss =\nself\n.args.alpha * student_loss + (\n1\n. -\nself\n.args.alpha) * loss_logits\nreturn\n(loss, outputs_student)\nif\nreturn_outputs\nelse\nloss\nHyperparameter Definition, Model Loading\nPython\nfrom\ntransformers\nimport\nAutoModelForSequenceClassification, DataCollatorWithPadding\nfrom\nhuggingface_hub\nimport\nHfFolder\n# create label2id, id2label dicts for nice outputs for the model\nlabels = tokenized_datasets[\n\"train\"\n].features[\n\"labels\"\n].names\nnum_labels = len(labels)\nlabel2id, id2label = dict(), dict()\nfor\ni, label\nin\nenumerate(labels):\nlabel2id[label] = str(i)\nid2label[str(i)] = label\n# define training args\ntraining_args =\nDistillationTrainingArguments\n(\noutput_dir\n=repo_name,\nnum_train_epochs\n=\n7\n,\nper_device_train_batch_size\n=\n128\n,\nper_device_eval_batch_size\n=\n128\n,\nfp16\n=\nTrue\n,\nlearning_rate\n=\n6e-5\n,\nseed\n=\n33\n,\n# logging & evaluation strategies\nlogging_dir\n=\nf\n\"\n{\nrepo_name\n}\n/logs\"\n,\nlogging_strategy\n=\n\"epoch\"\n,\n# to get more information to TB\nevaluation_strategy\n=\n\"epoch\"\n,\nsave_strategy\n=\n\"epoch\"\n,\nsave_total_limit\n=\n2\n,\nload_best_model_at_end\n=\nTrue\n,\nmetric_for_best_model\n=\n\"accuracy\"\n,\nreport_to\n=\n\"tensorboard\"\n,\n# push to hub parameters\npush_to_hub\n=\nTrue\n,\nhub_strategy\n=\n\"every_save\"\n,\nhub_model_id\n=repo_name,\nhub_token\n=HfFolder.\nget_token\n(),\n# distilation parameters\nalpha\n=\n0.5\n,\ntemperature\n=\n4.0\n)\n# define data_collator\ndata_collator =\nDataCollatorWithPadding\n(\ntokenizer\n=tokenizer)\n# define model\nteacher_model = AutoModelForSequenceClassification.\nfrom_pretrained\n(\nteacher_id,\nnum_labels\n=num_labels,\nid2label\n=id2label,\nlabel2id\n=label2id,\n)\n# define student model\nstudent_model = AutoModelForSequenceClassification.\nfrom_pretrained\n(\nstudent_id,\nnum_labels\n=num_labels,\nid2label\n=id2label,\nlabel2id\n=label2id,\n)\nEvaluation metric\nwe can create a\ncompute_metrics\nfunction to evaluate our model on the test set. This function will be used during the training process to compute the\naccuracy\n&\nf1\nof our model.\nPython\nfrom\ndatasets\nimport\nload_metric\nimport\nnumpy\nas\nnp\n# define metrics and metrics function\naccuracy_metric =\nload_metric\n(\n\"accuracy\"\n)\ndef\ncompute_metrics\n(\neval_pred\n):\npredictions, labels = eval_pred\npredictions = np.\nargmax\n(predictions,\naxis\n=\n1\n)\nacc = accuracy_metric.\ncompute\n(\npredictions\n=predictions,\nreferences\n=labels)\nreturn\n{\n\"accuracy\"\n: acc[\n\"accuracy\"\n],\n}\nTraining\nStart training with calling\ntrainer.train()\nPython\ntrainer =\nDistillationTrainer\n(\nstudent_model,\ntraining_args,\nteacher_model\n=teacher_model,\ntrain_dataset\n=tokenized_datasets[\n\"train\"\n],\neval_dataset\n=tokenized_datasets[\n\"validation\"\n],\ndata_collator\n=data_collator,\ntokenizer\n=tokenizer,\ncompute_metrics\n=compute_metrics,\n)\nstart training using the\nDistillationTrainer\n.\nPython\ntrainer.\ntrain\n()\nHyperparameter Search for Distillation parameter\nalpha\n&\ntemperature\nwith optuna\nThe parameter\nalpha\n&\ntemparature\nin the\nDistillationTrainer\ncan also be used when doing Hyperparamter search to maxizime our \"knowledge extraction\". As Hyperparamter Optimization framework are we using\nOptuna\n, which has a integration into the\nTrainer-API\n. Since we the\nDistillationTrainer\nis a sublcass of the\nTrainer\nwe can use the\nhyperparameter_search\nwithout any code changes.\nPython\n#%pip install optuna\nTo do Hyperparameter Optimization using\noptuna\nwe need to define our hyperparameter space. In this example we are trying to optimize/maximize the\nnum_train_epochs\n,\nlearning_rate\n,\nalpha\n&\ntemperature\nfor our\nstudent_model\n.\nPython\ndef\nhp_space\n(\ntrial\n):\nreturn\n{\n\"num_train_epochs\"\n: trial.\nsuggest_int\n(\n\"num_train_epochs\"\n,\n2\n,\n10\n),\n\"learning_rate\"\n: trial.\nsuggest_float\n(\n\"learning_rate\"\n,\n1e-5\n,\n1e-3\n,\nlog\n=\nTrue\n),\n\"alpha\"\n: trial.\nsuggest_float\n(\n\"alpha\"\n,\n0\n,\n1\n),\n\"temperature\"\n: trial.\nsuggest_int\n(\n\"temperature\"\n,\n2\n,\n30\n),\n}\nTo start our Hyperparmeter search we just need to call\nhyperparameter_search\nprovide our\nhp_space\nand number of trials to run.\nPython\ndef\nstudent_init\n():\nreturn\nAutoModelForSequenceClassification.\nfrom_pretrained\n(\nstudent_id,\nnum_labels\n=num_labels,\nid2label\n=id2label,\nlabel2id\n=label2id\n)\ntrainer =\nDistillationTrainer\n(\nmodel_init\n=student_init,\nargs\n=training_args,\nteacher_model\n=teacher_model,\ntrain_dataset\n=tokenized_datasets[\n\"train\"\n],\neval_dataset\n=tokenized_datasets[\n\"validation\"\n],\ndata_collator\n=data_collator,\ntokenizer\n=tokenizer,\ncompute_metrics\n=compute_metrics,\n)\nbest_run = trainer.\nhyperparameter_search\n(\nn_trials\n=\n50\n,\ndirection\n=\n\"maximize\"\n,\nhp_space\n=hp_space\n)\nprint(best_run)\nSince optuna is just finding the best hyperparameters we need to fine-tune our model again using the best hyperparamters from the\nbest_run\n.\nPython\n# overwrite initial hyperparameters with from the best_run\nfor\nk,v\nin\nbest_run.hyperparameters.\nitems\n():\nsetattr(training_args, k, v)\n# Define a new repository to store our distilled model\nbest_model_ckpt =\n\"tiny-bert-best\"\ntraining_args.output_dir = best_model_ckpt\nWe have overwritten the default Hyperparameters with the one from our\nbest_run\nand can start the training now.\nPython\n# Create a new Trainer with optimal parameters\noptimal_trainer =\nDistillationTrainer\n(\nstudent_model,\ntraining_args,\nteacher_model\n=teacher_model,\ntrain_dataset\n=tokenized_datasets[\n\"train\"\n],\neval_dataset\n=tokenized_datasets[\n\"validation\"\n],\ndata_collator\n=data_collator,\ntokenizer\n=tokenizer,\ncompute_metrics\n=compute_metrics,\n)\noptimal_trainer.\ntrain\n()\n# save best model, metrics and create model card\ntrainer.\ncreate_model_card\n(\nmodel_name\n=training_args.hub_model_id)\ntrainer.\npush_to_hub\n()\nPython\nfrom\nhuggingface_hub\nimport\nHfApi\nwhoami =\nHfApi\n().\nwhoami\n()\nusername = whoami[\n'name'\n]\nprint(\nf\n\"https://huggingface.co/\n{\nusername\n}\n/\n{\nrepo_name\n}\n\"\n)\nResults & Conclusion\nWe were able to achieve a\naccuracy\nof 0.8337, which is a very good result for our model. Our distilled\nTiny-Bert\nhas 96% less parameters than the teacher\nbert-base\nand runs ~46.5x faster while preserving over 90% of BERT’s performances as measured on the SST2 dataset.\nmodel\nParameter\nSpeed-up\nAccuracy\nBERT-base\n109M\n1x\n93.2%\ntiny-BERT\n4M\n46.5x\n83.4%\nNote: The\nFastFormers paper\nuncovered that the biggest boost in performance is observerd when having 6 or more layers in the student. The\ngoogle/bert_uncased_L-2_H-128_A-2\nwe used only had 2, which means when changing our student to, e.g.\ndistilbert-base-uncased\nwe should better performance in terms of accuracy.\nIf you are now planning to implement and add task-specific knowledge distillation to your models. I suggest to take a look at the\nsagemaker-distillation\n, which shows how to run task-specific knowledge distillation on Amazon SageMaker. For the example i created a script deriving this notebook to make it as easy as possible to use for you. You only need to define your\nteacher_id\n,\nstudent_id\nas well as your\ndataset\nconfig to run task-specific knowledge distillation for\ntext-classification\n.\nPython\nfrom\nsagemaker.huggingface\nimport\nHuggingFace\n# hyperparameters, which are passed into the training job\nhyperparameters={\n'teacher_id'\n:\n'textattack/bert-base-uncased-SST-2'\n,\n'student_id'\n:\n'google/bert_uncased_L-2_H-128_A-2'\n,\n'dataset_id'\n:\n'glue'\n,\n'dataset_config'\n:\n'sst2'\n,\n# distillation parameter\n'alpha'\n:\n0.5\n,\n'temparature'\n:\n4\n,\n# hpo parameter\n\"run_hpo\"\n:\nTrue\n,\n\"n_trials\"\n:\n100\n,\n}\n# create the Estimator\nhuggingface_estimator =\nHuggingFace\n(\n...\n,\nhyperparameters\n=hyperparameters)\n# start knwonledge distillation training\nhuggingface_estimator.\nfit\n()\nIn conclusion you can say that it is just incredible how easy Transformers and the\nTrainer API\ncan be used to implement task-specific knowledge distillation. We needed to write ~20 lines of custom code deriving the\nTrainer\ninto a\nDistillationTrainer\nto support task-specific knowledge distillation with leveraging all benefits of the\nTrainer API\nlike evaluation, hyperparameter tuning, and model card creation.\nIn addition, we used Amazon SageMaker to easily scale our Training with out thinking to much about the infrastructure and how we iterate on our experiments. At the end we created an example, which can be used for any Text-Classification dataset and teacher & student combination for task-specific knowledge distillation.\nI believe this will help companies improiving their production performance of Transformers even more by implementing task-specific knowledge distillation as one part of their MLOps pipeline.\nYou can find the code\nhere\nand feel free to open a thread on the\nforum\n.\nThanks for reading. If you have any questions, feel free to contact me, through\nGithub\n, or on the\nforum\n. You can also connect with me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker",
    "description": "Learn how to run apply task-specific knowledge distillation for BERT and text-classification using Hugging Face Transformers & Amazon SageMaker including Hyperparameter search.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker",
    "og:description": "Learn how to run apply task-specific knowledge distillation for BERT and text-classification using Hugging Face Transformers & Amazon SageMaker including Hyperparameter search.",
    "og:url": "https://www.philschmid.de/knowledge-distillation-bert-transformers",
    "og:image": "https://www.philschmid.de/api/og?title=Task-specific+knowledge+distillation+for+BERT+using+Transformers+%26+Amazon+SageMaker",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker",
    "twitter:description": "Learn how to run apply task-specific knowledge distillation for BERT and text-classification using Hugging Face Transformers & Amazon SageMaker including Hyperparameter search.",
    "twitter:image": "https://www.philschmid.de/api/og?title=Task-specific+knowledge+distillation+for+BERT+using+Transformers+%26+Amazon+SageMaker",
    "article:published_time": "2022-02-01"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2b67b4a16a3e/code/body-000.txt",
      "chars": 120,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-001.txt",
      "chars": 63,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-002.txt",
      "chars": 176,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-003.txt",
      "chars": 393,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-004.txt",
      "chars": 39,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-005.txt",
      "chars": 85,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-006.txt",
      "chars": 94,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-007.txt",
      "chars": 320,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-008.txt",
      "chars": 1626,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-009.txt",
      "chars": 1634,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-010.txt",
      "chars": 387,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-011.txt",
      "chars": 296,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-012.txt",
      "chars": 15,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-013.txt",
      "chars": 20,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-014.txt",
      "chars": 309,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-015.txt",
      "chars": 633,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-016.txt",
      "chars": 271,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-017.txt",
      "chars": 517,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-018.txt",
      "chars": 145,
      "language": "text"
    },
    {
      "file": "attachments/2b67b4a16a3e/code/body-019.txt",
      "chars": 589,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/knowledge-distillation-bert-transformers/knowledge-distillation.png",
      "bytes": 60416,
      "reused": false,
      "local_path": "attachments/2b67b4a16a3e/images/body-b8df4f1127adfe29a2cb.png",
      "src": "https://www.philschmid.de/static/blog/knowledge-distillation-bert-transformers/knowledge-distillation.png",
      "alt": "knowledge-distillation",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/textattack/bert-base-uncased-SST-2",
      "label": "BERT-base",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/google/bert_uncased_L-2_H-128_A-2",
      "label": "BERT-Tiny",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/tasks/text-classification",
      "label": "Text-Classification",
      "type": "a"
    },
    {
      "href": "https://paperswithcode.com/dataset/sst",
      "label": "Stanford Sentiment Treebank v2 (SST-2)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1910.01108.pdf",
      "label": "DistilBERT paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1910.01108",
      "label": "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.13382",
      "label": "FastFormers: Highly Efficient Transformer Models for Natural Language Understanding",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/lewis-tunstall/",
      "label": "Lewis Tunstall",
      "type": "a"
    },
    {
      "href": "https://lewtun.github.io/blog/weeknotes/nlp/huggingface/transformers/2021/01/17/wknotes-distillation-and-generation.html",
      "label": "Weeknotes: Distilling distilled transformers",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/models",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join",
      "label": "Hugging Face",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/textattack/bert-base-uncased-SST-2",
      "label": "BERT-base",
      "type": "a"
    },
    {
      "href": "https://gluebenchmark.com/",
      "label": "GLUE benchmark",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/course/chapter6/1?fw=tf",
      "label": "chapter 6",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/v4.16.1/en/main_classes/trainer",
      "label": "Trainer",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/c4ad38e5ac69e6d96116f39df789a2369dd33c21/src/transformers/trainer.py",
      "label": "compute_loss",
      "type": "a"
    },
    {
      "href": "https://optuna.org/",
      "label": "Optuna",
      "type": "a"
    },
    {
      "href": "https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker/blob/master/sagemaker-distillation.ipynb",
      "label": "sagemaker-distillation",
      "type": "a"
    },
    {
      "href": "https://discuss.huggingface.co/",
      "label": "forum",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers",
      "label": "Github",
      "type": "a"
    },
    {
      "href": "https://discuss.huggingface.co/c/sagemaker/17",
      "label": "forum",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:50Z",
  "source_url": "https://www.philschmid.de/knowledge-distillation-bert-transformers",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1910.01108.pdf",
      "bytes": 436085,
      "reused": false,
      "local_path": "papers/repair/30e0c28b2251c16c28b7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1910.01108",
      "bytes": 436085,
      "reused": true,
      "local_path": "papers/repair/9d327e2a83cac2e9902e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.13382",
      "bytes": 285692,
      "reused": false,
      "local_path": "papers/repair/53e577fa1c557c8d90a9.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 17230,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2b67b4a16a3e/pages/source.html",
  "local_html": "attachments/2b67b4a16a3e/pages/article.html",
  "body_base_path": "attachments/2b67b4a16a3e/pages/",
  "body_markdown_path": "attachments/2b67b4a16a3e/pages/article.md"
}