{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Fine-tune FLAN-T5 with PEFT",
  "keywords": "FLAN-T5,PEFT,LoRA,微调",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/fine-tune-flan-t5-peft",
  "publish_date": "2023-03",
  "corpus_stage": "预训练/SFT",
  "notes": "用PEFT微调FLAN-T5经典教程",
  "id": "31b9ad9f8fa1",
  "collected_at": "2026-10-02T14:29:07",
  "title": "Efficient Large Language Model training with LoRA and Hugging Face",
  "body_markdown": "Efficient Large Language Model training with LoRA and Hugging Face\nMarch 23, 2023\n10\nminute read\nView Code\nIn this blog, we are going to show you how to apply\nLow-Rank Adaptation of Large Language Models (LoRA)\nto fine-tune FLAN-T5 XXL (11 billion parameters) on a single GPU. We are going to leverage Hugging Face\nTransformers\n,\nAccelerate\n, and\nPEFT\n.\nYou will learn how to:\nSetup Development Environment\nLoad and prepare the dataset\nFine-Tune T5 with LoRA and bnb int-8\nEvaluate & run Inference with LoRA FLAN-T5\nQuick intro: PEFT or Parameter Efficient Fine-tuning\nPEFT\n, or Parameter Efficient Fine-tuning, is a new open-source library from Hugging Face to enable efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters. PEFT currently includes techniques for:\nLoRA:\nLORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS\nPrefix Tuning:\nP-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks\nP-Tuning:\nGPT Understands, Too\nPrompt Tuning:\nThe Power of Scale for Parameter-Efficient Prompt Tuning\nNote: This tutorial was created and run on a g5.2xlarge AWS EC2 Instance, including 1 NVIDIA A10G.\n1. Setup Development Environment\nIn our example, we use the\nPyTorch Deep Learning AMI\nwith already set up CUDA drivers and PyTorch installed. We still have to install the Hugging Face Libraries, including transformers and datasets. Running the following cell will install all the required packages.\nPython\n# install Hugging Face Libraries\n!\npip install\n\"peft==0.2.0\"\n!\npip install\n\"transformers==4.27.2\"\n\"datasets==2.9.0\"\n\"accelerate==0.17.1\"\n\"evaluate==0.4.0\"\n\"bitsandbytes==0.37.1\"\nloralib\n--\nupgrade\n--\nquiet\n# install additional dependencies needed for training\n!\npip install rouge-score tensorboard py7zr\n2. Load and prepare the dataset\nwe will use the\nsamsum\ndataset, a collection of about 16k messenger-like conversations with summaries. Conversations were created and written down by linguists fluent in English.\nPython\n{\n\"id\"\n:\n\"13818513\"\n,\n\"summary\"\n:\n\"Amanda baked cookies and will bring Jerry some tomorrow.\"\n,\n\"dialogue\"\n:\n\"Amanda: I baked cookies. Do you want some?\n\\r\\n\nJerry: Sure!\n\\r\\n\nAmanda: I'll bring you tomorrow :-)\"\n}\nTo load the\nsamsum\ndataset, we use the\nload_dataset()\nmethod from the 🤗 Datasets library.\nPython\nfrom\ndatasets\nimport\nload_dataset\n# Load dataset from the hub\ndataset =\nload_dataset\n(\n\"samsum\"\n)\nprint(\nf\n\"Train dataset size:\n{\nlen(dataset[\n'train'\n])\n}\n\"\n)\nprint(\nf\n\"Test dataset size:\n{\nlen(dataset[\n'test'\n])\n}\n\"\n)\n# Train dataset size: 14732\n# Test dataset size: 819\nTo train our model, we need to convert our inputs (text) to token IDs. This is done by a 🤗 Transformers Tokenizer. If you are not sure what this means, check out\nchapter 6\nof the Hugging Face Course.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer, AutoModelForSeq2SeqLM\nmodel_id=\n\"google/flan-t5-xxl\"\n# Load tokenizer of FLAN-t5-XL\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\nBefore we can start training, we need to preprocess our data. Abstractive Summarization is a text-generation task. Our model will take a text as input and generate a summary as output. We want to understand how long our input and output will take to batch our data efficiently.\nPython\nfrom\ndatasets\nimport\nconcatenate_datasets\nimport\nnumpy\nas\nnp\n# The maximum total input sequence length after tokenization.\n# Sequences longer than this will be truncated, sequences shorter will be padded.\ntokenized_inputs =\nconcatenate_datasets\n([dataset[\n\"train\"\n], dataset[\n\"test\"\n]]).\nmap\n(\nlambda\nx\n:\ntokenizer\n(x[\n\"dialogue\"\n],\ntruncation\n=\nTrue\n),\nbatched\n=\nTrue\n,\nremove_columns\n=[\n\"dialogue\"\n,\n\"summary\"\n])\ninput_lenghts = [len(x)\nfor\nx\nin\ntokenized_inputs[\n\"input_ids\"\n]]\n# take 85 percentile of max length for better utilization\nmax_source_length = int(np.\npercentile\n(input_lenghts,\n85\n))\nprint(\nf\n\"Max source length:\n{\nmax_source_length\n}\n\"\n)\n# The maximum total sequence length for target text after tokenization.\n# Sequences longer than this will be truncated, sequences shorter will be padded.\"\ntokenized_targets =\nconcatenate_datasets\n([dataset[\n\"train\"\n], dataset[\n\"test\"\n]]).\nmap\n(\nlambda\nx\n:\ntokenizer\n(x[\n\"summary\"\n],\ntruncation\n=\nTrue\n),\nbatched\n=\nTrue\n,\nremove_columns\n=[\n\"dialogue\"\n,\n\"summary\"\n])\ntarget_lenghts = [len(x)\nfor\nx\nin\ntokenized_targets[\n\"input_ids\"\n]]\n# take 90 percentile of max length for better utilization\nmax_target_length = int(np.\npercentile\n(target_lenghts,\n90\n))\nprint(\nf\n\"Max target length:\n{\nmax_target_length\n}\n\"\n)\nWe preprocess our dataset before training and save it to disk. You could run this step on your local machine or a CPU and upload it to the\nHugging Face Hub\n.\nPython\ndef\npreprocess_function\n(\nsample\n,\npadding\n=\n\"max_length\"\n):\n# add prefix to the input for t5\ninputs = [\n\"summarize: \"\n+ item\nfor\nitem\nin\nsample[\n\"dialogue\"\n]]\n# tokenize inputs\nmodel_inputs =\ntokenizer\n(inputs,\nmax_length\n=max_source_length,\npadding\n=padding,\ntruncation\n=\nTrue\n)\n# Tokenize targets with the `text_target` keyword argument\nlabels =\ntokenizer\n(\ntext_target\n=sample[\n\"summary\"\n],\nmax_length\n=max_target_length,\npadding\n=padding,\ntruncation\n=\nTrue\n)\n# If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore\n# padding in the loss.\nif\npadding ==\n\"max_length\"\n:\nlabels[\n\"input_ids\"\n] = [\n[(l\nif\nl != tokenizer.pad_token_id\nelse\n-\n100\n)\nfor\nl\nin\nlabel]\nfor\nlabel\nin\nlabels[\n\"input_ids\"\n]\n]\nmodel_inputs[\n\"labels\"\n] = labels[\n\"input_ids\"\n]\nreturn\nmodel_inputs\ntokenized_dataset = dataset.\nmap\n(preprocess_function,\nbatched\n=\nTrue\n,\nremove_columns\n=[\n\"dialogue\"\n,\n\"summary\"\n,\n\"id\"\n])\nprint(\nf\n\"Keys of tokenized dataset:\n{\nlist(tokenized_dataset[\n'train'\n].features)\n}\n\"\n)\n# save datasets to disk for later easy loading\ntokenized_dataset[\n\"train\"\n].\nsave_to_disk\n(\n\"data/train\"\n)\ntokenized_dataset[\n\"test\"\n].\nsave_to_disk\n(\n\"data/eval\"\n)\n3. Fine-Tune T5 with LoRA and bnb int-8\nIn addition to the LoRA technique, we will use\nbitsanbytes LLM.int8()\nto quantize out frozen LLM to int8. This allows us to reduce the needed memory for FLAN-T5 XXL ~4x.\nThe first step of our training is to load the model. We are going to use\nphilschmid/flan-t5-xxl-sharded-fp16\n, which is a sharded version of\ngoogle/flan-t5-xxl\n. The sharding will help us to not run off of memory when loading the model.\nPython\nfrom\ntransformers\nimport\nAutoModelForSeq2SeqLM\n# huggingface hub model id\nmodel_id =\n\"philschmid/flan-t5-xxl-sharded-fp16\"\n# load model from the hub\nmodel = AutoModelForSeq2SeqLM.\nfrom_pretrained\n(model_id,\nload_in_8bit\n=\nTrue\n,\ndevice_map\n=\n\"auto\"\n)\nNow, we can prepare our model for the LoRA int-8 training using\npeft\n.\nPython\nfrom\npeft\nimport\nLoraConfig, get_peft_model, prepare_model_for_int8_training, TaskType\n# Define LoRA Config\nlora_config =\nLoraConfig\n(\nr\n=\n16\n,\nlora_alpha\n=\n32\n,\ntarget_modules\n=[\n\"q\"\n,\n\"v\"\n],\nlora_dropout\n=\n0.05\n,\nbias\n=\n\"none\"\n,\ntask_type\n=TaskType.\nSEQ_2_SEQ_LM\n)\n# prepare int-8 model for training\nmodel =\nprepare_model_for_int8_training\n(model)\n# add LoRA adaptor\nmodel =\nget_peft_model\n(model, lora_config)\nmodel.\nprint_trainable_parameters\n()\n# trainable params: 18874368 || all params: 11154206720 || trainable%: 0.16921300163961817\nAs you can see, here we are only training 0.16% of the parameters of the model! This huge memory gain will enable us to fine-tune the model without memory issues.\nNext is to create a\nDataCollator\nthat will take care of padding our inputs and labels. We will use the\nDataCollatorForSeq2Seq\nfrom the 🤗 Transformers library.\nPython\nfrom\ntransformers\nimport\nDataCollatorForSeq2Seq\n# we want to ignore tokenizer pad token in the loss\nlabel_pad_token_id = -\n100\n# Data collator\ndata_collator =\nDataCollatorForSeq2Seq\n(\ntokenizer,\nmodel\n=model,\nlabel_pad_token_id\n=label_pad_token_id,\npad_to_multiple_of\n=\n8\n)\nThe last step is to define the hyperparameters (\nTrainingArguments\n) we want to use for our training.\nPython\nfrom\ntransformers\nimport\nSeq2SeqTrainer, Seq2SeqTrainingArguments\noutput_dir=\n\"lora-flan-t5-xxl\"\n# Define training args\ntraining_args =\nSeq2SeqTrainingArguments\n(\noutput_dir\n=output_dir,\nauto_find_batch_size\n=\nTrue\n,\nlearning_rate\n=\n1e-3\n,\n# higher learning rate\nnum_train_epochs\n=\n5\n,\nlogging_dir\n=\nf\n\"\n{\noutput_dir\n}\n/logs\"\n,\nlogging_strategy\n=\n\"steps\"\n,\nlogging_steps\n=\n500\n,\nsave_strategy\n=\n\"no\"\n,\nreport_to\n=\n\"tensorboard\"\n,\n)\n# Create Trainer instance\ntrainer =\nSeq2SeqTrainer\n(\nmodel\n=model,\nargs\n=training_args,\ndata_collator\n=data_collator,\ntrain_dataset\n=tokenized_dataset[\n\"train\"\n],\n)\nmodel.config.use_cache =\nFalse\n# silence the warnings. Please re-enable for inference!\nLet's now train our model and run the cells below. Note that for T5, some layers are kept in\nfloat32\nfor stability purposes.\nPython\n# train model\ntrainer.\ntrain\n()\nThe training took ~10:36:00 and cost\n~13.22$\nfor 10h of training. For comparison a\nfull fine-tuning on FLAN-T5-XXL\nwith the same duration (10h) requires 8x A100 40GBs and costs ~322$.\nWe can save our model to use it for inference and evaluate it. We will save it to disk for now, but you could also upload it to the\nHugging Face Hub\nusing the\nmodel.push_to_hub\nmethod.\nPython\n# Save our LoRA model & tokenizer results\npeft_model_id=\n\"results\"\ntrainer.model.\nsave_pretrained\n(peft_model_id)\ntokenizer.\nsave_pretrained\n(peft_model_id)\n# if you want to save the base model to call\n# trainer.model.base_model.save_pretrained(peft_model_id)\nOur LoRA checkpoint is only 84MB small and includes all of the learnt knowleddge for samsum.\n4. Evaluate & run Inference with LoRA FLAN-T5\nWe are going to use\nevaluate\nlibrary to evaluate the\nrogue\nscore. We can run inference using\nPEFT\nand\ntransformers\n. For our FLAN-T5 XXL model, we need at least 18GB of GPU memory.\nPython\nimport\ntorch\nfrom\npeft\nimport\nPeftModel, PeftConfig\nfrom\ntransformers\nimport\nAutoModelForSeq2SeqLM, AutoTokenizer\n# Load peft config for pre-trained checkpoint etc.\npeft_model_id =\n\"results\"\nconfig = PeftConfig.\nfrom_pretrained\n(peft_model_id)\n# load base LLM model and tokenizer\nmodel = AutoModelForSeq2SeqLM.\nfrom_pretrained\n(config.base_model_name_or_path,\nload_in_8bit\n=\nTrue\n,\ndevice_map\n={\n\"\"\n:\n0\n})\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(config.base_model_name_or_path)\n# Load the Lora model\nmodel = PeftModel.\nfrom_pretrained\n(model, peft_model_id,\ndevice_map\n={\n\"\"\n:\n0\n})\nmodel.\neval\n()\nprint(\n\"Peft model loaded\"\n)\nLet’s load the dataset again with a random sample to try the summarization.\nPython\nfrom\ndatasets\nimport\nload_dataset\nfrom\nrandom\nimport\nrandrange\n# Load dataset from the hub and get a sample\ndataset =\nload_dataset\n(\n\"samsum\"\n)\nsample = dataset[\n'test'\n][\nrandrange\n(len(dataset[\n\"test\"\n]))]\ninput_ids =\ntokenizer\n(sample[\n\"dialogue\"\n],\nreturn_tensors\n=\n\"pt\"\n,\ntruncation\n=\nTrue\n).input_ids.\ncuda\n()\n# with torch.inference_mode():\noutputs = model.\ngenerate\n(\ninput_ids\n=input_ids,\nmax_new_tokens\n=\n10\n,\ndo_sample\n=\nTrue\n,\ntop_p\n=\n0.9\n)\nprint(\nf\n\"input sentence:\n{\nsample[\n'dialogue'\n]\n}\n\\n\n{\n'---'\n*\n20}\n\"\n)\nprint(\nf\n\"summary:\n\\n\n{\ntokenizer.\nbatch_decode\n(outputs.\ndetach\n().\ncpu\n().\nnumpy\n(),\nskip_special_tokens\n=\nTrue\n)[\n0\n]\n}\n\"\n)\nNice! our model works! Now, lets take a closer look and evaluate it against the\ntest\nset of processed dataset from\nsamsum\n. Therefore we need to use and create some utilities to generate the summaries and group them together. The most commonly used metrics to evaluate summarization task is\nrogue_score\nshort for Recall-Oriented Understudy for Gisting Evaluation). This metric does not behave like the standard accuracy: it will compare a generated summary against a set of reference summaries.\nPython\nimport\nevaluate\nimport\nnumpy\nas\nnp\nfrom\ndatasets\nimport\nload_from_disk\nfrom\ntqdm\nimport\ntqdm\n# Metric\nmetric = evaluate.\nload\n(\n\"rouge\"\n)\ndef\nevaluate_peft_model\n(\nsample\n,\nmax_target_length\n=\n50\n):\n# generate summary\noutputs = model.\ngenerate\n(\ninput_ids\n=sample[\n\"input_ids\"\n].\nunsqueeze\n(\n0\n).\ncuda\n(),\ndo_sample\n=\nTrue\n,\ntop_p\n=\n0.9\n,\nmax_new_tokens\n=max_target_length)\nprediction = tokenizer.\ndecode\n(outputs[\n0\n].\ndetach\n().\ncpu\n().\nnumpy\n(),\nskip_special_tokens\n=\nTrue\n)\n# decode eval sample\n# Replace -100 in the labels as we can't decode them.\nlabels = np.\nwhere\n(sample[\n'labels'\n] != -\n100\n, sample[\n'labels'\n], tokenizer.pad_token_id)\nlabels = tokenizer.\ndecode\n(labels,\nskip_special_tokens\n=\nTrue\n)\n# Some simple post-processing\nreturn\nprediction, labels\n# load test dataset from distk\ntest_dataset =\nload_from_disk\n(\n\"data/eval/\"\n).\nwith_format\n(\n\"torch\"\n)\n# run predictions\n# this can take ~45 minutes\npredictions, references = [] , []\nfor\nsample\nin\ntqdm\n(test_dataset):\np,l =\nevaluate_peft_model\n(sample)\npredictions.\nappend\n(p)\nreferences.\nappend\n(l)\n# compute metric\nrogue = metric.\ncompute\n(\npredictions\n=predictions,\nreferences\n=references,\nuse_stemmer\n=\nTrue\n)\n# print results\nprint(\nf\n\"Rogue1:\n{\nrogue[\n'rouge1'\n]*\n100\n:2f\n}\n%\"\n)\nprint(\nf\n\"rouge2:\n{\nrogue[\n'rouge2'\n]*\n100\n:2f\n}\n%\"\n)\nprint(\nf\n\"rougeL:\n{\nrogue[\n'rougeL'\n]*\n100\n:2f\n}\n%\"\n)\nprint(\nf\n\"rougeLsum:\n{\nrogue[\n'rougeLsum'\n]*\n100\n:2f\n}\n%\"\n)\n# Rogue1: 50.386161%\n# rouge2: 24.842412%\n# rougeL: 41.370130%\n# rougeLsum: 41.394230%\nOur PEFT fine-tuned FLAN-T5-XXL achieved a rogue1 score of\n50.38%\non the test dataset. For comparison a\nfull fine-tuning of flan-t5-base achieved a rouge1 score of 47.23\n. That is a\n3%\nimprovements.\nIt is incredible to see that our LoRA checkpoint is only 84MB small and model achieves better performance than a smaller fully fine-tuned model.\nThanks for reading! If you have any questions, feel free to contact me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Efficient Large Language Model training with LoRA and Hugging Face",
    "description": "Learn how to fine-tune Google's FLAN-T5 XXL on a Single GPU using LoRA And Hugging Face Transformers.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Efficient Large Language Model training with LoRA and Hugging Face",
    "og:description": "Learn how to fine-tune Google's FLAN-T5 XXL on a Single GPU using LoRA And Hugging Face Transformers.",
    "og:url": "https://www.philschmid.de/fine-tune-flan-t5-peft",
    "og:image": "https://www.philschmid.de/static/blog/fine-tune-flan-t5-peft/thumbnail.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Efficient Large Language Model training with LoRA and Hugging Face",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Efficient Large Language Model training with LoRA and Hugging Face",
    "twitter:description": "Learn how to fine-tune Google's FLAN-T5 XXL on a Single GPU using LoRA And Hugging Face Transformers.",
    "twitter:image": "https://www.philschmid.de/static/blog/fine-tune-flan-t5-peft/thumbnail.png",
    "article:published_time": "2023-03-23"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/31b9ad9f8fa1/code/body-000.txt",
      "chars": 298,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-001.txt",
      "chars": 210,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-002.txt",
      "chars": 259,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-003.txt",
      "chars": 178,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-004.txt",
      "chars": 1198,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-005.txt",
      "chars": 1211,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-006.txt",
      "chars": 246,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-007.txt",
      "chars": 529,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-008.txt",
      "chars": 285,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-009.txt",
      "chars": 700,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-010.txt",
      "chars": 29,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-011.txt",
      "chars": 254,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-012.txt",
      "chars": 614,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-013.txt",
      "chars": 592,
      "language": "text"
    },
    {
      "file": "attachments/31b9ad9f8fa1/code/body-014.txt",
      "chars": 1457,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/peft-flan-t5-int8-summarization.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2106.09685",
      "label": "Low-Rank Adaptation of Large Language Models (LoRA)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/index",
      "label": "Transformers",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/accelerate/index",
      "label": "Accelerate",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/peft",
      "label": "PEFT",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2106.09685.pdf",
      "label": "LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2110.07602.pdf",
      "label": "P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2103.10385.pdf",
      "label": "GPT Understands, Too",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2104.08691.pdf",
      "label": "The Power of Scale for Parameter-Efficient Prompt Tuning",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-pytorch.html",
      "label": "PyTorch Deep Learning AMI",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/samsum",
      "label": "samsum",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/course/chapter6/1?fw=tf",
      "label": "chapter 6",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/hub/datasets-overview",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/hf-bitsandbytes-integration",
      "label": "bitsanbytes LLM.int8()",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16",
      "label": "philschmid/flan-t5-xxl-sharded-fp16",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/google/flan-t5-xxl",
      "label": "google/flan-t5-xxl",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/fine-tune-flan-t5-deepspeed",
      "label": "full fine-tuning on FLAN-T5-XXL",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/hub/main",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/ROUGE_(metric)",
      "label": "rogue_score",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/fine-tune-flan-t5",
      "label": "full fine-tuning of flan-t5-base achieved a rouge1 score of 47.23",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:46Z",
  "source_url": "https://www.philschmid.de/fine-tune-flan-t5-peft",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2106.09685",
      "bytes": 1609513,
      "reused": true,
      "local_path": "papers/repair/e7d2fed957472d7677ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2106.09685.pdf",
      "bytes": 1609513,
      "reused": false,
      "local_path": "papers/repair/a008d2dd68ac12ed789d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2110.07602.pdf",
      "bytes": 695040,
      "reused": false,
      "local_path": "papers/repair/50b146984aa2fc05603f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.10385.pdf",
      "bytes": 1583446,
      "reused": false,
      "local_path": "papers/repair/dbf7fff0aafc302632ac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2104.08691.pdf",
      "bytes": 547768,
      "reused": false,
      "local_path": "papers/repair/0b068a11229019c7a2ca.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 13585,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/31b9ad9f8fa1/pages/source.html",
  "local_html": "attachments/31b9ad9f8fa1/pages/article.html",
  "body_base_path": "attachments/31b9ad9f8fa1/pages/",
  "body_markdown_path": "attachments/31b9ad9f8fa1/pages/article.md"
}