{ "source_file": "0930", "source_platform": "Medium", "author": "Ransaka Ravihara", "title_listed": "How to Train BERT for Masked Language Modeling Tasks", "keywords": "BERT,MLM,掩码语言模型", "domain": "文本处理", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/how-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc/", "publish_date": "2023-10", "corpus_stage": "预训练/SFT", "notes": "", "id": "51e72f20c5f7", "collected_at": "2026-10-01T23:14:12", "title": "How to Train BERT for Masked Language Modeling Tasks | Towards Data Science", "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Programming](https://towardsdatascience.com/category/programming/)\n# How to Train BERT for Masked Language Modeling Tasks\nHands-on guide to building language model for MLM tasks from scratch using Python and Transformers library\n[Ransaka Ravihara](https://towardsdatascience.com/author/ransakaravihara/)\nOctober 17, 20237 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1kHJMclPKwWR8CTeM6CGjyQ.png)\n## Introduction\nIn recent years, large language models(LLMs) have taken all the attention from the machine learning community. Before LLMs came in, we had a crucial research phase on various language modeling techniques, including masked language modeling, causal language modeling, and sequence-to-sequence language modeling.\nFrom the above list, masked language models such as BERT became more usable in downstream NLP tasks such as classification and clustering. Thanks to libraries such as Hugging Face Transformers, adapting these models for downstream tasks became more accessible and manageable. Also thanks to the open-source community, we have plenty of language models to choose from covering widely used languages and domains.\nGitHub repository for this tutorial\n> [**GitHub - Ransaka/Train-BERT-for-Masked-Language-Modeling-Tasks: GitHub repo for TDS article \"How to...**](https://github.com/Ransaka/Train-BERT-for-Masked-Language-Modeling-Tasks)\nLearn this step by step with the interactive [Computer Science](https://roadmap.sh/computer-science) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n## Fine-tune or build one from scratch?\nWhen adapting existing language models to your specific use cases, sometimes we can use existing models without further tuning (so-called fine-tuning). For example, if you want an English sentiment/intent detection model, you can go into HuggingFace.co and find a suitable model for your use case.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1YbYz2uOkPBL3DzpCvQbX-Q-scaled.jpeg)\nHowever, you can only expect this for some of the tasks encountered in the real world. That's where we need an additional technique called fine-tuning. First, you must choose a base model that will be fine-tuned. Here, you must be careful about the selected model and your target language's lexical similarity.\nHowever, if you can't find a suitable model retrained on the desired language, consider building one from scratch. In this tutorial, we will implement the BERT model for the masked language model.\n## BERT Architecture\nEven though describing BERT architecture is out of the scope of this tutorial, for the sake of clarity, let's go through it very narrowly. BERT, or **B** idirectional **E** ncoder **R** epresentations from **T** ransformers, belongs to the encoder-only transformer family. It was introduced in the year 2018 by researchers at Google.\n### Paper abstract:\n> We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement). Paper: \nIn the above, we can see an interesting keyword, Bidirectional. Bidirectional nature gives human-like power to BERT. Assume you have to fill in a blank like the below one,\n> \"War may sometimes be a necessary evil. But no matter how necessary, it is always an _____, never a good.\"\nTo guess the word comes into a blank position, you should be aware of a few things: words before empty, words after blank, and the overall context of the sentence. Adapting this human nature, BERT works the same way. During training, we hide some words and ask BERT to try predicting those. When training is finished, BERT can predict masked tokens based on their before and after words. To do this, the model should allocate different attention to words presented in the input sequence, which may significantly impact predicting masked tokens.\n![Image by Author via https://huggingface.co/spaces/exbert-project/exbert](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/14LriARDWTx6M82sABg2TyQ.png)\nImage by Author via \nAs you can see here, the model sees a suitable word for the hidden position as _evil_ and the first sentence's _evil_ as necessary to make this prediction. This is a noticeable point and implies that the model understands the context of the input sequence. This context awareness allows BERT to generate meaningful sentence embeddings for given tasks. Further, these embeddings can be used in downstream tasks such as clustering and classification. Enough about BERT; let's build one from scratch.\n## Defining BERT model\nWe generally have BERT _(base)_ and BERT _(large)._ Both have 64 dimensions per head. The _large_ variant contains 24 encoder layers, while the _base_ variant only has 12. However, we are not limited to these configurations. Surprisingly, we have complete control over defining the model using Hugging Face Transformers library. All we have to do is define desired model configurations using the _BertConfig_ class.\nI chose 6 heads and 384 total model dimensions to comply with the original implementation. In this way, each head has 64 dimensions similar to the original implementation. Let's initialize our BERT model.\npython\n\n```\nfrom transformers import BertConfig, BertForMaskedLM\n\nconfig = BertConfig(\n hidden_size = 384,\n vocab_size= tokenizer.vocab_size,\n num_hidden_layers = 6,\n num_attention_heads = 6,\n intermediate_size = 1024,\n max_position_embeddings = 256\n)\n\nmodel = BertForMaskedLM(config=config)\nprint(model.num_parameters()) #10457864\n```\n\n## Training a tokenizer\nHere, I won't describe how tokenization works under the hood. Instead, let's train one from scratch using Hugging Face tokenizers library. Please note that the tokenizer used in the original BERT implementation is WordPiece tokenizer, yet another subword-based tokenization method. You can learn more about this tokenization using the neat HuggingFace resource below.\n> [**WordPiece tokenization - Hugging Face NLP Course**](https://huggingface.co/learn/nlp-course/chapter6/6?fw=pt)\nThe dataset used here is the Sinhala-400M dataset _(under apache-2.0)_. You can follow the same with any dataset you have.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/18wN2dxGCI_zMh8VUh1Jktg.png)\nAs you may notice, some Sinhalese words have been typed using English as well. Let's train a tokenizer for these corpora.\nLet's import the necessary modules first. The good thing about training tokenizers using Hugging Face Tokenizers library is that we can use existing tokenizers and replace only vocabulary (and merge where applicable) per our training corpus. This means tokenization steps such as pre-tokenization and post-tokenization will be preserved. For this, we can use a method, _train_new_from _iterator_ BertTokenizer class _._\npython\n\n```\nfrom tokenizers.implementations import ByteLevelBPETokenizer\nfrom tokenizers.processors import BertProcessing\nfrom transformers import AutoTokenizer\nfrom datasets import Dataset\nimport pandas as pd\n\n#load base tokenizer to train on dataset\ntokenizer_base = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n# convert pandas dataset to HF dataset\ndataset = Dataset.from_pandas(df.rename(columns={\"comment\":'text'}))\n\n# define iterator\ntraining_corpus = (\n dataset[i : i + 1000][\"text\"]\n for i in range(0, len(dataset), 1000)\n)\n\n#train the new tokenizer for dataset\ntokenizer = tokenizer_base.train_new_from_iterator(training_corpus, 5000)\n#test trained tokenizer for sample text\ntext = dataset['text'][123]\nprint(text)\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1sEVQuqssU_esL_tDgbNj8A.png)\nbash\n\n```\n# let's check tokenization process\ninput_ids = tokenizer(text).input_ids\nsubword_view = [tokenizer.convert_ids_to_tokens(id) for id in input_ids]\nnp.array(subword_view)\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1i-47yvm5Sdu-BbMgBVRfVA.png)\nYou can see words like 'cricketer' decomposed into _cricket_ and _##er,_ indicating that the tokenizer has been adequately trained. However, try out different vocab sizes; mine is 5000, which is relatively small but suitable for this toy example.\nFinally, we can save the trained tokenizer into our directory.\nbash\n\n```\ntokenizer.save_pretrained(\"tokenizer/sinhala-wordpiece-yt-comments\")\n```\n\n## Define data collator and tokenize dataset.\nLet's define a collator for MLM tasks. Here, we will mask 15% of tokens. Anyway, we can set different masking probabilities.\npython\n\n```\nfrom transformers import DataCollatorForLanguageModeling\n\ndata_collator = DataCollatorForLanguageModeling(\n tokenizer=tokenizer, mlm=True, mlm_probability=0.15\n)\n```\n\nLet's tokenize the dataset using a previously created tokenizer. I'm replacing the original _LineByLineTextDataset_ using my custom class utilizing Hugging Face accelerate.\npython\n\n```\nimport torch\nfrom torch.utils.data import Dataset\nfrom accelerate import Accelerator, DistributedType\n\nclass LineByLineTextDataset(Dataset):\n def __init__(self, tokenizer, raw_datasets, max_length: int):\n self.padding = \"max_length\"\n self.text_column_name = 'text'\n self.max_length = max_length\n self.accelerator = Accelerator(gradient_accumulation_steps=1)\n self.tokenizer = tokenizer\n\n with self.accelerator.main_process_first():\n self.tokenized_datasets = raw_datasets.map(\n self.tokenize_function,\n batched=True,\n num_proc=4,\n remove_columns=[self.text_column_name],\n desc=\"Running tokenizer on dataset line_by_line\",\n )\n self.tokenized_datasets.set_format('torch',columns=['input_ids'],dtype=torch.long)\n\n def tokenize_function(self,examples):\n examples[self.text_column_name] = [\n line for line in examples[self.text_column_name] if len(line[0]) > 0 and not line[0].isspace()\n ]\n return self.tokenizer(\n examples[self.text_column_name],\n padding=self.padding,\n truncation=True,\n max_length=self.max_length,\n return_special_tokens_mask=True,\n )\n def __len__(self):\n return len(self.tokenized_datasets)\n\n def __getitem__(self, i):\n return self.tokenized_datasets[i]\n```\n\nLet's tokenize the dataset.\nmakefile\n\n```\ntokenized_dataset_train = LineByLineTextDataset(\n tokenizer= tokenizer,\n raw_datasets = dataset,\n max_length=256,\n)\n```\n\nAlright, let's code our training loop.\npython\n\n```\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n output_dir=\"./model\",\n overwrite_output_dir=True,\n push_to_hub=True,\n hub_model_id=\"Ransaka/sinhala-bert-yt-comments\",\n num_train_epochs=2,\n per_device_train_batch_size=32,\n save_steps=5_000,\n logging_steps = 1000,\n save_total_limit=2,\n use_mps_device = True, # disable this if you're running non-mac env\n hub_private_repo = False, # please set true if you want to save model privetly\n save_safetensors= True,\n learning_rate = 1e-4,\n report_to='wandb'\n)\n\ntrainer = Trainer(\n model=model,\n args=training_args,\n data_collator=data_collator,\n train_dataset=tokenized_dataset_train\n)\n```\n\nWe can invoke the trainer using its _train()_ method.\nscss\n\n```\ntrainer.train()\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1_qFry17ZiJoMCaaOAhB12Q.png)\nAfter sufficient training, our model can be used for downstream tasks such as zero-shot classification and clustering. You may find the example using this Hugging Face space for more details.\n> [**Sinhala Embedding Space - a Hugging Face Space by Ransaka**](https://huggingface.co/spaces/Ransaka/sinhala-embedding-space)\n## Conclusion\nWith limited resources, pre-trained models may only recognize specific linguistic patterns, but they can still be helpful for particular use cases. It is highly recommended to fine-tune when possible.\n_In this article, all images, unless otherwise noted, are by the author._\n## **References**\n 1. An Explorable BERT - \n 2. BERT Paper - \n 3. Dataset - \n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657265628_qm714e.jpeg)\nWritten By\n## [Ransaka Ravihara](https://towardsdatascience.com/author/ransakaravihara/)\n[See all from Ransaka Ravihara](https://towardsdatascience.com/author/ransakaravihara/)\n[Bert](https://towardsdatascience.com/tag/bert/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Masked Language Model](https://towardsdatascience.com/tag/masked-language-model/)[Programming](https://towardsdatascience.com/tag/programming/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc%2F&title=How%20to%20Train%20BERT%20for%20Masked%20Language%20Modeling%20Tasks)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc%2F&title=How%20to%20Train%20BERT%20for%20Masked%20Language%20Modeling%20Tasks)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc%2F&text=How%20to%20Train%20BERT%20for%20Masked%20Language%20Modeling%20Tasks)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/05/1i7Rx4egymjgvq8r9fMWljA.png)](https://towardsdatascience.com/pandas-for-football-analysis-42c23b252995/)\n## [Pandas for Football Analysis](https://towardsdatascience.com/pandas-for-football-analysis-42c23b252995/)\n[Programming](https://towardsdatascience.com/category/programming/)\nA Merging and Scraping DataFrame example using Football League Data\n[Stephen Fordham](https://towardsdatascience.com/author/stephenfordham/)\nMay 23, 20191 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/04Fw68lkKHlKIGW-Y-scaled.jpg)](https://towardsdatascience.com/how-much-forecasting-performance-do-you-lose-during-model-selection-923889e2f2dc/)\n## [How Much Forecasting Performance Do You Lose During Model Selection?](https://towardsdatascience.com/how-much-forecasting-performance-do-you-lose-during-model-selection-923889e2f2dc/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow often does cross-validation pick the best forecasting model? What happens when it doesn't?\n[Vitor Cerqueira](https://towardsdatascience.com/author/vcerq/)\nJanuary 6, 20235 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0e4_7VxjBzDNel1H--scaled.jpg)](https://towardsdatascience.com/top-7-free-artificial-intelligence-courses-from-the-ivy-league-universities-7c951f787a55/)\n## [Top 7 FREE Artificial Intelligence Courses from the Ivy League Universities](https://towardsdatascience.com/top-7-free-artificial-intelligence-courses-from-the-ivy-league-universities-7c951f787a55/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA curated list of the top AI courses. Learn from the best minds in the field - Be selective with your time, energy and focus.\n[Roman Orac](https://towardsdatascience.com/author/romanorac/)\nOctober 12, 20206 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/1swd_PY6vTCyPnsgBYoFZfA.png)](https://towardsdatascience.com/a-visual-learners-guide-to-explain-implement-and-interpret-principal-component-analysis-cc9b345b75be/)\n## [A Visual Learner's Guide to Explain, Implement and Interpret Principal Component Analysis](https://towardsdatascience.com/a-visual-learners-guide-to-explain-implement-and-interpret-principal-component-analysis-cc9b345b75be/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLinear Algebra for Machine Learning - Covariance Matrix, Eigenvector and Principal Component\n[Destin Gong](https://towardsdatascience.com/author/destingong/)\nJanuary 25, 202311 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/09000CefaibzjFlw9.jpg)](https://towardsdatascience.com/python-exception-testing-clean-and-effective-methods-86799da86b90/)\n## [Python Exception Testing: Clean and Effective Methods](https://towardsdatascience.com/python-exception-testing-clean-and-effective-methods-86799da86b90/)\n[Coding](https://towardsdatascience.com/category/programming/coding/)\nBeyond Basics: Advanced Python Exception Testing for Pytest and Unittest\n[Naomi Kriger](https://towardsdatascience.com/author/naomikriger/)\nJuly 24, 20234 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/0kkl8DFQofxnBUl8u.jpg)](https://towardsdatascience.com/thread-your-python-program-with-two-lines-of-code-3b474407dbb8/)\n## [Thread Your Python Program with Two Lines of Code](https://towardsdatascience.com/thread-your-python-program-with-two-lines-of-code-3b474407dbb8/)\n[Programming](https://towardsdatascience.com/category/programming/)\nSpeed up your program by doing multiple things simultaneously\n[Mike Huls](https://towardsdatascience.com/author/mikehuls/)\nJanuary 10, 20238 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1eg-x0402hJatANCTGY2SeA.png)](https://towardsdatascience.com/how-to-get-a-job-programming-in-julia-7f24a2ee563f/)\n## [How to get a job programming in Julia?](https://towardsdatascience.com/how-to-get-a-job-programming-in-julia-7f24a2ee563f/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nTips to get hired programming with the Julia Language\n[Logan Kilpatrick](https://towardsdatascience.com/author/logankilpatrick/)\nJanuary 19, 20228 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/19NDSXLP81S-jruEcrthXAw-scaled.jpeg)](https://towardsdatascience.com/how-to-effortlessly-handle-class-imbalance-with-python-and-smote-9b715ca8e5a7/)\n## [How to Effortlessly Handle Class Imbalance with Python and SMOTE](https://towardsdatascience.com/how-to-effortlessly-handle-class-imbalance-with-python-and-smote-9b715ca8e5a7/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nClass-imbalanced dataset doesn't have to be a problem.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nDecember 5, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1BELbHSCU7nPS-rJovAT75w.png)](https://towardsdatascience.com/pharma-brand-volume-forecasting-fc85d0c10aaa/)\n## [Pharma Brand Sales Forecasting](https://towardsdatascience.com/pharma-brand-volume-forecasting-fc85d0c10aaa/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nUnderstanding Forecast from scratch to building first machine learning forecast model like a professional!\n[Surbhi Prasad](https://towardsdatascience.com/author/surbhi-prasad/)\nMay 24, 20215 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "metadata": { "title": "How to Train BERT for Masked Language Modeling Tasks | Towards Data Science", "description": "Hands-on guide to building language model for MLM tasks from scratch using Python and Transformers library", "keywords": null, "author": "Ransaka Ravihara", "og:locale": "en_US", "og:type": "article", "og:title": "How to Train BERT for Masked Language Modeling Tasks", "og:description": "Hands-on guide to building language model for MLM tasks from scratch using Python and Transformers library", "og:url": "https://towardsdatascience.com/how-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/14LriARDWTx6M82sABg2TyQ.png", "og:image:width": "1862", "og:image:height": "1378", "og:image:type": "image/png", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Ransaka Ravihara", "twitter:label2": "Est. reading time", "twitter:data2": "7 min read", "article:published_time": "2023-10-17T18:05:16.000Z", "article:modified_time": "2023-10-17T18:05:16.000Z", "article:author": "https://towardsdatascience.com/author/ransakaravihara/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_000.txt", "chars": 338 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_001.txt", "chars": 723 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_002.txt", "chars": 169 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_003.txt", "chars": 69 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_004.txt", "chars": 165 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_005.txt", "chars": 1440 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_006.txt", "chars": 125 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_007.txt", "chars": 723 }, { "language": "txt", "file": "attachments/51e72f20c5f7/code/block_008.txt", "chars": 16 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1kHJMclPKwWR8CTeM6CGjyQ.png", "alt": "", "local_path": "attachments/51e72f20c5f7/images/000.png", "bytes": 67329, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1YbYz2uOkPBL3DzpCvQbX-Q-scaled.jpeg", "alt": "", "local_path": "attachments/51e72f20c5f7/images/001.jpeg", "bytes": 423180, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/14LriARDWTx6M82sABg2TyQ.png", "alt": "Image by Author via https://huggingface.co/spaces/exbert-project/exbert", "local_path": "attachments/51e72f20c5f7/images/002.png", "bytes": 106515, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/05/1i7Rx4egymjgvq8r9fMWljA.png", "alt": "", "local_path": "attachments/51e72f20c5f7/images/003.png", "bytes": 378262, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/04Fw68lkKHlKIGW-Y-scaled.jpg", "alt": "", "local_path": "attachments/51e72f20c5f7/images/004.jpg", "bytes": 712158, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0e4_7VxjBzDNel1H--scaled.jpg", "alt": "", "local_path": "attachments/51e72f20c5f7/images/005.jpg", "bytes": 596266, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/1swd_PY6vTCyPnsgBYoFZfA.png", "alt": "", "local_path": "attachments/51e72f20c5f7/images/006.png", "bytes": 222765, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/09000CefaibzjFlw9.jpg", "alt": "", "local_path": "attachments/51e72f20c5f7/images/007.jpg", "bytes": 129400, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/0kkl8DFQofxnBUl8u.jpg", "alt": "", "local_path": "attachments/51e72f20c5f7/images/008.jpg", "bytes": 36222, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1eg-x0402hJatANCTGY2SeA.png", "alt": "", "local_path": "attachments/51e72f20c5f7/images/009.png", "bytes": 53629, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/19NDSXLP81S-jruEcrthXAw-scaled.jpeg", "alt": "", "local_path": "attachments/51e72f20c5f7/images/010.jpeg", "bytes": 313413, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1BELbHSCU7nPS-rJovAT75w.png", "alt": "", "local_path": "attachments/51e72f20c5f7/images/011.png", "bytes": 20307, "downloaded": true } ], "audio": [], "videos": [ { "src": "https://www.youtube.com/c/TowardsDataScience", "quality": "unavailable", "local_path": "", "note": "无1080p且无720p或下载失败: Command '['/opt/crawl4ai/bin/yt-dlp', '-f', 'bestvideo[height=1080]+bestaudio/best[height=1080]/bestvideo[height=720]+bestaudio/best[height=720]', '--no-playlist', '--max-filesize', '2G', '--socket-timeout', '30', '-o', '/www/wwwroot/dataset/xinxing-zimeiti-corpus/attachments/51e72f20c5f7/videos/%(id)s.%(ext)s', 'https://www.youtube.com/c/TowardsDataScience']' timed out after 240 seconds" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/programming/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/ransakaravihara/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/bert/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/masked-language-model/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/programming/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/python/" }, { "type": "internal", "href": "https://towardsdatascience.com/pandas-for-football-analysis-42c23b252995/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/stephenfordham/" }, { "type": "internal", "href": "https://towardsdatascience.com/how-much-forecasting-performance-do-you-lose-during-model-selection-923889e2f2dc/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/vcerq/" }, { "type": "internal", "href": "https://towardsdatascience.com/top-7-free-artificial-intelligence-courses-from-the-ivy-league-universities-7c951f787a55/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/romanorac/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://github.com/Ransaka/Train-BERT-for-Masked-Language-Modeling-Tasks" }, { "type": "external", "href": "https://roadmap.sh/computer-science" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://arxiv.org/abs/1810.04805" }, { "type": "external", "href": "https://huggingface.co/spaces/exbert-project/exbert" }, { "type": "external", "href": "https://huggingface.co/learn/nlp-course/chapter6/6?fw=pt" }, { "type": "external", "href": "https://huggingface.co/spaces/Ransaka/sinhala-embedding-space" }, { "type": "external", "href": "https://huggingface.co/datasets/Ransaka/Sinhala-400M" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=How+to+Train+BERT+for+Masked+Language+Modeling+Tasks&u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=How+to+Train+BERT+for+Masked+Language+Modeling+Tasks&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc%2F" }, { "type": "external", "href": "https://x.com/share?text=How+to+Train+BERT+for+Masked+Language+Modeling+Tasks&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" }, { "type": "external", "href": "https://bsky.app/profile/towardsdatascience.com" }, { "type": "external", "href": "https://thenewstack.io" }, { "type": "external", "href": "https://roadmap.sh" }, { "type": "external", "href": "https://insightmediagroup.io/creator-network" } ], "repair_version": 2, "repaired_at": "2026-10-14T11:11:49Z", "source_url": "https://towardsdatascience.com/how-to-train-bert-for-masked-language-modeling-tasks-3ccce07c6fdc/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }