{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "NeurIPS 2023 Primer",
  "keywords": "NeurIPS,LLM,论文",
  "domain": "机器学习",
  "content_type": "行业分析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/neurips-2023-primer/",
  "publish_date": "2023-12",
  "corpus_stage": "预训练/SFT",
  "notes": "NeurIPS 2023 LLM相关论文精读",
  "id": "c42b274a9df8",
  "collected_at": "2026-10-02T14:16:28",
  "title": "NeurIPS 2023 Primer",
  "body_markdown": "NeurIPS 2023 Primer\nA round-up of 20 exciting NeurIPS 2023 papers related to LLMs.\nSebastian Ruder\n01 Dec 2023\n•\n12 min read\nShare\nAn overview of the Parsel framework (\nZelikman et al., 2023\n), which decomposes algorithmic reasoning problems using LLMs and external tools.\nNeurIPS 2023\n, arguably this year’s biggest AI conference takes place in two weeks from Dec 10–16 in New Orleans.\n3586 papers\nwere accepted to the conference, which are\navailable online\n.\nIn this post, I’ll discuss a selection of 20 papers related to natural language processing (NLP) that caught my eye, with a focus on oral and spotlight papers. Here are the main trends I observed:\nMost NLP work at NeurIPS is related to large language models (LLMs).\nWhile there are some papers that do not employ LLMs or use a different setting (see\nSuhr & Artzi\nbelow, for instance), papers still presented their contributions in the context of LLMs.\nSynthetic setups to analyze LLM properties are becoming more common.\nThis is because it is computationally prohibitive to run many different pre-training experiments. Investigated properties range from the emergence of in-context learning and learning using global statistics to chain-of-thought reasoning.\nAligning models based on human preferences received a lot of attention.\nPapers particularly focused on improving RLHF and studying alignment to specific personality traits and beliefs.\nA comprehensive understanding of in-context learning still remains elusive.\nPapers studied different aspects of in-context learning such as whether it persists during training and using a Bayesian perspective.\nReasoning is still challenging with current models.\nPapers focused on improving performance on various types of reasoning tasks including pragmatic, graph-based, algorithmic, compositional, and planning-based reasoning.\nExternal tools are increasingly used to improve LLMs’ reasoning abilities.\nThese range from external verifiers to code execution modules.\nNote that some of the methods proposed in these papers such as\nDPO\nand\nQLoRA\nhave already been successfully used in LLM applications.\nThis post was first published in\nNLP News\n.\nRethinking LLMs\nThis is one of my favorite topics as these papers encourage us to rethink our fundamental assumptions regarding LLMs and provide new insights and perspectives on their inner workings.\nLexinvariant Language Models (Huang et al.).\nOne of the fundamental characteristics of LMs that hasn’t changed since the\nfirst neural LM paper\nis the one-to-one mapping between tokens and embeddings. This paper studies whether language modeling can also be done with models that are ‘lexinvariant’, i.e., that do not have fixed token embeddings but assign the\nlexical permutation\nof each sequence the same probability. This seems like a strong limitation—but it can serve as a useful inductive bias for recovering substitution cyphers (via an MLP probe) and in-context symbol manipulation. In practice, tokens are encoded using random Gaussian vectors and sampled so that the same token has the same representation\nwithin\na sequence but different representations\nacross\nsequences.\nWhile this method is mainly of theoretical interest, using it as regularization by using random embeddings only for a subset of tokens improves results on some BIG-bench tasks.\nLearning from Human Feedback\nGiven the proliferation of different pre-trained LLMs, researchers and practitioners are increasingly looking to improve the next step in the LLM pipeline: learning from human feedback, which is important for maximizing performance on downstream tasks but also for LLM alignment.\nDirect Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al.).\nReinforcement learning from human feedback (RLHF) is the preferred approach to update LLMs to align with target preferences but is quite complex (it requires first training a reward model and then updating the LLM with RL based on the reward model) and can be unstable. This paper proposes Direct Preference Optimization (DPO), which shows that the same objective can be optimized via a simple classification-based objective on the preference data—without any RL! An important component of the objective is a dynamic, per-example importance weight.\nDPO has the potential to make aligning LLMs with human preferences much more seamless—and will thus be important for safety research.\nFine-Grained Human Feedback Gives Better Rewards for Language Model Training (Wu et al.).\nThis paper addresses another limitation of RLHF, which does not allow the integration of more fine-grained feedback regarding which parts of the generated response are erroneous. The paper proposes Fine-grained RLHF, which\na)\nuses a dense reward model (a reward for every output sentence rather than for the entire output) and\nb)\nincorporates multiple reward models for diverse feedback. They experiment on detoxification and long-form question answering where they see improved results compared to RLHF and supervised fine-tuning. Importantly, as providing human preference judgements for RLHF is a complex annotation task, providing more fine-grained feedback is actually\nnot\nmore time-intensive.\nExpect to see more approaches experimenting with various reward models at different granularities.\nContinual Learning for Instruction Following from Realtime Feedback (Suhr & Artzi).\nThis paper tackles continual learning from human feedback in a collaborative 3D world environment. They demonstrate a simple approach using a contextual bandit to update the model’s policy using binary rewards. Over 11 rounds of training and deployment, instruction execution accuracy improves from 66.7% to 82.1%. Empirically, in this setting, the feedback data provides a similar amount of learning signal as the supervised data.\nWhile their setting differs from the standard text-based scenario, it provides a sketch for how instruction-following agents can continually learn from human feedback.\nIn the future, we will likely see approaches that utilize more expressive feedback such as via natural language.\nLLM Alignment\nIn order to ensure that LLMs are most useful, it is crucial to align them with the specific guidelines, safety policies, personality traits and beliefs that are relevant for a given downstream setting. To do this, we first need to understand what tendencies LLMs already encode—and then develop methods to steer them appropriately.\nEvaluating and Inducing Personality in Pre-trained Language Models (Jiang et al.).\nThis paper proposes to assess the personality of LLMs based on the\nBig Five personality traits\nknown from psychology. Building on existing questionnaires, they create multiple-choice question answering examples where the LLM must choose how accurately statements such as “You love to help others” describe it. Each statement is associated with a personality trait. Crucially, it is less important whether a model scores highly on a specific trait but whether it exhibits a\nconsistent\npersonality, that is, whether it responds similarly to all questions associated with the trait. Only the largest models exhibit consistent personality traits that are similar to those of humans.\nIt will be interesting to better understand under what conditions personality traits emerge and how consistent personalities can be encoded in smaller models.\nIn-Context Impersonation Reveals Large Language Models' Strengths and Biases (Salewski et al.).\nThere has been a lot of anecdotal evidence that prompting LLMs to impersonate domain experts (e.g., “you are an expert programmer”, etc) improved models’ capabilities. This paper studies such in-context impersonation across different tasks and finds indeed that LLMs impersonating domain experts perform better than LLMs impersonating non-domain experts. Impersonation is also useful to detect implicit biases. For instance, LLMs impersonating a man describe cars better than ones prompted to be a woman (based on CLIP’s ability to match an image to a category using the generated description of the category).\nOverall, impersonation is a useful tool to analyze LLMs—but may reinforce biases when used for (system) prompts.\nEvaluating the Moral Beliefs Encoded in LLMs (Scherer et al.).\nThis paper studies how moral beliefs are encoded in LLMs with regard to both high-ambiguity (“Should I tell a white lie?”) and low-ambiguity scenarios (“Should I stop for a pedestrian on the road?”) scenarios. They evaluate 28 (!) different LLMs and find that\na)\nin unambiguous scenarios most models align with commonsense while in ambiguous cases, most models express uncertainty;\nb)\nmodels are sensitive to the wording of the question; and\nc)\nsome models exhibit clear preferences in ambiguous scenarios—and closed-source models have similar preferences.\nWhile the evaluation relies on a heuristic mapping of output sequences to actions,\nthe data\nis useful for further research of LLMs’ moral beliefs.\nLLM Pre-training\nPre-training is the most compute-intensive part of LLM pipelines and is thus harder to study at scale. Nevertheless, innovations such as new scaling laws improve our understanding of pre-training and inform future training runs.\nScaling Data-Constrained Language Models (Muennighoff et al.).\nAs the amount of text online is limited, this paper investigates scaling laws in data-constrained regimes, in contrast to the scaling laws by\nHoffmann et al. (2022)\n, which focused on scaling without repeating data. The authors observe that training for up to 4 epochs on repeated data performs similarly to training on unique data. With more repetition, however, the value of additional training rapidly diminishes. In addition, augmenting the pre-training data with code meaningfully increases the pre-training data size.\nIn sum, whenever we don’t have infinite amounts of pre-training data, we should train smaller models for more (up to 4) epochs.\nLLM Fine-tuning\nFine-tuning large models with back-propagation is expensive so these papers propose methods to make fine-tuning more efficient, either using\nparameter-efficient fine-tuning\nmethods or without computing gradients (zeroth-order optimization).\nQLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al.).\nThis paper proposes QLoRA, a more memory-efficient (but\nslower\n) version of LoRA that uses several optimization tricks to save memory. They train a new model, Guanaco, that is fine-tuned only on a single GPU for 24h and outperforms previous models on the Vicuna benchmark.\nOverall, QLoRA enables using much fewer GPU memory for fine-tuning LLMs.\nConcurrently, other methods such as\n4-bit LoRA quantization\nhave been developed that achieve similar results.\nFine-Tuning Language Models with Just Forward Passes (Malladi et al.).\nThis paper proposes a memory-efficient zeroth-order optimizer (MeZO) as a more memory-efficient version of a classic zeroth-order optimizer that uses differences of loss values to estimate gradients. In practice, the method achieves similar performance to fine-tuning on several tasks but requires 100x more optimization steps (while being faster at each iteration).\nNevertheless, it is surprising that such zeroth-order optimization works with very large models in the first place, demonstrating the robustness of such models, and is an interesting direction for future work.\nEmergent Abilities and In-context Learning\nCertain abilities of LLMs such as in-context learning and arithmetic reasoning have been shown to be present only in the largest models. It is still unclear how these abilities are acquired during training and what specific properties lead to their emergence, motivating many studies in this area.\nAre Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al.)\n.\nEmergent abilities are abilities that are present in large-scale models but not in smaller models and are hard to predict. Rather than being a product of models’ scaling behavior, this paper argues that emergent abilities are mainly an artifact of the choice of metric used to evaluate them. Specifically, nonlinear and discontinuous metrics can lead to sharp and unpredictable changes in model performance. Indeed, the authors find that when accuracy is changed to a continuous metric for arithmetic tasks where emergent behavior was previously observed, performance improves smoothly instead.\nSo while emergent abilities may still exist, they should be properly controlled and researchers should consider how the chosen metric interacts with the model.\nThe Transient Nature of Emergent In-Context Learning in Transformers (Singh et al.).\nChan et al. (2022)\nhave previously shown that the distributional properties of language data (specifically, ‘burstiness’ and a highly skewed distribution) play an important role in the emergence of in-context learning (ICL) in LLMs. Prior work also generally assumes that once the ICL ability has been acquired, it is retained by the model as learning progresses. This paper uses\nOmniglot\n, a synthetic image few-shot dataset to show cases where ICL emerges—only to be subsequently lost while the loss continues to decrease. On the other hand, L2 regularization seems to help the model retain its ICL ability.\nIt is still unclear, however, how ICL emerges and if transience can be observed during LLM pre-training on real-world natural language data.\nWhy think step by step? Reasoning emerges from the locality of experience (Prystawski et al.).\nThis paper investigates why and how chain-of-thought reasoning\n(Wei et al., 2022)\nis useful in LLMs using a\nsynthetic setup\n. Similar to\nChan et al. (2022)\n, they study distributional properties of the pretraining data. They find that chain-of-thought reasoning is only useful when the training data is locally structured. In other words, when examples are about closely connected topics as is common in natural language. They find that chain-of-thought reasoning is helpful because it incrementally chains local statistical dependencies that are frequently observed in training.\nIt is still unclear, however, when chain-of-thought reasoning emerges during training and what are the properties of downstream tasks where it is most useful.\nLarge Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning (Wang et al.).\nThis paper frames in-context learning with LLMs as topic modeling where the generated tokens are conditioned on a latent topic (concept) variable, which captures format and task information. To make this computationally efficient, they use a smaller LM to learn the latent concepts via prompt tuning on the full demonstration data. They then select the examples that achieve the highest probability under the prompt-tuned model as demonstrations for in-context learning, which improves over other selection baselines.\nThis is a further data point that shows that examples that are probable based on a latent concept of a task are useful demonstrations.\nThis likely isn’t the full picture, however, and it will be interesting to see how this formulation relates to other data similarity and diversity measures.\nBirth of a Transformer: A Memory Viewpoint (Bietti et al.).\nThis study investigates how LLMs learn in-context learning as well as the ability to use more general knowledge in a synthetic setup. The setup consists of sequences generated by a bigram LM where some bigrams require the local context to infer them while others require global statistics. They find that two-layer transformers (but not one-layer transformers) develop an induction head consisting of a “circuit” of two attention heads to predict in-context bigrams. They then freeze some of the layers to study the model’s training dynamics, finding that global bigrams are learned first and that the induction head learns appropriate memories in a top-down fashion.\nOverall, this paper sheds further light on how in-context learning can emerge in LMs.\nReasoning\nReasoning tasks that require the systematic chaining or composition of different pieces of information are one of the most important problems for current LLMs to solve. Their challenging nature and the diversity of domains where they are relevant makes them a fruitful area for research.\nOn the Planning Abilities of Large Language Models - A Critical Investigation (Valmeekam et al.).\nPlanning and sequential decision making are important for a wide range of applications. This paper studies whether LLMs can generate simple plans for\ncommonsense planning tasks\n2\n. Used on their own, only 12% of the plans generated by the best LLM are directly executable. However, when combined with an automated planning algorithm that can identify and remove errors in the LLM-generated plan, LLMs do much better than using a random or empty initial plan. LLMs plans can also be improved via prompting based on feedback from an external verifier. On the other hand, the benefits of LLMs disappear when action names are unintelligible and cannot be easily inferred with common sense, which indicates a lack of abstract reasoning ability.\nOverall, while initial plans of LLMs can be useful as a starting point, LLM-based planning currently works best mainly in conjunction with external tools.\nCan Language Models Solve Graph Problems in Natural Language? (Wang et al.)\nThe authors create\nNLGraph\n, a benchmark of graph-based reasoning problems described in natural language and evaluate LLMs on it. They find that LLMs demonstrate impressive preliminary graph reasoning abilities—37–58% above random baselines. However, in-context learning and advanced prompting strategies (chain-of-thought prompting and others) are mostly ineffective on more complex graph problems and LLMs are susceptible to spurious correlations. More specialized graph prompting strategies, however, improve results.\nExpect to see combinations of standard graph-based methods (such as those applied to\nFB15k\n) and LLMs and research on methods scaling LLMs to larger graphs.\nThe Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs (Ruis et al.).\nThis paper studies whether LLMs exhibit a particular type of pragmatic inference,\nimplicature\n3\n. The authors evaluate whether models can understand implicature by measuring whether they assign a higher probability to a statement that contains the correct inference compared to the incorrect one. They find that both instruction tuning and chain-of-thought prompting are important for such pragmatic understanding and that the largest model, GPT-4 reaches human-level performance.\nWe will likely see more work on different types of pragmatic understanding as these are crucial for seamless and human-like conversations.\nParsel🐍: Algorithmic Reasoning with Language Models by Composing Decompositions (Zelikman et al.).\nThis paper introduces\nParsel\n, a framework to implement complex programs with code LLMs. An LLM is first used to generate natural language function descriptions in a simple intermediate language. For each function description, the model then generates implementation candidates. The function candidates are then tested against input-output constraints and composed to form the final program. Using GPT-4 as LLM, this framework increased performance on HumanEval from 67% to 85%.\nThis is a great example of how LLMs can be used as a building block together with domain-specific knowledge and tools for much improved performance.\nIn addition, expect to see more work on breaking down complex decision problems into subproblems that are more easily solvable using LLMs.\nFaith and Fate: Limits of Transformers on Compositionality (Dziri et al.).\nThis paper investigates the compositional reasoning abilities of LLMs. They formulate compositional reasoning tasks as computation graphs in order to quantify their complexity. They find that full computation subgraphs appear significantly more frequently in the training data for correctly predicted test examples than for incorrectly predicted ones, indicating that models learn to match subgraphs rather than developing systematic reasoning skills. Using a scratchpad and grokking (training beyond overfitting) similarly do not improve performance on more complex problems.\nOverall, current LLMs still struggle with composing operations into correct reasoning paths.",
  "metadata": {
    "title": "NeurIPS 2023 Primer",
    "description": "A round-up of 20 exciting NeurIPS 2023 papers related to LLMs.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "NeurIPS 2023 Primer",
    "og:description": "A round-up of 20 exciting NeurIPS 2023 papers related to LLMs.",
    "og:url": "https://www.ruder.io/neurips-2023-primer/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2023/12/parsel_image.png",
    "og:image:width": "1200",
    "og:image:height": "411",
    "twitter:card": "summary_large_image",
    "twitter:title": "NeurIPS 2023 Primer",
    "twitter:description": "A round-up of 20 exciting NeurIPS 2023 papers related to LLMs.",
    "twitter:url": "https://www.ruder.io/neurips-2023-primer/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2023/12/parsel_image.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2023-12-01T22:33:29.000Z",
    "article:modified_time": "2024-10-23T12:32:20.000Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2023/12/parsel_image.png",
      "bytes": 325338,
      "reused": true,
      "local_path": "attachments/c42b274a9df8/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2023/12/parsel_image.png",
      "alt": "NeurIPS 2023 Primer",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=qd9qcbVAwQ",
      "label": "Zelikman et al., 2023",
      "type": "a"
    },
    {
      "href": "https://nips.cc/?ref=ruder.io",
      "label": "NeurIPS 2023",
      "type": "a"
    },
    {
      "href": "https://blog.neurips.cc/2023/11/20/neurips-newsletter-october-2023/?ref=ruder.io",
      "label": "3586 papers",
      "type": "a"
    },
    {
      "href": "https://openreview.net/group?id=NeurIPS.cc/2023/Conference&ref=ruder.io",
      "label": "available online",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=ez6Cb0ZGzG&ref=ruder.io",
      "label": "Suhr & Artzi",
      "type": "a"
    },
    {
      "href": "https://github.com/eric-mitchell/direct-preference-optimization?ref=ruder.io",
      "label": "DPO",
      "type": "a"
    },
    {
      "href": "https://github.com/artidoro/qlora?ref=ruder.io",
      "label": "QLoRA",
      "type": "a"
    },
    {
      "href": "https://nlpnewsletter.substack.com/p/neurips-2023-primer",
      "label": "NLP News",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=NiQTy0NW1L&ref=ruder.io",
      "label": "Lexinvariant Language Models (Huang et al.).",
      "type": "a"
    },
    {
      "href": "https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf?ref=ruder.io",
      "label": "first neural LM paper",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=HPuSIXJaa9&ref=ruder.io",
      "label": "Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al.).",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=CSbGXyCswu&ref=ruder.io",
      "label": "Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (Wu et al.).",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=I9xE1Jsjfx&ref=ruder.io",
      "label": "Evaluating and Inducing Personality in Pre-trained Language Models (Jiang et al.).",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Big_Five_personality_traits?ref=ruder.io",
      "label": "Big Five personality traits",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=CbsJ53LdKc&ref=ruder.io",
      "label": "In-Context Impersonation Reveals Large Language Models' Strengths and Biases (Salewski et al.).",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=O06z2G18me&ref=ruder.io",
      "label": "Evaluating the Moral Beliefs Encoded in LLMs (Scherer et al.).",
      "type": "a"
    },
    {
      "href": "https://github.com/ninodimontalcino/moralchoice?ref=ruder.io",
      "label": "the data",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=j5BuTrEj35&ref=ruder.io",
      "label": "Scaling Data-Constrained Language Models (Muennighoff et al.).",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.15556?ref=ruder.io",
      "label": "Hoffmann et al. (2022)",
      "type": "a"
    },
    {
      "href": "https://www.modulardeeplearning.com/?ref=ruder.io",
      "label": "parameter-efficient fine-tuning",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=OUIFPHEgJU&ref=ruder.io",
      "label": "QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al.).",
      "type": "a"
    },
    {
      "href": "https://lightning.ai/pages/community/lora-insights/?ref=ruder.io",
      "label": "slower",
      "type": "a"
    },
    {
      "href": "https://github.com/johnsmith0031/alpaca_lora_4bit?ref=ruder.io",
      "label": "4-bit LoRA quantization",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=Vota6rFhBQ&ref=ruder.io",
      "label": "Fine-Tuning Language Models with Just Forward Passes (Malladi et al.).",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=ITw9edRDlD&ref=ruder.io",
      "label": "Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al.)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2311.08360?ref=ruder.io",
      "label": "The Transient Nature of Emergent In-Context Learning in Transformers (Singh et al.).",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.05055?ref=ruder.io",
      "label": "Chan et al. (2022)",
      "type": "a"
    },
    {
      "href": "https://github.com/brendenlake/omniglot?ref=ruder.io",
      "label": "Omniglot",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=rcXXNFVlEn&ref=ruder.io",
      "label": "Why think step by step? Reasoning emerges from the locality of experience (Prystawski et al.).",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2201.11903?ref=ruder.io",
      "label": "(Wei et al., 2022)",
      "type": "a"
    },
    {
      "href": "https://github.com/benpry/why-think-step-by-step?ref=ruder.io",
      "label": "synthetic setup",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2301.11916?ref=ruder.io",
      "label": "Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning (Wa",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=3X2EbBLNsk&ref=ruder.io",
      "label": "Birth of a Transformer: A Memory Viewpoint (Bietti et al.).",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=X6dEqXIsEW&ref=ruder.io",
      "label": "On the Planning Abilities of Large Language Models - A Critical Investigation (Valmeekam et al.).",
      "type": "a"
    },
    {
      "href": "https://github.com/karthikv792/LLMs-Planning?ref=ruder.io",
      "label": "commonsense planning tasks",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=UDqHhbqYJV&ref=ruder.io",
      "label": "Can Language Models Solve Graph Problems in Natural Language? (Wang et al.)",
      "type": "a"
    },
    {
      "href": "https://github.com/Arthur-Heng/NLGraph?ref=ruder.io",
      "label": "NLGraph",
      "type": "a"
    },
    {
      "href": "https://paperswithcode.com/dataset/fb15k?ref=ruder.io",
      "label": "FB15k",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=5bWW9Eop7l&ref=ruder.io",
      "label": "The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs (Ruis et al.)",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Implicature?ref=ruder.io",
      "label": "implicature",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=qd9qcbVAwQ&ref=ruder.io",
      "label": "Parsel🐍: Algorithmic Reasoning with Language Models by Composing Decompositions (Zelikman et al.).",
      "type": "a"
    },
    {
      "href": "https://github.com/ezelikman/parsel?ref=ruder.io",
      "label": "Parsel",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=Fkckkr3ya8&ref=ruder.io",
      "label": "Faith and Fate: Limits of Transformers on Compositionality (Dziri et al.).",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:04Z",
  "source_url": "https://www.ruder.io/neurips-2023-primer/",
  "papers": [
    {
      "status": "ok",
      "url": "https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf?ref=ruder.io",
      "bytes": 140095,
      "reused": false,
      "local_path": "papers/repair/5c1fc9615d60ef133eec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.15556",
      "bytes": 6004349,
      "reused": true,
      "local_path": "papers/repair/70766cef482201671d90.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.08360",
      "bytes": 4340411,
      "reused": true,
      "local_path": "papers/repair/3fca70ba18762ff9bbd8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.05055",
      "bytes": 1345306,
      "reused": true,
      "local_path": "papers/repair/e68190bcb875d901f794.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2201.11903",
      "bytes": 891773,
      "reused": true,
      "local_path": "papers/repair/52cc4c4f6cf368123b97.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2301.11916",
      "bytes": 1488359,
      "reused": false,
      "local_path": "papers/repair/d132425b5d93bd26c86b.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 20323,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c42b274a9df8/pages/source.html",
  "local_html": "attachments/c42b274a9df8/pages/article.html",
  "body_base_path": "attachments/c42b274a9df8/pages/",
  "body_markdown_path": "attachments/c42b274a9df8/pages/article.md"
}