{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "Open challenges in LLM research",
  "keywords": "LLM,研究挑战,前沿",
  "domain": "大模型算法",
  "content_type": "观点文章",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2023/08/16/llm-research-open-challenges.html",
  "publish_date": "2023-08",
  "corpus_stage": "预训练/SFT",
  "notes": "LLM研究开放挑战梳理",
  "id": "d721f7c874fe",
  "collected_at": "2026-10-02T14:14:08",
  "title": "Open challenges in LLM research",
  "body_markdown": "[\nLinkedIn discussion\n,\nTwitter thread\n]\nNever before in my life had I seen so many smart people working on the same goal: making LLMs better. After talking to many people working in both industry and academia, I noticed the 10 major research directions that emerged. The first two directions, hallucinations and context learning, are probably the most talked about today. I’m the most excited about numbers 3 (multimodality), 5 (new architecture), and 6 (GPU alternatives).\n1. Reduce and measure hallucinations\nHallucination\nis a heavily discussed topic already so I’ll be quick. Hallucination happens when an AI model makes stuff up. For many creative use cases, hallucination is a feature. However, for most other use cases, hallucination is a bug. I was at a panel on LLM with Dropbox, Langchain, Elastics, and Anthropic recently, and the #1 roadblock they see for companies to adopt LLMs in production is hallucination.\nMitigating hallucination and developing metrics to measure hallucination is a blossoming research topic, and I’ve seen many startups focus on this problem. There are also ad-hoc tips to reduce hallucination, such as adding more context to the prompt, chain-of-thought, self-consistency, or asking your model to be concise in its response.\nTo learn more about hallucination:\nSurvey of Hallucination in Natural Language Generation\n(Ji et al., 2022)\nHow Language Model Hallucinations Can Snowball\n(Zhang et al., 2023)\nA Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity\n(Bang et al., 2023)\nContrastive Learning Reduces Hallucination in Conversations\n(Sun et al., 2022)\nSelf-Consistency Improves Chain of Thought Reasoning in Language Models\n(Wang et al., 2022)\nSelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models\n(​​Manakul et al., 2023)\nA simple example of fact-checking and hallucination by\nNVIDIA’s NeMo-Guardrails\n2. Optimize context length and context construction\nA vast majority of questions require context. For example, if we ask ChatGPT: “What’s the best Vietnamese restaurant?”, the context needed would be “where” because the best Vietnamese restaurant in Vietnam would be different from the best Vietnamese in the US.\nAccording to this cool paper\nSituatedQA\n(Zhang & Choi, 2021), a significant proportion of information-seeking questions have context-dependent answers, e.g. roughly 16.5% of the\nNatural Questions NQ-Open dataset\n. Personally, I suspect that this percentage would be even higher for enterprise use cases. For example, say a company builds a chatbot for customer support, for this chatbot to answer any customer question about any product, the context needed might be that customer’s history or that product’s information.\nBecause the model “learns” from the context provided to it, this process is also called context learning.\nContext length is especially important for RAG –\nRetrieval Augmented Generation\n(Lewis et al., 2020) – which has emerged to be the predominant pattern for LLM industry use cases. For those not yet swept away in the RAG rage, RAG works in two phases:\nPhase 1: chunking (also known as indexing)\nGather all the documents you want your LLM to use\nDivide these documents into chunks that can be fed into your LLM to generate embeddings and store these embeddings in a vector database.\nPhase 2: querying\nWhen user sends a query, like “\nDoes my insurance policy pay for this drug X\n”, your LLM converts this query into an embedding, let’s call it QUERY_EMBEDDING\nYour vector database fetches the chunks whose embeddings are the most similar to QUERY_EMBEDDING\nScreenshot from\nJerry Liu’s talk on LlamaIndex\n(2023)\nThe longer the context length, the more chunks we can squeeze into the context. The more information the model has access to, the better its response will be, right?\nNot always. How much context a model can use and how efficiently that model will use it are two different questions. In parallel with the effort to increase model context length is the effort to make the context more efficient. Some people call it “prompt engineering” or “prompt construction”. For example, a paper that has made the rounds recently is about how models are much better at understanding information at the beginning and the end of the index rather than in the middle of it –\nLost in the Middle: How Language Models Use Long Contexts\n(Liu et al., 2023).\n3. Incorporate other data modalities\nMultimodality, IMO, is so powerful and yet so underrated. There are many reasons for multimodality.\nFirst, there are many use cases where multimodal data is required, especially in industries that deal with a mixture of data modalities such as healthcare, robotics, e-commerce, retail, gaming, entertainment, etc. Examples:\nOftentimes, medical predictions require both text (e.g. doctor’s notes, patients’ questionnaires) and images (e.g. CT, X-ray, MRI scans).\nProduct metadata often contains images, videos, descriptions, and even tabular data (e.g. production date, weight, color). You might want to automatically fill in missing product information based on users’ reviews or product photos. You might want to enable users to search for products using visual information, like shape or color.\nSecond, multimodality promises a big boost in model performance. Shouldn’t a model that can understand both text and images perform better than a model that can only understand text? Text-based models require so much text that there’s a realistic concern that\nwe’ll soon run out of Internet data to train text-based models\n. Once we run out of text, we’d need to leverage other data modalities.\nFlamingo architecture (Alayrac et al., 2022)\nOne use case I’m especially excited about is that multimodality can enable visually impaired people to browse the Internet and navigate the real world.\nCool multimodal work:\n[CLIP] Learning Transferable Visual Models From Natural Language Supervision\n(OpenAI, 2021)\nFlamingo: a Visual Language Model for Few-Shot Learning\n(DeepMind, 2022)\nBLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models\n(Salesforce, 2023)\nKOSMOS-1: Language Is Not All You Need: Aligning Perception with Language Models\n(Microsoft, 2023)\nPaLM-E: An embodied multimodal language model\n(Google, 2023)\nLLaVA: Visual Instruction Tuning\n(Liu et al., 2023)\nNeVA: NeMo Vision and Language Assistant\n(NVIDIA, 2023)\nI’ve been working on a post on multimodality that hopefully I can share soon!\n4. Make LLMs faster and cheaper\nWhen GPT-3.5 first came out in late November 2022, many people had concerns about latency and cost of using it in production. However, latency/cost analysis has changed rapidly since then. Within half a year, the community found a way to create a model that came pretty close to GPT-3.5 in terms of performance, yet required just under 2% of GPT-3.5’s memory footprint.\nMy takeaway: if you create something good enough, people will figure out a way to make it fast and cheap.\nDate\nModel\n# params\nQuantization\nMemory to finetune\nCan be trained on\nNov 2022\nGPT-3.5\n175B\n16-bit\n375GB\nMany, many machines\nMar 2023\nAlpaca 7B\n7B\n16-bit\n15GB\nGaming desktop\nMay 2023\nGuanaco 7B\n7B\n4-bit\n6GB\nAny Macbook\nBelow is Guanaco 7B’s performance compared to ChatGPT GPT-3.5 and GPT-4, as reported in the Guanco paper. Caveat: in general, the performance comparison is far from perfect. LLM evaluation is very, very hard.\nFour years ago, when I started working on the notes that would later become the section\nModel Compression\nfor the book\nDesigning Machine Learning Systems\n, I wrote about four major techniques for model optimization/compression:\nQuantization\n: by far the most general model optimization method. Quantization reduces a model’s size by using fewer bits to represent its parameters, e.g. instead of using 32 bits to represent a float, use only 16 bits, or even 4 bits.\nKnowledge distillation\n: a method in which a small model (student) is trained to mimic a larger model or ensemble of models (teacher).\nLow-rank factorization\n: the key idea here is to replace high-dimensional tensors with lower-dimensional tensors to reduce the number of parameters. For example, you can decompose a 3x3 tensor into the product of a 3x1 and a 1x3 tensor, so that instead of having 9 parameters, you have only 6 parameters.\nPruning\nAll these four techniques are still relevant and popular today. Alpaca was trained using knowledge distillation. QLoRA used a combination of low-rank factorization and quantization.\n5. Design a new model architecture\nSince AlexNet in 2012, we’ve seen many architectures go in and out of fashion, including LSTM, seq2seq. Compared to those, Transformer is incredibly sticky. It’s been around since 2017. It’s a big question mark how much longer this architecture will be in vogue.\nDeveloping a new architecture to outperform Transformer isn’t easy. Transformer has been so heavily optimized over the last 6 years. This new architecture has to be performing at the scale that people care about today, on the hardware that people care about. Side note:\nTransformer was originally designed by Google to run fast on TPUs\n, and only later optimized on GPUs.\nThere was a lot of excitement in 2021 around S4 from Chris Ré’s lab – see\nEfficiently Modeling Long Sequences with Structured State Spaces\n(Gu et al., 2021). I’m not quite sure what happened to it. Chris Ré’s lab is still very invested in developing new architecture, most recently with their architecture\nMonarch Mixer\n(Fu et al., 2023) in collaboration with the startup\nTogether\n.\nTheir key idea is that for the existing Transformer architecture, the complexity of attention is quadratic in sequence length and the complexity of an MLP is quadratic in model dimension. An architecture with subquadratic complexity would be more efficient.\nI’m sure many other labs are working on this idea, though I’m not aware of any attempt that has been made public. If you know of any, please let me know!\n6. Develop GPU alternatives\nGPU has been the dominating hardware for deep learning ever since AlexNet in 2012. In fact, one commonly acknowledged reason for AlexNet’s popularity is that it was the first paper to successfully use GPUs to train neural networks. Before GPUs, if you wanted to train a model at AlexNet’s scale, you’d have to use thousands of CPUs, like the one\nGoogle released just a few months before AlexNet\n. Compared to thousands of CPUs, a couple of GPUs were a lot more accessible to Ph.D. students and researchers, setting off the deep learning research boom.\nIn the last decade, many, many companies, both big corporations, and startups, have attempted to create new hardware for AI. The most notable attempts are Google’s\nTPUs\n, Graphcore’s\nIPUs\n(what’s happening with IPUs?), and\nCerebras\n. SambaNova raised over\na billion dollars to develop new AI chips\nbut seems to have pivoted to being a generative AI platform.\nFor a while, there has been a lot of anticipation around quantum computing, with key players being:\nIBM’s QPU\nGoogle’s Quantum computer reported\na major milestone in quantum error reduction\nearlier this year in Nature. Its quantum virtual machine is publicly accessible via\nGoogle Colab\nResearch labs such as\nMIT Center for Quantum Engineering\n,\nMax Planck Institute of Quantum Optics\n,\nChicago Quantum Exchange\n,\nOak Ridge National Laboratory\n, etc.\nAnother direction that is also super exciting is photonic chips. This is the direciton I know the least about – so please correct me if I’m wrong. Existing chips today use electricity to move data, which consumes a lot of power and also incurs latency. Photonic chips use photons to move data, harnessing the speed of light for faster and more efficient compute. Various startups in this space have raised hundreds of millions of dollars, including\nLightmatter\n($270M),\nAyar Labs\n($220M),\nLightelligence\n($200M+), and\nLuminous Computing\n($115M).\nBelow is the timeline of advances of the three major methods in photonic matrix computation, from the paper\nPhotonic matrix multiplication lights up photonic accelerator and beyond\n(Zhou et al., Nature 2022). The three different methods are plane light conversion (PLC), Mach–Zehnder interferometer (MZI), and wavelength division multiplexing (WDM).\n7. Make agents usable\nAgents are LLMs that can take actions, like browsing the Internet, sending emails, making reservations, etc. Compared to other research directions in this post, this might be the youngest direction.\nBecause of the novelty and the massive potential, there’s a feverish obsession with agents.\nAuto-GPT\nis now the 25th most popular GitHub repo ever by the number of stars.\nGPT-Engineering\nis another popular repo.\nDespite the excitement, there is still doubt about whether LLMs are reliable and performant enough to be entrusted with the power to act.\nOne use case that has emerged though is the use of agents for social studies, like the famous Stanford experiment that shows that a small society of generative agents produces emergent social behaviors:\nfor example, starting with only a single user-specified notion that one agent wants to throw a Valentine’s Day party, the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party …\n(\nGenerative Agents: Interactive Simulacra of Human Behavior\n, Park et al., 2023)\nThe most notable startup in this area is perhaps Adept, founded by two Transformer co-authors (though\nboth already left\n) and an ex-OpenAI VP, and has raised almost half a billion dollars to date. Last year, they had a demo showing their agent browsing the Internet and adding a new account to Salesforce. I’m looking forward to seeing their new demos 🙂\n8. Improve learning from human preference\nRLHF, Reinforcement Learning from Human Preference\n, is cool but kinda hacky. I wouldn’t be surprised if people figure out a better way to train LLMs. There are many open questions for RLHF, such as:\n1. How to mathematically represent human preference?\nCurrently, human preference is determined by comparison: human labeler determines if response A is better than response B. However, it doesn’t take into account how much better response A is than response B.\n2. What’s human preference?\nAnthropic measured the quality of their model’s responses along the three axes: helpful, honest, and harmless. See\nConstitutional AI: Harmlessness from AI Feedback\n(Bai et al., 2022).\nDeepMind tries to generate responses that please the most people. See\nFine-tuning language models to find agreement among humans with diverse preferences\n, (Bakker et al., 2022).\nAlso, do we want AIs that can take a stand or a vanilla AI that shies away from any potentially controversial topic?\n3. Whose preference is “human” preference, taking into account the differences in cultures, religions, political leanings, etc.?\nThere are a lot of challenges in obtaining training data that can be sufficiently representative of all the potential users.\nFor example, for OpenAI’s InstructGPT data, there was no labeler above 65 years old. Labelers are predominantly Filipino and Bangladeshi. See\nInstructGPT: Training language models to follow instructions with human feedback\n(Ouyang et al., 2022).\nCommunity-led efforts, while admirable in their intention, can lead to biased data. For example, for the OpenAssistant dataset, 201 out of 222 (90.5%) respondents identify as male.\nJeremy Howard has a great Twitter thread on this\n.\n9. Improve the efficiency of the chat interface\nEver since ChatGPT, there have been multiple discussions on whether chat is a suitable interface for a wide range of tasks.\nNatural language is the lazy user interface\n(Austin Z. Henley, 2023)\nWhy Chatbots Are Not the Future\n(Amelia Wattenberger, 2023)\nWhat Types of Questions Require Conversation to Answer? A Case Study of AskReddit Questions\n(Huang et al., 2023)\nAI chat interfaces could become the primary user interface to read documentation\n(Tom Johnson, 2023)\nInteracting with LLMs with Minimal Chat\n(Eugene Yan, 2023)\nHowever, this is not a new discussion. In many countries, especially in Asia, chat has been used as the interface for super apps for about a decade.\nDan Grover had this discussion back in 2014\n.\nChat as a universal interface for Chinese apps (Dan Grover, 2014)\nThe discussion again got tense in 2016, when many people thought apps were dead and chatbots would be the future.\nOn chat as interface\n(Alistair Croll, 2016)\nIs the Chatbot Trend One Big Misunderstanding?\n(Will Knight, 2016)\nBots won’t replace apps. Better apps will replace apps\n(Dan Grover, 2016)\nPersonally, I love the chat interface because of the following reasons:\nChat is an interface that everyone, even people without previous exposure to computers or the Internet, can learn to use quickly. When I volunteered at a low-income residential neighborhood (are we allowed to say slum?) in Kenya in the early 2010s, I was blown away by how comfortable everyone there was with doing banking on their phone, via texts. No one in that neighborhood had a computer.\nChat interface is accessible. You can use voice instead of text if your hands are busy.\nChat is also an incredibly robust interface – you can give it any request and it’ll give back a response, even if the response isn’t good.\nHowever, there are certain areas that I think the chat interface can be improved upon.\nMultiple messages per turn\nCurrently, we pretty much assume one message per turn. This is not how my friends and I text. Often, I need multiple messages to complete my thought, because I need to insert different data (e.g. images, locations, links), I forgot something in the previous messages, or I just don’t feel like putting everything into a massive paragraph.\nMultimodal input\nIn the realm of multimodal applications, most energy is spent on building better models, and very little on building better interfaces. Take\nNvidia’s NeVA chatbot\n. I’m not a UX expert, but I suspect there might be room for UX improvement here.\nP.S. Sorry the NeVA team for calling you out. Even with this interface, your work is super cool!\nIncorporating generative AI into your workflows\nLinus Lee covered this point well in his talk\nGenerative AI interface beyond chats\n. For example, if you want to ask a question about a column of a chart you’re working on, you should be able just point to that column and ask a question.\nEditing and deletion of messages\nHow would editing or deletion of a user input change the conversation flow with the chatbot?\n10. Build LLMs for non-English languages\nWe know that current English-first LLMs don’t work well for many other languages, both in terms of performance, latency, and speed. See:\nChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning\n(Lai et al., 2023)\nAll languages are NOT created (tokenized) equal\n(Yennie Jun, 2023)\nHere are some initiatives that I’m aware of. If you have pointers to others, I’d be happy to include them here.\nAya\n: An Open Science Initiative to Accelerate Multilingual AI Progress\nSymato\n: Vietnamese ChatGPT\nCabrita\n: Finetuning InstructLLaMA with portuguese data\nLuotuo-Chinese-LLM\nChinese-LLaMA-Alpaca\nChinese-Vicuna\nSeveral early readers of this post told me they don’t think I should include this direction for two reasons.\nThis is less of a research problem and more of a logistics problem. We already know how to do it. Someone just needs to put money and effort into it. This is not entirely true. Most languages are considered low-resource, e.g. they have far fewer high-quality data compared to English or Chinese, and might require different techniques to train a large language model. See:\nLow-resource Languages: A Review of Past Work and Future Challenges\n(Magueresse et al., 2020)\nJW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages\n(Agić et al., 2019)\nThose more pessimistic think that in the future, many languages will die out, and the Internet will consist of two universes in two languages: English and Mandarin. This school of thought isn’t new – anyone remembers Esperando?\nThe impact of AI tools, e.g. machine translation and chatbots, on language learning is still unclear. Will they help people learn new languages faster, or will they eliminate the need of learning new languages altogether?\nConclusion\nPhew, that was a lot of papers to reference, and I have no doubt that I still missed a ton. If there’s something you think I missed, please let me know.\nFor another perspective, check out this comprehsive paper\nChallenges and Applications of Large Language Models\n(Kaddour et al., 2023).\nSome of the problems mentioned above are harder than others. For example, I think that number 10, building LLMs for non-English languages, is more straightforward with enough time and resources.\nNumber 1, reducing hallucination, will be much harder, since hallucination is just LLMs doing their probabilistic thing.\nNumber 4, making LLMs faster and cheaper, will never be completely solved. There is already so much progress in this area, and there will be more, but we will never run out of room for improvement.\nNumber 5 and number 6, new architectures and new hardware, are very challenging, but they are inevitable with time. Because of the symbiosis between architecture and hardware – new architecture will need to be optimized for common hardware, and hardware will need to support common architecture – they might be solved by the same company.\nSome of these problems won’t be solved using only technical knowledge. For example, number 8, improving learning from human preference, might be more of a policy problem than a technical problem. Number 9, improving the efficiency of the chat interface, is more of a UX problem. We need more people with non-technical backgrounds to work with us to solve these problems.\nWhat research direction are you most excited about? What are the most promising solutions you see for these problems? I’d love to hear from you.",
  "metadata": {
    "title": "Open challenges in LLM research",
    "description": "[LinkedIn discussion, Twitter thread]",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "Open challenges in LLM research",
    "og:locale": "en_US",
    "og:description": "[LinkedIn discussion, Twitter thread]",
    "og:url": "https://huyenchip.com/2023/08/16/llm-research-open-challenges.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/llm-research/10-open-challenges-llm-research.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2023-08-16T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/2-context.png",
      "bytes": 297792,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-224d800e9c82b5acd6c2.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/2-context.png",
      "alt": "Context needed for a customer support query",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/2-rag.jpg",
      "bytes": 102089,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-c348dc64c4fcd23a0f3d.jpg",
      "src": "https://huyenchip.com/assets/pics/llm-research/2-rag.jpg",
      "alt": "Context needed for a customer support query",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/3-flamingo.png",
      "bytes": 349562,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-9e3877588414cf00a3c5.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/3-flamingo.png",
      "alt": "Multimodal Flamingo's architecture",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/4-llm-optimization.png",
      "bytes": 85446,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-62ae7f746b419bbc6493.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/4-llm-optimization.png",
      "alt": "Guanaco 7B's performance compared to ChatGPT GPT-3.5 and GPT-4",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/5-monarch-mixer.png",
      "bytes": 169285,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-9b5a605f00b2a6f40245.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/5-monarch-mixer.png",
      "alt": "Monarch Mixer architecture",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/6-photonic-matrix-multiplication.png",
      "bytes": 1611749,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-296e74d5bc62778e58b6.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/6-photonic-matrix-multiplication.png",
      "alt": "Timeline of advances of the three major methods in photonic matrix multiplication",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/8-instructgpt-demographics.png",
      "bytes": 78873,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-1fbde2528fbf2de53d49.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/8-instructgpt-demographics.png",
      "alt": "Demographics of labelers for InstructGPT",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/8-openassistant-demographics.png",
      "bytes": 209905,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-fef39fea84e0f948105e.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/8-openassistant-demographics.png",
      "alt": "Self-reported demographics of contributors to OpenAssistant dataset",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/9-superapp-chat-interface.png",
      "bytes": 270974,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-5450423afca73321c3f1.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/9-superapp-chat-interface.png",
      "alt": "Chat has been used as the universal interface for superapps in China for over a decade",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/9-neva.png",
      "bytes": 388839,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-92adcd01e0c3e63ef649.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/9-neva.png",
      "alt": "NVIDIA's NeVA interface",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/llm-research/10-non-english-tokens.png",
      "bytes": 205436,
      "reused": false,
      "local_path": "attachments/d721f7c874fe/images/body-91c20e3bbd775f7e8337.png",
      "src": "https://huyenchip.com/assets/pics/llm-research/10-non-english-tokens.png",
      "alt": "Tokenization for non-English languages",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=njzB6fm0U8g",
      "src": "https://www.youtube.com/watch?v=njzB6fm0U8g",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/d721f7c874fe/videos/repair-njzB6fm0U8g.mp4",
      "bytes": 152364817,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "3167.861000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=a7CXIE_Gyy8",
      "src": "https://www.youtube.com/embed/a7CXIE_Gyy8",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/d721f7c874fe/videos/repair-a7CXIE_Gyy8.mp4",
      "bytes": 13898689,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "210.161000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=rd-J3hmycQs",
      "src": "https://www.youtube.com/watch?v=rd-J3hmycQs",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/d721f7c874fe/videos/repair-rd-J3hmycQs.mp4",
      "bytes": 97763220,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1488.321000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.linkedin.com/posts/chiphuyen_llm-airesearch-generativeai-activity-7097619722363408385-s5Cp",
      "label": "LinkedIn discussion",
      "type": "a"
    },
    {
      "href": "https://twitter.com/chipro/status/1691858084824838427",
      "label": "Twitter thread",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2023/05/02/rlhf.html",
      "label": "Hallucination",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2202.03629",
      "label": "Survey of Hallucination in Natural Language Generation",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.13534",
      "label": "How Language Model Hallucinations Can Snowball",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.04023",
      "label": "A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.10400",
      "label": "Contrastive Learning Reduces Hallucination in Conversations",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.11171",
      "label": "Self-Consistency Improves Chain of Thought Reasoning in Language Models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.08896",
      "label": "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models",
      "type": "a"
    },
    {
      "href": "https://github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/grounding_rail/README.md",
      "label": "NVIDIA’s NeMo-Guardrails",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2109.06157.pdf",
      "label": "SituatedQA",
      "type": "a"
    },
    {
      "href": "https://ai.google.com/research/NaturalQuestions",
      "label": "Natural Questions NQ-Open dataset",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.11401",
      "label": "Retrieval Augmented Generation",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=njzB6fm0U8g",
      "label": "Jerry Liu’s talk on LlamaIndex",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2307.03172",
      "label": "Lost in the Middle: How Language Models Use Long Contexts",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.00020",
      "label": "[CLIP] Learning Transferable Visual Models From Natural Language Supervision",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.14198",
      "label": "Flamingo: a Visual Language Model for Few-Shot Learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2301.12597",
      "label": "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.14045",
      "label": "KOSMOS-1: Language Is Not All You Need: Aligning Perception with Language Models",
      "type": "a"
    },
    {
      "href": "https://ai.googleblog.com/2023/03/palm-e-embodied-multimodal-language.html",
      "label": "PaLM-E: An embodied multimodal language model",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.08485",
      "label": "LLaVA: Visual Instruction Tuning",
      "type": "a"
    },
    {
      "href": "https://catalog.ngc.nvidia.com/orgs/nvidia/teams/playground/models/neva",
      "label": "NeVA: NeMo Vision and Language Assistant",
      "type": "a"
    },
    {
      "href": "https://crfm.stanford.edu/2023/03/13/alpaca.html",
      "label": "Alpaca 7B",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.14314",
      "label": "Guanaco 7B",
      "type": "a"
    },
    {
      "href": "https://learning.oreilly.com/library/view/designing-machine-learning/9781098107956/ch07.html",
      "label": "Model Compression",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969",
      "label": "Designing Machine Learning Systems",
      "type": "a"
    },
    {
      "href": "https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/",
      "label": "Transformer was originally designed by Google to run fast on TPUs",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2111.00396",
      "label": "Efficiently Modeling Long Sequences with Structured State Spaces",
      "type": "a"
    },
    {
      "href": "https://together.ai/blog/monarch-mixer",
      "label": "Monarch Mixer",
      "type": "a"
    },
    {
      "href": "https://www.nytimes.com/2012/06/26/technology/in-a-big-network-of-computers-evidence-of-machine-learning.html",
      "label": "Google released just a few months before AlexNet",
      "type": "a"
    },
    {
      "href": "https://cloud.google.com/tpu/docs/intro-to-tpu",
      "label": "TPUs",
      "type": "a"
    },
    {
      "href": "https://www.graphcore.ai/products/ipu",
      "label": "IPUs",
      "type": "a"
    },
    {
      "href": "https://www.eetimes.com/cerebras-sells-100-million-ai-supercomputer-plans-8-more/",
      "label": "Cerebras",
      "type": "a"
    },
    {
      "href": "https://spectrum.ieee.org/sambanova-ceo-ai-interview",
      "label": "a billion dollars to develop new AI chips",
      "type": "a"
    },
    {
      "href": "https://www.ibm.com/quantum",
      "label": "IBM’s QPU",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/d41586-023-00536-w",
      "label": "a major milestone in quantum error reduction",
      "type": "a"
    },
    {
      "href": "https://quantumai.google/quantum-virtual-machine",
      "label": "Google Colab",
      "type": "a"
    },
    {
      "href": "https://cqe.mit.edu/",
      "label": "MIT Center for Quantum Engineering",
      "type": "a"
    },
    {
      "href": "https://www.mpq.mpg.de/en",
      "label": "Max Planck Institute of Quantum Optics",
      "type": "a"
    },
    {
      "href": "https://chicagoquantum.org/",
      "label": "Chicago Quantum Exchange",
      "type": "a"
    },
    {
      "href": "https://quantum-roadmap.ornl.gov/",
      "label": "Oak Ridge National Laboratory",
      "type": "a"
    },
    {
      "href": "https://lightmatter.co/",
      "label": "Lightmatter",
      "type": "a"
    },
    {
      "href": "https://ayarlabs.com/",
      "label": "Ayar Labs",
      "type": "a"
    },
    {
      "href": "https://www.lightelligence.ai/",
      "label": "Lightelligence",
      "type": "a"
    },
    {
      "href": "https://www.luminous.com/",
      "label": "Luminous Computing",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/s41377-022-00717-8",
      "label": "Photonic matrix multiplication lights up photonic accelerator and beyond",
      "type": "a"
    },
    {
      "href": "https://github.com/Significant-Gravitas/Auto-GPT",
      "label": "Auto-GPT",
      "type": "a"
    },
    {
      "href": "https://github.com/AntonOsika/gpt-engineer",
      "label": "GPT-Engineering",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.03442",
      "label": "Generative Agents: Interactive Simulacra of Human Behavior",
      "type": "a"
    },
    {
      "href": "https://www.theinformation.com/briefings/two-co-founders-of-adept-an-openai-rival-suddenly-left-to-start-another-company",
      "label": "both already left",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/embed/a7CXIE_Gyy8",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://arxiv.org/abs/2212.08073",
      "label": "Constitutional AI: Harmlessness from AI Feedback",
      "type": "a"
    },
    {
      "href": "https://www.deepmind.com/publications/fine-tuning-language-models-to-find-agreement-among-humans-with-diverse-preferences",
      "label": "Fine-tuning language models to find agreement among humans with diverse preferences",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.02155",
      "label": "InstructGPT: Training language models to follow instructions with human feedback",
      "type": "a"
    },
    {
      "href": "https://twitter.com/jeremyphoward/status/1647763133665271808/photo/1",
      "label": "Jeremy Howard has a great Twitter thread on this",
      "type": "a"
    },
    {
      "href": "https://austinhenley.com/blog/naturallanguageui.html",
      "label": "Natural language is the lazy user interface",
      "type": "a"
    },
    {
      "href": "https://wattenberger.com/thoughts/boo-chatbots",
      "label": "Why Chatbots Are Not the Future",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.17710",
      "label": "What Types of Questions Require Conversation to Answer? A Case Study of AskReddit Questions",
      "type": "a"
    },
    {
      "href": "https://idratherbewriting.com/blog/ai-chat-interfaces-are-the-new-user-interface-for-docs",
      "label": "AI chat interfaces could become the primary user interface to read documentation",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/llm-ux/",
      "label": "Interacting with LLMs with Minimal Chat",
      "type": "a"
    },
    {
      "href": "http://dangrover.com/blog/2014/12/01/chinese-mobile-app-ui-trends.html",
      "label": "Dan Grover had this discussion back in 2014",
      "type": "a"
    },
    {
      "href": "https://acroll.medium.com/on-chat-as-interface-92a68d2bf854",
      "label": "On chat as interface",
      "type": "a"
    },
    {
      "href": "https://www.technologyreview.com/2016/04/25/8510/is-the-chatbot-trend-one-big-misunderstanding/",
      "label": "Is the Chatbot Trend One Big Misunderstanding?",
      "type": "a"
    },
    {
      "href": "http://dangrover.com/blog/2016/04/20/bots-wont-replace-apps.html",
      "label": "Bots won’t replace apps. Better apps will replace apps",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=rd-J3hmycQs",
      "label": "Generative AI interface beyond chats",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.05613",
      "label": "ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning",
      "type": "a"
    },
    {
      "href": "https://blog.yenniejun.com/p/all-languages-are-not-created-tokenized",
      "label": "All languages are NOT created (tokenized) equal",
      "type": "a"
    },
    {
      "href": "https://aya.for.ai/",
      "label": "Aya",
      "type": "a"
    },
    {
      "href": "https://discord.gg/a2PCzB4AdE",
      "label": "Symato",
      "type": "a"
    },
    {
      "href": "https://github.com/22-hours/cabrita",
      "label": "Cabrita",
      "type": "a"
    },
    {
      "href": "https://github.com/LC1332/Luotuo-Chinese-LLM",
      "label": "Luotuo-Chinese-LLM",
      "type": "a"
    },
    {
      "href": "https://github.com/ymcui/Chinese-LLaMA-Alpaca",
      "label": "Chinese-LLaMA-Alpaca",
      "type": "a"
    },
    {
      "href": "https://github.com/Facico/Chinese-Vicuna",
      "label": "Chinese-Vicuna",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.07264",
      "label": "Low-resource Languages: A Review of Past Work and Future Challenges",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P19-1310/",
      "label": "JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2307.10169",
      "label": "Challenges and Applications of Large Language Models",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:48Z",
  "source_url": "https://huyenchip.com/2023/08/16/llm-research-open-challenges.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2202.03629",
      "bytes": 3996575,
      "reused": true,
      "local_path": "papers/repair/ce1261361041d2f3b02b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.13534",
      "bytes": 570726,
      "reused": false,
      "local_path": "papers/repair/b13687f776d1d9043ad3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.04023",
      "bytes": 2085222,
      "reused": false,
      "local_path": "papers/repair/759f199d2b7b19df1785.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.10400",
      "bytes": 705454,
      "reused": false,
      "local_path": "papers/repair/abc3532a670be8f9c595.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.11171",
      "bytes": 10505941,
      "reused": true,
      "local_path": "papers/repair/0a415c0e8756d0879469.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.08896",
      "bytes": 903117,
      "reused": true,
      "local_path": "papers/repair/6203039f8fe36d722964.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2109.06157.pdf",
      "bytes": 4822725,
      "reused": false,
      "local_path": "papers/repair/fbe2634507345a2f48ff.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.11401",
      "bytes": 885323,
      "reused": true,
      "local_path": "papers/repair/fc8d32f196e0f0c27ffb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2307.03172",
      "bytes": 747542,
      "reused": true,
      "local_path": "papers/repair/9a55f69abe89ff9892a1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.00020",
      "bytes": 6813639,
      "reused": true,
      "local_path": "papers/repair/7db0b9244bb1fd42ae98.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.14198",
      "bytes": 31579390,
      "reused": true,
      "local_path": "papers/repair/0957ce92a2e9701d5e26.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2301.12597",
      "bytes": 6737540,
      "reused": true,
      "local_path": "papers/repair/550756db727a4887cce7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.14045",
      "bytes": 3743156,
      "reused": false,
      "local_path": "papers/repair/9d1a94355769f2097414.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.08485",
      "bytes": 5906135,
      "reused": true,
      "local_path": "papers/repair/11628fe93e09857fc735.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.14314",
      "bytes": 1065470,
      "reused": true,
      "local_path": "papers/repair/df2f754581ddf63a5d62.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2111.00396",
      "bytes": 3386172,
      "reused": false,
      "local_path": "papers/repair/497239672faf5797298d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.03442",
      "bytes": 11947867,
      "reused": true,
      "local_path": "papers/repair/68f98ce7167c163414a4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.08073",
      "bytes": 2088111,
      "reused": true,
      "local_path": "papers/repair/b3e647be3bc180f47791.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.02155",
      "bytes": 1797405,
      "reused": true,
      "local_path": "papers/repair/4b8cf107df7118e0ffe8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.17710",
      "bytes": 3271543,
      "reused": false,
      "local_path": "papers/repair/3b02ad035fe32f284d77.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.05613",
      "bytes": 544409,
      "reused": false,
      "local_path": "papers/repair/21908735f2d3ec2fb40e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.07264",
      "bytes": 1038526,
      "reused": false,
      "local_path": "papers/repair/d66367ed770dd5c658e3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2307.10169",
      "bytes": 2919815,
      "reused": false,
      "local_path": "papers/repair/2d011378805e4f64fee2.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 22031,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d721f7c874fe/pages/source.html",
  "local_html": "attachments/d721f7c874fe/pages/article.html",
  "body_base_path": "attachments/d721f7c874fe/pages/",
  "body_markdown_path": "attachments/d721f7c874fe/pages/article.md"
}