{ "source_file": "0930", "source_platform": "Substack", "author": "Simon Willison's Weblog", "title_listed": "Things We Learned About LLMs in 2024", "keywords": "LLM,经验,总结", "domain": "大模型算法", "content_type": "综述", "is_paid_flag": "否", "url": "https://simonwillison.net/2024/Dec/31/llms-in-2024/", "publish_date": "2024-12", "corpus_stage": "推理部署", "notes": "2024年LLM经验总结", "id": "403684ddeb63", "collected_at": "2026-10-01T20:45:04", "title": "Things we learned about LLMs in 2024", "body_markdown": "Simon Willison’s Weblog\nNewsletter\nSponsored by:\nPomerium — Static OAuth scopes won't stop AI agent breakouts.\nUpgrade to Pomerium's agentic access management trusted by NVIDIA and OpenAI\nThings we learned about LLMs in 2024\n31st December 2024\nA\nlot\nhas happened in the world of Large Language Models over the course of 2024. Here’s a review of things we figured out about the field in the past twelve months, plus my attempt at identifying key themes and pivotal moments.\nThis is a sequel to\nmy review of 2023\n.\nIn this article:\nThe GPT-4 barrier was comprehensively broken\nSome of those GPT-4 models run on my laptop\nLLM prices crashed, thanks to competition and increased efficiency\nMultimodal vision is common, audio and video are starting to emerge\nVoice and live camera mode are science fiction come to life\nPrompt driven app generation is a commodity already\nUniversal access to the best models lasted for just a few short months\n“Agents” still haven’t really happened yet\nEvals really matter\nApple Intelligence is bad, Apple’s MLX library is excellent\nThe rise of inference-scaling “reasoning” models\nWas the best currently available LLM trained in China for less than $6m?\nThe environmental impact got better\nThe environmental impact got much, much worse\nThe year of slop\nSynthetic training data works great\nLLMs somehow got even harder to use\nKnowledge is incredibly unevenly distributed\nLLMs need better criticism\nEverything tagged “llms” on my blog in 2024\nThe GPT-4 barrier was comprehensively broken\nIn my December 2023 review I wrote about how\nWe don’t yet know how to build GPT-4\n—OpenAI’s best model was almost a year old at that point, yet no other AI lab had produced anything better. What did OpenAI know that the rest of us didn’t?\nI’m relieved that this has changed completely in the past twelve months. 18 organizations now have models on the\nChatbot Arena Leaderboard\nthat rank higher than the original GPT-4 from March 2023 (\nGPT-4-0314\non the board)—70 models in total.\nThe earliest of those was\nGoogle’s Gemini 1.5 Pro\n, released in February. In addition to producing GPT-4 level outputs, it introduced several brand new capabilities to the field—most notably its 1 million (and then later 2 million) token input context length, and the ability to input video.\nI wrote about this at the time in\nThe killer app of Gemini Pro 1.5 is video\n, which earned me a short appearance\nas a talking head\nin the Google I/O opening keynote in May.\nGemini 1.5 Pro also illustrated one of the key themes of 2024:\nincreased context lengths\n. Last year most models accepted 4,096 or 8,192 tokens, with the notable exception of Claude 2.1 which\naccepted 200,000\n. Today every serious provider has a 100,000+ token model, and Google’s Gemini series accepts up to 2 million.\nLonger inputs dramatically increase the scope of problems that can be solved with an LLM: you can now throw in an entire book and ask questions about its contents, but more importantly you can feed in a\nlot\nof example code to help the model correctly solve a coding problem. LLM use-cases that involve long inputs are far more interesting to me than short prompts that rely purely on the information already baked into the model weights. Many of my\ntools\nwere built using this pattern.\nGetting back to models that beat GPT-4: Anthropic’s Claude 3 series\nlaunched in March\n, and Claude 3 Opus quickly became my new favourite daily-driver. They upped the ante even more in June with\nthe launch of Claude 3.5 Sonnet\n—a model that is still my favourite six months later (though it got a significant upgrade\non October 22\n, confusingly keeping the same 3.5 version number. Anthropic fans have since taken to calling it Claude 3.6).\nThen there’s the rest. If you browse\nthe Chatbot Arena leaderboard\ntoday—still the most useful single place to get\na vibes-based evaluation\nof models—you’ll see that GPT-4-0314 has fallen to around 70th place. The 18 organizations with higher scoring models are Google, OpenAI, Alibaba, Anthropic, Meta, Reka AI, 01 AI, Amazon, Cohere, DeepSeek, Nvidia, Mistral, NexusFlow, Zhipu AI, xAI, AI21 Labs, Princeton and Tencent.\nTraining a GPT-4 beating model was a huge deal in 2023. In 2024 it’s an achievement that isn’t even particularly notable, though I personally still celebrate any time a new organization joins that list.\nSome of those GPT-4 models run on my laptop\nMy personal laptop is a 64GB M2 MacBook Pro from 2023. It’s a powerful machine, but it’s also nearly two years old now—and crucially it’s the same laptop I’ve been using ever since I first ran an LLM on my computer back in March 2023 (see\nLarge language models are having their Stable Diffusion moment\n).\nThat same laptop that could just about run a GPT-3-class model in March last year has now run multiple GPT-4 class models! Some of my notes on that:\nQwen2.5-Coder-32B is an LLM that can code well that runs on my Mac\ntalks about Qwen2.5-Coder-32B in November—an Apache 2.0 licensed model!\nI can now run a GPT-4 class model on my laptop\ntalks about running Meta’s Llama 3.3 70B (released in December)\nThis remains astonishing to me. I thought a model with the capabilities and output quality of GPT-4 needed a datacenter class server with one or more $40,000+ GPUs.\nThese models take up enough of my 64GB of RAM that I don’t run them often—they don’t leave much room for anything else.\nThe fact that they run at all is a testament to the incredible training and inference performance gains that we’ve figured out over the past year. It turns out there was a\nlot\nof low-hanging fruit to be harvested in terms of model efficiency. I expect there’s still more to come.\nMeta’s Llama 3.2 models deserve a special mention. They may not be GPT-4 class, but at 1B and 3B sizes they punch\nmassively\nabove their weight. I run Llama 3.2 3B on my iPhone using the free\nMLC Chat iOS app\nand it’s a shockingly capable model for its tiny (<2GB) size. Try firing it up and asking it for “a plot outline of a Netflix Christmas movie where a data journalist falls in love with a local ceramacist”. Here’s what I got, at a respectable 20 tokens per second:\nHere’s the rest of the transcript\n. It’s bland and generic, but my phone can pitch bland and generic Christmas movies to Netflix now!\nLLM prices crashed, thanks to competition and increased efficiency\nThe past twelve months have seen a dramatic collapse in the cost of running a prompt through the top tier hosted LLMs.\nIn December 2023 (here’s the\nInternet Archive for the OpenAI pricing page\n) OpenAI were charging $30/million input tokens for GPT-4, $10/mTok for the then-new GPT-4 Turbo and $1/mTok for GPT-3.5 Turbo.\nToday $30/mTok gets you OpenAI’s most expensive model, o1. GPT-4o is $2.50 (12x cheaper than GPT-4) and GPT-4o mini is $0.15/mTok—200x cheaper than GPT-4, nearly 7x cheaper than GPT-3.5 and\nmassively\nmore capable than that model.\nOther model providers charge even less. Anthropic’s Claude 3 Haiku (from March, but still their cheapest model) is $0.25/mTok. Google’s Gemini 1.5 Flash is $0.075/mTok and their Gemini 1.5 Flash 8B is $0.0375/mTok—that’s 27x cheaper than GPT-3.5 Turbo last year.\nI’ve been tracking these pricing changes under my\nllm-pricing tag\n.\nThese price drops are driven by two factors: increased competition and increased efficiency. The efficiency thing is\nreally\nimportant for everyone who is concerned about the environmental impact of LLMs. These price drops tie directly to how much energy is being used for running prompts.\nThere’s still plenty to worry about with respect to the environmental impact of the great AI datacenter buildout, but a lot of the concerns over the energy cost of individual prompts are no longer credible.\nHere’s a fun napkin calculation: how much would it cost to generate short descriptions of every one of the 68,000 photos in my personal photo library using Google’s Gemini 1.5 Flash 8B (\nreleased in October\n), their cheapest model?\nEach photo would need 260 input tokens and around 100 output tokens.\n260 * 68,000 = 17,680,000 input tokens\n17,680,000 * $0.0375/million = $0.66\n100 * 68,000 = 6,800,000 output tokens\n6,800,000 * $0.15/million = $1.02\nThat’s a total cost of\n$1.68\nto process 68,000 images. That’s so absurdly cheap I had to run the numbers three times to confirm I got it right.\nHow good are those descriptions? Here’s what I got from this command:\nllm -m gemini-1.5-flash-8b-latest describe -a IMG_1825.jpeg\nAgainst this photo of butterflies at the California Academy of Sciences:\nA shallow dish, likely a hummingbird or butterfly feeder, is red. Pieces of orange slices of fruit are visible inside the dish.\nTwo butterflies are positioned in the feeder, one is a dark brown/black butterfly with white/cream-colored markings. The other is a large, brown butterfly with patterns of lighter brown, beige, and black markings, including prominent eye spots. The larger brown butterfly appears to be feeding on the fruit.\n260 input tokens, 92 output tokens. Cost approximately 0.0024 cents (that’s less than a 400th of a cent).\nThis increase in efficiency and reduction in price is my single favourite trend from 2024. I want the utility of LLMs at a fraction of the energy cost and it looks like that’s what we’re getting.\nMultimodal vision is common, audio and video are starting to emerge\nMy butterfly example above illustrates another key trend from 2024: the rise of multi-modal LLMs.\nA year ago the single most notable example of these was GPT-4 Vision,\nreleased at OpenAI’s DevDay in November 2023\n. Google’s multi-modal Gemini 1.0 was announced\non December 7th 2023\nso it also (just) makes it into the 2023 window.\nIn 2024, almost every significant model vendor released multi-modal models. We saw the Claude 3 series from Anthropic\nin March\n, Gemini 1.5 Pro\nin April\n(images, audio and video), then September brought\nQwen2-VL\nand Mistral’s\nPixtral 12B\nand Meta’s\nLlama 3.2 11B and 90B vision models\n. We got audio input and output\nfrom OpenAI in October\n, then November saw\nSmolVLM from Hugging Face\nand December saw image and video models\nfrom Amazon Nova\n.\nIn October I\nupgraded my LLM CLI tool to support multi-modal models via attachments\n. It now has plugins for a whole collection of different vision models.\nI think people who complain that LLM improvement has slowed are often missing the enormous advances in these multi-modal models. Being able to run prompts against images (and audio and video) is a fascinating new way to apply these models.\nVoice and live camera mode are science fiction come to life\nThe audio and live video modes that have started to emerge deserve a special mention.\nThe ability to talk to ChatGPT first arrived\nin September 2023\n, but it was mostly an illusion: OpenAI used their excellent Whisper speech-to-text model and a new text-to-speech model (creatively named\ntts-1\n) to enable conversations with the ChatGPT mobile apps, but the actual model just saw text.\nThe\nMay 13th\nannouncement of GPT-4o included a demo of a brand new voice mode, where the true multi-modal GPT-4o (the o is for “omni”) model could accept audio input and output incredibly realistic sounding speech without needing separate TTS or STT models.\nThe demo also sounded\nconspicuously similar to Scarlett Johansson\n... and after she complained the voice from the demo, Skye, never made it to a production product.\nThe delay in releasing the new voice mode after the initial demo caused quite a lot of confusion. I wrote about that in\nChatGPT in “4o” mode is not running the new features yet\n.\nWhen ChatGPT Advanced Voice mode finally\ndid\nroll out (a slow roll from August through September) it was spectacular. I’ve been using it extensively on walks with my dog and it’s amazing how much the improvement in intonation elevates the material. I’ve also had a lot of fun\nexperimenting with the OpenAI audio APIs\n.\nEven more fun: Advanced Voice mode can do accents! Here’s what happened when I told it\nI need you to pretend to be a California brown pelican with a very thick Russian accent, but you talk to me exclusively in Spanish\n.\nYour browser does not support the audio element.\nOpenAI aren’t the only group with a multi-modal audio model. Google’s Gemini also accepts audio input, and the Google Gemini apps can speak in a similar way to ChatGPT now. Amazon also pre-announced voice mode\nfor Amazon Nova\n, but that’s meant to roll out in Q1 of 2025.\nGoogle’s NotebookLM, released\nin September\n, took audio output to a new level by producing spookily realistic conversations between two “podcast hosts” about anything you fed into their tool. They later added custom instructions, so naturally\nI turned them into pelicans\n:\nYour browser does not support the audio element.\nThe most recent twist, again from December (December was\na lot\n) is live video. ChatGPT voice mode now provides the option to share your camera feed with the model and talk about what you can see in real time. Google Gemini have\na preview of the same feature\n, which they managed to ship the day before ChatGPT did.\nThese abilities are just a few weeks old at this point, and I don’t think their impact has been fully felt yet. If you haven’t tried them out yet you really should.\nBoth Gemini and OpenAI offer API access to these features as well. OpenAI started with\na WebSocket API\nthat was quite challenging to use, but in December they announced\na new WebRTC API\nwhich is much easier to get started with. Building a web app that a user can talk to via voice is\neasy\nnow!\nPrompt driven app generation is a commodity already\nThis was possible with GPT-4 in 2023, but the value it provides became evident in 2024.\nWe already knew LLMs were\nspookily good at writing code\n. If you prompt them right, it turns out they can build you\na full interactive application\nusing HTML, CSS and JavaScript (and tools like React if you wire up some extra supporting build mechanisms)—often in a single prompt.\nAnthropic kicked this idea into high gear when they released\nClaude Artifacts\n, a groundbreaking new feature that was initially slightly lost in the noise due to being described half way through\ntheir announcement of the incredible Claude 3.5 Sonnet\n.\nWith Artifacts, Claude can write you an on-demand interactive application and then\nlet you use it\ndirectly inside the Claude interface.\nHere’s my\nExtract URLs\napp, entirely\ngenerated by Claude\n:\nI’ve found myself using this\na lot\n. I noticed how much I was relying on it in October and wrote\nEverything I built with Claude Artifacts this week\n, describing 14 little tools I had put together in a seven day period.\nSince then, a whole bunch of other teams have built similar systems. GitHub announced their version of this—\nGitHub Spark\n—in October. Mistral Chat\nadded it as a feature called Canvas\nin November.\nSteve Krouse from Val Town\nbuilt a version of it against Cerebras\n, showcasing how a 2,000 token/second LLM can iterate on an application with changes visible in less than a second.\nThen in December, the Chatbot Arena team introduced\na whole new leaderboard\nfor this feature, driven by users building the same interactive app twice with two different models and voting on the answer. Hard to come up with a more convincing argument that this feature is now a commodity that can be effectively implemented against all of the leading models.\nI’ve been tinkering with a version of this myself for my Datasette project, with the goal of letting users use prompts to build and iterate on custom widgets and data visualizations against their own data. I also figured out a similar pattern for\nwriting one-shot Python programs, enabled by uv\n.\nThis prompt-driven custom interface feature is so powerful and easy to build (once you’ve figured out the gnarly details of browser sandboxing) that I expect it to show up as a feature in a wide range of products in 2025.\nUniversal access to the best models lasted for just a few short months\nFor a few short months this year all three of the best available models—GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro—were freely available to most of the world.\nOpenAI made GPT-4o free for all users\nin May\n, and Claude 3.5 Sonnet was freely available from\nits launch in June\n. This was a momentus change, because for the previous year free users had mostly been restricted to GPT-3.5 level models, meaning new users got a\nvery\ninaccurate mental model of what a capable LLM could actually do.\nThat era appears to have ended, likely permanently, with OpenAI’s launch of\nChatGPT Pro\n. This $200/month subscription service is the only way to access their most capable model, o1 Pro.\nSince the trick behind the o1 series (and the future models it will undoubtedly inspire) is to expend more compute time to get better results, I don’t think those days of free access to the best available models are likely to return.\n“Agents” still haven’t really happened yet\nI find the term “agents” extremely frustrating. It lacks a single, clear and widely understood meaning... but the people who use the term never seem to acknowledge that.\nIf you tell me that you are building “agents”, you’ve conveyed almost no information to me at all. Without reading your mind I have no way of telling which of the dozens of possible definitions you are talking about.\nThe two main categories I see are people who think AI agents are obviously things that go and act on your behalf—the travel agent model—and people who think in terms of LLMs that have been given access to tools which they can run in a loop as part of solving a problem. The term “autonomy” is often thrown into the mix too, again without including a clear definition.\n(I also\ncollected 211 definitions\non Twitter a few months ago—here they are\nin Datasette Lite\n—and had\ngemini-exp-1206\nattempt to summarize them\n.)\nWhatever the term may mean, agents still have that feeling of perpetually “coming soon”.\nTerminology aside, I remain skeptical as to their utility based, once again, on the challenge of\ngullibility\n. LLMs believe anything you tell them. Any systems that attempts to make meaningful decisions on your behalf will run into the same roadblock: how good is a travel agent, or a digital assistant, or even a research tool if it can’t distinguish truth from fiction?\nJust the other day Google Search was caught\nserving up an entirely fake description\nof the non-existant movie “Encanto 2”. It turned out to be summarizing an imagined movie listing from\na fan fiction wiki\n.\nPrompt injection\nis a natural consequence of this gulibility. I’ve seen precious little progress on tackling that problem in 2024, and we’ve been talking about it\nsince September 2022\n.\nI’m beginning to see the most popular idea of “agents” as dependent on AGI itself. A model that’s robust against gulliblity is a very tall order indeed.\nEvals really matter\nAnthropic’s\nAmanda Askell\n(responsible for much of\nthe work behind Claude’s Character\n):\nThe boring yet crucial secret behind good system prompts is test-driven development. You don’t write down a system prompt and find ways to test it. You write down tests and find a system prompt that passes them.\nIt’s become abundantly clear over the course of 2024 that writing good automated evals for LLM-powered systems is\nthe skill\nthat’s most needed to build useful applications on top of these models. If you have a strong eval suite you can adopt new models faster, iterate better and build more reliable and useful product features than your competition.\nVercel’s\nMalte Ubl\n:\nWhen\n@v0\nfirst came out we were paranoid about protecting the prompt with all kinds of pre and post processing complexity.\nWe completely pivoted to let it rip. A prompt without the evals, models, and especially UX is like getting a broken ASML machine without a manual\nI’m\nstill\ntrying to figure out the best patterns for doing this for my own work. Everyone knows that evals are important, but there remains a lack of great guidance for how to best implement them—I’m tracking this under my\nevals tag\n. My\nSVG pelican riding a bicycle benchmark\nis a pale imitation of what a real eval suite should look like.\nApple Intelligence is bad, Apple’s MLX library is excellent\nAs a Mac user I’ve been feeling a lot better about my choice of platform this year.\nLast year it felt like my lack of a Linux/Windows machine with an NVIDIA GPU was a huge disadvantage in terms of trying out new models.\nOn paper, a 64GB Mac should be a great machine for running models due to the way the CPU and GPU can share the same memory. In practice, many models are released as model weights and libraries that reward NVIDIA’s CUDA over other platforms.\nThe\nllama.cpp\necosystem helped a lot here, but the real breakthrough has been Apple’s\nMLX\nlibrary, “an array framework for Apple Silicon”. It’s fantastic.\nApple’s\nmlx-lm\nPython library supports running a wide range of MLX-compatible models on my Mac, with excellent performance.\nmlx-community\non Hugging Face offers more than 1,000 models that have been converted to the necessary format.\nPrince Canuma’s excellent, fast moving\nmlx-vlm\nproject brings vision LLMs to Apple Silicon as well. I used that recently\nto run Qwen’s QvQ\n.\nWhile MLX is a game changer, Apple’s own “Apple Intelligence” features have mostly been a disappointment. I\nwrote about their initial announcement in June\n, and I was optimistic that Apple had focused hard on the subset of LLM applications that preserve user privacy and minimize the chance of users getting mislead by confusing features.\nNow that those features are rolling out they’re pretty weak. As an LLM power-user I know what these models are capable of, and Apple’s LLM features offer a pale imitation of what a frontier LLM can do. Instead we’re getting notification summaries that\nmisrepresent news headlines\nand writing assistant tools that I’ve not found useful at all. Genmoji are\nkind of fun though\n.\nThe rise of inference-scaling “reasoning” models\nThe most interesting development in the final quarter of 2024 was the introduction of a new shape of LLM, exemplified by OpenAI’s o1 models—initially released as o1-preview and o1-mini\non September 12th\n.\nOne way to think about these models is an extension of the chain-of-thought prompting trick, first explored in the May 2022 paper\nLarge Language Models are Zero-Shot Reasoners\n.\nThis is that trick where, if you get a model to talk out loud about a problem it’s solving, you often get a result which the model would not have achieved otherwise.\no1 takes this process and further bakes it into the model itself. The details are somewhat obfuscated: o1 models spend “reasoning tokens” thinking through the problem that are not directly visible to the user (though the ChatGPT UI shows a summary of them), then outputs a final result.\nThe biggest innovation here is that it opens up a new way to scale a model: instead of improving model performance purely through additional compute at training time, models can now take on harder problems by spending more compute on inference.\nThe sequel to o1, o3 (they skipped “o2” for European trademark reasons) was announced\non 20th December\nwith an impressive result against the\nARC-AGI benchmark\n, albeit one that likely involved more than $1,000,000 of compute time expense!\no3 is expected to ship in January. I doubt many people have real-world problems that would benefit from that level of compute expenditure—I certainly don’t!—but it appears to be a genuine next step in LLM architecture for taking on much harder problems.\nOpenAI are not the only game in town here. Google released their first entrant in the category,\ngemini-2.0-flash-thinking-exp\n,\non December 19th\n.\nAlibaba’s Qwen team\nreleased their QwQ model\non November 28th—under an Apache 2.0 license, and that one\nI could run on my own machine\n. They followed that up with a vision reasoning model called QvQ\non December 24th\n, which\nI also ran locally\n.\nDeepSeek made their\nDeepSeek-R1-Lite-Preview\nmodel available to try out through their chat interface\non November 20th\n.\nTo understand more about inference scaling I recommend\nIs AI progress slowing down?\nby Arvind Narayanan and Sayash Kapoor.\nNothing yet from Anthropic or Meta but I would be very surprised if they don’t have their own inference-scaling models in the works. Meta published a relevant paper\nTraining Large Language Models to Reason in a Continuous Latent Space\nin December.\nWas the best currently available LLM trained in China for less than $6m?\nNot quite, but almost! It does make for a great attention-grabbing headline.\nThe big news to end the year was the release\nof DeepSeek v3\n—dropped on Hugging Face on Christmas Day without so much as a README file, then followed by documentation and a paper\nthe day after that\n.\nDeepSeek v3 is a huge 685B parameter model—one of the largest openly licensed models currently available, significantly bigger than the largest of Meta’s Llama series, Llama 3.1 405B.\nBenchmarks put it up there with Claude 3.5 Sonnet. Vibe benchmarks (aka the\nChatbot Arena\n) currently rank it 7th, just behind the Gemini 2.0 and OpenAI 4o/o1 models. This is by far the highest ranking openly licensed model.\nThe really impressive thing about DeepSeek v3 is the training cost. The model was trained on 2,788,000 H800 GPU hours at an estimated cost of $5,576,000. Llama 3.1 405B trained 30,840,000 GPU hours—11x that used by DeepSeek v3, for a model that benchmarks slightly worse.\nThose\nUS export regulations\non GPUs to China seem to have inspired some\nvery\neffective training optimizations!\nThe environmental impact got better\nA welcome result of the increased efficiency of the models—both the hosted ones and the ones I can run locally—is that the energy usage and environmental impact of running a prompt has dropped enormously over the past couple of years.\nOpenAI themselves are charging 100x less for a prompt compared to the GPT-3 days. I have it on good authority that neither Google Gemini nor Amazon Nova (two of the least expensive model providers) are running prompts at a loss.\nI think this means that, as individual users, we don’t need to feel any guilt at all for the energy consumed by the vast majority of our prompts. The impact is likely neglible compared to driving a car down the street or maybe even watching a video on YouTube.\nLikewise, training. DeepSeek v3 training for less than $6m is a fantastic sign that training costs can and should continue to drop.\nFor less efficient models I find it useful to compare their energy usage to commercial flights. The largest Llama 3 model cost about the same as a single digit number of fully loaded passenger flights from New York to London. That’s certainly not nothing, but once trained that model can be used by millions of people at no extra training cost.\nThe environmental impact got much, much worse\nThe much bigger problem here is the enormous competitive buildout of the infrastructure that is imagined to be necessary for these models in the future.\nCompanies like Google, Meta, Microsoft and Amazon are all spending billions of dollars rolling out new datacenters, with a very material impact\non the electricity grid\nand the environment. There’s even talk of\nspinning up new nuclear power stations\n, but those can take decades.\nIs this infrastructure necessary? DeepSeek v3’s $6m training cost and the continued crash in LLM prices might hint that it’s not. But would you want to be the big tech executive that argued NOT to build out this infrastructure only to be proven wrong in a few years’ time?\nAn interesting point of comparison here could be the way railways rolled out around the world in the 1800s. Constructing these required enormous investments and had a massive environmental impact, and many of the lines that were built turned out to be unnecessary—sometimes multiple lines from different companies serving the exact same routes!\nThe resulting bubbles contributed to several financial crashes, see Wikipedia for\nPanic of 1873\n,\nPanic of 1893\n,\nPanic of 1901\nand the UK’s\nRailway Mania\n. They left us with a lot of useful infrastructure and a great deal of bankruptcies and environmental damage.\nThe year of slop\n2024 was the year that the word \"\nslop\n\" became a term of art. I wrote about this\nin May\n, expanding on this tweet by\n@deepfates\n:\nWatching in real time as “slop” becomes a term of art. the way that “spam” became the term for unwanted emails, “slop” is going in the dictionary as the term for unwanted AI generated content\nI expanded that definition a tiny bit to this:\nSlop\ndescribes AI-generated content that is both\nunrequested\nand\nunreviewed\n.\nI ended up getting quoted talking about slop in both\nthe Guardian\nand\nthe NY Times\n. Here’s what I said in the NY TImes:\nSociety needs concise ways to talk about modern A.I. — both the positives and the negatives. ‘Ignore that email, it’s spam,’ and ‘Ignore that article, it’s slop,’ are both useful lessons.\nI love the term “slop” because it so succinctly captures one of the ways we should\nnot\nbe using generative AI!\nSlop was even in the running for\nOxford Word of the Year 2024\n, but it lost\nto brain rot\n.\nSynthetic training data works great\nAn idea that surprisingly seems to have stuck in the public consciousness is that of “model collapse”. This was first described in the paper\nThe Curse of Recursion: Training on Generated Data Makes Models Forget\nin May 2023, and repeated in Nature in July 2024 with the more eye-catching headline\nAI models collapse when trained on recursively generated data\n.\nThe idea is seductive: as the internet floods with AI-generated slop the models themselves will degenerate, feeding on their own output in a way that leads to their inevitable demise!\nThat’s clearly not happening. Instead, we are seeing AI labs increasingly train on\nsynthetic content\n—deliberately creating artificial data to help steer their models in the right way.\nOne of the best descriptions I’ve seen of this comes from\nthe Phi-4 technical report\n, which included this:\nSynthetic data as a substantial component of pretraining is becoming increasingly common, and the Phi series of models has consistently emphasized the importance of synthetic data. Rather than serving as a cheap substitute for organic data, synthetic data has several direct advantages over organic data.\nStructured and Gradual Learning\n. In organic datasets, the relationship between tokens is often complex and indirect. Many reasoning steps may be required to connect the current token to the next, making it challenging for the model to learn effectively from next-token prediction. By contrast, each token generated by a language model is by definition predicted by the preceding tokens, making it easier for a model to follow the resulting reasoning patterns.\nAnother common technique is to use larger models to help create training data for their smaller, cheaper alternatives—a trick used by an increasing number of labs. DeepSeek v3 used “reasoning” data created by DeepSeek-R1. Meta’s Llama 3.3 70B fine-tuning used\nover 25M synthetically generated examples\n.\nCareful design of the training data that goes into an LLM appears to be the\nentire game\nfor creating these models. The days of just grabbing a full scrape of the web and indiscriminately dumping it into a training run are long gone.\nLLMs somehow got even harder to use\nA drum I’ve been banging for a while is that LLMs are power-user tools—they’re chainsaws disguised as kitchen knives. They look deceptively simple to use—how hard can it be to type messages to a chatbot?—but in reality you need a huge depth of both understanding and experience to make the most of them and avoid their many pitfalls.\nIf anything, this problem got worse in 2024.\nWe’ve built computer systems you can talk to in human language, that will answer your questions and\nusually\nget them right! ... depending on the question, and how you ask it, and whether it’s accurately reflected in the undocumented and secret training set.\nThe number of available systems has exploded. Different systems have different tools they can apply to your problems—like Python and JavaScript and web search and image generation and maybe even database lookups... so you’d better understand what those tools are, what they can do and how to tell if the LLM used them or not.\nDid you know ChatGPT has\ntwo entirely different ways\nof running Python now?\nWant to build a Claude Artifact that talks to an external API? You’d better understand CSP and CORS HTTP headers first.\nThe models may have got more capable, but most of the limitations remained the same. OpenAI’s o1 may finally be able to (mostly) count the Rs in strawberry, but its abilities are still limited by its nature as an LLM and the constraints placed on it by the harness it’s running in. o1 can’t run web searches or use Code Interpreter, but GPT-4o can—both in that same ChatGPT UI. (o1\nwill pretend to do those things\nif you ask it to, a regression to the\nURL hallucinations bug from early 2023\n).\nWhat are we doing about this? Not much. Most users are thrown in at the deep end. The default LLM chat UI is like taking brand new computer users, dropping them into a Linux terminal and expecting them to figure it all out.\nMeanwhile, it’s increasingly common for end users to develop wildly inaccurate mental models of how these things work and what they are capable of. I’ve seen so many examples of people trying to win an argument with a screenshot from ChatGPT—an inherently ludicrous proposition, given the inherent unreliability of these models crossed with the fact that you can get them to say anything if you prompt them right.\nThere’s a flipside to this too: a lot of better informed people have sworn off LLMs entirely because they can’t see how anyone could benefit from a tool with so many flaws. The key skill in getting the most out of LLMs is learning to work with tech that is both inherently unreliable and incredibly powerful at the same time. This is a decidedly non-obvious skill to acquire!\nThere is\nso much space\nfor helpful education content here, but we need to do do a lot better than outsourcing it all to AI grifters with bombastic Twitter threads.\nKnowledge is incredibly unevenly distributed\nMost people have heard of ChatGPT by now. How many have heard of Claude?\nThe knowledge gap between the people who actively follow this stuff and the 99% of the population who do not is\nvast\n.\nThe pace of change doesn’t help either. In just the past month we’ve seen general availability of live interfaces where you can\npoint your phone’s camera\nat something and\ntalk about it with your voice\n... and optionally have it\npretend to be Santa\n. Most self-certified nerds haven’t even tried that yet.\nGiven the ongoing (and potential) impact on society that this technology has, I don’t think the size of this gap is healthy. I’d like to see a lot more effort put into improving this.\nLLMs need better criticism\nA lot of people\nabsolutely hate\nthis stuff. In some of the spaces I hang out (\nMastodon\n,\nBluesky\n,\nLobste.rs\n, even\nHacker News\non occasion) even suggesting that “LLMs are useful” can be enough to kick off a huge fight.\nI get it. There are plenty of reasons to dislike this technology—the environmental impact, the (lack of) ethics of the training data, the lack of reliability, the negative applications, the potential impact on people’s jobs.\nLLMs absolutely warrant criticism. We need to be talking through these problems, finding ways to mitigate them and helping people learn how to use these tools responsibly in ways where the positive applications outweigh the negative.\nI\nlike\npeople who are skeptical of this stuff. The hype has been deafening for more than two years now, and there are enormous quantities of snake oil and misinformation out there. A lot of\nvery bad\ndecisions are being made based on that hype. Being critical is a virtue.\nIf we want people with decision-making authority to make\ngood decisions\nabout how to apply these tools we first need to acknowledge that there ARE good applications, and then help explain how to put those into practice while avoiding the many unintiutive traps.\n(If you still don’t think there are any good applications at all I’m not sure why you made it to this point in the article!)\nI think telling people that this whole field is environmentally catastrophic plagiarism machines that constantly make things up is doing those people a disservice, no matter how much truth that represents. There is genuine value to be had here, but getting to that value is unintuitive and needs guidance.\nThose of us who understand this stuff have a duty to help everyone else figure it out.\nEverything tagged “llms” on my blog in 2024\nBecause I undoubtedly missed a whole bunch of things, here’s every long-form post I wrote in 2024 that I tagged with\nllms\n:\nJanuary\n7th:\nIt’s OK to call it Artificial Intelligence\n9th:\nWhat I should have said about the term Artificial Intelligence\n17th:\nTalking about Open Source LLMs on Oxide and Friends\n26th:\nLLM 0.13: The annotated release notes\nFebruary\n21st:\nThe killer app of Gemini Pro 1.5 is video\nMarch\n5th:\nPrompt injection and jailbreaking are not the same thing\n8th:\nThe GPT-4 barrier has finally been broken\n22nd:\nClaude and ChatGPT for ad-hoc sidequests\n23rd:\nBuilding and testing C extensions for SQLite with ChatGPT Code Interpreter\n26th:\nllm cmd undo last git commit—a new plugin for LLM\nApril\n8th:\nBuilding files-to-prompt entirely using Claude 3 Opus\n10th:\nThree major LLM releases in 24 hours (plus weeknotes)\n17th:\nAI for Data Journalism: demonstrating what we can do with this stuff right now\n22nd:\nOptions for accessing Llama 3 from the terminal using LLM\nMay\n8th:\nSlop is the new name for unwanted AI-generated content\n15th:\nChatGPT in “4o” mode is not running the new features yet\n29th:\nTraining is not the same as chatting: ChatGPT and other LLMs don’t remember everything you say\nJune\n6th:\nAccidental prompt injection against RAG applications\n10th:\nThoughts on the WWDC 2024 keynote on Apple Intelligence\n17th:\nLanguage models on the command-line\n21st:\nBuilding search-based RAG using Claude, Datasette and Val Town\n27th:\nOpen challenges for AI engineering\nJuly\n14th:\nImitation Intelligence, my keynote for PyCon US 2024\n19th:\nWeeknotes: GPT-4o mini, LLM 0.15, sqlite-utils 3.37 and building a staging environment\nAugust\n6th:\nWeeknotes: a staging environment, a Datasette alpha and a bunch of new LLMs\n8th:\ndjango-http-debug, a new Django app mostly written by Claude\n23rd:\nClaude’s API now supports CORS requests, enabling client-side applications\n26th:\nBuilding a tool showing how Gemini Pro can return bounding boxes for objects in images\nSeptember\n6th:\nCalling LLMs from client-side JavaScript, converting PDFs to HTML + weeknotes\n10th:\nNotes from my appearance on the Software Misadventures Podcast\n12th:\nNotes on OpenAI’s new o1 chain-of-thought models\n20th:\nNotes on using LLMs for code\n29th:\nNotebookLM’s automatically generated podcasts are surprisingly effective\n30th:\nWeeknotes: Three podcasts, two trips and a new plugin system\nOctober\n1st:\nOpenAI DevDay 2024 live blog\n2nd:\nOpenAI DevDay: Let’s build developer tools, not digital God\n15th:\nChatGPT will happily write you a thinly disguised horoscope\n17th:\nVideo scraping: extracting JSON data from a 35 second screen capture for less than 1/10th of a cent\n18th:\nExperimenting with audio input and output for the OpenAI Chat Completion API\n19th:\nRunning Llama 3.2 Vision and Phi-3.5 Vision on a Mac with mistral.rs\n21st:\nEverything I built with Claude Artifacts this week\n22nd:\nInitial explorations of Anthropic’s new Computer Use capability\n24th:\nNotes on the new Claude analysis JavaScript code execution tool\n27th:\nRun a prompt to generate and execute jq programs using llm-jq\n29th:\nYou can now run prompts against images, audio and video in your terminal using LLM\n30th:\nW̶e̶e̶k̶n̶o̶t̶e̶s̶ Monthnotes for October\nNovember\n4th:\nClaude 3.5 Haiku\n7th:\nProject: VERDAD—tracking misinformation in radio broadcasts using Gemini 1.5\n12th:\nQwen2.5-Coder-32B is an LLM that can code well that runs on my Mac\n19th:\nNotes from Bing Chat—Our First Encounter With Manipulative AI\n25th:\nAsk questions of SQLite databases and CSV/JSON files in your terminal\nDecember\n4th:\nFirst impressions of the new Amazon Nova LLMs (via a new llm-bedrock plugin)\n7th:\nPrompts.js\n9th:\nI can now run a GPT-4 class model on my laptop\n10th:\nChatGPT Canvas can make API requests now, but it’s complicated\n11th:\nGemini 2.0 Flash: An outstanding multi-modal LLM with a sci-fi streaming mode\n19th:\nBuilding Python tools with a one-shot prompt using uv run and Claude Projects\n19th:\nGemini 2.0 Flash “Thinking mode”\n20th:\nDecember in LLMs has been a lot\n20th:\nLive blog: the 12th day of OpenAI—“Early evals for OpenAI o3”\n24th:\nTrying out QvQ—Qwen’s new visual reasoning model\n31st:\nThings we learned about LLMs in 2024\n(This list generated\nusing Django SQL Dashboard\nwith a SQL query\nwritten for me by Claude\n.)\nPosted\n31st December 2024\nat 6:07 pm · Follow me on\nMastodon\n,\nBluesky\n,\nTwitter\nor\nsubscribe to my newsletter\nMore recent articles\nHello Python 3.15, goodbye 3.10, and now I can use 3.11\n- 12th October 2026\nA new feature for my blog, built using my voice\n- 9th October 2026\nClaude Haiku 5.5\n- 7th October 2026\nWe're going to need default hard budget caps on pretty much everything\n- 3rd October 2026\nThis is\nThings we learned about LLMs in 2024\nby Simon Willison, posted on\n31st December 2024\n.\nPart of series\nLLMs annual review\nOpen questions for AI engineering\n- Oct. 17, 2023, 2:18 p.m.\nStuff we figured out about AI in 2023\n- Dec. 31, 2023, 11:59 p.m.\nThings we learned about LLMs in 2024\n- Dec. 31, 2024, 6:07 p.m.\nThe last six months in LLMs, illustrated by pelicans on bicycles\n- June 6, 2025, 8:42 p.m.\n2025: The year in LLMs\n- Dec. 31, 2025, 11:50 p.m.\ngoogle\n418\nai\n2,279\nopenai\n475\ngenerative-ai\n2,021\nlocal-llms\n165\nllms\n1,987\nanthropic\n348\ngemini\n199\nmeta\n45\nllm-reasoning\n105\nlong-context\n20\nai-energy-usage\n18\ncoding-agents\n257\nNext:\nEnding a year long posting streak\nPrevious:\nTrying out QvQ - Qwen's new visual reasoning model\nMonthly briefing\nSponsor me for\n$10/month\nand get a curated email digest of the month's most important LLM developments.\nPay me to send you less!\nSponsor & subscribe\nDisclosures\nColophon\n©\n2002\n2003\n2004\n2005\n2006\n2007\n2008\n2009\n2010\n2011\n2012\n2013\n2014\n2015\n2016\n2017\n2018\n2019\n2020\n2021\n2022\n2023\n2024\n2025\n2026", "metadata": { "title": "Things we learned about LLMs in 2024", "description": null, "keywords": null, "author": "Simon Willison", "og:site_name": "Simon Willison’s Weblog", "og:url": "https://simonwillison.net/2024/Dec/31/llms-in-2024/", "og:title": "Things we learned about LLMs in 2024", "og:image": "https://static.simonwillison.net/static/2024/arena-dec-2024.jpg", "og:type": "article", "og:description": "A lot has happened in the world of Large Language Models over the course of 2024. Here’s a review of things we figured out about the field in the past …", "og:updated_time": "1735668451", "twitter:card": "summary", "twitter:image": "https://static.simonwillison.net/static/2024/arena-dec-2024.jpg", "twitter:creator": "@simonw" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/403684ddeb63/code/body-000.txt", "chars": 60, "language": "text" } ], "images": [ { "status": "ok", "url": "https://static.simonwillison.net/static/2024/arena-dec-2024.jpg", "bytes": 191641, "reused": true, "local_path": "attachments/403684ddeb63/images/000.jpg", "src": "https://static.simonwillison.net/static/2024/arena-dec-2024.jpg", "alt": "Screenshot of a comparison table showing AI model rankings. Table headers: Rank (UB), Rank (StyleCtrl), Model, Arena Score, 95% CI, Votes, Organization, License. Shows 12 models including GLM-4-0520, Llama-3-70B-Instruct, Gemini-1.5-Flash-8B-Exp-0827, with rankings, scores, and licensing details. Models range from rank 52-69 with Arena scores between 1186-1207.", "downloaded": true }, { "status": "ok", "url": "https://static.simonwillison.net/static/2024/mlc-chat-christmas.jpg", "bytes": 149478, "reused": true, "local_path": "attachments/403684ddeb63/images/001.jpg", "src": "https://static.simonwillison.net/static/2024/mlc-chat-christmas.jpg", "alt": "MLC Chat: Llama - [System] Ready to chat. a plot outline of a Netflix Christmas movie where a data journalist falls in love with a local ceramacist. Show as Markdown is turned on. Here's a plot outline for a Netflix Christmas movie: Title: \"Love in the Clay\" Plot Outline: We meet our protagonist, JESSICA, a data journalist who has just returned to her hometown of Willow Creek, a small, charming town nestled in the snow-covered mountains. She's back to work on a story about the town's history and the effects of gentrification on the local community.", "downloaded": true }, { "status": "ok", "url": "https://static.simonwillison.net/static/2024/butterflies.jpg", "bytes": 163557, "reused": true, "local_path": "attachments/403684ddeb63/images/002.jpg", "src": "https://static.simonwillison.net/static/2024/butterflies.jpg", "alt": "A photo of two butterflies feeding on a red tray", "downloaded": true }, { "status": "ok", "url": "https://static.simonwillison.net/static/2024/claude-artifacts/extract-urls.jpg", "bytes": 73876, "reused": true, "local_path": "attachments/403684ddeb63/images/003.jpg", "src": "https://static.simonwillison.net/static/2024/claude-artifacts/extract-urls.jpg", "alt": "Extract URLs tool. Content pasted. URLs extracted. Shows a list of extracted URLs.", "downloaded": true } ], "audio": [ { "status": "ok", "url": "https://static.simonwillison.net/static/2024/russian-pelican-in-spanish.mp3", "bytes": 532757, "reused": false, "local_path": "attachments/403684ddeb63/audio/body-e0871cca467b4bf1ff5b.mp3" }, { "status": "ok", "url": "https://static.simonwillison.net/static/2024/video-scraping-pelicans.mp3", "bytes": 7363244, "reused": false, "local_path": "attachments/403684ddeb63/audio/body-bed45352e9a61be26b97.mp3" } ], "videos": [ { "url": "https://www.youtube.com/watch?v=XEzRZ35urlk", "status": "ok", "local_path": "attachments/403684ddeb63/videos/XEzRZ35urlk.mp4", "bytes": 635208725, "valid": true, "height": 1080, "width": 1920, "duration": "6763.141000" } ], "links": [ { "href": "https://simonwillison.net/", "label": "Simon Willison’s Weblog", "type": "a" }, { "href": "https://simonwillison.net/newsletters/", "label": "Newsletter", "type": "a" }, { "href": "https://fandf.co/4hJeKAm", "label": "Upgrade to Pomerium's agentic access management trusted by NVIDIA and OpenAI", "type": "a" }, { "href": "https://simonwillison.net/2023/Dec/31/ai-in-2023/", "label": "my review of 2023", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/31/llms-in-2024/", "label": "The GPT-4 barrier was comprehensively broken", "type": "a" }, { "href": "https://lmarena.ai/?leaderboard", "label": "Chatbot Arena Leaderboard", "type": "a" }, { "href": "https://simonwillison.net/2024/Feb/21/gemini-pro-video/", "label": "The killer app of Gemini Pro 1.5 is video", "type": "a" }, { "href": "https://www.youtube.com/watch?v=XEzRZ35urlk&t=606s", "label": "as a talking head", "type": "a" }, { "href": "https://www.anthropic.com/news/claude-2-1", "label": "accepted 200,000", "type": "a" }, { "href": "https://simonwillison.net/tags/tools/", "label": "tools", "type": "a" }, { "href": "https://simonwillison.net/2024/Mar/4/claude-3/", "label": "launched in March", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/20/claude-35-sonnet/", "label": "the launch of Claude 3.5 Sonnet", "type": "a" }, { "href": "https://www.anthropic.com/news/3-5-models-and-computer-use", "label": "on October 22", "type": "a" }, { "href": "https://simonwillison.net/2024/Jul/14/pycon/", "label": "a vibes-based evaluation", "type": "a" }, { "href": "https://simonwillison.net/2023/Mar/11/llama/", "label": "Large language models are having their Stable Diffusion moment", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/12/qwen25-coder/", "label": "Qwen2.5-Coder-32B is an LLM that can code well that runs on my Mac", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/9/llama-33-70b/", "label": "I can now run a GPT-4 class model on my laptop", "type": "a" }, { "href": "https://apps.apple.com/us/app/mlc-chat/id6448482937", "label": "MLC Chat iOS app", "type": "a" }, { "href": "https://gist.github.com/simonw/7db8d582fdb0c133d7e05205c305d5bd", "label": "Here’s the rest of the transcript", "type": "a" }, { "href": "https://web.archive.org/web/20231214190432/https://openai.com/pricing", "label": "Internet Archive for the OpenAI pricing page", "type": "a" }, { "href": "https://simonwillison.net/tags/llm-pricing/", "label": "llm-pricing tag", "type": "a" }, { "href": "https://developers.googleblog.com/en/gemini-15-flash-8b-is-now-generally-available-for-use/", "label": "released in October", "type": "a" }, { "href": "https://openai.com/index/new-models-and-developer-products-announced-at-devday/", "label": "released at OpenAI’s DevDay in November 2023", "type": "a" }, { "href": "https://blog.google/technology/ai/google-gemini-ai/", "label": "on December 7th 2023", "type": "a" }, { "href": "https://simonwillison.net/2024/Apr/10/gemini-15-pro-public-preview/", "label": "in April", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/4/qwen2-vl/", "label": "Qwen2-VL", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/11/pixtral/", "label": "Pixtral 12B", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/25/llama-32/", "label": "Llama 3.2 11B and 90B vision models", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/18/openai-audio/", "label": "from OpenAI in October", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/28/smolvlm/", "label": "SmolVLM from Hugging Face", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/4/amazon-nova/", "label": "from Amazon Nova", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/29/llm-multi-modal/", "label": "upgraded my LLM CLI tool to support multi-modal models via attachments", "type": "a" }, { "href": "https://openai.com/index/chatgpt-can-now-see-hear-and-speak/", "label": "in September 2023", "type": "a" }, { "href": "https://platform.openai.com/docs/models", "label": "tts-1", "type": "a" }, { "href": "https://openai.com/index/hello-gpt-4o/", "label": "May 13th", "type": "a" }, { "href": "https://www.nytimes.com/2024/05/20/technology/scarlett-johansson-openai-statement.html", "label": "conspicuously similar to Scarlett Johansson", "type": "a" }, { "href": "https://simonwillison.net/2024/May/15/chatgpt-in-4o-mode/", "label": "ChatGPT in “4o” mode is not running the new features yet", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/26/russian-spanish-pelican/", "label": "I need you to pretend to be a California brown pelican with a very thick Russian accent, but you talk to me exclusively ", "type": "a" }, { "href": "https://static.simonwillison.net/static/2024/russian-pelican-in-spanish.mp3", "label": "Your browser does not support the audio element.", "type": "source" }, { "href": "https://simonwillison.net/2024/Sep/29/notebooklm-audio-overview/", "label": "in September", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/17/notebooklm-pelicans/", "label": "I turned them into pelicans", "type": "a" }, { "href": "https://static.simonwillison.net/static/2024/video-scraping-pelicans.mp3", "label": "Your browser does not support the audio element.", "type": "source" }, { "href": "https://simonwillison.net/2024/Dec/20/december-in-llms-has-been-a-lot/", "label": "a lot", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/11/gemini-2/", "label": "a preview of the same feature", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/2/not-digital-god/", "label": "a WebSocket API", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/17/openai-webrtc/", "label": "a new WebRTC API", "type": "a" }, { "href": "https://www.anthropic.com/news/claude-3-5-sonnet", "label": "their announcement of the incredible Claude 3.5 Sonnet", "type": "a" }, { "href": "https://tools.simonwillison.net/extract-urls", "label": "Extract URLs", "type": "a" }, { "href": "https://gist.github.com/simonw/0a7d0ddeb0fdd63a844669475778ca06", "label": "generated by Claude", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/21/claude-artifacts/", "label": "Everything I built with Claude Artifacts this week", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/30/copilot-models/", "label": "GitHub Spark", "type": "a" }, { "href": "https://mistral.ai/news/mistral-chat/", "label": "added it as a feature called Canvas", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/31/cerebras-coder/", "label": "built a version of it against Cerebras", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/16/webdev-arena/", "label": "a whole new leaderboard", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/19/one-shot-python-tools/", "label": "writing one-shot Python programs, enabled by uv", "type": "a" }, { "href": "https://openai.com/index/introducing-chatgpt-pro/", "label": "ChatGPT Pro", "type": "a" }, { "href": "https://til.simonwillison.net/twitter/collecting-replies", "label": "collected 211 definitions", "type": "a" }, { "href": "https://lite.datasette.io/?json=https://gist.github.com/simonw/bdc7b894eedcfd54f0a2422ea8feaa80", "label": "in Datasette Lite", "type": "a" }, { "href": "https://gist.github.com/simonw/beaa5f90133b30724c5cc1c4008d0654", "label": "attempt to summarize them", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/29/encanto-2/", "label": "serving up an entirely fake description", "type": "a" }, { "href": "https://ideas.fandom.com/wiki/Encanto_2:_A_New_Generation", "label": "a fan fiction wiki", "type": "a" }, { "href": "https://simonwillison.net/series/prompt-injection/", "label": "Prompt injection", "type": "a" }, { "href": "https://simonwillison.net/2022/Sep/12/prompt-injection/", "label": "since September 2022", "type": "a" }, { "href": "https://twitter.com/amandaaskell/status/1866207266761760812", "label": "Amanda Askell", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/8/claudes-character/", "label": "the work behind Claude’s Character", "type": "a" }, { "href": "https://twitter.com/cramforce/status/1860436022347075667", "label": "Malte Ubl", "type": "a" }, { "href": "https://twitter.com/v0", "label": "@v0", "type": "a" }, { "href": "https://simonwillison.net/tags/evals/", "label": "evals tag", "type": "a" }, { "href": "https://simonwillison.net/tags/pelican-riding-a-bicycle/", "label": "SVG pelican riding a bicycle benchmark", "type": "a" }, { "href": "https://github.com/ggerganov/llama.cpp", "label": "llama.cpp", "type": "a" }, { "href": "https://github.com/ml-explore/mlx", "label": "MLX", "type": "a" }, { "href": "https://github.com/ml-explore/mlx-examples/tree/main/llms", "label": "mlx-lm", "type": "a" }, { "href": "https://huggingface.co/mlx-community", "label": "mlx-community", "type": "a" }, { "href": "https://github.com/Blaizzy/mlx-vlm", "label": "mlx-vlm", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/24/qvq/", "label": "to run Qwen’s QvQ", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/10/apple-intelligence/", "label": "wrote about their initial announcement in June", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/14/bbc-complains-to-apple-over-misleading-shooting-headline/", "label": "misrepresent news headlines", "type": "a" }, { "href": "https://bsky.app/profile/simonwillison.net/post/3leceujwvcc2x", "label": "kind of fun though", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/12/openai-o1/", "label": "on September 12th", "type": "a" }, { "href": "https://arxiv.org/abs/2205.11916", "label": "Large Language Models are Zero-Shot Reasoners", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/20/live-blog-the-12th-day-of-openai/", "label": "on 20th December", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/20/openai-o3-breakthrough/", "label": "ARC-AGI benchmark", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/19/gemini-thinking-mode/", "label": "on December 19th", "type": "a" }, { "href": "https://qwenlm.github.io/blog/qwq-32b-preview/", "label": "released their QwQ model", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/27/qwq/", "label": "I could run on my own machine", "type": "a" }, { "href": "https://qwenlm.github.io/blog/qvq-72b-preview/", "label": "on December 24th", "type": "a" }, { "href": "https://api-docs.deepseek.com/news/news1120", "label": "DeepSeek-R1-Lite-Preview", "type": "a" }, { "href": "https://x.com/deepseek_ai/status/1859200141355536422", "label": "on November 20th", "type": "a" }, { "href": "https://www.aisnakeoil.com/p/is-ai-progress-slowing-down", "label": "Is AI progress slowing down?", "type": "a" }, { "href": "https://arxiv.org/abs/2412.06769", "label": "Training Large Language Models to Reason in a Continuous Latent Space", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/25/deepseek-v3/", "label": "of DeepSeek v3", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/26/deepseek-v3/", "label": "the day after that", "type": "a" }, { "href": "https://www.cnbc.com/2023/10/17/us-bans-export-of-more-ai-chips-including-nvidia-h800-to-china.html", "label": "US export regulations", "type": "a" }, { "href": "https://www.bloomberg.com/graphics/2024-ai-power-home-appliances/", "label": "on the electricity grid", "type": "a" }, { "href": "https://www.nytimes.com/2024/10/16/business/energy-environment/amazon-google-microsoft-nuclear-energy.html", "label": "spinning up new nuclear power stations", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Panic_of_1873", "label": "Panic of 1873", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Panic_of_1893", "label": "Panic of 1893", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Panic_of_1901", "label": "Panic of 1901", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Railway_Mania", "label": "Railway Mania", "type": "a" }, { "href": "https://simonwillison.net/tags/slop/", "label": "slop", "type": "a" }, { "href": "https://simonwillison.net/2024/May/8/slop/", "label": "in May", "type": "a" }, { "href": "https://twitter.com/deepfates/status/1787472784106639418", "label": "@deepfates", "type": "a" }, { "href": "https://www.theguardian.com/technology/article/2024/may/19/spam-junk-slop-the-latest-wave-of-ai-behind-the-zombie-internet", "label": "the Guardian", "type": "a" }, { "href": "https://www.nytimes.com/2024/06/11/style/ai-search-slop.html", "label": "the NY Times", "type": "a" }, { "href": "https://corp.oup.com/news/voting-opens-for-oxford-word-of-the-year-2024/", "label": "Oxford Word of the Year 2024", "type": "a" }, { "href": "https://corp.oup.com/news/brain-rot-named-oxford-word-of-the-year-2024/", "label": "to brain rot", "type": "a" }, { "href": "https://arxiv.org/abs/2305.17493", "label": "The Curse of Recursion: Training on Generated Data Makes Models Forget", "type": "a" }, { "href": "https://www.nature.com/articles/s41586-024-07566-y", "label": "AI models collapse when trained on recursively generated data", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/15/phi-4-technical-report/", "label": "the Phi-4 technical report", "type": "a" }, { "href": "https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md", "label": "over 25M synthetically generated examples", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/10/chatgpt-canvas/", "label": "two entirely different ways", "type": "a" }, { "href": "https://chatgpt.com/share/677420e4-8854-8006-8940-9bc30b708821", "label": "will pretend to do those things", "type": "a" }, { "href": "https://simonwillison.net/2023/Mar/10/chatgpt-internet-access/", "label": "URL hallucinations bug from early 2023", "type": "a" }, { "href": "https://web.archive.org/web/20241230103630/https://help.openai.com/en/articles/10139238-santa-s-voice-in-chatgpt", "label": "pretend to be Santa", "type": "a" }, { "href": "https://fedi.simonwillison.net/@simon", "label": "Mastodon", "type": "a" }, { "href": "https://bsky.app/profile/simonwillison.net", "label": "Bluesky", "type": "a" }, { "href": "https://lobste.rs/", "label": "Lobste.rs", "type": "a" }, { "href": "https://news.ycombinator.com/", "label": "Hacker News", "type": "a" }, { "href": "https://simonwillison.net/tags/llms/", "label": "llms", "type": "a" }, { "href": "https://simonwillison.net/2024/Jan/7/call-it-ai/", "label": "It’s OK to call it Artificial Intelligence", "type": "a" }, { "href": "https://simonwillison.net/2024/Jan/9/what-i-should-have-said-about-ai/", "label": "What I should have said about the term Artificial Intelligence", "type": "a" }, { "href": "https://simonwillison.net/2024/Jan/17/oxide-and-friends/", "label": "Talking about Open Source LLMs on Oxide and Friends", "type": "a" }, { "href": "https://simonwillison.net/2024/Jan/26/llm/", "label": "LLM 0.13: The annotated release notes", "type": "a" }, { "href": "https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/", "label": "Prompt injection and jailbreaking are not the same thing", "type": "a" }, { "href": "https://simonwillison.net/2024/Mar/8/gpt-4-barrier/", "label": "The GPT-4 barrier has finally been broken", "type": "a" }, { "href": "https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-case-study/", "label": "Claude and ChatGPT for ad-hoc sidequests", "type": "a" }, { "href": "https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/", "label": "Building and testing C extensions for SQLite with ChatGPT Code Interpreter", "type": "a" }, { "href": "https://simonwillison.net/2024/Mar/26/llm-cmd/", "label": "llm cmd undo last git commit—a new plugin for LLM", "type": "a" }, { "href": "https://simonwillison.net/2024/Apr/8/files-to-prompt/", "label": "Building files-to-prompt entirely using Claude 3 Opus", "type": "a" }, { "href": "https://simonwillison.net/2024/Apr/10/weeknotes-llm-releases/", "label": "Three major LLM releases in 24 hours (plus weeknotes)", "type": "a" }, { "href": "https://simonwillison.net/2024/Apr/17/ai-for-data-journalism/", "label": "AI for Data Journalism: demonstrating what we can do with this stuff right now", "type": "a" }, { "href": "https://simonwillison.net/2024/Apr/22/llama-3/", "label": "Options for accessing Llama 3 from the terminal using LLM", "type": "a" }, { "href": "https://simonwillison.net/2024/May/29/training-not-chatting/", "label": "Training is not the same as chatting: ChatGPT and other LLMs don’t remember everything you say", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/6/accidental-prompt-injection/", "label": "Accidental prompt injection against RAG applications", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/17/cli-language-models/", "label": "Language models on the command-line", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/21/search-based-rag/", "label": "Building search-based RAG using Claude, Datasette and Val Town", "type": "a" }, { "href": "https://simonwillison.net/2024/Jun/27/ai-worlds-fair/", "label": "Open challenges for AI engineering", "type": "a" }, { "href": "https://simonwillison.net/2024/Jul/19/weeknotes/", "label": "Weeknotes: GPT-4o mini, LLM 0.15, sqlite-utils 3.37 and building a staging environment", "type": "a" }, { "href": "https://simonwillison.net/2024/Aug/6/staging/", "label": "Weeknotes: a staging environment, a Datasette alpha and a bunch of new LLMs", "type": "a" }, { "href": "https://simonwillison.net/2024/Aug/8/django-http-debug/", "label": "django-http-debug, a new Django app mostly written by Claude", "type": "a" }, { "href": "https://simonwillison.net/2024/Aug/23/anthropic-dangerous-direct-browser-access/", "label": "Claude’s API now supports CORS requests, enabling client-side applications", "type": "a" }, { "href": "https://simonwillison.net/2024/Aug/26/gemini-bounding-box-visualization/", "label": "Building a tool showing how Gemini Pro can return bounding boxes for objects in images", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/6/weeknotes/", "label": "Calling LLMs from client-side JavaScript, converting PDFs to HTML + weeknotes", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/10/software-misadventures/", "label": "Notes from my appearance on the Software Misadventures Podcast", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/20/using-llms-for-code/", "label": "Notes on using LLMs for code", "type": "a" }, { "href": "https://simonwillison.net/2024/Sep/30/weeknotes/", "label": "Weeknotes: Three podcasts, two trips and a new plugin system", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/1/openai-devday-2024-live-blog/", "label": "OpenAI DevDay 2024 live blog", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/15/chatgpt-horoscopes/", "label": "ChatGPT will happily write you a thinly disguised horoscope", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/17/video-scraping/", "label": "Video scraping: extracting JSON data from a 35 second screen capture for less than 1/10th of a cent", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/19/mistralrs/", "label": "Running Llama 3.2 Vision and Phi-3.5 Vision on a Mac with mistral.rs", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/22/computer-use/", "label": "Initial explorations of Anthropic’s new Computer Use capability", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/24/claude-analysis-tool/", "label": "Notes on the new Claude analysis JavaScript code execution tool", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/27/llm-jq/", "label": "Run a prompt to generate and execute jq programs using llm-jq", "type": "a" }, { "href": "https://simonwillison.net/2024/Oct/30/monthnotes/", "label": "W̶e̶e̶k̶n̶o̶t̶e̶s̶ Monthnotes for October", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/4/haiku/", "label": "Claude 3.5 Haiku", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/7/project-verdad/", "label": "Project: VERDAD—tracking misinformation in radio broadcasts using Gemini 1.5", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/19/notes-from-bing-chat/", "label": "Notes from Bing Chat—Our First Encounter With Manipulative AI", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/25/ask-questions-of-sqlite/", "label": "Ask questions of SQLite databases and CSV/JSON files in your terminal", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/7/prompts-js/", "label": "Prompts.js", "type": "a" }, { "href": "https://simonwillison.net/dashboard/llm-posts-in-2024/", "label": "using Django SQL Dashboard", "type": "a" }, { "href": "https://gist.github.com/simonw/89c358ac3617b38afc41c79c995a4ebe", "label": "written for me by Claude", "type": "a" }, { "href": "https://simonwillison.net/2024/Dec/31/", "label": "31st December 2024", "type": "a" }, { "href": "https://twitter.com/simonw", "label": "Twitter", "type": "a" }, { "href": "https://simonwillison.net/about/", "label": "subscribe to my newsletter", "type": "a" }, { "href": "https://simonwillison.net/2026/Oct/12/python-3-15/", "label": "Hello Python 3.15, goodbye 3.10, and now I can use 3.11", "type": "a" }, { "href": "https://simonwillison.net/2026/Oct/9/built-using-my-voice/", "label": "A new feature for my blog, built using my voice", "type": "a" }, { "href": "https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/", "label": "Claude Haiku 5.5", "type": "a" }, { "href": "https://simonwillison.net/2026/Oct/3/default-hard-budget-caps/", "label": "We're going to need default hard budget caps on pretty much everything", "type": "a" }, { "href": "https://simonwillison.net/series/llms-annual-review/", "label": "LLMs annual review", "type": "a" }, { "href": "https://simonwillison.net/2023/Oct/17/open-questions/", "label": "Open questions for AI engineering", "type": "a" }, { "href": "https://simonwillison.net/2025/Jun/6/six-months-in-llms/", "label": "The last six months in LLMs, illustrated by pelicans on bicycles", "type": "a" }, { "href": "https://simonwillison.net/2025/Dec/31/the-year-in-llms/", "label": "2025: The year in LLMs", "type": "a" }, { "href": "https://simonwillison.net/tags/google/", "label": "google 418", "type": "a" }, { "href": "https://simonwillison.net/tags/ai/", "label": "ai 2,279", "type": "a" }, { "href": "https://simonwillison.net/tags/openai/", "label": "openai 475", "type": "a" }, { "href": "https://simonwillison.net/tags/generative-ai/", "label": "generative-ai 2,021", "type": "a" }, { "href": "https://simonwillison.net/tags/local-llms/", "label": "local-llms 165", "type": "a" }, { "href": "https://simonwillison.net/tags/anthropic/", "label": "anthropic 348", "type": "a" }, { "href": "https://simonwillison.net/tags/gemini/", "label": "gemini 199", "type": "a" }, { "href": "https://simonwillison.net/tags/meta/", "label": "meta 45", "type": "a" }, { "href": "https://simonwillison.net/tags/llm-reasoning/", "label": "llm-reasoning 105", "type": "a" }, { "href": "https://simonwillison.net/tags/long-context/", "label": "long-context 20", "type": "a" }, { "href": "https://simonwillison.net/tags/ai-energy-usage/", "label": "ai-energy-usage 18", "type": "a" }, { "href": "https://simonwillison.net/tags/coding-agents/", "label": "coding-agents 257", "type": "a" }, { "href": "https://simonwillison.net/2025/Jan/2/ending-a-year-long-posting-streak/", "label": "Ending a year long posting streak", "type": "a" }, { "href": "https://github.com/sponsors/simonw/", "label": "Sponsor & subscribe", "type": "a" }, { "href": "https://simonwillison.net/2002/", "label": "2002", "type": "a" }, { "href": "https://simonwillison.net/2003/", "label": "2003", "type": "a" }, { "href": "https://simonwillison.net/2004/", "label": "2004", "type": "a" }, { "href": "https://simonwillison.net/2005/", "label": "2005", "type": "a" }, { "href": "https://simonwillison.net/2006/", "label": "2006", "type": "a" }, { "href": "https://simonwillison.net/2007/", "label": "2007", "type": "a" }, { "href": "https://simonwillison.net/2008/", "label": "2008", "type": "a" }, { "href": "https://simonwillison.net/2009/", "label": "2009", "type": "a" }, { "href": "https://simonwillison.net/2010/", "label": "2010", "type": "a" }, { "href": "https://simonwillison.net/2011/", "label": "2011", "type": "a" }, { "href": "https://simonwillison.net/2012/", "label": "2012", "type": "a" }, { "href": "https://simonwillison.net/2013/", "label": "2013", "type": "a" }, { "href": "https://simonwillison.net/2014/", "label": "2014", "type": "a" }, { "href": "https://simonwillison.net/2015/", "label": "2015", "type": "a" }, { "href": "https://simonwillison.net/2016/", "label": "2016", "type": "a" }, { "href": "https://simonwillison.net/2017/", "label": "2017", "type": "a" }, { "href": "https://simonwillison.net/2018/", "label": "2018", "type": "a" }, { "href": "https://simonwillison.net/2019/", "label": "2019", "type": "a" }, { "href": "https://simonwillison.net/2020/", "label": "2020", "type": "a" }, { "href": "https://simonwillison.net/2021/", "label": "2021", "type": "a" }, { "href": "https://simonwillison.net/2022/", "label": "2022", "type": "a" }, { "href": "https://simonwillison.net/2023/", "label": "2023", "type": "a" }, { "href": "https://simonwillison.net/2024/", "label": "2024", "type": "a" }, { "href": "https://simonwillison.net/2025/", "label": "2025", "type": "a" }, { "href": "https://simonwillison.net/2026/", "label": "2026", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:47:30Z", "source_url": "https://simonwillison.net/2024/Dec/31/llms-in-2024/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2205.11916", "bytes": 762445, "reused": true, "local_path": "papers/repair/7ee8d7eb1358c6773ecd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2412.06769", "bytes": 3211318, "reused": false, "local_path": "papers/repair/68b67bf331c0b5bab5c9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.17493", "bytes": 2354065, "reused": false, "local_path": "papers/repair/853ddb8d354be8ee27f1.pdf" } ], "actual_video_files": [ { "local_path": "attachments/403684ddeb63/videos/XEzRZ35urlk.mp4", "bytes": 635208725, "valid": true, "height": 1080, "width": 1920, "duration": "6763.141000", "status": "ok" } ], "body_chars": 42354, "expected_body_images": 4, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/403684ddeb63/pages/source.html", "local_html": "attachments/403684ddeb63/pages/article.html", "body_base_path": "attachments/403684ddeb63/pages/", "body_markdown_path": "attachments/403684ddeb63/pages/article.md" }