{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Simon Willison's Weblog",
  "title_listed": "Embeddings: What They Are and Why They Matter",
  "keywords": "嵌入,向量,语义",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://simonwillison.net/2023/Oct/23/embeddings/",
  "publish_date": "2023-10",
  "corpus_stage": "数据处理",
  "notes": "嵌入向量解析",
  "id": "a310d3f7beb9",
  "collected_at": "2026-10-01T20:45:06",
  "title": "Embeddings: What they are and why they matter",
  "body_markdown": "Simon Willison’s Weblog\nNewsletter\nSponsored by:\nPomerium — Static OAuth scopes won't stop AI agent breakouts.\nUpgrade to Pomerium's agentic access management trusted by NVIDIA and OpenAI\nEmbeddings: What they are and why they matter\n23rd October 2023\nEmbeddings are a really neat trick that often come wrapped in a pile of intimidating jargon.\nIf you can make it through that jargon, they unlock powerful and exciting techniques that can be applied to all sorts of interesting problems.\nI gave a talk about embeddings at\nPyBay 2023\n. This article represents an improved version of that talk, which should stand alone even without watching the video.\nIf you’re not yet familiar with embeddings I hope to give you everything you need to get started applying them to real-world problems.\nIn this article:\nThe 38 minute video version\nWhat are embeddings?\nRelated content using embeddings\nExploring how these things work with Word2Vec\nCalculating embeddings using my LLM tool\nVibes-based search\nEmbeddings for code using Symbex\nEmbedding text and images together using CLIP\nFaucet Finder: finding faucets with CLIP\nClustering embeddings\nVisualize in 2D with Principal Component Analysis\nScoring sentences using average locations\nAnswering questions with Retrieval-Augmented Generation\nQ&A\nFurther reading\nThe 38 minute video version\nHere’s\na video\nof the talk that I gave at PyBay:\nThe audio quality of the official video wasn’t great due to an issue with the microphone, but I ran that audio through Adobe’s\nEnhance Speech tool\nand uploaded my own video with the enhanced audio to YouTube.\nWhat are embeddings?\nEmbeddings are a technology that’s adjacent to the wider field of Large Language Models—the technology behind ChatGPT and Gemini and Claude.\nEmbeddings are based around one trick: take a piece of content—in this case\na blog entry\n—and turn that piece of content into an array of floating point numbers.\nThe key thing about that array is that it will always be the same length, no matter how long the content is. The length is defined by the embedding model you are using—an array might be 300, or 1,000, or 1,536 numbers long.\nThe best way to think about this array of numbers is to imagine it as co-ordinates in a very weird multi-dimensional space.\nIt’s hard to visualize 1,536 dimensional space, so here’s a 3D visualization of the same idea:\nWhy place content in this space? Because we can learn interesting things about that content based on its location—in particular, based on what else is nearby.\nThe location within the space represents the semantic meaning of the content, according to the embedding model’s weird, mostly incomprehensible understanding of the world. It might capture colors, shapes, concepts or all sorts of other characteristics of the content that has been embedded.\nNobody fully understands what those individual numbers mean, but we know that their locations can be used to find out useful things about the content.\nRelated content using embeddings\nOne of the first problems I solved with embeddings was to build a “related content” feature for\nmy TIL blog\n. I wanted to be able to show a list of related articles at the bottom of each page.\nI did this using embeddings—in this case, I used the OpenAI\ntext-embedding-ada-002\nmodel, which is available\nvia their API\n.\nI currently have 472 articles on my site. I calculated the 1,536 dimensional embedding vector (array of floating point numbers) for each of those articles, and stored those vectors in my site’s SQLite database.\nNow, if I want to find related articles for a given article, I can calculate the\ncosine similarity\nbetween the embedding vector for that article and every other article in the database, then return the 10 closest matches by distance.\nThere’s an example at\nthe bottom of this page\n. The top five related articles for\nGeospatial SQL queries in SQLite using TG, sqlite-tg and datasette-sqlite-tg\nare:\nGeopoly in SQLite\n—2023-01-04\nViewing GeoPackage data with SpatiaLite and Datasette\n—2022-12-11\nUsing SQL with GDAL\n—2023-03-09\nKNN queries with SpatiaLite\n—2021-05-16\nGUnion to combine geometries in SpatiaLite\n—2022-04-12\nThat’s a pretty good list!\nHere’s\nthe Python function\nI’m using to calculate those cosine similarity distances:\ndef\ncosine_similarity\n(\na\n,\nb\n):\ndot_product\n=\nsum\n(\nx\n*\ny\nfor\nx\n,\ny\nin\nzip\n(\na\n,\nb\n))\nmagnitude_a\n=\nsum\n(\nx\n*\nx\nfor\nx\nin\na\n)\n**\n0.5\nmagnitude_b\n=\nsum\n(\nx\n*\nx\nfor\nx\nin\nb\n)\n**\n0.5\nreturn\ndot_product\n/\n(\nmagnitude_a\n*\nmagnitude_b\n)\nMy TIL site runs on my\nDatasette\nPython framework, which supports building sites on top of a SQLite database. I wrote more about how that works in\nthe Baked Data architectural pattern\n.\nYou can browse the SQLite table that stores the calculated embeddings at\ntils/embeddings\n.\nThose are binary values. We can\nrun this SQL query\nto view them as hexadecimal:\nselect\nid, hex(embedding)\nfrom\nembeddings\nThat’s still not very readable though. We can use the\nllm_embed_decode()\ncustom SQL function to turn them into a JSON array:\nselect\nid, llm_embed_decode(embedding)\nfrom\nembeddings\nlimit\n10\nTry that here\n. It shows that each article is accompanied by that array of 1,536 floating point numbers.\nWe can use another custom SQL function,\nllm_embed_cosine(vector1, vector2)\n, to calculate those cosine distances and find the most similar content.\nThat SQL function\nis defined here\nin my\ndatasette-llm-embed\nplugin.\nHere’s a query returning the five most similar articles to my SQLite TG article:\nselect\nid,\n  llm_embed_cosine(\n    embedding,\n    (\nselect\nembedding\nfrom\nembeddings\nwhere\nid\n=\n'\nsqlite_sqlite-tg.md\n'\n)\n  )\nas\nscore\nfrom\nembeddings\norder by\nscore\ndesc\nlimit\n5\nExecuting that query\nreturns the following results:\nid\nscore\nsqlite_sqlite-tg.md\n1.0\nsqlite_geopoly.md\n0.8817322855676049\nspatialite_viewing-geopackage-data-with-spatialite-and-datasette.md\n0.8813094978399854\ngis_gdal-sql.md\n0.8799581261326747\nspatialite_knn.md\n0.8692992294266506\nAs expected, the similarity between the article and itself is 1.0. The other articles are all related to geospatial SQL queries in SQLite.\nThis query takes around 400ms to execute. To speed things up, I pre-calculate the top 10 similarities for every article and store them in a separate table called\ntils/similarities\n.\nI wrote a Python function to\nlook up related documents from that table\nand\ncalled it from the template\nthat’s used to render the article page.\nMy\nStoring and serving related documents with openai-to-sqlite and embeddings\nTIL explains how this all works in detail, including how GitHub Actions are used to fetch new embeddings\nas part of the build script\nthat deploys the site.\nI used the OpenAI embeddings API for this project. It’s extremely inexpensive—for my TIL website I embedded around 402,500 tokens, which at $0.0001 / 1,000 tokens comes to $0.04—just 4 cents!\nIt’s really easy to use: you POST it some text along with your API key, it gives you back that JSON array of floating point numbers.\nBut... it’s a proprietary model. A few months ago OpenAI\nshut down some of their older embeddings models\n, which is a problem if you’ve stored large numbers of embeddings from those models since you’ll need to recalculate them against a supported model if you want to be able to embed anything else new.\nTo OpenAI’s credit, they did promise to “cover the financial cost of users re-embedding content with these new models.”—but it’s still a reason to be cautious about relying on proprietary models.\nThe good news is that there are extremely powerful openly licensed models which you can run on your own hardware, avoiding any risk of them being shut down. We’ll talk about that more in a moment.\nExploring how these things work with Word2Vec\nGoogle Research put out an influential paper 10 years ago describing an early embedding model they created called Word2Vec.\nThat paper is\nEfficient Estimation of Word Representations in Vector Space\n, dated 16th January 2013. It’s a paper that helped kick off widespread interest in embeddings.\nWord2Vec is a model that takes single words and turns them into a list of 300 numbers. That list of numbers captures something about the meaning of the associated word.\nThis is best illustrated by a demo.\nturbomaze.github.io/word2vecjson\nis an interactive tool put together by\nAnthony Liu\nwith a 10,000 word subset of the Word2Vec corpus. You can view\nthis JavaScript file\nto see the JSON for those 10,000 words and their associated 300-long arrays of numbers.\nSearch for a word to find similar words based on cosine distance to their Word2Vec representation. For example, the word “france” returns the following related results:\nword\nsimilarity\nfrance\n1\nfrench\n0.7000748343471224\nbelgium\n0.6933180492111168\nparis\n0.6334910653433325\ngermany\n0.627075617939471\nitaly\n0.6135215284228007\nspain\n0.6064218103692152\nThat’s a mixture of french things and European geography.\nA really interesting thing you can do here is perform arithmetic on these vectors.\nTake the vector for “germany”, add “paris” and subtract “france”. The resulting vector is closest to “berlin”!\nSomething about this model has captured the idea of nationalities and geography to the point that you can use arithmetic to explore additional facts about the world.\nWord2Vec was trained on 1.6 billion words of content. The embedding models we use today are trained on much larger datasets and capture a much richer understanding of the underlying relationships.\nCalculating embeddings using my LLM tool\nI’ve been building a command-line utility and Python library called\nLLM\n.\nYou can read more about LLM here:\nllm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs\nThe LLM CLI tool now supports self-hosted language models via plugins\nLLM now provides tools for working with embeddings\nBuild an image search engine with llm-clip, chat with models with llm chat\nLLM is a tool for working with Large Language Models. You can install it like this:\npip install llm\nOr via Homebrew:\nbrew install llm\nYou can use it as a command-line tool for interacting with LLMs, or as\na Python library\n.\nOut of the box it can work with the OpenAI API. Set\nan API key\nand you can run commands like this:\nllm\n'\nten fun names for a pet pelican\n'\nWhere it gets really fun is when you start\ninstalling plugins\n. There are plugins that add entirely new language models to it, including models that run directly on your own machine.\nA few months ago\nI extended LLM\nto support plugins that can run embedding models as well.\nHere’s how to run the catchily titled\nall-MiniLM-L6-v2\nmodel using LLM:\nFirst, we install\nllm\nand then use that to install the\nllm-sentence-transformers\nplugin—a wrapper around the\nSentenceTransformers\nlibrary.\npip install llm\nllm install llm-sentence-transformers\nNext we need to register the\nall-MiniLM-L6-v2\nmodel. This will download the model from Hugging Face to your computer:\nllm sentence-transformers register all-MiniLM-L6-v2\nWe can test that out by embedding a single sentence like this:\nllm embed -m sentence-transformers/all-MiniLM-L6-v2 \\\n  -c\n'\nHello world\n'\nThis outputs a JSON array that starts like this:\n[-0.03447725251317024, 0.031023245304822922, 0.006734962109476328, 0.026108916848897934, -0.03936201333999634, ...\nEmbeddings like this on their own aren’t very interesting—we need to store and compare them to start getting useful results.\nLLM can store embeddings in a “collection”—a SQLite table. The\nembed-multi command\ncan be used to embed multiple pieces of content at once and store them in a collection.\nThat’s what this next command does:\nllm embed-multi readmes \\\n  --model sentence-transformers/all-MiniLM-L6-v2 \\\n  --files\n~\n/\n'\n**/README.md\n'\n--store\nHere we are populating a collection called “readmes”.\nThe\n--files\noption takes two arguments: a directory to search and a glob pattern to match against filenames. In this case I’m searching my home directory recursively for any file named\nREADME.md\n.\nThe\n--store\noption causes LLM to store the raw text in the SQLite table in addition to the embedding vector.\nThis command took around 30 minutes to run on my computer, but it worked! I now have a collection called\nreadmes\nwith 16,796 rows—one for each\nREADME.md\nfile it found in my home directory.\nVibes-based search\nNow that we have a collection of embeddings, we can run searches against it using the\nllm similar command\n:\nllm similar readmes -c\n'\nsqlite backup tools\n'\n|\njq .id\nWe are asking for items in the\nreadmes\ncollection that are similar to the embedding vector for the phrase “sqlite backup tools”.\nThis command outputs JSON by default, which includes the full text of the README files since we stored them using\n--store\nearlier.\nPiping the results through\njq .id\ncauses the command to output just the IDs of the matching rows.\nThe top matching results are:\n\"sqlite-diffable/README.md\"\n\"sqlite-dump/README.md\"\n\"ftstri/salite/ext/repair/README.md\"\n\"simonw/README.md\"\n\"sqlite-generate/README.md\"\n\"sqlite-history/README.md\"\n\"dbf-to-sqlite/README.md\"\n\"ftstri/sqlite/ext/README.md\"\n\"sqlite-utils/README.md\"\n\"ftstri/sqlite/README.md'\nThese are good results! Each of these READMEs either describes a tool for working with SQLite backups or a project that relates to backups ins ome way.\nWhat’s interesting about this is that it’s not guaranteed that the term “backups” appeared directly in the text of those READMEs. The content is semantically similar to that phrase, but might not be an exact textual match.\nWe can call this semantic search. I like to think of it as\nvibes-based search\n.\nThe vibes of those READMEs relate to our search term, according to this weird multi-dimensional space representation of the meaning of words.\nThis is absurdly useful. If you’ve ever built a search engine for a website, you know that exact matches don’t always help people find what they are looking for.\nWe can use this kind of semantic search to build better search engines for a whole bunch of different kinds of content.\nEmbeddings for code using Symbex\nAnother tool I’ve been building is called\nSymbex\n. It’s a tool for exploring the symbols in a Python codebase.\nI\noriginally built it\nto help quickly find Python functions and classes and pipe them into LLMs to help explain and rewrite them.\nThen I realized that I could use it to calculate embeddings for all of the functions in a codebase, and use those embeddings to build a code search engine.\nI added a feature that could output JSON or CSV representing the symbols it found, using the same output format that\nllm embed-multi\ncan use as an input.\nHere’s how I built a collection of all of the functions in my\nDatasette\nproject, using a newly released model called\ngte-tiny\n—just a 60MB file!\nllm sentence-transformers register TaylorAI/gte-tiny\ncd\ndatasette/datasette\n\nsymbex\n'\n*\n'\n'\n*:*\n'\n--nl\n|\n\\\n  llm embed-multi functions - \\\n  --model sentence-transformers/TaylorAI/gte-tiny \\\n  --format nl \\\n  --store\nsymbex '*' '*:*' --nl\nfinds all functions (\n*\n) and class methods (the\n*:*\npattern) in the current directory and outputs them as newline-delimited JSON.\nThe\nllm embed-multi ... --format nl\ncommand expects newline-delimited JSON as input, so we can pipe the output of\nsymbex\ndirectly into it.\nThis defaults to storing the embeddings in the default LLM SQLite database. You can add\n--database /tmp/data.db\nto specify an alternative location.\nAnd now... I can run vibes-based semantic search against my codebase!\nI could use the\nllm similar\ncommand for this, but I also have the ability to run these searches using Datasette itself.\nHere’s a SQL query for that, using the\ndatasette-llm-embed\nplugin from earlier:\nwith input\nas\n(\nselect\nllm_embed(\n'\nsentence-transformers/TaylorAI/gte-tiny\n'\n,\n      :input\n    )\nas\ne\n)\nselect\nid,\n  content\nfrom\nembeddings,\n  input\nwhere\ncollection_id\n=\n(\nselect\nid\nfrom\ncollections\nwhere\nname\n=\n'\nfunctions\n'\n)\norder by\nllm_embed_cosine(embedding,\ninput\n.\ne\n)\ndesc\nlimit\n5\nThe\n:input\nparameter is automatically turned into a form field by Datasette.\nWhen I run this, I get back functions that relate to the concept of listing plugins:\nThe key idea here is to use SQLite as an integration point—a substrate for combining together multiple tools.\nI can run separate tools that extract functions from a codebase, run them through an embedding model, write those embeddings to SQLite and then run queries against the results.\nAnything that can be piped into a tool can now be embedded and processed by the other components of this ecosystem.\nEmbedding text and images together using CLIP\nMy current favorite embedding model is\nCLIP\n.\nCLIP is a fascinating model released by OpenAI—back in January 2021, when they were still doing most things in the open—that can embed both text and images.\nCrucially, it embeds them both into the same vector space.\nIf you embed the string “dog”, you’ll get a location in 512 dimensional space (depending on your CLIP configuration).\nIf you embed a photograph of a dog, you’ll get a location in that same space... and it will be close in terms of distance to the location of the string “dog”!\nThis means we can search for related images using text, and search for related text using images.\nI built\nan interactive demo\nto help explain how this works. The demo is an Observable notebook that runs the CLIP model directly in the browser.\nIt’s a pretty heavy page—it has to load 158MB of resources (64.6MB for the CLIP text model and 87.6MB for the image model)—but once loaded you can use it to embed an image, then embed a string of text and calculate the distance between the two.\nI can give it this photo I took of a beach:\nThen type in different text strings to calculate a similarity score, here displayed as a percentage:\ntext\nscore\nbeach\n26.946%\ncity\n19.839%\nsunshine\n24.146%\nsunshine beach\n26.741%\ncalifornia\n25.686%\ncalifornia beach\n27.427%\nIt’s pretty amazing that we can do all of this in JavaScript running in the browser!\nThere’s an obvious catch: it’s not actually that useful to be able to take an arbitrary photo and say “how similar is this to the term ’city’?”.\nThe trick is to build additional interfaces on top of this. Once again, we have the ability to build vibes-based search engines.\nHere’s a great example of one of those.\nFaucet Finder: finding faucets with CLIP\nDrew Breunig\nused LLM and my\nllm-clip\nplugin to build a search engine for faucet taps.\nHe was renovating his bathroom, and he needed to buy new faucets. So he scraped 20,000 photographs of faucets from a faucet supply company and ran CLIP against all of them.\nHe used the result to build\nFaucet Finder\n—a custom tool (deployed using Datasette) for finding faucets that look similar to other faucets.\nAmong other things, this means you can find an expensive faucet you like and then look for visually similar cheaper options!\nDrew wrote more about his project in\nFinding Bathroom Faucets with Embeddings\n.\nDrew’s demo uses pre-calculated embeddings to display similar results without having to run the CLIP model on the server.\nInspired by this, I spent some time figuring out\nhow to deploy a server-side CLIP model\nhosted by my own\nFly.io\naccount.\nDrew’s Datasette instance\nincludes this table\nof embedding vectors, exposed via the Datasette API.\nI deployed my own instance with\nthis API\nfor embedding text strings, then built an Observable notebook demo that hits both APIs and combines the results.\nobservablehq.com/@simonw/search-for-faucets-with-clip-api\nNow I can search for things like “gold purple” and get back vibes-based faucet results:\nBeing able to spin up this kind of ultra-specific search engine in a few hours is exactly the kind of trick that excites me about having embeddings as a tool in my toolbox.\nClustering embeddings\nRelated content and semantic / vibes-based search are the two most comon applications of embeddings, but there are a bunch of other neat things you can do with them too.\nOne of those is clustering.\nI built a plugin for this called\nllm-cluster\nwhich implements this using\nsklearn.cluster\nfrom scikit-learn.\nTo demonstrate that, I used my\npaginate-json\ntool and the GitHub issues API to collect the titles of all of the issues in my\nsimonw/llm\nrepository into a collection called\nllm-issues\n:\npaginate-json\n'\nhttps://api.github.com/repos/simonw/llm/issues?state=all&filter=all\n'\n\\\n|\njq\n'\n[.[] | {id: .id, title: .title}]\n'\n\\\n|\nllm embed-multi llm-issues - \\\n  --store\nNow I can create 10 clusters of issues like this:\nllm install llm-cluster\n\nllm cluster llm-issues 10\nClusters are output as a JSON array, with output that looks something like this (truncated):\n[\n  {\n\"id\"\n:\n\"\n2\n\"\n,\n\"items\"\n: [\n      {\n\"id\"\n:\n\"\n1650662628\n\"\n,\n\"content\"\n:\n\"\nInitial design\n\"\n},\n      {\n\"id\"\n:\n\"\n1650682379\n\"\n,\n\"content\"\n:\n\"\nLog prompts and responses to SQLite\n\"\n}\n    ]\n  },\n  {\n\"id\"\n:\n\"\n4\n\"\n,\n\"items\"\n: [\n      {\n\"id\"\n:\n\"\n1650760699\n\"\n,\n\"content\"\n:\n\"\nllm web command - launches a web server\n\"\n},\n      {\n\"id\"\n:\n\"\n1759659476\n\"\n,\n\"content\"\n:\n\"\n`llm models` command\n\"\n},\n      {\n\"id\"\n:\n\"\n1784156919\n\"\n,\n\"content\"\n:\n\"\n`llm.get_model(alias)` helper\n\"\n}\n    ]\n  },\n  {\n\"id\"\n:\n\"\n7\n\"\n,\n\"items\"\n: [\n      {\n\"id\"\n:\n\"\n1650765575\n\"\n,\n\"content\"\n:\n\"\n--code mode for outputting code\n\"\n},\n      {\n\"id\"\n:\n\"\n1659086298\n\"\n,\n\"content\"\n:\n\"\nAccept PROMPT from --stdin\n\"\n},\n      {\n\"id\"\n:\n\"\n1714651657\n\"\n,\n\"content\"\n:\n\"\nAccept input from standard in\n\"\n}\n    ]\n  }\n]\nThese do appear to be related, but we can do better. The\nllm cluster\ncommand has a\n--summary\noption which causes it to pass the resulting cluster text through a LLM and use it to generate a descriptive name for each cluster:\nllm cluster llm-issues 10 --summary\nThis gives back names like “Log Management and Interactive Prompt Tracking” and “Continuing Conversation Mechanism and Management”. See\nthe README\nfor more details.\nVisualize in 2D with Principal Component Analysis\nThe problem with massively multi-dimensional space is that it’s really hard to visualize.\nWe can use a technique called Principal Component Analysis to reduce the dimensionality of the data to a more manageable size—and it turns out lower dimensions continue to capture useful semantic meaning about the content.\nMatt Webb\nused the OpenAI embedding model to generate embeddings for descriptions of every episode of the BBC’s In Our Time podcast. He used these to find related episodes, but also ran PCA against them to create\nan interactive 2D visualization\n.\nReducing 1,536 dimensions to just two still produces a meaningful way of exploring the data! Episodes about historic wars show up near each other, elsewhere there’s a cluster of episodes about modern scientific discoveries.\nMatt wrote more about this in\nBrowse the BBC In Our Time archive by Dewey decimal code\n.\nScoring sentences using average locations\nAnother trick with embeddings is to use them for classification.\nFirst calculate the average location for a group of embeddings that you have classified in a certain way, then compare embeddings of new content to those locations to assign it to a category.\nAmelia Wattenberger demonstrated a beautiful example of this in\nGetting creative with embeddings\n.\nShe wanted to help people improve their writing by encouraging a mixture of concrete and abstract sentences. But how do you tell if a sentence of text is concrete or abstract?\nHer trick was to generate samples of the two types of sentence, calculate their average locations and then score new sentences based on how close they are to either end of that newly defined spectrum.\nThis score can even be converted into a color loosely representing how abstract or concrete a given sentence is!\nThis is a really neat demonstration of the kind of creative interfaces you can start to build on top of this technology.\nAnswering questions with Retrieval-Augmented Generation\nI’ll finish with the idea that first got me excited about embeddings.\nEveryone who tries out ChatGPT ends up asking the same question: how could I use a version of this to answer questions based on my own private notes, or the internal documents owned by my company?\nPeople assume that the answer is to train a custom model on top of that content, likely at great expense.\nIt turns out that’s not actually necessary. You can use an off the shelf Large Language Model model (a hosted one or one that runs locally) and a trick called Retrieval Augmented Generation, or RAG.\nThe key idea is this: a user asks a question. You search your private documents for content that appears relevant to the question, then paste excerpts of that content into the LLM (respecting its size limit, usually between 3,000 and 6,000 words) along with the original question.\nThe LLM can then answer the question based on the additional content you provided.\nThis cheap trick is astonishingly effective. It’s trivial to get a basic version of this working—the challenge is in getting it to work as well as possible given the infinite set of questions a user might ask.\nThe key problem in RAG is figuring out the best possible excerpts of content to include in the prompt to the LLM.\n“Vibes-based” semantic search powered by embedding is exactly the kind of thing you need to gather potentially relevant content to help answer a user’s question.\nI built a version of this against content from my blog, described in detail in\nEmbedding paragraphs from my blog with E5-large-v2\n.\nI used a model called\nE5-large-v2\nfor this. It’s a model trained with this exact use-case in mind.\nA challenge with finding content relevant to questions is that the user’s question—“What is shot-scraper?”—isn’t guaranteed to be considered semantically similar to content that answers that question. Questions and assertions have a different grammar from each other.\nE5-large-v2 addresses this by supporting two types of content. You can embed phrases (factual sentences) and queries (questions) in the same space, similar to how CLIP supports both images and text.\nI embedded 19,000 paragraphs of text from my blog as phrases, and now I can embed a question as a query and use that to find the paragraphs most likely to answer that question.\nThe result is RAG implemented as a one-liner Bash script:\nllm similar blog-paragraphs -c\n\"\nquery:\n$1\n\"\n\\\n|\njq\n'\n.content | sub(\"passage: \"; \"\")\n'\n-r \\\n|\nllm -m mlc-chat-Llama-2-7b-chat-hf-q4f16_1 \\\n\"\n$1\n\"\n-s\n'\nYou answer questions as a single paragraph\n'\nThis example uses Llama 2 Chat 7B running on my laptop (with the\nllm-mlc\nplugin), so I’m able to answer questions entirely offline using code running on my laptop.\nRunning this:\n./blog-answer.sh\n'\nWhat is shot-scraper?\n'\nOutputs this:\nShot-scraper is a Python utility that wraps Playwright, providing both a command line interface and a YAML-driven configuration flow for automating the process of taking screenshots of web pages and scraping data from them using JavaScript. It can be used to take one-off screenshots or take multiple screenshots in a repeatable way by defining them in a YAML file. Additionally, it can be used to execute JavaScript on a page and return the resulting value.\nThat’s a really good description of my\nshot-scraper\ntool. I checked and none of that output is an exact match to content I had previously published on my blog.\nQ&A\nMy talk ended with a Q&A session. Here are the summarized questions and answers.\nHow does LangChain fit into this?\nLangChain\nis a popular framework for implementing features on top of LLMs. It covers a\nlot\nof ground—my only problem with LangChain is that you have to invest a lot of work in understanding how it works and what it can do for you. Retrieval Augmented Generation is one of the key features of LangChain, so a lot of the things I’ve shown you today could be built on top of LangChain if you invest the effort.\nMy philosophy around this stuff differs from LangChain in that I’m focusing on building a suite of small tools that can work together, as opposed to a single framework that solves everything in one go.\nHave you tried distance functions other than cosine similarity?\nI have not. Cosine similarity is the default function that everyone else seems to be using and I’ve not spent any time yet exploring other options.\nI actually got ChatGPT to write all of my different versions of cosine similarity, across both Python and JavaScript!\nA fascinating thing about RAG is that it has so many different knobs that you can tweak. You can try different distance functions, different embedding models, different prompting strategies and different LLMs. There’s a lot of scope for experimentation here.\nWhat do you need to adjust if you have 1 billion objects?\nThe demos I’ve shown today have all been on the small side—up to around 20,000 embeddings. This is small enough that you can run brute force cosine similarity functions against everything and get back results in a reasonable amount of time.\nIf you’re dealing with more data there are a growing number of options that can help.\nLots of startups are launching new “vector databases”—which are effectively databases that are custom built to answer nearest-neighbour queries against vectors as quickly as possible.\nI’m not convinced you need an entirely new database for this: I’m more excited about adding custom indexes to existing databases. For example, SQLite has\nsqlite-vss\nand PostgreSQL has\npgvector\n.\nI’ve also done some successful experiments with Facebook’s\nFAISS\nlibrary, including building a Datasette plugin that uses it called\ndatasette-faiss\n.\nWhat improvements to embedding models are you excited to see?\nI’m really excited about multi-modal models. CLIP is a great example, but I’ve also been experimenting with Facebook’s\nImageBind\n, which “learns a joint embedding across six different modalities—images, text, audio, depth, thermal, and IMU data.” It looks like we can go a lot further than just images and text!\nI also like the trend of these models getting smaller. I demonstrated a new model, gte-tiny, earlier which is just 60MB. Being able to run these things on constrained devices, or in the browser, is really exciting to me.\nFurther reading\nIf you want to dive more into the low-level details of how embeddings work, I suggest the following:\nWhat are embeddings?\nby Vicki Boykis\nText Embeddings Visually Explained\nby Meor Amer for Cohere\nThe Tensorflow Embedding Projector\n—an interactive tool for exploring embedding spaces\nLearn to Love Working with Vector Embeddings\nis a collection of tutorials from vector database vendor Pinecone\nPosted\n23rd October 2023\nat 1:36 pm · Follow me on\nMastodon\n,\nBluesky\n,\nTwitter\nor\nsubscribe to my newsletter\nMore recent articles\nHello Python 3.15, goodbye 3.10, and now I can use 3.11\n- 12th October 2026\nA new feature for my blog, built using my voice\n- 9th October 2026\nClaude Haiku 5.5\n- 7th October 2026\nWe're going to need default hard budget caps on pretty much everything\n- 3rd October 2026\nThis is\nEmbeddings: What they are and why they matter\nby Simon Willison, posted on\n23rd October 2023\n.\nmy-talks\n93\nai\n2,279\ngenerative-ai\n2,021\nvector-search\n21\nembeddings\n62\nllm\n639\nannotated-talks\n33\nrag\n35\nclip\n8\nNext:\nExecute Jina embeddings with a CLI using llm-embed-jina\nPrevious:\nWeeknotes: PyBay, AI Engineer Summit, Datasette metadata and JavaScript plugins\nMonthly briefing\nSponsor me for\n$10/month\nand get a curated email digest of the month's most important LLM developments.\nPay me to send you less!\nSponsor & subscribe\nDisclosures\nColophon\n©\n2002\n2003\n2004\n2005\n2006\n2007\n2008\n2009\n2010\n2011\n2012\n2013\n2014\n2015\n2016\n2017\n2018\n2019\n2020\n2021\n2022\n2023\n2024\n2025\n2026",
  "metadata": {
    "title": "Embeddings: What they are and why they matter",
    "description": null,
    "keywords": null,
    "author": "Simon Willison",
    "og:site_name": "Simon Willison’s Weblog",
    "og:url": "https://simonwillison.net/2023/Oct/23/embeddings/",
    "og:title": "Embeddings: What they are and why they matter",
    "og:image": "https://static.simonwillison.net/static/2023/embeddings/embeddings.003.jpeg",
    "og:type": "article",
    "og:description": "Embeddings are a really neat trick that often come wrapped in a pile of intimidating jargon. If you can make it through that jargon, they unlock powerful and exciting techniques …",
    "og:updated_time": "1698068181",
    "twitter:card": "summary",
    "twitter:image": "https://static.simonwillison.net/static/2023/embeddings/embeddings.003.jpeg",
    "twitter:creator": "@simonw"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/a310d3f7beb9/code/body-000.txt",
      "chars": 226,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-001.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-002.txt",
      "chars": 63,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-003.txt",
      "chars": 228,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-004.txt",
      "chars": 15,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-005.txt",
      "chars": 16,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-006.txt",
      "chars": 37,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-007.txt",
      "chars": 53,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-008.txt",
      "chars": 51,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-009.txt",
      "chars": 72,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-010.txt",
      "chars": 112,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-011.txt",
      "chars": 53,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-012.txt",
      "chars": 270,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-013.txt",
      "chars": 213,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-014.txt",
      "chars": 311,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-015.txt",
      "chars": 174,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-016.txt",
      "chars": 50,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-017.txt",
      "chars": 893,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-018.txt",
      "chars": 35,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-019.txt",
      "chars": 194,
      "language": "text"
    },
    {
      "file": "attachments/a310d3f7beb9/code/body-020.txt",
      "chars": 40,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/embeddings.002.jpeg",
      "bytes": 313959,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/000.jpeg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/embeddings.002.jpeg",
      "alt": "On the left, a blog entry titled Storing and serving related documents with oepnai-to-sqlite and embeddings. On the right, a JSON array of floating point numbers, with the caption Fixed zise: 300, 1000, 1536...",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/embeddings.003.jpeg",
      "bytes": 166055,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/001.jpeg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/embeddings.003.jpeg",
      "alt": "a 3D chart showing a location in many-multi-dimensional space. 400 randomly placed red dots are scattered around the chart.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils-embeddings.png",
      "bytes": 115606,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/002.png",
      "src": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils-embeddings.png",
      "alt": "Screenshot of the embeddings table in Datasette, it has 472, rows each of which consists of a text ID and a binary 6.144 bytes embedding",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils.png",
      "bytes": 140654,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/003.png",
      "src": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils.png",
      "alt": "Running that SQL query in Datasette returns text IDs and long hexadecimal strings for each embedding",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils.1.png",
      "bytes": 153885,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/004.png",
      "src": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils.1.png",
      "alt": "Now the SQL query returns a JSON array of floating point numbers for each ID",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils-similarities.png",
      "bytes": 133878,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/005.png",
      "src": "https://static.simonwillison.net/static/2023/embeddings/til-simonwillison-net-tils-similarities.png",
      "alt": "The similarities table has 4,922 rows each with an id, other_id and score column.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/embeddings.006.jpeg",
      "bytes": 134440,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/006.jpeg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/embeddings.006.jpeg",
      "alt": "Screenshot of curl against api.openai.com/v1/embeddings sending a Bearer token header and a JSON body specifying input text and the text-embedding-ada-002 model. The API responds with a JSON list of numbers.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/embeddings.007.jpeg",
      "bytes": 183841,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/007.jpeg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/embeddings.007.jpeg",
      "alt": "Screenshot of the OpenAI First-generation text embedding models list, showing the shutdown date of 4th April 2024 for 7 legacy models.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/word2vec.jpg",
      "bytes": 106687,
      "reused": true,
      "local_path": "attachments/a310d3f7beb9/images/008.jpg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/word2vec.jpg",
      "alt": "Screenshot of the Word to Vec JS Demo showing the results for france and the algebra results for germany + paris - france",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/embeddings.014.jpeg",
      "bytes": 140230,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-d7da62f204558073f66b.jpeg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/embeddings.014.jpeg",
      "alt": "Slide showing the commands listed below",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/embeddings.015.jpeg",
      "bytes": 176628,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-036d418897009af23ee2.jpeg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/embeddings.015.jpeg",
      "alt": "A terminal running llm similar and piping the results through jq",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/list-plugins.jpg",
      "bytes": 53759,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-4a98f88ab8f08030432b.jpg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/list-plugins.jpg",
      "alt": "Running that query in Datasette with an input of list plugins returns the plugins() function from the cli.py file on line 175",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/beach.jpg",
      "bytes": 148321,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-798510bf9bea98419d95.jpg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/beach.jpg",
      "alt": "A bright blue sky over a beach, with sandy cliffs and the Pacific ocean in the frame",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/clip.gif",
      "bytes": 64365,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-af7f8fabb4d0f03cd9dd.gif",
      "src": "https://static.simonwillison.net/static/2023/embeddings/clip.gif",
      "alt": "Animation showing different similarity scores for different text strings",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/faucet-finder.jpg",
      "bytes": 74676,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-4237fc7a108d977a4a04.jpg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/faucet-finder.jpg",
      "alt": "The Faucet Finder homepage - six faucets, each with a Find Similar button.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/clip-gold-purple.jpg",
      "bytes": 64407,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-deadf7e55ec30e168242.jpg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/clip-gold-purple.jpg",
      "alt": "Observable notebook: Search for Faucets with CLIP. The search term gold purple produces 8 alarmingly tasteless faucets in those combined colors.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/cpa.gif",
      "bytes": 173657,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-00a560488c35a5faa18c.gif",
      "src": "https://static.simonwillison.net/static/2023/embeddings/cpa.gif",
      "alt": "Animated screenshot of a cloud of points in 2D space. At one side hovering over them shows things like The War of 1812 and The Battle of Trafalgar - at the other side we get Quantum Gravity and Higgs Boson and Carbon",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://static.simonwillison.net/static/2023/embeddings/amelia.jpg",
      "bytes": 135550,
      "reused": false,
      "local_path": "attachments/a310d3f7beb9/images/body-ddf5c567ea49f032c5b9.jpg",
      "src": "https://static.simonwillison.net/static/2023/embeddings/amelia.jpg",
      "alt": "A document. Different sentences are displayed in different shades of green and purple, with a key on the right hand side showing that green means concreete and purple means abstract, with a gradient between them.",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=ArnMdc-ICCM",
      "status": "ok",
      "local_path": "attachments/a310d3f7beb9/videos/ArnMdc-ICCM.mp4",
      "bytes": 132760050,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "2317.901000"
    }
  ],
  "links": [
    {
      "href": "https://simonwillison.net/",
      "label": "Simon Willison’s Weblog",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/newsletters/",
      "label": "Newsletter",
      "type": "a"
    },
    {
      "href": "https://fandf.co/4hJeKAm",
      "label": "Upgrade to Pomerium's agentic access management trusted by NVIDIA and OpenAI",
      "type": "a"
    },
    {
      "href": "https://pybay.com/",
      "label": "PyBay 2023",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Oct/23/embeddings/",
      "label": "The 38 minute video version",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=ArnMdc-ICCM&t=50s",
      "label": "a video",
      "type": "a"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/ArnMdc-ICCM?start=50",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://podcast.adobe.com/enhance",
      "label": "Enhance Speech tool",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/llms/openai-embeddings-related-content",
      "label": "a blog entry",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/",
      "label": "my TIL blog",
      "type": "a"
    },
    {
      "href": "https://platform.openai.com/docs/guides/embeddings",
      "label": "via their API",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/sqlite/sqlite-tg",
      "label": "the bottom of this page",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/sqlite/geopoly",
      "label": "Geopoly in SQLite",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/spatialite/viewing-geopackage-data-with-spatialite-and-datasette",
      "label": "Viewing GeoPackage data with SpatiaLite and Datasette",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/gis/gdal-sql",
      "label": "Using SQL with GDAL",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/spatialite/knn",
      "label": "KNN queries with SpatiaLite",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/spatialite/gunion-to-combine-geometries",
      "label": "GUnion to combine geometries in SpatiaLite",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/llm/blob/bf229945fe57036fa75e8105e59d9e506a720156/llm/__init__.py",
      "label": "the Python function",
      "type": "a"
    },
    {
      "href": "https://datasette.io/",
      "label": "Datasette",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2021/Jul/28/baked-data/",
      "label": "the Baked Data architectural pattern",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/tils/embeddings",
      "label": "tils/embeddings",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/tils?sql=select+id%2C+hex%28embedding%29+from+embeddings",
      "label": "run this SQL query",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/tils?sql=select+id%2C+llm_embed_decode%28embedding%29+from+embeddings+limit+10",
      "label": "Try that here",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/datasette-llm-embed/blob/ebded67fa9ee19db2c4b1badb1895cef0d58ac4a/datasette_llm_embed.py",
      "label": "is defined here",
      "type": "a"
    },
    {
      "href": "https://datasette.io/plugins/datasette-llm-embed",
      "label": "datasette-llm-embed",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/tils?sql=select%0D%0A++id%2C%0D%0A++llm_embed_cosine%28%0D%0A++++embedding%2C%0D%0A++++%28%0D%0A++++++select%0D%0A++++++++embedding%0D%0A++++++from%0D%0A++++++++embeddings%0D%0A++++++where%0D%0A++++++++id+%3D+%27sqlite_sqlite-tg.md%27%0D%0A++++%29%0D%0A++%29+as+score%0D%0Afrom%0D%0A++embeddings%0D%0Aorder+by%0D%0A++score+desc%0D%0Alimit+5",
      "label": "Executing that query",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/tils/similarities",
      "label": "tils/similarities",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/til/blob/a244856c72000760c2939550ae2a78266dee4f07/plugins/template_vars.py",
      "label": "look up related documents from that table",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/til/blob/a244856c72000760c2939550ae2a78266dee4f07/templates/pages/%7Btopic%7D/%7Bslug%7D.html",
      "label": "called it from the template",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/til/blob/a244856c72000760c2939550ae2a78266dee4f07/.github/workflows/build.yml",
      "label": "as part of the build script",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/gpt-4-api-general-availability",
      "label": "shut down some of their older embeddings models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1301.3781",
      "label": "Efficient Estimation of Word Representations in Vector Space",
      "type": "a"
    },
    {
      "href": "https://turbomaze.github.io/word2vecjson/",
      "label": "turbomaze.github.io/word2vecjson",
      "type": "a"
    },
    {
      "href": "https://anthony.ai/",
      "label": "Anthony Liu",
      "type": "a"
    },
    {
      "href": "https://turbomaze.github.io/word2vecjson/data/wordvecs10000.js",
      "label": "this JavaScript file",
      "type": "a"
    },
    {
      "href": "https://llm.datasette.io/",
      "label": "LLM",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/May/18/cli-tools-for-llms/",
      "label": "llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Jul/12/llm/",
      "label": "The LLM CLI tool now supports self-hosted language models via plugins",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Sep/4/llm-embeddings/",
      "label": "LLM now provides tools for working with embeddings",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Sep/12/llm-clip-and-chat/",
      "label": "Build an image search engine with llm-clip, chat with models with llm chat",
      "type": "a"
    },
    {
      "href": "https://llm.datasette.io/en/stable/python-api.html",
      "label": "a Python library",
      "type": "a"
    },
    {
      "href": "https://llm.datasette.io/en/stable/setup.html",
      "label": "an API key",
      "type": "a"
    },
    {
      "href": "https://llm.datasette.io/en/stable/plugins/index.html",
      "label": "installing plugins",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2",
      "label": "all-MiniLM-L6-v2",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/llm-sentence-transformers",
      "label": "llm-sentence-transformers",
      "type": "a"
    },
    {
      "href": "https://www.sbert.net/",
      "label": "SentenceTransformers",
      "type": "a"
    },
    {
      "href": "https://llm.datasette.io/en/stable/embeddings/cli.html",
      "label": "embed-multi command",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/symbex",
      "label": "Symbex",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Jun/18/symbex/",
      "label": "originally built it",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/datasette",
      "label": "Datasette",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/TaylorAI/gte-tiny",
      "label": "gte-tiny",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/clip/",
      "label": "CLIP",
      "type": "a"
    },
    {
      "href": "https://observablehq.com/@simonw/openai-clip-in-a-browser",
      "label": "an interactive demo",
      "type": "a"
    },
    {
      "href": "https://www.dbreunig.com/",
      "label": "Drew Breunig",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/llm-clip",
      "label": "llm-clip",
      "type": "a"
    },
    {
      "href": "https://faucet-finder.fly.dev/",
      "label": "Faucet Finder",
      "type": "a"
    },
    {
      "href": "https://www.dbreunig.com/2023/09/26/faucet-finder.html",
      "label": "Finding Bathroom Faucets with Embeddings",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/fly/clip-on-fly",
      "label": "how to deploy a server-side CLIP model",
      "type": "a"
    },
    {
      "href": "https://fly.io/",
      "label": "Fly.io",
      "type": "a"
    },
    {
      "href": "https://faucet-finder.fly.dev/faucets/embeddings",
      "label": "includes this table",
      "type": "a"
    },
    {
      "href": "https://clip-datasette-on-fly.fly.dev/_memory?sql=select+hex(llm_embed(%27clip%27%2C+%3Aq))+as+x&q=purple",
      "label": "this API",
      "type": "a"
    },
    {
      "href": "https://observablehq.com/@simonw/search-for-faucets-with-clip-api",
      "label": "observablehq.com/@simonw/search-for-faucets-with-clip-api",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/llm-cluster",
      "label": "llm-cluster",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/clustering.html",
      "label": "sklearn.cluster",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/paginate-json",
      "label": "paginate-json",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/llm-cluster/blob/main/README.md",
      "label": "the README",
      "type": "a"
    },
    {
      "href": "https://interconnected.org/",
      "label": "Matt Webb",
      "type": "a"
    },
    {
      "href": "https://interconnected.org/more/2023/02/in_our_time-PCA-plot.html",
      "label": "an interactive 2D visualization",
      "type": "a"
    },
    {
      "href": "https://interconnected.org/home/2023/02/07/braggoscope",
      "label": "Browse the BBC In Our Time archive by Dewey decimal code",
      "type": "a"
    },
    {
      "href": "https://wattenberger.com/thoughts/yay-embeddings-math",
      "label": "Getting creative with embeddings",
      "type": "a"
    },
    {
      "href": "https://til.simonwillison.net/llms/embed-paragraphs",
      "label": "Embedding paragraphs from my blog with E5-large-v2",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/intfloat/e5-large-v2",
      "label": "E5-large-v2",
      "type": "a"
    },
    {
      "href": "https://github.com/simonw/llm-mlc",
      "label": "llm-mlc",
      "type": "a"
    },
    {
      "href": "https://shot-scraper.datasette.io/",
      "label": "shot-scraper",
      "type": "a"
    },
    {
      "href": "https://www.langchain.com/",
      "label": "LangChain",
      "type": "a"
    },
    {
      "href": "https://github.com/asg017/sqlite-vss",
      "label": "sqlite-vss",
      "type": "a"
    },
    {
      "href": "https://github.com/pgvector/pgvector",
      "label": "pgvector",
      "type": "a"
    },
    {
      "href": "https://github.com/facebookresearch/faiss",
      "label": "FAISS",
      "type": "a"
    },
    {
      "href": "https://datasette.io/plugins/datasette-faiss",
      "label": "datasette-faiss",
      "type": "a"
    },
    {
      "href": "https://github.com/facebookresearch/ImageBind",
      "label": "ImageBind",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/what_are_embeddings/",
      "label": "What are embeddings?",
      "type": "a"
    },
    {
      "href": "https://txt.cohere.com/text-embeddings/",
      "label": "Text Embeddings Visually Explained",
      "type": "a"
    },
    {
      "href": "https://projector.tensorflow.org/",
      "label": "The Tensorflow Embedding Projector",
      "type": "a"
    },
    {
      "href": "https://www.pinecone.io/learn/",
      "label": "Learn to Love Working with Vector Embeddings",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Oct/23/",
      "label": "23rd October 2023",
      "type": "a"
    },
    {
      "href": "https://fedi.simonwillison.net/@simon",
      "label": "Mastodon",
      "type": "a"
    },
    {
      "href": "https://bsky.app/profile/simonwillison.net",
      "label": "Bluesky",
      "type": "a"
    },
    {
      "href": "https://twitter.com/simonw",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/about/",
      "label": "subscribe to my newsletter",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2026/Oct/12/python-3-15/",
      "label": "Hello Python 3.15, goodbye 3.10, and now I can use 3.11",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2026/Oct/9/built-using-my-voice/",
      "label": "A new feature for my blog, built using my voice",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/",
      "label": "Claude Haiku 5.5",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2026/Oct/3/default-hard-budget-caps/",
      "label": "We're going to need default hard budget caps on pretty much everything",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/my-talks/",
      "label": "my-talks 93",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/ai/",
      "label": "ai 2,279",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/generative-ai/",
      "label": "generative-ai 2,021",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/vector-search/",
      "label": "vector-search 21",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/embeddings/",
      "label": "embeddings 62",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/llm/",
      "label": "llm 639",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/annotated-talks/",
      "label": "annotated-talks 33",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/rag/",
      "label": "rag 35",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/tags/clip/",
      "label": "clip 8",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Oct/26/llm-embed-jina/",
      "label": "Execute Jina embeddings with a CLI using llm-embed-jina",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/Oct/22/weeknotes/",
      "label": "Weeknotes: PyBay, AI Engineer Summit, Datasette metadata and JavaScript plugins",
      "type": "a"
    },
    {
      "href": "https://github.com/sponsors/simonw/",
      "label": "Sponsor & subscribe",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2002/",
      "label": "2002",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2003/",
      "label": "2003",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2004/",
      "label": "2004",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2005/",
      "label": "2005",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2006/",
      "label": "2006",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2007/",
      "label": "2007",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2008/",
      "label": "2008",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2009/",
      "label": "2009",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2010/",
      "label": "2010",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2011/",
      "label": "2011",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2012/",
      "label": "2012",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2013/",
      "label": "2013",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2014/",
      "label": "2014",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2015/",
      "label": "2015",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2016/",
      "label": "2016",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2017/",
      "label": "2017",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2018/",
      "label": "2018",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2019/",
      "label": "2019",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2020/",
      "label": "2020",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2021/",
      "label": "2021",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2022/",
      "label": "2022",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2023/",
      "label": "2023",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2024/",
      "label": "2024",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2025/",
      "label": "2025",
      "type": "a"
    },
    {
      "href": "https://simonwillison.net/2026/",
      "label": "2026",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:29Z",
  "source_url": "https://simonwillison.net/2023/Oct/23/embeddings/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1301.3781",
      "bytes": 228716,
      "reused": true,
      "local_path": "papers/repair/56b49aeaee8219bc1cd4.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/a310d3f7beb9/videos/ArnMdc-ICCM.mp4",
      "bytes": 132760050,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "2317.901000",
      "status": "ok"
    }
  ],
  "body_chars": 31426,
  "expected_body_images": 18,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/a310d3f7beb9/pages/source.html",
  "local_html": "attachments/a310d3f7beb9/pages/article.html",
  "body_base_path": "attachments/a310d3f7beb9/pages/",
  "body_markdown_path": "attachments/a310d3f7beb9/pages/article.md"
}