{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Vicki Boykis",
  "title_listed": "How big are our embeddings now, and why?",
  "keywords": "embedding,维度,架构",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://vickiboykis.com/2025/09/01/how-big-are-our-embeddings-now-and-why/",
  "publish_date": "2025-09",
  "corpus_stage": "数据处理",
  "notes": "embedding 规模分析",
  "id": "1cd757777bc3",
  "collected_at": "2026-10-02T15:22:57",
  "title": "How big are our embeddings now and why? | ✰Vicki Boykis✰",
  "body_markdown": "How big are our embeddings now and why?\nSep 1 2025\nA few years ago, I wrote\na paper on embeddings\n. At the time, I wrote that 200-300 dimension embeddings were fairly common in industry, and that adding more dimensions during training would create diminishing returns for the effectiveness of your downstream tasks (classification, recommendation, semantic search, topic modeling, etc.)\nI wrote the paper to be resilient to changes in the industry since it focuses on fundamentals and historical context rather than libraries or bleeding edge architectures, but this assumption about embedding size is now out of date and worth revisiting in the context of growing embedding dimensionality and embedding access patterns.\nAs a quick review, embeddings are compressed numerical representations of a variety of features (text, images, audio) that we can use for machine learning tasks like search, recommendations, RAG, and classification. The size of the embedding is how many features our item has.\nFor example, let’s say we have two butterflies. We can compare them among many dimensions, including wingspan, wing color, number of antennae. Let’s say we have a 3-dimensional feature for a butterfly, it might look like this.\nDoing a visual vibe scan, we can eyeball the data and see that\nbutterfly_1\nand\nbutterfly_3\nare more similar to each other than to\nbutterfly_2\nbecause their features are closer together.\nBut butterflies are 3-dimensional animals and some of these features are numerical. When we talk about embeddings in industry these days, we generally mean trying to understand the properties of text, so how does this concept work with words? We can’t directly compare words like “bird” and “butterfly” or “fly” in a given text but we can compare their numerical representations if we map them into the same shared space. We can see that “bird” and “flying” are more similar to each other than to “dog” through their numerical representations.\nIntuitively, we know this is true, but how do we artificially create these relationships?\nThere are different ways of creating text embeddings from a given word, all of which rely on analyzing that word in relation to the words around it in a given corpus of data. We can use traditional count-based approaches like TF-IDF based on term frequency in documents, or statistical approaches like PCA or LSA.\nWith the advent of deep learning models, we started learning representations generated from models like Word2Vec that maximized the probability that a left-out word would be next to other given words in the training dataset.\nWhen we learn the embedding representation of a given word using probabilistic models, we are comparing how similar these words are to other words. Each feature is not an explicit feature like “wing color”, but rather a vibe-based latent representation in the latent space that doesn’t have a clear explanation.\nFor example, one dimension might be “this word is an action word” or maybe “this word is related to other words about food”, but we generally don’t know exactly what the model thinks each feature represents. In fact, this is a\nfascinating area of study\nwe are just starting to understand how these latent representations work through ideas like\ncontrol vectors\n, a concept that Anthropic explored in the famous Golden Gate Claude paper.\nWhen we train a model, embedding size is initialized as a hyperparameter before model training, and we iterate on the size depending on our downstream evaluations after training. Picking the right hyperparameter is (\nalchemy\n) a combination of art and science and\ndepends on optimizing\ntraining throughput, final embedding storage size, and the performance of the embedding on your downstream task both qualitatively and wrt to latency of performing search on embeddings of different sizes.\nSince previous generations of models were smaller and trained in-house, hyperparameters were usually not published by companies, and as such, there was no standard agreement on embedding size. We generally, as an industry, understood\nthat somewhere around 300 dimensions for a given embedding model might be enough to compress all the nuance of a given textual dataset\n. 300 was the number of dimensions typically used by earlier models like Word2Vec and\nGloVE\n.\nAfter the publication of the attention paper\nBERT was released\nin 2018. This model’s architecture\nintroduced embeddings of 768 dimensions.\nAlthough previous\nRNN\nand LSTM models had been trained on GPUs, BERT was one of the first larger embedding models to be trained on GPUs (\nand TPUs\n), which meant that GPU optimization now became increasingly important.\nThe key behind\ntraining Transformer models efficiently\nis the ability to\nefficiently move data onto GPU\nand parallelize their matrix multiplication operations between several pipelines, aka attention heads, where each attention head can focus on understanding and defining a different part of the embedding feature space. As such, the embedding needs to be able to be partitioned evenly between the number of attention heads.\nBERT has 12 attention heads, so 768 dimensions was selected from a combination of trial and error and efficiently parallelizing computation to “attend” to different parts of the feature space, meaning that, in model training, each head operates on a 64-dimensional subspace of the original 768-dimensional input embedding. This itself comes from the Transformer paper, where each sub-embedding size per head\nis commonly chosen as 64\n.\nAs a result, many BERT-style models, and related model families, used 768 as a baseline for embedding dimensions. Although it had a much larger training dataset,\nGPT-2 also\nimplemented 768. And, although CLIP uses an embedding size of\n768\nas derived from the\nVision Transformer\narchitecture that CLIP uses for its image encoder, and for consistency, the text encoder also uses this dimension size[^1].\nEven though training cycles for BERT were fairly small (4 days for the original BERT model) compared to the months-long pretraining processes that LLMs require these days, it was still hard computationally to infer these embedding sizes, even with GPU optimizations.\nFor BERT, for example,\nFinding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours).\nSo in 2019,\nUKP created\nand optimized a model, SBERT, focusing on\nsentence-level representation\ns, which unlocked faster inference, and\nMinilm\nbecame the standard baseline model for document-level chunk embeddings at 384 dimensions and still performs extremely well for a number of tasks.\nWhat else changed was the rise of the open-source HuggingFace platform where model artifacts could be shared. Previously, they were either hosted in undiscoverable repos in GitHub or locked away in scientific repositories missing metadata. The rise of HuggingFace as a platform and the\ntransformers\nlibrary as a centralized API into training and inference for PyTorch-based models unlocked a level of standardization: now, many more people could simply download and replicate the existing models instead of rebuilding arcane research code from scratch. This led to much more standard embedding and architecture sizes used both in academia and industry.\nAlthough 768 held for a fairly long time, with upwards pressure from market competition on model sizes as the result of the release of GPT-2 (the backbone of ChatGPT), we started seeing standard embedding sizes increase.\nPart of this was the fact that companies no longer had to infer their own embeddings. Previously, embeddings had been custom-learned in labs or in\ncompanies whose core competency was processing large amounts of information\nthat could be culled in retrieval through search or recommendation.\nBut now, with the advent of ChatGPT and API-based model availability, embeddings became a commodity available with a GET request, and the most popular embeddings\nbecame OpenAI’s\n, which used 1536 dimensions, in line with GPT-3, which used much more training data than any previous model. (\n570GB versus GPT-2 which was 40GB, and 96 attention heads.)\nIt used to be the case that people only learned or fine-tune their own embeddings, but now all of the major AI providers have their own hosted sets of embeddings. Particularly common ones are OpenAI, with Cohere and Nomic close behind. Google also\nrecently released embeddings for Gemini\n, with both API and local versions being available.\nIn addition to standardization via HuggingFace and APIs, MTEB,\nwhich benchmarks embeddings publicly\n, has grown as a resource where you can compare embedding models.\nIf we look at MTEB,\nthe embedding benchmark today\n, we can see embedding sizes anywhere from our classic 768 to 4096 and beyond (note all of them are neatly divisible by 2, in line with architectural constraints)\nFinally, another consideration has been the change in the landscape with vector databases, which used to be huge, are now becoming more commoditized features in software and platforms like\nPostgres\n,\nS3\n, and\nElasticsearch\n, leading to less out of the box necessity in vector storage and performance tuning.\nWith the constant upward pressure on embedding sizes not limited by having to train models in-house, it’s not clear where we’ll slow down: Qwen-3, along with many others is already at 4096. It does appear that we are beginning to trim down on growth. OpenAI implemented a concept called\nmatryoshka representation learning\nthat trains embeddings with the “most important” concepts first, and additional embeddings adding incremental gains, meaning that an embedding learned in 1024 dimensions might be just as useful in 64, as long as the first 64\ndimensions compress most of the information efficiently\n, and also making sure we re-normalize them. There are also research that indicates that not all embeddings are necessary in retrieval + search tasks, and that we can\nin fact, in some cases, truncate 50% of them anyway.\nIt’s been fascinating to watch the rise of dimensionality and the constant struggle in tradeoffs between creating ever-larger models and then the need for those embedding sizes to perform at inference and storage time, aka continuously coming up against the age-old machine learning tradeoff of recall versus precision, and the engineering tradeoff of hardware limitations versus software versus business considerations.\nAs these architectures have matured and internal models have become inference points of public-facing paid APIs, embeddings have gone from a mysterious byproduct of internal machine learning systems in companies with lots of data to a commodity used across many AI-powered applications across numerous application stacks.\nAll in all, it’s been so much fun watching this space evolve, even if it means it looks like I’ll be having to update my paper over and over again.\n#embeddings\n#machine learning\n#deep learning",
  "metadata": {
    "title": "How big are our embeddings now and why? | ✰Vicki Boykis✰",
    "description": "Embedding sizes and architectures have changed remarkably over the past 5 years",
    "keywords": "embeddings,machine learning,deep learning,",
    "author": null,
    "og:type": "article",
    "og:url": "https://vickiboykis.com/2025/09/01/how-big-are-our-embeddings-now-and-why/",
    "og:title": "How big are our embeddings now and why?",
    "og:description": "Embedding sizes and architectures have changed remarkably over the past 5 years",
    "og:image": "https://vickiboykis.com/images/embeddings-sizes/img_1.png",
    "twitter:card": "summary",
    "twitter:url": "https://vickiboykis.com/2025/09/01/how-big-are-our-embeddings-now-and-why/",
    "twitter:title": "How big are our embeddings now and why?",
    "twitter:description": "Embedding sizes and architectures have changed remarkably over the past 5 years",
    "twitter:image": "https://vickiboykis.com/images/embeddings-sizes/img_1.png",
    "twitter:creator": "@vboykis",
    "twitter:site": "@vboykis",
    "article:published_time": "2025-09-01T00:00:00+00:00",
    "article:modified_time": "2025-09-01T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/1cd757777bc3/code/body-000.txt",
      "chars": 128,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://vickiboykis.com/images/embeddings-sizes/img_1.png",
      "bytes": 131036,
      "reused": false,
      "local_path": "attachments/1cd757777bc3/images/body-9b5a5ad8551819f10931.png",
      "src": "https://vickiboykis.com/images/embeddings-sizes/img_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://vickiboykis.com/images/embeddings-sizes/img_2.png",
      "bytes": 89625,
      "reused": false,
      "local_path": "attachments/1cd757777bc3/images/body-ddc222fa2a93ed7c58e9.png",
      "src": "https://vickiboykis.com/images/embeddings-sizes/img_2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://vickiboykis.com/images/embeddings-sizes/img_3.png",
      "bytes": 78387,
      "reused": false,
      "local_path": "attachments/1cd757777bc3/images/body-d1a4fd4d01b01c8d9b80.png",
      "src": "https://vickiboykis.com/images/embeddings-sizes/img_3.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://vickiboykis.com/images/embeddings-sizes/img_4.png",
      "bytes": 1085250,
      "reused": false,
      "local_path": "attachments/1cd757777bc3/images/body-597316b094d8ba6eb386.png",
      "src": "https://vickiboykis.com/images/embeddings-sizes/img_4.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://vickiboykis.com/images/embeddings-sizes/img_5.png",
      "bytes": 103499,
      "reused": false,
      "local_path": "attachments/1cd757777bc3/images/body-baa25ee330a6cd943a0b.png",
      "src": "https://vickiboykis.com/images/embeddings-sizes/img_5.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://vickiboykis.com/images/embeddings-sizes/img_6.png",
      "bytes": 334901,
      "reused": false,
      "local_path": "attachments/1cd757777bc3/images/body-9b326b66b8c1b64e196b.png",
      "src": "https://vickiboykis.com/images/embeddings-sizes/img_6.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://vickiboykis.com/what_are_embeddings/",
      "label": "a paper on embeddings",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/news/mapping-mind-language-model",
      "label": "fascinating area of study",
      "type": "a"
    },
    {
      "href": "https://vgel.me/posts/representation-engineering/",
      "label": "control vectors",
      "type": "a"
    },
    {
      "href": "https://archives.argmin.net/2017/12/05/kitchen-sinks/",
      "label": "alchemy",
      "type": "a"
    },
    {
      "href": "https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/genperf04-bp02.html",
      "label": "depends on optimizing",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/I17-2006/",
      "label": "that somewhere around 300 dimensions for a given embedding model might be enough to compress all the nuance of a given t",
      "type": "a"
    },
    {
      "href": "https://nlp.stanford.edu/projects/glove/",
      "label": "GloVE",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1810.04805",
      "label": "BERT was released",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/google-bert/bert-base-uncased",
      "label": "introduced embeddings of 768 dimensions.",
      "type": "a"
    },
    {
      "href": "https://svail.github.io/persistent_rnns/",
      "label": "RNN",
      "type": "a"
    },
    {
      "href": "https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/",
      "label": "and TPUs",
      "type": "a"
    },
    {
      "href": "https://jiegroup-genai.readthedocs-hosted.com/en/latest/resource/",
      "label": "training Transformer models efficiently",
      "type": "a"
    },
    {
      "href": "https://horace.io/brrr_intro.html",
      "label": "efficiently move data onto GPU",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1706.03762",
      "label": "is commonly chosen as 64",
      "type": "a"
    },
    {
      "href": "https://www.alignmentforum.org/posts/BMghmAxYxeSdAteDc/an-exploration-of-gpt-2-s-embedding-weights",
      "label": "GPT-2 also",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.11929",
      "label": "Vision Transformer",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.10084",
      "label": "For BERT, for example,",
      "type": "a"
    },
    {
      "href": "https://github.com/UKPLab/sentence-transformers",
      "label": "UKP created",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2",
      "label": "Minilm",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2007.03634",
      "label": "companies whose core competency was processing large amounts of information",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/new-and-improved-embedding-model/",
      "label": "became OpenAI’s",
      "type": "a"
    },
    {
      "href": "https://lambda.ai/blog/demystifying-gpt-3",
      "label": "570GB versus GPT-2 which was 40GB, and 96 attention heads.)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2503.07891",
      "label": "recently released embeddings for Gemini",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/mteb",
      "label": "which benchmarks embeddings publicly",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/spaces/mteb/leaderboard",
      "label": "the embedding benchmark today",
      "type": "a"
    },
    {
      "href": "https://github.com/pgvector/pgvector",
      "label": "Postgres",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/s3/features/vectors/",
      "label": "S3",
      "type": "a"
    },
    {
      "href": "https://www.elastic.co/docs/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example",
      "label": "Elasticsearch",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.13147",
      "label": "matryoshka representation learning",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/matryoshka",
      "label": "dimensions compress most of the information efficiently",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2508.17744",
      "label": "in fact, in some cases, truncate 50% of them anyway.",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/embeddings/",
      "label": "#embeddings",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/machine-learning/",
      "label": "#machine learning",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/deep-learning/",
      "label": "#deep learning",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:15Z",
  "source_url": "https://vickiboykis.com/2025/09/01/how-big-are-our-embeddings-now-and-why/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1810.04805",
      "bytes": 775166,
      "reused": true,
      "local_path": "papers/repair/33769e0bafc9f0c73baa.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.11929",
      "bytes": 3743814,
      "reused": true,
      "local_path": "papers/repair/81153ab91885fd77745c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.10084",
      "bytes": 548961,
      "reused": true,
      "local_path": "papers/repair/20151599fbbbabacaa9e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2007.03634",
      "bytes": 7936638,
      "reused": false,
      "local_path": "papers/repair/53a7ff1ae2f488498572.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2503.07891",
      "bytes": 836032,
      "reused": false,
      "local_path": "papers/repair/1cbc17ac09bb6e34eecb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.13147",
      "bytes": 7006007,
      "reused": false,
      "local_path": "papers/repair/152c0c8b980a7e7b0897.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2508.17744",
      "bytes": 1125166,
      "reused": false,
      "local_path": "papers/repair/bbbd1454bda9f70e8801.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 10915,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/1cd757777bc3/pages/source.html",
  "local_html": "attachments/1cd757777bc3/pages/article.html",
  "body_base_path": "attachments/1cd757777bc3/pages/",
  "body_markdown_path": "attachments/1cd757777bc3/pages/article.md"
}