{ "source_file": "0930", "source_platform": "个人博客", "author": "Jay Alammar", "title_listed": "The Illustrated Word2vec", "keywords": "word2vec,词嵌入,可视化", "domain": "文本处理", "content_type": "图解教程", "is_paid_flag": "否", "url": "https://jalammar.github.io/illustrated-word2vec/", "publish_date": "2019-03", "corpus_stage": "预训练/SFT", "notes": "word2vec图解", "id": "37477c68e81f", "collected_at": "2026-10-02T14:20:08", "title": "The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.", "body_markdown": "The Illustrated Word2vec\nDiscussions:\nHacker News (347 points, 37 comments)\n,\nReddit r/MachineLearning (151 points, 19 comments)\nTranslations:\nChinese (Simplified)\n,\nFrench\n,\nKorean\n,\nPortuguese\n,\nRussian\n“\nThere is in all things a pattern that is part of our universe. It has symmetry, elegance, and grace\n- those qualities you find always in that which the true artist captures. You can find it in the turning of the seasons, in the way sand trails along a ridge, in the branch clusters of the creosote\n bush or the pattern of its leaves.\nWe try to copy these patterns in our lives and our society,\n seeking the rhythms, the dances, the forms that comfort.\n Yet, it is possible to see peril in the finding of\n ultimate perfection. It is clear that the ultimate\n pattern contains it own fixity. In such\n perfection, all things move toward death.”\n ~ Dune (1965)\nI find the concept of embeddings to be one of the most fascinating ideas in machine learning. If you’ve ever used Siri, Google Assistant, Alexa, Google Translate, or even smartphone keyboard with next-word prediction, then chances are you’ve benefitted from this idea that has become central to Natural Language Processing models. There has been quite a development over the last couple of decades in using embeddings for neural models (Recent developments include contextualized word embeddings leading to cutting-edge models like\nBERT\nand GPT2).\nWord2vec is a method to efficiently create word embeddings and has been around since 2013. But in addition to its utility as a word-embedding method, some of its concepts have been shown to be effective in creating recommendation engines and making sense of sequential data even in commercial, non-language tasks. Companies like\nAirbnb\n,\nAlibaba\n,\nSpotify\n, and\nAnghami\nhave all benefitted from carving out this brilliant piece of machinery from the world of NLP and using it in production to empower a new breed of recommendation engines.\nIn this post, we’ll go over the concept of embedding, and the mechanics of generating embeddings with word2vec. But let’s start with an example to get familiar with using vectors to represent things. Did you know that a list of five numbers (a vector) can represent so much about your personality?\nPersonality Embeddings: What are you like?\n“I give you the desert chameleon, whose ability to blend itself into the background tells you all you need to know about the roots of ecology and the foundations of a personal identity” ~Children of Dune\nOn a scale of 0 to 100, how introverted/extraverted are you (where 0 is the most introverted, and 100 is the most extraverted)?\nHave you ever taken a personality test like MBTI – or even better, the\nBig Five Personality Traits\ntest? If you haven’t, these are tests that ask you a list of questions, then score you on a number of axes, introversion/extraversion being one of them.\nExample of the result of a Big Five Personality Trait test. It can really tell you a lot about yourself and is shown to have predictive ability in\nacademic\n,\npersonal\n, and\nprofessional success\n.\nThis\nis one place to find your results.\nImagine I’ve scored 38/100 as my introversion/extraversion score. we can plot that in this way:\nLet’s switch the range to be from -1 to 1:\nHow well do you feel you know a person knowing only this one piece of information about them? Not much. People are complex. So let’s add another dimension – the score of one other trait from the test.\nWe can represent the two dimensions as a point on the graph, or better yet, as a vector from the origin to that point. We have incredible tools to deal with vectors that will come in handy very shortly.\nI’ve hidden which traits we’re plotting just so you get used to not knowing what each dimension represents – but still getting a lot of value from the vector representation of a person’s personality.\nWe can now say that this vector partially represents my personality. The usefulness of such representation comes when you want to compare two other people to me. Say I get hit by a\nbus\nand I need to be replaced by someone with a similar personality. In the following figure, which of the two people is more similar to me?\nWhen dealing with vectors, a common way to calculate a similarity score is\ncosine_similarity\n:\nPerson #1\nis more similar to me in personality. Vectors pointing at the same direction (length plays a role as well) have a higher cosine similarity score.\nYet again, two dimensions aren’t enough to capture enough information about how different people are. Decades of psychology research have led to five major traits (and plenty of sub-traits). So let’s use all five dimensions in our comparison:\nThe problem with five dimensions is that we lose the ability to draw neat little arrows in two dimensions. This is a common challenge in machine learning where we often have to think in higher-dimensional space. The good thing is, though, that cosine_similarity still works. It works with any number of dimensions:\ncosine_similarity works for any number of dimensions. These are much better scores because they're calculated based on a higher resolution representation of the things being compared.\nAt the end of this section, I want us to come out with two central ideas:\nWe can represent people (and things) as vectors of numbers (which is great for machines!).\nWe can easily calculate how similar vectors are to each other.\nWord Embeddings\n“The gift of words is the gift of deception and illusion” ~Children of Dune\nWith this understanding, we can proceed to look at trained word-vector examples (also called word embeddings) and start looking at some of their interesting properties.\nThis is a word embedding for the word “king” (GloVe vector trained on Wikipedia):\n[ 0.50451 , 0.68607 , -0.59517 , -0.022801, 0.60046 , -0.13498 ,\n -0.08813 , 0.47377 , -0.61798 , -0.31012 , -0.076666, 1.493 ,\n -0.034189, -0.98173 , 0.68229 , 0.81722 , -0.51874 , -0.31503 ,\n -0.55809 , 0.66421 , 0.1961 , -0.13495 , -0.11476 , -0.30344 ,\n 0.41177 , -2.223 , -1.0756 , -1.0783 , -0.34354 , 0.33505 ,\n 1.9927 , -0.04234 , -0.64319 , 0.71125 , 0.49159 , 0.16754 ,\n 0.34344 , -0.25663 , -0.8523 , 0.1661 , 0.40102 , 1.1685 ,\n -1.0137 , -0.21585 , -0.15155 , 0.78321 , -0.91241 , -1.6106 ,\n -0.64426 , -0.51042 ]\nIt’s a list of 50 numbers. We can’t tell much by looking at the values. But let’s visualize it a bit so we can compare it other word vectors. Let’s put all these numbers in one row:\nLet’s color code the cells based on their values (red if they’re close to 2, white if they’re close to 0, blue if they’re close to -2):\nWe’ll proceed by ignoring the numbers and only looking at the colors to indicate the values of the cells. Let’s now contrast “King” against other words:\nSee how “Man” and “Woman” are much more similar to each other than either of them is to “king”? This tells you something. These vector representations capture quite a bit of the information/meaning/associations of these words.\nHere’s another list of examples (compare by vertically scanning the columns looking for columns with similar colors):\nA few things to point out:\nThere’s a straight red column through all of these different words. They’re similar along that dimension (and we don’t know what each dimensions codes for)\nYou can see how “woman” and “girl” are similar to each other in a lot of places. The same with “man” and “boy”\n“boy” and “girl” also have places where they are similar to each other, but different from “woman” or “man”. Could these be coding for a vague conception of youth? possible.\nAll but the last word are words representing people. I added an object (water) to show the differences between categories. You can, for example, see that blue column going all the way down and stopping before the embedding for “water”.\nThere are clear places where “king” and “queen” are similar to each other and distinct from all the others. Could these be coding for a vague concept of royalty?\nAnalogies\n\"Words can carry any burden we wish. All that's required is agreement and a tradition upon which to build.\" ~God Emperor of Dune\nThe famous examples that show an incredible property of embeddings is the concept of analogies. We can add and subtract word embeddings and arrive at interesting results. The most famous example is the formula: “king” - “man” + “woman”:\nUsing the\nGensim\nlibrary in python, we can add and subtract word vectors, and it would find the most similar words to the resulting vector. The image shows a list of the most similar words, each with its cosine similarity.\nWe can visualize this analogy as we did previously:\nThe resulting vector from \"king-man+woman\" doesn't exactly equal \"queen\", but \"queen\" is the closest word to it from the 400,000 word embeddings we have in this collection.\nNow that we’ve looked at trained word embeddings, let’s learn more about the training process. But before we get to word2vec, we need to look at a conceptual parent of word embeddings: the neural language model.\nLanguage Modeling\n“The prophet is not diverted by illusions of past, present and future.\nThe fixity of language determines such linear distinctions.\nProphets hold a key to the lock in a language.\nThis is not a mechanical universe. The linear progression of events is imposed by the observer. Cause and effect? That's not it at all.\nThe prophet utters fateful words.\nYou glimpse a thing \"destined to occur.\" But the prophetic instant releases something of infinite portent and power. The universe undergoes a ghostly shift.” ~God Emperor of Dune\nIf one wanted to give an example of an NLP application, one of the best examples would be the next-word prediction feature of a smartphone keyboard. It’s a feature that billions of people use hundreds of times every day.\nNext-word prediction is a task that can be addressed by a\nlanguage model\n. A language model can take a list of words (let’s say two words), and attempt to predict the word that follows them.\nIn the screenshot above, we can think of the model as one that took in these two green words (\nthou shalt\n) and returned a list of suggestions (“not” being the one with the highest probability):\nWe can think of the model as looking like this black box:\nBut in practice, the model doesn’t output only one word. It actually outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words). The keyboard application then has to find the words with the highest scores, and present those to the user.\nThe output of the neural language model is a probability score for all the words the model knows. We're referring to the probability as a percentage here, but 40% would actually be represented as 0.4 in the output vector.\nAfter being trained, early neural language models (\nBengio 2003\n) would calculate a prediction in three steps:\nThe first step is the most relevant for us as we discuss embeddings. One of the results of the training process was this matrix that contains an embedding for each word in our vocabulary. During prediction time, we just look up the embeddings of the input word, and use them to calculate the prediction:\nLet’s now turn to the training process to learn more about how this embedding matrix was developed.\nLanguage Model Training\n“A process cannot be understood by stopping it. Understanding must move with the flow of the process, must join it and flow with it.” ~Dune\nLanguage models have a huge advantage over most other machine learning models. That advantage is that we are able to train them on running text – which we have an abundance of. Think of all the books, articles, Wikipedia content, and other forms of text data we have lying around. Contrast this with a lot of other machine learning models which need hand-crafted features and specially-collected data.\n“You shall know a word by the company it keeps” J.R. Firth\nWords get their embeddings by us looking at which other words they tend to appear next to. The mechanics of that is that\nWe get a lot of text data (say, all Wikipedia articles, for example). then\nWe have a window (say, of three words) that we slide against all of that text.\nThe sliding window generates training samples for our model\nAs this window slides against the text, we (virtually) generate a dataset that we use to train a model. To look exactly at how that’s done, let’s see how the sliding window processes this phrase:\n“Thou shalt not make a machine in the likeness of a human mind” ~Dune\nWhen we start, the window is on the first three words of the sentence:\nWe take the first two words to be features, and the third word to be a label:\nWe now have generated the first sample in the dataset we can later use to train a language model.\nWe then slide our window to the next position and create a second sample:\nAn the second example is now generated.\nAnd pretty soon we have a larger dataset of which words tend to appear after different pairs of words:\nIn practice, models tend to be trained while we’re sliding the window. But I find it clearer to logically separate the “dataset generation” phase from the training phase. Aside from neural-network-based approaches to language modeling, a technique called N-grams was commonly used to train language models (see: Chapter 3 of\nSpeech and Language Processing\n). To see how this switch from N-grams to neural models reflects on real-world products,\nhere’s a 2015 blog post from Swiftkey\n, my favorite Android keyboard, introducing their neural language model and comparing it with their previous N-gram model. I like this example because it shows you how the algorithmic properties of embeddings can be described in marketing speech.\nLook both ways\n\"Paradox is a pointer telling you to look beyond it. If paradoxes bother you, that betrays your deep desire for absolutes. The relativist treats a paradox merely as interesting, perhaps amusing or even, dreadful thought, educational.\" ~God Emperor of Dune\nKnowing what you know from earlier in the post, fill in the blank:\nThe context I gave you here is five words before the blank word (and an earlier mention of “bus”). I’m sure most people would guess the word\nbus\ngoes into the blank. But what if I gave you one more piece of information – a word after the blank, would that change your answer?\nThis completely changes what should go in the blank. the word\nred\nis now the most likely to go into the blank. What we learn from this is the words both before and after a specific word carry informational value. It turns out that accounting for both directions (words to the left and to the right of the word we’re guessing) leads to better word embeddings. Let’s see how we can adjust the way we’re training the model to account for this.\nSkipgram\n“Intelligence takes chance with limited data in an arena where mistakes are not only possible but also necessary.” ~Chapterhouse: Dune\nInstead of only looking two words before the target word, we can also look at two words after it.\nIf we do this, the dataset we’re virtually building and training the model against would look like this:\nThis is called a\nContinuous Bag of Words\narchitecture and is described in\none of the word2vec papers\n[pdf]. Another architecture that also tended to show great results does things a little differently.\nInstead of guessing a word based on its context (the words before and after it), this other architecture tries to guess neighboring words using the current word. We can think of the window it slides against the training text as looking like this:\nThe word in the green slot would be the input word, each pink box would be a possible output.\nThe pink boxes are in different shades because this sliding window actually creates four separate samples in our training dataset:\nThis method is called the\nskipgram\narchitecture. We can visualize the sliding window as doing the following:\nThis would add these four samples to our training dataset:\nWe then slide our window to the next position:\nWhich generates our next four examples:\nA couple of positions later, we have a lot more examples:\nRevisiting the training process\n\"Muad'Dib learned rapidly because his first training was in how to learn. And the first lesson of all was the basic trust that he could learn. It's shocking to find how many people do not believe they can learn, and how many more believe learning to be difficult.\" ~Dune\nNow that we have our skipgram training dataset that we extracted from existing running text, let’s glance at how we use it to train a basic neural language model that predicts the neighboring word.\nWe start with the first sample in our dataset. We grab the feature and feed to the untrained model asking it to predict an appropriate neighboring word.\nThe model conducts the three steps and outputs a prediction vector (with a probability assigned to each word in its vocabulary). Since the model is untrained, it’s prediction is sure to be wrong at this stage. But that’s okay. We know what word it should have guessed – the label/output cell in the row we’re currently using to train the model:\nThe 'target vector' is one where the target word has the probability 1, and all other words have the probability 0.\nHow far off was the model? We subtract the two vectors resulting in an error vector:\nThis error vector can now be used to update the model so the next time, it’s a little more likely to guess\nthou\nwhen it gets\nnot\nas input.\nAnd that concludes the first step of the training. We proceed to do the same process with the next sample in our dataset, and then the next, until we’ve covered all the samples in the dataset. That concludes one\nepoch\nof training. We do it over again for a number of epochs, and then we’d have our trained model and we can extract the embedding matrix from it and use it for any other application.\nWhile this extends our understanding of the process, it’s still not how word2vec is actually trained. We’re missing a couple of key ideas.\nNegative Sampling\n“To attempt an understanding of Muad'Dib without understanding his mortal enemies, the Harkonnens, is to attempt seeing Truth without knowing Falsehood. It is the attempt to see the Light without knowing Darkness. It cannot be.” ~Dune\nRecall the three steps of how this neural language model calculates its prediction:\nThe third step is very expensive from a computational point of view – especially knowing that we will do it once for every training sample in our dataset (easily tens of millions of times). We need to do something to improve performance.\nOne way is to split our target into two steps:\nGenerate high-quality word embeddings (Don’t worry about next-word prediction).\nUse these high-quality embeddings to train a language model (to do next-word prediction).\nWe’ll focus on step 1. in this post as we’re focusing on embeddings. To generate high-quality embeddings using a high-performance model, we can switch the model’s task from predicting a neighboring word:\nAnd switch it to a model that takes the input and output word, and outputs a score indicating if they’re neighbors or not (0 for “not neighbors”, 1 for “neighbors”).\nThis simple switch changes the model we need from a neural network, to a logistic regression model – thus it becomes much simpler and much faster to calculate.\nThis switch requires we switch the structure of our dataset – the label is now a new column with values 0 or 1. They will be all 1 since all the words we added are neighbors.\nThis can now be computed at blazing speed – processing millions of examples in minutes. But there’s one loophole we need to close. If all of our examples are positive (target: 1), we open ourself to the possibility of a smartass model that always returns 1 – achieving 100% accuracy, but learning nothing and generating garbage embeddings.\nTo address this, we need to introduce\nnegative samples\nto our dataset – samples of words that are not neighbors. Our model needs to return 0 for those samples. Now that’s a challenge that the model has to work hard to solve – but still at blazing fast speed.\nFor each sample in our dataset, we add\nnegative examples\n. Those have the same input word, and a 0 label.\nBut what do we fill in as output words? We randomly sample words from our vocabulary\nThis idea is inspired by\nNoise-contrastive estimation\n[pdf]. We are contrasting the actual signal (positive examples of neighboring words) with noise (randomly selected words that are not neighbors). This leads to a great tradeoff of computational and statistical efficiency.\nSkipgram with Negative Sampling (SGNS)\nWe have now covered two of the central ideas in word2vec: as a pair, they’re called skipgram with negative sampling.\nWord2vec Training Process\n\"The machine cannot anticipate every problem of importance to humans. It is the difference between serial bits and an unbroken continuum. We have the one; machines are confined to the other.\" ~God Emperor of Dune\nNow that we’ve established the two central ideas of skipgram and negative sampling, we can proceed to look closer at the actual word2vec training process.\nBefore the training process starts, we pre-process the text we’re training the model against. In this step, we determine the size of our vocabulary (we’ll call this\nvocab_size\n, think of it as, say, 10,000) and which words belong to it.\nAt the start of the training phase, we create two matrices – an\nEmbedding\nmatrix and a\nContext\nmatrix. These two matrices have an embedding for each word in our vocabulary (So\nvocab_size\nis one of their dimensions). The second dimension is how long we want each embedding to be (\nembedding_size\n– 300 is a common value, but we’ve looked at an example of 50 earlier in this post).\nAt the start of the training process, we initialize these matrices with random values. Then we start the training process. In each training step, we take one positive example and its associated negative examples. Let’s take our first group:\nNow we have four words: the input word\nnot\nand output/context words:\nthou\n(the actual neighbor),\naaron\n, and\ntaco\n(the negative examples). We proceed to look up their embeddings – for the input word, we look in the\nEmbedding\nmatrix. For the context words, we look in the\nContext\nmatrix (even though both matrices have an embedding for every word in our vocabulary).\nThen, we take the dot product of the input embedding with each of the context embeddings. In each case, that would result in a number, that number indicates the similarity of the input and context embeddings\nNow we need a way to turn these scores into something that looks like probabilities – we need them to all be positive and have values between zero and one. This is a great task for\nsigmoid\n, the\nlogistic operation\n.\nAnd we can now treat the output of the sigmoid operations as the model’s output for these examples. You can see that\ntaco\nhas the highest score and\naaron\nstill has the lowest score both before and after the sigmoid operations.\nNow that the untrained model has made a prediction, and seeing as though we have an actual target label to compare against, let’s calculate how much error is in the model’s prediction. To do that, we just subtract the sigmoid scores from the target labels.\nerror\n=\ntarget\n-\nsigmoid_scores\nHere comes the “learning” part of “machine learning”. We can now use this error score to adjust the embeddings of\nnot\n,\nthou\n,\naaron\n, and\ntaco\nso that the next time we make this calculation, the result would be closer to the target scores.\nThis concludes the training step. We emerge from it with slightly better embeddings for the words involved in this step (\nnot\n,\nthou\n,\naaron\n, and\ntaco\n). We now proceed to our next step (the next positive sample and its associated negative samples) and do the same process again.\nThe embeddings continue to be improved while we cycle through our entire dataset for a number of times. We can then stop the training process, discard the\nContext\nmatrix, and use the\nEmbeddings\nmatrix as our pre-trained embeddings for the next task.\nWindow Size and Number of Negative Samples\nTwo key hyperparameters in the word2vec training process are the window size and the number of negative samples.\nDifferent tasks are served better by different window sizes. One\nheuristic\nis that smaller window sizes (2-15) lead to embeddings where high similarity scores between two embeddings indicates that the words are\ninterchangeable\n(notice that antonyms are often interchangable if we’re only looking at their surrounding words – e.g.\ngood\nand\nbad\noften appear in similar contexts). Larger window sizes (15-50, or even more) lead to embeddings where similarity is more indicative of\nrelatedness\nof the words. In practice, you’ll often have to provide\nannotations\nthat guide the embedding process leading to a useful similarity sense for your task. The Gensim default window size is 5 (five words before and five words after the input word, in addition to the input word itself).\nThe number of negative samples is another factor of the training process. The original paper prescribes 5-20 as being a good number of negative samples. It also states that 2-5 seems to be enough when you have a large enough dataset. The Gensim default is 5 negative samples.\nConclusion\n“If it falls outside your yardsticks, then you are engaged with intelligence, not with automation” ~God Emperor of Dune\nI hope that you now have a sense for word embeddings and the word2vec algorithm. I also hope that now when you read a paper mentioning “skip gram with negative sampling” (SGNS) (like the recommendation system papers at the top), that you have a better sense for these concepts. As always, all feedback is appreciated\n@JayAlammar\n.\nReferences & Further Readings\nDistributed Representations of Words and Phrases and their Compositionality\n[pdf]\nEfficient Estimation of Word Representations in Vector Space\n[pdf]\nA Neural Probabilistic Language Model\n[pdf]\nSpeech and Language Processing\nby Dan Jurafsky and James H. Martin is a leading resource for NLP. Word2vec is tackled in Chapter 6.\nNeural Network Methods in Natural Language Processing\nby\nYoav Goldberg\nis a great read for neural NLP topics.\nChris McCormick\nhas written some great blog posts about Word2vec. He also just released\nThe Inner Workings of word2vec\n, an E-book focused on the internals of word2vec.\nWant to read the code? Here are two options:\nGensim’s\npython implementation\nof word2vec\nMikolov’s original\nimplementation in C\n– better yet, this\nversion with detailed comments\nfrom Chris McCormick.\nEvaluating distributional models of compositional semantics\nOn word embeddings\n,\npart 2\nDune\nWritten on March 27, 2019", "metadata": { "title": "The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.", "description": "Discussions:\nHacker News (347 points, 37 comments), Reddit r/MachineLearning (151 points, 19 comments)\n\n\n\nTranslations: Chinese (Simplified), French, Korean, Portuguese, Russian\n\n\n\n \n \n\n\n\n “There is in all things a pattern that is part of our universe. It has symmetry, elegance, and grace - those qualities you find always in that which the true artist captures. You can find it in the turning of the seasons, in the way sand trails along a ridge, in the branch clusters of the creosote\n bush or the pattern of its leaves. \n\n We try to copy these patterns in our lives and our society,\n seeking the rhythms, the dances, the forms that comfort.\n Yet, it is possible to see peril in the finding of\n ultimate perfection. It is clear that the ultimate\n pattern contains it own fixity. In such\n perfection, all things move toward death.”\n ~ Dune (1965)\n\n\nI find the concept of embeddings to be one of the most fascinating ideas in machine learning. If you’ve ever used Siri, Google Assistant, Alexa, Google Translate, or even smartphone keyboard with next-word prediction, then chances are you’ve benefitted from this idea that has become central to Natural Language Processing models. There has been quite a development over the last couple of decades in using embeddings for neural models (Recent developments include contextualized word embeddings leading to cutting-edge models like BERT and GPT2).\n\n\n\nWord2vec is a method to efficiently create word embeddings and has been around since 2013. But in addition to its utility as a word-embedding method, some of its concepts have been shown to be effective in creating recommendation engines and making sense of sequential data even in commercial, non-language tasks. Companies like Airbnb, Alibaba, Spotify, and Anghami have all benefitted from carving out this brilliant piece of machinery from the world of NLP and using it in production to empower a new breed of recommendation engines.\n\nIn this post, we’ll go over the concept of embedding, and the mechanics of generating embeddings with word2vec. But let’s start with an example to get familiar with using vectors to represent things. Did you know that a list of five numbers (a vector) can represent so much about your personality?", "keywords": null, "author": "Jay Alammar", "og:description": "Discussions:\nHacker News (347 points, 37 comments), Reddit r/MachineLearning (151 points, 19 comments)\n\n\n\nTranslations: Chinese (Simplified), French, Korean, Portuguese, Russian\n\n\n\n \n \n\n\n\n “There is in all things a pattern that is part of our universe. It has symmetry, elegance, and grace - those qualities you find always in that which the true artist captures. You can find it in the turning of the seasons, in the way sand trails along a ridge, in the branch clusters of the creosote\n bush or the pattern of its leaves. \n\n We try to copy these patterns in our lives and our society,\n seeking the rhythms, the dances, the forms that comfort.\n Yet, it is possible to see peril in the finding of\n ultimate perfection. It is clear that the ultimate\n pattern contains it own fixity. In such\n perfection, all things move toward death.”\n ~ Dune (1965)\n\n\nI find the concept of embeddings to be one of the most fascinating ideas in machine learning. If you’ve ever used Siri, Google Assistant, Alexa, Google Translate, or even smartphone keyboard with next-word prediction, then chances are you’ve benefitted from this idea that has become central to Natural Language Processing models. There has been quite a development over the last couple of decades in using embeddings for neural models (Recent developments include contextualized word embeddings leading to cutting-edge models like BERT and GPT2).\n\n\n\nWord2vec is a method to efficiently create word embeddings and has been around since 2013. But in addition to its utility as a word-embedding method, some of its concepts have been shown to be effective in creating recommendation engines and making sense of sequential data even in commercial, non-language tasks. Companies like Airbnb, Alibaba, Spotify, and Anghami have all benefitted from carving out this brilliant piece of machinery from the world of NLP and using it in production to empower a new breed of recommendation engines.\n\nIn this post, we’ll go over the concept of embedding, and the mechanics of generating embeddings with word2vec. But let’s start with an example to get familiar with using vectors to represent things. Did you know that a list of five numbers (a vector) can represent so much about your personality?", "og:title": "The Illustrated Word2vec" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=ISPId9Lhc1g", "src": "https://www.youtube.com/embed/ISPId9Lhc1g", "status": "ok", "kind": "single_video", "local_path": "attachments/37477c68e81f/videos/repair-ISPId9Lhc1g.mp4", "bytes": 41541081, "valid": true, "height": 1080, "width": 1920, "duration": "523.761000", "reused": false }, { "url": "https://www.youtube.com/watch?v=tAxrlAVw-Tk", "src": "https://youtu.be/tAxrlAVw-Tk?t=648", "status": "ok", "kind": "single_video", "local_path": "attachments/37477c68e81f/videos/repair-tAxrlAVw-Tk.mp4", "bytes": 699025645, "valid": true, "height": 1080, "width": 1920, "duration": "2425.461000", "reused": false }, { "url": "https://www.youtube.com/watch?v=ao52o9l6KGw", "src": "https://youtu.be/ao52o9l6KGw?t=287", "status": "ok", "kind": "single_video", "local_path": "attachments/37477c68e81f/videos/repair-ao52o9l6KGw.mp4", "bytes": 313463152, "valid": true, "height": 1080, "width": 1920, "duration": "2184.101000", "reused": false } ], "links": [ { "href": "https://news.ycombinator.com/item?id=19498356", "label": "Hacker News (347 points, 37 comments)", "type": "a" }, { "href": "https://www.reddit.com/r/MachineLearning/comments/b60jtg/p_the_illustrated_word2vec/", "label": "Reddit r/MachineLearning (151 points, 19 comments)", "type": "a" }, { "href": "https://mp.weixin.qq.com/s?__biz=MjM5MTQzNzU2NA==&mid=2651669277&idx=2&sn=bc8f0590f9e340c1f1359982726c5a30&chksm=bd4c648e8a3bed9817f30c5a512e79fe0cc6fbc58544f97c857c30b120e76508fef37cae49bc&scene=0&xtrack=1", "label": "Chinese (Simplified)", "type": "a" }, { "href": "https://lbourdois.github.io/blog/nlp/word_embedding/", "label": "French", "type": "a" }, { "href": "https://databreak.netlify.com/2019-04-25-illustrated_word2vec/", "label": "Korean", "type": "a" }, { "href": "https://pessoalex.wordpress.com/2019/03/29/o-word2vec-ilustrado/", "label": "Portuguese", "type": "a" }, { "href": "https://habr.com/ru/post/446530/", "label": "Russian", "type": "a" }, { "href": "https://jalammar.github.io/illustrated-bert/", "label": "BERT", "type": "a" }, { "href": "https://www.youtube.com/embed/ISPId9Lhc1g", "label": "", "type": "iframe" }, { "href": "https://www.kdd.org/kdd2018/accepted-papers/view/real-time-personalization-using-embeddings-for-search-ranking-at-airbnb", "label": "Airbnb", "type": "a" }, { "href": "https://www.kdd.org/kdd2018/accepted-papers/view/billion-scale-commodity-embedding-for-e-commerce-recommendation-in-alibaba", "label": "Alibaba", "type": "a" }, { "href": "https://www.slideshare.net/AndySloane/machine-learning-spotify-madison-big-data-meetup", "label": "Spotify", "type": "a" }, { "href": "https://towardsdatascience.com/using-word2vec-for-music-recommendations-bb9649ac2484", "label": "Anghami", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Big_Five_personality_traits", "label": "Big Five Personality Traits", "type": "a" }, { "href": "http://psychology.okstate.edu/faculty/jgrice/psyc4333/FiveFactor_GPAPaper.pdf", "label": "academic", "type": "a" }, { "href": "https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-6570.1999.tb00174.x", "label": "personal", "type": "a" }, { "href": "https://www.massgeneral.org/psychiatry/assets/published_papers/soldz-1999.pdf", "label": "professional success", "type": "a" }, { "href": "https://projects.fivethirtyeight.com/personality-quiz/", "label": "This", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Cosine_similarity", "label": "cosine_similarity", "type": "a" }, { "href": "https://radimrehurek.com/gensim/", "label": "Gensim", "type": "a" }, { "href": "http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf", "label": "Bengio 2003", "type": "a" }, { "href": "http://web.stanford.edu/~jurafsky/slp3/", "label": "Speech and Language Processing", "type": "a" }, { "href": "https://blog.swiftkey.com/neural-networks-a-meaningful-leap-for-mobile-typing/", "label": "here’s a 2015 blog post from Swiftkey", "type": "a" }, { "href": "https://arxiv.org/pdf/1301.3781.pdf", "label": "one of the word2vec papers", "type": "a" }, { "href": "http://proceedings.mlr.press/v9/gutmann10a/gutmann10a.pdf", "label": "Noise-contrastive estimation", "type": "a" }, { "href": "https://jalammar.github.io/feedforward-neural-networks-visual-interactive/", "label": "sigmoid", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Logistic_function", "label": "logistic operation", "type": "a" }, { "href": "https://youtu.be/tAxrlAVw-Tk?t=648", "label": "heuristic", "type": "a" }, { "href": "https://youtu.be/ao52o9l6KGw?t=287", "label": "annotations", "type": "a" }, { "href": "https://twitter.com/JayAlammar", "label": "@JayAlammar", "type": "a" }, { "href": "https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf", "label": "Distributed Representations of Words and Phrases and their Compositionality", "type": "a" }, { "href": "https://web.stanford.edu/~jurafsky/slp3/", "label": "Speech and Language Processing", "type": "a" }, { "href": "https://www.amazon.com/Language-Processing-Synthesis-Lectures-Technologies/dp/1627052984", "label": "Neural Network Methods in Natural Language Processing", "type": "a" }, { "href": "https://twitter.com/yoavgo", "label": "Yoav Goldberg", "type": "a" }, { "href": "http://mccormickml.com/", "label": "Chris McCormick", "type": "a" }, { "href": "https://www.preview.nearist.ai/paid-ebook-and-tutorial", "label": "The Inner Workings of word2vec", "type": "a" }, { "href": "https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec.py", "label": "python implementation", "type": "a" }, { "href": "https://github.com/tmikolov/word2vec/blob/master/word2vec.c", "label": "implementation in C", "type": "a" }, { "href": "https://github.com/chrisjmccormick/word2vec_commented/blob/master/word2vec.c", "label": "version with detailed comments", "type": "a" }, { "href": "http://sro.sussex.ac.uk/id/eprint/61062/1/Batchkarov,%20Miroslav%20Manov.pdf", "label": "Evaluating distributional models of compositional semantics", "type": "a" }, { "href": "http://ruder.io/word-embeddings-1/index.html", "label": "On word embeddings", "type": "a" }, { "href": "http://ruder.io/word-embeddings-softmax/", "label": "part 2", "type": "a" }, { "href": "https://www.amazon.com/Dune-Frank-Herbert/dp/0441172717/", "label": "Dune", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:30Z", "source_url": "https://jalammar.github.io/illustrated-word2vec/", "papers": [ { "status": "http_403", "url": "http://psychology.okstate.edu/faculty/jgrice/psyc4333/FiveFactor_GPAPaper.pdf", "retried_at": "2026-10-20T17:45:46Z" }, { "status": "invalid_file_type", "url": "https://www.massgeneral.org/psychiatry/assets/published_papers/soldz-1999.pdf", "content_type": "text/html; charset=utf-8", "retried_at": "2026-10-20T17:45:48Z" }, { "status": "ok", "url": "http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf", "bytes": 140095, "reused": false, "local_path": "papers/repair/8ec2bcdfae6c83888cf3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1301.3781.pdf", "bytes": 228716, "reused": true, "local_path": "papers/repair/1e8e84311c23b0346b07.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v9/gutmann10a/gutmann10a.pdf", "bytes": 633631, "reused": true, "local_path": "papers/repair/4e238b55e8eb6f9638d6.pdf" }, { "status": "ok", "url": "https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf", "bytes": 111997, "reused": true, "local_path": "papers/repair/32de112509f802883250.pdf" }, { "status": "http_202", "url": "http://sro.sussex.ac.uk/id/eprint/61062/1/Batchkarov,%20Miroslav%20Manov.pdf", "retried_at": "2026-10-20T17:45:47Z" } ], "actual_video_files": [], "body_chars": 26648, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "http_202；http_403；invalid_file_type", "raw_html": "attachments/37477c68e81f/pages/source.html", "local_html": "attachments/37477c68e81f/pages/article.html", "body_base_path": "attachments/37477c68e81f/pages/", "body_markdown_path": "attachments/37477c68e81f/pages/article.md" }