{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Hacking Scale\nBetter Stack / Richard Oliver Bray",
  "title_listed": "How OpenAI Scaled Kubernetes to 7,500 Nodes by Removing One Plugin\n",
  "keywords": "Kubernetes, 分布式训练, 基础设施扩展, 训练集群",
  "domain": "数据处理",
  "content_type": "技术深潜",
  "is_paid_flag": "免费",
  "url": "https://newsletter.betterstack.com/p/how-openai-scaled-kubernetes-to-7500?utm_source=publication-search",
  "publish_date": "2024-08",
  "corpus_stage": "预训练",
  "notes": "OpenAI大规模K8s集群扩展经验，对LLM训练基础设施扩展的参考价值",
  "id": "ca6ac3c33e71",
  "collected_at": "2026-10-01T20:16:36",
  "title": "How OpenAI Scaled Kubernetes to 7,500 Nodes by Removing One Plugin",
  "body_markdown": "One of the biggest names in AI tripled the size of its infrastructure, going from\n2,500 nodes to 7,500\nin just a few years.\nTo put that into perspective, the average enterprise company will do fine with around\n50-100\nnodes.\nThis\nenormous increase\nin computing power worked because of\nlots of changes\n.\nBut the most important one was removing a plugin called\nFlannel\n.\nWhy? Let’s dive into it.\nEstimated reading time: 3 minutes 59 seconds\nWe do tech deep dives like this\nevery 2 weeks\n. Get the next one sent straight to your inbox.\nSubscribe\nWhat is Flannel?\nImagine we have a Kubernetes cluster that contains two\nnodes\n(virtual computers), one for a\nNode.js web server\n, another for a\nPostgres database\n.\nInside each of these nodes we'll have two pods (containers), one for the web server/database and something to collect logs like\nVector\n.\nSidenote: Node vs Pod\nPods usually contain a\nsingle application\n(multiple if needed), and a node provides resources for the pod to run, CPU, Memory etc.\nNodes get assigned an IP address by the cloud provider. Pods do get assigned IPs by default using\nkubnet\n, but are used to communicate\nonly within the same node\n.\nIf you want pods in different nodes to communicate, HTTP is a good approach.\nBut before you can even do that, you'll need to give each pod a\nunique IP address\nso they know where to send data.\nA bit like giving someone a\nunique postal address\nso you're able to send letters to them.\nPod-to-pod communication between nodes out-of-the-box with Kubernetes is\nnot straightforward\n, and this is where Flannel comes in.\nFlannel exists to make pod-to-pod communication really easy. It\nconfigures the network\nfor each pod and does things like\nassigns each pod an IP address\n.\nIn the world of Kubernetes, Flannel is called a\nCNI\nplugin\n(Container Network Interface). It's not the only one; there are others like\nCalico\nand\nCilium\n.\nBut Flannel is the\neasiest to setup\n.\nWhy Was it Removed?\nFlannel is a great solution, but it wasn't designed for\nthousands of nodes\n.\nOpenAI was already\nseeing slow speeds with Flannel for 2,500 nodes\n, so 7,500 nodes would have definitely been a struggle.\nTo fully understand what made it slow, let's go back to our previous example and\ndouble the number of nodes\n.\nFor communication between nodes, Flannel performs several tasks:\nAssigns\neach cluster a range of IP addresses for each node\nand a range of IPs for nodes to give to each pod (subnet allocation).\nCreates\nroute tables\nto keep track of which node has which IP address.\nWith this information, if data needs to be sent to a specific pod, Flannel will know the\nexact IP address of the node to which the pod belongs to\n(traffic routing).\nIt also\nlabels traveling data\n(packets) so that if it needs to pass through a cluster to reach its destination, it is not read by accident (packet encapsulation).\nThe aim of Flannel is to speed up communication, but with all these processes on top of thousands of nodes, it actually\nslows down the network\n.\nAs shown in the image below,\nwith 150,000\nrequests per second (RPS),\nFlannel’s VXLAN\ntends to have the\nworst performance\n.\nOf course, the team at OpenAI needed to do something about this.\nSidenote: VXLAN\nVXLAN (Virtual eXtensible Local Area Network) is used to create a\nvirtual network\nthat allows machines to communicate.\nLong before the internet, if you had two physical computers, you would have to\nconnect a cable between\nthem so they could communicate.\nA VXLAN simulates this using the\npower of the internet\nor a larger network.\nIf two computers are connected to the internet and have\nVXLAN software installed\n, they each get a unique IP address, and packets are encapsulated so that it’s not viewed by another machine.\nWhat Was Used Instead?\nIf you keep up with\nOpenAI news\n, or just AI news in general, you'll know that Microsoft has put a\nlot of money behind them\n.\nYou may also know that Microsoft owns Azure, which is a\nhuge cloud provider\n.\nSo it only makes sense that OpenAI hosts\nall of its infrastructure on Azure\n. I mean, it would be weird if they hosted it on Google Cloud or AWS, right?\nSince OpenAI was using the\nAzure Kubernetes Service (AKS)\nfor their infrastructure, using\nAzure's CNI\nwas the obvious choice, as both services work better together.\nAzure's CNI isn't just\na fork of Flannel with Microsoft branding\n; it's different in many ways:\nBoth nodes and pods are given\nIP addresses from the same range\n(subnet allocation). For example, if a node has IP 10.0.0.1, the pod gets 10.0.0.2, allowing direct pod-to-pod communication without going through a node.\nIt doesn't use a virtual network (VXLAN); it uses direct routing. This means\nno virtual routes\nneed to be created, and there's\nno encapsulation\nof packets.\nIt is not a general-purpose CNI; it is specifically designed for Azure Kubernetes Service, meaning it has\nnative integration and better performance\nwith an AKS cluster.\nThe red pods here are for the Azure CNI service and not Flannel\nAll of these changes made Azure CNI\nmuch faster\nthan Flannel, but it's not the same as Flannel.\nThe team had to use a different solution to keep track of packets and rely on a different\nrouting engine\n.\nHowever, the benefits\noutweigh the drawbacks\n.\nOpenAI even claimed that pod-to-pod communication between clusters was as fast as direct communication between\npods in the same node\n.\nAnd That’s a Wrap\nIt's amazing to think that this whole article was just\none paragraph\nof the\noriginal post\n.\nThere's a lot of good information in there, and I recommend you check it out.\nBut if you enjoyed this\nsimplified format\n, go ahead and\nsubscribe\nto get more like this.\nWe do tech deep dives like this\nevery 2 weeks\n. Get the next one sent straight to your inbox.\nSubscribe\nPS: Enjoyed this newsletter? Please forward it to a pal or follow us on socials (\nLinkedIn\n,\nTwitter\n,\nYouTube\n,\nInstagram\n). It only takes 10 seconds. Making this one took 20 hours.",
  "metadata": {
    "title": "How OpenAI Scaled Kubernetes to 7,500 Nodes by Removing One Plugin",
    "description": "The one change that improved OpenAI's network.",
    "keywords": null,
    "author": "Richard Oliver Bray",
    "og:type": "article",
    "og:title": "How OpenAI Scaled Kubernetes to 7,500 Nodes by Removing One Plugin",
    "og:description": "The one change that improved OpenAI's network.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!tBw6!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd0e7206-f377-41f9-84f1-8ca86197c981_1708x1193.png",
    "og:url": "https://newsletter.betterstack.com/p/how-openai-scaled-kubernetes-to-7500",
    "twitter:title": "How OpenAI Scaled Kubernetes to 7,500 Nodes by Removing One Plugin",
    "twitter:description": "The one change that improved OpenAI's network.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!lu9W!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fbetterstack.substack.com%2Fapi%2Fv1%2Fpost_preview%2F146640134%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2024-08-07T09:06:46.920Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!tBw6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd0e7206-f377-41f9-84f1-8ca86197c981_1708x1193.png",
      "bytes": 108798,
      "reused": true,
      "local_path": "attachments/ca6ac3c33e71/images/039.png",
      "src": "https://substackcdn.com/image/fetch/$s_!tBw6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd0e7206-f377-41f9-84f1-8ca86197c981_1708x1193.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!1Idh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfca89a6-9e49-4b3a-8560-2c6ee511bb76_1736x1459.png",
      "bytes": 148438,
      "reused": false,
      "local_path": "attachments/ca6ac3c33e71/images/body-32e28bc268ee6ae6522c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!1Idh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfca89a6-9e49-4b3a-8560-2c6ee511bb76_1736x1459.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!AoQT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe37240f6-6394-47e0-bb83-944c51ebe60e_1280x1194.png",
      "bytes": 96615,
      "reused": false,
      "local_path": "attachments/ca6ac3c33e71/images/body-15948eb75e9c175901ee.png",
      "src": "https://substackcdn.com/image/fetch/$s_!AoQT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe37240f6-6394-47e0-bb83-944c51ebe60e_1280x1194.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!oisb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb28eab21-74f7-4457-9102-b20846942ee0_1529x1194.png",
      "bytes": 122917,
      "reused": false,
      "local_path": "attachments/ca6ac3c33e71/images/body-46ba9a4b8ac8a4aa2cc9.png",
      "src": "https://substackcdn.com/image/fetch/$s_!oisb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb28eab21-74f7-4457-9102-b20846942ee0_1529x1194.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!UX_y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c0c0548-700b-41ed-b523-412da6e50c6e_2880x960.png",
      "bytes": 48909,
      "reused": false,
      "local_path": "attachments/ca6ac3c33e71/images/body-3a5e91052d24370e2956.png",
      "src": "https://substackcdn.com/image/fetch/$s_!UX_y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c0c0548-700b-41ed-b523-412da6e50c6e_2880x960.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!_ryb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78180de7-68ee-4aec-8629-4f1f38f936f6_1559x1194.png",
      "bytes": 121707,
      "reused": false,
      "local_path": "attachments/ca6ac3c33e71/images/body-aca73edc582a5eddd800.png",
      "src": "https://substackcdn.com/image/fetch/$s_!_ryb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78180de7-68ee-4aec-8629-4f1f38f936f6_1559x1194.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/@betterstack",
      "src": "https://www.youtube.com/@betterstack",
      "status": "reference_only",
      "kind": "channel_reference"
    }
  ],
  "links": [
    {
      "href": "https://openai.com/index/scaling-kubernetes-to-7500-nodes/",
      "label": "lots of changes",
      "type": "a"
    },
    {
      "href": "https://github.com/flannel-io/flannel",
      "label": "Flannel",
      "type": "a"
    },
    {
      "href": "https://vector.dev/",
      "label": "Vector",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!tBw6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd0e7206-f377-41f9-84f1-8ca86197c981_1708x1193.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!1Idh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcfca89a6-9e49-4b3a-8560-2c6ee511bb76_1736x1459.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://docs.tigera.io/",
      "label": "Calico",
      "type": "a"
    },
    {
      "href": "https://cilium.io/",
      "label": "Cilium",
      "type": "a"
    },
    {
      "href": "https://www.devopsschool.com/blog/list-of-cni-plugins-used-in-kubernetes/",
      "label": "easiest to setup",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/scaling-kubernetes-to-2500-nodes/",
      "label": "seeing slow speeds with Flannel for 2,500 nodes",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!AoQT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe37240f6-6394-47e0-bb83-944c51ebe60e_1280x1194.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!oisb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb28eab21-74f7-4457-9102-b20846942ee0_1529x1194.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://machinezone.github.io/research/networking-solutions-for-kubernetes/",
      "label": "worst performance",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!UX_y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c0c0548-700b-41ed-b523-412da6e50c6e_2880x960.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.cnbc.com/2023/04/08/microsofts-complex-bet-on-openai-brings-potential-and-uncertainty.html",
      "label": "lot of money behind them",
      "type": "a"
    },
    {
      "href": "https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni?tabs=configure-networking-portal",
      "label": "Azure's CNI",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!_ryb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78180de7-68ee-4aec-8629-4f1f38f936f6_1559x1194.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/company/betterstack/",
      "label": "LinkedIn",
      "type": "a"
    },
    {
      "href": "https://x.com/BetterStackHQ",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/@betterstack",
      "label": "YouTube",
      "type": "a"
    },
    {
      "href": "https://www.instagram.com/betterstackhq/",
      "label": "Instagram",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:12Z",
  "source_url": "https://newsletter.betterstack.com/p/how-openai-scaled-kubernetes-to-7500?utm_source=publication-search",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5909,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ca6ac3c33e71/pages/source.html",
  "local_html": "attachments/ca6ac3c33e71/pages/article.html",
  "body_base_path": "attachments/ca6ac3c33e71/pages/",
  "body_markdown_path": "attachments/ca6ac3c33e71/pages/article.md"
}