{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Wesley Kambale",
  "title_listed": "Scalable Model Serving with TensorFlow Serving",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://kambale.dev/scalable-model-serving-with-tensorflow-serving",
  "publish_date": "2025-02-10",
  "corpus_stage": "",
  "notes": "",
  "id": "5f8a664df4f8",
  "collected_at": "2026-10-02T14:11:21",
  "title": "Scalable Model Serving with TensorFlow Serving",
  "body_markdown": "Scalable Model Serving with TensorFlow Serving\nDeploying Scalable ML Models with TensorFlow Serving\nPublished\nFebruary 10, 2025\n•\n5\nmin read\n•\nView as Markdown\nW\nWesley Kambale\nI'm a Machine Learning Engineer passionate about building production-ready ML systems for the African market.\nWith experience in TensorFlow, Keras, and Python-based workflows, I help teams bridge the gap between machine learning research and real-world deployment—especially on resource-constrained devices.\nI'm also a Google Developer Expert in AI. I regularly speak at tech conferences including PyCon Africa, DevFest Kampala, DevFest Nairobi and more and also write technical articles on AI/ML here.\nPart of series\nMachine Learning\nIntroduction\nIn this article, we explore how to deploy machine learning models in a scalable and efficient manner using TensorFlow Serving. TensorFlow Serving is a flexible, high-performance serving system designed for production environments, enabling you to serve your machine learning models to a large number of clients efficiently. We will cover the basics of TensorFlow Serving, how to set it up, how to serve models, and best practices for scaling your deployment.\nWhat is TensorFlow Serving?\nTensorFlow Serving is an open-source serving system specifically designed for deploying machine learning models in production environments. It allows you to serve multiple models or multiple versions of the same model simultaneously, and it can be easily integrated with TensorFlow models.\nWhy Use TensorFlow Serving?\nScalability\n: TensorFlow Serving is designed to handle high-throughput predictions, making it suitable for large-scale deployments.\nFlexibility\n: It supports multiple models and versions, allowing for easy model management.\nEfficiency\n: TensorFlow Serving is optimized for performance, with low latency and high throughput.\nSetting Up TensorFlow Serving\nInstallation\nTensorFlow Serving can be installed on various platforms, including Linux, macOS, and Windows. However, the most common way to get TensorFlow Serving up and running is through Docker, which simplifies the process and ensures a consistent environment.\nInstalling TensorFlow Serving on Linux\nIf you prefer to install TensorFlow Serving directly on your system, you can follow these steps:\necho\n\"deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\"\n| sudo tee /etc/apt/sources.list.d/tensorflow-serving.list > /dev/null\ncurl -fsSL https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -\nsudo apt-get update && sudo apt-get install tensorflow-model-server\nVerify Installation\ntensorflow_model_server --version\nDocker Setup\nUsing Docker is the recommended way to set up TensorFlow Serving, as it provides a consistent environment across different platforms.\nPull the TensorFlow Serving Docker Image\ndocker pull tensorflow/serving\nRun the Docker Container\ndocker run -p 8501:8501 --name=tf_serving \\\n --mount\ntype\n=\nbind\n,\nsource\n=$(\npwd\n)/model,target=/models/model_name \\\n -e MODEL_NAME=model_name -t tensorflow/serving\nThis command starts a TensorFlow Serving container, serving the model located in the\nmodel\ndirectory.\nServing a TensorFlow Model\nExporting a TensorFlow Model\nBefore serving a model, you need to export it in a format that TensorFlow Serving can understand. Typically, TensorFlow models are saved in the SavedModel format.\nimport\ntensorflow\nas\ntf\n# Define a simple model\nmodel = tf.keras.Sequential([\n    tf.keras.layers.Dense(\n10\n, activation=\n'relu'\n, input_shape=(\n32\n,)),\n    tf.keras.layers.Dense(\n1\n)\n])\n# Save the model\nmodel.save(\n'/path/to/exported_model'\n)\nLoading the Model into TensorFlow Serving\nWith the model saved in the correct format, you can load it into TensorFlow Serving by pointing the server to the directory containing the exported model.\ndocker run -p 8501:8501 --name=tf_serving \\\n--mount\ntype\n=\nbind\n,\nsource\n=/path/to/exported_model,target=/models/my_model \\\n-e MODEL_NAME=my_model -t tensorflow/serving\nMaking Predictions via REST API\nTensorFlow Serving provides a REST API to interact with your models. You can make predictions by sending HTTP POST requests.\ncurl -d\n'{\"instances\": [[1.0, 2.0, 5.0, 1.0, 2.0, 5.0]]}'\n\\\n  -X POST http://localhost:8501/v1/models/my_model:predict\nThis request sends a JSON payload containing the input data and receives the model's predictions as a response.\nScaling TensorFlow Serving\nHorizontal and Vertical Scaling\nHorizontal Scaling\n: Involves adding more instances of TensorFlow Serving, distributing the load across multiple servers. This can be achieved using container orchestration platforms like Kubernetes.\nVertical Scaling\n: Involves increasing the resources (CPU, memory) of a single TensorFlow Serving instance. This can be done by allocating more resources to the Docker container.\nLoad Balancing\nLoad balancing is crucial for handling large volumes of requests efficiently. You can use a load balancer to distribute incoming requests across multiple TensorFlow Serving instances.\nMonitoring and Logging\nMonitoring and logging are essential for understanding the performance of your TensorFlow Serving deployment. TensorFlow Serving integrates well with monitoring tools like Prometheus and Grafana.\nExample Prometheus configuration:\nglobal:\nscrape_interval:\n15s\nscrape_configs:\n-\njob_name:\n'tensorflow_serving'\nstatic_configs:\n-\ntargets:\n[\n'localhost:8501'\n]\nThis configuration will scrape metrics from your TensorFlow Serving instance every 15 seconds.\nAdvanced Features of TensorFlow Serving\nModel Versioning\nTensorFlow Serving supports serving multiple versions of the same model. You can specify which version to serve using the\n--model_version_policy\nflag.\ndocker run -p 8501:8501 --name=tf_serving \\\n--mount\ntype\n=\nbind\n,\nsource\n=/path/to/exported_model,target=/models/my_model \\\n-e MODEL_NAME=my_model -e MODEL_VERSION_POLICY=\n\"latest\"\n-t tensorflow/serving\nBatch Prediction\nTensorFlow Serving can be configured to perform batch predictions, which can significantly improve performance for high-throughput scenarios.\ndocker run -p 8501:8501 --name=tf_serving \\\n--mount\ntype\n=\nbind\n,\nsource\n=/path/to/exported_model,target=/models/my_model \\\n-e MODEL_NAME=my_model -e TF_SERVING_BATCHING_PARAMETERS_FILE=\n\"/path/to/batching_parameters\"\n\\\n-t tensorflow/serving\nCustomizing TensorFlow Serving\nYou can customize TensorFlow Serving by adding custom code for preprocessing, postprocessing, or integrating with other systems.\nExample of a custom model handler:\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow_serving.apis\nimport\npredict_pb2, prediction_service_pb2_grpc\nclass\nCustomModelHandler\n:\ndef\n__init__\n(\nself, model_path\n):\nself.model = tf.keras.models.load_model(model_path)\ndef\npredict\n(\nself, request: predict_pb2.PredictRequest\n) -> predict_pb2.PredictResponse:\n# Custom preprocessing\ninputs = request.inputs[\n'input_tensor'\n].numpy()\n# Model prediction\npredictions = self.model.predict(inputs)\n# Custom postprocessing\nresponse = predict_pb2.PredictResponse()\n        response.outputs[\n'output_tensor'\n].CopyFrom(tf.make_tensor_proto(predictions))\nreturn\nresponse\nBest Practices\nSecurity Considerations\nAuthentication\n: Implement authentication mechanisms to ensure that only authorized clients can access your models.\nEncryption\n: Use TLS to encrypt data in transit between clients and TensorFlow Serving.\nResource Management\nCPU and Memory Limits\n: Set appropriate limits on CPU and memory usage to prevent resource exhaustion.\nAutoscaling\n: Use autoscaling to dynamically adjust the number of TensorFlow Serving instances based on demand.\nOptimizing Performance\nModel Optimization\n: Optimize your model using techniques like quantization to reduce latency.\nCaching\n: Implement caching mechanisms to store frequently requested predictions, reducing the load on your TensorFlow Serving instance.\nConclusion\nTensorFlow Serving provides a powerful and flexible solution for serving machine learning models in production environments. By following the steps outlined in this tutorial, you can set up a scalable TensorFlow Serving deployment that is capable of handling large volumes of requests efficiently. With advanced features like model versioning, batch prediction, and custom handlers, TensorFlow Serving can be tailored to meet the specific needs of your application.\n#\nscalable-models\n#\ntensorflow\n#\ntensorflow-serving",
  "metadata": {
    "title": "Scalable Model Serving with TensorFlow Serving",
    "description": "Deploying Scalable ML Models with TensorFlow Serving",
    "keywords": null,
    "author": null,
    "og:title": "Scalable Model Serving with TensorFlow Serving",
    "og:description": "Deploying Scalable ML Models with TensorFlow Serving",
    "og:url": "https://kambale.dev/scalable-model-serving-with-tensorflow-serving",
    "og:site_name": "Wesley Kambale",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1723164320089/ca0bea24-fd42-464d-9547-4cc9db9fedcd.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://twitter.com/WesKambale",
    "twitter:title": "Scalable Model Serving with TensorFlow Serving",
    "twitter:description": "Deploying Scalable ML Models with TensorFlow Serving",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1723164320089/ca0bea24-fd42-464d-9547-4cc9db9fedcd.png",
    "article:published_time": "2025-02-10T22:09:09.747Z",
    "article:author": "https://kambale.dev/@kambale"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/5f8a664df4f8/code/body-000.txt",
      "chars": 402,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-001.txt",
      "chars": 34,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-002.txt",
      "chars": 32,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-003.txt",
      "chars": 160,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-004.txt",
      "chars": 236,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-005.txt",
      "chars": 164,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-006.txt",
      "chars": 119,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-007.txt",
      "chars": 139,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-008.txt",
      "chars": 197,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-009.txt",
      "chars": 236,
      "language": "text"
    },
    {
      "file": "attachments/5f8a664df4f8/code/body-010.txt",
      "chars": 672,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1723164320089/ca0bea24-fd42-464d-9547-4cc9db9fedcd.png",
      "bytes": 104955,
      "reused": true,
      "local_path": "attachments/5f8a664df4f8/images/000.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1723164320089/ca0bea24-fd42-464d-9547-4cc9db9fedcd.png",
      "alt": "Scalable Model Serving with TensorFlow Serving",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://kambale.dev/scalable-model-serving-with-tensorflow-serving.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@kambale",
      "label": "W",
      "type": "a"
    },
    {
      "href": "https://twitter.com/WesKambale",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://linkedin.com/in/WesKambale",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/series/ml-series",
      "label": "Part of series Machine Learning",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/scalable-models",
      "label": "# scalable-models",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/tensorflow",
      "label": "# tensorflow",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/tensorflow-serving",
      "label": "# tensorflow-serving",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:49Z",
  "source_url": "https://kambale.dev/scalable-model-serving-with-tensorflow-serving",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8455,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/5f8a664df4f8/pages/source.html",
  "local_html": "attachments/5f8a664df4f8/pages/article.html",
  "body_base_path": "attachments/5f8a664df4f8/pages/",
  "body_markdown_path": "attachments/5f8a664df4f8/pages/article.md"
}