{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Wesley Kambale",
  "title_listed": "Introduction to TensorFlow Extended (TFX)",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://kambale.dev/tensorflow-extended-tfx",
  "publish_date": "2024-08-06",
  "corpus_stage": "",
  "notes": "",
  "id": "65d8ff4ba1a1",
  "collected_at": "2026-10-02T14:11:22",
  "title": "Introduction to TensorFlow Extended (TFX)",
  "body_markdown": "Introduction to TensorFlow Extended (TFX)\nRunning a TFX Pipeline: From Definition to Deployment\nPublished\nAugust 6, 2024\n•\n8\nmin read\n•\nView as Markdown\nW\nWesley Kambale\nI'm a Machine Learning Engineer passionate about building production-ready ML systems for the African market.\nWith experience in TensorFlow, Keras, and Python-based workflows, I help teams bridge the gap between machine learning research and real-world deployment—especially on resource-constrained devices.\nI'm also a Google Developer Expert in AI. I regularly speak at tech conferences including PyCon Africa, DevFest Kampala, DevFest Nairobi and more and also write technical articles on AI/ML here.\nPart of series\nMachine Learning\nTensorFlow Extended (TFX) is an end-to-end platform for deploying production machine learning (ML) pipelines. TFX allows data scientists and ML engineers to build, evaluate, and deploy ML models in a scalable, reliable, and reproducible manner. This article will introduce you to the core components of TFX, provide practical examples using the Iris dataset, and guide you through building a simple TFX pipeline.\nWhat is TFX?\nTFX is a production-ready ML platform designed to help you build, deploy, and manage ML models. It consists of a set of libraries and tools that help automate and manage the ML lifecycle. TFX pipelines are portable and can run on various platforms, including Apache Beam, Apache Airflow, and Kubeflow.\nKey Features of TFX\nScalability\n: TFX can handle large-scale data processing and training.\nPortability\n: Pipelines can run on different platforms and environments.\nModularity\n: TFX components are designed to be modular, allowing you to customize and extend them as needed.\nProduction-Ready\n: TFX is built with production deployment in mind, ensuring reliability and robustness.\nTFX Components\nTFX pipelines are composed of several components, each responsible for a specific part of the ML lifecycle. Here are the core components:\nExampleGen\nExampleGen is the first component in a TFX pipeline. It ingests and splits data into training and evaluation datasets. ExampleGen supports various data sources, such as CSV files, TFRecord files, and BigQuery.\nfrom\ntfx.components\nimport\nCsvExampleGen\n\nexample_gen = CsvExampleGen(input_base=\n'/content/iris.csv'\n)\nStatisticsGen\nStatisticsGen computes statistics over the data for data visualization and validation. It generates statistics using TensorFlow Data Validation (TFDV).\nfrom\ntfx.components\nimport\nStatisticsGen\n\nstatistics_gen = StatisticsGen(examples=example_gen.outputs[\n'examples'\n])\nSchemaGen\nSchemaGen generates a schema for the data based on the statistics computed by StatisticsGen. The schema includes information about the data types, domains, and constraints.\nfrom\ntfx.components\nimport\nSchemaGen\n\nschema_gen = SchemaGen(statistics=statistics_gen.outputs[\n'statistics'\n])\nExampleValidator\nExampleValidator detects anomalies in the data by comparing the data against the schema generated by SchemaGen. It helps ensure data quality.\nfrom\ntfx.components\nimport\nExampleValidator\n\nexample_validator = ExampleValidator(statistics=statistics_gen.outputs[\n'statistics'\n], schema=schema_gen.outputs[\n'schema'\n])\nTransform\nTransform performs feature engineering and data transformation using TensorFlow Transform (TFT). It preprocesses the data for model training and serving.\nfrom\ntfx.components\nimport\nTransform\ndef\npreprocessing_fn\n(\ninputs\n):\noutputs = {\n'sepal_length'\n: Transform.scale_to_z_score(inputs[\n'sepal.length'\n]),\n'sepal_width'\n: Transform.scale_to_z_score(inputs[\n'sepal.width'\n]),\n'petal_length'\n: Transform.scale_to_z_score(inputs[\n'petal.ength'\n]),\n'petal_width'\n: Transform.scale_to_z_score(inputs[\n'petal.width'\n]),\n'species'\n: inputs[\n'variety'\n]\n    }\nreturn\noutputs\n\ntransform = Transform(\n    examples=example_gen.outputs[\n'examples'\n],\n    schema=schema_gen.outputs[\n'schema'\n],\n    module_file=\n'/content/preprocessing.py'\n)\nTrainer\nTrainer trains an ML model using the preprocessed data. It supports various frameworks, including TensorFlow, Keras, and Estimator.\nfrom\ntfx.components\nimport\nTrainer\nfrom\ntfx.proto\nimport\ntrainer_pb2\n\ntrainer = Trainer(\n    module_file=\n'/content/trainer_module.py'\n,\n    examples=transform.outputs[\n'transformed_examples'\n],\n    schema=schema_gen.outputs[\n'schema'\n],\n    transform_graph=transform.outputs[\n'transform_graph'\n],\n    train_args=trainer_pb2.TrainArgs(num_steps=\n1000\n),\n    eval_args=trainer_pb2.EvalArgs(num_steps=\n100\n)\n)\nEvaluator\nEvaluator evaluates the trained model using TensorFlow Model Analysis (TFMA). It helps in validating and comparing different models.\nimport\ntensorflow_model_analysis\nas\ntfma\n\neval_config = tfma.EvalConfig(\n    slicing_specs=[tfma.SlicingSpec()],\n    metrics_specs=[\n        tfma.MetricsSpec(\n            metrics=[\n                tfma.MetricConfig(class_name=\n'SparseCategoricalAccuracy'\n)\n            ]\n        )\n    ]\n)\n\nevaluator = Evaluator(\n    examples=example_gen.outputs[\n'examples'\n],\n    model=trainer.outputs[\n'model'\n],\n    eval_config=eval_config\n)\nPusher\nPusher deploys the trained model to a serving infrastructure. It ensures that the model meets certain criteria before pushing it to production.\nfrom\ntfx.components\nimport\nPusher\nfrom\ntfx.proto\nimport\npusher_pb2\n\npusher = Pusher(\n    model=trainer.outputs[\n'model'\n],\n    model_blessing=evaluator.outputs[\n'blessing'\n],\n    push_destination=pusher_pb2.PushDestination(\n        filesystem=pusher_pb2.PushDestination.Filesystem(\n            base_directory=\n'/content/model'\n)\n    )\n)\nSetting Up TFX\nBefore building a TFX pipeline, it's essential to set up the environment. This involves installing the necessary packages and configuring the runtime environment.\nInstalling TFX\n: TFX can be installed via pip. The installation includes all the required libraries and dependencies for running TFX components.\npip install tfx\nConfiguring the Environment\n: Setting up the environment involves configuring paths for data, pipelines, and model artifacts. This ensures that all components can access the necessary resources and save outputs in the correct locations.\nBuilding a Simple TFX Pipeline\nTo illustrate the capabilities of TFX, we will build a pipeline using the Iris dataset, a well-known dataset for classification tasks. The Iris dataset contains 150 samples of iris flowers, each with four features (sepal length, sepal width, petal length, petal width) and a class label (species).\nData Ingestion\nThe first step in the TFX pipeline is to ingest the Iris dataset using\nExampleGen\n. This component reads the dataset, splits it into training and evaluation sets, and converts it into the TFX internal format.\nCSV ExampleGen\n: For the Iris dataset, we use the\nCsvExampleGen\ncomponent, which ingests data from CSV files. It automatically splits the data into training and evaluation sets based on a specified ratio.\nfrom\ntfx.components\nimport\nCsvExampleGen\n\nexample_gen = CsvExampleGen(input_base=\n'/content/iris.csv'\n)\nData Statistics\nStatisticsGen\ncomputes descriptive statistics for the dataset, providing insights into data distributions and detecting anomalies. It uses TensorFlow Data Validation (TFDV) to generate statistics such as mean, median, and standard deviation for each feature.\nfrom\ntfx.components\nimport\nStatisticsGen\n\nstatistics_gen = StatisticsGen(examples=example_gen.outputs[\n'examples'\n])\nImportance\n: Understanding the distribution of data is crucial for identifying potential issues and ensuring that the data is suitable for training. StatisticsGen helps detect anomalies such as outliers and missing values, which can affect model performance.\nSchema Generation\nBased on the statistics computed by StatisticsGen, SchemaGen generates a schema for the dataset. The schema includes information about feature types, value ranges, and presence constraints, serving as a blueprint for data validation and transformation.\nfrom\ntfx.components\nimport\nSchemaGen\n\nschema_gen = SchemaGen(statistics=statistics_gen.outputs[\n'statistics'\n])\nDefinition\n: The schema defines the expected structure of the data, including feature types (numeric, categorical, etc.), value ranges, and constraints (e.g., required features). This information is critical for ensuring data consistency and preparing it for model training.\nData Validation\nExampleValidator\nvalidates the dataset against the schema, identifying anomalies and missing values. It ensures that the data adheres to the expected format, which is essential for training reliable models.\nfrom\ntfx.components\nimport\nExampleValidator\n\nexample_validator = ExampleValidator(statistics=statistics_gen.outputs[\n'statistics'\n], schema=schema_gen.outputs[\n'schema'\n])\nAnomaly Detection\n: ExampleValidator detects anomalies such as outliers, missing values, and unexpected feature values. These issues can affect model performance and lead to unreliable predictions, making data validation a crucial step in the pipeline.\nData Transformation\nTransform\nperforms feature engineering and data preprocessing using TensorFlow Transform (TFT). It applies transformations such as scaling, normalization, and encoding, preparing the data for model training.\nfrom\ntfx.components\nimport\nTransform\ndef\npreprocessing_fn\n(\ninputs\n):\n# Normalize the numeric features\noutputs = {\n'sepal_length'\n: Transform.scale_to_z_score(inputs[\n'sepal.length'\n]),\n'sepal_width'\n: Transform.scale_to_z_score(inputs[\n'sepal.width'\n]),\n'petal_length'\n: Transform.scale_to_z_score(inputs[\n'petal.ength'\n]),\n'petal_width'\n: Transform.scale_to_z_score(inputs[\n'petal.width'\n]),\n'species'\n: inputs[\n'variety'\n]\n    }\nreturn\noutputs\n\ntransform = Transform(\n    examples=example_gen.outputs[\n'examples'\n],\n    schema=schema_gen.outputs[\n'schema'\n],\n    module_file=\n'/content/preprocessing.py'\n)\nPreprocessing Functions\n: Transform defines preprocessing functions that apply transformations to the raw data. These functions can include operations such as scaling numerical features, encoding categorical features, and generating new features based on existing ones.\nModel Training\nThe\nTrainer\ncomponent trains the ML model using the transformed data. It leverages TensorFlow's capabilities to define, train, and evaluate models.\nimport\ntensorflow\nas\ntf\nfrom\ntfx\nimport\nv1\nas\ntfxio\ndef\n_input_fn\n(\nfile_pattern, data_accessor, schema, batch_size=\n200\n):\nraw_data = data_accessor.tf_dataset_factory(\n        file_pattern, tfxio.TensorFlowDatasetOptions(batch_size=batch_size), schema)\n    transformed_data = raw_data.map(_parse_fn)\nreturn\ntransformed_data\ndef\n_build_keras_model\n():\nmodel = tf.keras.Sequential([\n        tf.keras.layers.InputLayer(input_shape=(\n4\n,)),\n        tf.keras.layers.Dense(\n10\n, activation=\n'relu'\n),\n        tf.keras.layers.Dense(\n10\n, activation=\n'relu'\n),\n        tf.keras.layers.Dense(\n3\n, activation=\n'softmax'\n)\n    ])\n    model.compile(optimizer=tf.keras.optimizers.Adam(lr=\n0.001\n),\n                  loss=\n'sparse_categorical_crossentropy'\n,\n                  metrics=[\n'accuracy'\n])\nreturn\nmodel\ndef\nrun_fn\n(\nfn_args\n):\nschema = tfx_bsl.public.tfxio_utils.get_tfx_schema_from_tensorflow_metadata_schema(\n        fn_args.schema)\n    train_dataset = _input_fn(fn_args.train_files, fn_args.data_accessor, schema, batch_size=\n200\n)\n    eval_dataset = _input_fn(fn_args.eval_files, fn_args.data_accessor, schema, batch_size=\n200\n)\n\n    model = _build_keras_model()\n    model.fit(train_dataset, steps_per_epoch=fn_args.train_steps,\n              validation_data=eval_dataset, validation_steps=fn_args.eval_steps)\n    model.save(fn_args.serving_model_dir, save_format=\n'tf'\n)\nModel Definition\n: Trainer uses a module file containing the model definition and training logic. This file defines the architecture of the model, the loss function, and the optimization algorithm. It also includes the training and evaluation steps, specifying the number of epochs, batch size, and evaluation metrics.\nTrainer\nTrainer trains an ML model using the preprocessed data. It supports various frameworks, including TensorFlow, Keras, and Estimator\nfrom\ntfx.components\nimport\nTrainer\nfrom\ntfx.proto\nimport\ntrainer_pb2\n\ntrainer = Trainer(\n    module_file=\n'/content/trainer_module.py'\n,\n    examples=transform.outputs[\n'transformed_examples'\n],\n    schema=schema_gen.outputs[\n'schema'\n],\n    transform_graph=transform.outputs[\n'transform_graph'\n],\n    train_args=trainer_pb2.TrainArgs(num_steps=\n1000\n),\n    eval_args=trainer_pb2.EvalArgs(num_steps=\n100\n)\n)\nModel Evaluation\nEvaluator evaluates the trained model using TensorFlow Model Analysis (TFMA). It performs a detailed analysis of model performance, identifying potential issues and ensuring that the model meets the desired criteria before deployment.\nimport\ntensorflow_model_analysis\nas\ntfma\n\neval_config = tfma.EvalConfig(\n    slicing_specs=[tfma.SlicingSpec()],\n    metrics_specs=[\n        tfma.MetricsSpec(\n            metrics=[\n                tfma.MetricConfig(class_name=\n'SparseCategoricalAccuracy'\n)\n            ]\n        )\n    ]\n)\n\nevaluator = Evaluator(\n    examples=example_gen.outputs[\n'examples'\n],\n    model=trainer.outputs[\n'model'\n],\n    eval_config=eval_config\n)\nEvaluation Configuration\n: Evaluator uses an evaluation configuration to specify the metrics and slicing specifications for model evaluation. Metrics such as accuracy, precision, and recall are used to assess model performance, while slicing specifications allow for analyzing performance across different subsets of the data.\nModel Deployment\nThe final step in the TFX pipeline is to deploy the validated model using\nPusher\n. This component ensures that only the best models are deployed to the serving infrastructure.\nfrom\ntfx.components\nimport\nPusher\nfrom\ntfx.proto\nimport\npusher_pb2\n\npusher = Pusher(\n    model=trainer.outputs[\n'model'\n],\n    model_blessing=evaluator.outputs[\n'blessing'\n],\n    push_destination=pusher_pb2.PushDestination(\n        filesystem=pusher_pb2.PushDestination.Filesystem(\n            base_directory=\n'/content/model'\n)\n    )\n)\nModel Deployment\n: Pusher deploys the model to a specified serving infrastructure, such as TensorFlow Serving. It ensures that the model is production-ready and meets the desired performance criteria, facilitating continuous model improvement.\nRunning the Pipeline\nThe pipeline is executed using an orchestrator like the Local DAG Runner, which processes the data, trains the model, evaluates its performance, and deploys it if it meets the required criteria.\nfrom\ntfx.orchestration\nimport\npipeline\nfrom\ntfx.orchestration.local.local_dag_runner\nimport\nLocalDagRunner\n# Define the pipeline\npipeline = pipeline.Pipeline(\n    pipeline_name=\n'iris_pipeline'\n,\n    pipeline_root=\n'/content/iris_pipeline'\n,\n    components=[example_gen, statistics_gen, schema_gen, example_validator, transform, trainer, evaluator, pusher],\n    enable_cache=\nTrue\n,\n    metadata_connection_config=\nNone\n)\n# Run the pipeline\nLocalDagRunner().run(pipeline)\nAfter execution, the pipeline produces outputs such as transformed data, trained models, evaluation results, and metadata, all stored in a specified directory for further analysis and deployment.\nMonitoring involves checking logs and reviewing evaluation metrics, while debugging includes inspecting artifacts and re-executing specific components to resolve any issues. This process ensures the pipeline runs smoothly, producing reliable and scalable machine learning models ready for deployment.\nConclusion\nIn this article, we introduced TensorFlow Extended (TFX) and its core components. We demonstrated how to set up a TFX environment, build a simple TFX pipeline using the Iris dataset, and run it. TFX provides a powerful and flexible framework for managing the end-to-end ML lifecycle, from data ingestion to model deployment. You should now have a solid foundation for building and deploying your own TFX pipelines.\nTFX's modularity and scalability make it suitable for a wide range of ML applications, ensuring that you can build robust and production-ready ML systems. Happy experimenting with TFX!\nResources\nGitHub Gist to the notebook\nhttps://gist.github.com/wkambale/ce26c4ab2a7f040372007d422ae0fbe2\n#\ntensorflow\n#\ntfx\n#\npipeline",
  "metadata": {
    "title": "Introduction to TensorFlow Extended (TFX)",
    "description": "Running a TFX Pipeline: From Definition to Deployment",
    "keywords": null,
    "author": null,
    "og:title": "Introduction to TensorFlow Extended (TFX)",
    "og:description": "Running a TFX Pipeline: From Definition to Deployment",
    "og:url": "https://kambale.dev/tensorflow-extended-tfx",
    "og:site_name": "Wesley Kambale",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1722902885939/c1d9f5b4-cbca-4a9d-b82e-734acae61f4c.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://twitter.com/WesKambale",
    "twitter:title": "Introduction to TensorFlow Extended (TFX)",
    "twitter:description": "Running a TFX Pipeline: From Definition to Deployment",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1722902885939/c1d9f5b4-cbca-4a9d-b82e-734acae61f4c.png",
    "article:published_time": "2024-08-06T00:11:38.137Z",
    "article:author": "https://kambale.dev/@kambale"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/65d8ff4ba1a1/code/body-000.txt",
      "chars": 102,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-001.txt",
      "chars": 115,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-002.txt",
      "chars": 110,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-003.txt",
      "chars": 168,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-004.txt",
      "chars": 602,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-005.txt",
      "chars": 396,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-006.txt",
      "chars": 423,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-007.txt",
      "chars": 340,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-008.txt",
      "chars": 16,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-009.txt",
      "chars": 102,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-010.txt",
      "chars": 115,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-011.txt",
      "chars": 110,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-012.txt",
      "chars": 168,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-013.txt",
      "chars": 639,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-014.txt",
      "chars": 1363,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-015.txt",
      "chars": 397,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-016.txt",
      "chars": 423,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-017.txt",
      "chars": 340,
      "language": "text"
    },
    {
      "file": "attachments/65d8ff4ba1a1/code/body-018.txt",
      "chars": 467,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1722774005457/afdce8da-141e-4909-8eca-3ab92381f512.png",
      "bytes": 49788,
      "reused": true,
      "local_path": "attachments/65d8ff4ba1a1/images/000.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1722774005457/afdce8da-141e-4909-8eca-3ab92381f512.png",
      "alt": "Introduction to TensorFlow Extended (TFX)",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://kambale.dev/tensorflow-extended-tfx.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@kambale",
      "label": "W",
      "type": "a"
    },
    {
      "href": "https://twitter.com/WesKambale",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://linkedin.com/in/WesKambale",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/series/ml-series",
      "label": "Part of series Machine Learning",
      "type": "a"
    },
    {
      "href": "https://gist.github.com/wkambale/ce26c4ab2a7f040372007d422ae0fbe2",
      "label": "https://gist.github.com/wkambale/ce26c4ab2a7f040372007d422ae0fbe2",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/tensorflow",
      "label": "# tensorflow",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/tfx",
      "label": "# tfx",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/pipeline",
      "label": "# pipeline",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:49Z",
  "source_url": "https://kambale.dev/tensorflow-extended-tfx",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 16162,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/65d8ff4ba1a1/pages/source.html",
  "local_html": "attachments/65d8ff4ba1a1/pages/article.html",
  "body_base_path": "attachments/65d8ff4ba1a1/pages/",
  "body_markdown_path": "attachments/65d8ff4ba1a1/pages/article.md"
}