{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "How to Test Machine Learning Code and Systems",
  "keywords": "测试,ML系统,质量",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/testing-ml/",
  "publish_date": "2020-11",
  "corpus_stage": "推理部署",
  "notes": "机器学习代码与系统测试方法",
  "id": "9b1965a26aac",
  "collected_at": "2026-10-02T14:17:02",
  "title": "How to Test Machine Learning Code and Systems",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nHow to Test Machine Learning Code and Systems\n[\nmachinelearning\nengineering\npython\n]\n· 14 min read\nTwo weeks ago,\nJeremy\nwrote a great post on\nEffective Testing for Machine Learning Systems\n. He distinguished between traditional software tests and machine learning (ML) tests; software tests check the\nwritten logic\nwhile ML tests check the\nlearned logic\n.\nML tests can be further split into\ntesting\nand\nevaluation\n. We’re familiar with ML\nevaluation\nwhere we train a model and evaluate its performance on an unseen validation set; this is done via metrics (e.g., accuracy,\nArea under Curve of Receiver Operating Characteristic (AUC ROC)\n) and visuals (e.g.,\nprecision-recall curve\n).\nOn the other hand, ML\ntesting\ninvolves checks on model behaviour.\nPre-train tests\n—which can be run without trained parameters—check if our\nwritten logic\nis correct. For example, is classification probability between 0 to 1?\nPost-train tests\ncheck if the\nlearned logic\nis expected. For example, on the\nTitanic dataset\n, we should expect females to have a higher survival probability (relative to males).\nWorkflow for testing machine learning (\nsource\n)\nTaken together, here’s how the workflow might look like. To complement this, we’ll implement a machine learning model and run the following tests on it:\nPre-train tests to ensure correct implementation\nPost-train tests to ensure expected learned behaviour\nEvaluation to ensure satisfactory model performance\nFollow along with the code in this Github repository:\ntesting-ml\nSetting up the context (algorithm and data)\nBefore we can do ML testing, we’ll need an\nalgorithm and some data\n. Our algorithm will be a\nnumpy\nimplementation of\nDecisionTree\nwhich predicts a probability for binary classification. (\nExtensions to make it support regression welcome!\n).\nTo run our tests, we’ll use the\nTitanic dataset\n. This tiny data set (~900 rows, 10 features) makes for fast testing (when model training is involved) and allows us to iterate quickly. (As part of performance evaluation, we run\nfit()\nand\npredict()\nhundreds of times to get the 99th percentile timing.) The simplicity and familiarity of the data also makes it easier to discuss the post-train (i.e., learned logic) tests.\n+ ------------+----------+--------+-----------------------------------------+--------+-----+-------+-------+-----------+---------+-------+----------+\n| PassengerId | Survived | Pclass | Name                                    | Sex    | Age | SibSp | Parch | Ticket    |    Fare | Cabin | Embarked |\n+ ------------+----------+--------+-----------------------------------------+--------+-----+-------+-------+-----------+---------+-------+----------|\n|           1 |        0 |      3 | Braund, Mr. Owen Harris                 | male   |  22 |     1 |     0 | A/5 21171 |    7.25 | nan   | S        |\n|           2 |        1 |      1 | Cumings, Mrs. John Bradley (Florence... | female |  38 |     1 |     0 | PC 17599  | 71.2833 | C85   | C        |\n|           3 |        1 |      3 | Heikkinen, Miss. Laina                  | female |  26 |     0 |     0 | STON/O2.  |   7.925 | nan   | S        |\n|           4 |        1 |      1 | Futrelle, Mrs. Jacques Heath (Lily M... | female |  35 |     1 |     0 | 113803    |    53.1 | C123  | S        |\n|           5 |        0 |      3 | Allen, Mr. William Henry                | male   |  35 |     0 |     0 | 373450    |    8.05 | nan   | S        |\n|           6 |        0 |      3 | Moran, Mr. James                        | male   | nan |     0 |     0 | 330877    |  8.4583 | nan   | Q        |\n|           7 |        0 |      1 | McCarthy, Mr. Timothy J                 | male   |  54 |     0 |     0 | 17463     | 51.8625 | E46   | S        |\n|           8 |        0 |      3 | Palsson, Master. Gosta Leonard          | male   |   2 |     3 |     1 | 349909    |  21.075 | nan   | S        |\n|           9 |        1 |      3 | Johnson, Mrs. Oscar W (Elisabeth Vil... | female |  27 |     0 |     2 | 347742    | 11.1333 | nan   | S        |\n|          10 |        1 |      2 | Nasser, Mrs. Nicholas (Adele Achem)     | female |  14 |     1 |     0 | 237736    | 30.0708 | nan   | C        |\n+ ------------+----------+--------+-----------------------------------------+--------+-----+-------+-------+-----------+---------+-------+----------+\nIf you're unfamiliar with the Titanic dataset, here's how it looks like (scroll to the right).\nAdopting testing habits from software engineering\nWe’ll adopt some good habits from software engineering. We won’t go through them in detail here though it was previously covered in another\npost\n:\nUnit test\nfixture reuse, exceptions testing, etc with\npytest\nCode coverage\nwith\nCoverage.py\nand\npytest-cov\nLinting\nto ensure code consistency with\npylint\nType checking\nto verify type correctness with\nmypy\n(Note: The tests below won’t include type hints though the\nimplementation code\ndoes.)\nPre-train tests to ensure correct implementation\nIn pre-train tests, we want to\ncatch errors in our implementation\n(i.e., written logic) before training an erroneous model. We can run these tests without a fully trained model.\nFirst, we’ll test our functions of\nGini impurity\nand\nGini gain\n. These will be used to\nsplit the data\nand grow our decision tree.\ndef\ntest_gini_impurity\n():\nassert\nround\n(\ngini_impurity\n([\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n1\n]),\n3\n)\n==\n0\nassert\nround\n(\ngini_impurity\n([\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n0\n]),\n3\n)\n==\n0.219\nassert\nround\n(\ngini_impurity\n([\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n]),\n3\n)\n==\n0.375\nassert\nround\n(\ngini_impurity\n([\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n]),\n3\n)\n==\n0.469\nassert\nround\n(\ngini_impurity\n([\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n]),\n3\n)\n==\n0.500\ndef\ntest_gini_gain\n():\nassert\nround\n(\ngini_gain\n([\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n],\n[[\n1\n,\n1\n,\n1\n,\n1\n],\n[\n0\n,\n0\n,\n0\n,\n0\n]]),\n3\n)\n==\n0.5\nassert\nround\n(\ngini_gain\n([\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n],\n[[\n1\n,\n1\n,\n1\n,\n0\n],\n[\n0\n,\n0\n,\n0\n,\n1\n]]),\n3\n)\n==\n0.125\nassert\nround\n(\ngini_gain\n([\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n],\n[[\n1\n,\n0\n,\n0\n,\n0\n],\n[\n0\n,\n1\n,\n1\n,\n1\n]]),\n3\n)\n==\n0.125\nassert\nround\n(\ngini_gain\n([\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n],\n[[\n1\n,\n1\n,\n0\n,\n0\n],\n[\n0\n,\n0\n,\n1\n,\n1\n]]),\n3\n)\n==\n0.0\nNext, we’ll check if the model prediction shape is expected. We should have the same number of rows as the input features.\ndef\ntest_dt_output_shape\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\ndt\n=\nDecisionTree\n()\ndt\n.\nfit\n(\nX_train\n,\ny_train\n)\npred_train\n=\ndt\n.\npredict\n(\nX_train\n)\npred_test\n=\ndt\n.\npredict\n(\nX_test\n)\nassert\npred_train\n.\nshape\n==\n(\nX_train\n.\nshape\n[\n0\n],),\n'DecisionTree output should be same as training labels.'\nassert\npred_test\n.\nshape\n==\n(\nX_test\n.\nshape\n[\n0\n],),\n'DecisionTree output should be same as testing labels.'\nWe’ll also want to check the output ranges. Given that we’re predicting probabilities, we should expect the output to range from 0 to 1 inclusive.\ndef\ntest_dt_output_range\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\ndt\n=\nDecisionTree\n()\ndt\n.\nfit\n(\nX_train\n,\ny_train\n)\npred_train\n=\ndt\n.\npredict\n(\nX_train\n)\npred_test\n=\ndt\n.\npredict\n(\nX_test\n)\nassert\n(\npred_train\n<=\n1\n).\nall\n()\n&\n(\npred_train\n>=\n0\n).\nall\n(),\n'Decision tree output should range from 0 to 1 inclusive'\nassert\n(\npred_test\n<=\n1\n).\nall\n()\n&\n(\npred_test\n>=\n0\n).\nall\n(),\n'Decision tree output should range from 0 to 1 inclusive'\nHere, we’ll check for test set leakage (i.e., duplicate rows in train/test splits) by concatenating train and test data, dropping duplicates, and checking the number of rows. (Note: Other leakages include\ntemporal leaks\nand\nfeature leaks\n; we won’t cover them here.)\ndef\ntest_data_leak_in_test_data\n(\ndummy_titanic_df\n):\ntrain\n,\ntest\n=\ndummy_titanic_df\nconcat_df\n=\npd\n.\nconcat\n([\ntrain\n,\ntest\n])\nconcat_df\n.\ndrop_duplicates\n(\ninplace\n=\nTrue\n)\nassert\nconcat_df\n.\nshape\n[\n0\n]\n==\ntrain\n.\nshape\n[\n0\n]\n+\ntest\n.\nshape\n[\n0\n]\nGiven perfectly separable data and unlimited depth, our decision tree should be able to “memorise” the training data and\noverfit\ncompletely\n. In other words, if we train\nand\nevaluate on the training data, we should get 100% accuracy. (Note: the Titanic data isn’t perfectly separable so we’ll create a small data sample for this.)\n@\npytest\n.\nfixture\ndef\ndummy_feats_and_labels\n():\nfeats\n=\nnp\n.\narray\n([[\n0.7057\n,\n-\n5.4981\n,\n8.3368\n,\n-\n2.8715\n],\n[\n2.4391\n,\n6.4417\n,\n-\n0.80743\n,\n-\n0.69139\n],\n[\n-\n0.2062\n,\n9.2207\n,\n-\n3.7044\n,\n-\n6.8103\n],\n[\n4.2586\n,\n11.2962\n,\n-\n4.0943\n,\n-\n4.3457\n],\n[\n-\n2.343\n,\n12.9516\n,\n3.3285\n,\n-\n5.9426\n],\n[\n-\n2.0545\n,\n-\n10.8679\n,\n9.4926\n,\n-\n1.4116\n],\n[\n2.2279\n,\n4.0951\n,\n-\n4.8037\n,\n-\n2.1112\n],\n[\n-\n6.1632\n,\n8.7096\n,\n-\n0.21621\n,\n-\n3.6345\n],\n[\n0.52374\n,\n3.644\n,\n-\n4.0746\n,\n-\n1.9909\n],\n[\n1.5077\n,\n1.9596\n,\n-\n3.0584\n,\n-\n0.12243\n]\n])\nlabels\n=\nnp\n.\narray\n([\n1\n,\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n])\nreturn\nfeats\n,\nlabels\ndef\ntest_dt_overfit\n(\ndummy_feats_and_labels\n):\nfeats\n,\nlabels\n=\ndummy_feats_and_labels\ndt\n=\nDecisionTree\n()\ndt\n.\nfit\n(\nfeats\n,\nlabels\n)\npred\n=\nnp\n.\nround\n(\ndt\n.\npredict\n(\nfeats\n))\nassert\nnp\n.\narray_equal\n(\nlabels\n,\npred\n),\n'DecisionTree should fit data perfectly and prediction should == labels.'\nLastly, we check if increasing tree depth leads to increased accuracy and AUC ROC on\ntraining\ndata (though it’ll overfit and perform poorly on\nvalidation\ndata). In the test below, we fit trees of depth one to 10 and ensure training accuracy and AUC increases consistently.\ndef\ntest_dt_increase_acc\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\n_\n,\n_\n=\ndummy_titanic\nacc_list\n=\n[]\nauc_list\n=\n[]\nfor\ndepth\nin\nrange\n(\n1\n,\n10\n):\ndt\n=\nDecisionTree\n(\ndepth_limit\n=\ndepth\n)\ndt\n.\nfit\n(\nX_train\n,\ny_train\n)\npred\n=\ndt\n.\npredict\n(\nX_train\n)\npred_binary\n=\nnp\n.\nround\n(\npred\n)\nacc_list\n.\nappend\n(\naccuracy_score\n(\ny_train\n,\npred_binary\n))\nauc_list\n.\nappend\n(\nroc_auc_score\n(\ny_train\n,\npred\n))\nassert\nsorted\n(\nacc_list\n)\n==\nacc_list\n,\n'Accuracy should increase as tree depth increases.'\nassert\nsorted\n(\nauc_list\n)\n==\nauc_list\n,\n'AUC ROC should increase as tree depth increases.'\nPost-train tests to ensure expected learned behaviour\nIn post-train tests, we want to\ncheck if the model is behaving correctly\n. Thus, we test our trained models for their\nlearned logic\n. I’m unaware of any frameworks for doing this so here’s my take on it (suggestions welcome!). I took inspiration from the excellent paper\nBeyond Accuracy: Behavioral Testing of NLP Models with CheckList\n.\nFirst, we check for\ninvariance\n. Keeping everything constant and tweaking one feature at a time, how would model predictions change? We should not expect survival probability to change due to the passenger’s name, ticket number, or port of embarkation.\n@\npytest\n.\nfixture\ndef\ndummy_passengers\n():\n# Based on passenger 2 (high passenger class female)\npassenger2\n=\n{\n'PassengerId'\n:\n2\n,\n'Pclass'\n:\n1\n,\n'Name'\n:\n' Mrs. John'\n,\n'Sex'\n:\n'female'\n,\n'Age'\n:\n38.0\n,\n'SibSp'\n:\n1\n,\n'Parch'\n:\n0\n,\n'Ticket'\n:\n'PC 17599'\n,\n'Fare'\n:\n71.2833\n,\n'Embarked'\n:\n'C'\n}\nreturn\npassenger2\ndef\ntest_dt_invariance\n(\ndummy_titanic_dt\n,\ndummy_passengers\n):\nmodel\n=\ndummy_titanic_dt\n_\n,\np2\n=\ndummy_passengers\n# Get original survival probability of passenger 2\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 1.0\n# Change name from John to Berns (without changing gender or title)\np2_name\n=\np2\n.\ncopy\n()\np2_name\n[\n'Name'\n]\n=\n' Mrs. Berns'\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2_name\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_name_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 1.0\n# Change ticket number from 'PC 17599' to 'A/5 21171'\np2_ticket\n=\np2\n.\ncopy\n()\np2_ticket\n[\n'ticket'\n]\n=\n'A/5 21171'\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2_ticket\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_ticket_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 1.0\nassert\np2_prob\n==\np2_name_prob\n==\np2_ticket_prob\nOn the other hand, we have\ndirectional expectations\nsuch as:\nFemales having higher survival probability (than males)\nHigher passenger class having higher survival probability (than lower classes)\nHigher fare having higher survival probability (than lower fare)\ndef\ntest_dt_directional_expectation\n(\ndummy_titanic_dt\n,\ndummy_passengers\n):\nmodel\n=\ndummy_titanic_dt\n_\n,\np2\n=\ndummy_passengers\n# Get original survival probability of passenger 2\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 1.0\n# Change gender from female to male\np2_male\n=\np2\n.\ncopy\n()\np2_male\n[\n'Name'\n]\n=\n' Mr. John'\np2_male\n[\n'Sex'\n]\n=\n'male'\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2_male\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_male_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 0.56\n# Change class from 1 to 3\np2_class\n=\np2\n.\ncopy\n()\np2_class\n[\n'Pclass'\n]\n=\n3\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2_class\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_class_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 0.0\n# Lower fare from 71.2833 to 5\np2_fare\n=\np2\n.\ncopy\n()\np2_fare\n[\n'Fare'\n]\n=\n5\ntest_df\n=\npd\n.\nDataFrame\n.\nfrom_dict\n([\np2_fare\n],\norient\n=\n'columns'\n)\nX\n,\ny\n=\nget_feats_and_labels\n(\nprep_df\n(\ntest_df\n))\np2_fare_prob\n=\nmodel\n.\npredict\n(\nX\n)[\n0\n]\n# 0.85\nassert\np2_prob\n>\np2_male_prob\n,\n'Changing gender from female to male should decrease survival probability.'\nassert\np2_prob\n>\np2_class_prob\n,\n'Changing class from 1 to 3 should decrease survival probability.'\nassert\np2_prob\n>\np2_fare_prob\n,\n'Changing fare from 72 to 5 should decrease survival probability.'\nModel evaluation to ensure satisfactory performance\nLast, we\nevaluate\nour model to ensure performance does not degrade, whether due to new data or updated implementation. Here, we assess model performance on\naccuracy\nand\nAUC\n.\ndef\ntest_dt_evaluation\n(\ndummy_titanic_dt\n,\ndummy_titanic\n):\nmodel\n=\ndummy_titanic_dt\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\npred_test\n=\nmodel\n.\npredict\n(\nX_test\n)\npred_test_binary\n=\nnp\n.\nround\n(\npred_test\n)\nacc_test\n=\naccuracy_score\n(\ny_test\n,\npred_test_binary\n)\nauc_test\n=\nroc_auc_score\n(\ny_test\n,\npred_test\n)\nassert\nacc_test\n>\n0.82\n,\n'Accuracy on test should be > 0.82'\nassert\nauc_test\n>\n0.84\n,\n'AUC ROC on test should be > 0.84'\nAs part of evaluation, we’ll also assess\ntraining and inference times\n. We should not expect algorithm updates to change these times greatly. Consumers of our predictions might have strict latency requirements. Here, we call\nfit()\nand\npredict()\nmultiple times and check the 99th percentile.\ndef\ntest_dt_training_time\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\n# Standardize to use depth = 10\ndt\n=\nDecisionTree\n(\ndepth_limit\n=\n10\n)\nlatency_array\n=\nnp\n.\narray\n([\ntrain_with_time\n(\ndt\n,\nX_train\n,\ny_train\n)[\n1\n]\nfor\ni\nin\nrange\n(\n100\n)])\ntime_p95\n=\nnp\n.\nquantile\n(\nlatency_array\n,\n0.95\n)\nassert\ntime_p95\n<\n1.0\n,\n'Training time at 95th percentile should be < 1.0 sec'\ndef\ntest_dt_serving_latency\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\n# Standardize to use depth = 10\ndt\n=\nDecisionTree\n(\ndepth_limit\n=\n10\n)\ndt\n.\nfit\n(\nX_train\n,\ny_train\n)\nlatency_array\n=\nnp\n.\narray\n([\npredict_with_time\n(\ndt\n,\nX_test\n)[\n1\n]\nfor\ni\nin\nrange\n(\n500\n)])\nlatency_p99\n=\nnp\n.\nquantile\n(\nlatency_array\n,\n0.99\n)\nassert\nlatency_p99\n<\n0.004\n,\n'Serving latency at 99th percentile should be < 0.004 sec'\nModel testing vs. evaluation; similar to white box vs black box?\nAs I write this, I’m seeing parallels with white box and black box testing. Here’s a brief comparison.\nWhite box testing: We test the\nlogic\nof an application with\nfull knowledge of the internals\n(i.e., structure, implementation).\nBlack box testing: We test the\noutcomes and behaviour\nof an application\nwithout knowledge\nabout internal code and implementation.\nHere, our pre-train tests are similar to white box testing. We have tests cases to check our written logic and implementation (e.g., Gini gain, output shape/range, etc.). On the other hand, our post-train tests and model evaluation are akin to black box tests. We\ncheck the model’s learned behaviour\nand\nevaluate performance\n.\nHere’s an\nexcellent comparison of white box vs black box testing\n.\nTesting\nRandomForest\n—does it improve accuracy/AUC?\nNext, we want to improve on\nDecisionTree\nby adding bagging and making it a\nRandomForest\n. We can do this by inheriting from\nDecisionTree\nand overriding the\nfit()\nand\npredict()\nmethods.\nFor pre-train tests, we should make sure that additional trees improve accuracy and AUC. In the example below, we train\nRandomForest\nmodels with 1, 3, 7, and 15 bagged trees and check if accuracy and AUC increases.\ndef\ntest_dt_increase_acc\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\nacc_list\n=\n[]\nauc_list\n=\n[]\nfor\nnum_trees\nin\n[\n1\n,\n3\n,\n7\n,\n15\n]:\nrf\n=\nRandomForest\n(\nnum_trees\n=\nnum_trees\n,\ndepth_limit\n=\n7\n,\ncol_subsampling\n=\n0.7\n,\nrow_subsampling\n=\n0.7\n)\nrf\n.\nfit\n(\nX_train\n,\ny_train\n)\npred\n=\nrf\n.\npredict\n(\nX_test\n)\npred_binary\n=\nnp\n.\nround\n(\npred\n)\nacc_list\n.\nappend\n(\naccuracy_score\n(\ny_test\n,\npred_binary\n))\nauc_list\n.\nappend\n(\nroc_auc_score\n(\ny_test\n,\npred\n))\nassert\nsorted\n(\nacc_list\n)\n==\nacc_list\n,\n'Accuracy should increase as number of trees increases.'\nassert\nsorted\n(\nauc_list\n)\n==\nauc_list\n,\n'AUC ROC should increase as number of trees increases.'\nWe’ll also want to check that, given the same tree depth,\nRandomForest\noutperforms\nDecisionTree\non a validation data set.\ndef\ntest_rf_better_than_dt\n(\ndummy_titanic\n):\nX_train\n,\ny_train\n,\nX_test\n,\ny_test\n=\ndummy_titanic\ndt\n=\nDecisionTree\n(\ndepth_limit\n=\n10\n)\ndt\n.\nfit\n(\nX_train\n,\ny_train\n)\nrf\n=\nRandomForest\n(\ndepth_limit\n=\n10\n,\nnum_trees\n=\n7\n,\ncol_subsampling\n=\n0.8\n,\nrow_subsampling\n=\n0.8\n)\nrf\n.\nfit\n(\nX_train\n,\ny_train\n)\npred_test_dt\n=\ndt\n.\npredict\n(\nX_test\n)\npred_test_binary_dt\n=\nnp\n.\nround\n(\npred_test_dt\n)\nacc_test_dt\n=\naccuracy_score\n(\ny_test\n,\npred_test_binary_dt\n)\nauc_test_dt\n=\nroc_auc_score\n(\ny_test\n,\npred_test_dt\n)\npred_test_rf\n=\nrf\n.\npredict\n(\nX_test\n)\npred_test_binary_rf\n=\nnp\n.\nround\n(\npred_test_rf\n)\nacc_test_rf\n=\naccuracy_score\n(\ny_test\n,\npred_test_binary_rf\n)\nauc_test_rf\n=\nroc_auc_score\n(\ny_test\n,\npred_test_rf\n)\nassert\nacc_test_rf\n>\nacc_test_dt\n,\n'RandomForest should have higher accuracy than DecisionTree on test set.'\nassert\nauc_test_rf\n>\nauc_test_dt\n,\n'RandomForest should have higher AUC ROC than DecisionTree on test set.'\nSimilar to\nDecisionTree\n, we run\ninvariance\nand\ndirectional expectation\ntests. We also do model evaluation on\naccuracy and AUC\n,\ntraining time\n, and\ninference time\n. (Note: Given a\nRandomForest\nof five trees, we should expect our training and inference time to be ~5x of\nDecisionTree\nas training and inference is not parallelized.)\nTry it for yourself and break something\nAlright enough reading; I urge you to clone the\nGitHub repository\nand try it for yourself. It’s as simple as:\n# Clone and setup environment\ngit clone https://github.com/eugeneyan/testing-ml.git\ncd\ntesting-ml\nmake setup\n# Run test suite\nmake check\nMess around with the implementation and tests. Break something! If you’re feeling adventurous, update\nDecisionTree\nto also do regression. You’ll need to:\nAdd an\n__init__()\nargument for\nclassification\nor\nregression\n. Alternatively, create it a new\nDecisionTree\nclass (via inheritance perhaps?) solely for regression.\nAdd a function to check for\nRMSE\nor\nMSE\nimprovement (instead of\nGini gain\n).\nAdd a test data set (perhaps\nBoston\n?) and basic data prep steps.\nAdd test cases and model evaluation checks.\nDo you know of (i) other frameworks for\nmachine learning tests on learned behaviour\n, or (ii) examples of\nbehaviour failures in machine learning applications\n?\nI would love to hear from you—please leave a comment below. Alternatively, here’s a\nsurvey\non machine learning checks and failures.\nPreviously, Jeremy shared how to test machine learning systems. He differentiated:\n• Testing vs. Evaluation\n• Implementation vs. learned behaviour tests\nTo follow-up, here are example test cases on a NumPy version of DecisionTree & RandomForest\nhttps://t.co/iXqyD2afJz\n— Eugene Yan (@eugeneyan)\nSeptember 9, 2020\nThanks\nto Yang Xinyi,\nJeremy Jordan\n,\nDavid Vargas\n, and\nJoel Christiansen\nfor reading drafts of this.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Sep 2020). How to Test Machine Learning Code and Systems. eugeneyan.com.\n        https://eugeneyan.com/writing/testing-ml/.\nor\n@article{yan2020testing,\n  title   = {How to Test Machine Learning Code and Systems},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2020},\n  month   = {Sep},\n  url     = {https://eugeneyan.com/writing/testing-ml/}\n}\nShare on:\nBrowse related tags:\n[\nmachinelearning\nengineering\npython\n]\nor\nSearch\n« Mailbag: Parsing Fields from PDFs—When to Use Machine Learning?\nMigrating Site Comments to Utterances »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "How to Test Machine Learning Code and Systems",
    "description": "Checking for correct implementation, expected learned behaviour, and satisfactory performance.",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "How to Test Machine Learning Code and Systems",
    "og:type": "article",
    "og:description": "Checking for correct implementation, expected learned behaviour, and satisfactory performance.",
    "og:url": "https://eugeneyan.com/writing/testing-ml/",
    "og:image": "https://eugeneyan.com/assets/og_image/testing-ml.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "How to Test Machine Learning Code and Systems",
    "twitter:description": "Checking for correct implementation, expected learned behaviour, and satisfactory performance.",
    "twitter:image": "https://eugeneyan.com/assets/og_image/testing-ml.jpg",
    "article:published_time": "2020-09-06T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "python"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9b1965a26aac/code/body-000.txt",
      "chars": 2100,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-001.txt",
      "chars": 776,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-002.txt",
      "chars": 435,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-003.txt",
      "chars": 458,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-004.txt",
      "chars": 237,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-005.txt",
      "chars": 1032,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-006.txt",
      "chars": 604,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-007.txt",
      "chars": 1429,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-008.txt",
      "chars": 1441,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-009.txt",
      "chars": 452,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-010.txt",
      "chars": 824,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-011.txt",
      "chars": 685,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-012.txt",
      "chars": 919,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-013.txt",
      "chars": 138,
      "language": "text"
    },
    {
      "file": "attachments/9b1965a26aac/code/body-014.txt",
      "chars": 237,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/testing-ml-flow.webp",
      "bytes": 8506,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-a240873e2d81a818b6e9.webp",
      "src": "https://eugeneyan.com/assets/testing-ml-flow.webp",
      "alt": "Workflow for testing machine learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/9b1965a26aac/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/machinelearning/",
      "label": "machinelearning",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/engineering/",
      "label": "engineering",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/python/",
      "label": "python",
      "type": "a"
    },
    {
      "href": "https://twitter.com/jeremyjordan",
      "label": "Jeremy",
      "type": "a"
    },
    {
      "href": "https://www.jeremyjordan.me/testing-ml/",
      "label": "Effective Testing for Machine Learning Systems",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Receiver_operating_characteristic",
      "label": "Area under Curve of Receiver Operating Characteristic (AUC ROC)",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/recommender-systems-baseline-pytorch/",
      "label": "precision-recall curve",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/titanic/data",
      "label": "Titanic dataset",
      "type": "a"
    },
    {
      "href": "https://github.com/eugeneyan/testing-ml",
      "label": "testing-ml",
      "type": "a"
    },
    {
      "href": "https://numpy.org/",
      "label": "numpy",
      "type": "a"
    },
    {
      "href": "https://github.com/eugeneyan/testing-ml/blob/master/src/tree/decision_tree.py",
      "label": "DecisionTree",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/setting-up-python-project-for-automation-and-collaboration/",
      "label": "post",
      "type": "a"
    },
    {
      "href": "https://docs.pytest.org/en/latest/",
      "label": "pytest",
      "type": "a"
    },
    {
      "href": "https://coverage.readthedocs.io/en/coverage-5.2.1/",
      "label": "Coverage.py",
      "type": "a"
    },
    {
      "href": "https://pytest-cov.readthedocs.io/en/latest/",
      "label": "pytest-cov",
      "type": "a"
    },
    {
      "href": "https://www.pylint.org/",
      "label": "pylint",
      "type": "a"
    },
    {
      "href": "http://mypy-lang.org/",
      "label": "mypy",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Decision_tree_learning",
      "label": "Gini impurity",
      "type": "a"
    },
    {
      "href": "https://www.fast.ai/2017/11/13/validation-sets/",
      "label": "temporal leaks",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Leakage_(machine_learning)",
      "label": "feature leaks",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Overfitting",
      "label": "overfit",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.04118",
      "label": "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList",
      "type": "a"
    },
    {
      "href": "https://www.geeksforgeeks.org/differences-between-black-box-testing-vs-white-box-testing/",
      "label": "excellent comparison of white box vs black box testing",
      "type": "a"
    },
    {
      "href": "https://github.com/eugeneyan/testing-ml/blob/master/src/tree/random_forest.py",
      "label": "RandomForest",
      "type": "a"
    },
    {
      "href": "https://github.com/eugeneyan/testing-ml/blob/master/tests/tree/test_random_forest_2post.py",
      "label": "invariance",
      "type": "a"
    },
    {
      "href": "https://github.com/eugeneyan/testing-ml/blob/master/tests/tree/test_random_forest_3eval.py",
      "label": "accuracy and AUC",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Root-mean-square_deviation",
      "label": "RMSE",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Mean_squared_error",
      "label": "MSE",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html",
      "label": "Boston",
      "type": "a"
    },
    {
      "href": "https://forms.gle/gDdQPTMiQNhyiN1N9",
      "label": "survey",
      "type": "a"
    },
    {
      "href": "https://t.co/iXqyD2afJz",
      "label": "https://t.co/iXqyD2afJz",
      "type": "a"
    },
    {
      "href": "https://twitter.com/eugeneyan/status/1303500014635810816?ref_src=twsrc%5Etfw",
      "label": "September 9, 2020",
      "type": "a"
    },
    {
      "href": "https://twitter.com/dvargas92495",
      "label": "David Vargas",
      "type": "a"
    },
    {
      "href": "https://joelc.io/",
      "label": "Joel Christiansen",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/mailbag-pdf-fields/",
      "label": "« Mailbag: Parsing Fields from PDFs—When to Use Machine Learning?",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/migrating-to-utterances/",
      "label": "Migrating Site Comments to Utterances »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:20Z",
  "source_url": "https://eugeneyan.com/writing/testing-ml/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.04118",
      "bytes": 367561,
      "reused": false,
      "local_path": "papers/repair/5b97e82f2bbab8d987a5.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 20980,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9b1965a26aac/pages/source.html",
  "local_html": "attachments/9b1965a26aac/pages/article.html",
  "body_base_path": "attachments/9b1965a26aac/pages/",
  "body_markdown_path": "attachments/9b1965a26aac/pages/article.md"
}