{ "source_file": "0930", "source_platform": "Medium", "author": "Benjamin Nweke", "title_listed": "I Trained Six Models for Fraud Detection, and the Best One Isn't in Production", "keywords": "", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/i-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production/", "publish_date": "2026-08-27", "corpus_stage": "", "notes": "", "id": "fb9c6fd5b42a", "collected_at": "2026-10-02T07:36:49", "title": "I Trained Six Models for Fraud Detection, and the Best One Isn't in Production | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# I Trained Six Models for Fraud Detection, and the Best One Isn't in Production\nWhat a final-year project taught me about the gap between evaluation metrics and production decisions\n[Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\nAugust 27, 202610 min read\n![](../images/000.jpg)\nPhoto by [Google DeepMind](https://unsplash.com/@googledeepmind?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/photos/a-large-group-of-colorful-blocks-lISkvdgfLEk?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)\nSomewhere in the middle of my final-year project, I trained six different models using the same fraud dataset, logged every training run, and picked a winner. Right now, the model that powers the current deployment is not the winning model.\nNairaShield is an AI-based fraud detection system that I built as my final year project for banking transactions. It started as a straightforward classification problem and ended up as something closer to a decision-support pipeline, partly because of that mismatch, and partly because my supervisor wasn't satisfied with the straightforward version.\nHe really liked the project, but there was one little thing; he had seen an anti-money laundering project by one of my classmates and felt that the two were too similar.\nI tried so hard to explain that the modelling problem was different, but he still wasn't convinced.\nWe had quite a few arguments over this matter before he told me what I needed to do to solve the problem at hand. And that single note is the reason the system ended up with a full regulatory review workflow and a second model family it wouldn't otherwise have had. It's also most of what this piece is actually about.\nThis is the first thing I've properly written since I left school a few weeks ago, capping off a two-month stretch away from writing while coursework and this project ate up everything else, so it felt right to come back with the project that has the most to actually show.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## Merging two datasets that don't agree with each other\nI trained on two public datasets that have no common schema. [PaySim](https://www.kaggle.com/datasets/ealaxi/paysim1) is a simulation of mobile money transactions and has attributes like `oldbalanceOrg` and `newbalanceOrig`. On the other hand, the [IEEE-CIS fraud detection dataset](https://www.kaggle.com/competitions/ieee-fraud-detection) covers card transactions and has none of that; instead, it has only `ProductCD`, `card1`, `card2`, and a long tail of other anonymized features. Before any of that merging actually happens, each dataset gets validated against its own schema first, so a malformed row fails loudly instead of quietly poisoning the training set:\npython\n\n```\nclass PaySimSchema(BaseModel):\n step: int\n type: str\n amount: float = Field(ge=0)\n nameOrig: str\n oldbalanceOrg: float\n newbalanceOrig: float\n nameDest: str\n oldbalanceDest: float\n newbalanceDest: float\n isFraud: int = Field(ge=0, le=1)\n isFlaggedFraud: int\n\nclass IeeeCisSchema(BaseModel):\n TransactionID: int\n isFraud: int = Field(ge=0, le=1)\n TransactionDT: int\n TransactionAmt: float = Field(ge=0)\n ProductCD: str\n card1: Optional[float] = None\n card2: Optional[float] = None\n addr1: Optional[float] = None\n P_emaildomain: Optional[str] = None\n```\n\nOnce both sides pass validation, mapping them into one shared structure looks like this:\npython\n\n```\ndf_paysim_aligned[\"transaction_id\"] = \"PAYSIM_\" + df_paysim[\"step\"].astype(str) + \"_\" + df_paysim.index.astype(str)\ndf_paysim_aligned[\"channel\"] = df_paysim[\"type\"].astype(str)\n\nproduct_map = {\"W\": \"CARD_WEB\", \"H\": \"CARD_HOST\", \"C\": \"CARD_PHONE\", \"S\": \"CARD_STORE\", \"R\": \"CARD_RECURRING\"}\ndf_ieee_aligned[\"channel\"] = df_ieee[\"ProductCD\"].map(product_map).fillna(\"CARD_OTHER\")\n```\n\nWhat took me way too long to realize was that having a unified schema meant the chosen features would have to be ones both datasets shared. \nThe paySim balances didn't make the cut because IEEE-CIS doesn't have a similar field. The actual deployed model is trained using three things: the amount of transaction, one-hot encoding of the channel, and one-hot encoding of the source dataset.\nI'm flagging that now because it matters a lot later in this piece, and ignoring it would undercut a very good thing I am trying to say here. \n## Balancing a dataset where fraud barely shows up\nSince there is very little fraud in both data sources, a model will be able to achieve high accuracy simply by predicting \"no fraud\" a lot of times while not predicting many fraud instances.\nI used SMOTE to oversample the minority class on the training split only, with a fallback to a hand-written interpolation version in case `imbalanced-learn` wasn't available in the deployment environment:\npython\n\n```\nk_neigh = min(2, sum(y_train == 1) - 1)\nif k_neigh >= 1:\n smote = SMOTE(random_state=42, k_neighbors=k_neigh)\n X_train_res, y_train_res = smote.fit_resample(X_train, y_train)\nelse:\n print(\"[Warning] Insufficient minority class samples to perform SMOTE. Skipping resampling.\")\n X_train_res, y_train_res = X_train, y_train\n```\n\nThe zero dependency fallback uses interpolation between a sample and its closest neighbors through Euclidean distance to produce synthetic minority samples, which is essentially a simpler version of how SMOTE works internally but left in the code for the sake of preventing failure when in a minimalistic setting.\n## Six models and a spreadsheet that argues with itself\nI trained Random Forest, Logistic Regression, XGBoost (both baseline and optimized versions), and LightGBM (again, both baseline and optimized). Every one of them ran through the same evaluation helper, so the numbers would actually be comparable instead of six slightly different measurement approaches wearing one result:\npython\n\n```\ndef fit_eval_and_log(model, name, filename, params, fit_kwargs=None):\n print(f\"\\n--- Training Model: {name} ---\")\n start_time = time.time()\n model.fit(X_train, y_train, **(fit_kwargs or {}))\n print(f\"[Training Info] Model training completed in {time.time() - start_time:.3f} seconds.\")\n\n preds = model.predict(X_test)\n probs = model.predict_proba(X_test)[:, 1] if hasattr(model, \"predict_proba\") else preds.astype(float)\n\n metrics = {\n \"Accuracy\": round(accuracy_score(y_test, preds), 6),\n \"Precision\": round(precision_score(y_test, preds, zero_division=0), 6),\n \"Recall\": round(recall_score(y_test, preds, zero_division=0), 6),\n \"AUC-ROC\": round(roc_auc_score(y_test, probs), 6),\n \"AUC-PR\": round(average_precision_score(y_test, probs), 6),\n }\n print(f\"{name} Test AUC-PR:\", metrics[\"AUC-PR\"])\n\n joblib.dump(model, filename)\n log_experiment_run(name, params, metrics)\n return metrics\n\n# Baseline vs. tuned, run back to back so the comparison is apples to apples\nxgb_base = xgb.XGBClassifier(use_label_encoder=False, eval_metric=\"logloss\", random_state=42)\nfit_eval_and_log(\n xgb_base, \"XGBoost (Baseline)\", \"xgboost_model.joblib\", xgb_base.get_params(),\n fit_kwargs={\"eval_set\": [(X_train, y_train), (X_test, y_test)], \"verbose\": 10}\n)\n\nxgb_tuned_params = tuned_params.get(\"xgboost\", {\n \"n_estimators\": 150, \"max_depth\": 6, \"learning_rate\": 0.1,\n \"subsample\": 0.8, \"colsample_bytree\": 0.8, \"random_state\": 42\n})\nxgb_tuned = xgb.XGBClassifier(**xgb_tuned_params, use_label_encoder=False, eval_metric=\"logloss\")\nfit_eval_and_log(xgb_tuned, \"XGBoost (Tuned)\", \"xgboost_model_tuned.joblib\", xgb_tuned_params)\n\nlgb_base = lgb.LGBMClassifier(random_state=42, verbose=-1)\nfit_eval_and_log(lgb_base, \"LightGBM (Baseline)\", \"lightgbm_model.joblib\", lgb_base.get_params())\n```\n\nEvery run, tuned or not, gets logged to `experiment_runs.json` with its own hyperparameters and metrics, which is the only reason it was possible to notice, months later, that the model in production wasn't the one at the top of that log:\n![Bar chart comparing AUC-PR across six fraud detection models](../images/001.png)\nThe model with the best AUC-PR isn't the one making the decisions in production. Image by author.\nA few things that are worth considering rather than skipping over. Logistic Regression shows the highest value of recall among all models at 0.95.\nBut don't just look at that; it also proves to be the least usable model of the six, as its low precision value of 0.14 means that most of its identified cases of fraud are false positives.\nTuning XGBoost improved AUC-PR slightly but actually made precision worse than the baseline, which is a reminder that hyperparameter search optimizes whatever metric you point it at, not necessarily the one that actually matters once the model is making real decisions.\nFinally, the baseline settings for LightGBM proved to be more efficient in terms of AUC-PR than optimized ones and were the best among all other models in the test.\nYou can always look back up to the diagram above to see what I'm talking about.\nThe production API still uses XGBoost (Tuned) as an active model, as this choice was made before the comparison of LightGBM models was completed.\nWhich, when you look at it now, is honestly worth revisiting.\nSHAP is what makes any of these numbers accountable to a human. Every flagged prediction comes with a list of features of what pushed it toward fraud, since \"the model said so\" isn't an answer a bank or a regulator should accept:\npython\n\n```\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(X_instance)\n```\n\n## What my supervisor actually wanted\nThe anti-money laundering project by a colleague of mine from the beginning of this piece is worth returning to, because the similarity my supervisor saw wasn't imagined. On the surface, his work and mine were doing the same basic job, both taking transaction data and deciding whether it looked suspicious, and my supervisor wasn't wrong to notice that.\nWhat he was actually pushing me to build wasn't another algorithm.\nNo. \nHe was pointing me toward the step that comes after a transaction gets flagged, the part where something actually happens next. From what I found digging into [other academic AML work](https://www.sciencedirect.com/org/science/article/pii/S1368520125000123), most of it stops right at classification. His comment was what got me thinking about what follows after the flag.\nThis led to the creation of a Regulatory Notification Center; a review layer built around role types modeled on Nigeria's Central Bank, the EFCC, and NDIC, each having a separate login and their own queue of flagged transactions.\nA regulatory reviewer can approve, ask for OTP verification, or block it outright, and each decision will be documented in the audit log with who acted and when. Building that workflow is also what forced me to include LightGBM in the comparison.\n## Where the confidence gate actually comes from\nThe API doesn't treat a fraud prediction as a single yes-or-no. The base classification threshold sits at 0.50, but what happens after that is split into bands:\npython\n\n```\nif response.get(\"prediction\") == 1:\n fp = response.get(\"fraud_probability\", 0.0)\n alert_payload = {\n \"alert_id\": f\"API-ALT-{_random.randint(100000, 999999)}\",\n \"transaction_id\": data.get(\"transaction_id\", f\"TXN-{_random.randint(10000, 99999)}\"),\n \"amount\": float(data.get(\"amount\", 0.0)),\n \"timestamp\": datetime.datetime.now().isoformat(),\n \"model_probability\": fp,\n \"triggered_rules\": [\"MODEL_HIGH_FRAUD_RISK\"] if fp >= 0.85 else [\"MODEL_SUSPICIOUS_RISK\"],\n \"status\": \"BLOCKED\" if fp >= 0.80 else \"PENDING_OTP\",\n \"rag_context\": response.get(\"rag_context\", []),\n \"sender_bank\": data.get(\"sender_bank\"),\n \"regulatory_body\": data.get(\"regulatory_body\")\n }\n\n # Dispatch to SMS/Email notifier if it is critical (>= 85%)\n if _api_notifier is not None and fp >= 0.85:\n threading.Thread(target=_api_notifier.dispatch, args=(alert_payload,)).start()\n```\n\nAnything that scores less than 0.50 passes without any alarm whatsoever going off. If the score falls between 0.50 and 0.80, then the transaction becomes flagged but is not stopped; it's routed to PENDING_OTP, which basically means that a human or a secondary verification step gets involved before anything final happens.\nScores from 0.80 and above block the transaction from happening, while at 0.85 an immediate alarm is sent either via SMS or email. The background thread sends it immediately without waiting for any request-response cycle.\n![Diagram of a transaction](../images/002.png)\nA transaction never gets a single up-or-down answer. It's routed to whoever should be looking at it next: an analyst, a regulator, or nobody at all. Image by author.\n···\n## Defense day\nThe panel started with the usual, or would I say expected, set of questions. Which models I used, what hyperparameters I tuned and how I arrived at them, where the datasets came from, and whether they were representative enough of transaction behavior specifically.\nI told them XGBoost (Tuned), because that's what was actually in production. What I left out, mostly because I hadn't sat with it properly myself yet, was that LightGBM's baseline had already beaten it on the metric that mattered most, days before I stood in that room. Nobody followed up on it. I'm the one following up on it now.\nThen it moved to the one I had not really prepared for: what is special about this fraud detection solution compared to all the others that already exist, and what problems with other solutions does it solve?\nThe scenario questions came next.\nIf the CEO of a big company decides one random afternoon to move a million dollars in a single transaction, would it get flagged? \nAnd if a student attempted the same transaction, what would happen?\nI told them the system studies patterns and behavior, meaning it would treat those two transactions differently depending on who was making them.\nWriting this months later, with the code in front of me instead of a panel across the table, I want to give the more accurate answer instead of the more comfortable one.\nWith the system as it is actually built now, those two transactions would receive an identical prediction. There's no account identity or transaction history in the thirteen features on which the model learns to look at how much money changed hands, through what channel, and how similar the transaction pattern is to either of the two source datasets. \nThe honest answer is that my system currently treats a million dollars the same regardless of whose account it left. It sees a million dollars, and that's all, and it does nothing with that knowledge except treat it like any other transaction.\nThe thing it does, though, is that it doesn't rely on that number alone to make a decision. In my system, if a million-dollar transaction is predicted to be fraud at a probability of 0.85, it is flagged, escalated, and in the regulatory workflow, is sent to a human for review. That human will have more context to work with than the model.\nThe confidence gate is doing the job that account-level reasoning would do if it existed yet. It's a real answer, just a more modest one than the answer I gave that day.\n## Final thoughts and what I'd actually do differently\nThat leads us straight to the next step, which is the one the defense question revealed: adding behavioral features per account, averaging transactions, and deviation from historical user amounts. With time, the model should then be able to make the distinction I spoke of, rather than approximating it through the confidence gate. \nAdditionally, I would revisit defaulting to XGBoost (Tuned) as the production model given LightGBM's baseline configuration scored better on the metric that matters most for this problem.\nBoth of those come from the same mistake: I locked in a production model before the comparison was even finished, and only really noticed while writing this down. \nIf there's one thing worth taking from all of it, it's that the best score on a spreadsheet isn't the same as the right production decision, and a model was never going to be the whole system by itself. The confidence gate, the audit log, the human reading a PENDING_OTP alert, that's doing as much of the real fraud-catching as the model is.\nI'm back to writing regularly after two months away, and this felt like the right project to be honest about first. There's a lot I haven't covered here: the RBAC setup, the audit logging, the regulatory notification pipeline itself; that's probably its own piece once this one's out in the world.\n···\n## Before you go!\nI write about the real, messy engineering behind AI systems, where abstractions help, where they hurt, and what it takes to build reliably.\nYou can [**subscribe to my newsletter**](https://thatcsguy.substack.com/?r=79lldp&utm_campaign=pub-share-checklist) if you'd like more of that.\n**Connect With Me**\n * [LinkedIn](http://www.linkedin.com/in/nwekebenjamin)\n * [Substack](https://thatcsguy.substack.com/)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-4ad89a2fc5e69dde007d.jpeg)\nWritten By\n## [Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\nBenjamin Nweke is a Computer Science graduate and technical writer exploring AI, machine learning, and data science. He writes about building reliable AI systems, model evaluation, agentic architectures, and the practical challenges that appear when machine learning moves beyond experiments. His work focuses on making technically complex ideas clear, useful, and grounded in real implementation.\n[See all from Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Fraud Detection](https://towardsdatascience.com/tag/fraud-detection/)[Mlops](https://towardsdatascience.com/tag/mlops/)[Model Deployment](https://towardsdatascience.com/tag/model-deployment/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&title=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&title=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&text=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/003.jpg)](https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/)\n## [How to simulate financial portfolios with Python](https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAn application of Multidimensional Geometric Brownian Motion\n[Riccardo D'Elia](https://towardsdatascience.com/author/riccardodelia95/)\nNovember 9, 20205 min read\n * [![](../images/004.jpeg)](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/)\n## [17 Quick Examples to Learn Python Lists](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nOne of the most commonly used Python data structures Data structures are the key components...\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nMay 21, 20216 min read\n * [![](../images/body-63beb50f664a055ed739.jpg)](https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/)\n## [Principal Component Analysis (PCA)](https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA Physically Intuitive Mathematical Introduction\n[Lydia Nemec](https://towardsdatascience.com/author/lydianemec/)\nJanuary 16, 20229 min read\n * [![](../images/006.jpg)](https://towardsdatascience.com/the-history-and-future-of-artificial-intelligence-through-the-lenses-of-computer-chess-and-legal-a-i-a7b7327f8800/)\n## [The History and Future of Artificial Intelligence through the lenses of Computer Chess and Legal A.I](https://towardsdatascience.com/the-history-and-future-of-artificial-intelligence-through-the-lenses-of-computer-chess-and-legal-a-i-a7b7327f8800/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn in-depth interview with Professor Jaap van den Herik\n[Emil Rijcken](https://towardsdatascience.com/author/emilrijcken/)\nDecember 30, 202115 min read\n * [![](../images/007.jpeg)](https://towardsdatascience.com/machine-learning-with-fuzzy-logic-52c85b46bfe4/)\n## [Machine Learning with Fuzzy Logic](https://towardsdatascience.com/machine-learning-with-fuzzy-logic-52c85b46bfe4/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow to extract control rules from data using Fuzzy logic\n[Nnamdi Uzoukwu](https://towardsdatascience.com/author/uzoukwu.nnamdi/)\nJuly 20, 20225 min read\n * [![](../images/008.jpg)](https://towardsdatascience.com/analysing-and-visualising-the-country-wise-population-from-1955-to-2020-with-pandas-matplotlib-70b3614eed6b/)\n## [Analysing and Visualising the Country wise Population from 1955 to 2020 with Pandas, Matplotlib...](https://towardsdatascience.com/analysing-and-visualising-the-country-wise-population-from-1955-to-2020-with-pandas-matplotlib-70b3614eed6b/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nDay 1, 2, 3 of 100 Days of Data Science.\n[Durgesh Samariya](https://towardsdatascience.com/author/themlphdstudent/)\nAugust 17, 20206 min read\n * [![](../images/009.png)](https://towardsdatascience.com/5-must-know-date-and-time-functions-of-sql-3079cdb1a71e/)\n## [5 Must-Know Date and Time Functions of SQL](https://towardsdatascience.com/5-must-know-date-and-time-functions-of-sql-3079cdb1a71e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nPractical guide with examples\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJanuary 31, 20215 min read\n * [![](../images/010.png)](https://towardsdatascience.com/why-you-are-not-a-data-scientist-f56b5dee68f4/)\n## [Why you are NOT a (Data) Scientist!](https://towardsdatascience.com/why-you-are-not-a-data-scientist-f56b5dee68f4/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIs Bill Gates a scientist? How about Linus Torvalds? If we don't call them scientists, why we call young data developers as data...\n[Shyam Sundar Dhanabalan](https://towardsdatascience.com/author/shyamdsundar/)\nApril 15, 20217 min read\n * [![](../images/011.png)](https://towardsdatascience.com/from-perceptron-to-densenet-an-introduction-to-convolutional-neural-networks-ab37e3b7872e/)\n## [Convolutional Neural Networks: An Introduction](https://towardsdatascience.com/from-perceptron-to-densenet-an-introduction-to-convolutional-neural-networks-ab37e3b7872e/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA brief journey from Perceptron to ResNet\n[Oliver Kramer](https://towardsdatascience.com/author/oliver_kramer/)\nOctober 21, 202212 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n", "metadata": { "title": "I Trained Six Models for Fraud Detection, and the Best One Isn't in Production | Towards Data Science", "description": "What a final-year project taught me about the gap between evaluation metrics and production decisions", "keywords": null, "author": "Benjamin Nweke", "og:locale": "en_US", "og:type": "article", "og:title": "I Trained Six Models for Fraud Detection, and the Best One Isn't in Production", "og:description": "What a final-year project taught me about the gap between evaluation metrics and production decisions", "og:url": "https://towardsdatascience.com/i-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/google-deepmind-lISkvdgfLEk-unsplash-scaled.jpg", "og:image:width": "2560", "og:image:height": "1440", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Benjamin Nweke", "twitter:label2": "Est. reading time", "twitter:data2": "10 min read", "article:published_time": "2026-08-27T17:05:01.904Z", "article:modified_time": "2026-08-27T17:05:01.904Z", "article:author": "https://towardsdatascience.com/author/benjaminnweke/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/fb9c6fd5b42a/code/block_000.txt", "chars": 608 }, { "language": "txt", "file": "attachments/fb9c6fd5b42a/code/block_001.txt", "chars": 377 }, { "language": "txt", "file": "attachments/fb9c6fd5b42a/code/block_002.txt", "chars": 333 }, { "language": "txt", "file": "attachments/fb9c6fd5b42a/code/block_003.txt", "chars": 1847 }, { "language": "txt", "file": "attachments/fb9c6fd5b42a/code/block_004.txt", "chars": 86 }, { "language": "txt", "file": "attachments/fb9c6fd5b42a/code/block_005.txt", "chars": 934 } ], "images": [ { "status": "ok", "url": "https://cdn-cookieyes.com/assets/images/revisit.svg", "bytes": 2339, "reused": false, "content_type": "image/svg+xml", "local_path": "attachments/fb9c6fd5b42a/images/body-df29beacd7496220cc94.svg", "src": "https://cdn-cookieyes.com/assets/images/revisit.svg", "alt": "Revisit consent button", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://cdn-cookieyes.com/assets/images/close.svg", "bytes": 1345, "reused": false, "content_type": "image/svg+xml", "local_path": "attachments/fb9c6fd5b42a/images/body-2159bb67f3de8ad870eb.svg", "src": "https://cdn-cookieyes.com/assets/images/close.svg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/google-deepmind-lISkvdgfLEk-unsplash-scaled.jpg", "bytes": 751774, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/000.jpg", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/google-deepmind-lISkvdgfLEk-unsplash-scaled.jpg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/model_comparison_chart-1024x622.png", "bytes": 74260, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/001.png", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/model_comparison_chart-1024x622.png", "alt": "Bar chart comparing AUC-PR across six fraud detection models", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/architecture_diagram-2-706x1024.png", "bytes": 150572, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/002.png", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/architecture_diagram-2-706x1024.png", "alt": "Diagram of a transaction", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/1787657271062_x78p7k.jpeg", "bytes": 194410, "reused": false, "content_type": "image/jpeg", "local_path": "attachments/fb9c6fd5b42a/images/body-4ad89a2fc5e69dde007d.jpeg", "src": "https://assets.insightmediagroup.io/media/1787657271062_x78p7k.jpeg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0CxfSE1h-N0GD1yGL-scaled.jpg", "bytes": 473287, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/003.jpg", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0CxfSE1h-N0GD1yGL-scaled.jpg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/13pPeo8VPDVQeUG6fN4fb1w-scaled.jpeg", "bytes": 883327, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/004.jpeg", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/13pPeo8VPDVQeUG6fN4fb1w-scaled.jpeg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://towardsdatascience.com/images/fallback-image.jpg", "bytes": 5092, "reused": false, "content_type": "image/jpeg", "local_path": "attachments/fb9c6fd5b42a/images/body-63beb50f664a055ed739.jpg", "src": "https://towardsdatascience.com/images/fallback-image.jpg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0GIm-HNNF550mWsrg-scaled.jpg", "bytes": 123956, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/006.jpg", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0GIm-HNNF550mWsrg-scaled.jpg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/1BcFMJ9ezpjDKCbZqhXGE5Q-scaled.jpeg", "bytes": 196722, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/007.jpeg", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/1BcFMJ9ezpjDKCbZqhXGE5Q-scaled.jpeg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/0ziR1fT67_yb7uQls-scaled.jpg", "bytes": 379326, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/008.jpg", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/0ziR1fT67_yb7uQls-scaled.jpg", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1xKnEpFn_NEfXVM2fq7dkBg.png", "bytes": 153223, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/009.png", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1xKnEpFn_NEfXVM2fq7dkBg.png", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1OS7xm5vgL3c2zPXlGeSsGw.png", "bytes": 93843, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/010.png", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1OS7xm5vgL3c2zPXlGeSsGw.png", "alt": "", "downloaded": true, "discovery": "saved_markdown" }, { "status": "ok", "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/1VoaXPBLlxY4KxP_BmnM0og.png", "bytes": 107772, "reused": true, "local_path": "attachments/fb9c6fd5b42a/images/011.png", "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/1VoaXPBLlxY4KxP_BmnM0og.png", "alt": "", "downloaded": true, "discovery": "saved_markdown" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/benjaminnweke/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/fraud-detection/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/mlops/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/model-deployment/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/python/" }, { "type": "internal", "href": "https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/riccardodelia95/" }, { "type": "internal", "href": "https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/sonery/" }, { "type": "internal", "href": "https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://unsplash.com/@googledeepmind" }, { "type": "external", "href": "https://unsplash.com/photos/a-large-group-of-colorful-blocks-lISkvdgfLEk" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://www.kaggle.com/datasets/ealaxi/paysim1" }, { "type": "external", "href": "https://www.kaggle.com/competitions/ieee-fraud-detection" }, { "type": "external", "href": "https://www.sciencedirect.com/org/science/article/pii/S1368520125000123" }, { "type": "external", "href": "https://thatcsguy.substack.com/?r=79lldp" }, { "type": "external", "href": "http://www.linkedin.com/in/nwekebenjamin" }, { "type": "external", "href": "https://thatcsguy.substack.com/" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=I+Trained+Six+Models+for+Fraud+Detection%2C+and+the+Best+One+Isn%27t+in+Production&u=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=I+Trained+Six+Models+for+Fraud+Detection%2C+and+the+Best+One+Isn%27t+in+Production&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F" }, { "type": "external", "href": "https://x.com/share?text=I+Trained+Six+Models+for+Fraud+Detection%2C+and+the+Best+One+Isn%27t+in+Production&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" }, { "type": "external", "href": "https://bsky.app/profile/towardsdatascience.com" }, { "type": "external", "href": "https://thenewstack.io" }, { "type": "external", "href": "https://roadmap.sh" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:23:10Z", "source_url": "https://towardsdatascience.com/i-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "http_403", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 15, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# I Trained Six Models for Fraud Detection, and the Best One Isn't in Production\nWhat a final-year project taught me about the gap between evaluation metrics and production decisions\n[Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\nAugust 27, 202610 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/google-deepmind-lISkvdgfLEk-unsplash-scaled.jpg)\nPhoto by [Google DeepMind](https://unsplash.com/@googledeepmind?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/photos/a-large-group-of-colorful-blocks-lISkvdgfLEk?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)\nSomewhere in the middle of my final-year project, I trained six different models using the same fraud dataset, logged every training run, and picked a winner. Right now, the model that powers the current deployment is not the winning model.\nNairaShield is an AI-based fraud detection system that I built as my final year project for banking transactions. It started as a straightforward classification problem and ended up as something closer to a decision-support pipeline, partly because of that mismatch, and partly because my supervisor wasn't satisfied with the straightforward version.\nHe really liked the project, but there was one little thing; he had seen an anti-money laundering project by one of my classmates and felt that the two were too similar.\nI tried so hard to explain that the modelling problem was different, but he still wasn't convinced.\nWe had quite a few arguments over this matter before he told me what I needed to do to solve the problem at hand. And that single note is the reason the system ended up with a full regulatory review workflow and a second model family it wouldn't otherwise have had. It's also most of what this piece is actually about.\nThis is the first thing I've properly written since I left school a few weeks ago, capping off a two-month stretch away from writing while coursework and this project ate up everything else, so it felt right to come back with the project that has the most to actually show.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## Merging two datasets that don't agree with each other\nI trained on two public datasets that have no common schema. [PaySim](https://www.kaggle.com/datasets/ealaxi/paysim1) is a simulation of mobile money transactions and has attributes like `oldbalanceOrg` and `newbalanceOrig`. On the other hand, the [IEEE-CIS fraud detection dataset](https://www.kaggle.com/competitions/ieee-fraud-detection) covers card transactions and has none of that; instead, it has only `ProductCD`, `card1`, `card2`, and a long tail of other anonymized features. Before any of that merging actually happens, each dataset gets validated against its own schema first, so a malformed row fails loudly instead of quietly poisoning the training set:\npython\n\n```\nclass PaySimSchema(BaseModel):\n step: int\n type: str\n amount: float = Field(ge=0)\n nameOrig: str\n oldbalanceOrg: float\n newbalanceOrig: float\n nameDest: str\n oldbalanceDest: float\n newbalanceDest: float\n isFraud: int = Field(ge=0, le=1)\n isFlaggedFraud: int\n\nclass IeeeCisSchema(BaseModel):\n TransactionID: int\n isFraud: int = Field(ge=0, le=1)\n TransactionDT: int\n TransactionAmt: float = Field(ge=0)\n ProductCD: str\n card1: Optional[float] = None\n card2: Optional[float] = None\n addr1: Optional[float] = None\n P_emaildomain: Optional[str] = None\n```\n\nOnce both sides pass validation, mapping them into one shared structure looks like this:\npython\n\n```\ndf_paysim_aligned[\"transaction_id\"] = \"PAYSIM_\" + df_paysim[\"step\"].astype(str) + \"_\" + df_paysim.index.astype(str)\ndf_paysim_aligned[\"channel\"] = df_paysim[\"type\"].astype(str)\n\nproduct_map = {\"W\": \"CARD_WEB\", \"H\": \"CARD_HOST\", \"C\": \"CARD_PHONE\", \"S\": \"CARD_STORE\", \"R\": \"CARD_RECURRING\"}\ndf_ieee_aligned[\"channel\"] = df_ieee[\"ProductCD\"].map(product_map).fillna(\"CARD_OTHER\")\n```\n\nWhat took me way too long to realize was that having a unified schema meant the chosen features would have to be ones both datasets shared. \nThe paySim balances didn't make the cut because IEEE-CIS doesn't have a similar field. The actual deployed model is trained using three things: the amount of transaction, one-hot encoding of the channel, and one-hot encoding of the source dataset.\nI'm flagging that now because it matters a lot later in this piece, and ignoring it would undercut a very good thing I am trying to say here. \n## Balancing a dataset where fraud barely shows up\nSince there is very little fraud in both data sources, a model will be able to achieve high accuracy simply by predicting \"no fraud\" a lot of times while not predicting many fraud instances.\nI used SMOTE to oversample the minority class on the training split only, with a fallback to a hand-written interpolation version in case `imbalanced-learn` wasn't available in the deployment environment:\npython\n\n```\nk_neigh = min(2, sum(y_train == 1) - 1)\nif k_neigh >= 1:\n smote = SMOTE(random_state=42, k_neighbors=k_neigh)\n X_train_res, y_train_res = smote.fit_resample(X_train, y_train)\nelse:\n print(\"[Warning] Insufficient minority class samples to perform SMOTE. Skipping resampling.\")\n X_train_res, y_train_res = X_train, y_train\n```\n\nThe zero dependency fallback uses interpolation between a sample and its closest neighbors through Euclidean distance to produce synthetic minority samples, which is essentially a simpler version of how SMOTE works internally but left in the code for the sake of preventing failure when in a minimalistic setting.\n## Six models and a spreadsheet that argues with itself\nI trained Random Forest, Logistic Regression, XGBoost (both baseline and optimized versions), and LightGBM (again, both baseline and optimized). Every one of them ran through the same evaluation helper, so the numbers would actually be comparable instead of six slightly different measurement approaches wearing one result:\npython\n\n```\ndef fit_eval_and_log(model, name, filename, params, fit_kwargs=None):\n print(f\"\\n--- Training Model: {name} ---\")\n start_time = time.time()\n model.fit(X_train, y_train, **(fit_kwargs or {}))\n print(f\"[Training Info] Model training completed in {time.time() - start_time:.3f} seconds.\")\n\n preds = model.predict(X_test)\n probs = model.predict_proba(X_test)[:, 1] if hasattr(model, \"predict_proba\") else preds.astype(float)\n\n metrics = {\n \"Accuracy\": round(accuracy_score(y_test, preds), 6),\n \"Precision\": round(precision_score(y_test, preds, zero_division=0), 6),\n \"Recall\": round(recall_score(y_test, preds, zero_division=0), 6),\n \"AUC-ROC\": round(roc_auc_score(y_test, probs), 6),\n \"AUC-PR\": round(average_precision_score(y_test, probs), 6),\n }\n print(f\"{name} Test AUC-PR:\", metrics[\"AUC-PR\"])\n\n joblib.dump(model, filename)\n log_experiment_run(name, params, metrics)\n return metrics\n\n# Baseline vs. tuned, run back to back so the comparison is apples to apples\nxgb_base = xgb.XGBClassifier(use_label_encoder=False, eval_metric=\"logloss\", random_state=42)\nfit_eval_and_log(\n xgb_base, \"XGBoost (Baseline)\", \"xgboost_model.joblib\", xgb_base.get_params(),\n fit_kwargs={\"eval_set\": [(X_train, y_train), (X_test, y_test)], \"verbose\": 10}\n)\n\nxgb_tuned_params = tuned_params.get(\"xgboost\", {\n \"n_estimators\": 150, \"max_depth\": 6, \"learning_rate\": 0.1,\n \"subsample\": 0.8, \"colsample_bytree\": 0.8, \"random_state\": 42\n})\nxgb_tuned = xgb.XGBClassifier(**xgb_tuned_params, use_label_encoder=False, eval_metric=\"logloss\")\nfit_eval_and_log(xgb_tuned, \"XGBoost (Tuned)\", \"xgboost_model_tuned.joblib\", xgb_tuned_params)\n\nlgb_base = lgb.LGBMClassifier(random_state=42, verbose=-1)\nfit_eval_and_log(lgb_base, \"LightGBM (Baseline)\", \"lightgbm_model.joblib\", lgb_base.get_params())\n```\n\nEvery run, tuned or not, gets logged to `experiment_runs.json` with its own hyperparameters and metrics, which is the only reason it was possible to notice, months later, that the model in production wasn't the one at the top of that log:\n![Bar chart comparing AUC-PR across six fraud detection models](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/model_comparison_chart-1024x622.png)\nThe model with the best AUC-PR isn't the one making the decisions in production. Image by author.\nA few things that are worth considering rather than skipping over. Logistic Regression shows the highest value of recall among all models at 0.95.\nBut don't just look at that; it also proves to be the least usable model of the six, as its low precision value of 0.14 means that most of its identified cases of fraud are false positives.\nTuning XGBoost improved AUC-PR slightly but actually made precision worse than the baseline, which is a reminder that hyperparameter search optimizes whatever metric you point it at, not necessarily the one that actually matters once the model is making real decisions.\nFinally, the baseline settings for LightGBM proved to be more efficient in terms of AUC-PR than optimized ones and were the best among all other models in the test.\nYou can always look back up to the diagram above to see what I'm talking about.\nThe production API still uses XGBoost (Tuned) as an active model, as this choice was made before the comparison of LightGBM models was completed.\nWhich, when you look at it now, is honestly worth revisiting.\nSHAP is what makes any of these numbers accountable to a human. Every flagged prediction comes with a list of features of what pushed it toward fraud, since \"the model said so\" isn't an answer a bank or a regulator should accept:\npython\n\n```\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(X_instance)\n```\n\n## What my supervisor actually wanted\nThe anti-money laundering project by a colleague of mine from the beginning of this piece is worth returning to, because the similarity my supervisor saw wasn't imagined. On the surface, his work and mine were doing the same basic job, both taking transaction data and deciding whether it looked suspicious, and my supervisor wasn't wrong to notice that.\nWhat he was actually pushing me to build wasn't another algorithm.\nNo. \nHe was pointing me toward the step that comes after a transaction gets flagged, the part where something actually happens next. From what I found digging into [other academic AML work](https://www.sciencedirect.com/org/science/article/pii/S1368520125000123), most of it stops right at classification. His comment was what got me thinking about what follows after the flag.\nThis led to the creation of a Regulatory Notification Center; a review layer built around role types modeled on Nigeria's Central Bank, the EFCC, and NDIC, each having a separate login and their own queue of flagged transactions.\nA regulatory reviewer can approve, ask for OTP verification, or block it outright, and each decision will be documented in the audit log with who acted and when. Building that workflow is also what forced me to include LightGBM in the comparison.\n## Where the confidence gate actually comes from\nThe API doesn't treat a fraud prediction as a single yes-or-no. The base classification threshold sits at 0.50, but what happens after that is split into bands:\npython\n\n```\nif response.get(\"prediction\") == 1:\n fp = response.get(\"fraud_probability\", 0.0)\n alert_payload = {\n \"alert_id\": f\"API-ALT-{_random.randint(100000, 999999)}\",\n \"transaction_id\": data.get(\"transaction_id\", f\"TXN-{_random.randint(10000, 99999)}\"),\n \"amount\": float(data.get(\"amount\", 0.0)),\n \"timestamp\": datetime.datetime.now().isoformat(),\n \"model_probability\": fp,\n \"triggered_rules\": [\"MODEL_HIGH_FRAUD_RISK\"] if fp >= 0.85 else [\"MODEL_SUSPICIOUS_RISK\"],\n \"status\": \"BLOCKED\" if fp >= 0.80 else \"PENDING_OTP\",\n \"rag_context\": response.get(\"rag_context\", []),\n \"sender_bank\": data.get(\"sender_bank\"),\n \"regulatory_body\": data.get(\"regulatory_body\")\n }\n\n # Dispatch to SMS/Email notifier if it is critical (>= 85%)\n if _api_notifier is not None and fp >= 0.85:\n threading.Thread(target=_api_notifier.dispatch, args=(alert_payload,)).start()\n```\n\nAnything that scores less than 0.50 passes without any alarm whatsoever going off. If the score falls between 0.50 and 0.80, then the transaction becomes flagged but is not stopped; it's routed to PENDING_OTP, which basically means that a human or a secondary verification step gets involved before anything final happens.\nScores from 0.80 and above block the transaction from happening, while at 0.85 an immediate alarm is sent either via SMS or email. The background thread sends it immediately without waiting for any request-response cycle.\n![Diagram of a transaction](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/architecture_diagram-2-706x1024.png)\nA transaction never gets a single up-or-down answer. It's routed to whoever should be looking at it next: an analyst, a regulator, or nobody at all. Image by author.\n···\n## Defense day\nThe panel started with the usual, or would I say expected, set of questions. Which models I used, what hyperparameters I tuned and how I arrived at them, where the datasets came from, and whether they were representative enough of transaction behavior specifically.\nI told them XGBoost (Tuned), because that's what was actually in production. What I left out, mostly because I hadn't sat with it properly myself yet, was that LightGBM's baseline had already beaten it on the metric that mattered most, days before I stood in that room. Nobody followed up on it. I'm the one following up on it now.\nThen it moved to the one I had not really prepared for: what is special about this fraud detection solution compared to all the others that already exist, and what problems with other solutions does it solve?\nThe scenario questions came next.\nIf the CEO of a big company decides one random afternoon to move a million dollars in a single transaction, would it get flagged? \nAnd if a student attempted the same transaction, what would happen?\nI told them the system studies patterns and behavior, meaning it would treat those two transactions differently depending on who was making them.\nWriting this months later, with the code in front of me instead of a panel across the table, I want to give the more accurate answer instead of the more comfortable one.\nWith the system as it is actually built now, those two transactions would receive an identical prediction. There's no account identity or transaction history in the thirteen features on which the model learns to look at how much money changed hands, through what channel, and how similar the transaction pattern is to either of the two source datasets. \nThe honest answer is that my system currently treats a million dollars the same regardless of whose account it left. It sees a million dollars, and that's all, and it does nothing with that knowledge except treat it like any other transaction.\nThe thing it does, though, is that it doesn't rely on that number alone to make a decision. In my system, if a million-dollar transaction is predicted to be fraud at a probability of 0.85, it is flagged, escalated, and in the regulatory workflow, is sent to a human for review. That human will have more context to work with than the model.\nThe confidence gate is doing the job that account-level reasoning would do if it existed yet. It's a real answer, just a more modest one than the answer I gave that day.\n## Final thoughts and what I'd actually do differently\nThat leads us straight to the next step, which is the one the defense question revealed: adding behavioral features per account, averaging transactions, and deviation from historical user amounts. With time, the model should then be able to make the distinction I spoke of, rather than approximating it through the confidence gate. \nAdditionally, I would revisit defaulting to XGBoost (Tuned) as the production model given LightGBM's baseline configuration scored better on the metric that matters most for this problem.\nBoth of those come from the same mistake: I locked in a production model before the comparison was even finished, and only really noticed while writing this down. \nIf there's one thing worth taking from all of it, it's that the best score on a spreadsheet isn't the same as the right production decision, and a model was never going to be the whole system by itself. The confidence gate, the audit log, the human reading a PENDING_OTP alert, that's doing as much of the real fraud-catching as the model is.\nI'm back to writing regularly after two months away, and this felt like the right project to be honest about first. There's a lot I haven't covered here: the RBAC setup, the audit logging, the regulatory notification pipeline itself; that's probably its own piece once this one's out in the world.\n···\n## Before you go!\nI write about the real, messy engineering behind AI systems, where abstractions help, where they hurt, and what it takes to build reliably.\nYou can [**subscribe to my newsletter**](https://thatcsguy.substack.com/?r=79lldp&utm_campaign=pub-share-checklist) if you'd like more of that.\n**Connect With Me**\n * [LinkedIn](http://www.linkedin.com/in/nwekebenjamin)\n * [Substack](https://thatcsguy.substack.com/)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657271062_x78p7k.jpeg)\nWritten By\n## [Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\nBenjamin Nweke is a Computer Science graduate and technical writer exploring AI, machine learning, and data science. He writes about building reliable AI systems, model evaluation, agentic architectures, and the practical challenges that appear when machine learning moves beyond experiments. His work focuses on making technically complex ideas clear, useful, and grounded in real implementation.\n[See all from Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Fraud Detection](https://towardsdatascience.com/tag/fraud-detection/)[Mlops](https://towardsdatascience.com/tag/mlops/)[Model Deployment](https://towardsdatascience.com/tag/model-deployment/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&title=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&title=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&text=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0CxfSE1h-N0GD1yGL-scaled.jpg)](https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/)\n## [How to simulate financial portfolios with Python](https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAn application of Multidimensional Geometric Brownian Motion\n[Riccardo D'Elia](https://towardsdatascience.com/author/riccardodelia95/)\nNovember 9, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/13pPeo8VPDVQeUG6fN4fb1w-scaled.jpeg)](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/)\n## [17 Quick Examples to Learn Python Lists](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nOne of the most commonly used Python data structures Data structures are the key components...\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nMay 21, 20216 min read\n * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/)\n## [Principal Component Analysis (PCA)](https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA Physically Intuitive Mathematical Introduction\n[Lydia Nemec](https://towardsdatascience.com/author/lydianemec/)\nJanuary 16, 20229 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0GIm-HNNF550mWsrg-scaled.jpg)](https://towardsdatascience.com/the-history-and-future-of-artificial-intelligence-through-the-lenses-of-computer-chess-and-legal-a-i-a7b7327f8800/)\n## [The History and Future of Artificial Intelligence through the lenses of Computer Chess and Legal A.I](https://towardsdatascience.com/the-history-and-future-of-artificial-intelligence-through-the-lenses-of-computer-chess-and-legal-a-i-a7b7327f8800/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn in-depth interview with Professor Jaap van den Herik\n[Emil Rijcken](https://towardsdatascience.com/author/emilrijcken/)\nDecember 30, 202115 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/1BcFMJ9ezpjDKCbZqhXGE5Q-scaled.jpeg)](https://towardsdatascience.com/machine-learning-with-fuzzy-logic-52c85b46bfe4/)\n## [Machine Learning with Fuzzy Logic](https://towardsdatascience.com/machine-learning-with-fuzzy-logic-52c85b46bfe4/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow to extract control rules from data using Fuzzy logic\n[Nnamdi Uzoukwu](https://towardsdatascience.com/author/uzoukwu.nnamdi/)\nJuly 20, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/0ziR1fT67_yb7uQls-scaled.jpg)](https://towardsdatascience.com/analysing-and-visualising-the-country-wise-population-from-1955-to-2020-with-pandas-matplotlib-70b3614eed6b/)\n## [Analysing and Visualising the Country wise Population from 1955 to 2020 with Pandas, Matplotlib...](https://towardsdatascience.com/analysing-and-visualising-the-country-wise-population-from-1955-to-2020-with-pandas-matplotlib-70b3614eed6b/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nDay 1, 2, 3 of 100 Days of Data Science.\n[Durgesh Samariya](https://towardsdatascience.com/author/themlphdstudent/)\nAugust 17, 20206 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1xKnEpFn_NEfXVM2fq7dkBg.png)](https://towardsdatascience.com/5-must-know-date-and-time-functions-of-sql-3079cdb1a71e/)\n## [5 Must-Know Date and Time Functions of SQL](https://towardsdatascience.com/5-must-know-date-and-time-functions-of-sql-3079cdb1a71e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nPractical guide with examples\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJanuary 31, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1OS7xm5vgL3c2zPXlGeSsGw.png)](https://towardsdatascience.com/why-you-are-not-a-data-scientist-f56b5dee68f4/)\n## [Why you are NOT a (Data) Scientist!](https://towardsdatascience.com/why-you-are-not-a-data-scientist-f56b5dee68f4/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIs Bill Gates a scientist? How about Linus Torvalds? If we don't call them scientists, why we call young data developers as data...\n[Shyam Sundar Dhanabalan](https://towardsdatascience.com/author/shyamdsundar/)\nApril 15, 20217 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/1VoaXPBLlxY4KxP_BmnM0og.png)](https://towardsdatascience.com/from-perceptron-to-densenet-an-introduction-to-convolutional-neural-networks-ab37e3b7872e/)\n## [Convolutional Neural Networks: An Introduction](https://towardsdatascience.com/from-perceptron-to-densenet-an-introduction-to-convolutional-neural-networks-ab37e3b7872e/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA brief journey from Perceptron to ResNet\n[Oliver Kramer](https://towardsdatascience.com/author/oliver_kramer/)\nOctober 21, 202212 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n", "body_base_path": "attachments/fb9c6fd5b42a/pages/", "body_markdown_path": "attachments/fb9c6fd5b42a/pages/archived-body.md", "local_html": "attachments/fb9c6fd5b42a/pages/archived-body.html", "body_source": "prior_capture" }