![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) # I Trained Six Models for Fraud Detection, and the Best One Isn't in Production What a final-year project taught me about the gap between evaluation metrics and production decisions [Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/) August 27, 202610 min read ![](../images/000.jpg) Photo by [Google DeepMind](https://unsplash.com/@googledeepmind?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/photos/a-large-group-of-colorful-blocks-lISkvdgfLEk?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) Somewhere in the middle of my final-year project, I trained six different models using the same fraud dataset, logged every training run, and picked a winner. Right now, the model that powers the current deployment is not the winning model. NairaShield is an AI-based fraud detection system that I built as my final year project for banking transactions. It started as a straightforward classification problem and ended up as something closer to a decision-support pipeline, partly because of that mismatch, and partly because my supervisor wasn't satisfied with the straightforward version. He really liked the project, but there was one little thing; he had seen an anti-money laundering project by one of my classmates and felt that the two were too similar. I tried so hard to explain that the modelling problem was different, but he still wasn't convinced. We had quite a few arguments over this matter before he told me what I needed to do to solve the problem at hand. And that single note is the reason the system ended up with a full regulatory review workflow and a second model family it wouldn't otherwise have had. It's also most of what this piece is actually about. This is the first thing I've properly written since I left school a few weeks ago, capping off a two-month stretch away from writing while coursework and this project ate up everything else, so it felt right to come back with the project that has the most to actually show. Learn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap. ## Merging two datasets that don't agree with each other I trained on two public datasets that have no common schema. [PaySim](https://www.kaggle.com/datasets/ealaxi/paysim1) is a simulation of mobile money transactions and has attributes like `oldbalanceOrg` and `newbalanceOrig`. On the other hand, the [IEEE-CIS fraud detection dataset](https://www.kaggle.com/competitions/ieee-fraud-detection) covers card transactions and has none of that; instead, it has only `ProductCD`, `card1`, `card2`, and a long tail of other anonymized features. Before any of that merging actually happens, each dataset gets validated against its own schema first, so a malformed row fails loudly instead of quietly poisoning the training set: python ``` class PaySimSchema(BaseModel): step: int type: str amount: float = Field(ge=0) nameOrig: str oldbalanceOrg: float newbalanceOrig: float nameDest: str oldbalanceDest: float newbalanceDest: float isFraud: int = Field(ge=0, le=1) isFlaggedFraud: int class IeeeCisSchema(BaseModel): TransactionID: int isFraud: int = Field(ge=0, le=1) TransactionDT: int TransactionAmt: float = Field(ge=0) ProductCD: str card1: Optional[float] = None card2: Optional[float] = None addr1: Optional[float] = None P_emaildomain: Optional[str] = None ``` Once both sides pass validation, mapping them into one shared structure looks like this: python ``` df_paysim_aligned["transaction_id"] = "PAYSIM_" + df_paysim["step"].astype(str) + "_" + df_paysim.index.astype(str) df_paysim_aligned["channel"] = df_paysim["type"].astype(str) product_map = {"W": "CARD_WEB", "H": "CARD_HOST", "C": "CARD_PHONE", "S": "CARD_STORE", "R": "CARD_RECURRING"} df_ieee_aligned["channel"] = df_ieee["ProductCD"].map(product_map).fillna("CARD_OTHER") ``` What took me way too long to realize was that having a unified schema meant the chosen features would have to be ones both datasets shared. The paySim balances didn't make the cut because IEEE-CIS doesn't have a similar field. The actual deployed model is trained using three things: the amount of transaction, one-hot encoding of the channel, and one-hot encoding of the source dataset. I'm flagging that now because it matters a lot later in this piece, and ignoring it would undercut a very good thing I am trying to say here. ## Balancing a dataset where fraud barely shows up Since there is very little fraud in both data sources, a model will be able to achieve high accuracy simply by predicting "no fraud" a lot of times while not predicting many fraud instances. I used SMOTE to oversample the minority class on the training split only, with a fallback to a hand-written interpolation version in case `imbalanced-learn` wasn't available in the deployment environment: python ``` k_neigh = min(2, sum(y_train == 1) - 1) if k_neigh >= 1: smote = SMOTE(random_state=42, k_neighbors=k_neigh) X_train_res, y_train_res = smote.fit_resample(X_train, y_train) else: print("[Warning] Insufficient minority class samples to perform SMOTE. Skipping resampling.") X_train_res, y_train_res = X_train, y_train ``` The zero dependency fallback uses interpolation between a sample and its closest neighbors through Euclidean distance to produce synthetic minority samples, which is essentially a simpler version of how SMOTE works internally but left in the code for the sake of preventing failure when in a minimalistic setting. ## Six models and a spreadsheet that argues with itself I trained Random Forest, Logistic Regression, XGBoost (both baseline and optimized versions), and LightGBM (again, both baseline and optimized). Every one of them ran through the same evaluation helper, so the numbers would actually be comparable instead of six slightly different measurement approaches wearing one result: python ``` def fit_eval_and_log(model, name, filename, params, fit_kwargs=None): print(f"\n--- Training Model: {name} ---") start_time = time.time() model.fit(X_train, y_train, **(fit_kwargs or {})) print(f"[Training Info] Model training completed in {time.time() - start_time:.3f} seconds.") preds = model.predict(X_test) probs = model.predict_proba(X_test)[:, 1] if hasattr(model, "predict_proba") else preds.astype(float) metrics = { "Accuracy": round(accuracy_score(y_test, preds), 6), "Precision": round(precision_score(y_test, preds, zero_division=0), 6), "Recall": round(recall_score(y_test, preds, zero_division=0), 6), "AUC-ROC": round(roc_auc_score(y_test, probs), 6), "AUC-PR": round(average_precision_score(y_test, probs), 6), } print(f"{name} Test AUC-PR:", metrics["AUC-PR"]) joblib.dump(model, filename) log_experiment_run(name, params, metrics) return metrics # Baseline vs. tuned, run back to back so the comparison is apples to apples xgb_base = xgb.XGBClassifier(use_label_encoder=False, eval_metric="logloss", random_state=42) fit_eval_and_log( xgb_base, "XGBoost (Baseline)", "xgboost_model.joblib", xgb_base.get_params(), fit_kwargs={"eval_set": [(X_train, y_train), (X_test, y_test)], "verbose": 10} ) xgb_tuned_params = tuned_params.get("xgboost", { "n_estimators": 150, "max_depth": 6, "learning_rate": 0.1, "subsample": 0.8, "colsample_bytree": 0.8, "random_state": 42 }) xgb_tuned = xgb.XGBClassifier(**xgb_tuned_params, use_label_encoder=False, eval_metric="logloss") fit_eval_and_log(xgb_tuned, "XGBoost (Tuned)", "xgboost_model_tuned.joblib", xgb_tuned_params) lgb_base = lgb.LGBMClassifier(random_state=42, verbose=-1) fit_eval_and_log(lgb_base, "LightGBM (Baseline)", "lightgbm_model.joblib", lgb_base.get_params()) ``` Every run, tuned or not, gets logged to `experiment_runs.json` with its own hyperparameters and metrics, which is the only reason it was possible to notice, months later, that the model in production wasn't the one at the top of that log: ![Bar chart comparing AUC-PR across six fraud detection models](../images/001.png) The model with the best AUC-PR isn't the one making the decisions in production. Image by author. A few things that are worth considering rather than skipping over. Logistic Regression shows the highest value of recall among all models at 0.95. But don't just look at that; it also proves to be the least usable model of the six, as its low precision value of 0.14 means that most of its identified cases of fraud are false positives. Tuning XGBoost improved AUC-PR slightly but actually made precision worse than the baseline, which is a reminder that hyperparameter search optimizes whatever metric you point it at, not necessarily the one that actually matters once the model is making real decisions. Finally, the baseline settings for LightGBM proved to be more efficient in terms of AUC-PR than optimized ones and were the best among all other models in the test. You can always look back up to the diagram above to see what I'm talking about. The production API still uses XGBoost (Tuned) as an active model, as this choice was made before the comparison of LightGBM models was completed. Which, when you look at it now, is honestly worth revisiting. SHAP is what makes any of these numbers accountable to a human. Every flagged prediction comes with a list of features of what pushed it toward fraud, since "the model said so" isn't an answer a bank or a regulator should accept: python ``` explainer = shap.TreeExplainer(model) shap_values = explainer.shap_values(X_instance) ``` ## What my supervisor actually wanted The anti-money laundering project by a colleague of mine from the beginning of this piece is worth returning to, because the similarity my supervisor saw wasn't imagined. On the surface, his work and mine were doing the same basic job, both taking transaction data and deciding whether it looked suspicious, and my supervisor wasn't wrong to notice that. What he was actually pushing me to build wasn't another algorithm. No. He was pointing me toward the step that comes after a transaction gets flagged, the part where something actually happens next. From what I found digging into [other academic AML work](https://www.sciencedirect.com/org/science/article/pii/S1368520125000123), most of it stops right at classification. His comment was what got me thinking about what follows after the flag. This led to the creation of a Regulatory Notification Center; a review layer built around role types modeled on Nigeria's Central Bank, the EFCC, and NDIC, each having a separate login and their own queue of flagged transactions. A regulatory reviewer can approve, ask for OTP verification, or block it outright, and each decision will be documented in the audit log with who acted and when. Building that workflow is also what forced me to include LightGBM in the comparison. ## Where the confidence gate actually comes from The API doesn't treat a fraud prediction as a single yes-or-no. The base classification threshold sits at 0.50, but what happens after that is split into bands: python ``` if response.get("prediction") == 1: fp = response.get("fraud_probability", 0.0) alert_payload = { "alert_id": f"API-ALT-{_random.randint(100000, 999999)}", "transaction_id": data.get("transaction_id", f"TXN-{_random.randint(10000, 99999)}"), "amount": float(data.get("amount", 0.0)), "timestamp": datetime.datetime.now().isoformat(), "model_probability": fp, "triggered_rules": ["MODEL_HIGH_FRAUD_RISK"] if fp >= 0.85 else ["MODEL_SUSPICIOUS_RISK"], "status": "BLOCKED" if fp >= 0.80 else "PENDING_OTP", "rag_context": response.get("rag_context", []), "sender_bank": data.get("sender_bank"), "regulatory_body": data.get("regulatory_body") } # Dispatch to SMS/Email notifier if it is critical (>= 85%) if _api_notifier is not None and fp >= 0.85: threading.Thread(target=_api_notifier.dispatch, args=(alert_payload,)).start() ``` Anything that scores less than 0.50 passes without any alarm whatsoever going off. If the score falls between 0.50 and 0.80, then the transaction becomes flagged but is not stopped; it's routed to PENDING_OTP, which basically means that a human or a secondary verification step gets involved before anything final happens. Scores from 0.80 and above block the transaction from happening, while at 0.85 an immediate alarm is sent either via SMS or email. The background thread sends it immediately without waiting for any request-response cycle. ![Diagram of a transaction](../images/002.png) A transaction never gets a single up-or-down answer. It's routed to whoever should be looking at it next: an analyst, a regulator, or nobody at all. Image by author. ··· ## Defense day The panel started with the usual, or would I say expected, set of questions. Which models I used, what hyperparameters I tuned and how I arrived at them, where the datasets came from, and whether they were representative enough of transaction behavior specifically. I told them XGBoost (Tuned), because that's what was actually in production. What I left out, mostly because I hadn't sat with it properly myself yet, was that LightGBM's baseline had already beaten it on the metric that mattered most, days before I stood in that room. Nobody followed up on it. I'm the one following up on it now. Then it moved to the one I had not really prepared for: what is special about this fraud detection solution compared to all the others that already exist, and what problems with other solutions does it solve? The scenario questions came next. If the CEO of a big company decides one random afternoon to move a million dollars in a single transaction, would it get flagged? And if a student attempted the same transaction, what would happen? I told them the system studies patterns and behavior, meaning it would treat those two transactions differently depending on who was making them. Writing this months later, with the code in front of me instead of a panel across the table, I want to give the more accurate answer instead of the more comfortable one. With the system as it is actually built now, those two transactions would receive an identical prediction. There's no account identity or transaction history in the thirteen features on which the model learns to look at how much money changed hands, through what channel, and how similar the transaction pattern is to either of the two source datasets. The honest answer is that my system currently treats a million dollars the same regardless of whose account it left. It sees a million dollars, and that's all, and it does nothing with that knowledge except treat it like any other transaction. The thing it does, though, is that it doesn't rely on that number alone to make a decision. In my system, if a million-dollar transaction is predicted to be fraud at a probability of 0.85, it is flagged, escalated, and in the regulatory workflow, is sent to a human for review. That human will have more context to work with than the model. The confidence gate is doing the job that account-level reasoning would do if it existed yet. It's a real answer, just a more modest one than the answer I gave that day. ## Final thoughts and what I'd actually do differently That leads us straight to the next step, which is the one the defense question revealed: adding behavioral features per account, averaging transactions, and deviation from historical user amounts. With time, the model should then be able to make the distinction I spoke of, rather than approximating it through the confidence gate. Additionally, I would revisit defaulting to XGBoost (Tuned) as the production model given LightGBM's baseline configuration scored better on the metric that matters most for this problem. Both of those come from the same mistake: I locked in a production model before the comparison was even finished, and only really noticed while writing this down. If there's one thing worth taking from all of it, it's that the best score on a spreadsheet isn't the same as the right production decision, and a model was never going to be the whole system by itself. The confidence gate, the audit log, the human reading a PENDING_OTP alert, that's doing as much of the real fraud-catching as the model is. I'm back to writing regularly after two months away, and this felt like the right project to be honest about first. There's a lot I haven't covered here: the RBAC setup, the audit logging, the regulatory notification pipeline itself; that's probably its own piece once this one's out in the world. ··· ## Before you go! I write about the real, messy engineering behind AI systems, where abstractions help, where they hurt, and what it takes to build reliably. You can [**subscribe to my newsletter**](https://thatcsguy.substack.com/?r=79lldp&utm_campaign=pub-share-checklist) if you'd like more of that. **Connect With Me** * [LinkedIn](http://www.linkedin.com/in/nwekebenjamin) * [Substack](https://thatcsguy.substack.com/) ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-4ad89a2fc5e69dde007d.jpeg) Written By ## [Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/) Benjamin Nweke is a Computer Science graduate and technical writer exploring AI, machine learning, and data science. He writes about building reliable AI systems, model evaluation, agentic architectures, and the practical challenges that appear when machine learning moves beyond experiments. His work focuses on making technically complex ideas clear, useful, and grounded in real implementation. [See all from Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/) [Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Fraud Detection](https://towardsdatascience.com/tag/fraud-detection/)[Mlops](https://towardsdatascience.com/tag/mlops/)[Model Deployment](https://towardsdatascience.com/tag/model-deployment/)[Python](https://towardsdatascience.com/tag/python/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&title=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&title=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fi-trained-six-models-for-fraud-detection-and-the-best-one-isnt-in-production%2F&text=I%20Trained%20Six%20Models%20for%20Fraud%20Detection%2C%20and%20the%20Best%20One%20Isn't%20in%20Production) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/003.jpg)](https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/) ## [How to simulate financial portfolios with Python](https://towardsdatascience.com/how-to-simulate-financial-portfolios-with-python-d0dc4b52a278/) [Data Science](https://towardsdatascience.com/category/data-science/) An application of Multidimensional Geometric Brownian Motion [Riccardo D'Elia](https://towardsdatascience.com/author/riccardodelia95/) November 9, 20205 min read * [![](../images/004.jpeg)](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/) ## [17 Quick Examples to Learn Python Lists](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) One of the most commonly used Python data structures Data structures are the key components... [Soner Yıldırım](https://towardsdatascience.com/author/sonery/) May 21, 20216 min read * [![](../images/body-63beb50f664a055ed739.jpg)](https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/) ## [Principal Component Analysis (PCA)](https://towardsdatascience.com/principal-component-analysis-pca-8133b02f11bd/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) A Physically Intuitive Mathematical Introduction [Lydia Nemec](https://towardsdatascience.com/author/lydianemec/) January 16, 20229 min read * [![](../images/006.jpg)](https://towardsdatascience.com/the-history-and-future-of-artificial-intelligence-through-the-lenses-of-computer-chess-and-legal-a-i-a7b7327f8800/) ## [The History and Future of Artificial Intelligence through the lenses of Computer Chess and Legal A.I](https://towardsdatascience.com/the-history-and-future-of-artificial-intelligence-through-the-lenses-of-computer-chess-and-legal-a-i-a7b7327f8800/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) An in-depth interview with Professor Jaap van den Herik [Emil Rijcken](https://towardsdatascience.com/author/emilrijcken/) December 30, 202115 min read * [![](../images/007.jpeg)](https://towardsdatascience.com/machine-learning-with-fuzzy-logic-52c85b46bfe4/) ## [Machine Learning with Fuzzy Logic](https://towardsdatascience.com/machine-learning-with-fuzzy-logic-52c85b46bfe4/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) How to extract control rules from data using Fuzzy logic [Nnamdi Uzoukwu](https://towardsdatascience.com/author/uzoukwu.nnamdi/) July 20, 20225 min read * [![](../images/008.jpg)](https://towardsdatascience.com/analysing-and-visualising-the-country-wise-population-from-1955-to-2020-with-pandas-matplotlib-70b3614eed6b/) ## [Analysing and Visualising the Country wise Population from 1955 to 2020 with Pandas, Matplotlib...](https://towardsdatascience.com/analysing-and-visualising-the-country-wise-population-from-1955-to-2020-with-pandas-matplotlib-70b3614eed6b/) [Data Science](https://towardsdatascience.com/category/data-science/) Day 1, 2, 3 of 100 Days of Data Science. [Durgesh Samariya](https://towardsdatascience.com/author/themlphdstudent/) August 17, 20206 min read * [![](../images/009.png)](https://towardsdatascience.com/5-must-know-date-and-time-functions-of-sql-3079cdb1a71e/) ## [5 Must-Know Date and Time Functions of SQL](https://towardsdatascience.com/5-must-know-date-and-time-functions-of-sql-3079cdb1a71e/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Practical guide with examples [Soner Yıldırım](https://towardsdatascience.com/author/sonery/) January 31, 20215 min read * [![](../images/010.png)](https://towardsdatascience.com/why-you-are-not-a-data-scientist-f56b5dee68f4/) ## [Why you are NOT a (Data) Scientist!](https://towardsdatascience.com/why-you-are-not-a-data-scientist-f56b5dee68f4/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Is Bill Gates a scientist? How about Linus Torvalds? If we don't call them scientists, why we call young data developers as data... [Shyam Sundar Dhanabalan](https://towardsdatascience.com/author/shyamdsundar/) April 15, 20217 min read * [![](../images/011.png)](https://towardsdatascience.com/from-perceptron-to-densenet-an-introduction-to-convolutional-neural-networks-ab37e3b7872e/) ## [Convolutional Neural Networks: An Introduction](https://towardsdatascience.com/from-perceptron-to-densenet-an-introduction-to-convolutional-neural-networks-ab37e3b7872e/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) A brief journey from Perceptron to ResNet [Oliver Kramer](https://towardsdatascience.com/author/oliver_kramer/) October 21, 202212 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)