保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Machine Learning

I Trained Six Models for Fraud Detection, and the Best One Isn't in Production

What a final-year project taught me about the gap between evaluation metrics and production decisions Benjamin Nweke August 27, 202610 min read Photo by Google DeepMind on Unsplash Somewhere in the middle of my final-year project, I trained six different models using the same fraud dataset, logged every training run, and picked a winner. Right now, the model that powers the current deployment is not the winning model. NairaShield is an AI-based fraud detection system that I built as my final year project for banking transactions. It started as a straightforward classification problem and ended up as something closer to a decision-support pipeline, partly because of that mismatch, and partly because my supervisor wasn't satisfied with the straightforward version. He really liked the project, but there was one little thing; he had seen an anti-money laundering project by one of my classmates and felt that the two were too similar. I tried so hard to explain that the modelling problem was different, but he still wasn't convinced. We had quite a few arguments over this matter before he told me what I needed to do to solve the problem at hand. And that single note is the reason the system ended up with a full regulatory review workflow and a second model family it wouldn't otherwise have had. It's also most of what this piece is actually about. This is the first thing I've properly written since I left school a few weeks ago, capping off a two-month stretch away from writing while coursework and this project ate up everything else, so it felt right to come back with the project that has the most to actually show. Learn this step by step with the interactive Machine Learning roadmap.

Merging two datasets that don't agree with each other

I trained on two public datasets that have no common schema. PaySim is a simulation of mobile money transactions and has attributes like oldbalanceOrg and newbalanceOrig. On the other hand, the IEEE-CIS fraud detection dataset covers card transactions and has none of that; instead, it has only ProductCD, card1, card2, and a long tail of other anonymized features. Before any of that merging actually happens, each dataset gets validated against its own schema first, so a malformed row fails loudly instead of quietly poisoning the training set: python

class PaySimSchema(BaseModel):
    step: int
    type: str
    amount: float = Field(ge=0)
    nameOrig: str
    oldbalanceOrg: float
    newbalanceOrig: float
    nameDest: str
    oldbalanceDest: float
    newbalanceDest: float
    isFraud: int = Field(ge=0, le=1)
    isFlaggedFraud: int

class IeeeCisSchema(BaseModel):
    TransactionID: int
    isFraud: int = Field(ge=0, le=1)
    TransactionDT: int
    TransactionAmt: float = Field(ge=0)
    ProductCD: str
    card1: Optional[float] = None
    card2: Optional[float] = None
    addr1: Optional[float] = None
    P_emaildomain: Optional[str] = None

Once both sides pass validation, mapping them into one shared structure looks like this: python

df_paysim_aligned["transaction_id"] = "PAYSIM_" + df_paysim["step"].astype(str) + "_" + df_paysim.index.astype(str)
df_paysim_aligned["channel"] = df_paysim["type"].astype(str)

product_map = {"W": "CARD_WEB", "H": "CARD_HOST", "C": "CARD_PHONE", "S": "CARD_STORE", "R": "CARD_RECURRING"}
df_ieee_aligned["channel"] = df_ieee["ProductCD"].map(product_map).fillna("CARD_OTHER")

What took me way too long to realize was that having a unified schema meant the chosen features would have to be ones both datasets shared. The paySim balances didn't make the cut because IEEE-CIS doesn't have a similar field. The actual deployed model is trained using three things: the amount of transaction, one-hot encoding of the channel, and one-hot encoding of the source dataset. I'm flagging that now because it matters a lot later in this piece, and ignoring it would undercut a very good thing I am trying to say here.

Balancing a dataset where fraud barely shows up

Since there is very little fraud in both data sources, a model will be able to achieve high accuracy simply by predicting "no fraud" a lot of times while not predicting many fraud instances. I used SMOTE to oversample the minority class on the training split only, with a fallback to a hand-written interpolation version in case imbalanced-learn wasn't available in the deployment environment: python

k_neigh = min(2, sum(y_train == 1) - 1)
if k_neigh >= 1:
    smote = SMOTE(random_state=42, k_neighbors=k_neigh)
    X_train_res, y_train_res = smote.fit_resample(X_train, y_train)
else:
    print("[Warning] Insufficient minority class samples to perform SMOTE. Skipping resampling.")
    X_train_res, y_train_res = X_train, y_train

The zero dependency fallback uses interpolation between a sample and its closest neighbors through Euclidean distance to produce synthetic minority samples, which is essentially a simpler version of how SMOTE works internally but left in the code for the sake of preventing failure when in a minimalistic setting.

Six models and a spreadsheet that argues with itself

I trained Random Forest, Logistic Regression, XGBoost (both baseline and optimized versions), and LightGBM (again, both baseline and optimized). Every one of them ran through the same evaluation helper, so the numbers would actually be comparable instead of six slightly different measurement approaches wearing one result: python

def fit_eval_and_log(model, name, filename, params, fit_kwargs=None):
    print(f"\n--- Training Model: {name} ---")
    start_time = time.time()
    model.fit(X_train, y_train, **(fit_kwargs or {}))
    print(f"[Training Info] Model training completed in {time.time() - start_time:.3f} seconds.")

    preds = model.predict(X_test)
    probs = model.predict_proba(X_test)[:, 1] if hasattr(model, "predict_proba") else preds.astype(float)

    metrics = {
        "Accuracy": round(accuracy_score(y_test, preds), 6),
        "Precision": round(precision_score(y_test, preds, zero_division=0), 6),
        "Recall": round(recall_score(y_test, preds, zero_division=0), 6),
        "AUC-ROC": round(roc_auc_score(y_test, probs), 6),
        "AUC-PR": round(average_precision_score(y_test, probs), 6),
    }
    print(f"{name} Test AUC-PR:", metrics["AUC-PR"])

    joblib.dump(model, filename)
    log_experiment_run(name, params, metrics)
    return metrics

# Baseline vs. tuned, run back to back so the comparison is apples to apples
xgb_base = xgb.XGBClassifier(use_label_encoder=False, eval_metric="logloss", random_state=42)
fit_eval_and_log(
    xgb_base, "XGBoost (Baseline)", "xgboost_model.joblib", xgb_base.get_params(),
    fit_kwargs={"eval_set": [(X_train, y_train), (X_test, y_test)], "verbose": 10}
)

xgb_tuned_params = tuned_params.get("xgboost", {
    "n_estimators": 150, "max_depth": 6, "learning_rate": 0.1,
    "subsample": 0.8, "colsample_bytree": 0.8, "random_state": 42
})
xgb_tuned = xgb.XGBClassifier(**xgb_tuned_params, use_label_encoder=False, eval_metric="logloss")
fit_eval_and_log(xgb_tuned, "XGBoost (Tuned)", "xgboost_model_tuned.joblib", xgb_tuned_params)

lgb_base = lgb.LGBMClassifier(random_state=42, verbose=-1)
fit_eval_and_log(lgb_base, "LightGBM (Baseline)", "lightgbm_model.joblib", lgb_base.get_params())

Every run, tuned or not, gets logged to experiment_runs.json with its own hyperparameters and metrics, which is the only reason it was possible to notice, months later, that the model in production wasn't the one at the top of that log: Bar chart comparing AUC-PR across six fraud detection models The model with the best AUC-PR isn't the one making the decisions in production. Image by author. A few things that are worth considering rather than skipping over. Logistic Regression shows the highest value of recall among all models at 0.95. But don't just look at that; it also proves to be the least usable model of the six, as its low precision value of 0.14 means that most of its identified cases of fraud are false positives. Tuning XGBoost improved AUC-PR slightly but actually made precision worse than the baseline, which is a reminder that hyperparameter search optimizes whatever metric you point it at, not necessarily the one that actually matters once the model is making real decisions. Finally, the baseline settings for LightGBM proved to be more efficient in terms of AUC-PR than optimized ones and were the best among all other models in the test. You can always look back up to the diagram above to see what I'm talking about. The production API still uses XGBoost (Tuned) as an active model, as this choice was made before the comparison of LightGBM models was completed. Which, when you look at it now, is honestly worth revisiting. SHAP is what makes any of these numbers accountable to a human. Every flagged prediction comes with a list of features of what pushed it toward fraud, since "the model said so" isn't an answer a bank or a regulator should accept: python

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_instance)

What my supervisor actually wanted

The anti-money laundering project by a colleague of mine from the beginning of this piece is worth returning to, because the similarity my supervisor saw wasn't imagined. On the surface, his work and mine were doing the same basic job, both taking transaction data and deciding whether it looked suspicious, and my supervisor wasn't wrong to notice that. What he was actually pushing me to build wasn't another algorithm. No. He was pointing me toward the step that comes after a transaction gets flagged, the part where something actually happens next. From what I found digging into other academic AML work, most of it stops right at classification. His comment was what got me thinking about what follows after the flag. This led to the creation of a Regulatory Notification Center; a review layer built around role types modeled on Nigeria's Central Bank, the EFCC, and NDIC, each having a separate login and their own queue of flagged transactions. A regulatory reviewer can approve, ask for OTP verification, or block it outright, and each decision will be documented in the audit log with who acted and when. Building that workflow is also what forced me to include LightGBM in the comparison.

Where the confidence gate actually comes from

The API doesn't treat a fraud prediction as a single yes-or-no. The base classification threshold sits at 0.50, but what happens after that is split into bands: python

if response.get("prediction") == 1:
    fp = response.get("fraud_probability", 0.0)
    alert_payload = {
        "alert_id": f"API-ALT-{_random.randint(100000, 999999)}",
        "transaction_id": data.get("transaction_id", f"TXN-{_random.randint(10000, 99999)}"),
        "amount": float(data.get("amount", 0.0)),
        "timestamp": datetime.datetime.now().isoformat(),
        "model_probability": fp,
        "triggered_rules": ["MODEL_HIGH_FRAUD_RISK"] if fp >= 0.85 else ["MODEL_SUSPICIOUS_RISK"],
        "status": "BLOCKED" if fp >= 0.80 else "PENDING_OTP",
        "rag_context": response.get("rag_context", []),
        "sender_bank": data.get("sender_bank"),
        "regulatory_body": data.get("regulatory_body")
    }

    # Dispatch to SMS/Email notifier if it is critical (>= 85%)
    if _api_notifier is not None and fp >= 0.85:
        threading.Thread(target=_api_notifier.dispatch, args=(alert_payload,)).start()

Anything that scores less than 0.50 passes without any alarm whatsoever going off. If the score falls between 0.50 and 0.80, then the transaction becomes flagged but is not stopped; it's routed to PENDING_OTP, which basically means that a human or a secondary verification step gets involved before anything final happens. Scores from 0.80 and above block the transaction from happening, while at 0.85 an immediate alarm is sent either via SMS or email. The background thread sends it immediately without waiting for any request-response cycle. Diagram of a transaction A transaction never gets a single up-or-down answer. It's routed to whoever should be looking at it next: an analyst, a regulator, or nobody at all. Image by author. ···

Defense day

The panel started with the usual, or would I say expected, set of questions. Which models I used, what hyperparameters I tuned and how I arrived at them, where the datasets came from, and whether they were representative enough of transaction behavior specifically. I told them XGBoost (Tuned), because that's what was actually in production. What I left out, mostly because I hadn't sat with it properly myself yet, was that LightGBM's baseline had already beaten it on the metric that mattered most, days before I stood in that room. Nobody followed up on it. I'm the one following up on it now. Then it moved to the one I had not really prepared for: what is special about this fraud detection solution compared to all the others that already exist, and what problems with other solutions does it solve? The scenario questions came next. If the CEO of a big company decides one random afternoon to move a million dollars in a single transaction, would it get flagged? And if a student attempted the same transaction, what would happen? I told them the system studies patterns and behavior, meaning it would treat those two transactions differently depending on who was making them. Writing this months later, with the code in front of me instead of a panel across the table, I want to give the more accurate answer instead of the more comfortable one. With the system as it is actually built now, those two transactions would receive an identical prediction. There's no account identity or transaction history in the thirteen features on which the model learns to look at how much money changed hands, through what channel, and how similar the transaction pattern is to either of the two source datasets. The honest answer is that my system currently treats a million dollars the same regardless of whose account it left. It sees a million dollars, and that's all, and it does nothing with that knowledge except treat it like any other transaction. The thing it does, though, is that it doesn't rely on that number alone to make a decision. In my system, if a million-dollar transaction is predicted to be fraud at a probability of 0.85, it is flagged, escalated, and in the regulatory workflow, is sent to a human for review. That human will have more context to work with than the model. The confidence gate is doing the job that account-level reasoning would do if it existed yet. It's a real answer, just a more modest one than the answer I gave that day.

Final thoughts and what I'd actually do differently

That leads us straight to the next step, which is the one the defense question revealed: adding behavioral features per account, averaging transactions, and deviation from historical user amounts. With time, the model should then be able to make the distinction I spoke of, rather than approximating it through the confidence gate. Additionally, I would revisit defaulting to XGBoost (Tuned) as the production model given LightGBM's baseline configuration scored better on the metric that matters most for this problem. Both of those come from the same mistake: I locked in a production model before the comparison was even finished, and only really noticed while writing this down. If there's one thing worth taking from all of it, it's that the best score on a spreadsheet isn't the same as the right production decision, and a model was never going to be the whole system by itself. The confidence gate, the audit log, the human reading a PENDING_OTP alert, that's doing as much of the real fraud-catching as the model is. I'm back to writing regularly after two months away, and this felt like the right project to be honest about first. There's a lot I haven't covered here: the RBAC setup, the audit logging, the regulatory notification pipeline itself; that's probably its own piece once this one's out in the world. ···

Before you go!

I write about the real, messy engineering behind AI systems, where abstractions help, where they hurt, and what it takes to build reliably. You can subscribe to my newsletter if you'd like more of that. Connect With Me

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Benjamin Nweke

Benjamin Nweke is a Computer Science graduate and technical writer exploring AI, machine learning, and data science. He writes about building reliable AI systems, model evaluation, agentic architectures, and the practical challenges that appear when machine learning moves beyond experiments. His work focuses on making technically complex ideas clear, useful, and grounded in real implementation. See all from Benjamin Nweke Artificial IntelligenceFraud DetectionMlopsModel DeploymentPython Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

How to simulate financial portfolios with Python

Data Science An application of Multidimensional Geometric Brownian Motion Riccardo D'Elia November 9, 20205 min read

17 Quick Examples to Learn Python Lists

Artificial Intelligence One of the most commonly used Python data structures Data structures are the key components... Soner Yıldırım May 21, 20216 min read

Principal Component Analysis (PCA)

Machine Learning A Physically Intuitive Mathematical Introduction Lydia Nemec January 16, 20229 min read

The History and Future of Artificial Intelligence through the lenses of Computer Chess and Legal A.I

Artificial Intelligence An in-depth interview with Professor Jaap van den Herik Emil Rijcken December 30, 202115 min read

Machine Learning with Fuzzy Logic

Machine Learning How to extract control rules from data using Fuzzy logic Nnamdi Uzoukwu July 20, 20225 min read

Analysing and Visualising the Country wise Population from 1955 to 2020 with Pandas, Matplotlib...

Data Science Day 1, 2, 3 of 100 Days of Data Science. Durgesh Samariya August 17, 20206 min read

5 Must-Know Date and Time Functions of SQL

Artificial Intelligence Practical guide with examples Soner Yıldırım January 31, 20215 min read

Why you are NOT a (Data) Scientist!

Machine Learning Is Bill Gates a scientist? How about Linus Torvalds? If we don't call them scientists, why we call young data developers as data... Shyam Sundar Dhanabalan April 15, 20217 min read

Convolutional Neural Networks: An Introduction

Deep Learning A brief journey from Perceptron to ResNet Oliver Kramer October 21, 202212 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network