{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Angela Shi & Kezhan Shi",
  "title_listed": "RAG Is Not Machine Learning, and the ML Toolkit Solves the Wrong Problem",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem/",
  "publish_date": "2026-06-01",
  "corpus_stage": "",
  "notes": "",
  "id": "8c14ee4978d7",
  "collected_at": "2026-10-02T01:14:53",
  "title": "RAG Is Not Machine Learning, and the ML Toolkit Solves the Wrong Problem | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\n# RAG Is Not Machine Learning, and the ML Toolkit Solves the Wrong Problem\nEnterprise Document Intelligence [Vol.1 #3] - Why the ML toolkit (hyperparameter sweeps, train/test splits, explainability frameworks) solves the wrong problem, and what to use instead\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nJune 1, 202625 min read\n![Why this image: an instrument for examining the small print, what this article does to the casual claim that RAG is an ML problem, looking closely at where the ML toolkit fits and where it does not.](../images/000.jpg)\n[image](https://unsplash.com/pt-br/fotografias/lupa-preta-em-papel-branco-da-impressora-O7LDeH0Qo_0) by Teslariu Mihai via Unsplash\n**A development team** dedicated six months to fine-tuning their RAG pipeline.\n  * They ran five Optuna sweeps.\n  * They added a custom reranker.\n  * They fine-tuned an embedding model on their own data.\n\n\nProduction accuracy never moved. Pilots kept complaining about the same wrong answers. Six months in, the bug was in the parser.\nThe team was lost, not stuck. **RAG is not machine learning, and the ML toolkit solves the wrong problem.** This is the single most expensive misconception in enterprise RAG today. It costs months of careful work, the wrong people on the wrong tasks, and a quiet erosion of trust in the system.\nRAG looks enough like machine learning that the ML toolkit feels like the natural next step. The instincts (hyperparameter optimization, evaluation datasets, explainability frameworks) are not wrong in isolation. They are imported from the wrong field. The methods that work for training models don’t work for assembling search systems.\nThe point is not that ML is bad. The embedding model that powers vector search is itself a deep learning model, but you don’t train it, you consume it. The point is that **the system you’re building around it is not a model** , and treating it as one wastes time, picks the wrong metrics, hires the wrong people, and hides the real failure modes.\nThe \"RAG is not ML\" position is one piece of _Enterprise Document Intelligence_ [Volume 1](https://towardsdatascience.com/document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-scale/), which builds enterprise RAG brick by brick. The four bricks (parsing, question parsing, retrieval, generation) are the engineering toolkit this article points to.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## 1. Two different problems\nMachine learning solves problems where the true answer is unknown and has to be predicted. Will this customer churn? What’s the probability this transaction is fraud? Is this image a cat? You don’t know the answer in advance. That’s why you train a model. The model learns from labeled examples, generalizes to new inputs, and produces a prediction. Performance is measured in aggregate, across thousands of test cases, because individual predictions can be wrong while the model is still useful overall.\nRAG solves a different problem. The answer to “what is the effective date of this contract?” exists, written on page one of the document, or it doesn’t exist anywhere. There’s nothing to predict. The system either finds the answer in the document and reports it faithfully, or it fails and should say so. Performance is binary at the question level (got it or didn’t) even if you measure aggregate rates across many questions.\nThese differences are concrete:\n  * In ML, “the model was wrong on 8% of cases” is a feature of the system. You build redundancy, downstream checks, human review for the borderline cases. In RAG, “the system gave a wrong answer 8% of the time” is a bug. Each of those 8% has a specific cause: the wrong passage was retrieved, the right passage was retrieved but the model paraphrased it badly, the answer wasn’t in the corpus and the system made one up. They aren’t statistical noise to optimize on average. They’re individually fixable failures.\n  * In ML, you can’t generally tell _why_ the model got a particular case wrong. That’s why explainability is a research field. In RAG, you can always tell. The retrieval logs which passages it returned. The generator saw exactly those passages. If the answer is wrong, you walk the chain backward and find the broken link. There’s nothing hidden.\n  * In ML, the model improves by training on more data. In RAG, the system improves by indexing better, parsing more carefully, retrieving more precisely, prompting more clearly. None of that is training. It’s engineering.\n\n\nThat difference changes which tools you reach for when something breaks.\nThe cases catalogued in Article 2 fall exactly here: negation, exact identifiers, internal acronyms, signal dilution in long context, topical proximity outranking the actual answer. None of those move when you swap embedding models or sweep chunk sizes. They aren’t bugs a model can learn its way out of, because there is no labeled signal saying _“this is the right line”_ for the model to train on. The fix is structural (question parsing, expert keywords, retrieval that knows the document’s structure), and the next sections walk through the three ML reflexes that pick the wrong tool instead.\n## 2. Three arguments that don’t apply\nThree ML methods get imported into RAG projects by default: hyperparameter optimization, evaluation datasets with train/test splits, and feature-attribution explainability. Each is reasonable inside ML. Each misfires here.\n### 2.1 The hyperparameter argument\nThe most common framing goes something like this: chunk size, overlap, top-k, similarity threshold. These are hyperparameters, and you should optimize them the way you optimize ML models, using tools like Optuna or Ray Tune. Run a sweep, plot the curves, pick the best configuration.\nIn these setups, `top_k` is the number of passages the retriever keeps, and `similarity_threshold` is the minimum cosine score a passage must reach to qualify. The code below declares all four as numbers to optimize:\npython\n\n```\n# What teams typically write (and why it's the wrong activity)\nimport optuna\ndef objective(trial):\n    chunk_size    = trial.suggest_int(\"chunk_size\", 100, 2000)\n    chunk_overlap = trial.suggest_int(\"chunk_overlap\", 0, 200)\n    top_k         = trial.suggest_int(\"top_k\", 1, 20)\n    threshold     = trial.suggest_float(\"threshold\", 0.5, 0.95)\n    accuracy = run_rag_pipeline_and_score(\n        chunk_size, chunk_overlap, top_k, threshold\n    )\n    return accuracy\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective, n_trials=200)  # two weeks of compute later...\n```\n\nThere’s a grain of truth here. These variables do affect retrieval quality, and they are worth tuning. The trouble starts with the word “hyperparameter,” which brings in a metaphor with hidden assumptions.\nIn machine learning, a hyperparameter controls how a model learns: learning rate, regularization strength, number of layers. The model itself is what changes during training; the hyperparameter shapes that change. In RAG, there is no learning. The chunk size doesn’t control how something learns. It controls how a function splits text, the same way every time, regardless of what you’ve fed it before.\nWhat looks like a hyperparameter is a configuration choice, the kind you’d make when configuring a search engine. The expertise needed to tune it well isn’t statistical optimization. It’s understanding the structure of your documents and the shape of your questions. Chunk size of 512 tokens may work beautifully on dense academic papers and disastrously on insurance contracts where a single clause spans 800 tokens and breaking it in half loses the conditional that gives the clause its meaning. No grid search will tell you that. You need to read your documents.\nThis is why teams who grid-search chunk size often find a “best” value that performs marginally better on the test set and identically on production data. The optimum on the test set was an artifact of the test set, not a genuine improvement in the underlying system. They’ve optimized a number, not solved a problem.\n> **Common pitfall:** A team running Optuna over chunk_size, top_k, and similarity_threshold for two weeks, ending up at chunk_size=487 with no idea why. The honest answer to “why 487?” is “because Optuna said so.” That answer doesn’t survive a real production failure, and it doesn’t generalize when the document distribution shifts. A chunk size of 500 chosen because that’s roughly the size of a paragraph in this corpus is more defensible than 487 chosen because a sweep landed there.\nThe right activity isn’t tuning numbers. It’s deciding **structurally** how to chunk. By section? By paragraph? By the table of contents entries? By question type, with different chunkers for short lookups vs long clauses? Answered by looking at documents and questions, not by optimization curves.\nThere’s a deeper reason chunk size resists optimization: by construction, no single chunk size can serve every question. Take two questions on the same insurance contract:\n  * _“What is the effective date?”_ The answer is one line, somewhere on page one. It wants a chunk small enough to pin down a single line precisely.\n  * _“What are the exclusions of the policy?”_ The answer might be one page, or three pages, depending on how the insurer wrote it. It wants a chunk large enough to capture an entire section.\n\n\nThere is no number that satisfies both. A chunk size of 200 tokens chops the exclusions section into incoherent fragments. A chunk size of 2000 tokens buries the effective date in surrounding noise.\nTrying to find “the best chunk size” is therefore not a tuning problem. The framing itself is broken: no single number can serve a distribution of questions whose answers have different lengths.\nYou could, in principle, make chunk size respond to the question by training a small model that predicts the right chunker from the question’s features: classify the intent, regress over the expected answer length, output a strategy. That would be machine learning applied legitimately, on a problem where something is being learned.\nBut you don’t need to. You can write the rule down. Look at a question and you can tell whether it asks for a date, a section, or a comparison. So can a domain expert. So can ten lines of Python with hand-written conditions over keywords. The deeper reason RAG isn’t machine learning is that, for most of the decisions inside the system, **you already know the answer** , or someone on your team does. Machine learning is the tool for problems where nobody knows the answer in advance.\nThe right approach is to stop looking for one chunk size and start routing different question types to different retrieval strategies:\npython\n\n```\n# What to do instead: route by question type\ndef chunk_for_question(question: str, line_df, toc_df):\n    intent = classify_intent(question)\n    if intent == \"point_lookup\":          # \"what is the effective date?\"\n        return chunk_by_line(line_df)\n    elif intent == \"section_retrieval\":   # \"what are the exclusions?\"\n        return chunk_by_toc_section(line_df, toc_df)\n    elif intent == \"comparison\":          # \"compare clauses A and B\"\n        return chunk_by_full_section(line_df, toc_df)\n```\n\nThe two code blocks above are the entire argument of this section. The first runs Optuna over four numbers for two weeks and produces a value nobody can defend. The second makes one structural decision per question type and produces a system whose behavior anyone can explain.\nLater articles develop how to classify intent (Article 6, on question understanding) and how the different retrieval methods and granularities are implemented (Article 7, on retrieval). The point here is just that the activity isn’t tuning, it’s routing.\n### 2.2 The evaluation dataset argument\nThe next ML import is evaluation method. The reasoning goes: RAG, like any ML system, needs a proper evaluation dataset: questions paired with expected answers, split into train and test sets, scored with precision and recall. Frameworks like RAGAS have made this even more tempting, offering metrics for faithfulness, answer relevancy, and context recall that look satisfyingly ML-ish.\nEvaluation is useful. The issue isn’t whether to evaluate. It’s what the metrics mean. In machine learning, evaluation tells you whether a model has generalized from training data to unseen examples. The train/test split exists because you want to detect overfitting: a model that memorized the training set rather than learning a transferable pattern.\nIn RAG, there is nothing to generalize. Overfitting (when a model memorizes training examples rather than learning a pattern that transfers to new data) cannot happen here: the system does not change between queries. The retriever computes the same cosine distances every time. The generator follows the same prompt template. There is no model adjusting to data.\nWhat evaluation measures in RAG is three things, all of which are coverage and quality questions, not statistical generalization:\n  * **Does my corpus contain the answer?** If not, the system can’t find it. This is a content question, not a model question.\n  * **Does my retriever find the right passage?** If the answer is in the corpus but the retriever missed it, the system fails. This is a search question.\n  * **Does my generator stay faithful to what was retrieved?** If the right passage was retrieved but the model paraphrased it incorrectly or hallucinated extras, the system fails. This is a generation discipline question.\n\n\nEach one points to a specific fix. Mixing them up under an aggregate “accuracy” score loses information. A 75% accuracy from “corpus is missing 25% of the documented topics” demands different action than a 75% accuracy from “retriever misses the right passage 25% of the time.” The first calls for ingesting more documents. The second calls for fixing the retriever. An aggregate metric that treats them the same hides the diagnostic.\nThis also explains why teams using RAGAS-style frameworks sometimes report great metrics on a held-out test set and then watch the system fail in production. The test set covered topics where the corpus had answers and the retriever happened to find them. Production has questions whose answers are not in the corpus at all, and the system either hallucinates or fails to say “not found.” The metric was high on the test set because the test set was friendly. The system isn’t broken. The evaluation was.\nWhat you need to evaluate, broken down by question type, takes about ten lines:\npython\n\n```\n# Retrieval recall, per question, per intent\ndef evaluate_retrieval(reference_set, retrieve_fn):\n    rows = []\n    for ref in reference_set:\n        retrieved_lines = retrieve_fn(ref.question)\n        recall = len(set(retrieved_lines) & set(ref.expected_lines)) / len(ref.expected_lines)\n        rows.append({\n            \"question\": ref.question,\n            \"intent\":   ref.intent,\n            \"recall\":   recall,\n            \"hit\":      recall > 0,\n        })\n    return pd.DataFrame(rows)\n# Always break down by question type, never just an aggregate\ndf.groupby(\"intent\")[\"hit\"].mean()\n# point_lookup        0.92\n# section_retrieval   0.41   <-- this is the real problem\n# comparison          0.55\n```\n\nA single aggregate accuracy of 63% would have hidden the catastrophe on section_retrieval. The per-intent breakdown reveals it instantly. Recall here means: on questions where the answer exists in the corpus, did the retriever find the right passage? Grouping by intent (point_lookup, section_retrieval, …) shows _which kind_ of question fails, and therefore _which part_ of the pipeline to fix.\nRAG has two evaluation surfaces with very different shapes.\n**The retrieval surface** is a search problem: did the right passage land in front of the model? Measuring this means checking, on a reference set of questions, whether the relevant lines or pages were retrieved at all. The metric is recall at the level you care about (recall at line, at page, at section) and it’s specific to your corpus. Nobody else can run this evaluation for you. Your corpus is unique. This is where the bulk of evaluation effort belongs.\n**The generation surface** is different. Once the right passage has been retrieved, the question becomes: did the model produce a faithful answer, in the right format, with proper citations, and a clean “not found” when the passage didn’t contain the answer? Some of this you do evaluate yourself, but a large part is already evaluated by the LLM vendors. OpenAI, Anthropic, and Mistral spend enormous resources testing whether their models follow JSON schemas, refuse to invent, and respect prompt instructions. Those are the dimensions on which they improve their models. As a RAG builder, you’re not training the generator. You’re **consuming** it. If the model fails badly at returning structured JSON or stays unfaithful to its inputs, you’ll notice within an hour of integration. That’s not a metric to set up; it’s a sanity check that’s either obvious or fine.\nWhat this means in practice: most of your evaluation time should go into retrieval (which is corpus-specific and only you can do it), not into generation (which is mostly the vendor’s problem, and which shows obvious failures fast). Teams that spend weeks building elaborate generation evaluation suites are usually putting off the harder retrieval work that would improve the result.\n> **Going further:** _Evaluating Your System_ (later in the series) walks through how to build a reference set for your specific corpus, the four metrics that matter, and why per-question-type metrics are essential while aggregate metrics are misleading.\n### 2.3 The explainability argument\nMachine learning has its own toolkit for explainability. SHAP values to attribute predictions to features. LIME for local approximations of complex models. Attention visualization for transformers. When people start asking for RAG explainability (“why did the system give this answer?”) they naturally turn to these tools. They want to score retrieval relevance, weight document contributions, visualize which tokens influenced the output.\nThe irony is that RAG is more explainable by design than most ML models. There’s no need for SHAP. There’s no opacity to crack open. The system retrieved these specific passages from these specific sources, and the answer was built on top of them. That **is** the explanation. It’s documentary, not statistical.\nThis points to a deeper asymmetry between machine learning and RAG. In machine learning, the human has intuition but cannot quantify. Ask who survived the Titanic and people say _wealth_ , _age_ , _class_ : none wrong, none precise. The model has no such doubt: fit a decision tree and the root split is _sex_ , the next cut is an exact age threshold nobody would have guessed, then class. Every split is a number intuition alone could not have produced. The model exists to put those numbers down.\n![](../images/001.png)\n_A real sklearn decision tree on Titanic data. Every threshold is a number intuition couldn’t produce - Image by author_\nFor text data, the direction reverses. The user can read the source. A lawyer scanning a contract sees the conditions, the exceptions, the dates. A compliance officer reads a policy and knows whether a behavior breaches it. The text doesn’t hide its meaning, and the expert is already a fluent reader.\nThere are exceptions: sarcasm and irony are the classic ones, where modern LLMs sometimes catch what a literal reader misses. But in enterprise contexts the user is the domain expert.\nThe model isn’t there to explain the text. It’s there to do the reading at corpus scale, and a citation is enough to let the expert verify any answer in seconds.\nWhen a user asks “why this answer?”, the right response isn’t a heatmap of attention weights or a feature attribution score. It’s: “I looked at pages 12, 47, and 89 of this contract. Here’s the exact text I used. The answer follows from that text.” If the user disagrees with the answer, they can read the source themselves and judge. They don’t need an explainability framework. They need a citation.\nThe fifty-line pipeline from Article 1 already showed this. The prompt asked the model to return the start and end line numbers (with their pages) alongside the answer, in a structured JSON; the annotator then highlighted those exact lines on the PDF. No SHAP, no LIME, no attention visualization, no specialized observability platform. The “explanation” was a side product of how the prompt was written. The citation is part of the answer, not an analysis layer added on top.\nThe trace is the explanation. Reading it requires no interpretation, just reading.\nImporting ML explainability into RAG is solving a problem that doesn’t exist. SHAP on a retrieval score is using a scalpel to open a mailbox. The retrieval score is already a number you computed on inputs you can read. There’s nothing to attribute that you don’t already see.\nThe deeper failure of the ML-explainability framing is that it makes you focus on the wrong thing. You start trying to explain _why_ a particular passage scored higher than another in vector space, a near-impossible question that doesn’t matter. What matters is whether the right passage was retrieved at all, and whether the answer faithfully reflects it. Those are questions you can answer by reading the logs and the source. No tooling needed.\n## 3. What changes when you see RAG correctly\nOnce you stop treating RAG as ML, two things change. The day-to-day tools, metrics and people reorganize around search rather than training. And a deeper question (_where the intelligence sits_) moves from the model to the team. Both come from the same framing.\n### 3.1 Tools, metrics, people\nThree concrete things change.\n**The tools change:** You don’t need PyTorch, or a training cluster, or hyperparameter optimization frameworks for the system itself. You need a good parser, a flexible retriever, careful prompt engineering, and structured logging of everything that happens. The components that _are_ ML (the embedding model, the LLM) you consume as services. They’re commodity inputs, not things you build or train.\n**The metrics change:** Aggregate accuracy gives way to per-failure-mode metrics: retrieval recall (did we find the right passage?), answer faithfulness (did the model stick to it?), extraction accuracy (when extracting structured data, did the values match?), not-found rate (when the answer isn’t in the corpus, did we say so cleanly?). Each measures something specific, each maps to a specific part of the pipeline you can fix.\n**The people change:** A pure ML team trying to ship a RAG system often misses what makes it work, and what makes it fail. The skills that matter most are software engineering (the system has many moving parts that need to compose cleanly), domain expertise (someone has to know what a good answer to a domain question even looks like), and information retrieval intuition (someone has to think like a search engine designer, not a model trainer). ML expertise is useful, but it’s not the dominant skill. A team of ML researchers and no domain expert will produce a beautifully tuned system that misses the point. A team with one ML-aware engineer, two software engineers, and one domain expert will usually outperform it.\n### 3.2 Where the intelligence sits\nThe shift in people points to a deeper question: where does the intelligence of the system live?\nIn an ML system the intelligence lives in the model. The model holds the patterns. The team feeds it training data and tunes the loss function. In a RAG system the intelligence lives in the team. The lawyer knows which clauses to look at first. The underwriter knows what “deductible” means, and which page usually carries it. The compliance officer knows which regulation applies to which product. None of that lives inside the embedding model. None of it comes out of a hyperparameter sweep. It already lives in the heads of people who have read these documents for years.\nWatch an underwriter open a new policy. She doesn’t read it linearly. She jumps to the exclusions section first because she’s read five hundred of these and knows that’s where the trap usually lives. She checks the schedule of benefits for the deductibles and ceilings. She checks the territory clause. Three minutes in, she has a clearer view of the contract than any embedding model would produce on a thousand of those contracts. That habit is what the system has to amplify.\n### 3.3 Amplifying the expert, brick by brick\nThe job of an enterprise RAG system is to **amplify** that expertise at scale, not replace it. What that looks like depends on the brick.\n**Parsing** comes first. If the parser turns a contract’s PDF into scrambled text, no downstream cleverness recovers it. If the document has a working table of contents, the parser has to extract it cleanly, because the TOC is what the expert relies on to navigate. When a document has no TOC at all (scanned faxes, slide decks exported to PDF, old typewritten policies), reconstructing one becomes a job in itself, often more useful than any retrieval tweak.\n**Question understanding** carries the team’s vocabulary across the gap between how a user phrases a question and how the document writes the answer. The pilot user types _kettle_ , the contract says _small electrical appliance_. The compliance officer types _data breach_ , the policy says _unauthorized disclosure of personal information_. The expert knows the mapping. The question parser turns that mapping into a lookup table: translations across languages, spelling variants, plural forms, internal acronyms. None of it is learned from data, it is dictated by the expert and written down.\n**Retrieval** amplifies what the expert already does by hand. The expert searches keywords; that part is already easy. What the expert cannot do at scale is run regex patterns over thousands of pages, check whether two terms co-occur inside the same paragraph, or combine boolean conditions across the whole corpus. The retriever does that work fast, then hands candidates back so the expert can verify.\n**Generation** does the two things the expert would otherwise do by hand: cite the exact passage that supports the answer, and format the raw value into something usable. The string `3455434` on the page becomes `€3,455,434` in the answer. `20260516` becomes `May 16, 2026`. _thirty days from the date of the loss_ stays verbatim, with a citation back to the clause so the expert can verify in one click.\nArticles 5, 6, 7, and 8 develop each brick in turn: the parser that extracts TOC structure, the expert dictionary that maps vocabulary, the TOC-aware retriever, the typed-answer generator. Same principle every time: pick up a piece of human expertise and move the repetitive part to the machine.\nThis is also why the series is careful with autonomous agents. It prefers keyword retrieval to embedding similarity by default. It treats reranker tuning as a last resort. Each of those defaults assumes there is no expert to consult. In enterprise contexts the expert is always there. The system should listen to them.\nIf you work in a setting with no expert, with unbounded questions, with very different documents, this series will not be your best guide. General-purpose retrieval and autonomous agents are a better fit there.\n## 4. Two parts, two failure modes\nA useful way to picture RAG is **as a search engine, plus an LLM that writes the answer**. Two parts, each with a clear job, each with its own way of breaking.\n**The search engine** retrieves passages from documents. Given a question, return the lines, paragraphs, or sections most likely to contain the answer. This is a pure search problem: selectivity, recall, ranking. Decades of information retrieval theory apply. The fact that part of it uses neural embeddings doesn’t change its nature; embedding similarity is just one ranking signal among several.\n**The LLM** takes a passage and a question and produces a natural-language answer with a citation. The LLM doesn’t _find_ the answer. The search engine already did that. The LLM _writes_ the answer from a passage that’s been placed in front of it. It’s closer to a translator or a scribe than to an oracle.\nMapping back to the four bricks from Article 1: parsing, question understanding, and retrieval together make up the search engine; generation is the LLM. The brick view is the operational one (one box of code per brick); the two-part view is the mental model you carry in your head when something goes wrong.\nThe two parts fail in different ways, and the diagnosis starts at the seam between them. Pull the trace from a failing query: were the retrieved passages in front of the model, and did they contain the answer?\n**If the answer wasn’t in the retrieved passages** , the search engine is the culprit, and the fix is upstream. Was the right page corrupted by the parser (OCR errors, multi-word terms split across lines, two-column interleaving)? Did the question parser miss a synonym the expert vocabulary should have expanded? Did the retrieval mechanism rank the right page out of `top_k`, or break on punctuation that needed a regex? Or is the relevant document just not in the corpus? Four very different fixes, all upstream. _“Tune the retriever”_ is meaningless until you’ve localized which one. The same four bricks that amplify the expert when working (section 3.3) break in their own ways here, each with its own deep-dive article (Articles 5, 6, 7).\n**If the answer was in the retrieved passages but the response is wrong** , the LLM is the culprit, and the fix is downstream. Common patterns: the model paraphrased and lost a conditional, returned the raw `3455434` because the schema left the answer free-form, cited the wrong line numbers, invented a value not in the passage, or produced an answer when it should have said _“not found”_. Five generation bugs, five different fixes, all in the prompt, schema, or post-validation layer (Article 8). None of them get better by tuning the retriever.\nHere’s what that diagnosis looks like in practice. A user asks _“how many heads does the base Transformer use?”_ (answer: 8, page 5 of the [_Attention Is All You Need_](https://arxiv.org/abs/1706.03762) paper, Vaswani et al. 2017; arXiv non-exclusive distribution license, declared on the [arXiv abstract page](https://arxiv.org/abs/1706.03762)). The system reports _“16”_. Pull the trace.\nRetrieval returned pages 4, 7, 8. None of them contain the base-model configuration: page 8 describes the _big_ model (which does use 16 heads), pages 4 and 7 describe encoder structure. The generator read the wrong pages and returned the number it found there. The bug is retrieval, not generation.\nWhy did retrieval miss page 5? The keywords were `['heads', 'base', 'model']`. Page 7 has _heads_ six times; page 5 has it twice. The keyword retriever ranked page 7 higher because it scored by raw term frequency, without checking whether `base`, `model`, and `heads` co-occur on the same line. Five lines of Python in the keyword retriever fix it.\nWhat didn’t happen: nobody fine-tuned anything. Nobody ran a sweep. Nobody added a reranker. The diagnostic took five minutes; the fix took an afternoon.\nThis separation is what makes RAG workable in practice. Each failure has a specific part to fix. There’s no training loop where retrieval and generation get tangled together. They’re independent components, composed cleanly, each replaceable on its own. Production systems gain a lot from this property: you can swap embedding models, swap LLMs, swap parsers, all without retraining anything.\n> **The whole pipeline is configuration, not model.**\nWhen something goes wrong, you change a configuration: the retrieval method, the prompt, the schema, a validation rule. You don’t retrain. You change a Python file, you ship, you measure the per-question-type metric for the affected category, and you confirm the fix. Iteration cycle: hours, not weeks.\nOnce you see RAG as configuration to assemble rather than behavior to learn, the rest of the series’ choices follow naturally.\n## 5. Six months on the wrong problem\nA team at a mid-size enterprise is given six months to deliver a RAG system over a few thousand internal documents. They start by building an evaluation dataset of 500 questions, splitting it 70/30 into train and test. They set up Optuna to sweep chunk size, overlap, top-k, and similarity threshold. The first sweep takes a week of compute, comes back with a “best” configuration, and the team ships it for internal testing.\nThe pilot users complain immediately. The system answers fluently but is wrong half the time on questions that the evaluators clearly know: questions about specific clauses, specific dates, specific numerical limits. The team’s response is to expand the evaluation dataset, run another sweep, fine-tune the embedding model on synthetic question-document pairs, and add a reranker. Three more months go by. Production accuracy doesn’t move.\nWhat was wrong: the parser was treating scanned pages with degraded OCR layers as if they were native text. About 30% of the corpus was effectively unreadable, but the team’s evaluation set happened to be drawn from the readable 70%. No amount of chunk size optimization, embedding fine-tuning, or reranker integration could fix it: a third of the documents were producing garbage. A two-day investment in checking each page (the work of Article 5, on parsing) would have caught this on day one.\nThe team had spent six months in ML mode (sweeping hyperparameters, growing evaluation sets, fine-tuning models) when the fix was a parser change.\n![](../images/body-1e4f5fc533847d80cc4e.png)\n_ix months of ML activity on the TEAM lane; the corpus bug sat untouched on the CORPUS lane - Image by author_\nThis story is composite, but every element of it has happened in real projects. The pattern is consistent: ML reflexes drive the team toward optimization activities that feel productive, while the structural problems sit untouched in the parser, the corpus, or the not-found logic. The first instinct on a struggling RAG system shouldn’t be “let’s tune”. It should be “let’s trace what happens to a failing query, end to end, and find the broken link.”\n## 6. Conclusion\nRAG looks like machine learning. The resemblance is shallow. The answer exists in the document or it does not. There is no statistical generalisation, no learning curve, no train/test split that maps to real failures. The right framing is search engine assembly: a search engine plus an LLM, two parts you can fix independently, with per-failure-mode metrics replacing aggregate accuracy.\nThe cost of holding on to the ML framing is not intellectual. It is six months of careful work on the wrong problem. Article 4 turns the right framing into a working diagnostic: RAG problems sit on a grid of document complexity by question control, and each cell calls for a different stack.\nArticle 4 is one entry point into _Enterprise Document Intelligence_ [Volume 1](https://towardsdatascience.com/document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-scale/), which builds enterprise RAG brick by brick across parsing, question parsing, retrieval, and generation: every brick handled with the engineering toolkit, not the ML one.\n![](../images/body-9a32f4b9e0e772a9255c.png)\n## 7. Sources and further reading\nThe article puts RAG in the 50-year IR tradition (Manning, Raghavan, Schütze, [Introduction to Information Retrieval](https://nlp.stanford.edu/IR-book/information-retrieval-book.html), 2008) rather than the ML tradition. The empirical claim that BM25 often beats dense retrievers out-of-distribution comes from Thakur et al. ([BEIR](https://arxiv.org/abs/2104.08663), NeurIPS 2021). The per-failure-mode framing is the same direction as Barnett et al. ([Seven Failure Points](https://arxiv.org/abs/2401.05856), 2024). The honest concession is that the reranker is a thin learned layer where ML methodology applies. The framing the article uses for explainability is **citation as the explanation** : a RAG answer carries its source lines, so the explainability tooling ML projects budget for becomes unnecessary.\n**Same direction as the article:**\n  * Manning, Raghavan, Schütze, [_Introduction to Information Retrieval_](https://nlp.stanford.edu/IR-book/information-retrieval-book.html) (Cambridge, 2008). The 50-year IR tradition the article puts RAG in.\n  * Thakur et al., _BEIR_ benchmark, NeurIPS 2021 ([arXiv:2104.08663](https://arxiv.org/abs/2104.08663)). Dense retrievers tuned on MS MARCO often lose to BM25 out-of-distribution. Empirical support for the _IR, not ML_ framing.\n  * Barnett et al., _Seven Failure Points When Engineering a RAG System_ , 2024 ([arXiv:2401.05856](https://arxiv.org/abs/2401.05856)). Practitioner taxonomy of where RAG breaks. Same direction as the per-failure-mode framing.\n  * Kamradt, [_Needle in a Haystack_](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) (2023). The canonical long-context retrieval benchmark. Research-only: tests a single verbatim fact in a long context, not the aggregating questions enterprise users ask. Discussed in Article 1 and developed in Article 7.\n\n\n**Different angle, different context:**\n  * Es et al., _RAGAS: Automated Evaluation of Retrieval Augmented Generation_ , EACL 2024 ([arXiv:2309.15217](https://arxiv.org/abs/2309.15217)). Treats RAG with aggregate ML metrics (faithfulness, answer relevance, context precision / recall) on benchmark datasets. The context is research benchmarks; the article’s framing is per-failure-mode rates on a fixed enterprise corpus.\n  * Saad-Falcon et al., _ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems_ , NAACL 2024 ([arXiv:2311.09476](https://arxiv.org/abs/2311.09476)). ML-style RAG evaluation framework with synthetic train / dev / test splits. Same context as RAGAS; the article argues the train / test split paradigm does not fit enterprise RAG where the answer either exists in the document or does not.\n  * Lewis et al., _Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks_ , NeurIPS 2020 ([arXiv:2005.11401](https://arxiv.org/abs/2005.11401)). The paper that named RAG, and the one that trained retriever and generator jointly. A useful borderline reference: the _original_ RAG paper was an ML paper, even though the engineering pattern that inherited the name is not.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-249d024535392fb76067.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Rag](https://towardsdatascience.com/tag/rag/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F&title=RAG%20Is%20Not%20Machine%20Learning%2C%20and%20the%20ML%20Toolkit%20Solves%20the%20Wrong%20Problem)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F&title=RAG%20Is%20Not%20Machine%20Learning%2C%20and%20the%20ML%20Toolkit%20Solves%20the%20Wrong%20Problem)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F&text=RAG%20Is%20Not%20Machine%20Learning%2C%20and%20the%20ML%20Toolkit%20Solves%20the%20Wrong%20Problem)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/002.png)](https://towardsdatascience.com/5-frequently-asked-questions-in-data-scientist-interviews-6fb3eeeb497/)\n## [5 Frequently Asked Questions in Data Scientist Interviews](https://towardsdatascience.com/5-frequently-asked-questions-in-data-scientist-interviews-6fb3eeeb497/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nRecently, I was asked by some new graduate students about how to prepare for Data Scientist interviews. Generally, the Data Scientist or...\n[Alina Zhang](https://towardsdatascience.com/author/alina-li-zhang/)\nOctober 18, 20185 min read\n  * [![](../images/003.png)](https://towardsdatascience.com/dummy-classifier-explained-a-visual-guide-with-code-examples-for-beginners-009ff95fc86e/)\n## [Dummy Classifier Explained: A Visual Guide with Code Examples for Beginners](https://towardsdatascience.com/dummy-classifier-explained-a-visual-guide-with-code-examples-for-beginners-009ff95fc86e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSetting the Bar in Machine Learning with Simple Baseline Models\n[Samy Baladram](https://towardsdatascience.com/author/samybaladram/)\nAugust 14, 20247 min read\n  * [![](../images/004.png)](https://towardsdatascience.com/diffusion-transformer-explained-e603c4770f7e/)\n## [Diffusion Transformer Explained](https://towardsdatascience.com/diffusion-transformer-explained-e603c4770f7e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nExploring the architecture that brought transformers into image generation\n[Mario Larcher](https://towardsdatascience.com/author/mnslarcher/)\nFebruary 28, 202412 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/how-to-use-gemini-3-pro-efficiently/)\n## [How to Use Gemini 3 Pro Efficiently](https://towardsdatascience.com/how-to-use-gemini-3-pro-efficiently/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nLearn the pros and cons of Gemini 3 Pro, from testing with both coding and console usage\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nNovember 20, 20257 min read\n  * [![](../images/006.jpg)](https://towardsdatascience.com/on-the-challenge-of-converting-tensorflow-models-to-pytorch/)\n## [On the Challenge of Converting TensorFlow Models to PyTorch](https://towardsdatascience.com/on-the-challenge-of-converting-tensorflow-models-to-pytorch/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHow to upgrade and optimize legacy AI/ML models\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nDecember 5, 202517 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/enhance-your-python-codes-readability-with-pycodestyle-9838976077cb/)\n## [Enhance your Python code's readability with pycodestyle](https://towardsdatascience.com/enhance-your-python-codes-readability-with-pycodestyle-9838976077cb/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAutomatically review the readability and quality of your Python scripts based on PEP-8 style conventions\n[Kenneth Leung](https://towardsdatascience.com/author/kennethleungty/)\nJuly 4, 20214 min read\n  * [![](../images/008.jpg)](https://towardsdatascience.com/are-you-still-using-lora-to-fine-tune-your-llm/)\n## [Are You Still Using LoRA to Fine-Tune Your LLM?](https://towardsdatascience.com/are-you-still-using-lora-to-fine-tune-your-llm/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nA look at this year’s crop of LoRA alternatives\n[Martin Görner](https://towardsdatascience.com/author/martin.gorner/)\nMarch 13, 20255 min read\n  * [![](../images/009.png)](https://towardsdatascience.com/20-lessons-learned-going-from-junior-data-scientist-to-chief-data-scientist-6fe09ff89d78/)\n## [20 Lessons learned going from Junior Data Scientist to Chief Data Scientist](https://towardsdatascience.com/20-lessons-learned-going-from-junior-data-scientist-to-chief-data-scientist-6fe09ff89d78/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\n3 years spent eating, sleeping, and breathing data science\n[Mathias Gruber](https://towardsdatascience.com/author/nanomathias/)\nJune 9, 202116 min read\n  * [![](../images/010.jpg)](https://towardsdatascience.com/llms-pandas-how-i-use-generative-ai-to-generate-pandas-dataframe-summaries-2/)\n## [LLMs + Pandas: How I Use Generative AI to Generate Pandas DataFrame Summaries](https://towardsdatascience.com/llms-pandas-how-i-use-generative-ai-to-generate-pandas-dataframe-summaries-2/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nLocal Large Language Models can convert massive DataFrames to presentable Markdown reports — here's how.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nJune 2, 202510 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "metadata": {
    "title": "RAG Is Not Machine Learning, and the ML Toolkit Solves the Wrong Problem | Towards Data Science",
    "description": "Enterprise Document Intelligence [Vol.1 #3] - Why the ML toolkit (hyperparameter sweeps, train/test splits, explainability frameworks) solves the wrong problem, and what to use instead",
    "keywords": null,
    "author": "Angela and Kezhan Shi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "RAG Is Not Machine Learning, and the ML Toolkit Solves the Wrong Problem",
    "og:description": "Enterprise Document Intelligence [Vol.1 #3] - Why the ML toolkit (hyperparameter sweeps, train/test splits, explainability frameworks) solves the wrong problem, and what to use instead",
    "og:url": "https://towardsdatascience.com/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/loupe_O7LDeH0Qo_v3_card.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Angela and Kezhan Shi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "25 min read",
    "article:published_time": "2026-06-01T12:50:01.000Z",
    "article:modified_time": "2026-06-01T12:50:01.000Z",
    "article:author": "https://towardsdatascience.com/author/angela.shi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/8c14ee4978d7/code/block_000.txt",
      "chars": 587
    },
    {
      "language": "txt",
      "file": "attachments/8c14ee4978d7/code/block_001.txt",
      "chars": 500
    },
    {
      "language": "txt",
      "file": "attachments/8c14ee4978d7/code/block_002.txt",
      "chars": 702
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:36:15Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:36:15Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:36:15Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:36:15Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/angela.shi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-scale/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/document-processing/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/llm-applications/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/5-frequently-asked-questions-in-data-scientist-interviews-6fb3eeeb497/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/alina-li-zhang/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/dummy-classifier-explained-a-visual-guide-with-code-examples-for-beginners-009ff95fc86e/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/samybaladram/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/pt-br/fotografias/lupa-preta-em-papel-branco-da-impressora-O7LDeH0Qo_0"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1706.03762"
    },
    {
      "type": "external",
      "href": "https://nlp.stanford.edu/IR-book/information-retrieval-book.html"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2104.08663"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2401.05856"
    },
    {
      "type": "external",
      "href": "https://github.com/gkamradt/LLMTest_NeedleInAHaystack"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2309.15217"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2311.09476"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2005.11401"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=RAG+Is+Not+Machine+Learning%2C+and+the+ML+Toolkit+Solves+the+Wrong+Problem&u=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=RAG+Is+Not+Machine+Learning%2C+and+the+ML+Toolkit+Solves+the+Wrong+Problem&url=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=RAG+Is+Not+Machine+Learning%2C+and+the+ML+Toolkit+Solves+the+Wrong+Problem&url=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:23:08Z",
  "source_url": "https://towardsdatascience.com/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\n# RAG Is Not Machine Learning, and the ML Toolkit Solves the Wrong Problem\nEnterprise Document Intelligence [Vol.1 #3] - Why the ML toolkit (hyperparameter sweeps, train/test splits, explainability frameworks) solves the wrong problem, and what to use instead\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nJune 1, 202625 min read\n![Why this image: an instrument for examining the small print, what this article does to the casual claim that RAG is an ML problem, looking closely at where the ML toolkit fits and where it does not.](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/loupe_O7LDeH0Qo_v3_card.jpg)\n[image](https://unsplash.com/pt-br/fotografias/lupa-preta-em-papel-branco-da-impressora-O7LDeH0Qo_0) by Teslariu Mihai via Unsplash\n**A development team** dedicated six months to fine-tuning their RAG pipeline.\n  * They ran five Optuna sweeps.\n  * They added a custom reranker.\n  * They fine-tuned an embedding model on their own data.\n\n\nProduction accuracy never moved. Pilots kept complaining about the same wrong answers. Six months in, the bug was in the parser.\nThe team was lost, not stuck. **RAG is not machine learning, and the ML toolkit solves the wrong problem.** This is the single most expensive misconception in enterprise RAG today. It costs months of careful work, the wrong people on the wrong tasks, and a quiet erosion of trust in the system.\nRAG looks enough like machine learning that the ML toolkit feels like the natural next step. The instincts (hyperparameter optimization, evaluation datasets, explainability frameworks) are not wrong in isolation. They are imported from the wrong field. The methods that work for training models don’t work for assembling search systems.\nThe point is not that ML is bad. The embedding model that powers vector search is itself a deep learning model, but you don’t train it, you consume it. The point is that **the system you’re building around it is not a model** , and treating it as one wastes time, picks the wrong metrics, hires the wrong people, and hides the real failure modes.\nThe \"RAG is not ML\" position is one piece of _Enterprise Document Intelligence_ [Volume 1](https://towardsdatascience.com/document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-scale/), which builds enterprise RAG brick by brick. The four bricks (parsing, question parsing, retrieval, generation) are the engineering toolkit this article points to.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## 1. Two different problems\nMachine learning solves problems where the true answer is unknown and has to be predicted. Will this customer churn? What’s the probability this transaction is fraud? Is this image a cat? You don’t know the answer in advance. That’s why you train a model. The model learns from labeled examples, generalizes to new inputs, and produces a prediction. Performance is measured in aggregate, across thousands of test cases, because individual predictions can be wrong while the model is still useful overall.\nRAG solves a different problem. The answer to “what is the effective date of this contract?” exists, written on page one of the document, or it doesn’t exist anywhere. There’s nothing to predict. The system either finds the answer in the document and reports it faithfully, or it fails and should say so. Performance is binary at the question level (got it or didn’t) even if you measure aggregate rates across many questions.\nThese differences are concrete:\n  * In ML, “the model was wrong on 8% of cases” is a feature of the system. You build redundancy, downstream checks, human review for the borderline cases. In RAG, “the system gave a wrong answer 8% of the time” is a bug. Each of those 8% has a specific cause: the wrong passage was retrieved, the right passage was retrieved but the model paraphrased it badly, the answer wasn’t in the corpus and the system made one up. They aren’t statistical noise to optimize on average. They’re individually fixable failures.\n  * In ML, you can’t generally tell _why_ the model got a particular case wrong. That’s why explainability is a research field. In RAG, you can always tell. The retrieval logs which passages it returned. The generator saw exactly those passages. If the answer is wrong, you walk the chain backward and find the broken link. There’s nothing hidden.\n  * In ML, the model improves by training on more data. In RAG, the system improves by indexing better, parsing more carefully, retrieving more precisely, prompting more clearly. None of that is training. It’s engineering.\n\n\nThat difference changes which tools you reach for when something breaks.\nThe cases catalogued in Article 2 fall exactly here: negation, exact identifiers, internal acronyms, signal dilution in long context, topical proximity outranking the actual answer. None of those move when you swap embedding models or sweep chunk sizes. They aren’t bugs a model can learn its way out of, because there is no labeled signal saying _“this is the right line”_ for the model to train on. The fix is structural (question parsing, expert keywords, retrieval that knows the document’s structure), and the next sections walk through the three ML reflexes that pick the wrong tool instead.\n## 2. Three arguments that don’t apply\nThree ML methods get imported into RAG projects by default: hyperparameter optimization, evaluation datasets with train/test splits, and feature-attribution explainability. Each is reasonable inside ML. Each misfires here.\n### 2.1 The hyperparameter argument\nThe most common framing goes something like this: chunk size, overlap, top-k, similarity threshold. These are hyperparameters, and you should optimize them the way you optimize ML models, using tools like Optuna or Ray Tune. Run a sweep, plot the curves, pick the best configuration.\nIn these setups, `top_k` is the number of passages the retriever keeps, and `similarity_threshold` is the minimum cosine score a passage must reach to qualify. The code below declares all four as numbers to optimize:\npython\n\n```\n# What teams typically write (and why it's the wrong activity)\nimport optuna\ndef objective(trial):\n    chunk_size    = trial.suggest_int(\"chunk_size\", 100, 2000)\n    chunk_overlap = trial.suggest_int(\"chunk_overlap\", 0, 200)\n    top_k         = trial.suggest_int(\"top_k\", 1, 20)\n    threshold     = trial.suggest_float(\"threshold\", 0.5, 0.95)\n    accuracy = run_rag_pipeline_and_score(\n        chunk_size, chunk_overlap, top_k, threshold\n    )\n    return accuracy\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective, n_trials=200)  # two weeks of compute later...\n```\n\nThere’s a grain of truth here. These variables do affect retrieval quality, and they are worth tuning. The trouble starts with the word “hyperparameter,” which brings in a metaphor with hidden assumptions.\nIn machine learning, a hyperparameter controls how a model learns: learning rate, regularization strength, number of layers. The model itself is what changes during training; the hyperparameter shapes that change. In RAG, there is no learning. The chunk size doesn’t control how something learns. It controls how a function splits text, the same way every time, regardless of what you’ve fed it before.\nWhat looks like a hyperparameter is a configuration choice, the kind you’d make when configuring a search engine. The expertise needed to tune it well isn’t statistical optimization. It’s understanding the structure of your documents and the shape of your questions. Chunk size of 512 tokens may work beautifully on dense academic papers and disastrously on insurance contracts where a single clause spans 800 tokens and breaking it in half loses the conditional that gives the clause its meaning. No grid search will tell you that. You need to read your documents.\nThis is why teams who grid-search chunk size often find a “best” value that performs marginally better on the test set and identically on production data. The optimum on the test set was an artifact of the test set, not a genuine improvement in the underlying system. They’ve optimized a number, not solved a problem.\n> **Common pitfall:** A team running Optuna over chunk_size, top_k, and similarity_threshold for two weeks, ending up at chunk_size=487 with no idea why. The honest answer to “why 487?” is “because Optuna said so.” That answer doesn’t survive a real production failure, and it doesn’t generalize when the document distribution shifts. A chunk size of 500 chosen because that’s roughly the size of a paragraph in this corpus is more defensible than 487 chosen because a sweep landed there.\nThe right activity isn’t tuning numbers. It’s deciding **structurally** how to chunk. By section? By paragraph? By the table of contents entries? By question type, with different chunkers for short lookups vs long clauses? Answered by looking at documents and questions, not by optimization curves.\nThere’s a deeper reason chunk size resists optimization: by construction, no single chunk size can serve every question. Take two questions on the same insurance contract:\n  * _“What is the effective date?”_ The answer is one line, somewhere on page one. It wants a chunk small enough to pin down a single line precisely.\n  * _“What are the exclusions of the policy?”_ The answer might be one page, or three pages, depending on how the insurer wrote it. It wants a chunk large enough to capture an entire section.\n\n\nThere is no number that satisfies both. A chunk size of 200 tokens chops the exclusions section into incoherent fragments. A chunk size of 2000 tokens buries the effective date in surrounding noise.\nTrying to find “the best chunk size” is therefore not a tuning problem. The framing itself is broken: no single number can serve a distribution of questions whose answers have different lengths.\nYou could, in principle, make chunk size respond to the question by training a small model that predicts the right chunker from the question’s features: classify the intent, regress over the expected answer length, output a strategy. That would be machine learning applied legitimately, on a problem where something is being learned.\nBut you don’t need to. You can write the rule down. Look at a question and you can tell whether it asks for a date, a section, or a comparison. So can a domain expert. So can ten lines of Python with hand-written conditions over keywords. The deeper reason RAG isn’t machine learning is that, for most of the decisions inside the system, **you already know the answer** , or someone on your team does. Machine learning is the tool for problems where nobody knows the answer in advance.\nThe right approach is to stop looking for one chunk size and start routing different question types to different retrieval strategies:\npython\n\n```\n# What to do instead: route by question type\ndef chunk_for_question(question: str, line_df, toc_df):\n    intent = classify_intent(question)\n    if intent == \"point_lookup\":          # \"what is the effective date?\"\n        return chunk_by_line(line_df)\n    elif intent == \"section_retrieval\":   # \"what are the exclusions?\"\n        return chunk_by_toc_section(line_df, toc_df)\n    elif intent == \"comparison\":          # \"compare clauses A and B\"\n        return chunk_by_full_section(line_df, toc_df)\n```\n\nThe two code blocks above are the entire argument of this section. The first runs Optuna over four numbers for two weeks and produces a value nobody can defend. The second makes one structural decision per question type and produces a system whose behavior anyone can explain.\nLater articles develop how to classify intent (Article 6, on question understanding) and how the different retrieval methods and granularities are implemented (Article 7, on retrieval). The point here is just that the activity isn’t tuning, it’s routing.\n### 2.2 The evaluation dataset argument\nThe next ML import is evaluation method. The reasoning goes: RAG, like any ML system, needs a proper evaluation dataset: questions paired with expected answers, split into train and test sets, scored with precision and recall. Frameworks like RAGAS have made this even more tempting, offering metrics for faithfulness, answer relevancy, and context recall that look satisfyingly ML-ish.\nEvaluation is useful. The issue isn’t whether to evaluate. It’s what the metrics mean. In machine learning, evaluation tells you whether a model has generalized from training data to unseen examples. The train/test split exists because you want to detect overfitting: a model that memorized the training set rather than learning a transferable pattern.\nIn RAG, there is nothing to generalize. Overfitting (when a model memorizes training examples rather than learning a pattern that transfers to new data) cannot happen here: the system does not change between queries. The retriever computes the same cosine distances every time. The generator follows the same prompt template. There is no model adjusting to data.\nWhat evaluation measures in RAG is three things, all of which are coverage and quality questions, not statistical generalization:\n  * **Does my corpus contain the answer?** If not, the system can’t find it. This is a content question, not a model question.\n  * **Does my retriever find the right passage?** If the answer is in the corpus but the retriever missed it, the system fails. This is a search question.\n  * **Does my generator stay faithful to what was retrieved?** If the right passage was retrieved but the model paraphrased it incorrectly or hallucinated extras, the system fails. This is a generation discipline question.\n\n\nEach one points to a specific fix. Mixing them up under an aggregate “accuracy” score loses information. A 75% accuracy from “corpus is missing 25% of the documented topics” demands different action than a 75% accuracy from “retriever misses the right passage 25% of the time.” The first calls for ingesting more documents. The second calls for fixing the retriever. An aggregate metric that treats them the same hides the diagnostic.\nThis also explains why teams using RAGAS-style frameworks sometimes report great metrics on a held-out test set and then watch the system fail in production. The test set covered topics where the corpus had answers and the retriever happened to find them. Production has questions whose answers are not in the corpus at all, and the system either hallucinates or fails to say “not found.” The metric was high on the test set because the test set was friendly. The system isn’t broken. The evaluation was.\nWhat you need to evaluate, broken down by question type, takes about ten lines:\npython\n\n```\n# Retrieval recall, per question, per intent\ndef evaluate_retrieval(reference_set, retrieve_fn):\n    rows = []\n    for ref in reference_set:\n        retrieved_lines = retrieve_fn(ref.question)\n        recall = len(set(retrieved_lines) & set(ref.expected_lines)) / len(ref.expected_lines)\n        rows.append({\n            \"question\": ref.question,\n            \"intent\":   ref.intent,\n            \"recall\":   recall,\n            \"hit\":      recall > 0,\n        })\n    return pd.DataFrame(rows)\n# Always break down by question type, never just an aggregate\ndf.groupby(\"intent\")[\"hit\"].mean()\n# point_lookup        0.92\n# section_retrieval   0.41   <-- this is the real problem\n# comparison          0.55\n```\n\nA single aggregate accuracy of 63% would have hidden the catastrophe on section_retrieval. The per-intent breakdown reveals it instantly. Recall here means: on questions where the answer exists in the corpus, did the retriever find the right passage? Grouping by intent (point_lookup, section_retrieval, …) shows _which kind_ of question fails, and therefore _which part_ of the pipeline to fix.\nRAG has two evaluation surfaces with very different shapes.\n**The retrieval surface** is a search problem: did the right passage land in front of the model? Measuring this means checking, on a reference set of questions, whether the relevant lines or pages were retrieved at all. The metric is recall at the level you care about (recall at line, at page, at section) and it’s specific to your corpus. Nobody else can run this evaluation for you. Your corpus is unique. This is where the bulk of evaluation effort belongs.\n**The generation surface** is different. Once the right passage has been retrieved, the question becomes: did the model produce a faithful answer, in the right format, with proper citations, and a clean “not found” when the passage didn’t contain the answer? Some of this you do evaluate yourself, but a large part is already evaluated by the LLM vendors. OpenAI, Anthropic, and Mistral spend enormous resources testing whether their models follow JSON schemas, refuse to invent, and respect prompt instructions. Those are the dimensions on which they improve their models. As a RAG builder, you’re not training the generator. You’re **consuming** it. If the model fails badly at returning structured JSON or stays unfaithful to its inputs, you’ll notice within an hour of integration. That’s not a metric to set up; it’s a sanity check that’s either obvious or fine.\nWhat this means in practice: most of your evaluation time should go into retrieval (which is corpus-specific and only you can do it), not into generation (which is mostly the vendor’s problem, and which shows obvious failures fast). Teams that spend weeks building elaborate generation evaluation suites are usually putting off the harder retrieval work that would improve the result.\n> **Going further:** _Evaluating Your System_ (later in the series) walks through how to build a reference set for your specific corpus, the four metrics that matter, and why per-question-type metrics are essential while aggregate metrics are misleading.\n### 2.3 The explainability argument\nMachine learning has its own toolkit for explainability. SHAP values to attribute predictions to features. LIME for local approximations of complex models. Attention visualization for transformers. When people start asking for RAG explainability (“why did the system give this answer?”) they naturally turn to these tools. They want to score retrieval relevance, weight document contributions, visualize which tokens influenced the output.\nThe irony is that RAG is more explainable by design than most ML models. There’s no need for SHAP. There’s no opacity to crack open. The system retrieved these specific passages from these specific sources, and the answer was built on top of them. That **is** the explanation. It’s documentary, not statistical.\nThis points to a deeper asymmetry between machine learning and RAG. In machine learning, the human has intuition but cannot quantify. Ask who survived the Titanic and people say _wealth_ , _age_ , _class_ : none wrong, none precise. The model has no such doubt: fit a decision tree and the root split is _sex_ , the next cut is an exact age threshold nobody would have guessed, then class. Every split is a number intuition alone could not have produced. The model exists to put those numbers down.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/05/image-399-1024x556.png)\n_A real sklearn decision tree on Titanic data. Every threshold is a number intuition couldn’t produce - Image by author_\nFor text data, the direction reverses. The user can read the source. A lawyer scanning a contract sees the conditions, the exceptions, the dates. A compliance officer reads a policy and knows whether a behavior breaches it. The text doesn’t hide its meaning, and the expert is already a fluent reader.\nThere are exceptions: sarcasm and irony are the classic ones, where modern LLMs sometimes catch what a literal reader misses. But in enterprise contexts the user is the domain expert.\nThe model isn’t there to explain the text. It’s there to do the reading at corpus scale, and a citation is enough to let the expert verify any answer in seconds.\nWhen a user asks “why this answer?”, the right response isn’t a heatmap of attention weights or a feature attribution score. It’s: “I looked at pages 12, 47, and 89 of this contract. Here’s the exact text I used. The answer follows from that text.” If the user disagrees with the answer, they can read the source themselves and judge. They don’t need an explainability framework. They need a citation.\nThe fifty-line pipeline from Article 1 already showed this. The prompt asked the model to return the start and end line numbers (with their pages) alongside the answer, in a structured JSON; the annotator then highlighted those exact lines on the PDF. No SHAP, no LIME, no attention visualization, no specialized observability platform. The “explanation” was a side product of how the prompt was written. The citation is part of the answer, not an analysis layer added on top.\nThe trace is the explanation. Reading it requires no interpretation, just reading.\nImporting ML explainability into RAG is solving a problem that doesn’t exist. SHAP on a retrieval score is using a scalpel to open a mailbox. The retrieval score is already a number you computed on inputs you can read. There’s nothing to attribute that you don’t already see.\nThe deeper failure of the ML-explainability framing is that it makes you focus on the wrong thing. You start trying to explain _why_ a particular passage scored higher than another in vector space, a near-impossible question that doesn’t matter. What matters is whether the right passage was retrieved at all, and whether the answer faithfully reflects it. Those are questions you can answer by reading the logs and the source. No tooling needed.\n## 3. What changes when you see RAG correctly\nOnce you stop treating RAG as ML, two things change. The day-to-day tools, metrics and people reorganize around search rather than training. And a deeper question (_where the intelligence sits_) moves from the model to the team. Both come from the same framing.\n### 3.1 Tools, metrics, people\nThree concrete things change.\n**The tools change:** You don’t need PyTorch, or a training cluster, or hyperparameter optimization frameworks for the system itself. You need a good parser, a flexible retriever, careful prompt engineering, and structured logging of everything that happens. The components that _are_ ML (the embedding model, the LLM) you consume as services. They’re commodity inputs, not things you build or train.\n**The metrics change:** Aggregate accuracy gives way to per-failure-mode metrics: retrieval recall (did we find the right passage?), answer faithfulness (did the model stick to it?), extraction accuracy (when extracting structured data, did the values match?), not-found rate (when the answer isn’t in the corpus, did we say so cleanly?). Each measures something specific, each maps to a specific part of the pipeline you can fix.\n**The people change:** A pure ML team trying to ship a RAG system often misses what makes it work, and what makes it fail. The skills that matter most are software engineering (the system has many moving parts that need to compose cleanly), domain expertise (someone has to know what a good answer to a domain question even looks like), and information retrieval intuition (someone has to think like a search engine designer, not a model trainer). ML expertise is useful, but it’s not the dominant skill. A team of ML researchers and no domain expert will produce a beautifully tuned system that misses the point. A team with one ML-aware engineer, two software engineers, and one domain expert will usually outperform it.\n### 3.2 Where the intelligence sits\nThe shift in people points to a deeper question: where does the intelligence of the system live?\nIn an ML system the intelligence lives in the model. The model holds the patterns. The team feeds it training data and tunes the loss function. In a RAG system the intelligence lives in the team. The lawyer knows which clauses to look at first. The underwriter knows what “deductible” means, and which page usually carries it. The compliance officer knows which regulation applies to which product. None of that lives inside the embedding model. None of it comes out of a hyperparameter sweep. It already lives in the heads of people who have read these documents for years.\nWatch an underwriter open a new policy. She doesn’t read it linearly. She jumps to the exclusions section first because she’s read five hundred of these and knows that’s where the trap usually lives. She checks the schedule of benefits for the deductibles and ceilings. She checks the territory clause. Three minutes in, she has a clearer view of the contract than any embedding model would produce on a thousand of those contracts. That habit is what the system has to amplify.\n### 3.3 Amplifying the expert, brick by brick\nThe job of an enterprise RAG system is to **amplify** that expertise at scale, not replace it. What that looks like depends on the brick.\n**Parsing** comes first. If the parser turns a contract’s PDF into scrambled text, no downstream cleverness recovers it. If the document has a working table of contents, the parser has to extract it cleanly, because the TOC is what the expert relies on to navigate. When a document has no TOC at all (scanned faxes, slide decks exported to PDF, old typewritten policies), reconstructing one becomes a job in itself, often more useful than any retrieval tweak.\n**Question understanding** carries the team’s vocabulary across the gap between how a user phrases a question and how the document writes the answer. The pilot user types _kettle_ , the contract says _small electrical appliance_. The compliance officer types _data breach_ , the policy says _unauthorized disclosure of personal information_. The expert knows the mapping. The question parser turns that mapping into a lookup table: translations across languages, spelling variants, plural forms, internal acronyms. None of it is learned from data, it is dictated by the expert and written down.\n**Retrieval** amplifies what the expert already does by hand. The expert searches keywords; that part is already easy. What the expert cannot do at scale is run regex patterns over thousands of pages, check whether two terms co-occur inside the same paragraph, or combine boolean conditions across the whole corpus. The retriever does that work fast, then hands candidates back so the expert can verify.\n**Generation** does the two things the expert would otherwise do by hand: cite the exact passage that supports the answer, and format the raw value into something usable. The string `3455434` on the page becomes `€3,455,434` in the answer. `20260516` becomes `May 16, 2026`. _thirty days from the date of the loss_ stays verbatim, with a citation back to the clause so the expert can verify in one click.\nArticles 5, 6, 7, and 8 develop each brick in turn: the parser that extracts TOC structure, the expert dictionary that maps vocabulary, the TOC-aware retriever, the typed-answer generator. Same principle every time: pick up a piece of human expertise and move the repetitive part to the machine.\nThis is also why the series is careful with autonomous agents. It prefers keyword retrieval to embedding similarity by default. It treats reranker tuning as a last resort. Each of those defaults assumes there is no expert to consult. In enterprise contexts the expert is always there. The system should listen to them.\nIf you work in a setting with no expert, with unbounded questions, with very different documents, this series will not be your best guide. General-purpose retrieval and autonomous agents are a better fit there.\n## 4. Two parts, two failure modes\nA useful way to picture RAG is **as a search engine, plus an LLM that writes the answer**. Two parts, each with a clear job, each with its own way of breaking.\n**The search engine** retrieves passages from documents. Given a question, return the lines, paragraphs, or sections most likely to contain the answer. This is a pure search problem: selectivity, recall, ranking. Decades of information retrieval theory apply. The fact that part of it uses neural embeddings doesn’t change its nature; embedding similarity is just one ranking signal among several.\n**The LLM** takes a passage and a question and produces a natural-language answer with a citation. The LLM doesn’t _find_ the answer. The search engine already did that. The LLM _writes_ the answer from a passage that’s been placed in front of it. It’s closer to a translator or a scribe than to an oracle.\nMapping back to the four bricks from Article 1: parsing, question understanding, and retrieval together make up the search engine; generation is the LLM. The brick view is the operational one (one box of code per brick); the two-part view is the mental model you carry in your head when something goes wrong.\nThe two parts fail in different ways, and the diagnosis starts at the seam between them. Pull the trace from a failing query: were the retrieved passages in front of the model, and did they contain the answer?\n**If the answer wasn’t in the retrieved passages** , the search engine is the culprit, and the fix is upstream. Was the right page corrupted by the parser (OCR errors, multi-word terms split across lines, two-column interleaving)? Did the question parser miss a synonym the expert vocabulary should have expanded? Did the retrieval mechanism rank the right page out of `top_k`, or break on punctuation that needed a regex? Or is the relevant document just not in the corpus? Four very different fixes, all upstream. _“Tune the retriever”_ is meaningless until you’ve localized which one. The same four bricks that amplify the expert when working (section 3.3) break in their own ways here, each with its own deep-dive article (Articles 5, 6, 7).\n**If the answer was in the retrieved passages but the response is wrong** , the LLM is the culprit, and the fix is downstream. Common patterns: the model paraphrased and lost a conditional, returned the raw `3455434` because the schema left the answer free-form, cited the wrong line numbers, invented a value not in the passage, or produced an answer when it should have said _“not found”_. Five generation bugs, five different fixes, all in the prompt, schema, or post-validation layer (Article 8). None of them get better by tuning the retriever.\nHere’s what that diagnosis looks like in practice. A user asks _“how many heads does the base Transformer use?”_ (answer: 8, page 5 of the [_Attention Is All You Need_](https://arxiv.org/abs/1706.03762) paper, Vaswani et al. 2017; arXiv non-exclusive distribution license, declared on the [arXiv abstract page](https://arxiv.org/abs/1706.03762)). The system reports _“16”_. Pull the trace.\nRetrieval returned pages 4, 7, 8. None of them contain the base-model configuration: page 8 describes the _big_ model (which does use 16 heads), pages 4 and 7 describe encoder structure. The generator read the wrong pages and returned the number it found there. The bug is retrieval, not generation.\nWhy did retrieval miss page 5? The keywords were `['heads', 'base', 'model']`. Page 7 has _heads_ six times; page 5 has it twice. The keyword retriever ranked page 7 higher because it scored by raw term frequency, without checking whether `base`, `model`, and `heads` co-occur on the same line. Five lines of Python in the keyword retriever fix it.\nWhat didn’t happen: nobody fine-tuned anything. Nobody ran a sweep. Nobody added a reranker. The diagnostic took five minutes; the fix took an afternoon.\nThis separation is what makes RAG workable in practice. Each failure has a specific part to fix. There’s no training loop where retrieval and generation get tangled together. They’re independent components, composed cleanly, each replaceable on its own. Production systems gain a lot from this property: you can swap embedding models, swap LLMs, swap parsers, all without retraining anything.\n> **The whole pipeline is configuration, not model.**\nWhen something goes wrong, you change a configuration: the retrieval method, the prompt, the schema, a validation rule. You don’t retrain. You change a Python file, you ship, you measure the per-question-type metric for the affected category, and you confirm the fix. Iteration cycle: hours, not weeks.\nOnce you see RAG as configuration to assemble rather than behavior to learn, the rest of the series’ choices follow naturally.\n## 5. Six months on the wrong problem\nA team at a mid-size enterprise is given six months to deliver a RAG system over a few thousand internal documents. They start by building an evaluation dataset of 500 questions, splitting it 70/30 into train and test. They set up Optuna to sweep chunk size, overlap, top-k, and similarity threshold. The first sweep takes a week of compute, comes back with a “best” configuration, and the team ships it for internal testing.\nThe pilot users complain immediately. The system answers fluently but is wrong half the time on questions that the evaluators clearly know: questions about specific clauses, specific dates, specific numerical limits. The team’s response is to expand the evaluation dataset, run another sweep, fine-tune the embedding model on synthetic question-document pairs, and add a reranker. Three more months go by. Production accuracy doesn’t move.\nWhat was wrong: the parser was treating scanned pages with degraded OCR layers as if they were native text. About 30% of the corpus was effectively unreadable, but the team’s evaluation set happened to be drawn from the readable 70%. No amount of chunk size optimization, embedding fine-tuning, or reranker integration could fix it: a third of the documents were producing garbage. A two-day investment in checking each page (the work of Article 5, on parsing) would have caught this on day one.\nThe team had spent six months in ML mode (sweeping hyperparameters, growing evaluation sets, fine-tuning models) when the fix was a parser change.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/05/image-400-1024x778.png)\n_ix months of ML activity on the TEAM lane; the corpus bug sat untouched on the CORPUS lane - Image by author_\nThis story is composite, but every element of it has happened in real projects. The pattern is consistent: ML reflexes drive the team toward optimization activities that feel productive, while the structural problems sit untouched in the parser, the corpus, or the not-found logic. The first instinct on a struggling RAG system shouldn’t be “let’s tune”. It should be “let’s trace what happens to a failing query, end to end, and find the broken link.”\n## 6. Conclusion\nRAG looks like machine learning. The resemblance is shallow. The answer exists in the document or it does not. There is no statistical generalisation, no learning curve, no train/test split that maps to real failures. The right framing is search engine assembly: a search engine plus an LLM, two parts you can fix independently, with per-failure-mode metrics replacing aggregate accuracy.\nThe cost of holding on to the ML framing is not intellectual. It is six months of careful work on the wrong problem. Article 4 turns the right framing into a working diagnostic: RAG problems sit on a grid of document complexity by question control, and each cell calls for a different stack.\nArticle 4 is one entry point into _Enterprise Document Intelligence_ [Volume 1](https://towardsdatascience.com/document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-scale/), which builds enterprise RAG brick by brick across parsing, question parsing, retrieval, and generation: every brick handled with the engineering toolkit, not the ML one.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/05/enterprise_document_intelligence_rag_series_overview-1024x538.png)\n## 7. Sources and further reading\nThe article puts RAG in the 50-year IR tradition (Manning, Raghavan, Schütze, [Introduction to Information Retrieval](https://nlp.stanford.edu/IR-book/information-retrieval-book.html), 2008) rather than the ML tradition. The empirical claim that BM25 often beats dense retrievers out-of-distribution comes from Thakur et al. ([BEIR](https://arxiv.org/abs/2104.08663), NeurIPS 2021). The per-failure-mode framing is the same direction as Barnett et al. ([Seven Failure Points](https://arxiv.org/abs/2401.05856), 2024). The honest concession is that the reranker is a thin learned layer where ML methodology applies. The framing the article uses for explainability is **citation as the explanation** : a RAG answer carries its source lines, so the explainability tooling ML projects budget for becomes unnecessary.\n**Same direction as the article:**\n  * Manning, Raghavan, Schütze, [_Introduction to Information Retrieval_](https://nlp.stanford.edu/IR-book/information-retrieval-book.html) (Cambridge, 2008). The 50-year IR tradition the article puts RAG in.\n  * Thakur et al., _BEIR_ benchmark, NeurIPS 2021 ([arXiv:2104.08663](https://arxiv.org/abs/2104.08663)). Dense retrievers tuned on MS MARCO often lose to BM25 out-of-distribution. Empirical support for the _IR, not ML_ framing.\n  * Barnett et al., _Seven Failure Points When Engineering a RAG System_ , 2024 ([arXiv:2401.05856](https://arxiv.org/abs/2401.05856)). Practitioner taxonomy of where RAG breaks. Same direction as the per-failure-mode framing.\n  * Kamradt, [_Needle in a Haystack_](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) (2023). The canonical long-context retrieval benchmark. Research-only: tests a single verbatim fact in a long context, not the aggregating questions enterprise users ask. Discussed in Article 1 and developed in Article 7.\n\n\n**Different angle, different context:**\n  * Es et al., _RAGAS: Automated Evaluation of Retrieval Augmented Generation_ , EACL 2024 ([arXiv:2309.15217](https://arxiv.org/abs/2309.15217)). Treats RAG with aggregate ML metrics (faithfulness, answer relevance, context precision / recall) on benchmark datasets. The context is research benchmarks; the article’s framing is per-failure-mode rates on a fixed enterprise corpus.\n  * Saad-Falcon et al., _ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems_ , NAACL 2024 ([arXiv:2311.09476](https://arxiv.org/abs/2311.09476)). ML-style RAG evaluation framework with synthetic train / dev / test splits. Same context as RAGAS; the article argues the train / test split paradigm does not fit enterprise RAG where the answer either exists in the document or does not.\n  * Lewis et al., _Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks_ , NeurIPS 2020 ([arXiv:2005.11401](https://arxiv.org/abs/2005.11401)). The paper that named RAG, and the one that trained retriever and generator jointly. A useful borderline reference: the _original_ RAG paper was an ML paper, even though the engineering pattern that inherited the name is not.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Rag](https://towardsdatascience.com/tag/rag/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F&title=RAG%20Is%20Not%20Machine%20Learning%2C%20and%20the%20ML%20Toolkit%20Solves%20the%20Wrong%20Problem)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F&title=RAG%20Is%20Not%20Machine%20Learning%2C%20and%20the%20ML%20Toolkit%20Solves%20the%20Wrong%20Problem)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Frag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem%2F&text=RAG%20Is%20Not%20Machine%20Learning%2C%20and%20the%20ML%20Toolkit%20Solves%20the%20Wrong%20Problem)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/10/1arztzSOJ1lWN2Dr4yWeGuA.png)](https://towardsdatascience.com/5-frequently-asked-questions-in-data-scientist-interviews-6fb3eeeb497/)\n## [5 Frequently Asked Questions in Data Scientist Interviews](https://towardsdatascience.com/5-frequently-asked-questions-in-data-scientist-interviews-6fb3eeeb497/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nRecently, I was asked by some new graduate students about how to prepare for Data Scientist interviews. Generally, the Data Scientist or...\n[Alina Zhang](https://towardsdatascience.com/author/alina-li-zhang/)\nOctober 18, 20185 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1eVxxKT4DKvRVuAHBGknJ7w.png)](https://towardsdatascience.com/dummy-classifier-explained-a-visual-guide-with-code-examples-for-beginners-009ff95fc86e/)\n## [Dummy Classifier Explained: A Visual Guide with Code Examples for Beginners](https://towardsdatascience.com/dummy-classifier-explained-a-visual-guide-with-code-examples-for-beginners-009ff95fc86e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSetting the Bar in Machine Learning with Simple Baseline Models\n[Samy Baladram](https://towardsdatascience.com/author/samybaladram/)\nAugust 14, 20247 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1hWXJ-CkWDFeF8owOW15kg.png)](https://towardsdatascience.com/diffusion-transformer-explained-e603c4770f7e/)\n## [Diffusion Transformer Explained](https://towardsdatascience.com/diffusion-transformer-explained-e603c4770f7e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nExploring the architecture that brought transformers into image generation\n[Mario Larcher](https://towardsdatascience.com/author/mnslarcher/)\nFebruary 28, 202412 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/image-204.jpg)](https://towardsdatascience.com/how-to-use-gemini-3-pro-efficiently/)\n## [How to Use Gemini 3 Pro Efficiently](https://towardsdatascience.com/how-to-use-gemini-3-pro-efficiently/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nLearn the pros and cons of Gemini 3 Pro, from testing with both coding and console usage\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nNovember 20, 20257 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/olga-thelavart-lIg6fu2yxFk-unsplash.jpg)](https://towardsdatascience.com/on-the-challenge-of-converting-tensorflow-models-to-pytorch/)\n## [On the Challenge of Converting TensorFlow Models to PyTorch](https://towardsdatascience.com/on-the-challenge-of-converting-tensorflow-models-to-pytorch/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHow to upgrade and optimize legacy AI/ML models\n[Chaim Rand](https://towardsdatascience.com/author/chaimrand/)\nDecember 5, 202517 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0ZCYWvH0rYWxZHKVM-scaled.jpg)](https://towardsdatascience.com/enhance-your-python-codes-readability-with-pycodestyle-9838976077cb/)\n## [Enhance your Python code's readability with pycodestyle](https://towardsdatascience.com/enhance-your-python-codes-readability-with-pycodestyle-9838976077cb/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAutomatically review the readability and quality of your Python scripts based on PEP-8 style conventions\n[Kenneth Leung](https://towardsdatascience.com/author/kennethleungty/)\nJuly 4, 20214 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/LoRA-1-1.jpg)](https://towardsdatascience.com/are-you-still-using-lora-to-fine-tune-your-llm/)\n## [Are You Still Using LoRA to Fine-Tune Your LLM?](https://towardsdatascience.com/are-you-still-using-lora-to-fine-tune-your-llm/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nA look at this year’s crop of LoRA alternatives\n[Martin Görner](https://towardsdatascience.com/author/martin.gorner/)\nMarch 13, 20255 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/1vOmAjcLDK90U1rR6io_2aQ.png)](https://towardsdatascience.com/20-lessons-learned-going-from-junior-data-scientist-to-chief-data-scientist-6fe09ff89d78/)\n## [20 Lessons learned going from Junior Data Scientist to Chief Data Scientist](https://towardsdatascience.com/20-lessons-learned-going-from-junior-data-scientist-to-chief-data-scientist-6fe09ff89d78/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\n3 years spent eating, sleeping, and breathing data science\n[Mathias Gruber](https://towardsdatascience.com/author/nanomathias/)\nJune 9, 202116 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/06/headway-5QgIuuBxKwM-unsplash-scaled-1.jpg)](https://towardsdatascience.com/llms-pandas-how-i-use-generative-ai-to-generate-pandas-dataframe-summaries-2/)\n## [LLMs + Pandas: How I Use Generative AI to Generate Pandas DataFrame Summaries](https://towardsdatascience.com/llms-pandas-how-i-use-generative-ai-to-generate-pandas-dataframe-summaries-2/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nLocal Large Language Models can convert massive DataFrames to presentable Markdown reports — here's how.\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nJune 2, 202510 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "body_base_path": "attachments/8c14ee4978d7/pages/",
  "body_markdown_path": "attachments/8c14ee4978d7/pages/archived-body.md",
  "local_html": "attachments/8c14ee4978d7/pages/archived-body.html",
  "body_source": "prior_capture"
}