{ "source_file": "0930", "source_platform": "个人博客", "author": "Jason Liu", "title_listed": "Hard Truths From the AI Trenches", "keywords": "AI工程,经验,成本", "domain": "大模型算法", "content_type": "观点文章", "is_paid_flag": "否", "url": "https://jxnl.co/writing/2025/03/05/ai-from-the-trenches/", "publish_date": "2025-03", "corpus_stage": "推理部署", "notes": "从一线看AI工程残酷真相", "id": "8fa9dab6c9a5", "collected_at": "2026-10-02T14:29:29", "title": "Hard Truths From the AI Trenches - Jason Liu", "body_markdown": "Hard Truths From the AI Trenches\n¶\nI never planned to become a consultant. But somewhere between building machine learning systems and growing my Twitter following, companies started sliding into my DMs with the same message: \"Help, our AI isn't working.\"\nSo I started charging to join their stand-ups. Sometimes I didn't even code. I just asked uncomfortable questions.\nHere's what I've learned watching companies burn millions on AI.\nYour AI problems aren't AI problems\n¶\nThe head of AI at a major company once showed me their evaluation framework. They had 23 evals. One of them tested if their AI could \"talk like a pirate.\"\nI shit you not.\nMeanwhile, 75% of their users were just trying to make their AI read Excel spreadsheets. But they had three engineers focused on a fancy document integration that less than 10% of users touched.\nThis is the pattern everywhere I go. Companies obsess over sophisticated AI capabilities while ignoring what users actually do with their product.\nThe truth? Your AI problem is almost always a data problem. Or a process problem. Or a people problem. But rarely is it actually an AI problem.\nStop waiting for the model to save you\n¶\n\"When GPT-5 comes out, all our problems will be solved.\"\nI've heard this exact sentiment in different words from CTOs who should know better. It's magical thinking, and it's everywhere.\nHere's a hard truth: capabilities are monotonically increasing. Each new model will be better than the last. But if your recall sucks now, ChatGPT 47.0 isn't going to save you.\nFix your retrieval. Fix your data pipeline. Fix your evaluation process. These are things you control today.\nWhen I tell clients this, they often look disappointed. They wanted me to tweak some prompt magic or recommend a fancy new model. Instead, I'm telling them to do the unsexy work of:\nBuilding better test sets\nImplementing proper feedback loops\nActually looking at their data\nOne client went from 27% recall to 85% in four hours once they stopped praying to the model gods and started making targeted improvements to their data preprocessing. It wasn't glamorous, but it worked.\nMeasure what matters (and it's not what you think)\n¶\nMost AI teams are measuring the wrong things. They obsess over model perplexity or ROUGE scores while their business burns.\nAt a sales tech company, engineers spent weeks trying to perfect their call transcript summarization accuracy. Meanwhile, sales managers didn't care about perfect summaries — they wanted to know which prospects had buying objections.\nClassification first, extraction second. Always.\nWhen we shifted from \"how accurately can we summarize this call?\" to \"can we identify sales objections with 95% recall?\", suddenly the business value was obvious. The sales team could follow up on objections, coach their reps better, and close more deals.\nThe best metric isn't technical. It's: \"Does this solve a real problem someone will pay for?\"\nExperimentation speed is your only moat\n¶\nAsk most AI teams how long it takes to run an experiment, and you'll get embarrassed silence. Four days? A week? \"Well, we have to coordinate with three teams and...\"\nThe companies winning with AI can run 10 experiments before lunch.\nOne client went from spending weeks fine-tuning prompts to implementing a system where they could:\nSample real-time traffic\nTest multiple approaches\nCompare metrics\nDeploy winners\nAll in the same day.\nYour ability to learn fast is everything. Build for learning velocity above all else.\nAs I tell my clients: how do you get better at jiu-jitsu? Mat time. More experiments, more data, more learning.\nTestimonials > Metrics\n¶\nNobody cares that your RAG system has 78.9% recall. They care that FancyCorp saved $2M using your product.\nI was advising a construction tech company where we dramatically improved their document retrieval. But the testimonial that closed their next funding round wasn't \"They improved our retrieval precision by 40%.\"\nIt was: \"Their system automatically identifies contractors who haven't signed liability waivers 4 days before they're scheduled on site. This prevented $80,000 in construction delays last month alone.\"\nWhen marketing your AI, translate your metrics into outcomes people actually care about.\nFind the hidden value patterns\n¶\nThe highest-leverage consulting I do isn't technical at all. It's identifying where the actual value is.\nAt a document management company, we discovered that 60% of questions weren't about document content at all. Users were asking: \"Who last modified this document?\" They needed contact information to verify decisions.\nSo we modified every text chunk to include metadata about creation date, modification date, and owner contact info. Suddenly, the most valuable feature wasn't fancy RAG—it was helping people find who to call.\nAt another client, we found that what users really wanted wasn't answering questions about blueprints. They wanted to count rooms. We could have spent months improving general QA, or we could build a specialized counter. We built the counter.\nYour users are telling you what they value. But you have to listen.\nNetwork effects are everything (and AI doesn't have them yet)\n¶\nHere's the biggest challenge facing AI companies: getting more customers doesn't make your product better. It just creates more opportunities to disappoint people.\nNetflix gets better when more people use it. Spotify gets better. Doordash gets better.\nBut most AI apps? Adding users just adds load without improving quality.\nWhat if having 1,000 users made your recall go up by 20%? You'd have a flywheel. Right now, most companies are just pushing a boulder.\nThis is why few AI companies have achieved escape velocity. The ones that do figure out how to create virtuous loops where more usage makes their systems smarter in ways users can feel.\nThe 'Oh shit' moment\n¶\nYou know you've found product-market fit when user growth starts hurting. One client went viral with their AI assistant, hitting 400,000 MRR almost overnight.\nThe next week? 340,000. Week after? 280,000.\nWhy? Their product wasn't ready for scale. They blew their first impression with thousands of users who will never come back.\nAI products often focus so much on capabilities that they forget fundamentals like stability, speed, and reliability. The most elegant RAG system means nothing if it takes 10 seconds to respond or breaks under load.\nMost consultants solve the wrong problem (including me sometimes)\n¶\nI've fired clients when I realized I couldn't help them. Not because their AI challenges were too hard, but because their organizational challenges were too deep.\nWhen a client books meetings with me at 5am while I'm about to board a flight, that's not an AI problem. When a team is rebuilding the same database integration for the third time, that's not an AI problem.\nSometimes the highest-value thing I can do is tell clients: \"Your issue isn't that your AI is bad. It's that your company's processes are broken.\"\nAnd sometimes the best consulting is just giving smart people permission to trust their instincts. Like when an engineer tells me: \"The only way forward is to try random permutations of these parameters to see what works,\" and I say, \"Yes, that's called hyperparameter optimization, and it's exactly what you should be doing.\"\nLessons learned\n¶\nThe pattern I see across every client is this: companies spend too much time on capabilities and not enough on connections. They build amazing AI features without connecting them to business outcomes.\nThe best AI isn't the most impressive in a demo. It's the one that solves a real problem customers will pay for.\nWhen a client asks how they're doing, I don't start with technical metrics. I ask: \"Are pilots converting to paid? Are usage numbers climbing? Are customers telling their friends?\"\nThose are the only metrics that matter.\nRemember:\nStart with the business outcome, work backwards\nMeasure what matters (hint: it's rarely model performance)\nOptimize for learning velocity\nConnect your AI directly to problems people will pay to solve\nAnd if all else fails, remember what I tell prospective clients: \"If we meet at a party six months from now, I don't want you to say 'Jason helped us improve our AI answers.' I want you to say 'Jason helped us close our Series A.'\"\nThat's the difference between solving AI problems and solving business problems.\nIf you found this useful, follow me on Twitter\n@jxnlco\nfor more AI insights. And if you're struggling with your AI strategy, DM me. I might be able to help.\nComments", "metadata": { "title": "Hard Truths From the AI Trenches - Jason Liu", "description": "A collection of hard truths from the AI trenches.", "keywords": null, "author": "Jason Liu", "og:type": "website", "og:title": "Hard Truths From the AI Trenches - Jason Liu", "og:description": "A collection of hard truths from the AI trenches.", "og:image": "https://jxnl.co/assets/images/social/writing/2025/03/05/ai-from-the-trenches.png", "og:image:type": "image/png", "og:image:width": "1200", "og:image:height": "630", "og:url": "https://jxnl.co/writing/2025/03/05/ai-from-the-trenches/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/jxnl/blog/edit/main/docs/writing/posts/ai-truths.md", "label": "", "type": "a" }, { "href": "https://github.com/jxnl/blog/raw/main/docs/writing/posts/ai-truths.md", "label": "", "type": "a" }, { "href": "https://twitter.com/jxnlco", "label": "@jxnlco", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:57Z", "source_url": "https://jxnl.co/writing/2025/03/05/ai-from-the-trenches/", "papers": [], "actual_video_files": [], "body_chars": 8547, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/8fa9dab6c9a5/pages/source.html", "local_html": "attachments/8fa9dab6c9a5/pages/article.html", "body_base_path": "attachments/8fa9dab6c9a5/pages/", "body_markdown_path": "attachments/8fa9dab6c9a5/pages/article.md" }