{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "Fuck You, Show Me The Prompt.",
  "keywords": "Prompt,提示,透明",
  "domain": "大模型算法",
  "content_type": "观点文章",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/blog/posts/prompt/",
  "publish_date": "2024-02",
  "corpus_stage": "预训练/SFT",
  "notes": "关于Prompt透明度的尖锐论述",
  "id": "ad95e5b5dc64",
  "collected_at": "2026-10-02T14:28:54",
  "title": "Fuck You, Show Me The Prompt. – Hamel's Blog",
  "body_markdown": "Fuck You, Show Me The Prompt.\nllms\nml\nQuickly understand inscrutable LLM frameworks by intercepting API calls.\nAuthor\nHamel Husain\nPublished\nFebruary 14, 2024\nBackground\nThere are many libraries that aim to make the output of your LLMs better by\nre-writing or constructing the prompt for you\n. These libraries purport to make the output of your LLMs:\nsafer\n(ex: guardrails)\ndeterministic\n(ex: guidance)\nstructured\n(ex: instructor)\nresilient\n(ex: langchain)\n… or even optimized for an arbitrary metric\n(ex: DSPy)\n.\nA common theme among\nsome\nof these tools is they encourage users to disintermediate themselves from prompting.\nDSPy\n: “This is a new paradigm in which LMs and their prompts fade into the background …. you can compile your program again DSPy will create new effective prompts”\nguidance\n“guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting …”\nEven when tools don’t discourage prompting, I’ve often found it difficult to retrieve the final prompt(s) these tools send to the language model.\nThe prompts sent by these tools to the LLM is a natural language description of what these tools are doing, and is the fastest way to understand how they work.\nFurthermore, some tools have\ndense terminology\nto describe internal constructs which can further obfuscate what they are doing.\nFor reasons I’ll explain below, I think most people would benefit from the following mindset:\nIn this blog post, I’ll show you how you can\nintercept API calls w/prompts for any tool, without having to fumble through docs or read source code.\nI’ll show you how to setup and operate\nmitmproxy\nwith examples from the LLM the tools I previously mentioned.\nMotivation: Minimize accidental complexity\nBefore adopting an abstraction, its important to consider the dangers of taking on\naccidental complexity\n. This danger is acute for LLM abstractions relative to programming abstractions. With LLM abstractions, we often force the user to regress towards writing code instead of conversing with the AI in natural language, which can run counter to the purpose of LLMs:\nProgramming abstraction -> a human-like language you can use to translate your task into machine code\nLLM abstraction -> an unintelligible framework you can use to translate your task into human language\n— Hamel Husain (\n@HamelHusain\n)\nFebruary 5, 2024\nWhile this is a cheeky comment, it’s worth keeping this in mind while evaluating tools. There are two primary types of automation that tools provide:\nInterleaving code and LLMs:\nExpressing this automation is often best done through code, since code must be run to carry out the task. Examples include routing, executing functions, retries, chaining, etc.\nRe-Writing and constructing prompts\n: Expressing your intent is often best done through natural language. However, there are exceptions! For example, it is convenient to express a function definition or schema from code instead of natural language.\nMany frameworks offer both types of automation. However, going too far with the second type can have negative consequences. Seeing the prompt allows you decide:\nIs this framework really necessary?\nShould I just steal the final prompt (a string) and jettison the framework?\nCan we write a better prompt than this (shorter, aligned with your intent, etc)?\nIs this the best approach (do the # of API calls seem appropriate)?\nIn my experience, seeing the prompts and API calls are essential to making informed decisions.\nIntercepting LLM API calls\nThere are many possible ways to intercept LLM API calls, such as monkey patching source code or finding a user-facing option. I’ve found that those approaches take far too much time since the quality of source code and documentation can vary greatly. After all, I just want to see API calls without worrying about how the code works!\nA framework agnostic way to see API calls is to setup a proxy that logs your outgoing API requests. This is easy to do with\nmitmproxy\n, an free, open-source HTTPS proxy.\nSetting Up mitmproxy\nThis is an opinionated way to setup\nmitmproxy\nthat’s beginner-friendly for our intended purposes:\nFollow the installation instructions\non the website\nStart the interactive UI by running\nmitmweb\nin the terminal. Pay attention to the url of the interactive UI in the logs which will look something like this:\nWeb server listening at http://127.0.0.1:8081/\nNext, you need to configure your device (i.e. your laptop) to route all traffic through\nmitproxy\n, which listens on\nhttp://localhost:8080\n. Per the documentation:\nWe recommend to simply search the web on how to configure an HTTP proxy for your system. Some operating system have a global settings, some browser have their own, other applications use environment variables, etc.\nIn my case, A\ngoogle search for “set proxy for macos”\nreturned these results:\nchoose Apple menu > System Settings, click Network in the sidebar, click a network service on the right, click Details, then click Proxies.\nI then insert\nlocalhost\nand\n8080\nin the following places in the UI:\nNext, navigate to\nhttp://mitm.it\nand it will give you instructions on how to install the mitmproxy Certificate Authority (CA), which you will need for intercepting HTTPS requests. (You can also do this manually\nhere\n.) Also, take note of the location of the CA file as we will reference it later.\nYou can test that everything works by browsing to a website like\nhttps://mitmproxy.org/\n, and seeing the corresponding output in the mtimweb UI which for me is located at\nhttp://127.0.0.1:8081/\n(look at the logs in your terminal to get the URL).\nNow that you set everything up, you can disable the proxy that you previously enabled on your network. I do this on my mac by toggling the proxy buttons in the screenshot I showed above. This is because we want to scope the proxy to only the python program to eliminate unnecessary noise.\nTip\nNetworking related software commonly allows you to proxy outgoing requests by setting environment variables. This is the approach we will use to scope our proxy to specific Python programs. However, I encourage you to play with other types of programs to see what you find after you are comfortable!\nEnvironment variables for Python\nWe need to set the following environment variables so that the\nrequests\nand\nhttpx\nlibraries will direct traffic to the proxy and reference the CA file for HTTPS traffic:\nImportant\nMake sure you set these environment variables before running any of the code snippets in this blog post.\nimport\nos\n# The location of my CA File\ncert_file\n=\n'/Users/hamel/Downloads/mitmproxy-ca-cert.pem'\nos.environ[\n'REQUESTS_CA_BUNDLE'\n]\n=\ncert_file\nos.environ[\n'SSL_CERT_FILE'\n]\n=\ncert_file\nos.environ[\n'HTTPS_PROXY'\n]\n=\n'http://127.0.0.1:8080'\nYou can do a minimal test by running the following code:\nimport\nrequests\nrequests.post(\n'https://httpbin.org/post'\n,\ndata\n=\n{\n'key'\n:\n'value'\n})\n<Response [200]>\nThis will appear in the UI like so:\nExamples\nNow for the fun part, let’s run through some examples of LLM libraries and intercept their API calls!\nGuardrails\nGuardrails allows you specify structure and types, which it uses to validate and correct the outputs of large language models. This is a hello world example from the\nguardrails-ai/guardrails\nREADME\n:\nfrom\npydantic\nimport\nBaseModel, Field\nfrom\nguardrails\nimport\nGuard\nimport\nopenai\nclass\nPet(BaseModel):\npet_type:\nstr\n=\nField(description\n=\n\"Species of pet\"\n)\nname:\nstr\n=\nField(description\n=\n\"a unique pet name\"\n)\nprompt\n=\n\"\"\"\nWhat kind of pet should I get and what should I name it?\n$\n{gr.complete_json_suffix_v2}\n\"\"\"\nguard\n=\nGuard.from_pydantic(output_class\n=\nPet, prompt\n=\nprompt)\nvalidated_output,\n*\nrest\n=\nguard(\nllm_api\n=\nopenai.completions.create,\nengine\n=\n\"gpt-3.5-turbo-instruct\"\n)\nprint\n(\nf\"\n{\nvalidated_output\n}\n\"\n)\n{\n    \"pet_type\": \"dog\",\n    \"name\": \"Buddy\nWhat is happening here? How is this structured output and validation working? Looking at the mitmproxy UI, I can see that the above code resulted in two LLM API calls, the first one with this prompt:\nWhat kind of pet should I get and what should I name it?\nGiven below is XML that describes the information to extract from this document and the tags to extract it into.\n<output>\n<string name=\"pet_type\" description=\"Species of pet\"/>\n<string name=\"name\" description=\"a unique pet name\"/>\n</output>\nONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise.\nHere are examples of simple (XML, JSON) pairs that show the expected behavior:\n- `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}`\n- `<list name='bar'><string format='upper-case' /></list>` => `{\"bar\": ['STRING ONE', 'STRING TWO', etc.]}`\n- `<object name='baz'><string name=\"foo\" format=\"capitalize two-words\" /><integer name=\"index\" format=\"1-indexed\" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}`\nFollowed by another call with this prompt:\nI was given the following response, which was not parseable as JSON.\n\"{\\n    \\\"pet_type\\\": \\\"dog\\\",\\n    \\\"name\\\": \\\"Buddy\"\nHelp me correct this by making it valid JSON.\nGiven below is XML that describes the information to extract from this document and the tags to extract it into.\n<output>\n<string name=\"pet_type\" description=\"Species of pet\"/>\n<string name=\"name\" description=\"a unique pet name\"/>\n</output>\nONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter `null`.\nWoof. That’s a whole lot of ceremony to get structured output! We learned that this library’s approach to structured output uses XML schemas (while others use function calling). It’s worth considering if you can fashion a better or simpler approach now that the magic has been lifted. Either way, we now have insight into how it works without dragging you into unnecessary complexity, which is a win.\nGuidance\nGuidance offers constrained generation and programming constructs for writing prompts. Let’s dive into a\nchat example from their tutorials\n:\nimport\nguidance\ngpt35\n=\nguidance.models.OpenAI(\n\"gpt-3.5-turbo\"\n)\nimport\nre\nfrom\nguidance\nimport\ngen, select, system, user, assistant\n@guidance\ndef\nplan_for_goal(lm, goal:\nstr\n):\n# This is a helper function which we will use below\ndef\nparse_best(prosandcons, options):\nbest\n=\nre.search(\nr'Best=\n(\n\\d\n+\n)\n'\n, prosandcons)\nif\nnot\nbest:\nbest\n=\nre.search(\nr'Best\n.\n*?\n(\n\\d\n+\n)\n'\n,\n'Best= option is 3'\n)\nif\nbest:\nbest\n=\nint\n(best.group(\n1\n))\nelse\n:\nbest\n=\n0\nreturn\noptions[best]\n# Some general instruction to the model\nwith\nsystem():\nlm\n+=\n\"You are a helpful assistant.\"\n# Simulate a simple request from the user\n# Note that we switch to using 'lm2' here, because these are intermediate steps (so we don't want to overwrite the current lm object)\nwith\nuser():\nlm2\n=\nlm\n+\nf\"\"\"\n\\\nI want to\n{\ngoal\n}\nCan you please generate one option for how to accomplish this?\nPlease make the option very short, at most one line.\"\"\"\n# Generate several options. Note that this means several sequential generation requests\nn_options\n=\n5\nwith\nassistant():\noptions\n=\n[]\nfor\ni\nin\nrange\n(n_options):\noptions.append((lm2\n+\ngen(name\n=\n'option'\n, temperature\n=\n1.0\n, max_tokens\n=\n50\n))[\n\"option\"\n])\n# Have the user request pros and cons\nwith\nuser():\nlm2\n+=\nf\"\"\"\n\\\nI want to\n{\ngoal\n}\nCan you please comment on the pros and cons of each of the following options, and then pick the best option?\n---\n\"\"\"\nfor\ni, opt\nin\nenumerate\n(options):\nlm2\n+=\nf\"Option\n{\ni\n}\n:\n{\nopt\n}\n\\n\n\"\nlm2\n+=\nf\"\"\"\n\\\n---\nPlease discuss each option very briefly (one line for pros, one for cons), and end by saying Best=X, where X is the number of the best option.\"\"\"\n# Get the pros and cons from the model\nwith\nassistant():\nlm2\n+=\ngen(name\n=\n'prosandcons'\n, temperature\n=\n0.0\n, max_tokens\n=\n600\n, stop\n=\n\"Best=\"\n)\n+\n\"Best=\"\n+\ngen(\n\"best\"\n, regex\n=\n\"[0-9]+\"\n)\n# The user now extracts the one selected as the best, and asks for a full plan\n# We switch back to 'lm' because this is the final result we want\nwith\nuser():\nlm\n+=\nf\"\"\"\n\\\nI want to\n{\ngoal\n}\nHere is my plan:\n{\noptions[\nint\n(lm2[\n\"best\"\n])]\n}\nPlease elaborate on this plan, and tell me how to best accomplish it.\"\"\"\n# The plan is generated\nwith\nassistant():\nlm\n+=\ngen(name\n=\n'plan'\n, max_tokens\n=\n500\n)\nreturn\nlm\nresults\n=\ngpt35\n+\nplan_for_goal(goal\n=\n\"read more books\"\n)\nsystem\nYou are a helpful assistant.\nuser\nI want to read more books\nHere is my plan: Set aside 30 minutes of dedicated reading time each day.\nPlease elaborate on this plan, and tell me how to best accomplish it.\nassistant\nSetting\naside\n30\nminutes\nof\ndedicated\nreading\ntime\neach\nday\nis\na\ngreat\nplan\nto\nread\nmore\nbooks\n.\nHere\nare\nsome\ntips\nto\nhelp\nyou\naccomplish\nthis\ngoal\n:\n1\n.\nEstablish\na\nroutine\n:\nChoose\na\nspecific\ntime\nof\nday\nthat\nworks\nbest\nfor\nyou\n,\nwhether\nit\n's\nin\nthe\nmorning\n,\nduring\nlunch\nbreak\n,\nor\nbefore\nbed\n.\nCons\nistency\nis\nkey\nto\nforming\na\nhabit\n.\n2\n.\nCreate\na\nreading\n-friendly\nenvironment\n:\nFind\na\nquiet\nand\ncomfortable\nspot\nwhere\nyou\ncan\nfocus\non\nyour\nreading\nwithout\ndistractions\n.\nIt\ncould\nbe\na\ncozy\ncorner\nin\nyour\nhome\n,\na\npark\nbench\n,\nor\na\nlocal\nlibrary\n.\n3\n.\nMini\nmi\nze\ndistractions\n:\nPut\naway\nyour\nphone\n,\nturn\noff\nthe\nTV\n,\nand\navoid\nany\nother\npotential\ninterruptions\nduring\nyour\ndedicated\nreading\ntime\n.\nThis\nwill\nhelp\nyou\nstay\nfocused\nand\nfully\nimmer\nse\nyourself\nin\nthe\nbook\n.\n4\n.\nChoose\nbooks\nthat\ninterest\nyou\n:\nSelect\nbooks\nthat\nalign\nwith\nyour\npersonal\ninterests\n,\nhobbies\n,\nor\ngoals\n.\nWhen\nyou\n're\ngenuinely\ninterested\nin\nthe\nsubject\nmatter\n,\nyou\n'll\nbe\nmore\nmotivated\nto\nread\nregularly\n.\n5\n.\nStart\nwith\nmanageable\ngoals\n:\nIf\nyou\n're\nnew\nto\nreading\nor\nhave\na\nbusy\nschedule\n,\nstart\nwith\na\nsmaller\ntime\ncommitment\n,\nsuch\nas\n15\nminutes\n,\nand\ngradually\nincrease\nit\nto\n30\nminutes\nor\nmore\nas\nyou\nbecome\nmore\ncomfortable\n.\n6\n.\nSet\na\ntimer\n:\nUse\na\ntimer\nor\na\nreading\napp\nthat\nallows\nyou\nto\ntrack\nyour\nreading\ntime\n.\nThis\nwill\nhelp\nyou\nstay\naccountable\nand\nensure\nthat\nyou\ndedicate\nthe\nfull\n30\nminutes\nto\nreading\n.\n7\n.\nMake\nreading\nenjoyable\n:\nCreate\na\ncozy\nreading\natmosphere\nby\nlighting\na\ncandle\n,\nsip\nping\na\ncup\nof\ntea\n,\nor\nplaying\nsoft\nbackground\nmusic\n.\nEng\naging\nall\nyour\nsenses\ncan\nenhance\nyour\nreading\nexperience\n.\n8\n.\nJoin\na\nbook\nclub\nor\nreading\ngroup\n:\nConsider\njoining\na\nbook\nclub\nor\nparticipating\nin\na\nreading\ngroup\nto\nconnect\nwith\nfellow\nbook\nlovers\n.\nThis\ncan\nprovide\nadditional\nmotivation\n,\ndiscussion\nopportunities\n,\nand\nbook\nrecommendations\n.\n9\n.\nKeep\na\nreading\nlog\n:\nMaintain\na\nrecord\nof\nthe\nbooks\nyou\n've\nread\n,\nalong\nwith\nyour\nthoughts\nand\nreflections\n.\nThis\ncan\nhelp\nyou\ntrack\nyour\nprogress\n,\ndiscover\npatterns\nin\nyour\nreading\npreferences\n,\nand\nserve\nas\na\nsource\nof\ninspiration\nfor\nfuture\nreading\n.\n10\n.\nBe\nflexible\n:\nWhile\nit\n's\nimportant\nto\nhave\na\ndedicated\nreading\ntime\n,\nbe\nflexible\nand\nadaptable\n.\nLife\ncan\nsometimes\nget\nbusy\n,\nso\nif\nyou\nmiss\na\nday\n,\ndon\n't\nbe\ndiscouraged\n.\nSimply\npick\nup\nwhere\nyou\nleft\noff\nand\ncontinue\nwith\nyour\nreading\nroutine\n.\nRemember\n,\nthe\ngoal\nis\nto\nenjoy\nthe\nprocess\nof\nreading\nand\nmake\nit\na\nregular\npart\nof\nyour\nlife\n.\nHappy\nreading\n!\nThis looks pretty neat! But what is it doing exactly?\nThis makes a total of 7 calls to OpenAI\n, which I have put in\nthis gist\n.\n5 of 7 of these API calls are “internal” thoughts asking the LLM to generate ideas.\nEven though the temperature is set to 1.0,\nthese “ideas” are mostly redundant.\nThe penultimate call to OpenAI enumerates these “ideas” which I’ve included below:\nI want to read more books\nCan you please comment on the pros and cons of each of the following options, and then pick the best option?\n---\nOption 0: Set aside dedicated time each day for reading.\nOption 1: Set aside 30 minutes of dedicated reading time each day.\nOption 2: Set aside dedicated time each day for reading.\nOption 3: Set aside dedicated time each day for reading.\nOption 4: Join a book club.\n---\nPlease discuss each option very briefly (one line for pros, one for cons), and end by saying Best=X, where X is the number of the best option.\nI know from experience that you are likely to get better results if you tell the language model to generate ideas in one shot. That way, the LLM can reference previous ideas and achieve more diversity. This is a good example of accidental complexity: its very tempting to take this design pattern and apply it blindly. This is less of a critique of this particular framework, since the code makes it clear that 5 independent calls will happen. Either way, its good idea to check your work by inspecting API calls!.\nLangchain\nLangchain is a multi-tool for all things LLM. Lots of people rely on Langchain when get started with LLMs. The core LangChain library doesn’t generally hide prompts from you, however there are experimental features that do. Let’s take a look at one of these features called\nSmartLLMChain\n:\nfrom\nlangchain.prompts\nimport\nPromptTemplate\nfrom\nlangchain_experimental.smart_llm\nimport\nSmartLLMChain\nfrom\nlangchain_openai\nimport\nChatOpenAI\nhard_question\n=\n\"I have a 12 liter jug and a 6 liter jug.\n\\\nI want to measure 6 liters. How do I do it?\"\nprompt\n=\nPromptTemplate.from_template(hard_question)\nllm\n=\nChatOpenAI(temperature\n=\n0\n, model_name\n=\n\"gpt-3.5-turbo\"\n)\nchain\n=\nSmartLLMChain(llm\n=\nllm, prompt\n=\nprompt,\nn_ideas\n=\n2\n,\nverbose\n=\nTrue\n)\nresult\n=\nchain.run({})\nprint\n(result)\nIdea 1: 1. Fill the 12 liter jug completely.\n2. Pour the contents of the 12 liter jug into the 6 liter jug. This will leave you with 6 liters in the 12 liter jug.\n3. Empty the 6 liter jug.\n4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.\n5. You now have 6 liters in the 6 liter jug.\n\nIdea 2: 1. Fill the 12 liter jug completely.\n2. Pour the contents of the 12 liter jug into the 6 liter jug. This will leave you with 6 liters in the 12 liter jug.\n3. Empty the 6 liter jug.\n4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.\n5. You now have 6 liters in the 6 liter jug.\n\nImproved Answer:\n1. Fill the 12 liter jug completely.\n2. Pour the contents of the 12 liter jug into the 6 liter jug until the 6 liter jug is full. This will leave you with 6 liters in the 12 liter jug and the 6 liter jug completely filled.\n3. Empty the 6 liter jug.\n4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.\n5. You now have 6 liters in the 6 liter jug.\n\nFull Answer:\nTo measure 6 liters using a 12 liter jug and a 6 liter jug, follow these steps:\n1. Fill the 12 liter jug completely.\n2. Pour the contents of the 12 liter jug into the 6 liter jug until the 6 liter jug is full. This will leave you with 6 liters in the 12 liter jug and the 6 liter jug completely filled.\n3. Empty the 6 liter jug.\n4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.\n5. You now have 6 liters in the 6 liter jug.\nNeat! So what happened exactly? While this API emits logs that show you a lot of information (available on\nthis gist\n), the API request pattern is interesting:\nTwo\nseperate\napi calls for each “idea”.\nAnother API call that incorporates the two ideas as context, with the prompt:\nYou are a researcher tasked with investigating the 2 response options provided. List the flaws and faulty logic of each answer options. Let’w work this out in a step by step way to be sure we have all the errors:”\nA final API call that that takes the critique from step 2 and generates an answer.\nIts not clear that this approach is optimal. I am not sure it should take 4 separate API calls to accomplish this task. Perhaps the critique and the final answer could be generated in one step? Furthermore, the prompt has a spelling error (\nLet'w\n) and also overly focuses on the negative about identifying errors - which makes me skeptical that this prompt has been optimized or tested.\nInstructor\nInstructor\nis a framework for structured outputs.\nStructred data extraction with Pydantic\nHere is a basic example from the project’s\nREADME\nthat allows you to extract structured data by using Pydantic to define your schema.\nimport\ninstructor\nfrom\nopenai\nimport\nOpenAI\nfrom\npydantic\nimport\nBaseModel\nclient\n=\ninstructor.patch(OpenAI())\nclass\nUserDetail(BaseModel):\nname:\nstr\nage:\nint\nuser\n=\nclient.chat.completions.create(\nmodel\n=\n\"gpt-3.5-turbo\"\n,\nresponse_model\n=\nUserDetail,\nmessages\n=\n[{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n:\n\"Extract Jason is 25 years old\"\n}])\nWe can see how this works by inspecting the API call logged to mitmproxy:\n{\n\"function_call\"\n:\n{\n\"name\"\n:\n\"UserDetail\"\n},\n\"functions\"\n:\n[\n{\n\"description\"\n:\n\"Correctly extracted `UserDetail` with all the required parameters with correct types\"\n,\n\"name\"\n:\n\"UserDetail\"\n,\n\"parameters\"\n:\n{\n\"properties\"\n:\n{\n\"age\"\n:\n{\n\"title\"\n:\n\"Age\"\n,\n\"type\"\n:\n\"integer\"\n},\n\"name\"\n:\n{\n\"title\"\n:\n\"Name\"\n,\n\"type\"\n:\n\"string\"\n}\n},\n\"required\"\n:\n[\n\"age\"\n,\n\"name\"\n]\n,\n\"type\"\n:\n\"object\"\n}\n}\n]\n,\n\"messages\"\n:\n[\n{\n\"content\"\n:\n\"Extract Jason is 25 years old\"\n,\n\"role\"\n:\n\"user\"\n}\n]\n,\n\"model\"\n:\n\"gpt-3.5-turbo\"\n}\nThis is great. For structured output -\nIt does exactly what I want, and it correctly uses the OpenAI API the way I would use it\nif I were writing this manually (by defining a function schema). I would consider this specific API a zero-cost abstraction, meaning it does exactly what I expect it to with a minimal surface area.\nValidation\nHowever, instructor has other APIs that are more aggressive and write prompts for you. For example, consider this\nvalidation example\n. Running through that example should trigger similar questions to the exploration of\nLangchain’s SmartLLMChain\nabove. In this example, you will observe 3 LLM API calls to get the right answer, with the final payload looking like this:\n{\n\"function_call\"\n:\n{\n\"name\"\n:\n\"Validator\"\n},\n\"functions\"\n:\n[\n{\n\"description\"\n:\n\"Validate if an attribute is correct and if not,\n\\n\nreturn a new value with an error message\"\n,\n\"name\"\n:\n\"Validator\"\n,\n\"parameters\"\n:\n{\n\"properties\"\n:\n{\n\"fixed_value\"\n:\n{\n\"anyOf\"\n:\n[\n{\n\"type\"\n:\n\"string\"\n}\n,\n{\n\"type\"\n:\n\"null\"\n}\n]\n,\n\"default\"\n:\nnull\n,\n\"description\"\n:\n\"If the attribute is not valid, suggest a new value for the attribute\"\n,\n\"title\"\n:\n\"Fixed Value\"\n},\n\"is_valid\"\n:\n{\n\"default\"\n:\ntrue\n,\n\"description\"\n:\n\"Whether the attribute is valid based on the requirements\"\n,\n\"title\"\n:\n\"Is Valid\"\n,\n\"type\"\n:\n\"boolean\"\n},\n\"reason\"\n:\n{\n\"anyOf\"\n:\n[\n{\n\"type\"\n:\n\"string\"\n}\n,\n{\n\"type\"\n:\n\"null\"\n}\n]\n,\n\"default\"\n:\nnull\n,\n\"description\"\n:\n\"The error message if the attribute is not valid, otherwise None\"\n,\n\"title\"\n:\n\"Reason\"\n}\n},\n\"required\"\n:\n[]\n,\n\"type\"\n:\n\"object\"\n}\n}\n]\n,\n\"messages\"\n:\n[\n{\n\"content\"\n:\n\"You are a world class validation model. Capable to determine if the following value is valid for the statement, if it is not, explain why and suggest a new value.\"\n,\n\"role\"\n:\n\"system\"\n}\n,\n{\n\"content\"\n:\n\"Does `According to some perspectives, the meaning of life is to find purpose, happiness, and fulfillment. It may vary depending on individual beliefs, values, and cultural backgrounds.` follow the rules: don't say objectionable things\"\n,\n\"role\"\n:\n\"user\"\n}\n]\n,\n\"model\"\n:\n\"gpt-3.5-turbo\"\n,\n\"temperature\"\n:\n0\n}\nConcretely, I’m curious if these steps could be collapsed into two LLM calls instead of three. Furthermore, I wonder if generic validation functions (as supplied in the above payload) are the right way to critique output? I don’t know the answer, but this is an interesting design pattern that is worth poking at.\nNote\nAs far as LLM frameworks go, I really like this one. The core functionality of defining schemas with Pydantic is very convenient. The code is also very readable and easy to understand. Despite this, I still found it helpful to intercept instructor’s API calls to get another perspective.\nThere is a way to set a logging level in instructor to see the raw API calls, however, I like using a framework agnostic approach :)\nDSPy\nDSPy\nis the framework that helps you optimize your prompts to optimize any arbitrary metric. There is a fairly steep learning curve to DSPy, partly because it introduces many new technical terms specific to its framework like compilers and teleprompters. However, we can quickly peel back the complexity by looking at the API calls that it makes!\nLet’s run the\nminimal working example\n:\nimport\ntime\nimport\ndspy\nfrom\ndspy.datasets.gsm8k\nimport\nGSM8K, gsm8k_metric\nstart_time\n=\ntime.time()\n# Set up the LM\nturbo\n=\ndspy.OpenAI(model\n=\n'gpt-3.5-turbo-instruct'\n, max_tokens\n=\n250\n)\ndspy.settings.configure(lm\n=\nturbo)\n# Load math questions from the GSM8K dataset\ngms8k\n=\nGSM8K()\ntrainset, devset\n=\ngms8k.train, gms8k.dev\nclass\nCoT(dspy.Module):\ndef\n__init__\n(\nself\n):\nsuper\n().\n__init__\n()\nself\n.prog\n=\ndspy.ChainOfThought(\n\"question -> answer\"\n)\ndef\nforward(\nself\n, question):\nreturn\nself\n.prog(question\n=\nquestion)\nfrom\ndspy.teleprompt\nimport\nBootstrapFewShotWithRandomSearch\n# Set up the optimizer: we want to \"bootstrap\" (i.e., self-generate) 8-shot examples of our CoT program.\n# The optimizer will repeat this 10 times (plus some initial attempts) before selecting its best attempt on the devset.\nconfig\n=\ndict\n(max_bootstrapped_demos\n=\n8\n, max_labeled_demos\n=\n8\n, num_candidate_programs\n=\n10\n, num_threads\n=\n4\n)\n# Optimize! Use the `gms8k_metric` here. In general, the metric is going to tell the optimizer how well it's doing.\nteleprompter\n=\nBootstrapFewShotWithRandomSearch(metric\n=\ngsm8k_metric,\n**\nconfig)\noptimized_cot\n=\nteleprompter.\ncompile\n(CoT(), trainset\n=\ntrainset, valset\n=\ndevset)\nWarning\nThis was not very minimal\nDespite this being the official\nquick-start/minimal working\nexample, this code took\nmore than 30 minutes to run, and made hundreds of calls to OpenAI!\nThis cost non-trivial time (and money), especially as an entry-point to the library for someone trying to take a look. There was no prior warning that this would happen.\nDSPy made 100s of API calls because it was iteratively sampling examples for a few-shot prompt and selecting the best ones according to the\ngsm8k_metric\non a validation set. I was able to quickly understand this by scanning through the API requests logged to mitmproxy.\nDSPy offers an\ninspect_history\nmethod which allows you to see the the last\nn\nprompts and their completions:\nturbo.inspect_history(n\n=\n1\n)\nI was able to verify that these prompts matched the last few API calls being made in mitmproxy. Overall, I would be motivated to potentially keep the prompt and and jettison the library. That being said, I think I am curious to see how this library evolves.\nMy Personal Experience\nDo I hate LLM libraries? No! I think many of the libraries in this blog post could be helpful if used thoughtfully in the right situations. However, I’ve witnessed too many people fall into the trap of using these libraries without understanding what they are doing.\nOne thing I focus on as an independent consultant is to make sure my clients don’t take on accidental complexity. It’s very tempting to adopt additional tools given all the excitement around LLMs. Looking at prompts is one way to mitigate that temptation.\nI’m wary of frameworks that distance the human too far from LLMs. By whispering\n“Fuck you, show me the prompt!”\nwhen using these tools, you are empowered to decide for yourself.\n1\nAcknowledgments: Thanks to\nJeremy Howard\nand\nBen Clavie\nfor thoughtfully reviewing this post.\nFootnotes\nYou don’t have to whisper. Saying it out loud is fine too - let others know!\n↩︎",
  "metadata": {
    "title": "Fuck You, Show Me The Prompt. – Hamel's Blog",
    "description": "Quickly understand inscrutable LLM frameworks by intercepting API calls.",
    "keywords": null,
    "author": "Hamel Husain",
    "og:type": "article",
    "og:title": "Fuck You, Show Me The Prompt. –",
    "og:description": "Quickly understand inscrutable LLM frameworks by intercepting API calls.",
    "og:image": "https://hamel.dev/blog/posts/prompt/slap_3.png",
    "og:locale": "en_US",
    "og:image:height": "254",
    "og:image:width": "508",
    "twitter:title": "Fuck You, Show Me The Prompt. –",
    "twitter:description": "Quickly understand inscrutable LLM frameworks by intercepting API calls.",
    "twitter:image": "https://hamel.dev/blog/posts/prompt/slap_3.png",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image",
    "twitter:image-height": "254",
    "twitter:image-width": "508"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ad95e5b5dc64/code/body-000.txt",
      "chars": 235,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-001.txt",
      "chars": 95,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-002.txt",
      "chars": 16,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-003.txt",
      "chars": 530,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-004.txt",
      "chars": 43,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-005.txt",
      "chars": 1138,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-006.txt",
      "chars": 842,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-007.txt",
      "chars": 2497,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-008.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-009.txt",
      "chars": 2634,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-010.txt",
      "chars": 551,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-011.txt",
      "chars": 361,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-012.txt",
      "chars": 139,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-013.txt",
      "chars": 13,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-014.txt",
      "chars": 1508,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-015.txt",
      "chars": 343,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-016.txt",
      "chars": 911,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-017.txt",
      "chars": 2385,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-018.txt",
      "chars": 323,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-019.txt",
      "chars": 218,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-020.txt",
      "chars": 663,
      "language": "text"
    },
    {
      "file": "attachments/ad95e5b5dc64/code/body-021.txt",
      "chars": 26,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/prompt/slap_3.jpeg",
      "bytes": 85595,
      "reused": false,
      "local_path": "attachments/ad95e5b5dc64/images/body-47cf10ef4c735e87259c.jpeg",
      "src": "https://hamel.dev/blog/posts/prompt/slap_3.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/prompt/mac.png",
      "bytes": 54990,
      "reused": false,
      "local_path": "attachments/ad95e5b5dc64/images/body-d2108de143cbb1651e0d.png",
      "src": "https://hamel.dev/blog/posts/prompt/mac.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/prompt/mitm_01.png",
      "bytes": 76332,
      "reused": false,
      "local_path": "attachments/ad95e5b5dc64/images/body-a5aa5eed175e18ec8088.png",
      "src": "https://hamel.dev/blog/posts/prompt/mitm_01.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://github.com/guardrails-ai/guardrails",
      "label": "(ex: guardrails)",
      "type": "a"
    },
    {
      "href": "https://github.com/guidance-ai/guidance",
      "label": "(ex: guidance)",
      "type": "a"
    },
    {
      "href": "https://github.com/jxnl/instructor",
      "label": "(ex: instructor)",
      "type": "a"
    },
    {
      "href": "https://www.langchain.com/",
      "label": "(ex: langchain)",
      "type": "a"
    },
    {
      "href": "https://github.com/stanfordnlp/dspy",
      "label": "(ex: DSPy)",
      "type": "a"
    },
    {
      "href": "https://github.com/stanfordnlp/dspy?tab=readme-ov-file",
      "label": "dense terminology",
      "type": "a"
    },
    {
      "href": "https://mitmproxy.org/",
      "label": "mitmproxy",
      "type": "a"
    },
    {
      "href": "https://dev.to/alexbunardzic/software-complexity-essential-accidental-and-incidental-3i4d",
      "label": "accidental complexity",
      "type": "a"
    },
    {
      "href": "https://twitter.com/HamelHusain/status/1754315254413361553",
      "label": "February 5, 2024",
      "type": "a"
    },
    {
      "href": "https://www.google.com/search?q=set+proxy+for+macos&sca_esv=c51a80de1a7d45f0&rlz=1C5CHFA_enUS1048US1049&sxsrf=ACQVn0_ysjr6Kma2_lX8WbB06iPbDi5gUQ%3A1707764982232&ei=9mzKZYXoDcfy0PEPpJqb2Ao&ved=0ahUKEwiFu4CpwKaEAxVHOTQIHSTNBqsQ4dUDCBA&uact=5&oq=set+proxy+for+macos&gs_lp=Egxnd3Mtd2l6LXNlcnAiE3NldCBwcm94eSBmb3IgbWFjb3MyBBAjGCcyBhAAGBYYHjIGEAAYFhgeMgYQABgWGB4yBhAAGBYYHjILEAAYgAQYigUYhgMyCxAAGIAEGIoFGIYDSMk-UMU7WMU7cAd4AZABAJgBVaABVaoBATG4AQPIAQD4AQHCAgoQABhHGNYEGLAD4gMEGAAgQYgGAZAGCA&sclient=gws-wiz-serp",
      "label": "google search for “set proxy for macos”",
      "type": "a"
    },
    {
      "href": "http://mitm.it/",
      "label": "http://mitm.it",
      "type": "a"
    },
    {
      "href": "https://docs.mitmproxy.org/stable/concepts-certificates/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "http://127.0.0.1:8081/",
      "label": "http://127.0.0.1:8081/",
      "type": "a"
    },
    {
      "href": "https://github.com/guidance-ai/guidance/blob/main/notebooks/tutorials/chat.ipynb",
      "label": "chat example from their tutorials",
      "type": "a"
    },
    {
      "href": "https://gist.github.com/hamelsmu/d0d75bf702e56987f35cb715f7da4d6a",
      "label": "this gist",
      "type": "a"
    },
    {
      "href": "https://api.python.langchain.com/en/latest/smart_llm/langchain_experimental.smart_llm.base.SmartLLMChain.html",
      "label": "SmartLLMChain",
      "type": "a"
    },
    {
      "href": "https://gist.github.com/hamelsmu/abfb14b0af4c70e8532f9d4e0ef3e54e",
      "label": "this gist",
      "type": "a"
    },
    {
      "href": "https://python.useinstructor.com/learning/validation/basics/",
      "label": "validation example",
      "type": "a"
    },
    {
      "href": "https://dspy.ai/",
      "label": "minimal working example",
      "type": "a"
    },
    {
      "href": "https://twitter.com/jeremyphoward",
      "label": "Jeremy Howard",
      "type": "a"
    },
    {
      "href": "https://twitter.com/bclavie",
      "label": "Ben Clavie",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:42Z",
  "source_url": "https://hamel.dev/blog/posts/prompt/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 27774,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ad95e5b5dc64/pages/source.html",
  "local_html": "attachments/ad95e5b5dc64/pages/article.html",
  "body_base_path": "attachments/ad95e5b5dc64/pages/",
  "body_markdown_path": "attachments/ad95e5b5dc64/pages/article.md"
}