{ "source_file": "0930", "source_platform": "Ghost", "author": "Achal Shah / In the tubes", "title_listed": "Realtime Compute and AWS Lambda", "keywords": "", "domain": "数据处理", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://blog.achals.com/realtime-compute-and-aws-lambda/", "publish_date": "2026-02", "corpus_stage": "", "notes": "", "id": "4a42ae0c6ea1", "collected_at": "2026-10-02T12:28:00", "title": "Realtime Compute and AWS Lambda", "body_markdown": "Realtime Compute and AWS Lambda\nAchal Shah\n20 Feb 2026\n—\n6 min read\nShare\nPhoto by\nH&CO\n/\nUnsplash\nThis is followup to\nhttps://blog.achals.com/some-thoughts-on-building-realtime-compute-at-tecton/\n- focused on how we built a platform for user code execution at Tecton. If you haven't read it yet - you probably should, cause this post will make more sense if you're caught up. 😄\nWe left off at a pretty big question - how we let users run code to transform their data before it gets written into online/offline storage?\nThere's a few dimensions along which we're constrained:\nHow do users express their transformations? Broadly, the spectrum is declarative (i.e. SQL) to imperative (i.e. a fully fledged programming language like Python).\nWhere do these transformations run? Are they run as a separate service, or colocated with existing services?\nThis problem describes not just an aspect of the problem that Tecton had to solve, but something very general purpose - while I have never worked at places like Modal, it seems intuitively that the problems - and options - they face are quite similar.\nV0 Implementation\nAt Tecton, there was a V0 implementation that found its way into production - we allows users to express transformations as\nPython UDFs\n. These UDFs would run in a Python sub-process, colocated next to every single\nFeature Server\nprocess. The Feature Server would retrieve records from the online store, pass them into the python interpreter for processing, and retrieve the response and return it callers.\nThis\nworked. In a manner of speaking.\nIt wasn't really a reusable component (yet) and had quite a few points of concern.\nExtremely tied to feature serving architecturally - by which I mean it ran as a subprocess managed by the feature server process. So:\nThere was no independent service that we could make an RPC call to - it couldn't scale independently.\nIn fact, since the feature server and the transformation server ran in the same container - there were challenges with resource contention between the two processes.\nAdditionally, the transformation server binary was built and managed by Tecton - this meant that we were in control of what dependencies were available at transformation time - users had no control over this. They couldn't bring in 3p dependencies - or even other 1st party dependencies. In fact, adding any dependency was adding baggage to all the customers who\ndidn't\nneed said dependency - and increasing our support burden.\nFinally, and probably most importantly - if you think about the permission model, you'll notice that the python process is effectively running with the same credentials as the feature server process. But.. the feature server process is a privileged process, but the code the transform server is running can be.. well, arbitrary user code. This is a security nightmare!\nV0 implementation of Realtime Feature Views\nSo, where does that leave us? Lets look at things from the lens of an ideal system to solve this problem:\nWe want to decouple transformations and other callers (feature server, ingestion, etc). We want to scale these separately.\nGoing further down this path, we want to have a different set of credentials for the transform server. This allows us to effectively isolate user code from the rest of production environment and nerf and security constraints.\nSecondary - we want the environment to be buildable\nby users\n. This will allow users to dictate the environment and dependencies - and have full control of the execution environment.\nThis has an interesting corollary - if users need to build the environment with their dependencies, they need to include the an entrypoint/runtime component that will allow our services to talk to the transformation server. Which implies that the transformation server needs to be a standalone library that users can include in their python environments and Docker images.\nV1 - Transformations in AWS Lambda\nTo satisfy these requirements, we re-architected the our feature serving system to decouple the transformation server from the feature server. The natural question being - where do transformations get executed now?\nFor this pass, we did some benchmarking on some architectures - running as a separate standalone service in our existing Kubernetes cluster, vs running the transformations on AWS Lambda.\nOur speed of iteration on AWS Lambda was actually pretty fantastic. We were able to get up and running rather quickly. Early rudimentary benchmarks promised easy scaling, but a minimal latency hit (~20ms P99 latency degradation). In our view, this latency disruption was an acceptable tradeoff given the potential cost savings (since we'd only pay for usage on transformations, and would need fewer cores for feature serving). We would also have a different, nerfed IAM role while executing the transformation Lambda. It's a natural use for this kind of use case.\nV1 implementation of Realtime Feature Views\nSo, we introduced the concept of\nenvironments\n. Quoting the docs:\nTecton allows users to create custom Python Environments for Realtime Feature Views. These environments can be created using a python\nrequirements file\nthat includes all the dependencies required in the custom environment.\nUsers could create environments via the following command:\ntecton environment create --name \"my-custom-env\" --description \"This is a custom environment\" --requirements /path/to/requirements.txt\nWhere the requirements would be something like:\n# PyPI packages\nfuzzywuzzy==0.18.0\ntecton-runtime==1.0.1\n\n# Packages from GitHub repositories\ngit+https://@github.com//.git@#egg=\n\n# Local wheel files\n./wheels/my_local_package.whl\n../other_project/wheels/another_local_package.whl\n\n# Wheel files from S3\nhttps://s3.amazonaws.com/my-bucket/wheels/s3_package_1.whl\nhttps://s3.amazonaws.com/my-bucket/wheels/s3_package_2.whl\nOur CLI would slurp up all these packages, upload them to Tecton, which would bake them into a docker image, and prepare a Lambda function that fronted the Image.\nLambda functions require\nattached execution roles\n. Under the hood, we created a new role for the function, with no permissions other than the assumeRole policy required by Lambda. This ensured that the user code couldn't modify our infrastructure, by working on the principle of least-privilege. We also ran the Lambda in a separate VPC which did\nnot\nhave an internet gateway/NAT gateway. This way we could also ensure no outbound connections to the general internet.\nThen, users could specify the names of this environment constraints in the definition of a\nRealtime Feature View\n.\nfrom tecton import realtime_feature_view, RequestSource, Attribute\nfrom tecton.types import Field, Int64, String\n\nrequest_schema = [Field(\"text\", String)]\nsimilarity_request = RequestSource(schema=request_schema)\n\n\n@realtime_feature_view(\n sources=[similarity_request],\n mode=\"python\",\n features=[Attribute(\"similarity\", Int64), Attribute(\"partial_similarity\", Int64)],\n environments=[\"my-custom-env\"],\n)\ndef fuzzy_similarity(request):\n from fuzzywuzzy import fuzz\n\n baseline = \"Mocha Cheesecake Fudge Brownie Bars\"\n result = {\n \"similarity\": fuzz.ratio(baseline, request[\"text\"]),\n \"partial_similarity\": fuzz.partial_ratio(baseline, request[\"text\"]),\n }\n return result\nUsers could then set the execution environment for a Feature Service containing the above RTFV, and under the hood we would route the transformation request to be executed on the Lambda function instead of the transformation server subprocess:\nfrom tecton import FeatureService\nfrom ads.features.realtime_feature_views.similarity import fuzzy_similarity\n\nsimilarity_feature_service = FeatureService(\n name=\"similarity_feature_service\",\n realtime_environment=\"my-custom-env\",\n online_serving_enabled=True,\n features=[fuzzy_similarity],\n)\nSo, how'd it perform?\nTerribly. 🫠\nThe problem we encountered in production was the latency impact was much,\nmuch\nworse than demonstrated in our benchmarks. Specifically with spiky traffic, cold start problems would manifest starkly, dragging our P99 latencies down by 700-1000ms. In some cases, even worse!\nThe other interesting thing we noticed was that latency would actually grow quadratically with the size of the Lambda payload. This increase in latency couldn't be tracked to our code - it was before the lambda instance even started executing our code. Our best guess is that there's some kind of request copying/buffering that's happening somewhere in Lambda machinery before our the function is ready to execute, which is introducing this latency.\nThis basically would not even support some use cases that were\nalready\nin production. A complete no-go from a product perspective. We underestimated how much of a penalty bursty traffic would incur when using Lambda, and how often we would run into cold start issues. Our benchmarks were not entirely representative of the traffic patterns we saw in production.\nWe knew we were on to something with the idea of decoupling the transformations, but we just hadn't managed to get the architecture quite right.\nThe things we\ndid\nlike:\nIndependent scaling\nLow privilege code execution\nCustom dependencies\nThe things we\ndid not\nlike:\nThe black-box nature of serverless providers.\nBasically, the challenge didn't exist in our code - which was pretty lean. The performance challenges existed\nbefore our code even started executing\n, and there was no control we had over that - that was all Lambda and how it functioned.\nLesson learned: latency sensitive use cases with spiky traffic are not a good fit for serverless approaches. The main challenge we faced is not necessarily that the latency was high - it was - but that it was\nunpredictable\n. We could not offer a strong SLO. And since the variance was due to factors out of control - back to the drawing board we went. I'll describe our next (and successful) solution next post.", "metadata": { "title": "Realtime Compute and AWS Lambda", "description": null, "keywords": null, "author": null, "og:site_name": "In the tubes", "og:type": "article", "og:title": "Realtime Compute and AWS Lambda", "og:description": "This is followup to https://blog.achals.com/some-thoughts-on-building-realtime-compute-at-tecton/ - focused on how we built a platform for user code execution at Tecton. If you haven't read it yet - you probably should, cause this post will make more sense if you're caught", "og:url": "https://blog.achals.com/realtime-compute-and-aws-lambda/", "og:image": "https://images.unsplash.com/photo-1650177926671-86ffe32b95d4?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDF8fExhbWJkYXxlbnwwfHx8fDE3NzE1NDM3ODh8MA&ixlib=rb-4.1.0&q=80&w=2000", "og:image:width": "1200", "og:image:height": "800", "twitter:card": "summary_large_image", "twitter:title": "Realtime Compute and AWS Lambda", "twitter:description": "This is followup to https://blog.achals.com/some-thoughts-on-building-realtime-compute-at-tecton/ - focused on how we built a platform for user code execution at Tecton. If you haven't read it yet - you probably should, cause this post will make more sense if you're caught", "twitter:url": "https://blog.achals.com/realtime-compute-and-aws-lambda/", "twitter:image": "https://images.unsplash.com/photo-1650177926671-86ffe32b95d4?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDF8fExhbWJkYXxlbnwwfHx8fDE3NzE1NDM3ODh8MA&ixlib=rb-4.1.0&q=80&w=2000", "twitter:label1": "Written by", "twitter:data1": "Achal Shah", "article:published_time": "2026-02-20T19:43:03.000Z", "article:modified_time": "2026-02-21T07:47:59.000Z" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/4a42ae0c6ea1/code/body-000.txt", "chars": 134, "language": "text" }, { "file": "attachments/4a42ae0c6ea1/code/body-001.txt", "chars": 416, "language": "text" }, { "file": "attachments/4a42ae0c6ea1/code/body-002.txt", "chars": 710, "language": "text" }, { "file": "attachments/4a42ae0c6ea1/code/body-003.txt", "chars": 304, "language": "text" } ], "images": [ { "status": "ok", "url": "https://images.unsplash.com/photo-1650177926671-86ffe32b95d4?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDF8fExhbWJkYXxlbnwwfHx8fDE3NzE1NDM3ODh8MA&ixlib=rb-4.1.0&q=80&w=1200", "bytes": 51315, "reused": true, "local_path": "attachments/4a42ae0c6ea1/images/000.jpg", "src": "https://images.unsplash.com/photo-1650177926671-86ffe32b95d4?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDF8fExhbWJkYXxlbnwwfHx8fDE3NzE1NDM3ODh8MA&ixlib=rb-4.1.0&q=80&w=1200", "alt": "Realtime Compute and AWS Lambda", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/7c/e6/7ce64ba2-383c-4c9e-896e-4d163a93db20/content/images/2026/02/image.png", "bytes": 114091, "reused": true, "local_path": "attachments/4a42ae0c6ea1/images/005.png", "src": "https://storage.ghost.io/c/7c/e6/7ce64ba2-383c-4c9e-896e-4d163a93db20/content/images/2026/02/image.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/7c/e6/7ce64ba2-383c-4c9e-896e-4d163a93db20/content/images/2026/02/image-2.png", "bytes": 125379, "reused": true, "local_path": "attachments/4a42ae0c6ea1/images/009.png", "src": "https://storage.ghost.io/c/7c/e6/7ce64ba2-383c-4c9e-896e-4d163a93db20/content/images/2026/02/image-2.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://blog.achals.com/author/achal/", "label": "", "type": "a" }, { "href": "https://unsplash.com/@hngstrm?utm_source=ghost&utm_medium=referral&utm_campaign=api-credit", "label": "H&CO", "type": "a" }, { "href": "https://unsplash.com/?utm_source=ghost&utm_medium=referral&utm_campaign=api-credit", "label": "Unsplash", "type": "a" }, { "href": "https://blog.achals.com/some-thoughts-on-building-realtime-compute-at-tecton/", "label": "https://blog.achals.com/some-thoughts-on-building-realtime-compute-at-tecton/", "type": "a" }, { "href": "https://docs.tecton.ai/docs/defining-features/feature-views/realtime-feature-view/realtime-feature-view-examples?ref=blog.achals.com", "label": "Python UDFs", "type": "a" }, { "href": "https://docs.tecton.ai/docs/running-in-production/scaling-feature-server?ref=blog.achals.com", "label": "Feature Server", "type": "a" }, { "href": "https://docs.tecton.ai/docs/defining-features/feature-views/realtime-feature-view/realtime-feature-view-environments/custom-realtime-feature-view-environments?ref=blog.achals.com", "label": "environments", "type": "a" }, { "href": "https://pip.pypa.io/en/stable/reference/requirements-file-format/?ref=blog.achals.com", "label": "requirements file", "type": "a" }, { "href": "https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html?ref=blog.achals.com", "label": "attached execution roles", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:47:48Z", "source_url": "https://blog.achals.com/realtime-compute-and-aws-lambda/", "papers": [], "actual_video_files": [], "body_chars": 9981, "expected_body_images": 3, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/4a42ae0c6ea1/pages/source.html", "local_html": "attachments/4a42ae0c6ea1/pages/article.html", "body_base_path": "attachments/4a42ae0c6ea1/pages/", "body_markdown_path": "attachments/4a42ae0c6ea1/pages/article.md" }