{ "source_file": "0930", "source_platform": "个人博客", "author": "Vicki Boykis", "title_listed": "Everything I did in 2024", "keywords": "年度总结,LLM,评估", "domain": "机器学习", "content_type": "年度回顾", "is_paid_flag": "否", "url": "https://vickiboykis.com/2025/01/03/everything-i-did-in-2024/", "publish_date": "2025-01", "corpus_stage": "机器学习", "notes": "2024 年度总结", "id": "ed0d35811bfd", "collected_at": "2026-10-02T15:23:03", "title": "Everything I did in 2024 | ✰Vicki Boykis✰", "body_markdown": "Everything I did in 2024\nJan 3 2025\nI want to get back into writing more regularly this year, so in light of that, here’s my last year in review.\nEvaluating LLMs\nLike many of us in tech, I spent a large portion of 2024 thinking about and working with LLMs, but I was lucky enough to do it for work. I spent the year designing, building, open-sourcing, (and naming! 🐊)\nan application to evaluate LLMs\n,\nLumigator.\nIn support of that work, I\ndid open-source work in the LLM ecosystem\nand learned a ton of stuff along the way. Just when I thought I had\nreached machine learning enlightenment\n, I learned a ton about the weird\nnondeterministic properties of LLMs\n, their evaluation methods, Ray + Ray Serve, vLLM, Llamafile,\nGGUF\n, FastAPI, OpenAPI-compatible APIs, and much, much, much more. I am hoping to translate some of these into blog posts as well.\nThere is always more to learn in machine learning, particularly in the fast-moving world at the bleeding edge.\nLucikly, the fundamentals always remain the same.\nMy biggest takeaways from this year are that:\nMachine learning in industry is mostly still made up of good engineering\n, and in order to work with LLMs, you need to have both machine learning context and engineering discipline. There’s some debate whether the “old” machine learning skills (gradient boosted trees anyone? 👵) still apply in this Brave New World of\nNonTabular Scraped Internet\n, and\nI am of the opinion\nthat these skills are more valuable than ever, particularly as engineers without ML context enter the frustrating world of building with non-deterministic experimental loops.\nBut what’s even more valuable is being able to\nship things that hold engineering rigor\n. Even if you are deep in academic research,\ntaking an engineering approach\nwill make your life significantly easier.\nAs part of this, I spent a significant amount of the year\nwidening my own context window\nabout engineering best practices: Over the summer, I\nlearned C\nand this fall, I learned JavaScript and Go and\nbuilt my first working project, Gitfeed\n. I am particularly enamored with Go: it’s so boring, so clean, and so easy to build things that are simple and move quickly out of the box compared to the Python ecosystem.\nThis is not to say that the\nPython world\n, which is my first love, is not\nchanging and evolving\n. I\nmoved to\nuv\nfrom\npyenv\nfor both work and personal projects in 2024 and have never looked back, particularly as\nuv\ncontinues to work\nto gain support for PyTorch\nand all of its cuda-related install issues.\nThere is too much going on in the model landscape:\ntoo many models, too much choice, too many platforms, too much money, too much drama. My main strategy is to follow what’s going on on a daily basis to keep my finger on the pulse and then basically ignore it. If people mention it again over the course of 2-3 months, continue to pay attention.\nIn light of this, some contradictary advice if you’re in the space is that you should also\nalways be playing around with stuff\n, trying it out, building it and breakign it. There are some\nclassic papers and books\nfor understanding the context and theory, but most of the stuff I read is still in\nr/LocalLLaMA\nor in various tweets/skeets and blog posts. We are not at the level yet for this stuff where cannon exists, although I have noticed that there are technical books being published about AI engineering this year, which means we are starting to solidify.\nSome LLM tools I tried out and loved this year were\nllamafile\n, for getting started with GGUF transformer and embedding models with a server extremely quickly, and\nollama + openwebui\nfor an experience that is nearly identical to using Claude. In fact, I’ve switched over from Claude to using\nmistral:latest\nlocally for most of my LLM usage, which is basically code search. Mistral is a model that has consistently worked well for me both at work and for my own purposes, and I’ve enjoyed the chance to pit it against newer models like\nQwen2.5 Coder\n, Llama2, and phi.\nI am still having a hard time wrapping my mind around what LLM models are useful for, in general, and for me personally. I\nspend a lot\nof time\nreading blog posts\nabout\nhow other (smarter) people use LLMS\nand against academic papers that study the success of these models at machin learning tasks stemming in their roots as NLP models focusing on problems like text completion, classification, translation, and summarization.\nA very cool and fun related thing I did while reviewing the “classics” was edit this wonderful series of posts by Katharine about\nmodel memorization\nwhich you should read if you’re interested in their internals.\nI also spent a significant part of the year working on web development. I wrote earlier in the year that there is a\nbifurcation in how we consume LLMs\n:\nThe LLM ecosystem is currently bifurcated between HuggingFace and OpenAI compatibility: An interesting pattern has developed in my development work on open-source in LLMs. It’s become clear to me that, in this new space of developer tooling around transformer-style language models at an industrial scale, you are generally conforming to be downstream of one of two interfaces:\nmodels that are trained and hosted using HuggingFace libraries\nand particularly the HuggingFace hub as infrastructure - in practicality, this means dependence on PyTorch’s programming paradigm, which HuggingFace tools wrap (although they now also provide interop between Tensorflow and JAX)\nModels that are available via API endpoints\n, particularly as hosted by OpenAI. Given that OpenAI was a first mover in the product LLM space, they currently have the API advantage, and many tools that have developed have developed OpenAI-compatible endpoints which don’t always mean using OpenAI, but conform to the same set of patterns that the Chat Completions API v1/chat/completions offers. For example, adding OpenAI-style interop chat completions allowed us to stand up our own vLLM OpenAI-compatible server that works against models we’ve started with on HuggingFace and fine-tuned locally.\nIf you want to be successful in this space today and you’d like to cater to a broad audience, you as a library or service provider have to be able to interface with both of these.\nI still believe this is true and it will be interesting to see how this develops over the course of the next year as APIs and their model ecosystems become even broader.\nThese models will integrate with, not replace traditional machine learning systems.\nIn the beginning, everyone thought that we could replace whole classes of engineering and machine learning problems with LLMs. It’s becoming increasingly clearer that one big-ass model is not as a good as many smaller models for specific tasks performed in industry, which makes complete sense because LLMs as a concept rose out of academic labs whose goals are to generalize on out-of-sample problems, whereas the task of industry is to specialize in healthcare or OCR document scanning for legal, or sentiment analysis for social media. Even if fine-tuning did decline as a practice this year, people are making models specialized in a lot of deifferent ways, which I think the rise of agents at the end of this year really makes clear.\nAn area that has particularly interested me is the integration of LLMs into recommender systems, my favorite area of applied machine learning. Earlier last year, I, along with\nJames\nand\nRavi\n, started a position paper on what will change in recommender systems in the light of LLMs based on a joke tweet I had made,\npersonalized recommendations based on implicit matrix factorization from data acquired through large log streaming architectures were a low interest rate phenomenon\nMy hypothesis was that companies used to collect a lot of streaming user data, particularly in social and other B2C settings, that were then used for personalization. Now, all the data collection happens publicly at the level of the interet,\nwhere it is re-compressed into LLMs.\nWhat does this mean for personalization, now?\nUnfortunately we didn’t finish the post (maybe someday?), but our main hypothesis was that traditional recsys will not be replaced with, but instead augmented with LLMs, that LLMs could help fill the cold-start gap because they’re so good at zero-shot retrieval and general topic classification and can serve as places to augment cold-start problems and in onboarding, and that LLMs could offer more explainability in recommendations. This has\nso far\nproven to be true.\nItaly + Learning Italian\nI had the extreme pleasure of being invited to give a keynote at\nPyCon Italia\nin Florence in May. I took all these learnings that had been running on crazed hamster wheels in my brain and threw them into a\ntalk based loosely on the Decameron\n.\nIn addition to subjecting my poor audience to jokes about the Medicis, I also gave the intro in Italian to an audience of +100, which was extremely nervewracking. I’ve been wanting to study Italian since I was 18 and the stars aligned two years ago for me to seriously start. and I’ve been learning ever singe. This year, I also managed to read my first (Level A1/A2) books in Italian.\nMy family also came to Italy. It was the kids’ first trip to Europe. Everyone had an amazing time (at least after my talk was over), ate a lot of gelato, and experienced Rome and Florence in the spring. One of my favorite moments of the trip (and of my year) was listening to a\nRicchi e Poveri\ncover band playing at a disco outside our hotel at 11:30 at night in Florence.\nInspired by all of this, and to take a break from LLMs, I also started learning how to mix music, and hopefully will have something more concrete to write/share about DJing this year.\nI also\nmanaged to read some books.\nShift in Social Strategy\nAs I spent the year trying my best to keep up with the wild, unruly, and growing LLM landscape, another part of my ecosystem was wilting. It’s clear for all intents and purposes that Twitter as we knew her is dead. It doesn’t have the juice anymore, as the kids say. I’m surprised that it died from human intervention rather than engineering failure, but as anyone who’s read my newsletter knows, this is by far the most likely outcome for techno-social systems. I have been wanting to write a long eulogy on my personal experinece and what Twitter meant to me: it was a place where I made friends, learned to be a serious programmer, worked out my best ideas, got leads for jobs, started a newsletter,\norganized Normconf\n, and most importantly, had fun. But in the end, it all died with a whimper, and I, like many others, just quietly left.\nI’ve always been a firm believer in owning your own internet space, and I continued to do that by blogging, and moving from Substack to Buttondown, but I also need micro social interaction because I get energy and ideas from it. So, I engaged more on Bluesky, where I’ve been since last year, and was pleasantly surprised to see the machine learning community start to migrate there. Amost immediately as soon as I came back to the platform, I had a conversation that\nled to an idea for a post.\nA lot of people have (rightfully so) sworn off agglomerated social\nin favor of group chats.\nThis makes complete sense to me, yet as a creator and someone who thrives on community, I was sorely missing this space until I came back to Bluesky. There is a lot of conversation around protocol versus platform and what it means for Bluesky the app versus Bluesky the protocol to succeed.\nIn particular, I encourage anyone interested in the space to read\nthis series of exchanges\non the question\nbut I remain hopeful that that space grows and remains a place where users can choose and experiment. It allowed me, at the end of the year,\nto hack on gitfeed\n, which is very, very very cool.\n2025\nWhat next? Who knows, who can even predict the future? Not machine learning models, and certainly not me. I’m just excited to keep learning and building and making and dreaming, and, hopefully writing more about it all.\n#machine learning\n#career advice\n#golang\n#learning", "metadata": { "title": "Everything I did in 2024 | ✰Vicki Boykis✰", "description": "A LOT of LLMs", "keywords": "machine learning,career advice,golang,learning,", "author": null, "og:type": "article", "og:url": "https://vickiboykis.com/2025/01/03/everything-i-did-in-2024/", "og:title": "Everything I did in 2024", "og:description": "A LOT of LLMs", "og:image": "https://vickiboykis.com/images/902697ae-eefa-4315-bf30-f4603dc1489b.jpg", "twitter:card": "summary", "twitter:url": "https://vickiboykis.com/2025/01/03/everything-i-did-in-2024/", "twitter:title": "Everything I did in 2024", "twitter:description": "A LOT of LLMs", "twitter:image": "https://vickiboykis.com/images/902697ae-eefa-4315-bf30-f4603dc1489b.jpg", "twitter:creator": "@vboykis", "twitter:site": "@vboykis", "article:published_time": "2025-01-03T00:00:00+00:00", "article:modified_time": "2025-01-03T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://vickiboykis.com/images/902697ae-eefa-4315-bf30-f4603dc1489b.jpg", "bytes": 233517, "reused": false, "local_path": "attachments/ed0d35811bfd/images/body-f495a1b0084ecff4a74b.jpg", "src": "https://vickiboykis.com/images/902697ae-eefa-4315-bf30-f4603dc1489b.jpg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://vickiboykis.com/images/bceed358-ffa0-4909-a353-834b30d215e4.jpg", "bytes": 153289, "reused": false, "local_path": "attachments/ed0d35811bfd/images/body-90a0ec6d0f50305910b3.jpg", "src": "https://vickiboykis.com/images/bceed358-ffa0-4909-a353-834b30d215e4.jpg", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=6xTTHC2Y18E", "src": "https://www.youtube.com/watch?v=6xTTHC2Y18E", "status": "ok", "kind": "single_video", "local_path": "attachments/ed0d35811bfd/videos/repair-6xTTHC2Y18E.mp4", "bytes": 14890099, "valid": true, "height": 1080, "width": 1920, "duration": "210.941000", "reused": false } ], "links": [ { "href": "https://blog.mozilla.ai/lets-build-an-app-for-evaluating-llms/", "label": "an application to evaluate LLMs", "type": "a" }, { "href": "https://github.com/mozilla-ai/lumigator", "label": "Lumigator.", "type": "a" }, { "href": "https://blog.mozilla.ai/open-source-in-the-age-of-llms/", "label": "did open-source work in the LLM ecosystem", "type": "a" }, { "href": "https://vickiboykis.com/2021/09/23/reaching-mle-machine-learning-enlightenment/", "label": "reached machine learning enlightenment", "type": "a" }, { "href": "https://vickiboykis.com/2024/05/06/weve-been-put-in-the-vibe-space/", "label": "nondeterministic properties of LLMs", "type": "a" }, { "href": "https://vickiboykis.com/2024/02/28/gguf-the-long-way-around/", "label": "GGUF", "type": "a" }, { "href": "https://vickiboykis.com/2022/11/10/how-i-learn-machine-learning/", "label": "Lucikly, the fundamentals always remain the same.", "type": "a" }, { "href": "https://m-clark.github.io/posts/2021-07-15-dl-for-tabular/index.html", "label": "NonTabular Scraped Internet", "type": "a" }, { "href": "https://timkellogg.me/blog/2024/12/10/ml-liability", "label": "I am of the opinion", "type": "a" }, { "href": "https://vickiboykis.com/2022/12/05/the-cloudy-layers-of-modern-day-programming/", "label": "ship things that hold engineering rigor", "type": "a" }, { "href": "https://bsky.app/profile/eugenevinitsky.bsky.social/post/3ldm5i4ljks2z", "label": "taking an engineering approach", "type": "a" }, { "href": "https://vickiboykis.com/2023/09/13/build-and-keep-your-context-window/", "label": "widening my own context window", "type": "a" }, { "href": "https://publish.obsidian.md/learning-c/Learning+C/Learn+C+Programming+and+OOP", "label": "learned C", "type": "a" }, { "href": "https://github.com/veekaybee/gitfeed", "label": "built my first working project, Gitfeed", "type": "a" }, { "href": "https://pydevtools.com/blog/effective-python-developer-tooling-in-december-2024/", "label": "Python world", "type": "a" }, { "href": "https://thedataquarry.com/posts/towards-a-unified-python-toolchain/", "label": "changing and evolving", "type": "a" }, { "href": "https://bsky.app/profile/vickiboykis.com/post/3lazmcuftus25", "label": "moved to uv from pyenv", "type": "a" }, { "href": "https://docs.astral.sh/uv/guides/integration/pytorch/", "label": "to gain support for PyTorch", "type": "a" }, { "href": "https://gist.github.com/veekaybee/be375ab33085102f9027853128dc5f0e", "label": "classic papers and books", "type": "a" }, { "href": "https://github.com/Mozilla-Ocho/llamafile", "label": "llamafile", "type": "a" }, { "href": "https://simonwillison.net/2024/Nov/12/qwen25-coder/", "label": "Qwen2.5 Coder", "type": "a" }, { "href": "https://mathstodon.xyz/@tao/113132502735585408", "label": "spend a lot", "type": "a" }, { "href": "https://antirez.com/news/140", "label": "reading blog posts", "type": "a" }, { "href": "https://nicholas.carlini.com/writing/2024/how-i-use-ai.html", "label": "how other (smarter) people use LLMS", "type": "a" }, { "href": "https://blog.kjamistan.com/a-deep-dive-into-memorization-in-deep-learning.html", "label": "model memorization", "type": "a" }, { "href": "https://jfkirk.github.io/posts/trustworthiness-ai/", "label": "James", "type": "a" }, { "href": "https://bsky.app/profile/ravimody.bsky.social", "label": "Ravi", "type": "a" }, { "href": "https://vickiboykis.com/2024/01/15/whats-new-with-ml-in-production/", "label": "where it is re-compressed into LLMs.", "type": "a" }, { "href": "https://arxiv.org/html/2410.17152v1", "label": "so far", "type": "a" }, { "href": "https://arxiv.org/abs/2307.02046", "label": "proven to be true.", "type": "a" }, { "href": "https://2025.pycon.it/en", "label": "PyCon Italia", "type": "a" }, { "href": "https://vickiboykis.com/2024/05/20/dont-worry-about-llms/", "label": "talk based loosely on the Decameron", "type": "a" }, { "href": "https://www.youtube.com/watch?v=6xTTHC2Y18E", "label": "Ricchi e Poveri", "type": "a" }, { "href": "https://vickiboykis.com/essays/2024-12-31-favorite-books/", "label": "managed to read some books.", "type": "a" }, { "href": "https://vickiboykis.com/2022/12/22/everything-i-learned-about-accidentally-running-a-successful-tech-conference/", "label": "organized Normconf", "type": "a" }, { "href": "https://vickiboykis.com/2024/11/09/why-are-we-using-llms-as-calculators/", "label": "led to an idea for a post.", "type": "a" }, { "href": "https://vickiboykis.com/2024/09/19/dead-internet-souls/", "label": "in favor of group chats.", "type": "a" }, { "href": "https://dustycloud.org/blog/how-decentralized-is-bluesky/", "label": "this series of exchanges", "type": "a" }, { "href": "https://whtwnd.com/bnewbold.net/3lbvbtqrg5t2t", "label": "on the question", "type": "a" }, { "href": "https://vickiboykis.com/tags/machine-learning/", "label": "#machine learning", "type": "a" }, { "href": "https://vickiboykis.com/tags/career-advice/", "label": "#career advice", "type": "a" }, { "href": "https://vickiboykis.com/tags/golang/", "label": "#golang", "type": "a" }, { "href": "https://vickiboykis.com/tags/learning/", "label": "#learning", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:53:15Z", "source_url": "https://vickiboykis.com/2025/01/03/everything-i-did-in-2024/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2307.02046", "bytes": 8074212, "reused": false, "local_path": "papers/repair/75f4f0b5ec21c50393a0.pdf" } ], "actual_video_files": [], "body_chars": 12078, "expected_body_images": 2, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/ed0d35811bfd/pages/source.html", "local_html": "attachments/ed0d35811bfd/pages/article.html", "body_base_path": "attachments/ed0d35811bfd/pages/", "body_markdown_path": "attachments/ed0d35811bfd/pages/article.md" }