{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "Key trends from NeurIPS 2019",
  "keywords": "NeurIPS,趋势,研究",
  "domain": "机器学习",
  "content_type": "行业分析",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2019/12/18/key-trends-neurips-2019.html",
  "publish_date": "2019-12",
  "corpus_stage": "预训练/SFT",
  "notes": "NeurIPS 2019关键趋势总结",
  "id": "660be759316e",
  "collected_at": "2026-10-02T14:14:29",
  "title": "Key trends from NeurIPS 2019",
  "body_markdown": "[\nTwitter thread\n]\nWith 51 workshops, 1428 accepted papers, and 13k attendees, saying that NeurIPS is overwhelming is an understatement. I did my best to summarize the key trends I got from the conference. This post is generously edited by the wonderful\nAndrey Kurenkov\n.\nDisclaimer: This post doesn’t reflect the view of any of the organizations I’m associated with. NeurIPS is huge with a lot to take in, so I might get something wrong. Feedback is welcome!\nTable of Contents\nDeconstructing the deep learning black box\nNew approaches to deep learning\n2.1 Deep learning with Bayesian principles\n2.2 Graph neural networks\n2.3 Convex optimization\nNeuroscience x Machine Learning\nKeyword analysis\nNeurIPS by numbers\nConclusion\n1. Deconstructing the deep learning black box\nLately, there has been a lot of reflection on the limitations of deep learning. A few examples:\nFacebook’s director of AI is worried about the computational wall. Companies should not expect to keep making progress just with bigger deep learning systems because “\nright now, an experiment might be in seven figures, but it’s not going to go to nine or ten figures .. nobody can afford that.\n”\nYoshua Bengio gave Gary Marcus as an example of someone who frequently points out deep learning’s limitations. Bengio summarized Gary Marcus’s view as “\nLook, deep learning doesn’t work.\n” Gary Marcus\ndisputed this characterization\n.\nYann Lecun addressed this trend: “\nI don’t get why, all of a sudden, we read all these stories and tweets that claim ‘progress in AI is slowing’ and ‘Deep Learning is hitting a wall’ … I have been pointing out those two limitations and challenges in pretty much every single one of my talks of the last 5 years … So, no, the identification of these limitations is not new. And there is no slow down.\n”\nIn this climate, it’s nice to see an explosion of papers exploring the theory behind deep learning, why and how it works. At this year’s NeurIPS, there are 31 papers on the convergence of various techniques. The outstanding new directions paper award goes to Vaishnavh Nagarajan and J. Zico Kolter’s\nUniform convergence may be unable to explain generalization in deep learning\n, with the thesis that the uniform convergence theory by itself can’t explain the ability of deep learning to generalize. As the dataset size increases, theoretical bounds on generalization gap (the gap between a model’s performance on seen and unseen data) also increase while the empirical generalization gap decreases.\nImage from Vaishnavh Nagarajan's oral presentation\nThe\nneural tangent kernel (NTK)\nis a recent research direction which aims to understand the optimization and generalization of neural networks. It came up in several spotlight talks and many conversations I had with people at NeuIPS. By building on the well-known notion that fully connected neural nets are equivalent to Gaussian processes in the infinite-width limit, Arthur Jacot et al. were able to study their training dynamics in function space instead of parameter space. They proved that “\nduring gradient descent on the parameters of an ANN, the network function (which maps input vectors to output vectors) follows the kernel gradient of the functional cost w.r.t. a new kernel: the NTK\n.” They also showed that when a finite layer version of the NTK is trained with gradient descent, its performance converges to the infinite-width limit NTK and then stays constant during training.\nSome NeurIPS papers that build upon NTK:\nLearning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers\nOn the Inductive Bias of Neural Tangent Kernels\nHowever, many believe that NTK can’t fully explain deep learning. The hyperparameter settings needed for a neural net to approach the NTK regime – small learning rate, large initialization, no weight decay – aren’t often used to train neural networks in practice. The NTK perspective also states that neural networks would only generalize as well as kernel methods, but empirically they have been observed to generalize better.\nColin Wei et. al’s paper\nRegularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel\ntheoretically demonstrates that neural networks with weight decay can generalize much better than NTK, suggesting that studying L2-regularized neural networks could offer better insights into generalization. The following papers from this NeurIPS also show that conventional neural networks can outperform NTK:\nWhat Can ResNet Learn Efficiently, Going Beyond Kernels?\nLimitations of Lazy Training of Two-layers Neural Network\nMany papers analyze the behavior of different components of neural networks. Chulhee Yun et al. presented\nSmall ReLU networks are powerful memorizers: a tight analysis of memorization capacity\nand showed that “\n3-layer ReLU networks with Omega(sqrt(N)) hidden nodes can perfectly memorize most datasets with N points\n.”\nShirin Jalali et al.’s paper\nEfficient Deep Learning of Gaussian Mixture Models\nstarted with the question: “\nThe universal approximation theorem states that any regular function can be approximated closely using a single hidden layer neural network. Can depth make it more efficient?\n” They showed that in the case of optimal Bayesian classification of Gaussian Mixture Models, such functions can be approximated with arbitrary precision using O(exp(n)) nodes in a neural network with one hidden layer, but only O(n) nodes in a two-layer network.\nIn one of the more practical papers,\nControl Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence\n, Fengxiang He and team trained 1,600 ResNet-110 and VGG-19 models using SGD on CIFAR datasets and found that the generalization capacity of these models correlate negatively with batch size, positively with learning rate, and negatively with batch size/learning rate ratio.\nImage by He et al.\nAt the same time, Yuanzhi Li et al.’s\nTowards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks\nstates that “\na two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start … because the small learning rate model first memorizes low noise, hard-to-fit patterns, it generalizes worse on higher noise, easier-to-fit patterns than its large learning rate counterpart.\n”\nWhile these theoretical analyses are fascinating and important, it can be hard to aggregate them into a big picture because each of them focuses on one narrow aspect of the system.\n2. New approaches to deep learning\nNeurIPS this year featured a wide range of methods outside stacking layers on top of each other. The three directions I’m excited about: Bayesian learning, graph neural networks, and convex optimization.\n2.1 Deep learning with Bayesian principles\nBayesian learning and deep learning are very different, as highlighted by Emtiyaz Khan in his invited talk\nDeep Learning with Bayesian Principles\n. According to Khan, deep learning uses a ‘trial and error’ approach – let’s see where experiments take us – whereas Bayesian principles force you to think of a hypothesis (priors) beforehand.\nCompared to regular deep learning, there are two main appeals to Bayesian deep learning: uncertainty estimation and better generalization on small datasets. In real world applications, it’s not enough that a system makes a prediction. It’s important to know how certain it’s about each prediction. For example, a prediction of cancer with 50.1% certainty requires different treatments from the same prediction with 99.9% certainty. With Bayesian learning, uncertainty estimation is a built-in feature.\nTraditional neural networks give single point estimates – they output a prediction on a datapoint using a single set of weights. Bayesian neural networks, on the other hand, use a probability distribution over the network weights and output an average prediction of all sets of weights in that distribution, which has the same effect as averaging over many neural networks. Thus, Bayesian neural networks are natural ensembles, which act like regularization and can prevent overfitting.\nBayesian neural networks with millions of parameters are still computationally expensive to train. Converging to a posterior might take weeks, so approximating methods such as variational inference have become popular. The\nProbabilistic Methods – Variational Inference\nsession features 10 papers on this variational Bayesian approach.\nSome NeurIPS papers on Bayesian deep learning that I enjoyed reading:\nImportance Weighted Hierarchical Variational Inference\nA Simple Baseline for Bayesian Uncertainty in Deep Learning\nPractical Deep Learning with Bayesian Principles\n2.2 Graph neural networks (GNNs)\nFor years, I’ve been talking about how graph theory is among the most underrated topics in machine learning. I’m glad to see graphs are all the rage at NeurIPS this year.\nGraph representation learning is the most popular workshop of the day at\n#NeurIPS2019\n. Amazing how far the field has advanced. I did not imagine so many people would get into this when I started working on graph neural nets back in 2015 during an internship. Time flies...\npic.twitter.com/H8LM0BMvK4\n— Yujia Li (@liyuajia)\nDecember 13, 2019\nGraphs are beautiful and natural representations for many types of data such as social networks, knowledge bases, game states. The user-item data used for recommendation systems can be represented as a bipartite graph in which one disjoint set consists of users and another consists of items.\nGraphs can also represent outputs of neural networks. As Yoshua Bengio reminded us in his invited talk, any joint distribution can be represented as a factor graph.\nThis makes graph neural networks perfect for tasks such as combinatorial optimization (e.g. travelling salesman, scheduling), identity matching (is this Twitter user the same as this Facebook user?), recommendation systems.\nThe most popular graph neural network is graph convolutional neural network (GCNN), which is expected since they both encode local information. Convolutions encode a bias towards finding relationships between neighboring parts of the inputs. Graphs encode the most closely related parts of the input via edges.\nImage by Gasse et al.\nSome papers on GNNs that I like:\nExact Combinatorial Optimization with Graph Convolutional Neural Networks\nYes, there’s a paper that fuses two hottest trends this year:\nGraph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels\nMy favorite poster presentation at NeurIPS:\n(Nearly) Efficient Algorithms for the Graph Matching Problem on Correlated Random Graphs\nRecommended reading outside NeurIPS:\nThomas N. Kipf’s\nGraph Convolutional Networks blog post\nKung-Hsiang, Huang’s\nA Gentle Introduction to Graph Neural Networks (Basics, DeepWalk, and GraphSage)\n2.3 Convex optimization\nI’ve low-key worshipped Stephen Boyd’s work on\nconvex optimization\nfrom afar, so it was delightful to see it getting more popular at NeurIPS – there are 32 papers related to the topic (\n1\n,\n2\n). Stephen Boyd and J. Zico Kolter’s labs also presented their paper\nDifferentiable Convex Optimization Layers\nwhich showed how to differentiate through the solutions of convex optimization problems, making it possible to embed them in differentiable programs (e.g. neural networks) and learn them from data.\nConvex optimization problems are appealing because they can be solved exactly (1e-10 error tolerance is attainable) and quickly. They also don’t produce weird/unexpected outputs, which is crucial for real-world applications. Even though many problems encountered in the wild are nonconvex, decomposing them into a sequence of convex problems can work well.\nNeural networks are also trained using algorithms for convex optimization. However, while the emphasis in neural networks is to learn things from scratch, in an end-to-end fashion, applications of convex optimization problems emphasize modeling systems explicitly, using domain-specific knowledge. When it’s possible to model a system explicitly, in a convex way, usually much less data is needed. The work on differentiable convex optimization layers is one way to mix the benefits of end-to-end learning and explicit modeling.\nConvex optimization is especially useful when you want to control a system’s outputs. For example,\nSpaceX uses convex optimization to land its rockets\n, and BlackRock uses it for trading algorithms. It’d be really cool to see convex optimization used in deep learning, the way Bayesian learning is now.\nSome NeurIPS papers on convex optimization recommended by Akshay Agrawal.\nAcceleration via Symplectic Discretization of High-Resolution Differential Equations\nHamiltonian descent for composite objectives\n3. Neuroscience x Machine Learning\nAccording to\nthe analysis of NeurIPS 2019 Program Chair Hugo Larochelle\n, the category that got the highest pump in acceptance rate is Neuroscience. In their invited talks, both Yoshua Bengio (\nFrom System 1 Deep Learning to System 2 Deep Learning\n) and Blaise Aguera y Arcas (\nSocial Intelligence\n) urged the machine learning community to think more about the biological roots of natural intelligence.\nBengio’s talk introduced consciousness into the mainstream machine learning vocabulary. The core ingredient of Bengio’s consciousness is attention. He compared machine attention mechanism to the way our brains choose what to pay attention to: “\nMachine learning can be used to help brain scientists better understand consciousness, but what we understand of consciousness can also help machine learning develop better capabilities.\n” According to Bengio, a consciousness-inspired approach is the way to go if we want machine learning algorithms that can generalize to out-of-distribution samples.\nAguera y Arcas’s talk is my favorite at the conference. It’s theoretically rigorous but practical. He argued how optimization is inadequate to capture human-like intelligence: “\nOptimization is not how life works… Brains don’t just evaluate a function. They develop. They’re self-modifying. They learn through experience. A function doesn’t have those things.\n” He called for “\na more general, biologically inspired synapse update rule that allows but doesn’t require a loss function and gradient descent.\n”\nThis NeurIPS trend is consistent with my observation that many people in AI are moving to neuroscience. They bring neuroscience back into machine learning.\nSome of the smartest people I know are leaving AI research for engineering/neuroscience. Their reasons?\n1. We need to understand how humans learn to teach machines to learn.\n2. Research should be hypothesis -> experiments, but AI research rn is experiments -> justifying results.\n— Chip Huyen (@chipro)\nNovember 15, 2019\n4. Keyword analysis\nLet’s check out a more global view of what papers at the conference were about. I first visualized all the 1,011 paper titles from NeurIPS 2018 and 1,428 papers titles from NeurIPS 2019 with\nvennclouds\n. The black area in the middle is the list of words that are common in both 2018 and 2019 papers.\nI then measured the proportional percentage change of these keywords from 2018 to 2019. For example, if in 2018, 1% of all the accepted papers have the keyword ‘X’ and in 2019, that number is 2%, then the proportional change is (2 - 1) / 1 = 100%. I plotted the keywords with the absolute proportional change of at least 20%.\nKey points\n:\nReinforcement learning is gaining ground even outside robotics. Some of the keywords with significant positive change are bandit, feedback, regret, control.\nGenerative models are still popular. GANs still capture our imagination, but are much less hype-y.\nRecurrent and convolutional neural networks are literally so last year.\nHardware keyword is also on the rise, signaling more hardware-aware algorithms. This is the answer to the concern that hardware is the bottleneck in machine learning.\nI’m sad that data is on the decline. I was so excited to check out the\nAlgorithms – Missing Data poster session\nonly to find out it has one poster\nMissing Not at Random in Matrix Completion: The Effectiveness of Estimating Missingness Probabilities Under a Low Nuclear Norm Assumption\n.\nMeta sees the highest increase in proportion this year. Meta-meme by\nJesse Mu\n.\nEven though Bayesian goes down, uncertainty goes up. Last year, there were many papers that use Bayesian principles, but not for deep learning.\n5. NeurIPS by numbers\n7k\npapers submitted to the main conference.\n1,428\naccepted papers.\n21%\nacceptance rate.\n13k\nattendees, which, by my estimation, means at least half of all attendees don’t present a paper there.\n57\nworkshops, 4 focused on inclusion: Black in AI, Women in Machine Learning, LatinX in AI, Queer in AI, New In Machine Learning, Machine Learning Competitions for All.\n16k\npages of conference proceedings.\n12%\nof all the accepted papers have at least one author from Google or Deepmind.\n87\npapers are from Stanford, making it the academic institution with the highest number of accepted papers.\nStuck at the airport so I browsed\n#neurips2019\npapers.\nOut of 1429 accepted papers, 167 (~12%) have at least one author from Google/DeepMind, same as Microsoft, Facebook, IBM, & Amazon combined.\nIs there any stats on the % of reviewers who are Google/DeepMind affiliated?\npic.twitter.com/bXPoB135PA\n— Chip Huyen (@chipro)\nSeptember 7, 2019\n250\npapers, 16.7%, are on applications.\n648\nis the number of citations as of Dec 2019 for the test of time award paper, Lin Xiao’s\nDual Averaging Method for Regularized Stochastic Learning and Online Optimization\n. This is a proof that citation count and contribution don’t necessarily correlate.\n75%\nof the papers with link to code in the camera-ready version, compared to only 50% from last year.\n2,255\nreviews mention looking at the code accompanying submission.\n173\npapers claimed for the\nReproducibility Challenge on OpenReview\n.\n31\nposters at the\nNeurIPS Workshop on Machine Learning for Creativity and Design\n. Several people have told me it’s their favorite part of NeurIPS. Shout out to Good Kid the band for their amazing performance at the closing banquet. Check out\ntheir music on Spotify\nif you haven’t already!\nSome days they are ML researchers, other days they are rockstars; tonight they are both! - Good Kid Band\npic.twitter.com/Y8Y7eYrGkI\n— Daniel Nkemelu (@DanielNkemelu)\nDecember 15, 2019\n11\ntalks at the first\nRetrospectives: A Venue for Self-Reflection in ML Research\nworkshop, which was another favorite.\nThe atmosphere of NeurIPS is captured well by atteendess on Twitter.\nA fun little demonstration of the scale of\n#NeurIPS2019\n- video of people heading into keynote talk.\nThis is 9 minutes condensed down to 15 seconds, and this is not even close to all the attendees!\npic.twitter.com/1VqAHZoqtj\n— Andrey Kurenkov 🤖 @ Neurips (@andrey_kurenkov)\nDecember 12, 2019\nI went to NeurIPS for the first time in 2015. I believe it was in Montreal. I wrote about my terrible experience and was going to share anonymously but friends told me that it would be obvious who wrote it :) 4 years later, this conference has changed a lot 1\\\n— Timnit Gebru (@timnitGebru)\nDecember 15, 2019\nConclusion\nI found NeurIPS overwhelming, both knowledge-wise and people-wise. I don’t think anyone can read 16k pages of the conference proceedings. The poster sessions are oversubscribed which makes it hard to talk to the authors. I undoubtedly missed out on a lot. If there’s a paper you think I should read, please let me know!\nHowever, the massiveness of the conference also means a confluence of many research directions and people to talk to. It was nice to be exposed to work outside my subfield and learn from researchers whose backgrounds and interests are different from my own.\nIt was also great to see the research community moving away from the ‘bigger, better’ approach. My impression walking around the poster sessions is that many papers only experimented on small datasets such as MNIST and CIFAR. The best paper award, Ilias Diakonikolas et al.’s\nDistribution-Independent PAC Learning of Halfspaces with Massart Noise\n, doesn’t have any experiment.\nI’ve often heard young researchers worrying over having to join big research labs to get access to computing resources, but NeurIPS proves that you can make important contributions without throwing data and compute at a problem.\nDuring a\nNewInML\npanel that I was a part of, someone said he didn’t see how the majority of papers at NeurIPS could be used in production.\nNeil Lawrence\nsaid maybe he should look into other conferences. NeurIPS is more theoretical than many other machine learning conferences, and it’s okay. It’s important to pursue fundamental research.\nOverall, I had a great time at NeurIPS, and am planning on attending it again next year. However, for those new in machine learning, I’d recommend\nICLR\nas their first academic conference. It’s smaller, shorter, and more application-oriented. Next year, ICLR will be in Ethiopia, an amazing country to visit.\nAcknowledgment\nI’d like to thank\nAkshay Agrawal\n, Andrey Kurenkov, and\nColin Wei\nfor helping me with this draft. Fun fact: Colin Wei is one of a few PhD candidates with 4 papers accepted at NeurIPS this year – 3 of them made spotlight!",
  "metadata": {
    "title": "Key trends from NeurIPS 2019",
    "description": "[Twitter thread]",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "Key trends from NeurIPS 2019",
    "og:locale": "en_US",
    "og:description": "[Twitter thread]",
    "og:url": "https://huyenchip.com/2019/12/18/key-trends-neurips-2019.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/neurips2019/9_keywords.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2019-12-18T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/1_generation_bounds.png",
      "bytes": 563086,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-1bcc2845ea5c34584d57.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/1_generation_bounds.png",
      "alt": "Uniform convergence theory",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/2_convergence.png",
      "bytes": 207469,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-821ec496fca8a629680d.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/2_convergence.png",
      "alt": "Generalization with respect to learning rate and batch size",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/3_bayesian.png",
      "bytes": 319434,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-955379ec55f10c2551f6.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/3_bayesian.png",
      "alt": "Bayesian Deep Learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/4_graph.png",
      "bytes": 117670,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-a55e1836a160751a85e9.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/4_graph.png",
      "alt": "Graph neural networks",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/5_graph.jpg",
      "bytes": 90551,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-71c27afa96bbccb9c21f.jpg",
      "src": "https://huyenchip.com/assets/pics/neurips2019/5_graph.jpg",
      "alt": "Graph matching",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/6_convex.png",
      "bytes": 136917,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-f15d3d6beb92a3a3c0e7.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/6_convex.png",
      "alt": "Hamiltonian descent convex",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/7_neuroscience_acceptance.png",
      "bytes": 245379,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-aabb051b24a143b2b8ed.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/7_neuroscience_acceptance.png",
      "alt": "Neuroscience is the category with the highest acceptance rate",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/8_conscious.png",
      "bytes": 1035100,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-c1ce1db7463e459cd151.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/8_conscious.png",
      "alt": "Whaddup it's consciousness ya boi",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/9_keywords.png",
      "bytes": 770256,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-fff0c6f2ad4c529d9efd.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/9_keywords.png",
      "alt": "NeurIPS keyword cloud",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/10_keywords_percent.png",
      "bytes": 262255,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-738a6034473dbb90a4fa.png",
      "src": "https://huyenchip.com/assets/pics/neurips2019/10_keywords_percent.png",
      "alt": "NeurIPS keyword percentage change",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/neurips2019/11_metameme.jpg",
      "bytes": 76441,
      "reused": false,
      "local_path": "attachments/660be759316e/images/body-31f9cace8fcbb769b8c1.jpg",
      "src": "https://huyenchip.com/assets/pics/neurips2019/11_metameme.jpg",
      "alt": "Meta-meme",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://twitter.com/chipro/status/1207506277443629056",
      "label": "Twitter thread",
      "type": "a"
    },
    {
      "href": "https://twitter.com/andrey_kurenkov",
      "label": "Andrey Kurenkov",
      "type": "a"
    },
    {
      "href": "https://www.wired.com/story/facebooks-ai-says-field-hit-wall/",
      "label": "right now, an experiment might be in seven figures, but it’s not going to go to nine or ten figures .. nobody can afford",
      "type": "a"
    },
    {
      "href": "https://spectrum.ieee.org/tech-talk/robotics/artificial-intelligence/yoshua-bengio-revered-architect-of-ai-has-some-ideas-about-what-to-build-next",
      "label": "Look, deep learning doesn’t work.",
      "type": "a"
    },
    {
      "href": "https://twitter.com/GaryMarcus/status/1204783961848737793",
      "label": "disputed this characterization",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/yann.lecun/posts/10156437493897143",
      "label": "I don’t get why, all of a sudden, we read all these stories and tweets that claim ‘progress in AI is slowing’ and ‘Deep ",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.04742",
      "label": "Uniform convergence may be unable to explain generalization in deep learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1806.07572",
      "label": "neural tangent kernel (NTK)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.04918",
      "label": "Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9449-on-the-inductive-bias-of-neural-tangent-kernels",
      "label": "On the Inductive Bias of Neural Tangent Kernels",
      "type": "a"
    },
    {
      "href": "https://nips.cc/Conferences/2019/Schedule?showEvent=14579",
      "label": "Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9103-what-can-resnet-learn-efficiently-going-beyond-kernels",
      "label": "What Can ResNet Learn Efficiently, Going Beyond Kernels?",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9111-limitations-of-lazy-training-of-two-layers-neural-network",
      "label": "Limitations of Lazy Training of Two-layers Neural Network",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1810.07770",
      "label": "Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8704-efficient-deep-approximation-of-gmms",
      "label": "Efficient Deep Learning of Gaussian Mixture Models",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8398-control-batch-size-and-learning-rate-to-generalize-well-theoretical-and-empirical-evidence",
      "label": "Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.04595",
      "label": "Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks",
      "type": "a"
    },
    {
      "href": "https://slideslive.com/38921489/deep-learning-with-bayesian-principles",
      "label": "Deep Learning with Bayesian Principles",
      "type": "a"
    },
    {
      "href": "https://nips.cc/Conferences/2019/Schedule?showParentSession=15542",
      "label": "Probabilistic Methods – Variational Inference",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.03290",
      "label": "Importance Weighted Hierarchical Variational Inference",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.02476",
      "label": "A Simple Baseline for Bayesian Uncertainty in Deep Learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.02506",
      "label": "Practical Deep Learning with Bayesian Principles",
      "type": "a"
    },
    {
      "href": "https://twitter.com/hashtag/NeurIPS2019?src=hash&ref_src=twsrc%5Etfw",
      "label": "#NeurIPS2019",
      "type": "a"
    },
    {
      "href": "https://t.co/H8LM0BMvK4",
      "label": "pic.twitter.com/H8LM0BMvK4",
      "type": "a"
    },
    {
      "href": "https://twitter.com/liyuajia/status/1205538110773190656?ref_src=twsrc%5Etfw",
      "label": "December 13, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.01629",
      "label": "Exact Combinatorial Optimization with Graph Convolutional Neural Networks",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.13192",
      "label": "Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.02349",
      "label": "(Nearly) Efficient Algorithms for the Graph Matching Problem on Correlated Random Graphs",
      "type": "a"
    },
    {
      "href": "https://tkipf.github.io/graph-convolutional-networks/",
      "label": "Graph Convolutional Networks blog post",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/a-gentle-introduction-to-graph-neural-network-basics-deepwalk-and-graphsage-db5d540d50b3",
      "label": "A Gentle Introduction to Graph Neural Networks (Basics, DeepWalk, and GraphSage)",
      "type": "a"
    },
    {
      "href": "https://web.stanford.edu/~boyd/cvxbook/",
      "label": "convex optimization",
      "type": "a"
    },
    {
      "href": "https://nips.cc/Conferences/2019/Schedule?showParentSession=15535",
      "label": "1",
      "type": "a"
    },
    {
      "href": "https://nips.cc/Conferences/2019/Schedule?showParentSession=15608",
      "label": "2",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9152-differentiable-convex-optimization-layers",
      "label": "Differentiable Convex Optimization Layers",
      "type": "a"
    },
    {
      "href": "http://www.larsblackmore.com/nae_bridge_2016.pdf",
      "label": "SpaceX uses convex optimization to land its rockets",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8811-acceleration-via-symplectic-discretization-of-high-resolution-differential-equations",
      "label": "Acceleration via Symplectic Discretization of High-Resolution Differential Equations",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9590-hamiltonian-descent-for-composite-objectives",
      "label": "Hamiltonian descent for composite objectives",
      "type": "a"
    },
    {
      "href": "https://medium.com/@NeurIPSConf/what-we-learned-from-neurips-2019-data-111ab996462c",
      "label": "the analysis of NeurIPS 2019 Program Chair Hugo Larochelle",
      "type": "a"
    },
    {
      "href": "https://slideslive.com/38921750/from-system-1-deep-learning-to-system-2-deep-learning",
      "label": "From System 1 Deep Learning to System 2 Deep Learning",
      "type": "a"
    },
    {
      "href": "https://slideslive.com/38921748/social-intelligence",
      "label": "Social Intelligence",
      "type": "a"
    },
    {
      "href": "https://twitter.com/chipro/status/1195425389587226624?ref_src=twsrc%5Etfw",
      "label": "November 15, 2019",
      "type": "a"
    },
    {
      "href": "https://github.com/Coppersmith/vennclouds",
      "label": "vennclouds",
      "type": "a"
    },
    {
      "href": "https://nips.cc/Conferences/2019/Schedule?showParentSession=15500",
      "label": "Algorithms – Missing Data poster session",
      "type": "a"
    },
    {
      "href": "https://nips.cc/Conferences/2019/Schedule?showEvent=14398",
      "label": "Missing Not at Random in Matrix Completion: The Effectiveness of Estimating Missingness Probabilities Under a Low Nuclea",
      "type": "a"
    },
    {
      "href": "https://twitter.com/jayelmnop/status/1206637800537362432",
      "label": "Jesse Mu",
      "type": "a"
    },
    {
      "href": "https://twitter.com/hashtag/neurips2019?src=hash&ref_src=twsrc%5Etfw",
      "label": "#neurips2019",
      "type": "a"
    },
    {
      "href": "https://t.co/bXPoB135PA",
      "label": "pic.twitter.com/bXPoB135PA",
      "type": "a"
    },
    {
      "href": "https://twitter.com/chipro/status/1170174478304759809?ref_src=twsrc%5Etfw",
      "label": "September 7, 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/3882-dual-averaging-method-for-regularized-stochastic-learning-and-online-optimization",
      "label": "Dual Averaging Method for Regularized Stochastic Learning and Online Optimization",
      "type": "a"
    },
    {
      "href": "https://openreview.net/group?id=NeurIPS.cc/2019/Reproducibility_Challenge",
      "label": "Reproducibility Challenge on OpenReview",
      "type": "a"
    },
    {
      "href": "https://slideslive.com/38922087/neurips-workshop-on-machine-learning-for-creativity-and-design-30-2",
      "label": "NeurIPS Workshop on Machine Learning for Creativity and Design",
      "type": "a"
    },
    {
      "href": "https://open.spotify.com/artist/38SKxCyfrmNWqWunb9wGHP",
      "label": "their music on Spotify",
      "type": "a"
    },
    {
      "href": "https://t.co/Y8Y7eYrGkI",
      "label": "pic.twitter.com/Y8Y7eYrGkI",
      "type": "a"
    },
    {
      "href": "https://twitter.com/DanielNkemelu/status/1206057759017299969?ref_src=twsrc%5Etfw",
      "label": "December 15, 2019",
      "type": "a"
    },
    {
      "href": "https://slideslive.com/38921897/retrospectives-a-venue-for-selfreflection-in-ml-research-4",
      "label": "Retrospectives: A Venue for Self-Reflection in ML Research",
      "type": "a"
    },
    {
      "href": "https://t.co/1VqAHZoqtj",
      "label": "pic.twitter.com/1VqAHZoqtj",
      "type": "a"
    },
    {
      "href": "https://twitter.com/andrey_kurenkov/status/1205266951792029696?ref_src=twsrc%5Etfw",
      "label": "December 12, 2019",
      "type": "a"
    },
    {
      "href": "https://twitter.com/timnitGebru/status/1206321822850936832?ref_src=twsrc%5Etfw",
      "label": "December 15, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.10075",
      "label": "Distribution-Independent PAC Learning of Halfspaces with Massart Noise",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/feed/update/urn:li:activity:6611360431829323776/",
      "label": "NewInML",
      "type": "a"
    },
    {
      "href": "https://twitter.com/lawrennd",
      "label": "Neil Lawrence",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2019/05/12/top-8-trends-from-iclr-2019.html",
      "label": "ICLR",
      "type": "a"
    },
    {
      "href": "https://www.akshayagrawal.com/",
      "label": "Akshay Agrawal",
      "type": "a"
    },
    {
      "href": "https://sites.google.com/view/colinwei",
      "label": "Colin Wei",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:52Z",
  "source_url": "https://huyenchip.com/2019/12/18/key-trends-neurips-2019.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.04742",
      "bytes": 1050189,
      "reused": false,
      "local_path": "papers/repair/c90c98d42c2891419c6c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1806.07572",
      "bytes": 519056,
      "reused": false,
      "local_path": "papers/repair/6f1716341e1000cae100.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.04918",
      "bytes": 1617404,
      "reused": false,
      "local_path": "papers/repair/ef4a145c192da6276d8b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1810.07770",
      "bytes": 409622,
      "reused": false,
      "local_path": "papers/repair/60adee255faaf2433007.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.04595",
      "bytes": 1157278,
      "reused": false,
      "local_path": "papers/repair/6c7927522c12056fe567.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.03290",
      "bytes": 625919,
      "reused": false,
      "local_path": "papers/repair/cf2f17faa07d28552601.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.02476",
      "bytes": 1428180,
      "reused": false,
      "local_path": "papers/repair/5efdc0bc42fa3498579b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.02506",
      "bytes": 2279744,
      "reused": false,
      "local_path": "papers/repair/b0d5813f5197dfb262d2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.01629",
      "bytes": 820826,
      "reused": false,
      "local_path": "papers/repair/97501bd9acb0a9618e20.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.13192",
      "bytes": 1160921,
      "reused": false,
      "local_path": "papers/repair/8b53414722650996b238.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.02349",
      "bytes": 524522,
      "reused": false,
      "local_path": "papers/repair/3c7e0a6ddf53af2aa4b1.pdf"
    },
    {
      "status": "download_error",
      "url": "http://www.larsblackmore.com/nae_bridge_2016.pdf",
      "reason": "ConnectionError: HTTPConnectionPool(host='www.larsblackmore.com', port=80): Max retries exceeded with url: /nae_bridge_2016.pdf (Caused by NewConnectionError(\"HTTPConnection(host='www.larsblackmore",
      "retried_at": "2026-10-20T22:23:37Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.10075",
      "bytes": 508804,
      "reused": false,
      "local_path": "papers/repair/658a04d18afd6719653f.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 21339,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "download_error",
  "raw_html": "attachments/660be759316e/pages/source.html",
  "local_html": "attachments/660be759316e/pages/article.html",
  "body_base_path": "attachments/660be759316e/pages/",
  "body_markdown_path": "attachments/660be759316e/pages/article.md"
}