An environment for studying counterfactuals

Proposes a class of environments in which the agent is tasked with predicting the utility of every action, in addition to maximizing expected utility. It is evaluated on the utility achieved as well as correctly predicting the utility it gets. Epsilon-exploration is required, so for every action there is always some chance that the agent will be tested on predicting the utility of that action. The agent is also provided a prior P about the world, including what the agent will do (which exists due to a fixed-point theorem).

I'm confused (I'm not an expert in this field), but I'm not sure what I'm confused about. Is there a dynamics model? Given that the agent gets access to a prior, can it find Pr(U | o, a) and choose the a with maximum expected utility? Why are we including reflection? There are often many fixed points, which one do we pick?


Choosing to Choose?

If it is possible for your utility function to change, then should you optimize for your current utility function, or your expected future utility function? The post gives an argument for both sides, and ultimately says that you should optimize for your current utility function, but notes some problems with the proposed argument for it.

I think that it is correct to optimize for your current utility function, and I didn't find the argument for the other side convincing (and wrote a comment on the post with more details).

Self-Modification of Policy and Utility Function in Rational Agents


Conditioning, Counterfactuals, Exploration, and Gears

One way that you can think about counterfactuals is to _condition_ on some low probability state, and then look at the probability distribution that implies. This seems like the most general version of counterfactuals, but it doesn't match what we intuitively mean by counterfactuals, which is more like "suppose that by fiat this constraint were met, but don't consider what would have caused it, now predict the consequences". This sort of imputing only works because there are very simple rules governing our universe, so that there are strong correlations between different experiences and so it actually is possible to generalize to very new situations. It seems very important to use this idea in order to advance beyond epsilon-exploration for new situations.

I agree that this is an important idea, and it has arisen elsewhere -- in ML, this is part of the thinking on the problem of generalization. There are no-free-lunch theorems that say you cannot do well in arbitrary environments, where the constructions typically violate the "strong correlation between different experiences" heuristic. In philosophy, this is the problem of induction.


Conditions under which misaligned subagents can (not) arise in classifiers

Agents or subagents with "goals" are only likely to arise when you are considering tasks where it is important to keep state/memory, because past inputs are informative about future inputs. So, unaligned subagents are unlikely to arise for eg. classification tasks where it is not necessary to model how things change over time.

I do think that classifiers with a bounded task that run for a bounded amount of time are unlikely to develop unaligned subagents with memory. However, I still feel very unclear on the term "unaligned subagent", so I'm not very confident in this assessment.


Dependent Type Theory and Zero-Shot Reasoning

Humans can do zero-shot reasoning (in the sense of writing down proofs) by "running a type checker in their head" (analogous to theorem provers like Lean). The post gives an example of this, using Lean syntax. However, humans seem to have very different ways of thinking -- for example, you could either generate ideas for solutions to a problem, see if they work, and iterate, or you could start proving some facts about the problem, and keep on proving things until you have proved a solution. These feel like many-shot reasoning and zero-shot reasoning respectively, even though they are both attempting a zero-shot task. This is one way to understand the difference between Iterated distillation and amplification, and Agent foundations -- the former is many-shot and the latter is zero-shot, even though both are attempting a zero-shot task.

I found the part about how people prove things to be the most interesting part of the post, because my own method seems different from both. I usually alternate between searching for solutions, counterexamples to solutions, and proving that solutions must satisfy some property.


Mathematical Mindset

Introduces a new term, "mathematical mindset", which is about finding good _definitions_ or _models_ that make it easier for you to reason about them. For example, you expect proofs with a newer definition to be shorter or more general. Key quote: "Having a “mathematical mindset” means being comfortable with words being redefined. This is because it means being comfortable with models being upgraded -- in particular, with models being related and compared to each other: the activity of theorization."

I'm all for having better definitions that make things clearer and easier to reason about. I don't know if "ease of proofs" is the right thing to aim for -- "ease of reasoning" is closer to what I care about, even if it's informal reasoning.


Monk Treehouse: some problems defining simulation

Some approaches to AI alignment require you to identify copies of programs in the environment, and it is not clear how to do this in full generality. Proposals so far have attempted to define two programs to be equivalent if they do the same thing now and would also do the same thing in counterfactual worlds. This post argues that such definitions don't work using an analogy where there are monks computing by moving heavy stones in a treehouse, that could unbalance it. In this setting, there are lots of checks and balances to make sure that the program does one and only one thing; any counterfactual you specify would lead to weird results (like the treehouse falling over from unbalanced stones, or monks noticing that something is off and correcting the result, etc.) and so it wouldn't be considered equivalent to the same program on a silicon-based computer.

I don't know where a proposed definition is supposed to be used so it's hard for me to comment on how relevant this objection is.


No, I won't go there, it feels like you're trying to Pascal-mug me

One explanation for why [Pascal's mugging](https://en.wikipedia.org/wiki/Pascal%27s_mugging) feels intuitively wrong is that if we were to pay the mugger, we would open ourselves up to exploitation by any other agent. [Logical induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/) puts uncertainties on statements in such a way that it isn't exploitable by polynomial-time traders. Perhaps there is a connection here that can help us create AIs that don't get mugged.

Non-exploitability is my preferred resolution to Pascal's mugging. However, it seems like such an obvious solution, yet there's very little discussion of it, which makes me think that there's some fatal flaw that I'm not seeing.


A universal score for optimizers

We can measure the optimization power of an agent as the log probability that a random agent matches the outcome that the agent achieves.

Seems like a reasonable starting point to measure optimization power. As Alex Mennen [notes](https://www.lesswrong.com/posts/vWfFDahpnY3tnCLrp/a-universal-score-for-optimizers#e3Hsaqkc2xuWsHY2t), it's dependent on the specific action set chosen, and doesn't take into account the strength of preferences, only their ranking.


Alignment problems for economists

What AI alignment problems could we outsource to economists? There are some who would be interested in working on alignment, but don't because it would be too much of a career risk.

Unfortunately, the "desirable properties" for these problems all seem to conspire to make any particular problem fairly low impact.


On the Role of Counterfactuals in Learning

This post hypothesizes that since humans are computationally bounded, we infer causal models using approximate inference (eg. Gibbs sampling), as opposed to a full Bayesian update. However, approximate inference algorithms depend a lot on choosing a good initialization. Counterfactuals fill this role.

I think I've summarizes this post badly, because I didn't really understand it. In particular, I didn't understand the jump from "humans do approximate inference over the space of models" to "counterfactuals form the initialization".


The Intentional Agency Experiment

In order to determine whether an agent has some intention, we can check to see whether the agent would take actions that achieve the intent under a wide range of circumstances (either counterfactuals, or actual changes to the environment). For example, to show that an ant has agency and intends to find sugar, we could block its route to the sugar and notice that it finds a path around the obstacle.

The motivation was to use this to deduce the intentions of a superintelligent AI system, but it seems that such an AI system could figure out it is being tested and respond in the "expected" way.


Two agents can have the same source code and optimise different utility functions

Even if you have two agents with identical source code, their goals are in relation to themselves, so each agent will, for example, try to gain resources for itself. Since the two agents are now competing, they clearly have different utility functions.

I'm somewhat confused -- I'm not sure what the point is here.


AlphaFold: Using AI for scientific discovery

Andrew Senior
John Jumper and Demis Hassabis

This post briefly describes AlphaFold, a system that does well at the protein folding problem. They train neural networks that can be used to evaluate how good a particular proposed protein structure is. This can be used to guide an evolutionary search that repeatedly replaces pieces of a protein structure with new protein fragments from a generative model. Alternatively, it can be used to construct a loss function for entire proteins, which allows us to use gradient descent to optimize the protein structure.

The approach here is to learn heuristics that can guide a top-level search algorithm, which is the sort of thing that I think deep learning is particularly well poised to improve right now. Note that gradient descent is a top-level search algorithm here, because a separate loss function is constructed _for every protein_, rather than having a single loss function that is used to train a network that works on all proteins. However, unlike other applications such as SMT solvers, the top-level search algorithm does not have some sort of "correctness" guarantee.


A major milestone for the treatment of eye disease

DeepMind's partnership with Moorfields Eye Hospital has resulted in an AI system that can recognize features of eye disease and recommend treatment. Interestingly, in order to get interpretability, they train two networks instead of one, where one predicts the features of eye disease for all of the tissue (eg. haemorrhages, lesions and irregular fluids), and the other then makes a recommendation for treatment. This required them to label a subset of the dataset with feature markers in order to train the first model.

As interpretability goes, using a modular model with human-interpretable intermediate representations seems quite good -- it decouples the problem of understanding the model's output into two smaller problems. The big downside is that it requires a lot more labeling (877 segmented images in this case), and that the human-interpretable representation may not be the best one for the job. For example, if there are other visual cues besides the specific features DeepMind used that help with recommending treatment, this model will not be able to take advantage of them, while an end-to-end trained system could.


The Machine Learning Behind Android Smart Linkify

Android now has Smart Linkify technology, which allows it to automatically find pieces of text that should link to another app (for example, addresses should link to Maps, dates and times to Calendar, etc). There are a lot of interesting details on what had to be done to get this to actually work in the real world. The system has two separate nets -- one which generates candidate entities, and another which says what kind of entity each one is. In between these two nets, we have a regular program that takes the set of proposed entities, and prunes it so that no two entities overlap, and then sends it off to the entity classification net. There are a few tricks to get the memory requirements down, and many dataset augmentation tricks to get the nets to learn particular rules that it would not otherwise have learned.

I take this as an example of what advanced AI systems will look like -- a system of different modules, each with its own job, passing around information appropriately in order to perform some broad task. Some of the modules could be neural nets (which can learn hard-to-program functions), while others could be classic programs (which generalize much better and are more efficient). OpenAI Five also has elements of this -- the drafting system is a classic program operating on the win probabilities from the neural net. It's also interesting how many tricks are required to get Smart Linkify to work -- I don't know whether to think that this means generally intelligent AI is further away, or that the generally intelligent AI that we build will rely on these sorts of tricks.


Tackling Climate Change with Machine Learning

David Rolnick
Priya L. Donti
Lynn H. Kaack
Kelly Kochanski
Alexandre Lacoste
Kris Sankaran
Andrew Slavin Ross
Nikola Milojevic-Dupont
Natasha Jaques
Anna Waldman-Brown
Alexandra Luccioni
Tegan Maharaj
Evan D. Sherwin
S. Karthik Mukkavilli
Konrad P. Kording
Carla Gomes
Andrew Y. Ng
Demis Hassabis
John C. Platt
Felix Creutzig
Jennifer Chayes
Yoshua Bengio


Artificial Intelligence — The Revolution Hasn’t Happened Yet

There is a lot of hype at the moment around AI, particularly around creating AI systems that have human intelligence, since the thrill (and fear) of creating human intelligence in silicon causes overexuberance and excessive media attention. However, we _actually_ want to create AI systems that can help us improve our lives, often by doing things that humans are not capable of. In order to accomplish this, it is likely better to work directly on these problems, since human-like intelligence is neither necessary nor sufficient to build such systems. However, as with all new technologies, there are associated challenges and opportunities with these AI systems, and we are currently at risk of not seeing these because we are too focused on human intelligence in particular.

There certainly is a lot of hype both around putting human intelligence in silicon, as well as the risks that surround such an endeavor. Even though I focus on such risks, I agree with Jordan that these are overhyped in the media and we would benefit from having more faithful coverage of them. I do disagree on some specific points. For example, he says that human-imitative AI is not sufficient to build some AI systems such as self-driving cars, but why couldn't an AI with human intelligence just do whatever humans would do to build self-driving cars? (I can think of answers, such as "we don't know how to give the AI system access to all the data that humans have access to", but I wish he had engaged more with this argument.) I do agree with the overall conclusion that in the near future humans will make progress on building such systems, and not by trying to give the systems "human intelligence". I also suspect that we disagree either on how close we are to human-imitative AI, or at what point it is worth it to start thinking about the associated risks, but it's hard to tell more from the article.


To Build Truly Intelligent Machines, Teach Them Cause and Effect

An interview with Judea Pearl about causality, deep learning, and where the field is going.

This is fairly superficial, if you've read any of the other things that Pearl himself has written about deep learning, you'll know all of this already.


Ben Garfinkel on scrutinising classic AI risk arguments

In this podcast, Ben Garfinkel goes through several reasons why he is skeptical of classic AI risk arguments (some previously discussed <@here@>(@How Sure are we about this AI Stuff?@)). The podcast has considerably more detail and nuance than this summary.

Ben thinks that historically, it has been hard to affect transformative technologies in a way that was foreseeably good for the long-term-- it's hard e.g. to see what you could have done around the development of agriculture or industrialization that would have an impact on the world today. He thinks some potential avenues for long-term influence could be through addressing increased political instability or the possibility of lock-in, though he thinks that it’s unclear what we could do today to influence the outcome of a lock-in, especially if it’s far away.

In terms of alignment, Ben focuses on the standard set of arguments outlined in Nick Bostrom’s Superintelligence, because they are broadly influential and relatively fleshed out. Ben has several objections to these arguments:
- He thinks it isn't likely that there will be a sudden jump to extremely powerful and dangerous AI systems, and he thinks we have a much better chance of correcting problems as they come up if capabilities grow gradually.
- He thinks that making AI systems capable and making AI systems have the right goals are likely to go together.
- He thinks that just because there are many ways to create a system that behaves destructively doesn't mean that the engineering process creating that system is likely to be attracted to those destructive systems; it seems like we are unlikely to accidentally create systems that are destructive enough to end humanity.

Ben also spends a little time discussing <@mesa-optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), a much newer argument for AI risk. He largely thinks that the case for mesa-optimization hasn’t yet been fleshed out sufficiently. He also thinks it’s plausible that learning incorrect goals may be a result of having systems that are insufficiently sophisticated to represent goals appropriately. With sufficient training, we may in fact converge to the system we want.

Given the current state of argumentation, Ben thinks that it's worth EA time to flesh out newer arguments around AI risk, but also thinks that EAs who don't have a comparative advantage in AI-related topics shouldn't necessarily switch into AI. Ben thinks it's a moral outrage that we have spent less money on AI safety and governance than the 2017 movie 'The Boss Baby', starring Alec Baldwin.

This podcast covers a really impressive breadth of the existing argumentation. A lot of the reasoning is similar to <@that I’ve heard from other researchers@>(@Takeaways from safety by default interviews@). I’m really glad that Ben and others are spending time critiquing these arguments; in addition to showing us where we’re wrong, it helps us steer towards more plausible risky scenarios.

I largely agree with Ben’s criticisms of the Bostrom AI model; I think mesa-optimization is the best current case for AI risk and am excited to see more work on it. The parts of the podcast where I most disagreed with Ben were:
- I think even in the absence of solid argumentation, I feel good about a prior where AI has a non-trivial chance of being existentially threatening, partially because I think it’s reasonable to put AI in the reference class of ‘new intelligent species’ in addition to ‘new technology’.
- I’m not sure that institutions will address failures sufficiently, <@even if progress is gradual and there are warnings@>(@Possible takeaways from the coronavirus pandemic for slow AI takeoff@).

**Rohin's opinion:** I recommend listening to the full podcast, as it contains a lot of detail that wouldn't fit in this summary. Overall I agree pretty strongly with Ben. I do think that some of the counterarguments are coming from a different frame than the classic arguments. For example, a lot of the counterarguments involve an attempt to generalize from current ML practice to make claims about future AI systems. However, I usually imagine that the classic arguments are basically ignoring current ML, and instead claiming that if an AI system is superintelligent, then it must be goal-directed and have convergent instrumental subgoals. If current ML systems don't lead to goal-directed behavior, I expect that proponents of the classic arguments would say that they also won't lead to superintelligent AI systems. I'm not particularly sold on this intuition either, but I can see its appeal.


We Shouldn’t be Scared by ‘Superintelligent A.I.’

This review of <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) argues that people worried about superintelligent AI are making a mistake by assuming that an AI system "could surpass the generality and flexibility of human intelligence while seamlessly retaining the speed, precision and programmability of a computer". It seems likely that human intelligence is strongly integrated, such that our emotions, desires, sense of autonomy, etc. are all _necessary_ for intelligence, and so general intelligence can't be separated from so-called "irrational" biases. Since we know so little about what intelligence actually looks like, we don't yet have enough information to create AI policy for the real world.

The only part of this review I disagree with is the title -- every sentence in the text seems quite reasonable. I in fact do not want policy that advocates for particular solutions now, precisely because it's not yet clear what the problem actually is. (More "field-building" type policy, such as increased investment in research, seems fine.)

The review never actually argues for its title -- you need some additional argument, such as "and therefore, we will never achieve superintelligence", or "and since superintelligent AI will be like humans, they will be aligned by default". For the first one, while I could believe that we'll never build ruthlessly goal-pursuing agents for the reasons outlined in the article, I'd be shocked if we couldn't build agents that were more intelligent than us. For the second one, I agree with the outside view argument presented in _Human Compatible_: while humans might be aligned with each other (debatable, but for now let's accept it), humans are certainly not aligned with gorillas. We don't have a strong reason to say that our situation with superintelligent AI will be different from the gorillas' situation with us. (Obviously, we get to design AI systems, while gorillas didn't design us, but this is only useful if we actually have an argument why our design for AI systems will avoid the gorilla problem, and so far we don't have such an argument.)


The seven deadly sins of AI predictions

This is an older article I was sent recently, that argues against AI risk and the idea that we will have AGI soon. It generally argues that AGI proponents are mistaken about current capabilities of AI and how long it will take to make progress in AGI research.

This article is aimed at refuting the superintelligent perfectly-rational agent model of AGI, and so feels to me like it's attacking a strawman of the argument for AI risk, but it does seem to me that many people do have beliefs similar to the ones he's arguing against. I partially agree with some of his criticisms and disagree with others, but overall I think most of the arguments are reasonable ones and worth knowing about.


ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters

This paper introduces ZeRO and DeepSpeed, system optimizations that enable training significantly larger models than we have before.

_Data parallelism_ is a way of splitting data across multiple machines to increase training throughput. Instead of training a model sequentially on one dataset, the dataset is split and models are trained in parallel. Resulting gradients on every machine are combined centrally and then used for back propagation. Previously, data parallelism approaches were memory-constrained because the entire model still had to fit on each GPU, which becomes infeasible for billion to trillion-parameter models.

Instead of replicating each model on each machine, ZeRO partitions each model across machines and shares states, resulting in a per-machine memory reduction that is linear with the number of machines. (E.g., splitting across 64 GPUs yields a 64x memory reduction).

In addition to ZeRO, Microsoft is releasing DeepSpeed, a library which offers ZeRO as well as several other performance optimizations in an easy-to-use library for PyTorch, a popular open-source machine learning framework. They purport that their library allows for models that are 10x bigger, up to 5x faster to train, and up to 5x cheaper. They use DeepSpeed to train a [17-billion-parameter language model](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft) which exceeds state-of-the-art results in natural language processing.

I think this is a significant step in machine learning performance which may not be used heavily until average model sizes in general increase. The technique itself is pretty straightforward, which makes me think that as model sizes increase there may be a lot of similar "low-hanging fruit" that yield large performance gains.


Deep Double Descent

Preetum Nakkiran
Gal Kaplun
Yamini Bansal
Tristan Yang
Boaz Barak
Ilya Sutskever

This blog post provides empirical evidence for the existence of the _double descent_ phenomenon, proposed in an earlier paper summarized below. Define the _effective model complexity_ (EMC) of a training procedure and a dataset to be the maximum size of training set such that the training procedure achieves a _train_ error of at most ε (they use ε = 0.1). Let's suppose you start with a small, underparameterized model with low EMC. Then initially, as you increase the EMC, the model will achieve a better fit to the data, leading to lower test error. However, once the EMC is approximately equal to the size of the actual training set, then the model can "just barely" fit the training set, and the test error can increase or decrease. Finally, as you increase the EMC even further, so that the training procedure can easily fit the training set, the test error will once again _decrease_, causing a second descent in test error. This unifies the perspectives of statistics, where larger models are predicted to overfit, leading to increasing test error with higher EMC, and modern machine learning, where the common empirical wisdom is to make models as big as possible and test error will continue decreasing.

They show that this pattern arises in a variety of simple settings. As you increase the width of a ResNet up to 64, you can observe double descent in the final test error of the trained model. In addition, if you fix a large overparameterized model and change the number of epochs for which it is trained, you see another double descent curve, which means that simply training longer can actually _correct overfitting_. Finally, if you fix a training procedure and change the size of the dataset, you can see a double descent curve as the size of the dataset decreases. This actually implies that there are points in which _more data is worse_, because the training procedure is in the critical interpolation region where test error can increase. Note that most of these results only occur when there is _label noise_ present, that is, some proportion of the training set (usually 10-20%) is given random incorrect labels. Some results still occur without label noise, but the resulting double descent peak is quite small. The authors hypothesize that label noise leads to the effect because double descent occurs when the model is misspecified, though it is not clear to me what it means for a model to be misspecified in this context.

While I previously didn't think that double descent was a real phenomenon (see summaries later in this email for details), these experiments convinced me that I was wrong and in fact there is something real going on. Note that the settings studied in this work are still not fully representative of typical use of neural nets today; the label noise is the most obvious difference, but also e.g. ResNets are usually trained with higher widths than studied in this paper. So the phenomenon might not generalize to neural nets as used in practice, but nonetheless, there's _some_ real phenomenon here, which flies in the face of all of my intuitions.

The authors don't really suggest an explanation; the closest they come is speculating that at the interpolation threshold there's only ~one model that can fit the data, which may be overfit, but then as you increase further the training procedure can "choose" from the various models that all fit the data, and that "choice" leads to better generalization. But this doesn't make sense to me, because whatever is being used to "choose" the better model applies throughout training, and so even at the interpolation threshold the model should have been selected throughout training to be the type of model that generalized well. (For example, if you think that regularization is providing a simplicity bias that leads to better generalization, the regularization should also help models at the interpolation threshold, since you always regularize throughout training.)

Perhaps one explanation could be that in order for the regularization to work, there needs to be a "direction" in the space of model parameters that doesn't lead to increased training error, so that the model can move along that direction towards a simpler model. Each training data point defines a particular direction in which training error will increase. So, when the number of training points is equal to the number of parameters, the training points just barely cover all of the directions, and then as you increase the number of parameters further, that starts creating new directions that are not constrained by the training points, allowing the regularization to work much better. (In fact, the [original paper](https://arxiv.org/abs/1812.11118), summarized below, _defined_ the interpolation threshold as the point where number of parameters equals the size of the training dataset.) However, while this could explain model-wise double descent and training-set-size double descent, it's not a great explanation for epoch-wise double descent.

Paper: Deep Double Descent: Where Bigger Models and More Data Hurt


Are Deep Neural Networks Dramatically Overfitted?

The concepts of underfitting and overfitting, and their relation to the bias-variance tradeoff, are fundamental to standard machine learning theory. Roughly, for a fixed amount of data, there is an optimal model complexity for learning from that data: any less complex and the model won't be able to fit the data, and any more complex and it will overfit to noise in the data. This means that as you increase model complexity, training error will go down to zero, but validation error will go down and then start turning back up once the model is overfitting.

We know that neural networks are much more expressive than the theory would predict is optimal, both from theorems showing that neural networks can learn any function (including one that provides a rather tight bound on number of parameters), as well as a [paper](https://arxiv.org/abs/1611.03530) showing that neural nets can learn random noise. Yet they work well in practice, achieving good within-distribution generalization.

The post starts with a brief summary of topics that readers of this newsletter are probably familiar with: Occam's razor, the Minimum Description Length principle, Kolmogorov Complexity, and Solomonoff Induction. If you don't know these, I strongly recommend learning them if you care about understanding within-distribution generalization. The post then looks at a few recent informative papers, and tries to reproduce them.

The [first one](https://arxiv.org/abs/1812.11118) is the most surprising: they find that as you increase the model complexity, your validation error goes down and then back up, as expected, but then at some point it enters a new regime and goes down again. However, the author notes that you have to set up the experiments just right to get the smooth curves the paper got, and her own attempts at reproducing the result are not nearly as dramatic.

Another [paper](https://arxiv.org/abs/1804.08838) measures the difficulty of a task based on its "intrinsic dimension", which Cody has summarized separately in this newsletter.

The [last paper](https://arxiv.org/abs/1902.01996) looks at what happens if you (a) reset some layer's parameters to the initial parameters and (b) randomize some layer's parameters. They find that randomizing always destroys performance, but resetting to initial parameters doesn't make much of a difference for later layers, while being bad for earlier layers. This was easy to reproduce, and the findings reemerge very clearly.

I'm very interested in this problem, and this post does a great job of introducing it and summarizing some of the recent work. I especially appreciated the attempts at reproducing the results.

On the papers themselves, a regime where you already have ~zero training error but validation error goes _down_ as you increase model expressivity is exceedingly strange. Skimming the paper, it seems that the idea is that in the normal ML regime, you are only minimizing training error -- but once you can get the training error to zero, you can then optimize for the "simplest" model with zero training error, which by Occam's Razor-style arguments should be the best one and lead to better validation performance. This makes sense in the theoretical model that they use, but it's not clear to me how this applies to neural nets, where you aren't explicitly optimizing for simplicity after getting zero training error. (Techniques like regularization don't result in one-after-the-other optimization -- you're optimizing for both simplicity and low training error simultaneously, so you wouldn't expect this critical point at which you enter a new regime.) So I still don't understand these results. That said, given the difficulty with reproducing them, I'm not going to put too much weight on these results now.

I tried to predict the results of the last paper and correctly predicted that randomizing would always destroy performance, but predicted that resetting to initialization would be okay for _early_ layers instead of later layers. I had a couple of reasons for the wrong prediction. First, there had been a few papers that showed good results even with random features, suggesting the initial layers aren't too important, and so maybe don't get updated too much. Second, the gradient of the loss w.r.t later layers requires only a few backpropagation steps, and so probably provides a clear, consistent direction moving it far away from the initial configuration, while the gradient w.r.t earlier layers factors through the later layers which may have weird or wrong values and so might push in an unusual direction that might get cancelled out across multiple gradient updates. I skimmed the paper and it doesn't really speculate on why this happens, and my thoughts still seem reasonable to me, so this is another fact that I have yet to explain.


Better Language Models and Their Implications

Alec Radford
Jeffrey Wu
Dario Amodei
Daniela Amodei
Jack Clark
Miles Brundage
and Ilya Sutskever

OpenAI has trained a scaled up GPT model using unsupervised learning (specifically, predicting the next word given a very large context) on a very large dataset with presumably very large compute. The resulting language model can produce impressive language samples (with some cherry-picking) that to my eye are particularly good at handling long-range dependencies, which makes sense since it is based on the [Transformer](https://arxiv.org/abs/1706.03762) (see Transformer-XL entry in [AN #44](https://mailchi.mp/6bfac400a0c3/alignment-newsletter-44)). It sets new state of the art performance on 7 out of 8 language modeling tasks, including difficult datasets such as [LAMBADA](https://arxiv.org/abs/1606.06031), _without using the training data for those tasks_. It can also be used for more structured tasks by providing a particular context -- for example, to summarize a document, you can provide the document followed by "TL;DR:" in order to induce GPT-2 to "predict" a summary. (They use a different prediction algorithm in order to improve summarization results, but I suspect even with regular prediction you'd get something in the right ballpark.) On these more structured tasks, it doesn't get anywhere near the state of the art set by specialized systems -- but again, this is without any finetuning for the specific task that we are testing.

The [paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) argues that in order to get generally capable AI systems, we will need to train them on many different tasks, as in meta-learning. However, we might expect that we need hundreds of thousands of tasks in order to learn something general, just as we need hundreds of thousands of examples in order to develop good classifiers. Prediction of the next word in natural language is particularly good for this, because in order to predict well across a huge variety of text, you need to become good at many different tasks such as question answering, summarization, and even translation. The biggest challenge is in creating a dataset that has sufficient diversity -- they do this by scraping all outbound links from Reddit with at least 3 karma.

Unusually for research, but in accordance with [its charter](https://blog.openai.com/openai-charter/) ([AN #2](https://mailchi.mp/14782876a85d/alignment-newsletter-2)), OpenAI has decided not to release the model publicly, citing the possibility of malicious uses of the model. This has been controversial, with the debate raging for days on Twitter. I haven't paid enough attention to the debate to give a reasonable summary so you'll have to rely on other sources for that.

These are some pretty impressive results. I'm surprised that all of this came from a single order of magnitude more data and model size, I would have expected it to take more than that. I think this lends a lot of support to the hypothesis that unsupervised learning with sufficient amounts of compute and diverse data can lead to generally capable AI systems. (See this [SlateStarCodex post](https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/) for a more detailed version of this take.) This is also some evidence that we will have AI systems that can pass the Turing Test before we have general AI systems, that is, the Turing Test is not AI-complete.

Language Models are Unsupervised Multitask Learners


Transformer-XL: Unleashing the Potential of Attention Models

[Transformer](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) architectures have become all the rage recently, showing better performance on many tasks compared to CNNs and RNNs. This post introduces Transformer-XL, an improvement on the Transformer architecture for very long sequences.

The key idea with the original Transformer architecture is to use self-attention layers to analyze sequences instead of something recurrent like an RNN, which has problems with vanishing and exploding gradients. An attention layer takes as input a query q and key-value pairs (K, V). The query q is "compared" against every key k, and that is used to decide whether to return the corresponding value v. In their particular implementation, for each key k, you take the dot product of q and k to get a "weight", which is then used to return the weighted average of all of the values. So, you can think of the attention layer as taking in a query q, and returning the "average" value corresponding to keys that are "similar" to q (since dot product is a measure of how aligned two vectors are). Typically, in an attention layer, some subset of Q, K and V will be learned. With _self-attention_, Q, K and V are all sourced from _the same place_ -- the result of the previous layer (or the input if this is the first layer). Of course, it's not exactly the output from the previous layer -- if that were the case, there would be no parameters to learn. They instead learn three _linear projections_ (i.e. matrices) that map from the output of the previous layer to Q, K and V respectively, and then feed the generated Q, K and V into a self-attention layer to compute the final output. And actually, instead of having a single set of projections, they have multiple sets that each contain three learned linear projections, that are all then used for attention, and then combined together for the next layer by another learned matrix. This is called _multi-head attention_.

Of course, with attention, you are treating your data as a set of key-value pairs, which means that the order of the key value pairs does not matter. However, the order of words in a sentence is obviously important. To allow the model to make use of position information, they augment each word and add position information to it. You could do this just by literally appending a single number to each word embedding representing its absolute position, but then it would be hard for the neural net to ask about a word that was "3 words prior". To make this easier for the net to learn, they create a vector of numbers to represent the absolute position based on sinusoids such that "go back 3 words" can be computed by a linear function, which should be easy to learn, and add _(not concatenate!)_ it elementwise to the word embedding.

This model works great when you are working with a single sentence, where you can attend over the entire sentence at once, but doesn't work as well when you are working with eg. entire documents. So far, people have simply broken up documents into segments of a particular size N and trained Transformer models over these segments. Then, at test time, for each word, they use the past N - 1 words as context and run the model over all N words to get the output. This cannot model any dependencies that have range larger than N. The Transformer-XL model fixes this issue by taking the segments that vanilla Transformers use, and adding recurrence. Now, in addition to the normal output predictions we get from segments, we also get as output a new hidden state, that is then passed in to the next segment's Transformer layer. This allows for arbitrarily far long-range dependencies. However, this screws up our position information -- each word in each segment is augmented with _absolute_ position information, but this doesn't make sense across segments, since there will now be multiple words at (say) position 2 -- one for each segment. At this point, we actually want _relative_ positions instead of absolute ones. They show how to do this -- it's quite cool but I don't know how to explain it without going into the math and this has gotten long already. Suffice it to say that they look at the interaction between arbitrary words x_i and x_j, see the terms that arise in the computation when you add absolute position embeddings to each of them, and then change the terms so that they only depend on the difference j - i, which is a relative position.

This new model is state of the art on several tasks, though I don't know what the standard benchmarks are here so I don't know how impressed I should be.

It's quite interesting that even though the point of Transformer was to get away from recurrent structures, adding them back in leads to significant improvements. Of course, the recurrent structure is now at the higher level of segments, rather than at the word or character level. This reminds me a lot of hierarchy -- it seems like we're using the Transformer as a basic building block that works on the ~sentence level so that our RNN-like structure can deal with a higher level of abstraction (which of course also helps with vanishing/exploding gradients).

There's an interesting pattern where hierarchy and structure seem to be a good inductive bias, that let you get good performance with limited compute and data, but as those limits subside, you're better off doing something that has less bias. This would predict that as we get more data and compute, we would want larger Transformer models (i.e. longer segments) and less recurrence. It would be interesting to see if that actually holds.


Reptile: A Scalable Meta-Learning Algorithm

I somehow forgot to include this in past emails, so I'm including it now. Reptile is an algorithm for meta-learning, and in this paper is applied to few-shot classification, where given a few examples of different classes, you must learn a good classification algorithm for those classes. The authors show using a Taylor expansion that [MAML](https://arxiv.org/abs/1703.03400) and Reptile have very similar gradients to first order in alpha, the step size. Their evaluation shows that for the few-shot classification case, Reptile and MAML perform similarly (though they do not evaluate on reinforcement learning tasks, as in the MAML paper).

This seems like an important advance in meta-learning, as it is much more computationally efficient than MAML while still achieving similar levels of performance.

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks


Relational inductive biases, deep learning, and graph networks

Peter W. Battaglia
Jessica B. Hamrick
Victor Bapst
Alvaro Sanchez-Gonzalez
Vinicius Zambaldi
Mateusz Malinowski
Andrea Tacchetti
David Raposo
Adam Santoro
Ryan Faulkner
Caglar Gulcehre
Francis Song
Andrew Ballard
Justin Gilmer
George Dahl
Ashish Vaswani
Kelsey Allen
Charles Nash
Victoria Langston
Chris Dyer
Nicolas Heess
Daan Wierstra
Pushmeet Kohli
Matt Botvinick
Oriol Vinyals
Yujia Li
Razvan Pascanu

"Part position paper, part review, and part unification", this paper emphasises the importance of combinatorial generalisation, which is key to how humans understand the world. It argues for approaches which perform computation over discrete entities and the relations between them, such as graph networks. The authors claim that CNNs and RNNs are so successful due to relational inductive biases - for example, the bias towards local structure induced by convolutional layers. Graph networks are promising because they can express arbitrary relational biases: any nodes can be connected with any others depending on the structure of the problem. Further, since graph networks learn functions which are reused for all nodes and edges, each one can be applied to graphs of any shape and size: a form of combinatorial generalisation.

In this paper's framework, each 'graph block' does computations over an input graph and returns an output graph. The relevant part of the output might be the values of edges, or those of nodes, or 'global' properties of the overall graph. Graph blocks can be implemented by standard neural network architectures or more unusual ones such as message-passing neural networks or non-local neural networks. The authors note some major open questions: how to generate the graphs in the first place, and how to adaptively modify them during the course of computation.

This paper is an excellent holistic discussion of graph networks and reasons to think they are promising. I'm glad that it also mentioned the open problems, though, since I think they're pretty crucial to using graphs in deep learning, and current approaches in this area (e.g. capsule networks' dynamic control flow) aren't satisfactory.


Seedbank — discover machine learning examples

Seedbank provides interactive machine learning examples in Colab notebooks (think Jupyter notebooks in the cloud). This makes it really easy to just run example code without any setup, and even to modify it to play around with it. Google even provides a free GPU to make the training and inference faster!

I haven't explored it yet, but this seems great, especially if you want to learn ML. I have used Colab notebooks before and recommend them highly for small projects (maybe even large ones, I'm not sure), especially if you're familiar with Jupyter notebooks.


The Scaling Hypothesis

This post centers around the **scaling hypothesis**:

_Once we find a scalable architecture which can be applied fairly uniformly, we can simply train ever larger networks and ever more sophisticated behavior will emerge naturally as the easiest way to optimize for all the tasks and data. More powerful NNs are “just” scaled-up weak NNs, in much the same way that human brains look much like scaled-up primate brains._

Importantly, we can get this sophisticated behavior just by training on simple objectives, such as “predict the next word”, as long as the data is sufficiently diverse. So, a priori, why might we expect the scaling hypothesis to be true?

The core reason is that optimal (or human-level) prediction of text really does require knowledge, reasoning, causality, etc. If you don’t know how to perform addition, you are probably not going to be able to predict the next word in the sentence “Though he started with six eggs, he found another fourteen, bringing his total to \_\_\_\_”. However, since any specific fact is only useful in a tiny, tiny number of cases, it only reduces the expected loss by a tiny amount. So, you’ll only see models learn this sort of behavior once they have exhausted all the other “easy wins” for predicting text; this will only happen when the models and dataset are huge.

Consider a model tasked with predicting characters in text with a set of 64 characters (52 uppercase and lowercase letters, along with some punctuation). Initially it outputs random characters, assigning a probability of 1/64 to the correct character, resulting in a loss of 6 bits. Once you start training, the easiest win is to simply notice how frequent each character is; just noticing that uppercase letters are rare, spaces are common, vowels are common, etc. could get your error down to 4-5 bits. After this, it might start to learn what words actually exist; this might take 10^5 - 10^6 samples since each word is relatively rare and there are thousands of words to learn, but this is a drop in the bucket given our huge dataset. After this step, it may have also learned punctuation along the way, and might now be down to 3-4 bits. At this point, if you sample from the model, you might get correctly spelled English words, but they won’t make any sense.

With further training the model now has to pick up on associations between adjacent words to make progress. Now it needs to look at things 10 characters ago to predict the next character -- a far cry from our initial letter frequencies where it didn’t even need to look at other characters! For example, it might learn that “George W” tends to be followed by “ashington”. It starts to learn grammar, being able to correctly put verbs in relation to subjects and objects (that are themselves nouns). It starts to notice patterns in how words like “before” and “after” are used; these can then be used to better predict words in the future; at this point it’s clear that the model is starting to learn semantics. Now the loss is around 2 bits per character. A little more training and your model starts to produce sentences that sound human-like in isolation, but don’t fit together: a model might start a story about a very much alive protagonist and then talk about how she is dead in the next sentence. Training is now about fixing errors like these and each such fix gains a tiny amount of accuracy -- think ten thousandths of a bit. Every further 0.1 bits you gain represents the model learning a huge amount of relevant knowledge (and correspondingly each subsequent 0.1 bits takes a much larger amount of training and data). The final few fractions of a bit are the most important and comprise most of what we call “intelligence”.

(The human baseline is a loss of 0.7 bits, with lots of uncertainty on that figure.)

So far this is a clever argument, but doesn’t really establish that this will work _in practice_ -- for example, maybe your model has to have 10^100 parameters to learn all of this, or maybe existing models and algorithms are not sophisticated enough to _find_ the right parameters (and instead just plateau at, say, 2 bits of loss). But recent evidence provides strong support for the scaling hypothesis:

1. The <@scaling laws@>(@Scaling Laws for Neural Language Models@) line of work demonstrated that models could be expected to reach the interesting realm of loss at amounts of compute, data, and model capacity that seemed feasible in the near future.
2. Various projects have trained large models and demonstrated that this allows them to solve tasks that they weren’t explicitly trained for, often in a more human-like way and with better performance than a more supervised approach. Examples include <@GPT-3@>(@Language Models are Few-Shot Learners@), [Image GPT](https://openai.com/blog/image-gpt/), [BigGAN](https://arxiv.org/abs/1809.11096), <@AlphaStar@>(@AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning@), etc. (The full post has something like 25 examples.)

The author then argues that it seems like most researchers seem to be completely ignoring this phenomenon. OpenAI is the only actor that really has the conviction needed to put a large amount of resources behind a project based on the scaling hypothesis (such as GPT-3); DeepMind seems to believe in a weaker version where we need to build a bunch of “modules” similar to those in the human brain, but that those modules can then be scaled up indefinitely. Other actors seem to not take either scaling hypothesis very seriously.

In my view, the scaling hypothesis is easily the most important hypothesis relevant to AI forecasting and AI development models, and this is the best public writeup of it that I know of. (For example, it seems to be an implicit assumption in the <@bio anchors framework@>(@Draft report on AI timelines@).) I broadly agree with the author that it’s a bit shocking how few people seem to be taking it seriously after OpenAI Five, AlphaStar, GPT-3, Copilot, etc.

I *think* this includes the AI safety space, where as far as I can tell the primary effect has been that it is even more fashionable to have shorter timelines, whereas it hasn’t affected AI safety research very much. However, I do know around 3-4 researchers who changed what they were working on based on changing their mind about the scaling hypothesis, so it’s possible there are several others I don’t know about.

As a simple example of how the scaling hypothesis affects AI safety research, it suggests that the training objective (“predict the next word”) is relatively unimportant in determining properties of the trained agent; in contrast, the dataset is much more important. This suggests that analyses based on the “reward function used to train the agent” are probably not going to be very predictive of the systems we actually build.


Feature-wise transformations

Vincent Dumoulin
Ethan Perez
Nathan Schucher
Florian Strub
Harm de Vries
Aaron Courville and Yoshua Bengio

This Distill article is about transformations on features using FiLM (feature-wise linear modulation). A FiLM layer is used to "condition" a neural network on auxiliary information, which just means providing the input to the neural network in a way that it can use it effectively. This can be used to integrate multiple sources of information -- for example, in visual question answering (VQA), the main part of the network can be an image processing pipeline, and FiLM can be used to turn the natural language question about the image into a task representation and integrate it into the pipeline, and the full network can be trained end-to-end. The FiLM layer works by first using a subnetwork to turn the auxiliary information (such as the question in VQA) into a "task representation" (a new representation chosen by the neural network), which is then used as the parameters for an affine transformation of the features in the main pipeline. Importantly, each feature is treated independently of other features, so the FiLM layer can't create interactions between features. Yet, this still works well in many different contexts.

Since it is a Distill paper, it then goes into a ton of detail about lots of interesting details, such as how architectures in a variety of ML tasks can be thought of as FiLM, how FiLM relates to other ideas such as attention, how we can often interpolate between different auxiliary information by taking a weighted combination of the corresponding task information, how conditioning through concatenation is equivalent to FiLM with only a bias and no scaling, etc.

I really enjoy Distill articles, they are consistently far more readable and understandable than typical papers (or even blog posts), even without including the interactive visualizations. This article is no exception. I didn't have particularly strong opinions on how to condition neural nets before, but now I think I will think about FiLM and how it could apply.


Kevin Lu
Aditya Grover
Pieter Abbeel
Igor Mordatch

We’ve seen some very impressive few-shot learning results from <@GPT-3@>(@Language Models are Few-Shot Learners@) and [CLIP](https://openai.com/blog/clip/). These work by training a large Transformer model on a giant pile of data in a particular modality (such as language or images), and then we express tasks within that modality (e.g. summarization  for a language model). This paper asks the question: could those models also help with tasks in a _different_ modality? Surprisingly, the answer seems to be yes!

Specifically, the authors take the pretrained GPT-2 models and finetune on very different tasks, changing only the following parameters (which make up just ~0.1% of the model):
1. Input layer: This is a linear layer that transforms the input tokens before they go through the attention layers.
2. Output layer: This is a linear layer that uses the final representations to solve some downstream tasks.
3. Layer norm: These parameters are meant to mimic the statistics of the data distribution, and so need to be finetuned.
4. Positional embeddings. (They say that it only makes a slight difference to finetune these.)

For downstream tasks, they consider tasks like memorizing bit sequences, computing XORs, MNIST and CIFAR (where each image is represented as a sequence of 64 tokens, and each token is a 4x4 patch of the image), and protein folding. None of these tasks involve any use of natural language -- the input modality is completely different.

The headline result: these sorts of models tend to achieve similar performance as Transformer models trained from scratch on the same tasks, and better performance than models initialized with random weights and then finetuned using the method above. This suggests that _even for new data modalities_ the GPT-2 pretraining helps, suggesting that the model has learned some “universal computation” in its attention layers (hence the title). Note though that the differences from the random initialization are not that large (2-6 percentage points, except 25 percentage points in Bit Memory), suggesting that a lot of this might be the inductive bias of the Transformer architecture itself.

The rest of the paper delves into this more, running several experiments to learn more empirical facts. For example:
1. If the Transformers are pretrained on images instead of language, you do better on image tasks like CIFAR, but not as well on the other tasks.
2. Transformers do a _lot_ better than LSTMs.
3. Pretrained Transformers also learn significantly faster than randomly initialized Transformers.

This is a pretty cool result. I’m not sure what I would have predicted ahead of time -- the gains are small enough that I could believe I might have predicted them on a general basis of “probably training on realistic data gives you slightly better patterns of thought, so probably if you try hard enough you can find a small set of parameters to finetune that would work well”.

However, another possible line of reasoning would be “the attention heuristics learned for language would probably throw away lots of information if we applied them directly to the input tokens, and the input linear layer may not be enough to handle this issue, so probably this just destroys any good performance of the model”. I could see myself being convinced by that too.

Paper: Pretrained Transformers as Universal Computation Engines


Fast and Easy Infinitely Wide Networks with Neural Tangents

Roman Novak*
Lechao Xiao*
Samuel S. Schoenholz*
Jiri Hron
Jaehoon Lee
Alexander A. Alemi
Jascha Sohl-Dickstein

The success of Deep Learning has led researchers to explore why they're such effective function approximators. One key insight is that increasing the width of the network layers makes it *easier* to understand. More precisely, as the width is sent to infinity the network's learning dynamics can be approximated with a Taylor expansion and become a kernel problem. This kernel has an exact form in the limit and is referred to as the neural tangent kernel (NTK). Ultimately, this allows us to model the network with a simpler model known as a Gaussian process. Unfortunately, showing this analytically is difficult and creating efficient implementations is cumbersome. **The authors address this problem by introducing "Neural Tangents", a library that makes creating infinite-width networks as easy as creating their finite counterparts with libraries such as PyTorch or TensorFlow.** They include support for convolutions with full-padding, residual-connections, feed-forward networks, and support for a variety of activation functions. Additionally, there is out-of-the-box support for CPU, GPU, and TPU. Moreover, uncertainty comparisons with finite ensembles are possible via exact Bayesian inference.

I took a look at the repository and found there to be ample documentation available making it easy for me to try training my own infinite-width network. The authors derive a practical way to compute the exact convolutional NTK which I find  impressive and which seems to be the main technical contribution of this paper. While the authors note that there are some conditions necessary to enter the so-called "kernel regime", in practice it seems as though you can often get away with merely large network widths. If for nothing else, I'd recommend at least perusing the notebooks they have available or taking a look at the visualization they present of a neural network converging to a Gaussian process, which relies on a subtle application of the law of large numbers.

Paper: Neural Tangents: Fast and Easy Infinite Neural Networks in Python


Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

Deep neural networks trained with gradient descent do well at generalizing from their training set, but the field currently has relatively little understanding of why that is. Large networks have enough parameters to fully memorize the training set and can do so even if trained on data with entirely random labels. This allows for many functions that would fit the training set well, but not generalize, a phenomenon known as overfitting. The question is how gradient descent picks out one of a small subset of functions that will generalize well.

The *Coherent Gradients* hypothesis, introduced here and tested further in [this paper](http://arxiv.org/abs/2003.07422), is that this results from per-example gradients being averaged during gradient descent. For each example, some of the gradient points in a direction that is idiosyncratic to that example, but some of it points towards a more general solution. When the average is taken across these gradients, the more general directions reinforce each other while the example-specific directions cancel out. As a result, the training process moves faster towards more general directions.

In order to test this hypothesis, they run two experiments. First they use varying amounts of label noise (corrupting a fraction of the dataset to have random labels). They predict and find that:
1. More label noise leads to slower learning.
2. The uncorrupted examples will be learned faster.

The next experiment tests a novel form of regularization, called winsorization, where they clip the gradients on a per-example and per-parameter basis to prevent a single example from dominating the gradient, effectively curtailing the component of the gradient that is example-specific. Since the computation of per-example gradients is expensive, when scaling this up to larger networks, they instead use the median of 3 mini-batches to address outliers. Theexperiments suggest that winsorization reduces overfitting and in particular prevents neural nets from learning randomly labeled data.

The hypothesis makes sense to me and the experiments do seem to bear out their conclusions. However, none of the results of the experiments were surprising to me and seem to me like they could be consistent with other explanations for generalization. I would be more convinced if the Coherent Gradients hypothesis made predictions that were different from other leading theories and then those turned out to be true.

Explaining Memorization and Generalization: A Large-Scale Study with Coherent Gradients


SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Beidi Chen
Tharun Medini
James Farwell
Sameh Gobriel
Charlie Tai
Anshumali Shrivastava

This paper presents an algorithmic technique called SLIDE (Sub-LInear Deep learning Engine) which takes advantage of sparsity in inputs and activations to speed up the training of large neural networks.

Suppose that activations at layer k are a_k. Then, the ith element of a_{k+1} is given by the dot product of a_k and w_i for some weight vector w_i. Call w_i the ith neuron of layer k + 1. The largest activations in a_{k+1} are the ones for whom w_i has high magnitude and points in the same direction as a_k. The core proposal of SLIDE is to only compute the largest elements of a_{k+1}, which they call the “activated neurons”, and approximate all of the others are zero, allowing us to avoid a lot of computation.

In order to do this, we maintain a data structure called a _locality-sensitive hash table_, which when given an activation a_k can tell us which neurons (w_is) are most similar. We can then compute the outputs for just those neurons to get a_{k+1}. In this way, we can effectively ‘sparsify’ the network, calculating the activations and updating the weights of only a small subset of the neurons. This is what gives us our computational gains.

SLIDE randomly initializes weights in the network and generates the locality-sensitive hash table that maps activations to activated neurons. To take a gradient step on an input, it calculates the activated neurons in a forward pass, then backpropagates through the activated neurons, and then updates the locality-sensitive hash table. The hash table update is computationally expensive, and SLIDE uses several mechanisms to make it less costly, such as updating hash tables less frequently later in the training process since gradients are likely to change less then. Due to the sparsity, the gradients for different inputs are often changing different neurons, and so SLIDE asynchronously parallelizes gradient updates without worrying about race conditions, allowing for much better scaling with additional cores.

The paper evaluates SLIDE on large multi-label classification tasks, which must run on neural networks with extremely wide final layers. It finds that the CPUs running SLIDE are 1.8 times faster in clock-time than the GPU on the Delicious 200k dataset, and 2.7 times faster than the GPU on the Amazon-670K dataset, with an additional ~1.3x speed-up after performing cache optimization on SLIDE. Scalability tests suggest that the SLIDE CPUs beat GPU performance even when using only 8 cores. The paper claims that SLIDE’s computational benefits come because the number of neurons sampled in the wide final layer is extremely small-- fewer than 0.5% of active neurons.

The tasks they test on are _extremely_ sparse: since there are hundreds of thousands of possible labels, even if you take the top ~thousand predictions in the final layer (which corresponds to most of the computation), that’s only 1% of the total number of predictions, saving you 99% of the arithmetic you would have had to do. The input features are also very sparse: in both datasets, less than 0.06% (yes, percent) of features are non-zero. It’s cool that under such conditions you can design an algorithm that is ~an order of magnitude better on cost, but it’s not going to be “the death of NVIDIA” or anything like that — without further optimizations, SLIDE will be worse than regular Tensorflow on GPU for something like ImageNet.

I'm also not sure I agree with the 'thesis' of the paper that smart algorithms beat hardware acceleration-- it seems to me like there are large gains from investing in the combination of the two. Even if GPUs aren't optimized to run SLIDE, I can imagine specialized hardware optimized for SLIDE creating even bigger performance gains.


A new model and dataset for long-range memory

Jack W. Rae
Anna Potapenko
Siddhant M. Jayakumar
Timothy P. Lillicrap

A central challenge in language modeling is capturing long-range dependencies. For example, a model needs to be able to identify the antecedent of a pronoun even if it is much earlier in the text. Existing datasets consist of news and Wikipedia articles, where articles have average lengths ranging from 27 to 3,600 words. This paper introduces a dataset of Project Gutenberg books, PG-19, where each book has a much longer average length of 69,000 words. This benchmark enables comparison of how well algorithms can make use of information that is spread out across a much larger context.

They then introduce the *Compressive Transformer*, which builds on the <@*TransformerXL*@>(@Transformer-XL: Unleashing the Potential of Attention Models@). The *TransformerXL* saves old activations into a FIFO queue, discarding them when the queue is full. The *Compressive Transformer* instead has two FIFO queues: the first stores the activations just like *TransformerXL*, but when activations are ejected, they are compressed and added to the second queue. This functions as a sort of long-term memory, storing information from a longer period of time but in a compressed format.

They try a number of types of compression function and find that it is best to use a 1D convolutional compression function with an auxiliary loss that leads to lossy compression, where information that is not attended to can be removed. The compression network and the Transformer optimize independent losses without any mixing.

They find that the *Compressive Transformer* improves on *TransformerXL* on their new PG-19 dataset and is state of the art on the already existing WikiText-103 and Enwik8 benchmarks. They also inspect where the network attends to and find that more attention is paid to the compressed memory than the oldest activations in regular memory, showing that the network is preserving some valuable information.

I like the idea of saving long-term memory in a more efficient but lower-dimensional format than short-term memory. The current <@trend@>(@Scaling Laws for Neural Language Models@) in language modelling is that more computation leads to better results, so I think that algorithms that target computation on the most relevant information are promising. I’d be interested to see (and curious if the authors tried) more continuous variants of this where older information is compressed at a higher rate than newer information, since it seems rather arbitrary to split into two FIFO queues where one has a fixed compression rate.

I’m not well calibrated on the meaning of the evaluation metrics for NLP, so I don’t have a sense of how much of an improvement this is over the *TransformerXL*. I looked through some of the example text they gave in the blog post and thought it was impressive but has clear room for improvement.

Paper: Compressive Transformers for Long-Range Sequence Modelling


The Quiet Semi-Supervised Revolution

Historically, semi-supervised learning that uses small amounts of labelled data combined with a lot of unlabeled data only helped when there was very little labelled data available. In this regime, both supervised and semi-supervised learning were too inaccurate to be useful. Furthermore, approaches like using a representation learnt by an autoencoder for classification empirically limited asymptotic performance. This is strange because using more data should not lead to worse performance.

Recent trends suggest that this might change soon: semi-supervised systems have begun to outperform supervised systems by larger and larger margins in the low data regime and their advantage now extends into regimes with more and more data. An important driver of this trend is the idea of using data augmentation for more consistent self-labelling.

Better semi-supervised learning might for example be useful for  federated learning which attempts to respect privacy by learning locally on (labelled) user data and sending the models trained by different users to be combined in a central server. One problem with this approach is that the central model might memorize some of the private models' idiosyncracies such that inference about the private labels is possible. Semi-supervised learning makes this harder by reducing the amount of influence private data has on the aggregate model.

Because the way humans classify things are strongly influenced by our priors about how classes "should" behave, learning with limited data most likely requires some information about these priors. Semi-supervised learning that respects that data augmentation does not change the correct classification might be an efficient and scalable way to force some of these priors onto a model. Thus it seems likely that more diverse and sophisticated data augmentation could lead to further improvements in the near term. On the other hand, it seems like a lot of our priors would be very hard to capture only using automatic data augmentation, such that other methods to transfer our priors are still important.


Uniform convergence may be unable to explain generalization in deep learning

This post argues that existing generalization bounds cannot explain the empirical success of neural networks at generalizing to the test set.

"What?", you say if you're like me, "didn't we already know this? Generalization bounds depend on your hypothesis space being sufficiently small, but [neural nets can represent any reasonable function](https://en.wikipedia.org/wiki/Universal_approximation_theorem)? And even if you avoid that by considering the size of the neural net, we know that empirically [neural nets can learn randomly labeled data](https://arxiv.org/abs/1611.03530), which can never generalize; surely this means that you can't explain generalization without reference to some property of the dataset, which generalization bounds typically don't do?"

It turns out that the strategy has been to prove generalization bounds that depend on the _norm of the weights of the trained model_ (for some norm that depends on the specific bound), which gets around both these objections, since the resulting bounds are independent of the number of parameters, and depend on the trained model (which itself depends on the dataset). However, when these bounds are evaluated on a simple sphere-separation task, they _increase_ with the size of the training dataset, because the norms of the trained models increase.

Okay, but can we have a stronger argument than mere empirical results? Well, all of these bounds depend on a _uniform convergence bound_: a number that bounds the absolute difference between the train and test error for _any_ model in your hypothesis space. (I assume the recent generalization bounds only consider the hypothesis space "neural nets with norms at most K", or some suitable overapproximation of that, and this is how they get a not-obviously-vacuous generalization bound that depends on weight norms. However, I haven't actually read those papers.)

However, no matter what hypothesis space these bounds choose, to get a valid generalization bound the hypothesis space must contain (nearly) all of the models that would occur by training the neural net on a dataset sampled from the underlying distribution. What if we had the actual smallest such hypothesis space, which only contained the models that resulted from an actual training run? The authors show that, at least on the sphere-separation task, the uniform convergence bound is still extremely weak. Let's suppose we have a training dataset S. Our goal is now to find a model in the hypothesis space which has a high absolute difference between actual test error, and error in classifying S. (Recall that uniform convergence requires you to bound the absolute difference for _all_ models in your hypothesis class, not just the one trained on S.) The authors do so by creating an "adversarial" training dataset S' that also could have been sampled from the underlying distribution, and training a model on S'. This model empirically gets S almost completely wrong. Thus, this model has low test error, but high error in classifying S, which forces the uniform convergence bound to be very high.

I enjoyed this blog post a lot (though it took some time to digest it, since I know very little about generalization bounds). It constrains the ways in which we can try to explain the empirical generalization of neural networks, which I for one would love to understand. Hopefully future work will explore new avenues for understanding generalization, and hit upon a more fruitful line of inquiry.


Understanding the generalization of ‘lottery tickets’ in neural networks

The <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis at Scale@) states that a randomly initialized dense or convolutional neural network contains (sparse) subnetworks, called "winning tickets", which can be trained to achieve performance similar to the trained base network while requiring a lot less compute.

The blogpost summarizes facebook AI's recent investigations of the generalization of winning tickets and the generality of the hypothesis. Because winning tickets are hard to find, we would like to reuse the ones we have found for similar tasks. To test whether this works, the authors trained classifiers, pruned and reset them to obtain winning tickets on different image datasets and then trained these on other datasets. Winning tickets derived from similar datasets relevantly outperform random subnetworks after training and ones derived from larger or more complex datasets generalize better. For example, tickets from ImageNet are consistently among the best and tickets from CIFAR-100 generalize better than those from CIFAR-10.

Experiments in natural language processing and reinforcement learning suggest that the lottery ticket hypothesis is not just a peculiarity of image classification: for example, the performance of a large transformer model could be recovered from a winning ticket with just a third of the original weights, whereas random tickets with that amount of weights performed quite a bit worse. The analysis of simple shallow neural networks in a student-teacher setting is used as a toy model: when a larger student network is trained to mimic a smaller teacher with the same amount of layers, **student specialization** happens: some of the student's neurons learn to imitate single neurons of the teacher. This can be seen to happen more often and faster if the student neuron is already close to the teacher neuron at initialization. If the student network is large enough, every teacher neuron will be imitated by some student neuron and these student neurons collectively form a winning ticket.

I enjoyed reading this blogpost and like the idea of using winning tickets for transfer learning. I would have been quite surprised if they had found that the lottery ticket hypothesis was specific to image classification, as similar to pretraining, winning tickets seem to provide an inductive bias constraining the set of features that can be learnt during training to more useful ones. I do not think that further research into that direction will directly help with quickly training models for novel tasks unless the tickets can be identified very efficiently which seems like a harder optimization problem than just training a network by gradient descent.


SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver

Po-Wei Wang
Priya L. Donti
Bryan Wilder
Zico Kolter

Historically, deep learning architectures have struggled with problems that involve logical reasoning, since they often impose non-local constraints that gradient descent has a hard time learning. This paper presents a new technique, SATNet, which allows neural nets to solve logical reasoning problems by encoding them explicitly as MAXSAT-solving neural network layers. A MAXSAT problem provides a large set of logical constraints on an exponentially large set of options, and the goal is to find the option that satisfies as many logical constraints as possible. Since MaxSAT is NP-complete, the authors design a layer that solves a relaxation of the MaxSAT problem in its forward pass (that can be solved quickly, unlike MaxSAT), while the backward pass computes gradients as usual.

In experiment, SATNet is given bit representations of 9,000 9 x 9 Sudoku boards which it uses to learn the logical constraints of Sudoku, then presented with 1,000 test boards to solve. SATNet vastly outperforms traditional convolutional neural networks given the same training / test setup, achieving 98.3% test accuracy where the convolutional net achieves 0%. It performs similarly well on a "Visual" Sudoku problem where the trained network consists of initial layers that perform digit recognition followed by SATNet layers, achieving 63.2% accuracy where the convolutional net achieves 0.1%.

My impression is this is a big step forward in being able to embed logical reasoning in current deep learning techniques. From an engineering perspective, it seems extremely useful to be able to train systems that encorporate these layers end-to-end. It's worth being clear that in systems like these, a lot of generality is lost since part of the network is explicitly carved out for solving a particular problem of logical constraints-- it would be hard to use the same network to learn a different problem.


the transformer … “explained”?

This is an excellent explanation of the intuitions and ideas behind self-attention and the [Transformer](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) <@architecture@>(@Transformer-XL: Unleashing the Potential of Attention Models@).


Do we still need models or just more data and compute?

This is a response to <@The Bitter Lesson@>, that emphasizes the importance of data in addition to compute. It brings up a number of considerations that seem important to me, and is worth reading if you want to better understand my position on the bitter lesson.


Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park
Ming-Yu Liu
Ting-Chun Wang and Jun-Yan Zhu

This paper shows how to create somewhat realistic images specified by semantic segmentation maps. They accomplish this by modifying batch normalization. Batch normalization modifications can be quite powerful for image generation, even enough to [control style](https://arxiv.org/abs/1703.06868). Their modification is that normalization is a direct function of the semantic segmentation map throughout the network, so that the semantic segmentation map is readily available to each ResBlock. Visualizations produced by this method are [here](https://nvlabs.github.io/SPADE/).


Measuring the Intrinsic Dimension of Objective Landscapes

Chunyuan Li
Rosanne Liu
and Jason Yosinski

This paper proposes and defines a quantity called "intrinsic dimension", a geometrically-informed metric of how many degrees of freedom are actually needed to train a given model on a given dataset.  They calculate this by picking a set of random directions that span some subspace of dimension d, and taking gradient steps only along that lower-dimensional subspace. They consider the intrinsic dimension of a model and a dataset to be the smallest value d at which performance reaches 90% of a baseline, normally trained model on the dataset. The geometric intuition of this approach is that the dimensionality of parameter space can be, by definition, split into intrinsic dimension and its codimension, the dimension of the solution set. In this framing, higher solution set dimension (and lower intrinsic dimension) corresponds to proportionally more of the search space containing reasonable solution points, and therefore a situation where a learning agent will be more likely to find such a solution point. There are some interesting observations here that correspond with our intuitions about model trainability: on MNIST, intrinsic dimensionality for a CNN is lower than for a fully connected network, but if you randomize pixel locations, CNN's intrinsic dimension shoots up above FC, matching the intuition that CNNs are appropriate when their assumption of local structure holds.

Overall, I find this an interesting and well-articulated paper, and am curious to see future work that addresses some of the extrapolations and claims implied by this paper, particularly their claim, surprising relative to my intuitions, that increasing n_parameters will, maybe monotonically, reduce difficulty of training, because it simply increases the dimensionality of the solution set. I'm also not sure how to feel about their simply asserting that a solution exists when a network reaches 90% of baselines performance, since we may care about that "last mile" performance and it might also be the harder to reach.


Measuring the Limits of Data Parallel Training for Neural Networks

Consider the relationship between the size of a single batch and the number of batches needed to reach a specific performance bound when using deep learning. If all that mattered for performance was the total number of examples that you take gradient steps on (i.e. the product of these two numbers), then you would expect a perfect inverse relationship between these two quantities, which would look like a line with negative slope on a log-log plot. In this case, we could scale batch sizes up arbitrarily far, and distribute them across as many machines as necessary, in order to reduce wall clock training time. A 2x increase in batch size with twice as many machines would lead to a 2x decrease in training time. However, as you make batch sizes really large, you face the problem of stale gradients: if you had updated on the first half of the batch and then computed gradients on the second half of the batch, the gradients for the second half would be "better", because they were computed with respect to a better set of parameters. When this effect becomes significant, you no longer get the nice linear scaling from parallelization.

This post studies the relationship empirically across a number of datasets, architectures, and optimization algorithms. They find that universally, there is initially an era of perfect linear scaling as you increase batch size, followed by a region of diminishing marginal returns that ultimately leads to an asymptote where increasing batch size doesn't help at all with reducing wall-clock training time. However, the transition points between these regimes vary wildly, suggesting that there may be low hanging fruit in the design of algorithms or architectures that explicitly aim to achieve very good scaling.

OpenAI <@found@>(@How AI Training Scales@) that the best predictor of the maximum useful batch size was how noisy the gradient is. Presumably when you have noisy gradients, a larger batch size helps "average out" the noise across examples. Rereading their post, I notice that they mentioned the study I've summarized here and said that their results can help explain why there's so much variance in the transition points _across datasets_. However, I don't think it can explain the variance in transition points _across architectures_. Noisy gradients are typically a significant problem, and so it would be weird if the variance in transition points across architectures were explained by the noisiness of the gradient: that would imply that two architectures reach the same final performance even though one had the problem of noisy gradients while the other didn't. So there seems to be something left to explain here.

That said, I haven't looked in depth at the data, so the explanation could be very simple. For example, maybe the transition points don't vary much across architecture and vary much more across datasets, and the variance across architecture is small enough that its effect on performance is dwarfed by all the other things that can affect the performance of deep learning systems. Or perhaps while the noisiness of the gradient is a good predictor of the maximum batch size, it still only explains say 40% of the effect, and so variance across architectures is totally compatible with factors other than the gradient noise affecting the maximum batch size.


How AI Training Scales

Sam McCandlish
Jared Kaplan and Dario Amodei

OpenAI has done an empirical investigation into the performance of AI systems, and found that the maximum useful batch size for a particular task is strongly influenced by the noise in the gradient. (Here, the noise in the gradient comes from the fact that we are using _stochastic_ gradient descent -- any difference in the gradients across batches counts as "noise".) They also found some preliminary results showing the more powerful ML techniques tend to have more gradient noise, and even a single model tends to have increased gradient noise over time as they get better at the task.

While OpenAI doesn't speculate on why this relationship exists, it seems to me that as you get larger batch sizes, you are improving the gradient by reducing noise by averaging over a larger batch. This predicts the results well: as the task gets harder and the noise in the gradients gets larger, there's more noise to get rid of by averaging over data points, and so there's more opportunity to have _even larger_ batch sizes.


Relational Deep Reinforcement Learning

Vinicius Zambaldi
David Raposo
Adam Santoro
Victor Bapst
Yujia Li
Igor Babuschkin
Karl Tuyls
David Reichert
Timothy Lillicrap
Edward Lockhart
Murray Shanahan
Victoria Langston
Razvan Pascanu
Matthew Botvinick
Oriol Vinyals
Peter Battaglia

This paper uses the self-attention mechanism discussed in 'Relational recurrent neural networks' to compute relationships between entities extracted from input data. The system was tested on the Box-World environment, in which an agent needs to use keys to open boxes in a certain order. It generalised very well to test environments which required much longer sequences of actions than any training examples, and improved slightly on a baseline for Starcraft mini-games.

Getting neural networks to generalise to longer versions of training problems is often surprisingly difficult, so I'm impressed by the Box-World results; I would have liked to see what happened on even longer problems.


Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Alethea Power
Yuri Burda
Harri Edwards
Igor Babuschkin
Vedant Misra

This paper presents an interesting empirical phenomenon with deep learning: **grokking**.

Consider tasks of the form “a ◦ b = ?”, where “◦” is some operation on modular arithmetic. For example, in the task of addition mod 97, an example problem would be “32 + 77 = ?”. There are exactly 97 possible operands, each of which gets its own token, and so there are 97^2 possible problems that are defined by pairs of tokens. We will train a neural net on some fraction of all possible problems and then ask how well it performs on the remaining problems it didn’t see: that is, we’re asking it to fill in the missing entries in the 97x97 table that defines addition mod 97.

It turns out that in these cases, the neural net memorizes the training dataset pretty quickly (in around 10^3 updates), at which point it has terrible generalization performance. However, if you continue to train it all the way out to 10^6 updates, then it will often hit a phase transition where you go from random chance to perfect generalization almost immediately. Intuitively, at the point of the phase transition, the network has “grokked” the function and can run it on new inputs as well. Some relevant details about grokking:

1. It isn’t specific to group or ring operations: you also see grokking for tasks like “a/b if b is odd, otherwise a − b”.
2. It is quite sensitive to the choice of hyperparameters, especially learning rate; the learning rate can only vary over about a single order of magnitude.
3. The time till perfect generalization is reduced by weight decay and by adding noise to the optimization process.
4. When you have 25-30% of possible examples as training data, a decrease of 1 percentage point leads to an increase of 40-50% in the median time to generalization.
5. As problems become more intuitively complicated, time till generalization increases (and sometimes generalization doesn’t happen at all). For example, models failed to grok the task x^3 + xy^2 + y (mod 97) even when provided with 95% of the possible examples as training data.
6. Grokking mostly still happens even when adding 1,000 “outliers” (points that could be incorrectly labeled), but mostly stops happening at 2,000 “outliers”.

1st Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021

Another interesting fact about neural net generalization! Like <@double descent@>(@Deep Double Descent@), this can’t easily be explained by appealing to the diversity model. I don’t really have a good theory for either of these phenomena, but one guess for grokking is that:

1. Functions that perfectly memorize the data without generalizing (i.e. probability 1 on the true answer and 0 elsewhere) are very complicated, nonlinear, and wonky. The memorizing functions learned by deep learning don’t get all the way there and instead assign a probability of (say) 0.95 to the true answer.
2. The correctly generalizing function is much simpler and for that reason can be easily pushed by deep learning to give a probability of 0.99 to the true answer.
3. Gradient descent quickly gets to a memorizing function, and then moves mostly randomly through the space, but once it hits upon the correctly generalizing function (or something close enough to it), it very quickly becomes confident in it, getting to probability 0.99 and then never moving very much again.

A similar theory could explain deep double descent: the worse your generalization, the more complicated, nonlinear and wonky you are, and so the more you explore to find a better generalizing function. The biggest problem with this theory is that it suggests that making the neural net larger should primarily advantage the memorizing functions, but in practice I expect it will actually advantage the correctly generalizing function. You might be able to rescue the theory by incorporating aspects of the <@lottery ticket hypothesis@>(@The Lottery Ticket Hypothesis at Scale@).


Branch Specialization

Chelsea Voss
Gabriel Goh
Nick Cammarata
Michael Petrov
Ludwig Schubert
Chris Olah

Neural network architectures sometimes have different “branches”, where later features can depend on earlier features _within the same branch_, but cannot depend on features in parallel branches. This post presents evidence showing that in these architectures, branches often tend to specialize in particular types of features. For example:

1. The first two layers in AlexNet are split into two branches. In one branch, the first layer tends to learn black and white Gabor filters, while in the other branch, the first layer tends to learn low-frequency color detectors. This persists across retraining, or even training on a different dataset of natural images, such as Places (rather than ImageNet).
2. All 9 of the black and white vs. color detectors in mixed3a are in mixed3a_5x5 (p < 1e-8). All 30 of the curve-related features in mixed3b are in mixed3b_5x5 (p < 1e-20). There are confounds here, but also good reasons to expect that it is in fact branch specialization.

Given that branch specialization seems to be robust and consistent even across datasets, a natural hypothesis is that it is reflecting a structure that already exists. Even if you didn’t have branching, it seems likely that the model would still learn very similar neurons, and it seems plausible that e.g. the weights connecting the first-layer black-and-white Gabor filters to the second-layer color detectors are effectively zero. With branching, you learn the same features in such a way that all the weights that previously were effectively zero now don’t exist because they would be crossing branches. This would look like having the Gabor filters in one branch and the color detectors in the other branch.

I find the hypothesis the authors propose quite compelling (and this is very similar to the hypothesis that neural networks tend to be modular, which we discuss more below). Partly, this is because it has a common-sense explanation: when designing an organization, you want to put related functions in the same group to minimize the communication across groups. Here, the full network is the organization, the branches are an explicit constraint on communication, and so you want to put related functions in the same branch.

At the end of the article, the authors also suggest that there could be a connection with the way that different regions of the brain are specialized to particular tasks. I’ll go further than the authors in my speculation: it seems plausible to me that this specialization is simply the result of the brain’s learning algorithm reflecting the structure of the world through specialization. (Though it seems likely that the different areas of the brain must at least have different “architectures”, in order for the same tasks to be routed to the same brain regions across humans.) But the case of AlexNet demonstrates that in theory, the only thing you need for specialization to arise is a restriction on the communication between one part of the architecture and the other.