The Paths Perspective on Value Learning

Introduction
------------

 In the last few years, reinforcement learning (RL) has made remarkable progress, including [beating world-champion Go players](https://deepmind.com/research/alphago/), [controlling robotic hands](https://blog.openai.com/learning-dexterity/), and even [painting pictures](https://deepmind.com/blog/learning-to-generate-images/).

 One of the key sub-problems of RL is value estimation – learning the long-term consequences of being in a state.

 This can be tricky because future returns are generally noisy, affected by many things other than the present state. The further we look into the future, the more this becomes true.

 But while difficult, estimating value is also essential to many approaches to RL.For many approaches (policy-value iteration), estimating value essentially is the whole problem, while in other approaches (actor-critic models), value estimation is essential for reducing noise.

 The natural way to estimate the value of a state is as the average return you observe from that state. We call this Monte Carlo value estimation.

**Cliff World**
 is a classic RL example, where the agent learns to
 walk along a cliff to reach a goal.

![](figures/cliffworld-path1.svg)

 Sometimes the agent reaches its goal.

![](figures/cliffworld-path2.svg)

 Other times it falls off the cliff.

![](figures/cliffworld-mc.svg)

 Monte Carlo averages over trajectories where they intersect.

 If a state is visited by only one episode, Monte Carlo says its value is the return of that episode. If multiple episodes visit a state, Monte Carlo estimates its value as the average over them.

 Let’s write Monte Carlo a bit more formally.
 In RL, we often describe algorithms with update rules, which tell us how estimates change with one more episode.
 We’ll use an “updates toward” (↩\hookleftarrow↩) operator to keep equations simple.

 In tabular settings such as the Cliff World example, this “update towards” operator computes a running average. More specifically, the nthn^{th}nth Monte Carlo update is V(st)=V(st−1)+1n[Rn−V(st)] V(s\_t) = V(s\_{t-1}) + \frac{1}{n} \bigr[ R\_{n} - V(s\_t) \bigl] V(st)=V(st−1)+n1[Rn−V(st)] and we could just as easily use the “+=” notation. But when using parameteric function approximators such as neural networks, our “update towards” operator may represent a gradient step, which cannot be written in “+=” notation. In order to keep our notation clean and general, we chose to use the ↩\hookleftarrow↩ operator throughout.

V(st)  V(s\_t)~~V(st)
↩  \hookleftarrow~~↩
RtR\_tRt
 State value
 Return

 The term on the right is called the return and we use it to measure the amount of long-term reward an agent earns. The return is just a weighted sum of future rewards rt+γrt+1+γ2rt+2+...r\_{t} + \gamma r\_{t+1} + \gamma^2 r\_{t+2} + …rt+γrt+1+γ2rt+2+... where γ\gammaγ is a discount factor which controls how much short term rewards are worth relative to long-term rewards. Estimating value by updating towards return makes a lot of sense. After all, the *definition* of value is expected return. It might be surprising that we can do better.

Beating Monte Carlo
-------------------

 But we can do better! The trick is to use a method called *Temporal Difference (TD) learning*, which bootstraps off of nearby states to make value updates.

V(st)  V(s\_t)~~V(st)
↩  \hookleftarrow~~↩
rtr\_{t} rt
+++
γV(st+1)\gamma V(s\_{t+1})γV(st+1)
 State value
Reward
Next state value

 Intersections between two trajectories are handled differently under this update. Unlike Monte Carlo, TD updates merge intersections so that the return flows backwards to all preceding states.

![](figures/cliffworld-path1.svg)

 Sometimes the agent reaches its goal.

![](figures/cliffworld-path2.svg)

 Other times it falls off the cliff.

![](figures/cliffworld-td.svg)

 TD learning merges paths where they intersect.

 What does it mean to “merge trajectories” in a more formal sense? Why might it be a good idea? One thing to notice is that V(st+1)V(s\_{t+1})V(st+1) can be written as the expectation over all of its TD updates:

V(st+1)  V(s\_{t+1})~~V(st+1)
≃  \simeq~~≃
E[rt+1′ + γV(st+2′)]\mathop{\mathbb{E}} \bigr[ r’\_{t+1} ~+~ \gamma V(s’\_{t+2}) \bigl] E[rt+1′ + γV(st+2′)]
≃  \simeq~~≃
E[rt+1′]  +  γE[V(st+2′)]\mathop{\mathbb{E}} \bigr[ r’\_{t+1} \bigl] ~~+~~ \gamma \mathop{\mathbb{E}} \bigr[ V(s’\_{t+2}) \bigl] E[rt+1′]  +  γE[V(st+2′)]

 Now we can use this equation to expand the TD update rule recursively:

V(st) V(s\_t)~V(st)
↩ \hookleftarrow~↩
rtr\_{t} rt
+++
γV(st+1)\gamma V(s\_{t+1})γV(st+1)

↩ \hookleftarrow~↩
rtr\_{t} rt
+++
γE[rt+1′]\gamma \mathop{\mathbb{E}} \bigr[ r’\_{t+1} \bigl]γE[rt+1′]
+++
γ2E[V(st+2′′)]\gamma^2 \mathop{\mathbb{E}} \bigr[ V(s’’\_{t+2}) \bigl]γ2E[V(st+2′′)]

↩ \hookleftarrow~↩
rtr\_{t} rt
+++
γE [rt+1′]\gamma \mathop{\mathbb{E}} ~ \bigr[ r’\_{t+1} \bigl]γE [rt+1′]
+++
γ2EE [rt+2′′]\gamma^2 \mathop{\mathbb{EE}} ~ \bigr[ r’’\_{t+2} \bigl]γ2EE [rt+2′′]
+ +~+
...  …~~...

 This gives us a strange-looking sum of nested expectation values. At first glance, it’s not clear how to compare them with the more simple-looking Monte Carlo update. More importantly, it’s not clear that we *should* compare the two; the updates are so different that it feels a bit like comparing apples to oranges. Indeed, it’s easy to think of Monte Carlo and TD learning as two entirely different approaches.

 But they are not so different after all. Let’s rewrite the Monte Carlo update in terms of reward and place it beside the expanded TD update.

**MC update**
V(st) V(s\_t)~V(st)
 ↩  ~\hookleftarrow~~ ↩
rtr\_{t}rt
+ +~+
γ rt+1\gamma ~ r\_{t+1}γ rt+1
+ +~+
γ2 rt+2\gamma^2 ~ r\_{t+2}γ2 rt+2
+ +~+
...…...
Reward from present path.
Reward from present path.
Reward from present path…
**TD update**
V(st) V(s\_t)~V(st)
 ↩  ~\hookleftarrow~~ ↩
rtr\_{t}rt
+ +~+
γE [rt+1′]\gamma \mathop{\mathbb{E}} ~ \bigr[ r’\_{t+1} \bigl]γE [rt+1′]
+ +~+
γ2EE [rt+2′′]\gamma^2 \mathop{\mathbb{EE}} ~ \bigr[ r’’\_{t+2} \bigl]γ2EE [rt+2′′]
+ +~+
...…...
Reward from present path.
Expectation over paths intersecting present path.
Expectation over paths intersecting *paths intersecting* present path…

 A pleasant correspondence has emerged. The difference between Monte Carlo and TD learning comes down to the nested expectation operators. It turns out that there is a nice visual interpretation for what they are doing. We call it the *paths perspective* on value learning.

The Paths Perspective
---------------------

 We often think about an agent’s experience as a series of trajectories. The grouping is logical and easy to visualize.

![](figures/cliffworld-path-1of4.svg)

 Trajectory 1

![](figures/cliffworld-path-2of4.svg)

 Trajectory 2

 But this way of organizing experience de-emphasizes relationships *between* trajectories. Wherever two trajectories intersect, both outcomes are valid futures for the agent. So even if the agent has followed Trajectory 1 to the intersection, it could *in theory* follow Trajectory 2 from that point onward. We can dramatically expand the agent’s experience using these simulated trajectories or “paths.”

 #cliffworld-paths {
 display: grid;
 grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));
 grid-gap: 20px;
 }

![](figures/cliffworld-path-1of4.svg)

 Path 1

![](figures/cliffworld-path-2of4.svg)

 Path 2

![](figures/cliffworld-path-3of4.svg)

 Path 3

![](figures/cliffworld-path-4of4.svg)

 Path 4

**Estimating value.** It turns out that Monte Carlo is averaging over real trajectories whereas TD learning is averaging over all possible paths. The nested expectation values we saw earlier correspond to the agent averaging across *all possible future paths*.

 #compare-mctd {
 display: grid;
 grid-gap: 40px;
 grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
 }
 #compare-mctd .subfigure {
 display: grid;
 grid-gap: 20px;
 grid-template-columns: repeat(2, minmax(160px, 1fr));
 /\* grid-auto-rows: min-content; \*/
 /\* grid-template-rows: min-content auto; \*/
 }
 #compare-mctd .subfigure .column-heading {
 grid-column: 1 / -1;
 }

 #compare-mctd .figcaption {
 border-top: 1px solid rgba(0, 0, 0, 0.1);
 padding-top: 5px;
 margin-top: 5px;
 /\* min-height: 179px; \*/
 /\* min-width: 179px; \*/
 /\* flex: 1; \*/
 }

#### Monte Carlo Estimation

![](figures/traj-thumbnails.svg)

 Averages over **real trajectories**

![](figures/cliffworld-mc.svg)

 Resulting MC estimate

#### Temporal Difference Estimation

![](figures/path-thumbnails.svg)

 Averages over **possible paths**

![](figures/cliffworld-td.svg)

 Resulting TD estimate

**Comparing the two.** Generally speaking, the best value estimate is the one with the lowest variance. Since tabular TD and Monte Carlo are empirical averages, the method that gives the better estimate is the one that averages over more items. This raises a natural question: Which estimator averages over more items?

Var[V(s)]  Var[V(s)]~~Var[V(s)]
∝  \propto~~∝
1N\frac{1}{N} N1
 Variance of estimate
Inverse of the number of items in the average

 First off, TD learning never averages over fewer trajectories than Monte Carlo because there are never fewer simulated trajectories than real trajectories. On the other hand, when there are *more* simulated trajectories, TD learning has the chance to average over more of the agent’s experience.

 This line of reasoning suggests that TD learning is the better estimator and helps explain why TD tends to outperform Monte Carlo in tabular environments.

Introducing Q-functions
-----------------------

 An alternative to the value function is the Q-function. Instead of estimating the value of a state, it estimates the value of a state and an action. The most obvious reason to use Q-functions is that they allow us to compare different actions.

 #qlearning-intro {
 display: grid;
 grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));
 grid-column-gap: 30px;
 }

![](figures/policy.svg)

 Many times we’d like to compare the value of actions under a policy.

![](figures/value.svg)

 It’s hard to do this with a value function.

![](figures/qvalue.svg)

 It’s easier to use Q-functions, which estimate joint state-action values.

 There are some other nice properties of Q-functions. In order to see them, let’s write out the Monte Carlo and TD update rules.

**Updating Q-functions.** The Monte Carlo update rule looks nearly identical to the one we wrote down for V(s)V(s)V(s):

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)
↩  \hookleftarrow~~↩
RtR\_tRt
 State-action value
 Return

 We still update towards the return. Instead of updating towards the return of being in some state, though, we update towards the return of being in some state *and* selecting some action.

 Now let’s try doing the same thing with the TD update:

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)
↩  \hookleftarrow~~↩
rtr\_{t} rt
+++
γQ(st+1,at+1)\gamma Q(s\_{t+1}, a\_{t+1})γQ(st+1,at+1)
 State-action value
Reward
Next state value

 This version of the TD update rule requires a tuple of the form (st,at,rt,st+1,at+1)(s\_t, a\_t, r\_{t}, s\_{t+1}, a\_{t+1})(st,at,rt,st+1,at+1), so we call it the *Sarsa* algorithm.
 Sarsa may be the simplest way to write this TD update, but it’s not the most efficient.
 The problem with Sarsa is that it uses Q(st+1,at+1)Q(s\_{t+1},a\_{t+1})Q(st+1,at+1) for the next state value when it really should be using V(st+1)V(s\_{t+1})V(st+1).

 What we need is a better estimate of V(st+1)V(s\_{t+1})V(st+1).

Q(st,at)  Q(s\_t, a\_t)~~Q(st,at)
↩  \hookleftarrow~~↩
rtr\_{t} rt
+++
γV(st+1)\gamma V(s\_{t+1})γV(st+1)
 State-action value
Reward
Next state value
V(st+1)  V(s\_{t+1})~~V(st+1)
=  =~~=
?? ?

 There are many ways to recover V(st+1)V(s\_{t+1})V(st+1) from Q-functions. In the next section, we’ll take a close look at four of them.

Learning Q-functions with reweighted paths
------------------------------------------

**Expected Sarsa.**
 A better way of estimating the next state’s value is with a weighted sumAlso written as an expectation value, hence “Expected Sarsa”. over its Q-values. We call this approach Expected Sarsa:

**Sarsa** uses the Q-value associated with at+1a\_{t+1}at+1 to estimate the next state’s value.

![](figures/sarsa.svg)

**Expected Sarsa** uses an expectation over Q-values to estimate the next state’s value.

![](figures/expected-sarsa.svg)

 Here’s a surprising fact about Expected Sarsa: the value estimate it gives is often *better* than a value estimate computed straight from the experience. This is because the expectation value weights the Q-values by the true policy distribution rather than the empirical policy distribution. In doing this, Expected Sarsa *corrects for the difference between the empirical policy distribution and the true policy distribution.*

**Off-policy value learning.** We can push this idea even further. Instead of weighting Q-values by the true policy distribution, we can weight them by an arbitrary policy, πoff\pi^{off}πoff:

**Off-policy value learning** weights Q-values by an arbitrary policy.

![](figures/off-policy.svg)

 This slight modification lets us estimate value under any policy we like. It’s interesting to think about Expected Sarsa as a special case of off-policy learning that’s used for on-policy estimation.

**Re-weighting path intersections.** What does the paths perspective say about off-policy learning? To answer this question, let’s consider some state where multiple paths of experience intersect.

 #reweighting {
 display: grid;
 grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));
 grid-gap: 30px;
 }
 /\* #reweighting .wrapper {
 min-width: 160px;
 } \*/
 #reweighting-full .cls-1 {
 fill: #2d307b;
 }

 #reweighting-full .cls-2 {
 fill: #e7ebe8;
 }

 #reweighting-full .cls-3 {
 fill: #cac9cc;
 }

 #reweighting-full .cls-4 {
 fill: #bd5f35;
 }

 #reweighting-full .cls-5, .cls-6, .cls-7 {
 fill: none;
 stroke-width: 10px;
 }

 #reweighting-full .cls-5 {
 stroke: #bd5f35;
 }

 #reweighting-full .cls-5, .cls-7 {
 stroke-miterlimit: 10;
 }

 #reweighting-full .cls-6 {
 stroke: #2d307b;
 stroke-linecap: round;
 stroke-linejoin: round;
 }

 #reweighting-full .cls-7 {
 stroke: #8191c9;
 }

 #reweighting-full .cls-8 {
 fill: #8191c9;
 }

 #reweighting-full .cls-9 {
 font-size: 24.25341px;
 fill: #d1d3d4;
 font-family: Arial-BoldMT, Arial;
 font-weight: 700;
 }

![](figures/reweighting-1.svg)

 Multiple paths of experience intersect at this state.

![](figures/reweighting-2.svg)

 Paths that exit the state through different actions get associated with different Q-values.

+2
-1

Weight of upward path:

 Whenever we re-weight Q-values, we also re-weight the paths that pass through them.

 Wherever intersecting paths are re-weighted, the paths that are most representative of the off-policy distribution end up making larger contributions to the value estimate. Meanwhile, paths that have low probability make smaller contributions.

**Q-learning.** There are many cases where an agent needs to collect experience under a sub-optimal policy (e.g. to improve exploration) while estimating value under an optimal one. In these cases, we use a version of off-policy learning called Q-learning.

**Q-learning** estimates value under the optimal policy by choosing the max Q-value.

![](figures/q-learning.svg)

 Q-learning prunes away all but the highest-valued paths. The paths that remain are the paths that the agent will follow at test time; they are the only ones it needs to pay attention to. This sort of value learning often leads to faster convergence than on-policy methodsTry using the Playground at the end of this article to compare between approaches..

**Double Q-Learning.** The problem with Q-learning is that it gives biased value estimates. More specifically, it is over-optimistic in the presence of noisy rewards. Here’s an example where Q-learning fails:

*You go to a casino and play a hundred slot machines. It’s your lucky day: you hit the jackpot on machine 43. Now, if you use Q-learning to estimate the value of being in the casino, you will choose the best outcome over the actions of playing slot machines. You’ll end up thinking that the value of the casino is the value of the jackpot…and decide that the casino is a great place to be!*

 Sometimes the largest Q-value of a state is large *just by chance*; choosing it over others makes the value estimate biased.
 One way to reduce this bias is to have a friend visit the casino and play the same set of slot machines. Then, ask them what their winnings were at machine 43 and use their response as your value estimate. It’s not likely that you both won the jackpot on the same machine, so this time you won’t end up with an over-optimistic estimate. We call this approach *Double Q-learning*.

**Putting it together.**  It’s easy to think of Sarsa, Expected Sarsa, Q-learning, and Double Q-learning as different algorithms. But as we’ve seen, they are simply different ways of estimating V(st+1)V(s\_{t+1})V(st+1) in a TD update.

#### On-policy methods

**Sarsa** uses the Q-value associated with at+1a\_{t+1}at+1 to estimate the next state’s value.

![](figures/sarsa.svg)

**Expected Sarsa** uses an expectation over Q-values to estimate the next state’s value.

![](figures/expected-sarsa.svg)
#### Off-policy methods

**Off-policy value learning** weights Q-values by an arbitrary policy.

![](figures/off-policy.svg)

**Q-learning** estimates value under the optimal policy by choosing the max Q-value.

![](figures/q-learning.svg)

**Double Q-learning** selects the best action with QAQ\_AQA and then estimates the value of that action with
 QBQ\_BQB.

![](figures/double-q-learning.svg)

 The intuition behind all of these approaches is that they re-weight path intersections.

**Re-weighting paths with Monte Carlo.** At this point, a natural question is: Could we accomplish the same re-weighting effect with Monte Carlo? We could, but it would be messier and involve re-weighting all of the agent’s experience. By working at intersections, TD learning re-weights individual transitions instead of episodes as a whole. This makes TD methods much more convenient for off-policy learning.

Merging Paths with Function Approximators
-----------------------------------------

 Up until now, we’ve learned one parameter — the value estimate — for every state or every state-action pair. This works well for the Cliff World example because it has a small number of states. But most interesting RL problems have a large or infinite number of states. This makes it hard to store value estimates for each state.

 #figure-fnapprox-intro {
 display: grid;
 grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
 grid-column-gap: 20px;
 grid-row-gap: 10px;
 /\* grid-auto-flow: column; \*/
 }

![](figures/large-cliffworld-states.svg)

**Large or infinite state spaces** are a characteristic of many interesting RL problems. Value estimation
 in these spaces often requires function approximation.

![](figures/large-cliffworld-path.svg)

**Tabular value functions** keep value estimates for each individual state. They consume a great deal of
 memory and don’t generalize.

![](figures/large-cliffworld-approx.svg)

**Euclidean averagers** – which are a type of function approximator – save memory and let agents
 generalize to states they haven’t visited yet.

 Instead, we must force our value estimator to have fewer parameters than there are states. We can do this with machine learning methods such as linear regression, decision trees, or neural networks. All of these methods fall under the umbrella of function approximation.

**Merging nearby paths.** From the paths perspective, we can interpret function approximation as a way of merging nearby paths. But what do we mean by “nearby”? In the figure above, we made an implicit decision to measure “nearby” with Euclidean distance. This was a good idea because the Euclidean distance between two states is highly correlated with the probability that the agent will transition between them.

 However, it’s easy to imagine cases where this implicit assumption breaks down. By adding a single long barrier, we can construct a case where the Euclidean distance metric leads to bad generalization. The problem is that we have merged the wrong paths.

 #fnapprox-barrier .wrapper {
 margin-left: auto;
 margin-right: auto;
 display: grid;
 grid-template-columns: 1fr 1.464fr;
 grid-template-rows: auto auto;
 grid-column-gap: 20px;
 grid-row-gap: 10px;
 grid-auto-flow: column;
 }

![](figures/large-cliffworld-barrier-intro.svg)

 Imagine changing the Cliff World setup by adding a long barrier.

![](figures/large-cliffworld-barrier.svg)

 Now, using the Euclidean averager leads to bad value updates.

**Merging the wrong paths.** The diagram below shows the effects of merging the wrong paths a bit more explicitly. Since the Euclidean averager is to blame for poor generalization, both Monte Carlo and TD make bad value updates. However, TD learning amplifies these errors dramatically whereas Monte Carlo does not.

![](figures/compare-function-approx.svg)

 We’ve seen that TD learning makes more efficient value updates. The price we pay is that these updates end up being much more sensitive to bad generalization.

Implications for deep reinforcement learning
--------------------------------------------

**Neural networks.** Deep neural networks are perhaps the most popular function approximators for reinforcement learning. These models are exciting for many reasons, but one particularly nice property is that they don’t make implicit assumptions about which states are “nearby.”

 Early in training, neural networks, like averagers, tend to merge the wrong paths of experience. In the Cliff Walking example, an untrained neural network might make the same bad value updates as the Euclidean averager.

 But as training progresses, neural networks can actually learn to overcome these errors. They learn which states are “nearby” from experience. In the Cliff World example, we might expect a fully-trained neural network to have learned that value updates to states *above* the barrier should never affect the values of states *below* the barrier. This isn’t something that most other function approximators can do. It’s one of the reasons deep RL is so interesting!

![](figures/latent-distance.png)

 A distance metric learned by a neural network . **Lighter blue →\rightarrow→ more distant**. The agent, which was trained to grasp objects using the robotic arm, takes into account obstacles and arm length when it measures the distance between two states.

**TD or not TD?** So far, we’ve seen how TD learning can outperform Monte Carlo by merging paths of experience where they intersect. We’ve also seen that merging paths is a double-edged sword: when function approximation causes bad value updates, TD can end up doing worse.

 Over the last few decades, most work in RL has preferred TD learning to Monte Carlo. Indeed, many approaches to RL use TD-style value updates. With that being said, there are many other ways to use Monte Carlo for reinforcement learning. Our discussion centers around Monte Carlo for value estimation in this article, but it can also be used for policy selection as in Silver et al.

 Since Monte Carlo and TD learning both have desirable properties, why not try building a value estimator that is a mixture of the two? That’s the reasoning behind TD(λ\lambdaλ) learning. It’s a technique that simply interpolates (using the coefficient λ\lambdaλ) between Monte Carlo and TD updatesIn the limit λ=0\lambda=0λ=0, we recover the TD update rule. Meanwhile, when λ=1\lambda=1λ=1, we recover Monte Carlo.. Often, TD(λ\lambdaλ) works better than either Monte Carlo or TD learning aloneResearchers often keep the λ\lambdaλ coefficient constant as they train a deep RL model. However, if we think Monte Carlo learning is best early in training (before the agent has learned a good state representation) and TD learning is best later on (when it’s easier to benefit from merging paths), maybe the best approach is to anneal λ\lambdaλ over the course of training..

Conclusion
----------

 In this article we introduced a new way to think about TD learning. It helps us see why TD learning can be beneficial, why it can be effective for off-policy learning, and why there can be challenges in combining TD learning with function approximators.

 We encourage you to use the playground below to build on these intuitions, or to try an experiment of your own.

#### Gridworld playground

A closer look at how Temporal Difference Learning merges paths of experience for greater statistical efficiency

Reinforcement Learning: An Introduction

Double Q-learning

Universal Planning Networks


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'

On May 6th, Andrew Ilyas and colleagues [published a paper](http://gradientscience.org/adv/)
 outlining two sets of experiments.
 Firstly, they showed that models trained on adversarial examples can transfer to real data,
 and secondly that models trained on a dataset derived from the representations of robust neural networks
 seem to inherit non-trivial robustness.
 They proposed an intriguing interpretation for their results:
 adversarial examples are due to “non-robust features” which are highly predictive but imperceptible to
 humans.

 The paper was received with intense interest and discussion
 on social media, mailing lists, and reading groups around the world.
 How should we interpret these experiments?
 Would they replicate?
 Adversarial example research is particularly vulnerable to a certain kind of non-replication among
 disciplines of machine learning,
 because it requires researchers to play both attack and defense.
 It’s easy for even very rigorous researchers to accidentally use a weak attack.
 However, as we’ll see, Ilyas et al’s results have held up to initial scrutiny.

 And if non-robust features exist… what are they?

 To explore these questions, Distill decided to run an experimental “discussion article.”
 Running a discussion article is something Distill has wanted to try for several years.
 It was originally suggested to us by Ferenc Huszár, who writes many lovely discussions of papers on [his blog](https://www.inference.vc/).

 Why not just have everyone write private blog posts like Ferenc?
 Distill hopes that providing a more organized forum for many people to participate
 can give more researchers license to invest energy in discussing other’s work
 and make sure there’s an opportunity for all parties to comment and respond before the final version is
 published.

 We invited a number of researchers
 to write comments on the paper and organized discussion and responses from the original authors.

 The Machine Learning community
 [sometimes](https://www.machinelearningdebates.com/program)
[worries](https://medium.com/syncedreview/cvpr-paper-controversy-ml-community-reviews-peer-review-79bf49eb0547)
 that peer review isn’t thorough enough.
 In contrast to this, we were struck by how deeply respondents engaged.
 Some respondents literally invested weeks in replicating results, running new experiments, and thinking
 deeply about the original paper.
 We also saw respondents update their views on non-robust features as they ran experiments — sometimes back
 and forth!
 The original authors similarly deeply engaged in discussing their results, clarifying misunderstandings, and
 even running new experiments in response to comments.

 We think this deep engagement and discussion is really exciting, and hope to experiment with more such
 discussion articles in the future.

Discussion Themes
-----------------

**Clarifications**:
 Discussion between the respondents and original authors was able
 to surface several misunderstandings or opportunities to sharpen claims.
 The original authors summarize this in their rebuttal.

**Successful Replication**:
 Respondents successfully reproduced many of the experiments in Ilyas et al  and had no unsuccessful replication attempts.
 This was significantly facilitated by the release of code, models, and datasets by the original authors.
 Gabriel Goh and Preetum Nakkiran both independently reimplemented and replicated
 the non-robust dataset experiments.
 Preetum reproduced the D^det\widehat{\mathcal{D}}\_{det}D
det non-robust dataset experiment as described in the
 paper, for L∞L\_\inftyL∞ and L2L\_2L2 attacks.

 Gabriel repproduced both D^det\widehat{\mathcal{D}}\_{det}D
det and D^rand\widehat{\mathcal{D}}\_{rand}D
rand for L2L\_2L2
 attacks.

 Preetum also replicated part of the robust dataset experiment by
 training models on the provided robust dataset and finding that they seemed non-trivially robust.
 It seems epistemically notable that both Preetum and Gabriel were initially skeptical.
 Preetum emphasizes that he found it easy to make the phenomenon work and that it was robust to many variants
 and hyperparameters he tried.

**Exploring the Boundaries of Non-Robust Transfer**:
 Three of the comments focused on variants of the “non-robust dataset” experiment,
 where training on adversarial examples transfers to real data.
 When, how, and why does it happen?
 Gabriel Goh explores an alternative mechanism for the results,
 Preetum Nakkiran shows a special construction where it doesn’t happen,
 and Eric Wallace shows that transfer can happen for other kinds of incorrectly labeled data.

**Properties of Robust and Non-Robust Features**:
 The other three comments focused on the properties of robust and non-robust models.
 Gabriel Goh explores what non-robust features might look like in the case of linear models,
 while Dan Hendrycks and Justin Gilmer discuss how the results relate to the broader problem of robustness to
 distribution shift,
 and Reiichiro Nakano explores the qualitative differences of robust models in the context of style transfer.

Comments
--------

 Distill collected six comments on the original paper.
 They are presented in alphabetical order by the author’s last name,
 with brief summaries of each comment and the corresponding response from the original authors.

###
[Adversarial Example Researchers Need to Expand What is Meant by
 “Robustness”](response-1/)

### Authors

### Affiliations

[Justin Gilmer](https://www.linkedin.com/in/jmgilmer)

[Google Brain Team](https://g.co/brain)

[Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/)

[UC Berkeley](https://www.berkeley.edu/)

 Justin and Dan discuss “non-robust features” as a special case
 of models being non-robust because they latch on to superficial correlations,
 a view often found in the distributional robustness literature.
 As an example, they discuss recent analysis of how neural networks behave in frequency space.
 They emphasize we should think about a broader notion of robustness.
 [Read Full Article](response-1/)

#### Comment from original authors:

 The demonstration of models that learn from only high-frequency components of the data is
 an interesting finding that provides us with another way our models can learn from data that
 appears “meaningless” to humans.
 The authors fully agree that studying a wider notion of robustness will become increasingly
 important in ML, and will help us get a better grasp of features we actually want our models
 to rely on.

###
[Robust Feature Leakage](response-2/)

### Authors

### Affiliations

[Gabriel Goh](https://gabgoh.github.io)

[OpenAI](https://openai.com)

 Gabriel explores an alternative mechanism that could contribute to the non-robust transfer
 results.
 He establishes a lower-bound showing that this mechanism contributes a little bit to the
 D^rand\widehat{\mathcal{D}}\_{rand}D
rand experiment,
 but finds no evidence for it effecting the D^det\widehat{\mathcal{D}}\_{det}D
det experiment.
 [Read Full Article](response-2/)

#### Comment from original authors:

 This is a nice in-depth investigation that highlights (and neatly visualizes) one of the
 motivations for designing the D^det\widehat{\mathcal{D}}\_{det}D
det dataset.

###
[Two Examples of Useful, Non-Robust Features](response-3/)

### Authors

### Affiliations

[Gabriel Goh](https://gabgoh.github.io)

[OpenAI](https://openai.com)

 Gabriel explores what non-robust useful features might look like in the linear case.
 He provides two constructions:
 “contaminated” features which are only non-robust due to a non-useful feature being mixed in,
 and “ensembles” that could be candidates for true useful non-robust features.
 [Read Full Article](response-3/)

#### Comment from original authors:

 These experiments with linear models are a great first step towards visualizing non-robust
 features for real datasets (and thus a neat corroboration of their existence).
 Furthermore, the theoretical construction of “contaminated” non-robust features opens an
 interesting direction of developing a more fine-grained definition of features.

###
[Adversarially Robust Neural Style Transfer](response-4/)

### Authors

###

[Reiichiro Nakano](https://reiinakano.com/)

 Reiichiro shows that adversarial robustness makes neural style transfer
 work by default on a non-VGG architecture.
 He finds that matching robust features makes style transfer’s outputs look perceptually better
 to humans.
 [Read Full Article](response-4/)

#### Comment from original authors:

 Very interesting results that highlight the potential role of non-robust features and the
 utility of robust models for downstream tasks. We’re excited to see what kind of impact robustly
 trained models will have in neural network art!
 Inspired by these findings, we also take a deeper dive into (non-robust) VGG, and find some
 interesting links between robustness and style transfer.

###
[Adversarial Examples are Just Bugs, Too](response-5/)

### Authors

### Affiliations

[Preetum Nakkiran](https://preetum.nakkiran.org/)

[OpenAI](https://openai.com) &
 [Harvard University](https://www.harvard.edu/)

 Preetum constructs a family of adversarial examples with no transfer to real data,
 suggesting that some adversarial examples are “bugs” in the original paper’s framing.
 Preetum also demonstrates that adversarial examples can arise even if the underlying distribution
 has no “non-robust features”.
 [Read Full Article](response-5/)

#### Comment from original authors:

 A fine-grained look at adversarial examples that neatly our thesis (i.e. that non-robust
 features exist and adversarial examples arise from them, see Takeaway #1) while providing an
 example of adversarial examples that arise from “bugs”.
 The fact that the constructed “bugs”-based adversarial examples don’t transfer constitutes
 another evidence for the link between transferability and (non-robust) features.

###
[Learning from Incorrectly Labeled Data](response-6/)

### Authors

### Affiliations

[Eric Wallace](https://www.ericswallace.com/)

[Allen Institute for AI](https://allenai.org/)

 Eric shows that training on a model’s training errors,
 or on how it predicts examples form an unrelated dataset,
 can both transfer to the true test set.
 These experiments are analogous to the original paper’s non-robust transfer results — all three results are examples of a kind of “learning from incorrectly labeled data.”
 [Read Full Article](response-6/)

#### Comment from original authors:

 These experiments are a creative demonstration of the fact that the underlying phenomenon of
 learning features from “human-meaningless” data can actually arise in a broad range of
 settings.

Original Author Discussion and Responses
----------------------------------------

###
[Discussion and Author Responses](original-authors/)

### Authors

### Affiliations

[Logan Engstrom](http://loganengstrom.com/),
 [Andrew Ilyas](http://andrewilyas.com/),
 [Aleksander Madry](https://people.csail.mit.edu/madry/),
 [Shibani Santurkar](http://people.csail.mit.edu/shibani/),
 Brandon Tran,
 [Dimitris Tsipras](http://people.csail.mit.edu/tsipras/)

MIT

 The original authors describe their takeaways and some clarifcations that resulted from the
 conversation.
 This article also contains their responses to each comment.
 [Read Full Article](original-authors/)

Logan Engstrom
Justin Gilmer
Gabriel Goh
Dan Hendrycks
Andrew Ilyas
Aleksander Madry
Reiichiro Nakano
Shibani Santurkar
Dimitris Tsipras
Eric Wallace
Justin Gilmer
Dan Hendrycks
Gabriel Goh
Gabriel Goh
Reiichiro Nakano
Preetum Nakkiran
Eric Wallace
Logan Engstrom
Andrew Ilyas
Aleksander Madry
Shibani Santurkar
Dimitris Tsipras

Six comments from the community and responses from the original authors

Adversarial examples are not bugs, they are features


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Example Researchers Need to Expand What is Meant by 'Robustness'

#rebuttal,
 .comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[Other Comments](/2019/advex-bugs-discussion/#commentaries)
[Comment by Ilyas et al.](#rebuttal)

 The hypothesis in Ilyas et. al. is a special case of a more general principle that is well accepted in the
 distributional robustness literature — models lack robustness to distribution shift because they latch onto
 superficial correlations in the data. Naturally, the same principle also explains adversarial examples
 because they arise from a worst-case analysis of distribution shift. To obtain a more complete understanding
 of robustness, adversarial example researchers should connect their work to the more general problem of
 distributional robustness rather than remaining solely fixated on small gradient perturbations.

Detailed Response
-----------------

 The main hypothesis in Ilyas et al. (2019) happens to be a special case of a more general principle that is
 commonly accepted in the robustness to distributional shift literature
: a model’s lack of
 robustness is largely because the model latches onto superficial statistics in the data. In the image
 domain, these statistics may be unused by — and unintuitive to — humans, yet they may be useful for
 generalization in i.i.d. settings. Separate experiments eschewing gradient perturbations and studying
 robustness beyond adversarial perturbations show similar results. For example, a recent work
 demonstrates that models can generalize to the test examples by learning from high-frequency information
 that is both naturally occurring and also inconspicuous. Concretely, models were trained and tested with an
 extreme high-pass filter applied to the data. The resulting high-frequency features appear completely
 grayscale to humans, yet models are able to achieve 50% top-1 accuracy on ImageNet-1K solely from these
 natural features that usually are “invisible.” These hard-to-notice features can be made conspicuous by
 normalizing the filtered image to have unit variance pixel statistics in the figure below.

![](images/figure-1-cropped.png)

[1](#figure-1)
 Models can achieve high accuracy using information from the input that would be unrecognizable
 to humans. Shown above are models trained and tested with aggressive high and low pass filtering applied
 to the inputs. With aggressive low-pass filtering, the model is still above 30% on ImageNet when the
 images appear to be simple globs of color. In the case of high-pass (HP) filtering, models can achieve
 above 50% accuracy using features in the input that are nearly invisible to humans. As shown on the
 right hand side, the high pass filtered images needed be normalized in order to properly visualize the
 high frequency features.

 Given the plethora of useful correlations that exist in natural data, we should expect that our models will
 learn to exploit them. However, models relying on superficial statistics can poorly generalize should these
 same statistics become corrupted after deployment. To obtain a more complete understanding of model
 robustness,  measured test error after perturbing every image in the test set by a
 Fourier basis vector,
 as shown in Figure 2. The naturally trained model is robust to low-frequency perturbations, but,
 interestingly, lacks robustness in the mid to high frequencies. In contrast, adversarial training improves
 robustness to mid- and high-frequency perturbations, while sacrificing performance on low frequency
 perturbations. For instance adversarial training degrades performance on the low-frequency fog corruption
  from 85.7% to 55.3%. Adversarial training similarly degrades robustness to
 contrast and low-pass
 filtered noise. By taking a broader view of robustness beyond tiny ℓp\ell\_pℓp norm perturbations, we discover
 that adversarially trained models are actually not “robust.” They are instead biased towards different kinds
 of superficial statistics. As a result, adversarial training can sacrifice robustness in real-world
 settings.

![](images/figure-2-cropped.png)

[2](#figure-2)
 Model sensitivity to additive noise aligned with different Fourier basis vectors on CIFAR-10.
 We fix the additive noise to have ℓ2\ell\_2ℓ2 norm 4 and evaluate three models: a naturally trained model,
 an
 adversarially trained model, and a model trained with Gaussian data augmentation. Error rates are
 averaged over 1000 randomly sampled images from the test set. In the bottom row we show images perturbed
 with noise along the corresponding Fourier basis vector. The naturally trained model is highly sensitive
 to additive noise in all but the lowest frequencies. Both adversarial training and Gaussian data
 augmentation dramatically improve robustness in the higher frequencies while sacrificing the robustness
 of the naturally trained model in the lowest frequencies (i.e. in both models, blue area in the middle
 is smaller compared to that of the naturally trained model).

 How, then, can the research community create models that robustly generalize in the real world, given that
 adversarial training can harm robustness to distributional shift? To do so, the research community must take
 a broader view of robustness and accept that ℓp\ell\_pℓp adversarial robustness is highly limited and mostly
 detached from security and real-world robustness . While often thought an
 idiosyncratic quirk of deep
 neural network classifiers, adversarial examples are not a counterintuitive mystery plaguing otherwise
 superhuman classifiers. Instead, adversarial examples are in fact expected of models which lack robustness
 to noise . They should not be surprising given the brittleness observed in
 numerous synthetic — and even
 natural  — conditions. Models reliably exhibit poor performance when they are
 evaluated on distributions
 slightly different from the training distribution. For all that, current benchmarks do not expose these
 failure modes. The upshot is that we need to design harder and more diverse test sets, and we should not
 continue to be singularly fixated on studying specific gradient perturbations. As we move forward in
 robustness research, we should focus on the various ways in which models are fragile, and design more
 comprehensive benchmarks accordingly . As long as models lack
 robustness to
 distributional shift, there will always be errors to find adversarially.

 To cite Ilyas et al.’s response, please cite their
 [collection of responses](/2019/advex-bugs-discussion/original-authors/#citation).

**Response Summary**: The demonstration of models that learn from
 high-frequency components of the data is interesting and nicely aligns with our
 findings. Now, even though susceptibility to noise could indeed arise from
 non-robust useful features, this kind of brittleness (akin to adversarial examples)
 of ML models has been so far predominantly viewed as a consequence of model
 “bugs” that will be eliminated by “better” models. Finally, we agree that our
 models need to be robust to a much broader set of perturbations — expanding the
 set of relevant perturbations will help identify even more non-robust features
 and further distill the useful features we actually want our models to rely on.

**Response**: The fact that models can learn to classify correctly based
 purely on the high-frequency component of the training set is neat! This nicely
 complements one of our [takeaways](/2019/advex-bugs-responses/rebuttal/#takeaway1): models
 will rely on useful features even if these features appear incomprehensible to humans.

 Also, while non-robustness to noise can be an indicator of models using
 non-robust useful features, this is not how the phenomenon was predominantly viewed.
 More often than not, the brittleness of ML models to noise was instead regarded
 as an innate shortcoming of the models, e.g., due to poor margins. (This view is
 even more prevalent in the adversarial robustness community.) Thus, it was often
 expected that progress towards “better”/”bug-free” models will lead to them
 being more robust to noise and adversarial examples.

 Finally, we fully agree that the set of LpL\_pLp-bounded perturbations is a very
 small subset of the perturbations we want our models to be robust to. Note,
 however, that the focus of our work is human-alignment — to that end, we
 demonstrate that models rely on features sensitive to patterns that are
 imperceptible to humans. Thus, the existence of other families of
 incomprehensible but useful features would provide even more support for our
 thesis — identifying and characterizing such features is an interesting area for
 future research.

 You can find more responses in the  [main discussion article](/2019/advex-bugs-discussion/).

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Measuring the tendency of CNNs to Learn Surface Statistical Regularities

Nightmare at Test Time: Robust Learning by Feature Deletion

A Robust Minimax Approach to Classification

Generalisation in humans and deep neural networks

A Fourier Perspective on Model Robustness in Computer Vision

Motivating the Rules of the Game for Adversarial Example Research

Adversarial Examples Are a Natural Consequence of Test Error in Noise

Robustness of classifiers: from adversarial to random noise

Natural Adversarial Examples

{MNIST-C:} {A} Robustness Benchmark for Computer Vision

{NICO:} {A} Dataset Towards Non-I.I.D. Image Classification

Do ImageNet Classifiers Generalize to ImageNet?

The Elephant in the Room

Using Videos to Evaluate Image Model Robustness


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Robust Feature Leakage

#rebuttal,
 .comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[Other Comments](/2019/advex-bugs-discussion/#commentaries)
[Comment by Ilyas et al.](#rebuttal)

 Ilyas et al.  report a surprising result: a model trained on
 adversarial examples is effective on clean data. They suggest this transfer is driven by adverserial
 examples containing geuinely useful non-robust cues. But an alternate mechanism for the transfer could be a
 kind of “robust feature leakage” where the model picks up on faint robust cues in the attacks.

 We show that at least 23.5% (out of 88%) of the accuracy can be explained by robust features in
 DrandD\_\text{rand}Drand. This is a weak lower bound, established by a linear model, and does not perclude the
 possibility of further leakage. On the other hand, we find no evidence of leakage in DdetD\_\text{det}Ddet.

### Lower Bounding Leakage

 Our technique for quantifying leakage consisting of two steps:

1. First, we construct features fi(x)=wiTxf\_i(x) = w\_i^Txfi(x)=wiTx that are provably robust, in a sense we will soon
 specify.
2. Next, we train a linear classifier as per ,
 Equation 3 on the datasets D^det\hat{\mathcal{D}}\_{\text{det}}D^det and
 D^rand\hat{\mathcal{D}}\_{\text{rand}}D^rand (Defined , Table 1) on
 these robust features *only*.

 Since Ilyas et al.  only specify robustness in the two class
 case, we propose two possible specifications for what constitutes a *robust feature* in the multiclass
 setting:

 **Specification 1**
For at least one of the
 classes, the feature is γ\gammaγ-robustly useful   with
 γ=0\gamma = 0γ=0, and the set of valid perturbations equal to an L2L\_2L2 norm ball with radius 0.25.

 **Specification 2**

 The feature comes from a robust model for which at least 80% of points in the test set have predictions
 that remain static in a neighborhood of radius 0.25 on the L2L\_2L2 norm ball.

 We find features that satisfy *both* specifications by using the 10 linear features of a robust linear
 model trained on CIFAR-10. Because the features are linear, the above two conditions can be certified
 analytically. We leave the reader to inspect the weights corresponding to the features manually:

 10 Features, FCF\_CFC, of robust linear classifier CCC. Each feature is γi\gamma\_iγi-robustly-useful with
 respect to label iii. Visualized are the weights wiw\_iwi of features fi(x)=wiTxf\_i(x) = w\_i^Txfi(x)=wiTx.

 Training a linear model on the above robust features on D^rand\hat{\mathcal{D}}\_{\text{rand}}D^rand and testing on the
 CIFAR test set incurs an accuracy of **23.5%** (out of 88%). Doing the same on
 D^det\hat{\mathcal{D}}\_{\text{det}}D^det incurs an accuracy of **6.81%** (out of 44%).

 The contrasting results suggest that the the two experiements should be interpreted differently. The
 transfer results of D^rand\hat{\mathcal{D}}\_{\text{rand}}D^rand in Table 1 of
 should approached with caution: A non-trivial portion of the accuracy can be attributed to robust
 features. Note that this bound is weak: this bound could be possibly be improved if we used nonlinear
 features, e.g. from a robust deep neural network.

 The results of D^det\hat{\mathcal{D}}\_{\text{det}}D^det in Table 1 of
 however, are on stronger footing. We find no evidence of feature leakage (in fact, we find negative leakage — an influx!). We thus conclude that it is plausible the majority of the accuracy is driven by
 non-robust features, exactly the thesis of .

 To cite Ilyas et al.’s response, please cite their
 [collection of responses](/2019/advex-bugs-discussion/original-authors/#citation).

**Response Summary**: This
 is a valid concern that was actually one of our motivations for creating the
 D^det\widehat{\mathcal{D}}\_{det}D
det dataset (which, as the comment notes, actually
 has *misleading* robust features). The provided experiment further
 improves our understanding of the underlying phenomenon.
**Response**: This comment raises a valid concern which was in fact one of
 the primary reasons for designing the D^det\widehat{\mathcal{D}}\_{det}D
det dataset.
 In particular, recall the construction of the D^rand\widehat{\mathcal{D}}\_{rand}D
rand
 dataset: assign each input a random target label and do PGD towards that label.
 Note that unlike the D^det\widehat{\mathcal{D}}\_{det}D
det dataset (in which the
 target class is deterministically chosen), the D^rand\widehat{\mathcal{D}}\_{rand}D
rand
 dataset allows for robust features to actually have a (small) positive
 correlation with the label.

To see how this can happen, consider the following simple setting: we have a
 single feature f(x)f(x)f(x) that is 111 for cats and −1-1−1 for dogs. If ϵ=0.1\epsilon = 0.1ϵ=0.1
 then f(x)f(x)f(x) is certainly a robust feature. However, randomly assigning labels
 (as in the dataset D^rand\widehat{\mathcal{D}}\_{rand}D
rand) would make this feature
 uncorrelated with the assigned label, i.e., we would have that E[f(x)⋅y]=0E[f(x)\cdot y] = 0E[f(x)⋅y]=0. Performing a
 targeted attack might in this case induce some correlation with the
 assigned label, as we could have E[f(x+η⋅∇f(x))⋅y]>E[f(x)⋅y]=0\mathbb{E}[f(x+\eta\cdot\nabla
 f(x))\cdot y] > \mathbb{E}[f(x)\cdot y] = 0E[f(x+η⋅∇f(x))⋅y]>E[f(x)⋅y]=0, allowing a model to learn
 to correctly classify new inputs.

In other words, starting from a dataset with no features, one can encode
 robust features within small perturbations. In contrast, in the
 D^det\widehat{\mathcal{D}}\_{det}D
det dataset, the robust features are *correlated
 with the original label* (since the labels are permuted) and since they are
 robust, they cannot be flipped to correlate with the newly assigned (wrong)
 label. Still, the D^rand\widehat{\mathcal{D}}\_{rand}D
rand dataset enables us to show
 that (a) PGD-based adversarial examples actually alter features in the data and
 (b) models can learn from human-meaningless/mislabeled training data. The
 D^det\widehat{\mathcal{D}}\_{det}D
det dataset, on the other hand, illustrates that the
 non-robust features are actually sufficient for generalization and can be
 preferred over robust ones in natural settings.

The experiment put forth in the comment is a clever way of showing that such
 leakage is indeed possible. However, we want to stress (as the comment itself
 does) that robust feature leakage does *not* have an impact on our main
 thesis — the D^det\widehat{\mathcal{D}}\_{det}D
det dataset explicitly controls
 for robust
 feature leakage (and in fact, allows us to quantify the models’ preference for
 robust features vs non-robust features — see Appendix D.6 in the
 [paper](https://arxiv.org/abs/1905.02175)).

 You can find more responses in the  [main discussion article](/2019/advex-bugs-discussion/).

An example project using webpack and svelte-loader and ejs to inline SVGs


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Two Examples of Useful, Non-Robust Features

#rebuttal,
 .comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[Other Comments](/2019/advex-bugs-discussion/#commentaries)
[Comment by Ilyas et al.](#rebuttal)

 Ilyas et al.  define a *feature* as a function fff that
 takes xxx from the *data distribution* (x,y)∼D(x,y) \sim \mathcal{D}(x,y)∼D into a real number, restricted to have
 mean zero and unit variance. A feature is said to be *useful* if it has high correlation with the
 label. But in the presence of an adversary Ilyas et al.  argues
 the metric that truly matters is a feature’s *robust usefulness*,

E[inf∥δ∥≤ϵyf(x+δ)],
 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right],
 E[∥δ∥≤ϵinfyf(x+δ)],

 its correlation with the label while under attack. Ilyas et al.
 suggests that in addition to the pedestrian, robust features we know and love (such as the color of the
 sky), our models may also be taking advantage of useful, non-robust features, some of which may even lie
 beyond the threshold of human intuition. This begs the question: what might such non-robust features look
 like?

### Non-Robust Features in Linear Models

 Our search is simplified when we realize the following: non-robust features are not unique to the complex,
 nonlinear models encountered in deep learning. As Ilyas et al
 observe, they arise even in the humblest of models — the linear one. Thus, we restrict our attention
 to linear features of the form:

f(x)=aTx∥a∥ΣwhereΣ=E[xxT]andE[x]=0.f(x) = \frac{a^Tx}{\|a\|\_\Sigma}\qquad \text{where} \qquad \Sigma = \mathbf{E}[xx^T] \quad
 \text{and} \quad \mathbf{E}[x] = 0.
 f(x)=∥a∥ΣaTxwhereΣ=E[xxT]andE[x]=0.

 The robust usefulness of a linear feature admits an elegant decomposition
  This
 E[inf∥δ∥≤ϵyf(x+δ)]=E[yf(x)+inf∥δ∥≤ϵyf(δ)]=E[yf(x)+inf∥δ∥≤ϵyaTδ∥a∥Σ]=E[yf(x)+inf∥δ∥≤ϵaTδ∥a∥Σ]=E[yf(x)]−ϵ∥a∥∗∥a∥Σ
 \begin{aligned}
 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right] &
 =\mathbf{E}\left[yf(x)+\inf\_{\|\delta\|\leq\epsilon}yf(\delta)\right]\\
 & =\mathbf{E}\left[yf(x)+\inf\_{\|\delta\|\leq\epsilon}y\frac{a^{T}\delta}{\|a\|\_{\Sigma}}\right]\\
 &
 =\mathbf{E}\left[yf(x)+\frac{\inf\_{\|\delta\|\leq\epsilon}a^{T}\delta}{\|a\|\_{\Sigma}}\right]=\mathop{\mathbf{E}[yf(x)]}-\epsilon\frac{\|a\|\_{\*}}{\|a\|\_{\Sigma}}
 \end{aligned}
 E[∥δ∥≤ϵinfyf(x+δ)]=E[yf(x)+∥δ∥≤ϵinfyf(δ)]=E[yf(x)+∥δ∥≤ϵinfy∥a∥ΣaTδ]=E[yf(x)+∥a∥Σinf∥δ∥≤ϵaTδ]=E[yf(x)]−ϵ∥a∥Σ∥a∥∗
 into two terms:

 .undomargin {
 position: relative;
 left: -1em;
 top: 0.2em;
 }

E[inf∥δ∥≤ϵyf(x+δ)]
 \mathbf{E}\left[\inf\_{\|\delta\|\leq\epsilon}yf(x+\delta)\right]
 E[∥δ∥≤ϵinfyf(x+δ)]

===

E[yf(x)]\mathop{\mathbf{E}[yf(x)]}E[yf(x)]

−-−

ϵ∥a∥∗∥a∥Σ\epsilon\frac{\|a\|\_{\*}}{\|a\|\_{\Sigma}}ϵ∥a∥Σ∥a∥∗

 The robust usefulness of a feature

 the correlation of the feature with the label

 the feature’s non-robustness

 In the above equation ∥⋅∥∗\|\cdot\|\_\*∥⋅∥∗ deontes the dual norm of ∥⋅∥\|\cdot\|∥⋅∥.
 This decomposition gives us an instrument for visualizing any set of linear features aia\_iai in a two
 dimensional plot.

 Plotted below is the binary classification task of separating *truck* and *frog* in CIFAR-10 on
 the set of features aia\_iai corresponding to the ithi^{th}ith singular vector of the data.

 The elusive non-robust useful features, however, seem conspicuously absent in the above plot.
 Fortunately, we can construct such features by strategically combining elements of this basis.

 We demonstrate two constructions:

 It is surprising, thus, that the experiments of Madry et al.
 (with deterministic perturbations) *do* distinguish between the non-robust useful
 features generated from ensembles and containments. A succinct definition of a robust feature that peels
 these two worlds apart is yet to exist, and remains an open problem for the machine learning community.

 To cite Ilyas et al.’s response, please cite their
 [collection of responses](/2019/advex-bugs-discussion/original-authors/#citation).

**Response Summary**: The construction of explicit non-robust features is
 very interesting and makes progress towards the challenge of visualizing some of
 the useful non-robust features detected by our experiments. We also agree that
 non-robust features arising as “distractors” is indeed not precluded by our
 theoretical framework, even if it is precluded by our experiments.
 This simple theoretical framework sufficed for reasoning about and
 predicting the outcomes of our experiments
 We also presented a theoretical setting where we can
 analyze things fully rigorously in Section 4 of our paper..
 However, this comment rightly identifies finding a more comprehensive
 definition of feature as an important future research direction.

**Response**: These experiments (visualizing the robustness and
 usefulness of different linear features) are very interesting! They both further
 corroborate the existence of useful, non-robust features and make progress
 towards visualizing what these non-robust features actually look like.

We also appreciate the point made by the provided construction of non-robust
 features (as defined in our theoretical framework) that are combinations of
 useful+robust and useless+non-robust features. Our theoretical framework indeed
 enables such a scenario, even if — as the commenter already notes — our
 experimental results do not. (In this sense, the experimental results and our [main takeaway](/2019/advex-bugs-discussion/rebuttal/#takeaway1)  are actually stronger than our theoretical
 framework technically captures.) Specifically, in such a scenario, during the
 construction of the D^det\widehat{\mathcal{D}}\_{det}D
det dataset, only the non-robust
 and useless term of the feature would be flipped. Thus, a classifier trained on
 such a dataset would associate the predictive robust feature with the
 *wrong* label and would thus not generalize on the test set. In contrast,
 our experiments show that classifiers trained on D^det\widehat{\mathcal{D}}\_{det}D
det
 do generalize.

Overall, our focus while developing our theoretical framework was on
 enabling us to formally describe and predict the outcomes of our experiments. As
 the comment points out, putting forth a theoretical framework that captures
 non-robust features in a very precise way is an important future research
 direction in itself.

 You can find more responses in the  [main discussion article](/2019/advex-bugs-discussion/).


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarially Robust Neural Style Transfer

#rebuttal,
 .comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[Other Comments](/2019/advex-bugs-discussion/#commentaries)
[Comment by Ilyas et al.](#rebuttal)

 A figure in Ilyas, et. al. that struck me as particularly
 interesting
 was the following graph showing a correlation between adversarial transferability between architectures and
 their
 tendency to learn similar non-robust features.

![](images/transferability.png)

 Adversarial transferability vs test accuracy of different architectures trained on ResNet-50′s
 non-robust features.

 One way to interpret this graph is that it shows how well a particular architecture is able to capture
 non-robust features in an image.
 Since the non-robust features are defined by the non-robust features ResNet-50 captures,
 NRFresnetNRF\_{resnet}NRFresnet, what this graph really shows is how well an architecture captures NRFresnetNRF\_{resnet}NRFresnet.

 Notice how far back VGG is compared to the other models.

 In the unrelated field of neural style transfer, VGG-based neural networks are also quite special since non-VGG architectures are
 known to not work very well This phenomenon is discussed at length in [this
 Reddit thread](https://www.reddit.com/r/MachineLearning/comments/7rrrk3/d_eat_your_vggtables_or_why_does_neural_style/). without some sort of parameterization trick .
 The above interpretation of the graph provides an alternative explanation for this phenomenon.
 **Since VGG is unable to capture non-robust features as well as other architectures, the outputs for style
 transfer actually look more correct to humans!**
To follow this argument, note that the perceptual losses used in neural style transfer are
 dependent on matching features learned by a separately trained image classifier. If these learned
 features don’t make sense to humans (non-robust features), the outputs for neural style transfer won’t
 make sense either.

 Before proceeding, let’s quickly discuss the results obtained by Mordvintsev, et. al. in [Differentiable Image
 Parameterizations](https://distill.pub/2018/differentiable-parameterizations/), where they show that non-VGG architectures can work for style transfer by using a
 simple technique previously established in feature visualization.
 In their experiment, instead of optimizing the output image in RGB space, they optimize it in Fourier space,
 and run the image through a series of transformations (e.g jitter, rotation, scaling) before passing it
 through the neural network.

 Can we reconcile this result with our hypothesis linking neural style transfer and non-robust features?

 One possible theory is that all of these image transformations *weaken* or even *destroy*
 non-robust features.
 Since the optimization can no longer reliably manipulate non-robust features to bring down the loss, it is
 forced to use robust features instead, which are presumably more resistant to the applied image
 transformations (a rotated and jittered flappy ear still looks like a flappy ear).

A quick experiment
------------------

 Testing our hypothesis is fairly straightforward:
 Use an adversarially robust classifier for neural style transfer and see
 what happens.

 I evaluated a regularly trained (non-robust) ResNet-50 with a robustly trained ResNet-50 from Engstrom, et.
 al. on their performance on neural style transfer.
 For comparison, I performed the same algorithm with a regular VGG-19
  .

 To ensure a fair comparison despite the different networks having different optimal hyperparameters, I
 performed a small grid search for each image and manually picked the best output per network.
 Further details can be read in a footnote

 L-BFGS was used for optimization as it showed faster convergence
 over Adam.
 For ResNet-50, the style layers used were the ReLu outputs after each of the 4 residual blocks,
 [relu2\_x,relu3\_x,relu4\_x,relu5\_x][relu2\\_x, relu3\\_x, relu4\\_x, relu5\\_x][relu2\_x,relu3\_x,relu4\_x,relu5\_x] while the content layer used was relu4\_xrelu4\\_xrelu4\_x.
 For VGG-19, style layers [relu1\_1,relu2\_1,relu3\_1,relu4\_1,relu5\_1][relu1\\_1,relu2\\_1,relu3\\_1,relu4\\_1,relu5\\_1][relu1\_1,relu2\_1,relu3\_1,relu4\_1,relu5\_1] were used with a content layer
 relu4\_2relu4\\_2relu4\_2.
 In VGG-19, max pooling layers were replaced with avg pooling layers, as stated in Gatys, et. al.

 or observed in the accompanying Colaboratory notebook.

 The results of this experiment can be explored in the diagram below.

 #style-transfer-slider.juxtapose {
 max-height: 512px;
 max-width: 512px;
 }

**Content image**

**Style image**

  Compare VGG or Robust
 ResNet

'use strict';

// I don't know how to write JavaScript without a bundler. Please someone save me.

(function () {

 // Initialize slider
 var currentContent = 'ben';
 var currentStyle = 'scream';
 var currentLeft = 'nonrobust';

 var compareVGGCheck = document.getElementById("check-compare-vgg");
 var styleTransferSliderDiv = document.getElementById("style-transfer-slider");

 function refreshSlider() {
 while (styleTransferSliderDiv.firstChild) {
 styleTransferSliderDiv.removeChild(styleTransferSliderDiv.firstChild);
 }
 var imgPath1 = 'images/style-transfer/' + currentContent + '\_' + currentStyle + '\_' + currentLeft + '.jpg';
 var imgPath2 = 'images/style-transfer/' + currentContent + '\_' + currentStyle + '\_robust.jpg';
 new juxtapose.JXSlider('#style-transfer-slider', [{
 src: imgPath1, // TODO: Might need to use absolute\_url?
 label: currentLeft === 'nonrobust' ? 'Non-robust ResNet50' : 'VGG'
 }, {
 src: imgPath2,
 label: 'Robust ResNet50'
 }], {
 animate: true,
 showLabels: true,
 showCredits: false,
 startingPosition: "50%",
 makeResponsive: true
 });
 }

 refreshSlider();

 compareVGGCheck.onclick = function (evt) {
 currentLeft = evt.target.checked ? 'vgg' : 'nonrobust';
 refreshSlider();
 };

 // Initialize selector
 $("#content-select").imagepicker({
 changed: function changed(oldVal, newVal, event) {
 currentContent = newVal;
 refreshSlider();
 }
 });
 $("#style-select").imagepicker({
 changed: function changed(oldVal, newVal, event) {
 currentStyle = newVal;
 refreshSlider();
 }
 });
})();

 Success!
 The robust ResNet shows drastic improvement over the regular ResNet.
 Remember, all we did was switch the ResNet’s weights, the rest of the code for performing style transfer is
 exactly the same!

 A more interesting comparison can be done between VGG-19 and the robust ResNet.
 At first glance, the robust ResNet’s outputs seem on par with VGG-19.
 Looking closer, however, the ResNet’s outputs seem slightly noisier and exhibit some artifacts
 This is more obvious when the output image is initialized not with the content image, but with
 Gaussian noise..

![](images/zoom/vgg_texture.jpg)

![](images/zoom/vgg_texture.jpg)

 Texture synthesized with VGG.

*Mild artifacts.*

![](images/zoom/resnet_texture.jpg)

![](images/zoom/resnet_texture.jpg)

 Texture synthesized with robust ResNet.

*Severe artifacts.*

 A comparison of artifacts between textures synthesized by VGG and ResNet.
 Interact by hovering around the images.
 This diagram was repurposed from
 [Deconvolution and Checkerboard Artifacts](https://distill.pub/2016/deconv-checkerboard/)
 by Odena, et. al.

 It is currently unclear exactly what causes these artifacts.
 One theory is that they are checkerboard artifacts
 caused by
 non-divisible kernel size and stride in the convolution layers.
 They could also be artifacts caused by the presence of max pooling layers
 in ResNet.
 An interesting implication is that these artifacts, while problematic, seem orthogonal to the
 problem that
 adversarial robustness solves in neural style transfer.

VGG remains a mystery
---------------------

 Although this experiment started because of an observation about a special characteristic of VGG
 nets, it
 did not provide an explanation for this phenomenon.
 Indeed, if we are to accept the theory that adversarial robustness is the reason VGG works out of
 the box
 with neural style transfer, surely we’d find some indication in existing literature that VGG is
 naturally
 more robust than other architectures.

 A few papers
 indeed show
 that VGG architectures are slightly more robust than ResNet.
 However, they also show that AlexNet, not known to work well
 for
 neural style transferAs shown by Dávid Komorowicz
 in
 this [blog post](https://dawars.me/neural-style-transfer-deep-learning/).
 , is
 *above* VGG in terms of this “natural robustness”.

 Perhaps adversarial robustness just happens to incidentally fix or cover up the true reason non-VGG
 architectures fail at style transfer (or other similar algorithms

 In fact, neural style transfer is not the only pretrained classifier-based iterative image
 optimization
 technique that magically works better with adversarial robustness. In Engstrom, et. al., they show that feature visualization via activation
 maximization works on robust classifiers *without*
 enforcing
 any priors or regularization (e.g. image transformations and decorrelated parameterization) used
 by
 previous work. In a recent chat with Chris
 Olah, he
 pointed out that the aforementioned feature visualization techniques actually work well on VGG
 *without* these priors, just like style transfer!

 ) i.e. adversarial robustness is a sufficient but unnecessary condition for good style transfer.
 Whatever the reason, I believe that further examination of VGG is a very interesting direction for
 future
 work.

 To cite Ilyas et al.’s response, please cite their
 [collection of responses](/2019/advex-bugs-discussion/original-authors/#citation).

**Response Summary**: Very interesting
 results, highlighting the effect of non-robust features and the utility of
 robust models for downstream tasks. We’re excited to see what kind of impact
 robustly trained models will have in neural network art! We were also really
 intrigued by the mysteriousness of VGG in the context of style transfer
. As such, we took a
 deeper dive which found some interesting links between robustness and style
 transfer that suggest that perhaps robustness does indeed play a role here.

**Response**: These experiments are really cool! It is interesting that
 preventing the reliance of a model on non-robust features improves performance
 on style transfer, even without an explicit task-related objective (i.e. we
 didn’t train the networks to be better for style transfer).

 We also found the discussion of VGG as a “mysterious network” really
 interesting — it would be valuable to understand what factors drive style transfer
 performance more generally. Though not a complete answer, we made a couple of
 observations while investigating further:

*Style transfer does work with AlexNet:* One wrinkle in the idea that
 robustness is the “secret ingredient” to style transfer could be that VGG is not
 the most naturally robust network — AlexNet is. However, based on our own
 testing, style transfer does seem to work with AlexNet out-of-the-box, as
 long as we use a few early layers in the network (in a similar manner to
 VGG):

![](images/alexnetworks.png)

 Style transfer using AlexNet, using conv\_1 through conv\_4.

 Observe that even though style transfer still works, there are checkerboard
 patterns emerging — this seems to be a similar phenomenon to the one noticed
 in the comment in the context of robust models.
 This might be another indication that these two phenomena (checkerboard
 patterns and style transfer working) are not as intertwined as previously
 thought.

*From prediction robustness to layer robustness:*  Another
 potential wrinkle here is that both AlexNet and VGG are not that
 much more robust than ResNets (for which style transfer completely fails),
 and yet seem to have dramatically better performance. To try to
 explain this, recall that style transfer is implemented as a minimization of a
 combined objective consisting of a style loss and a content loss. We found,
 however, that the network we use to compute the
 style loss is far more important
 than the one for the content loss. The following demo illustrates this — we can
 actually use a non-robust ResNet for the content loss and everything works just
 fine:

![](images/stylematters.png)

Style transfer seems to be rather
 invariant to the choice of content network used, and very sensitive
 to the style network used.

Therefore, from now on, we use a fixed ResNet-50 for the content loss as a
 control, and only worry about the style loss.

Now, note that the way that style loss works is by using the first few
 layers of the relevant network. Thus, perhaps it is not about the robustness of
 VGG’s predictions, but instead about the robustness of the layers that we actually use
 for style transfer?

 To test this hypothesis, we measure the robustness of a layer fff as:

R(f)=Ex1∼D[maxx′∥f(x′)−f(x1)∥2]Ex1,x2∼D[∥f(x1)−f(x2)∥2]
 R(f) = \frac{\mathbb{E}\_{x\_1\sim D}\left[\max\_{x’} \|f(x’) - f(x\_1)\|\_2 \right]}
 {\mathbb{E}\_{x\_1, x\_2 \sim D}\left[\|f(x\_1) - f(x\_2)\|\_2\right]}
 R(f)=Ex1,x2∼D[∥f(x1)−f(x2)∥2]Ex1∼D[maxx′∥f(x′)−f(x1)∥2]
 Essentially, this quantity tells us how much we can change the
 output of that layer f(x)f(x)f(x) within a small ball, normalized by how far apart
 representations are between images in general. We’ve plotted this value for
 the first few layers in a couple of different networks below:

![](images/robustnesses.png)

The robustness R(f)R(f)R(f) of the first
 four layers of VGG16, AlexNet, and robust/standard ResNet-50
 trained on ImageNet.

 Here, it becomes clear that, the first few layers of VGG and AlexNet are
 actually almost as robust as the first few layers of the robust ResNet!
 This is perhaps a more convincing indication that robustness might have
 something to with VGG’s success in style transfer after all.

 Finally, suppose we restrict style transfer to only use a single layer of
 the network when computing the style lossUsually style transfer uses
 several layers in the loss function to get the most visually appealing results — here we’re only interested in whether or not style transfer works (i.e.
 actually confers some style onto the image).. Again, the more
 robust layers seem to indeed work better for style transfer! Since all of the
 layers in the robust ResNet are robust, style transfer yields non-trivial
 results even using the last layer alone. Conversely, VGG and AlexNet seem to
 excel in the earlier layers (where they are non-trivially robust) but fail when
 using exclusively later (non-robust) layers:

![](images/styletransfer.png)

Style transfer using a single layer. The
 names of the layers and their robustness R(f)R(f)R(f) are printed below
 each style transfer result. We find that for both networks, the robust
 layers seem to work (for the robust ResNet, every layer is robust).

 Of course, there is much more work to be done here, but we are excited
 to see further work into understanding the role of both robustness and the VGG
 in network-based image manipulation.

 You can find more responses in the  [main discussion article](/2019/advex-bugs-discussion/).

An experiment showing adversarial robustness makes neural style transfer work on a non-VGG architecture

A Neural Algorithm of Artistic Style

Differentiable Image Parameterizations

Feature Visualization

Deconvolution and checkerboard artifacts

ImageNet Classification with Deep Convolutional Neural Networks

Neural Style transfer with Deep Learning

The Building Blocks of Interpretability


A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too

.comment-info {
 background-color: hsl(54, 78%, 96%);
 border-left: solid hsl(54, 33%, 67%) 1px;
 padding: 1em;
 color: hsla(0, 0%, 0%, 0.67);
 }

 #header-info {
 margin-top: 0;
 margin-bottom: 1.5rem;
 display: grid;
 grid-template-columns: 65px max-content 1fr;
 grid-template-areas:
 "icon explanation explanation"
 "icon back comment";
 grid-column-gap: 1.5em;
 }

 #header-info .icon-multiple-pages {
 grid-area: icon;
 padding: 0.5em;
 content: url(images/multiple-pages.svg);
 }

 #header-info .explanation {
 grid-area: explanation;
 font-size: 85%;
 color: hsl(0, 0%, 0.33);
 }

 #header-info .back {
 grid-area: back;
 }

 #header-info .back::before {

 content: "←";
 margin-right: 0.5em;
 }

 #header-info .comment {
 grid-area: comment;
 scroll-behavior: smooth;
 }

 #header-info .comment::before {
 content: "↓";
 margin-right: 0.5em;
 }

 #header-info a.back,
 #header-info a.comment {
 font-size: 80%;
 font-weight: 600;
 border-bottom: none;
 text-transform: uppercase;
 color: #2e6db7;
 display: block;
 margin-top: 0.25em;
 letter-spacing: 0.25px;
 }

 This article is part of a discussion of the Ilyas et al. paper
 *“Adversarial examples are not bugs, they are features”.*
 You can learn more in the
 [main discussion article](/2019/advex-bugs-discussion/) .

[All Responses](/2019/advex-bugs-discussion/#articles)
[Comment by Ilyas et al.](#rebuttal)

 We demonstrate that there exist adversarial examples which are just “bugs”:
 aberrations in the classifier that are not intrinsic properties of the data distribution.
 In particular, we give a new method for constructing adversarial examples which:

1. Do not transfer between models, and
2. Do not leak “non-robust features” which allow for learning, in the
 sense of Ilyas-Santurkar-Tsipras-Engstrom-Tran-Madry
 .

 We replicate the Ilyas et al.
 experiment of training on mislabeled adversarially-perturbed images
 (Section 3.2 of ),
 and show that it fails for our construction of adversarial perturbations.

 The message is, whether adversarial examples are features or bugs depends
 on how you find them — standard PGD finds features, but bugs are abundant as well.

 We also give a toy example of a data distribution which has no “non-robust features”
 (under any reasonable definition of feature), but for which standard training yields a highly non-robust
 classifier.
 This demonstrates, again, that adversarial examples can occur even if the data distribution does not
 intrinsically have any vulnerable directions.

### Background

 Many have understood Ilyas et al.
 to claim that adversarial examples are not “bugs”, but are “features”.
 Specifically, Ilyas et al. postulate the following two worlds:
 As communicated to us by the original authors.

* **World 1: Adversarial examples exploit directions irrelevant for classification (“bugs”).**
 In this world, adversarial examples occur because classifiers behave
 poorly off-distribution,
 when they are evaluated on inputs that are not natural images.
 Here, adversarial examples would occur in arbitrary directions,
 having nothing to do with the true data distribution.
* **World 2: Adversarial examples exploit useful directions for classification (“features”).**
 In this world, adversarial examples occur in directions that are still “on-distribution”,
 and which contain features of the target class.
 For example, consider the perturbation that
 makes an image of a dog to be classified as a cat.
 In World 2, this perturbation is not purely random, but has something to do with cats.
 Moreover, we expect that this perturbation transfers to other classifiers trained to distinguish cats
 vs. dogs.

 Our main contribution is demonstrating that these worlds are not mutually exclusive — and in fact, we
 are in both.
 Ilyas et al.
 show that there exist adversarial examples in World 2, and we show there exist
 examples in World 1.

 Constructing Non-transferrable Targeted Adversarial Examples
--------------------------------------------------------------

 We propose a method to construct targeted adversarial examples for a given classifier
 $f$,
 which do not transfer to other classifiers trained for the same problem.

 Recall that for a classifier $f$, an input example $(x, y)$, and target class $y\_{targ}$,
 a *targeted adversarial example* is an $x’$ such that $||x - x’||\leq \eps$ and
 $f(x’) = y\_{targ}$.

 The standard method of constructing adversarial examples is via Projected Gradient Descent (PGD)

 PGD is described in the appendix.

 which starts at input $x$, and iteratively takes steps $\{x\_t\}$
 to minimize the loss $L(f, x\_t, y\_{targ})$.
 That is, we take steps in the direction
 $$-\nabla\_x L(f, x\_t, y\_{targ})$$
 where $L(f, x, y)$ is the loss of $f$ on input $x$, label $y$.

 Note that since PGD steps in the gradient direction towards the target class,
 we may expect these adversarial examples have *feature leakage* from the target class.
 For example, suppose we are perturbing an image of a dog into a plane (which usually appears against a blue
 background).
 It is plausible that the gradient direction tends to make the dog image more blue,
 since the “blue” direction is correlated with the plane class.
 In our construction below, we attempt to eliminate such feature leakage.

![](./manifold.svg)

 An illustration of the image-manifold for adversarially perturbing a dog to a plane.
 The gradient of the loss can be thought of as having an on-manifold “feature component”
 and an off-manifold “random component”.
 PGD steps along both components, hence causing feature-leakage in adversarial examples.
 Our construction below attempts to step only in the off-manifold direction.

### Our Construction

 Let $\{f\_i : \R^n \to \cY\}\_i$ be an ensemble of classifiers
 for the same classification problem as $f$.
 For example, we can let $\{f\_i\}$ be a collection of ResNet18s trained from
 different random initializations.

 For input example $(x, y)$ and target class $y\_{targ}$,
 we perform iterative updates to find adversarial attacks — as in PGD.
 However, instead of stepping directly in the gradient direction, we
 step in the direction

 Formally, we replace the iterative step with
 $$x\_{t+1} \gets \Pi\_\eps\left( x\_t
 - \alpha( \nabla\_x L(f, x\_t, y\_{targ}) + \E\_i[ \nabla\_x L(f\_i, x\_t, y)]) \right)$$
 where $\Pi\_\eps$ is the projection onto the $\eps$-ball around $x$.

 $$-\left( \nabla\_x L(f, x\_t, y\_{targ}) + \E\_i[ \nabla\_x L(f\_i, x\_t, y)] \right)$$

 That is, instead of taking gradient steps to minimize $L(f, x, y\_{targ})$,
 we minimize the “disentangled loss”

 We could also consider explicitly using the ensemble to decorrelate,
 by stepping in direction
 $\nabla\_x L(f, x, y\_{targ}) - \E\_i[ \nabla\_x L(f\_i, x, y\_{targ})]$.
 This works well for small $\epsilon$,
 but the given loss has better optimization properties for larger $\epsilon$.

 $$L(f, x, y\_{targ}) + \E\_i[L(f\_i, x, y)]$$
 This loss encourages finding an $x\_t$ which is adversarial for $f$,
 but not for the ensemble $\{f\_i\}$.

 These adversarial examples will not be adversarial for the ensemble $\{f\_i\}$. But perhaps surprisingly,
 these examples are also not adversarial for
 *new* classifiers trained for the same problem.

### Experiments

 We train a ResNet18 on CIFAR10 as our target classifier $f$.
 For our ensemble, we train 10 ResNet18s on CIFAR10, from fresh random initializations.
 We then test the probability that
 a targeted attack for $f$
 transfers to a new (freshly-trained) ResNet18, with the same targeted class.
 Our construction yields adversarial examples which do not transfer well to new models.

 For $L\_{\infty}$ attacks:

|  |  |  |
| --- | --- | --- |
|  | Attack Success | Transfer Success |
| PGD | 99.6% | 52.1% |
| Ours | 98.6% | 0.8% |

 For $L\_2$ attacks:

|  |  |  |
| --- | --- | --- |
|  | Attack Success | Transfer Success |
| PGD | 99.9% | 82.5% |
| Ours | 99.3% | 1.7% |

Adversarial Examples With No Features
-------------------------------------

 Using the above, we can construct adversarial examples
 which *do not suffice* for learning.
 Here, we replicate the Ilyas et al. experiment
 that “Non-robust features suffice for standard classification”
 (Section 3.2 of ),
 but show that it fails for our construction of adversarial examples.

 To review, the Ilyas et al. non-robust experiment was:

1. Train a standard classifier $f$ for CIFAR.
2. From the CIFAR10 training set $S = \{(X\_i, Y\_i)\}$,
 construct an alternate train set $S’ = \{(X\_i^{Y\_i \to (Y\_i + 1)}, Y\_i + 1)\}$,
 where $X\_i^{Y\_i \to (Y\_i +1)}$ denotes an adversarial example for
 $f$, perturbing $X\_i$ from its true class $Y\_i$ towards target class $Y\_i+1 (\text{mod }10)$.
 Note that $S’$ appears to humans as “mislabeled examples”.
3. Train a new classifier $f’$ on train set $S’$.
 Observe that this classifier has non-trivial accuracy on the original CIFAR distribution.

 Ilyas et al. use Step (3) to argue that
 adversarial examples have a meaningful “feature” component.

 However, for adversarial examples constructed using our method, Step (3) fails.
 In fact, $f’$ has good accuracy with respect to the “label-shifted” distribution
 $(X, Y+1)$, which is intuitively what we trained on.

For $L\_{\infty}$ attacks:

|  |  |  |
| --- | --- | --- |
|  | Test Acc on CIFAR: $(X, Y)$ | Test Acc on Shifted-CIFAR: $(X, Y+1)$ |
| PGD | 23.7% | 40.4% |
| Ours | 2.5% | 75.9% |

 Table: Test Accuracies of $f’$

For $L\_2$ attacks:

|  |  |  |
| --- | --- | --- |
|  | Test Acc on CIFAR: $(X, Y)$ | Test Acc on Shifted-CIFAR: $(X, Y+1)$ |
| PGD | 33.2% | 27.3% |
| Ours | 2.8% | 70.8% |

 Table: Test Accuracies of $f’$

Adversarial Squares: Adversarial Examples from Robust Features
--------------------------------------------------------------

 To further illustrate that adversarial examples can be “just bugs”,
 we show that they can arise even when the true data distribution has no “non-robust features” — that is, no intrinsically vulnerable directions.

 We are unaware of a satisfactory definition of “non-robust feature”, but we claim that for any
 reasonable
 *intrinsic* definition, this problem has no non-robust features.
 Intrinsic here meaning, a definition which depends only on geometric properties of the data
 distribution, and not on the family of classifiers, or the finite-sample training set.

 We do not use the Ilyas et al. definition of “non-robust features,” because we believe it is vacuous.
 In particular, by the Ilyas et al. definition, **every** distribution
 has “non-robust features” — so the definition does not discern structural properties of the
 distribution.
 Moreover, for every “robust feature” $f$, there exists a corresponding “non-robust feature” $f’$, such
 that $f$ and $f’$ agree on the data distribution — so the definition depends strongly on the
 family of classifiers being considered.

 In the following toy problem, adversarial vulnerability arises as a consequence of finite-sample
 overfitting, and
 label noise.

 The problem is to distinguish between CIFAR-sized images that are either all-black or all-white,
 with a small amount of random pixel noise and label noise.

![](./twosquares.png)

 A sample of images from the distribution.

![](./data.png)

 Formally, let the distribution be as follows.
 Pick label $Y \in \{\pm 1\}$ uniformly,
 and let $$X :=
 \begin{cases}
 (+\vec{\mathbb{1}} + \vec\eta\_\eps) \cdot \eta & \text{if $Y=1$}\\
 (-\vec{\mathbb{1}} + \vec\eta\_\eps) \cdot \eta & \text{if $Y=-1$}\\
 \end{cases}$$

 where $\vec\eta\_\eps \sim [-0.1, +0.1]^d$ is uniform $L\_\infty$ pixel noise,
 and
 $\eta \in \{\pm 1\} \sim Bernoulli(0.1)$ is the 10% label noise.

 A plot of samples from a 2D-version of this distribution is shown to the right.

 Notice that there exists a robust linear classifier for this problem which achieves perfect robust
 classification, with up to $\eps = 0.9$ magnitude $L\_\infty$ attacks.
 However, if we sample 10000 training images from this distribution, and train
 a ResNet18 to 99.9% train accuracy,

 We optimize using Adam with learning-rate $0.00001$ and batch size $128$ for 20 epochs.

 the resulting classifier is highly non-robust:
 an $\eps=0.01$ perturbation suffices to flip the class of almost all test examples.

 The input-noise and label noise are both essential for this construction.
 One intuition for what is happening is: in the initial stage of training
 the optimization learns the “correct” decision boundary (indeed, stopping after 1 epoch results in a robust
 classifier).
 However, optimizing for close to 0 train-error requires a network with high Lipshitz constant
 to fit the label-noise, which hurts robustness.

![](./data.png)
![](./step10.png)
![](./step10000.png)

 Left: The training set (labels color-coded). Middle: The classifier after 10 SGD steps.
 Right: The classifier at the end of training. Note that it is overfit, and not robust.

 Figure adapted from .

Addendum: Data Poisoning via Adversarial Examples
-------------------------------------------------

 As an addendum, we observe that the “non-robust features”
 experiment of  (Section 3.2)
 directly implies data-poisoning attacks:
 An adversary that is allowed to imperceptibly change every image in the training set can destroy the
 accuracy of the learnt classifier — and can moreover apply an arbitrary permutation
 to the classifier output labels (e.g. swapping cats and dogs).

 To see this, recall that the original “non-robust features” experiment shows:Using our previous
 notation, and also using vanilla PGD to find adversarial examples.

 1. If we train on distribution $(X^{Y \to (Y+1)}, Y+ 1)$ the classifier learns to predict well
 on distribution $(X, Y)$.

 By permutation-symmetry of the labels, this implies that:

 2. If we train on distribution $(X^{Y \to (Y+1)}, Y)$ the classifier learns to predict well
 on distribution $(X, Y-1)$.

 Note that in case (2), we are training with correct labels, just perturbing the inputs imperceptibly,
 but the classifier learns to predict the cyclically-shifted labels.
 Concretely, using the original numbers of
 Table 1 in , this reduction implies that
 **an adversary can perturb the CIFAR10 train set by $\eps=0.5$ in $L\_2$,
 and cause the learnt classifier to output shifted-labels
 43.7% of the time
 (cats classified as birds, dogs as deers, etc).**

 This should extend to attacks that force arbitrary desired permutations of the labels.

 To cite Ilyas et al.’s response, please cite their
 [collection of responses](/2019/advex-bugs-discussion/original-authors/#citation).

**Response Summary**: We note that as discussed in
 more detail in [Takeaway #1](/2019/advex-bugs-discussion/original-authors/#takeaway1), the
 mere existence of adversarial
 examples
 that are “features” is sufficient to corroborate our main thesis. This comment
 illustrates, however, that we can indeed craft adversarial examples that are
 based on “bugs” in realistic settings. Interestingly, such examples don’t
 transfer, which provides further support for the link between transferability
 and non-robust features.

**Response**: As mentioned [above](/2019/advex-bugs-discussion/original-authors/#nonclaim1),
 we did not intend to claim
 that adversarial examples arise *exclusively* from (useful) features but rather
 that useful non-robust features exist and are thus (at least
 partially) responsible for adversarial vulnerability. In fact,
 prior work already shows how in theory adversarial examples can arise from
 insufficient samples  or finite-sample overfitting
 , and the experiments
 presented here (particularly, the adversarial squares) constitute a neat
 real-world demonstration of these facts.

 Our main thesis that “adversarial examples will not just go away as we fix
 bugs in our models” is not contradicted by the existence of adversarial examples
 stemming from “bugs.” As long as adversarial examples can stem from non-robust
 features (which the commenter seems to agree with), fixing these bugs will not
 solve the problem of adversarial examples.

Moreover, with regards to feature “leakage” from PGD, recall that in
 or D\_det dataset, the non-robust features are associated with the
 correct label whereas the robust features are associated with the wrong
 one. We wanted to emphasize that, as
 [shown in Appendix D.6](https://arxiv.org/abs/1905.02175) ,
 models trained on our $D\_{det}$ dataset actually generalize *better* to
 the non-robust feature-label association that to the robust
 feature-label association. In contrast, if PGD introduced a small
 “leakage” of non-robust features, then we would expect the trained model
 would still predominantly use the robust feature-label association.

 That said, the experiments cleverly zoom in on some more fine-grained
 nuances in our understanding of adversarial examples. One particular thing that
 stood out to us is that by creating a set of adversarial examples that are
 *explicitly* non-transferable, one also prevents new classifiers from learning
 features from that dataset. This finding thus makes the connection between
 transferability of adversarial examples and their containing generalizing
 features even stronger! Indeed, we can add the constructed dataset into our
 “$\widehat{\mathcal{D}}\_{det}$ learnability vs transferability” plot
 (Figure 3 in the paper) — the point
 corresponding to this dataset fits neatly onto the trendline!

![](transfer.png)

 Relationship between models reliance on non-robust features and their susceptibility to transfer
 attacks

 You can find more responses in the  [main discussion article](/2019/advex-bugs-discussion/).

Refining the source of adversarial examples

SGD on Neural Networks Learns Functions of Increasing Complexity