Differentiable Image Parameterizations

Neural networks trained to classify images have a remarkable — and surprising! — capacity to generate images.
 Techniques such as DeepDream , style transfer, and feature visualization leverage this capacity as a powerful tool for exploring the inner workings of neural networks, and to fuel a small artistic movement based on neural art.

 All these techniques work in roughly the same way.
 Neural networks used in computer vision have a rich internal representation of the images they look at.
 We can use this representation to describe the properties we want an image to have (e.g. style), and then optimize the input image to have those properties.
 This kind of optimization is possible because the networks are differentiable with respect to their inputs: we can slightly tweak the image to better fit the desired properties, and then iteratively apply such tweaks in gradient descent.

 Typically, we parameterize the input image as the RGB values of each pixel, but that isn’t the only way.
 As long as the mapping from parameters to images is differentiable, we can still optimize alternative parameterizations with gradient descent.

[1](#figure-differentiable-parameterizations):
 As long as an
image parameterization
is differentiable, we can backpropagate
(    )
through it.

MappingParametersimage/RGB spaceLossFunction

 Differentiable image parameterizations invite us to ask “what kind of image generation process can we backpropagate through?”
 The answer is quite a lot, and some of the more exotic possibilities can create a wide range of interesting effects, including 3D neural art, images with transparency, and aligned interpolation.
 Previous work using specific unusual image parameterizations  has shown exciting results — we think that zooming out and looking at this area as a whole suggests there’s even more potential.

### Why Does Parameterization Matter?

 It may seem surprising that changing the parameterization of an optimization problem can significantly change the result, despite the objective function that is actually being optimized remaining the same.
 We see four reasons why the choice of parameterization can have a significant effect:

**(1) - Improved Optimization** -
Transforming the input to make an optimization problem easier — a technique called “preconditioning” — is a staple of optimization.

 Preconditioning is most often presented as a transformation of the gradient
 (usually multiplying it by a positive definite “preconditioner” matrix).
 However, this is equivalent to optimizing an alternate parameterization of the input.

We find that simple changes in parameterization make image optimization for neural art and image optimization much easier.

**(2) - Basins of Attraction** -
When we optimize the input to a neural network, there are often many different solutions, corresponding to different local minima.

 Training deep neural networks characterized by complex optimization landscapes , which may have many equally good local minima for a given objective.
 (Note that finding the global minimum is not always desirable as it may result in an overfitted model .)
 Thus, it’s probably not surprising that optimizing the input to a neural network would also have many local minima.

The probability of our optimization process falling into any particular local minima is controlled by its basin of attraction (i.e., the region of the optimization landscape under the influence of the minimum).
Changing the parameterization of an optimization problem is known to change the sizes of different basins of attraction, influencing the likely result.

**(3) - Additional Constraints**  -
Some parameterizations cover only a subset of possible inputs, rather than the entire space.
An optimizer working in such a parameterization will still find solutions that minimize or maximize the objective function, but they’ll be subject to the constraints of the parameterization.
By picking the right set of constraints, one can impose a variety of constraints, ranging from simple constraints (e.g., the boundary of the image must be black), to rich, subtle constraints.

**(4) - Implicitly Optimizing other Objects** -
 A parameterization may internally use a different kind of object than the one it outputs and we optimize for.
 For example, while the natural input to a vision network is an RGB image, we can parameterize that image as a rendering of a 3D object and, by backpropagating through the rendering process, optimize that instead.
 Because the 3D object has more degrees of freedom than the image, we generally use a *stochastic* parameterization that produces images rendered from different perspectives.

In the rest of the article we give concrete examples where such approaches are beneficial and lead to surprising and interesting visual results.

---

[1](#section-aligned-interpolation)

[Aligned Feature Visualization Interpolation](#section-aligned-interpolation)
-----------------------------------------------------------------------------

 Feature visualization is most often used to visualize individual neurons,
 but it can also be used to [visualize combinations of neurons](https://distill.pub/2017/feature-visualization/#interaction), in order to study how they interact .
 Instead of optimizing an image to make a single neuron fire, one optimizes it to make multiple neurons fire.

 When we want to really understand the interaction between two neurons,
 we can go a step further and create multiple visualizations,
 gradually shifting the objective from optimizing one neuron to putting more weight on the other neuron firing.
 This is in some ways similar to interpolation in the latent spaces of generative models like GANs.

 Despite this, there is a small challenge: feature visualization is stochastic.
 Even if you optimize for the exact same objective, the visualization will be laid out differently each time.
 Normally, this isn’t a problem, but it does detract from the interpolation visualizations.
 If we make them naively, the resulting visualizations will be *unaligned*:
 visual landmarks such as eyes appear in different locations in each image.
 This lack of alignment can make it harder to see the difference due to slightly different objectives,
 because they’re swamped by the much larger differences in layout.

 We can see the issue with independent optimization if we look at the interpolated frames as an animation:

[2](#figure-aligned-interpolation-comparison)

 How can we achieve this aligned interpolation, where visual landmarks do not move between frames?
 There are a number of possible approaches one could try

 For example, one could explicitly penalize differences between adjacent frames. Our final result and our colab notebook use this technique in combination with a shared parameterization.

  , one of which is using a *shared parameterization*: each frame is parameterized as a combination of its own unique parameterization, and a single shared one.

[3](#figure-aligned-interpolation-examples)

 By partially sharing a parameterization between frames, we encourage the resulting visualizations to naturally align.
 Intuitively, the shared parameterization provides a common reference for the displacement of visual landmarks, while the unique one gives to each frame its own visual appeal based on its interpolation weights.

 Concretely, we combine a usually lower-resolution shared parameterization Pshared P\_{\text{shared}}Pshared and full-resolution independent parameterizations PuniqueiP\_{\text{unique}}^iPuniquei that are unique to each frame iii of the visualization.
 Each individual frame iii is then parameterized as a combination PiP^iPi of the two, Pi=σ(Puniquei+Pshared)P^i = \sigma(P\_{\text{unique}}^i + P\_{\text{shared}})Pi=σ(Puniquei+Pshared), where σ\sigmaσ is the logistic sigmoid function.

 This parameterization doesn’t change the objective, but it does enlarge the **(2) basins of attraction** where the visualizations are aligned.

 We can explicitly visualize how shared parameterization affects the basins of attraction in a toy example.
 Let’s consider optimizing two variables xxx and yyy to both minimize L(z)=(z2−1)2L(z)= (z^2-1)^2L(z)=(z2−1)2.
 Since L(z)L(z)L(z) has two basins of attraction z=1z=1z=1 or z=−1z=-1z=−1, the pair of optimization problems has four solutions:
 (x,y)=(1,1)(x,y) = (1,1)(x,y)=(1,1), (x,y)=(−1,1)(x,y) = (-1,1)(x,y)=(−1,1), (x,y)=(1,−1)(x,y) = (1,-1)(x,y)=(1,−1), or (x,y)=(−1,−1)(x,y) = (-1,-1)(x,y)=(−1,−1).
 Let’s consider randomly initializing xxx and yyy, and then optimizing them.
 Normally, the optimization problems are independent, so xxx and yyy are equally likely to come to unaligned solutions (where they have different signs) as aligned ones.
 But if we add a shared parameterization, the problems become coupled and the basin of attraction where they’re aligned becomes bigger.

![](images/diagrams/basin-alignment.png)

 This is an initial example of how differentiable parameterizations in general can be a useful additional tool in visualizing neural networks.

---

[2](#section-styletransfer)

[Style Transfer with non-VGG architectures](#section-styletransfer)
-------------------------------------------------------------------

 Neural style transfer has a mystery:
 despite its remarkable success, almost all style transfer is done with variants of the **VGG architecture**.
 This isn’t because no one is interested in doing style transfer on other architectures, but because attempts to do it on other architectures consistently work poorly.

 Examples of experiments performed with different architectures can be found on [Medium](https://medium.com/mlreview/getting-inception-architectures-to-work-with-style-transfer-767d53475bf8), [Reddit](https://www.reddit.com/r/MachineLearning/comments/7rrrk3/d_eat_your_vggtables_or_why_does_neural_style/) and [Twitter](https://twitter.com/hardmaru/status/954173051330904065).

 Several hypotheses have been proposed to explain why VGG works so much better than other models.
 One suggested explanation is that VGG’s large size causes it to capture information that other models discard.
 This extra information, the hypothesis goes, isn’t helpful for classification, but it does cause the model to work better for style transfer.
 An alternate hypothesis is that other models downsample more aggressively than VGG, losing spatial information.
 We suspect that there may be another factor: most modern vision models have checkerboard artifacts in their gradient , which could make optimization of the stylized image more difficult.

 In previous work we found that a [decorrelated parameterization can significantly improve optimization](//distill.pub/2017/feature-visualization/#preconditioning).
 We find the same approach also improves style transfer, allowing us to use a model that did not otherwise produce visually appealing style transfer results:

[4](#figure-style-transfer-examples):
 Move the slider under “final image optimization” to compare optimization in pixel space with optimization in a decorrelated space. Both images were created with the same objective and differ only in their parameterization.

 Let’s consider this change in a bit more detail. Style transfer involves three images: a content image, a style image, and the image we optimize.
 All three of these feed into the CNN, and the style transfer objective is based on the differences in how these images activate the CNN.
 The only change we make is how we parameterize the optimized image. Instead of parameterizing it in terms of pixels (which are highly correlated with their neighbors), we use a scaled Fourier transform.

[5](#figure-style-transfer-diagram)

content imagelearned imageLossContentStyleCNNCNNCNNstyle imageParameterizing the learned image in a decorrelated space makes style transfer more robust to choice of model.The content objective aims to get neurons to fire in the same position as they did for the content image.The style objective aims to create similar patterns of neuron activation as in the style image — without regard to position.inverse2D FFT

 Our exact implementation can be found in the accompanying notebook. Note that it also uses [transformation robustness](https://distill.pub/2017/feature-visualization/#regularizer-playground-robust), which not all implementations of style transfer use.

---

[3](#section-xy2rgb)

[Compositional Pattern Producing Networks](#section-xy2rgb)
-----------------------------------------------------------

 So far, we’ve explored image parameterizations that are relatively close to how we normally think of images, using pixels or Fourier components.
 In this section, we explore the possibility of  **(3) adding additional constraints** to the optimization process by using a different parameterization.
 More specifically, we parameterize our image as a neural network  — in particular, a Compositional Pattern Producing Network (CPPN) .

 CPPNs are neural networks that map (x,y)(x,y)(x,y) positions to image colors:

(x,y) →CPPN (r,g,b)(x,y) ~\xrightarrow{\tiny CPPN}~ (r,g,b)(x,y) CPPN (r,g,b)

 By applying the CPPN to a grid of positions, one can make arbitrary resolution images.
 The parameters of the CPPN network — the weights and biases — determine what image is produced.
 Depending on the architecture chosen for the CPPN, pixels in the resulting image are constraint to share, up to a certain degree, the color of their neighbors.

 Random parameters can produce aesthetically interesting images , but we can produce more interesting images by learning the parameters of the CPPN .
 Often this is done by evolution ; here we explore the possibility to backpropagate some objective function, such a feature visualization objective.
 This is easily done since the CPPN network is differentiable as the convolutional neural network and the objective function can be propagated also through the CPPN to update its parameters accordingly.
 That is to say, CPPNs are a differentiable image parameterization — a general tool for parameterizing images in any neural art or visualization task.

WeightsChannelCNNCPPNimage/RGB

[6](#figure-xy2rgb-diagram):
 CPPNs are a differentiable image parameterization. We can use them for neural art or visualization tasks by backpropagating past the image, through the CPPN to its parameters.

 Using CPPNs as image parameterization can add an interesting artistic quality to neural art, vaguely reminiscent of light-paintings.Light-painting is an artistic medium where images are created by manipulating colorful light beams with prisms and mirrors. Notable examples of this technique are the [work of Stephen Knapp](http://www.lightpaintings.com/).

Note that light-painting metaphor here is rather fragile: for example light composition is an additive process, while CPPNs can have negative-weighted connections between layers.
 At a more theoretical level, they can be seen as constraining the compositional complexity of your images.
 When used to optimize a feature visualization objective, they produce distinctive images:

[7](#figure-xy2rgb-examples):
 A Compositional Pattern Producing Network (CPPN) is used as differentiable parameterization for visualizing features at different layers.

 The visual quality of the generated images is heavily influenced by the architecture of the chosen CPPN.
 Not only the shape of the network, i.e., the number of layers and filters, plays a role, but also the chosen activation functions and normalization. For example, deeper networks produce more fine grained details compared to shallow ones.
 We encourage readers to experiment in generating different images by changing the architecture of the CPPN. This can be easily done by changing the code in the supplementary notebook.

 The evolution of the patterns generated by the CPPN are artistic artifacts themselves.
 To maintain the metaphor of light-paintings, the optimization process correspond to an iterative adjustments of the beam directions and shapes.
 Because the iterative changes have a more global effect compared to, for example, a pixel parameterization, at the beginning of the optimization only major patterns are visible.
 By iteratively adjusting the weights, our imaginary beams are positioned in such a way that fine details emerge.

![](images/pointer.svg)

[8](#figure-xy2rgb-training):
 Output of CPPNs during training. *Control each video by hovering, or tapping it if you are on a mobile device.*

 By playing with this metaphor, we can also create a new kind of animation that morph one of the above images into a different one.
 Intuitively, we start from one of the light-paintings and we move the beams to create a different one.
 This result is in fact achieved by interpolating the weights of the CPPN representations of the two patterns. A number of intermediate frames are then generated by generating an image given the interpolated CPPN representation.
 As before, changes in the parameter have a global effect and create visually appealing intermediate frames.

![](images/pointer.svg)

[9](#figure-xy2rgb-interpolation):
 Interpolating CPPN weights between two learned points.

 In this section we presented a parameterization that goes beyond a standard image representation.
 Neural networks, a CPPN in this case, can be used to parameterize an image that is optimized for a given objective function.
 More specifically, we combined a feature-visualization objective function with a CPPN parameterization to create infinite-resolution images of distinctive visual style.

---

[4](#section-rgba)

[Generation of Semi-Transparent Patterns](#section-rgba)
--------------------------------------------------------

 The neural networks used in this article were trained to receive 2D RGB images as input.
 Is it possible to use the same network to synthesize artifacts that span  **(4) beyond this domain**?
 It turns out that we can do so by making our differentiable parameterization define a *family* of images instead of a single image, and then sampling one or a few images from that family at each optimization step.
 This is important because many of the objects we’ll explore optimizing have more degrees of freedom than the images going into the network.

 To be concrete, let’s consider the case of semi-transparent images. These images have, in addition to the RGB channels, an alpha channel that encodes each pixel’s opacity (in the range [0,1][0,1][0,1]). In order to feed such images into a neural network trained on RGB images, we need to somehow collapse the alpha channel. One way to achieve this is to overlay the RGBA image III on top of a background image BGBGBG using the standard alpha blending formula In our experiments we use  [gamma-corrected blending](https://en.wikipedia.org/wiki/Alpha_compositing#Composing_alpha_blending_with_gamma_correction)

Orgb  =  Irgb∗Ia  +  BGrgb∗(1−Ia)O\_{rgb} ~~=~~ I\_{rgb} \* I\_a ~~+~~ BG\_{rgb} \* (1 - I\_a)Orgb  =  Irgb∗Ia  +  BGrgb∗(1−Ia),

 where IaI\_aIa is the alpha channel of the image III.

 If we used a static background BGBGBG, such as black, the transparency would merely indicate pixel positions in which that background contributes directly to the optimization objective.
 In fact, this is equivalent to optimizing an RGB image and making it transparent in areas where its color matches with the background!
 Intuitively, we’d like transparent areas to correspond to something like “the content of this area could be anything.”
 Building on this intuition, we use a different random background at every optimization step.

 We have tried both sampling from real images, and using different types of noise.
 As long as they were sufficiently randomized, the different distributions did not meaningfully influence the resulting optimization.
 Thus, for simplicity, we use a smooth 2D gaussian noise.

ChannelCNNrandom noise

[10](#figure-rgba-diagram):
 Adding an alpha channel to the image parameterization allows it to represent transparency.
 Transparent areas are blended with a random background at each step of the optimization.

 By default, optimizing our semi-transparent image will make the image fully opaque, so the network can always get its optimal input.
 To avoid this, we need to change our objective with an objective that encourages some transparency.
 We find it effective to replace the original objective with:

objnew  =  objold  ∗  (1−mean(Ia))\text{obj}\_{\text{new}} ~~=~~ \text{obj}\_{\text{old}} ~~\*~~ (1-\text{mean}(I\_a))objnew  =  objold  ∗  (1−mean(Ia)),

 This new objective automatically balances the original objective objold\text{obj}\_{\text{old}}objold with reducing the mean transparency.
 If the image becomes very transparent, it will focus on the original objective. If it becomes too opaque, it will temporarily stop caring about the original objective and focus on decreasing the average opacity.

[11](#figure-rgba-examples):
 Examples of the optimization of semi transparent images for different layers and units.

 It turns out that the generation of semi-transparent images is useful in feature visualization.
 Feature visualization aims to understand what neurons in a vision model are looking for, by creating images that maximally activate them.
 Unfortunately, there is no way for these visualizations to distinguish which areas of an image strongly influence a neuron’s activation and those which only marginally do so.
 This issue does not occur when optimizing for the activation of entire channels, because in that case every pixel has multiple neurons that are close to centered on it. As a consequence, the entire input image gets filled with copies of what those neurons care about strongly.

 Ideally, we would like a way for our visualizations to make this distinction in importance — one natural way to represent that a part of the image doesn’t matter is for it to be transparent.
 Thus, if we optimize an image with an alpha channel and encourage the overall image to be transparent, parts of the image that are unimportant according to the feature visualization objective should become transparent.

---

[5](#section-featureviz-3d)

[Efficient Texture Optimization through 3D Rendering](#section-featureviz-3d)
-----------------------------------------------------------------------------

 In the previous section, we were able to use a neural network for RGB images to create a semi-transparent RGBA image.
 Can we push this even further, creating **(4) other kinds of objects** even further removed from the RGB input?
 In this section we explore optimizing **3D objects** for a feature-visualization objective.
 We use a 3D rendering process to turn them into 2D RGB images that can be fed into the network, and backpropagate through the rendering process to optimize the texture of the 3D object.

 Our technique is similar to the approach that Athalye et al. used for the creation of real-world adversarial examples, as we rely on the backpropagation of the objective function to randomly sampled views of the 3D model.
 We differ from existing approaches for artistic texture generation, as we do not modify the geometry of the object during back-propagation.
 By disentangling the generation of the texture from the position of their vertices, we can create very detailed texture for complex objects.

Before we can describe our approach, we first need to understand how a 3D object is stored and rendered on screen. The object’s geometry is usually saved as a collection of interconnected triangles called **triangle mesh** or, simply, mesh. To render a realistic model, a **texture** is painted over the mesh. The texture is saved as an image that is applied to the model by using the so called **UV-mapping**. Every vertex cic\_ici in the mesh is associated to a (ui,vi)(u\_i,v\_i)(ui,vi) coordinate in the texture image. The model is then rendered, i.e. drawn on screen, by coloring every triangle with the region of the image that is delimited by the (u,v)(u,v)(u,v) coordinates of its vertices.

 A simple naive way to create the 3D object texture would be to optimize an image the normal way and then use it as a texture to paint on the object.
 However, this approach generates a texture that does not consider the underlying UV-mapping and, therefore, will create a variety of visual artifacts in the rendered object.
 First, **seams** are visible on the rendered texture, because the optimization is not aware of the underlying UV-mapping and, therefore, does not optimize the texture consistently along the split patches of the texture.
 Second, the generated patterns are **randomly oriented** on different parts of the object (see, e.g., the vertical and wiggly patterns) because they are not consistently oriented in the underlying UV-mapping.
 Finally generated patterns are **inconsistently scaled** because the UV-mapping does not enforce a consistent scale between triangle areas and their mapped triangle in the texture.

[13](#figure-featureviz-3d-explanation):
 3D model of the famous Stanford Bunny. You can interact with the model by rotating and zooming. Moreover, you can unfold the object to its two-dimensional texture representation. This unfolding reveals the UV mapping used to store the texture in the texture image. Note how the render-based optimized texture is divided in several patches that allows for a complete and undistorted coverage of the object.

 We take a different approach.
 Instead of directly optimizing the texture, we optimize the texture *through* renderings of the 3D object, like those the user would eventually see.
 The following diagram presents an overview of the proposed pipeline:

ChannelCNNRenderlearned imagelearned textureEvery optimization step, the 3D model is rendered from a random angle aimed at the center of the object.random view3D model

[14](#figure-featureviz-3d-diagram):
 We optimize a texture by backpropagating through the rendering process. This is possible because we know how pixels in the rendered image correspond to pixels in the texture.

We start the process by randomly initializing the texture with a Fourier parameterization.
At every training iteration we sample a random camera position, which is oriented towards the center of the bounding box of the object, and we render the textured object as an image.
We then backpropagate the gradient of the desired objective function, i.e., the feature of interest in the neural network, to the rendered image.

However, an update of the rendered image does not correspond to an update to the texture that we aim at optimizing. Hence, we need to further propagate the changes to the object’s texture.
The propagation is easily implemented by applying a reverse UV-mapping, as for each pixel on screen we know its coordinate in the texture.
By modifying the texture, during the following optimization iterations, the rendered image will incorporate the changes applied in the previous iterations.

[15](#figure-featureviz-3d-examples):
 Textures are generated by optimizing for a feature visualization objective function.
 Seams in the textures are hardly visible and the patterns are correctly oriented.

The resulting textures are consistently optimized along the cuts, hence removing the seams and enforcing an uniform orientation for the rendered object.
Morever, since the function optimization is disentangled by the geometry of the object, the resolution of the texture can be arbitrary high.
In the next section we will se how this framework can be reused for performing an artistic style transfer to the object’s texture.

---

[6](#section-style-transfer-3d)

[Style Transfer for Textures through 3D Rendering](#section-style-transfer-3d)
------------------------------------------------------------------------------

Now that we have established a framework for efficient backpropagation into the UV-mapped texture, we can use it to adapt existing style transfer techniques for 3D objects.
Similarly to the 2D case, we aim at redrawing the original object’s texture with the style of a user-provided image.
The following diagram presents an overview of the approach:

[16](#figure-style-transfer-3d-diagram)

content imagelearned imageLossContentStyleCNNCNNCNNRenderrandom view3D modeloriginal texturestyle imageRenderThe content objective aims to get neurons to fire in the same position as they did for a random view of the 3D model with the original texture.Every optimization step, the 3D model is rendered from a random angle in two different textures: the original one and the learned one.The style objective aims to create similar patterns of neuron activation as in the style image without regard to position.By backpropagating through the rendering process, we can optimize a texture.

The algorithm works in similar way to the one presented in the previous section, starting from a randomly initialized texture.
At each iteration, we sample a random view point oriented toward the center of the bounding box of the object and we render two images of it: one with the original texture, the *content image*, and one with the texture that we are currently optimizing, the *learned image*.

After the *content image* and *learned image* are rendered, we optimize for the style-transfer objective function introduced by Gatys et al. and we map the parameterization back in the UV-mapped texture as introduced in the previous section.
The procedure is then iterated until the desired blend of content and style is obtained in the target texture.

[17](#figure-style-transfer-3d-examples):
 Style Transfer onto various 3D models. Note that visual landmarks in the content texture, such as eyes, show up correctly in the generated texture.

Because every view is optimized independently, the optimization is forced to try to add all the style’s elements at every iteration.
For example, if we use as style image the Van Gogh’s “Starry Night” painting, stars will be added in every single view.
We found we obtain more pleasing results, such as those presented above, by introducing a sort of “memory” of the style of
previous views. To this end, we maintain moving averages of style-representing Gram matrices
over the recently sampled viewpoints. On each optimization iteration we compute the style loss against those averaged matrices,
instead of the ones computed for that particular view.

 We use TensorFlow’s `tf.stop_gradient` method to substitute current Gram matrices
 with their moving averages on forward pass, while still propagating the correct gradients
 to the current Gram matrices.

 An alternative approach, such as the one employed by ,
 would require sampling multiple viewpoints of the scene at each step,
 increasing memory requirements. In contrast, our substitution trick can be also
 used to apply style transfer to high resolution (>10M pixels) images on a
 single consumer-grade GPU.

The resulting textures combine elements of the desired style, while preserving the characteristics of the original texture.
Take as an example the model created by imposing Van Gogh’s starry night as style image.
The resulting texture contains the rythmic and vigorous brush strokes that characterize Van Gogh’s work.
However, despite the style image’s primarily cold tones, the resulting fur has a warm orange undertone as it is preserved from the original texture.
Even more interesting is how the eyes of the bunny are preserved when different styles are transfered.
For example, when the style is obtained from the Van Gogh’s painting, the eyes are transformed in a star-like swirl, while if Kandinsky’s work is used, they become abstract patterns that still resemble the original eyes.

![](images/printed_bunny_extended.jpg)

[18](#figure-style-transfer-3d-picture):
 3D print of a style transfer of ”[La grand parade sur fond rouge](https://www.wikiart.org/en/fernand-leger/the-large-one-parades-on-red-bottom-1953)″ (Fernand Leger, 1953) onto the ”[Stanford Bunny](http://alice.loria.fr/index.php/software/7-data/37-unwrapped-meshes.html)″ by Greg Turk & Marc Levoy.

 Textured models produced with the presented method can be easily used with popular 3D modeling software or game engines. To show this, we 3D printed one of the designs as a real-world physical artifact We used the [Full-color Sandstone](https://www.shapeways.com/materials.sandstone) material..

---

[Conclusions](#conclusions)
---------------------------

 For the creative artist or researcher, there’s a large space of ways to parameterize images for optimization.
 This opens up not only dramatically different image results, but also animations and 3D objects!
 We think the possibilities explored in this article only scratch the surface.
 For example, one could explore extending the optimization of 3D object textures to optimizing the material or reflectance — or even go the direction of Kato et al. and optimize the mesh vertex positions.

 This article focused on *differentiable* image parameterizations, because they are easy to optimize and cover a wide range of possible applications.
 But it’s certainly possible to optimize image parameterizations that aren’t differentiable, or are only partly differentiable, using reinforcement learning or evolutionary strategies .
 Using non-differentiable parameterizations could open up exciting possibilities for image or scene generation.

Alexander Mordvintsev
Nicola Pezzotti
Ludwig Schubert
Chris Olah

A powerful, under-explored tool for neural network visualizations and art.

Inceptionism: Going deeper into neural networks

A Neural Algorithm of Artistic Style

Feature Visualization

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Synthesizing robust adversarial examples

The loss surfaces of multilayer networks

Very deep convolutional networks for large-scale image recognition

Deconvolution and checkerboard artifacts

Deep Image Prior

Compositional pattern producing networks: A novel abstraction of development

Neural Network Generative Art in Javascript

Image Regression

Generating Large Images from Latent Vectors

Artificial Evolution for Computer Graphics

Neural 3D Mesh Renderer

The Stanford Bunny

Natural evolution strategies.

Evolution strategies as a scalable alternative to reinforcement learning

Tensorflow: Large-scale machine learning on heterogeneous distributed systems


Feature-wise transformations

Many real-world problems require integrating multiple sources of information.
 Sometimes these problems involve multiple, distinct modalities of
 information — vision, language, audio, etc. — as is required
 to understand a scene in a movie or answer a question about an image.
 Other times, these problems involve multiple sources of the same
 kind of input, i.e. when summarizing several documents or drawing one
 image in the style of another.

 Video and audio must be understood in the context of each  other to understand the scene.  Credit: still frame from the movie Charade. “Are you sure there’s no mistake?” An image needs to be processed in the context of  a question being asked.  Credit: image-question pair from the CLEVR dataset. Are there an equalnumber of largethings and metalspheres?

 When approaching such problems, it often makes sense to process one source
 of information *in the context of* another; for instance, in the
 right example above, one can extract meaning from the image in the context
 of the question. In machine learning, we often refer to this context-based
 processing as *conditioning*: the computation carried out by a model
 is conditioned or *modulated* by information extracted from an
 auxiliary input.

 Finding an effective way to condition on or fuse sources of information
 is an open research problem, and

 in this article, we concentrate on a specific family of approaches we call
 *feature-wise transformations*.

 We will examine the use of feature-wise transformations in many neural network
 architectures to solve a surprisingly large and diverse set of problems;

 their success, we will argue, is due to being flexible enough to learn an
 effective representation of the conditioning input in varied settings.
 In the language of multi-task learning, where the conditioning signal is
 taken to be a task description, feature-wise transformations
 learn a task representation which allows them to capture and leverage the
 relationship between multiple sources of information, even in remarkably
 different problem settings.

---

Feature-wise transformations
----------------------------

 To motivate feature-wise transformations, we start with a basic example,
 where the two inputs are images and category labels, respectively. For the
 purpose of this example, we are interested in building a generative model of
 images of various classes (puppy, boat, airplane, etc.). The model takes as
 input a class and a source of random noise (e.g., a vector sampled from a
 normal distribution) and outputs an image sample for the requested class.

 A decoder-basedgenerative model maps a source of  noise to a sample  in the context of the  “puppy” class. noise“puppy”decoder

 Our first instinct might be to build a separate model for each
 class. For a small number of classes this approach is not too bad a solution,
 but for thousands of classes, we quickly run into scaling issues, as the number
 of parameters to store and train grows with the number of classes.
 We are also missing out on the opportunity to leverage commonalities between
 classes; for instance, different types of dogs (puppy, terrier, dalmatian,
 etc.) share visual traits and are likely to share computation when
 mapping from the abstract noise vector to the output image.

 Now let’s imagine that, in addition to the various classes, we also need to
 model attributes like size or color. In this case, we can’t
 reasonably expect to train a separate network for *each* possible
 conditioning combination! Let’s examine a few simple options.

 A quick fix would be to concatenate a representation of the conditioning
 information to the noise vector and treat the result as the model’s input.
 This solution is quite parameter-efficient, as we only need to increase
 the size of the first layer’s weight matrix. However, this approach makes the implicit
 assumption that the input is where the model needs to use the conditioning information.
 Maybe this assumption is correct, or maybe it’s not; perhaps the
 model does not need to incorporate the conditioning information until late
 into the generation process (e.g., right before generating the final pixel
 output when conditioning on texture). In this case, we would be forcing the model to
 carry this information around unaltered for many layers.

 Because this operation is cheap, we might as well avoid making any such
 assumptions and concatenate the conditioning representation to the input of
 *all* layers in the network. Let’s call this approach
 *concatenation-based conditioning*.

Concatenation-based conditioning simply concatenates the conditioning  representation to the input.  The result is passed  through a linear layer  to produce the output.  input conditioningrepresentationconcatenatelinear output

 Another efficient way to integrate conditioning information into the network
 is via *conditional biasing*, namely, by adding a *bias* to
 the hidden layers based on the conditioning representation.

Conditional biasing first maps  the conditioning representation to a bias vector.  The bias vector is then  added to the input. inputoutputconditioningrepresentationlinear

 Interestingly, conditional biasing can be thought of as another way to
 implement concatenation-based conditioning. Consider a fully-connected
 linear layer applied to the concatenation of an input
 x\mathbf{x}x and a conditioning representation
 z\mathbf{z}z:

 The same argument applies to convolutional networks, provided we ignore
 the border effects due to zero-padding.

Concatenation-based conditioning is equivalent to conditional biasing.  We can decompose the matrix-  vector product into two matrix-  vector subproducts.  We can then add the  resulting two vectors.  The z-dependent vector  is a conditional bias. Wxzxzconditional bias

 Yet another efficient way to integrate class information into the network is
 via *conditional scaling*, i.e., scaling hidden layers
 based on the conditioning representation.

Conditional scaling first maps the conditioning representation to a  scaling vector.  The scaling vector is then multiplied  with the input. inputoutputconditioningrepresentationlinear

 A special instance of conditional scaling is feature-wise sigmoidal gating:
 we scale each feature by a value between 000 and
 111 (enforced by applying the logistic function), as a
 function of the conditioning representation. Intuitively, this gating allows
 the conditioning information to select which features are passed forward
 and which are zeroed out.

 Given that both additive and multiplicative interactions seem natural and
 intuitive, which approach should we pick? One argument in favor of
 *multiplicative* interactions is that they are useful in learning
 relationships between inputs, as these interactions naturally identify
 “matches”: multiplying elements that agree in sign yields larger values than
 multiplying elements that disagree. This property is why dot products are
 often used to determine how similar two vectors are.

 Multiplicative interactions alone have had a history of success in various
 domains — see [Bibliographic Notes](#bibliographic-notes).

 One argument in favor of *additive* interactions is that they are
 more natural for applications that are less strongly dependent on the
 joint values of two inputs, like feature aggregation or feature detection
 (i.e., checking if a feature is present in either of two inputs).

 In the spirit of making as few assumptions about the problem as possible,
 we may as well combine *both* into a
 conditional *affine transformation*.

 An affine transformation is a transformation of the form
 y=m∗x+by = m \* x + by=m∗x+b.

 All methods outlined above share the common trait that they act at the
 *feature* level; in other words, they leverage *feature-wise*
 interactions between the conditioning representation and the conditioned
 network. It is certainly possible to use more complex interactions,
 but feature-wise interactions often strike a happy compromise between
 effectiveness and efficiency: the number of scaling and/or shifting
 coefficients to predict scales linearly with the number of features in the
 network. Also, in practice, feature-wise transformations (often compounded
 across multiple layers) frequently have enough capacity to model complex
 phenomenon in various settings.

 Lastly, these transformations only enforce a limited inductive bias and
 remain domain-agnostic. This quality can be a downside, as some problems may
 be easier to solve with a stronger inductive bias. However, it is this
 characteristic which also enables these transformations to be so widely
 effective across problem domains, as we will later review.

### Nomenclature

 To continue the discussion on feature-wise transformations we need to
 abstract away the distinction between multiplicative and additive
 interactions. Without losing generality, let’s focus on feature-wise affine
 transformations, and let’s adopt the nomenclature of Perez et al.
 , which formalizes conditional affine
 transformations under the acronym *FiLM*, for Feature-wise Linear
 Modulation.

 Strictly speaking, *linear* is a misnomer, as we allow biasing, but
 we hope the more rigorous-minded reader will forgive us for the sake of a
 better-sounding acronym.

 We say that a neural network is modulated using FiLM, or *FiLM-ed*,
 after inserting *FiLM layers* into its architecture. These layers are
 parametrized by some form of conditioning information, and the mapping from
 conditioning information to FiLM parameters (i.e., the shifting and scaling
 coefficients) is called the *FiLM generator*.
 In other words, the FiLM generator predicts the parameters of the FiLM
 layers based on some auxiliary input.
 Note that the FiLM parameters are parameters in one network but predictions
 from another network, so they aren’t learnable parameters with fixed
 weights as in the fully traditional sense.
 For simplicity, you can assume that the FiLM generator outputs the
 concatenation of all FiLM parameters for the network architecture.

 The FiLM generator processes the conditioning information  and produces parameters that describe how the target network  should alter its computation.  Here, the FiLM-ed network’s computation  is conditioned by two FiLM layers. outputsub-networkFiLMsub-networkFiLMsub-networkinputconditioningFiLM generatorFiLM parameters

 As the name implies, a FiLM layer applies a feature-wise affine
 transformation to its input. By *feature-wise*, we mean that scaling
 and shifting are applied element-wise, or in the case of convolutional
 networks, feature map -wise.

 To expand a little more on the convolutional case, feature maps can be
 thought of as the same feature detector being evaluated at different
 spatial locations, in which case it makes sense to apply the same affine
 transformation to all spatial locations.

 In other words, assuming x\mathbf{x}x is a FiLM layer’s
 input, z\mathbf{z}z is a conditioning input, and
 γ\gammaγ and β\betaβ are
 z\mathbf{z}z-dependent scaling and shifting vectors,

 FiLM(x)=γ(z)⊙x+β(z).
 \textrm{FiLM}(\mathbf{x}) = \gamma(\mathbf{z}) \odot \mathbf{x}
 + \beta(\mathbf{z}).
 FiLM(x)=γ(z)⊙x+β(z).

 You can interact with the following fully-connected and convolutional FiLM
 layers to get an intuition of the sort of modulation they allow:

 In a fully-connected network,  FiLM applies a different affine  transformation to each feature.  First, each feature (or channel)  is scaled by the corresponding γ parameter.  Then, each feature (or channel)  is shifted by the corresponding β parameter. γβ In a convolutional network,  FiLM applies a different affine  transformation to each channel,  consistent across spatial locations. γβ

 In addition to being a good abstraction of conditional feature-wise
 transformations, the FiLM nomenclature lends itself well to the notion of a
 *task representation*. From the perspective of multi-task learning,
 we can view the conditioning signal as the task description. More
 specifically, we can view the concatenation of all FiLM scaling and shifting
 coefﬁcients as both an instruction on *how to modulate* the
 conditioned network and a *representation* of the task at hand. We
 will explore and illustrate this idea later on.

---

Feature-wise transformations in the literature
----------------------------------------------

 Feature-wise transformations find their way into methods applied to many
 problem settings, but because of their simplicity, their effectiveness is
 seldom highlighted in lieu of other novel research contributions. Below are
 a few notable examples of feature-wise transformations in the literature,
 grouped by application domain. The diversity of these applications
 underscores the flexible, general-purpose ability of feature-wise
 interactions to learn effective task representations.

expand all

Visual question-answering+

 Perez et al.  use
 FiLM layers to build a visual reasoning model
 trained on the CLEVR dataset  to
 answer multi-step, compositional questions about synthetic images.

 The linguistic pipeline acts as the FiLM generator.  FiLM layers in each residual  block modulate the visualpipeline. feature extractorsub-networkFiLM layerReLU Each residual block has a  FiLM layer added to it. sub-networkFiLM layer…linearFiLM parametersAreGRUthereGRUmoreGRUcubesGRUthanGRUyellowGRUthingsGRU

 The model’s linguistic pipeline is a FiLM generator which
 extracts a question representation that is linearly mapped to
 FiLM parameter values. Using these values, FiLM layers inserted within each
 residual block condition the visual pipeline. The model is trained
 end-to-end on image-question-answer triples. Strub et al.
  later on improved on the model by
 using an attention mechanism to alternate between attending to the language
 input and generating FiLM parameters layer by layer. This approach was
 better able to scale to settings with longer input sequences such as
 dialogue and was evaluated on the GuessWhat?!
 and ReferIt  datasets.

 de Vries et al.  leverage FiLM
 to condition a pre-trained network. Their model’s linguistic pipeline
 modulates the visual pipeline via conditional batch normalization,
 which can be viewed as a special case of FiLM. The model learns to answer natural language questions about
 real-world images on the GuessWhat?!
 and VQAv1  datasets.

 The linguistic pipeline acts  as the FiLM generator and also  directly passes the question  representation to the rest  of the network.  FiLM layers modulate the pre-  trained visual pipeline by  making the batch normalization  parameters query-dependent. sub-networknormalizationFiLM layersub-networknormalizationFiLM layer… conditional batch  normalization  conditional batch  normalization MLPFiLM parameters…IsLSTMtheLSTMumbrellaLSTMupsideLSTMdownLSTM

 The visual pipeline consists of a pre-trained residual network that is
 fixed throughout training. The linguistic pipeline manipulates the visual
 pipeline by perturbing the residual network’s batch normalization
 parameters, which re-scale and re-shift feature maps after activations
 have been normalized to have zero mean and unit variance. As hinted
 earlier, conditional batch normalization can be viewed as an instance of
 FiLM where the post-normalization feature-wise affine transformation is
 replaced with a FiLM layer.

Style transfer+

 Dumoulin et al.  use
 feature-wise affine transformations — in the form of conditional
 instance normalization layers — to condition a style transfer
 network on a chosen style image. Like conditional batch normalization
 discussed in the previous subsection,
 conditional instance normalization can be seen as an instance of FiLM
 where a FiLM layer replaces the post-normalization feature-wise affine
 transformation. For style transfer, the network models each style as a separate set of
 instance normalization parameters, and it applies normalization with these
 style-specific parameters.

 The FiLM generator predicts parameters  describing the target style.  The style transfer network is conditioned by making  the instance normalization  parameters style-dependent. FiLM generatorFiLM parameterssub-networknormalizationFiLM layersub-networknormalizationFiLM layer… conditional instance  normalization  conditional instance  normalization

 Dumoulin et al.  use a simple
 embedding lookup to produce instance normalization parameters, while
 Ghiasi et al.  further
 introduce a *style prediction network*, trained jointly with the
 style transfer network to predict the conditioning parameters directly from
 a given style image. In this article we opt to use the FiLM nomenclature
 because it is decoupled from normalization operations, but the FiLM
 layers used by Perez et al.  were
 themselves heavily inspired by the conditional normalization layers used
 by Dumoulin et al. .

 Yang et al.  use a related
 architecture for video object segmentation — the task of segmenting a
 particular object throughout a video given that object’s segmentation in the
 first frame. Their model conditions an image segmentation network over a
 video frame on the provided first frame segmentation using feature-wise
 scaling factors, as well as on the previous frame using position-wise
 biases.

 So far, the models we covered have two sub-networks: a primary
 network in which feature-wise transformations are applied and a secondary
 network which outputs parameters for these transformations. However, this
 distinction between *FiLM-ed network* and *FiLM generator*
 is not strictly necessary. As an example, Huang and Belongie
  propose an alternative
 style transfer network that uses adaptive instance normalization layers,
 which compute normalization parameters using a simple heuristic.

 The model processes  content and style images  up to the adaptive instance  normalization layer.  FiLM parameters are  computed as the spatial  mean and standard  deviation statistics of the  style feature maps.  The FiLM-ed feature maps  are fed to the remainder  of the network to produce  the stylized image. sub-networksub-networkγ, β acrossspatial axesFiLMparametersnormalizationFiLM layer adaptive instance  normalization sub-network

 Adaptive instance normalization can be interpreted as inserting a FiLM
 layer midway through the model. However, rather than relying
 on a secondary network to predict the FiLM parameters from the style
 image, the main network itself is used to extract the style features
 used to compute FiLM parameters. Therefore, the model can be seen as
 *both* the FiLM-ed network and the FiLM generator.

Image recognition+

 As discussed in previous subsections, there is nothing preventing us from considering a
 neural network’s activations *themselves* as conditioning
 information. This idea gives rise to
 *self-conditioned* models.

 The FiLM generator predicts FiLM  parameters conditioned on the  network’s internal activations.  An arbitrary input vector (or feature map)  modulates downstream activations. inputsub-networkFiLM layerFiLM generatoroutput

 Highway Networks  are a prime
 example of applying this self-conditioning principle. They take inspiration
 from the LSTMs’  heavy use of
 feature-wise sigmoidal gating in their input, forget, and output gates to
 regulate information flow:

inputsub-networksigmoidal layer1 - xoutput

 The ImageNet 2017 winning model  also
 employs feature-wise sigmoidal gating in a self-conditioning manner, as a
 way to “recalibrate” a layer’s activations conditioned on themselves.

 The squeeze-and-excitation block  uses sigmoidal gating. First, the  network maps input feature maps  to a gating vector.  The gating vector is then multiplied  with the input feature maps. inputoutputglobal poolingReLU layersigmoidal layer

Natural language processing+

 For statistical language modeling (i.e., predicting the next word
 in a sentence), the LSTM
 constitutes a popular class of recurrent network architectures. The LSTM
 relies heavily on feature-wise sigmoidal gating to control the
 information flow in and out of the memory or context cell
 c\mathbf{c}c, based on the hidden states
 h\mathbf{h}h and inputs x\mathbf{x}x at
 every timestep t\mathbf{t}t.

ct-1tanhcthtsigmoidsigmoidtanhsigmoidlinearlinearlinearlinearht-1xtconcatenate

 Also in the domain of language modeling, Dauphin et al.  use sigmoidal
 gating in their proposed *gated linear unit*, which uses half of the
 input features to apply feature-wise sigmoidal gating to the other half.
 Gehring et al.  adopt this
 architectural feature, introducing a fast, parallelizable model for machine
 translation in the form of a fully convolutional network.

 The gated linear unit activation function  uses sigmoidal gating. Half of the input  features go through a sigmoid function  to produce a gating vector.  The gating vector is then multiplied  with the second half of the features. inputsigmoidoutput

 The Gated-Attention Reader
 uses feature-wise scaling, extracting information
 from text by conditioning a document-reading network on a query. Its
 architecture consists of multiple Gated-Attention modules, which involve
 element-wise multiplications between document representation tokens and
 token-specific query representations extracted via soft attention on the
 query representation tokens.

 Dhingra et al. use conditional scaling to  integrate query information into a document  processing network. Applying soft attention  to the query representation tokens produces the scaling vector.  The scaling vector is then multiplied with  the input document representation tokendocumentrepresentationtokenoutputtokenqueryrepresentationtokenssoft attention

Reinforcement learning+

 The Gated-Attention architecture
 uses feature-wise sigmoidal gating to fuse linguistic and visual
 information in an agent trained to follow simple “go-to” language
 instructions in the VizDoom  3D
 environment.

 Chaplot et al. use sigmoidal gating as a multimodal fusion mechanism.  An instruction representation is mapped  to a scaling vector via a sigmoid layer.  The scaling vector is then multiplied with  the input feature maps. A policy network  uses the result to decide the next action. inputoutputinstructionrepresentationsigmoid

 Bahdanau et al. use FiLM
 layers to condition Neural Module Network
 and LSTM -based policies to follow
 basic, compositional language instructions (arranging objects and going
 to particular locations) in a 2D grid world. They train this policy
 in an adversarial manner using rewards from another FiLM-based network,
 trained to discriminate between ground-truth examples of achieved
 instruction states and failed policy trajectories states.

 Outside instruction-following, Kirkpatrick et al.
  also use
 game-specific scaling and biasing to condition a shared policy network
 trained to play 10 different Atari games.

Generative modeling+

 The conditional variant of DCGAN ,
 a well-recognized network architecture for generative adversarial networks
 , uses concatenation-based
 conditioning. The class label is broadcasted as a feature map and then
 concatenated to the input of convolutional and transposed convolutional
 layers in the discriminator and generator networks.

Concatenation-based conditioning is used in the class-conditional  DCGAN model. Each convolutional layer is concatenated with the  broadcased label along the channel axis.  The resulting stack of feature maps  is then convolved to produce the  conditioned output. inputconcatenateinputclass labelconvolutionoutputbroadcastclass label

 For convolutional layers, concatenation-based conditioning requires the
 network to learn redundant convolutional parameters to interpret each
 constant, conditioning feature map; as a result, directly applying a
 conditional bias is more parameter efficient, but the two approaches are
 still mathematically equivalent.

 PixelCNN
 and WaveNet  — two recent
 advances in autoregressive, generative modeling of images and audio,
 respectively — use conditional biasing. The simplest form of
 conditioning in PixelCNN adds feature-wise biases to all convolutional layer
 outputs. In FiLM parlance, this operation is equivalent to inserting FiLM
 layers after each convolutional layer and setting the scaling coefficients
 to a constant value of 1.

 The authors also describe a location-dependent biasing scheme which
 cannot be expressed in terms of FiLM layers due to the absence of the
 feature-wise property.

 PixelCNN uses conditional biasing.  The model first maps a high-level  image description to a bias vector.  Then, it adds the bias vector to  the input stack of feature maps  to condition convolutional layers. inputoutputimagedescriptionlinear

 WaveNet describes two ways in which conditional biasing allows external
 information to modulate the audio or speech generation process based on
 conditioning input:

1. **Global conditioning** applies the same conditional bias
 to the whole generated sequence and is used e.g. to condition on speaker
 identity.
2. **Local conditioning** applies a conditional bias which
 varies across time steps of the generated sequence and is used e.g. to
 let linguistic features in a text-to-speech model influence which sounds
 are produced.

 As in PixelCNN, conditioning in WaveNet can be viewed as inserting FiLM
 layers after each convolutional layer. The main difference lies in how
 the FiLM-generating network is defined: global conditioning
 expresses the FiLM-generating network as an embedding lookup which is
 broadcasted to the whole time series, whereas local conditioning expresses
 it as a mapping from an input sequence of conditioning information to an
 output sequence of FiLM parameters.

Speech recognition+

 Kim et al. modulate a deep
 bidirectional LSTM using a form
 of conditional normalization. As discussed in the
 *Visual question-answering* and *Style transfer* subsections,
 conditional normalization can be seen as an instance of FiLM where
 the post-normalization feature-wise affine transformation is replaced
 with a FiLM layer.

 Kim et al. achieve speaker adaptation by adapting the usual  LSTM architecture to condition its various gates on an  utterance summarization. ct-1tanhcthtsigmoidsigmoidtanhsigmoidht-1 utterance  summarization xt Each gate uses FiLM to condition on the utterancesummarization. linearnormalizationFiLMlinearnormalizationFiLMht-1xtFiLM generator utterance  summarization

 The key difference here is that the conditioning signal does not come from
 an external source but rather from utterance
 summarization feature vectors extracted in each layer to adapt the model.

Domain adaptation and few-shot learning+

 For domain adaptation, Li et al.
 find it effective to update the per-channel batch normalization
 statistics (mean and variance) of a network trained on one domain with that
 network’s statistics in a new, target domain. As discussed in the
 *Style transfer* subsection, this operation is akin to using the network as
 both the FiLM generator and the FiLM-ed network. Notably, this approach,
 along with Adaptive Instance Normalization, has the particular advantage of
 not requiring any extra trainable parameters.

 For few-shot learning, Oreshkin et al.
  explore the use of FiLM layers to
 provide more robustness to variations in the input distribution across
 few-shot learning episodes. The training set for a given episode is used to
 produce FiLM parameters which modulate the feature extractor used in a
 Prototypical Networks
 meta-training procedure.

---

Related ideas
-------------

 Aside from methods which make direct use of feature-wise transformations,
 the FiLM framework connects more broadly with the following methods and
 concepts.

expand all

Zero-shot learning+

 The idea of learning a task representation shares a strong connection with
 zero-shot learning approaches. In zero-shot learning, semantic task
 embeddings may be learned from external information and then leveraged to
 make predictions about classes without training examples. For instance, to
 generalize to unseen object categories for image classification, one may
 construct semantic task embeddings from text-only descriptions and exploit
 objects’ text-based relationships to make predictions for unseen image
 categories. Frome et al. , Socher et
 al. , and Norouzi et al.
  are a few notable exemplars
 of this idea.

HyperNetworks+

 The notion of a secondary network predicting the parameters of a primary
 network is also well exemplified by HyperNetworks , which predict weights for entire layers
 (e.g., a recurrent neural network layer). From this perspective, the FiLM
 generator is a specialized HyperNetwork that predicts the FiLM parameters of
 the FiLM-ed network. The main distinction between the two resides in the
 number and specificity of predicted parameters: FiLM requires predicting far
 fewer parameters than Hypernetworks, but also has less modulation potential.
 The ideal trade-off between a conditioning mechanism’s capacity,
 regularization, and computational complexity is still an ongoing area of
 investigation, and many proposed approaches lie on the spectrum between FiLM
 and HyperNetworks (see [Bibliographic Notes](#bibliographic-notes)).

Attention+

 Some parallels can be drawn between attention and FiLM, but the two operate
 in different ways which are important to disambiguate.

Attention computes a  probability distribution  over locations. Attention pools over locations. Attention summarizes  the input into a vector. FiLM computes a scaling  vector applied to the feature axis. FiLM conserves  input dimensions. αΣ…(β omitted for clarity)γ

 This difference stems from distinct intuitions underlying attention and
 FiLM: the former assumes that specific spatial locations or time steps
 contain the most useful information, whereas the latter assumes that
 specific features or feature maps contain the most useful information.

Bilinear transformations+

 With a little bit of stretching, FiLM can be seen as a special case of a
 bilinear transformation
  with low-rank weight
 matrices. A bilinear transformation defines the relationship between two
 inputs x\mathbf{x}x and z\mathbf{z}z and the
 kthk^{th}kth output feature yky\_kyk as

 yk=xTWkz.
 y\_k = \mathbf{x}^T W\_k \mathbf{z}.
 yk=xTWkz.

 Note that for each output feature yky\_kyk we have a separate
 matrix WkW\_kWk, so the full set of weights forms a
 multi-dimensional array.

 Each element yk of the output vector y is the  result of a distinct vector-matrix-vector product.  This enables multiplicative interactions between  any pair of elements of x and z. yxW1 zxW2 zxW3 zW1zW2zW3z

 If we view z\mathbf{z}z as the concatenation of the scaling
 and shifting vectors γ\gammaγ and β\betaβ and
 if we augment the input x\mathbf{x}x with a 1-valued feature,

 As is commonly done to turn a linear transformation into an affine
 transformation.

 we can represent FiLM using a bilinear transformation by zeroing out the
 appropriate weight matrix entries:

 FiLM computes elements of the output  vector as yk = γk xk + βk.  This can be expressed as a dot product  between a 1-augmented x and a sparse  vector containing γk and βk.  (Shaded  cells have a zero value.)  The sparse vector is given by multiplying  a low-rank weight matrix with the  concatenation of γ and β. (Shaded cells  again have a zero value.) y1xW1 z1xW2 z1xW3 zW111zγβW211zγβW311zγβ

 For some applications of bilinear transformations,
 see the [Bibliographic Notes](#bibliographic-notes).

---

Properties of the learned task representation
---------------------------------------------

 As hinted earlier, in adopting the FiLM perspective we implicitly introduce
 a notion of *task representation*: each task — be it a question
 about an image or a painting style to imitate — elicits a different
 set of FiLM parameters via the FiLM generator which can be understood as its
 representation in terms of how to modulate the FiLM-ed network. To help
 better understand the properties of this representation, let’s focus on two
 FiLM-ed models used in fairly different problem settings:

* The visual reasoning model of Perez et al.
 , which uses FiLM
 to modulate a visual processing pipeline based off an input question.

 The linguistic pipeline acts as the FiLM generator.  FiLM layers in each residual  block modulate the visualpipeline. feature extractorsub-networkFiLM layerReLU Each residual block has a  FiLM layer added to it. sub-networkFiLM layer…linearFiLM parametersAreGRUthereGRUmoreGRUcubesGRUthanGRUyellowGRUthingsGRU
* The artistic style transfer model of Ghiasi et al.
 , which uses FiLM to modulate a
 feed-forward style transfer network based off an input style image.

 The FiLM generator predicts parameters  describing the target style.  The style transfer network is conditioned by making  the instance normalization  parameters style-dependent. FiLM generatorFiLM parameterssub-networknormalizationFiLM layersub-networknormalizationFiLM layer… conditional instance  normalization  conditional instance  normalization

 As a starting point, can we discern any pattern in the FiLM parameters as a
 function of the task description? One way to visualize the FiLM parameter
 space is to plot γ\gammaγ against β\betaβ,
 with each point corresponding to a specific task description and a specific
 feature map. If we color-code each point according to the feature map it
 belongs to we observe the following:

 FiLM parameters for 256 tasks and for 16 feature maps, chosen randomly.  Visual reasoning model  Style transfer model γβ Feature map γβ Feature map

 The plots above allow us to make several interesting observations. First,
 FiLM parameters cluster by feature map in parameter space, and the cluster
 locations are not uniform across feature maps. The orientation of these
 clusters is also not uniform across feature maps: the main axis of variation
 can be γ\gammaγ-aligned, β\betaβ-aligned, or
 diagonal at varying angles. These findings suggest that the affine
 transformation in FiLM layers is not modulated in a single, consistent way,
 i.e., using γ\gammaγ only, β\betaβ only, or
 γ\gammaγ and β\betaβ together in some specific
 way. Maybe this is due to the affine transformation being overspecified, or
 maybe this shows that FiLM layers can be used to perform modulation
 operations in several distinct ways.

 Nevertheless, the fact that these parameter clusters are often somewhat
 “dense” may help explain why the style transfer model of Ghiasi et al.
  is able to perform style
 interpolations: any convex combination of FiLM parameters is likely to
 correspond to a meaningful parametrization of the FiLM-ed network.

Style 1Style 2 w InterpolationContent Image

 To some extent, the notion of interpolating between tasks using FiLM
 parameters can be applied even in the visual question-answering setting.
 Using the model trained in Perez et al. ,
 we interpolated between the model’s FiLM parameters for two pairs of CLEVR
 questions. Here we visualize the input locations responsible for
 the globally max-pooled features fed to the visual pipeline’s output classifier:

 What shape is  the red thing left  of the sphere?  What shape is  the red thing right  of the sphere?  How many brown  things are there?  How many yellow  things are there?

 The network seems to be softly switching where in the image it is looking,
 based on the task description. It is quite interesting that these semantically
 meaningful interpolation behaviors emerge, as the network has not been
 trained to act this way.

 Despite these similarities across problem settings, we also observe
 qualitative differences in the way in which FiLM parameters cluster as a
 function of the task description. Unlike the style transfer model, the
 visual reasoning model sometimes exhibits several FiLM parameter
 sub-clusters for a given feature map.

 FiLM parameters of the visual reasoning model for 256 questions chosen randomly. Feature map 26 of the first FiLM layer. Feature map 76 of the first FiLM layer. γβγβ

 At the very least, this may indicate that FiLM learns to operate in ways
 that are problem-specific, and that we should not expect to find a unified
 and problem-independent explanation for FiLM’s success in modulating FiLM-ed
 networks. Perhaps the compositional or discrete nature of visual reasoning
 requires the model to implement several well-defined modes of operation
 which are less necessary for style transfer.

 Focusing on individual feature maps which exhibit sub-clusters, we can try
 to infer how questions regroup by color-coding the scatter plots by question
 type.

 FiLM parameters of the visual reasoning model for 256 questions chosen randomly. Feature map 26 of the first FiLM layer. Feature map 76 of the first FiLM layer.  Question type γβγβ

 Sometimes a clear pattern emerges, as in the right plot, where color-related
 questions concentrate in the top-right cluster — we observe that
 questions either are of type *Query color* or *Equal color*,
 or contain concepts related to color. Sometimes it is harder to draw a
 conclusion, as in the left plot, where question types are scattered across
 the three clusters.

 In cases where question types alone cannot explain the clustering of the
 FiLM parameters, we can turn to the conditioning content itself to gain
 an understanding of the mechanism at play. Let’s take a look at two more
 plots: one for feature map 26 as in the previous figure, and another
 for a different feature map, also exhibiting several subclusters. This time
 we regroup points by the words which appear in their associated question.

 FiLM parameters of the visual reasoning model for 256 questions chosen randomly. Feature map 26 suggests an object position  separation mechanism. Feature map 92 suggests an object material  separation mechanism.  Word in question γβγβ

 In the left plot, the left subcluster corresponds to questions involving
 objects positioned *in front* of other objects, while the right
 subcluster corresponds to questions involving objects positioned
 *behind* other objects. In the right plot we see some evidence of
 separation based on object material: the left subcluster corresponds to
 questions involving *matte* and *rubber* objects, while the
 right subcluster contains questions about *shiny* or
 *metallic* objects.

 The presence of sub-clusters in the visual reasoning model also suggests
 that question interpolations may not always work reliably, but these
 sub-clusters don’t preclude one from performing arithmetic on the question
 representations, as Perez et al.
 report.

 The model incorrectly answers a question  which involves an unseen combination  of concepts (in bold).  Rather than using the FiLM parameters  of the FiLM generator, we can use those  produced by combining questions with  familiar combinations of concepts (in bold). This corrects the model’s answer.  Q: What is the blue big cylinder made of?  Q ? Rubber ✘ QA : What is the blue big sphere made of?  QB : What is the green big cylinder made of?  QC : What is the green big sphere made of? QA+ QB- QC ?Metal ✔

 Perez et al.  report that this sort of
 task analogy is not always successful in correcting the model’s answer, but
 it does point to an interesting fact about FiLM-ed networks: sometimes the
 model makes a mistake not because it is incapable of computing the correct
 output, but because it fails to produce the correct FiLM parameters for a
 given task description. The reverse can also be true: if the set of tasks
 the model was trained on is insufficiently rich, the computational
 primitives learned by the FiLM-ed network may be insufficient to ensure good
 generalization. For instance, a style transfer model may lack the ability to
 produce zebra-like patterns if there are no stripes in the styles it was
 trained on. This could explain why Ghiasi et al.
  report that their style transfer
 model’s ability to produce pastiches for new styles degrades if it has been
 trained on an insufficiently large number of styles. Note however that in
 that case the FiLM generator’s failure to generalize could also play a role,
 and further analysis would be needed to draw a definitive conclusion.

 This points to a separation between the various computational
 primitives learned by the FiLM-ed network and the “numerical recipes”
 learned by the FiLM generator: the model’s ability to generalize depends
 both on its ability to parse new forms of task descriptions and on it having
 learned the required computational primitives to solve those tasks. We note
 that this multi-faceted notion of generalization is inherited directly from
 the multi-task point of view adopted by the FiLM framework.

 Let’s now turn our attention back to the overal structural properties of FiLM
 parameters observed thus far. The existence of this structure has already
 been explored, albeit more indirectly, by Ghiasi et al.
  as well as Perez et al.
 , who applied t-SNE
  on the FiLM parameter values.

 t-SNE projection of FiLM parameters for many task descriptions.

Visual reasoning model Question type Reset pan / zoomStyle transfer model Artist name Reset pan / zoom

 The projection on the left is inspired by a similar projection done by Perez
 et al.  for their visual reasoning
 model trained on CLEVR and shows how questions group by question type.
 The projection on the right is inspired by a similar projection done by
 Ghiasi et al.  for their style
 transfer network. The projection does not cluster artists as neatly as the
 projection on the left, but this is to be expected, given that an artist’s
 style may vary widely over time. However, we can still detect interesting
 patterns in the projection: note for instance the isolated cluster (circled
 in the figure) in which paintings by Ivan Shishkin and Rembrandt are
 aggregated. While these two painters exhibit fairly different styles, the
 cluster is a grouping of their sketches.

 Rembrandt’s Woman with aPink.  Shishkin’s Woman with a boyin the forest.  Sketches by Rembrandt and  Shishkin found in the same t-SNE  cluster.

 To summarize, the way neural networks learn to use FiLM layers seems to
 vary from problem to problem, input to input, and even from feature to
 feature; there does not seem to be a single mechanism by which the
 network uses FiLM to condition computation. This flexibility may
 explain why FiLM-related methods have been successful across such a
 wide variety of domains.

---

Discussion
----------

 Looking forward, there are still many unanswered questions.
 Do these experimental observations on FiLM-based architectures generalize to
 other related conditioning mechanisms, such as conditional biasing, sigmoidal
 gating, HyperNetworks, and bilinear transformations? When do feature-wise
 transformations outperform methods with stronger inductive biases and vice
 versa? Recent work combines feature-wise transformations with stronger
 inductive bias methods
 ,
 which could be an optimal middle ground. Also, to what extent are FiLM’s
 task representation properties
 inherent to FiLM, and to what extent do they emerge from other features
 of neural networks (i.e. non-linearities, FiLM generator
 depth, etc.)? If you are interested in exploring these or other
 questions about FiLM, we recommend looking into the code bases for
 FiLM models for [visual reasoning](https://github.com/ethanjperez/film)
 and [style transfer](https://github.com/tensorflow/magenta/tree/master/magenta/models/arbitrary_image_stylization)
 which we used as a
 starting point for our experiments here.

 Finally, the fact that changes on the feature level alone are able to
 compound into large and meaningful modulations of the FiLM-ed network is
 still very surprising to us, and hopefully future work will uncover deeper
 explanations. For now, though, it is a question that
 evokes the even grander mystery of how neural networks in general compound
 simple operations like matrix multiplications and element-wise
 non-linearities into semantically meaningful transformations.

Vincent Dumoulin
Ethan Perez
Nathan Schucher
Florian Strub
Harm de Vries
Aaron Courville
Yoshua Bengio

A simple and surprisingly effective family of conditioning mechanisms.

FiLM: Visual Reasoning with a General Conditioning Layer

Learning visual reasoning without strong priors

CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning

GuessWhat?! Visual object discovery through multi-modal dialogue

ReferItGame: Referring to objects in photographs of natural scenes

Modulating early visual processing by language

VQA: visual question answering

A learned representation for artistic style

Exploring the structure of a real-time, arbitrary neural artistic stylization network

Efficient video object segmentation via network modulation

Arbitrary style transfer in real-time with adaptive instance normalization

Highway networks

Long short-term memory

Squeeze-and-Excitation networks

On the state of the art of evaluation in neural language models

Language modeling with gated convolutional networks

Convolution sequence-to-sequence learning

Gated-attention readers for text comprehension

Gated-attention architectures for task-oriented language grounding

Vizdoom: A doom-based AI research platform for visual reinforcement learning

Learning to follow language instructions with adversarial reward induction

Neural module networks

Overcoming catastrophic forgetting in neural networks

Unsupervised representation learning with deep convolutional generative adversarial networks

Generative adversarial nets

Conditional image generation with PixelCNN decoders

WaveNet: A generative model for raw audio

Dynamic layer normalization for adaptive neural acoustic modeling in speech recognition

Adaptive batch normalization for practical domain adaptation

TADAM: Task dependent adaptive metric for improved few-shot learning

Prototypical networks for few-shot learning

Devise: A deep visual-semantic embedding model

Zero-shot learning through cross-modal transfer

Zero-shot learning by convex combination of semantic embeddings

HyperNetworks

Separating style and content with bilinear models

Visualizing data using t-SNE

A dataset and architecture for visual reasoning with a working memory

A parallel computation that assigns canonical object-based frames of reference

The correlation theory of brain function

Generating text with recurrent neural networks

Robust boltzmann machines for recognition and denoising

Factored conditional restricted Boltzmann machines for modeling motion style

Combining discriminative features to infer complex trajectories

Learning where to attend with deep architectures for image tracking

Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis

Convolutional learning of spatio-temporal features

Learning to relate images

Incorporating side information by adaptive convolution

Learning multiple visual domains with residual adapters

Predicting deep zero-shot convolutional neural networks using textual descriptions

Zero-shot task generalization with multi-task deep reinforcement learning

Separating style and content

Facial expression space learning

Personalized recommendation on dynamic content using predictive bilinear models

Like like alike: joint friendship and interest propagation in social networks

Matrix factorization techniques for recommender systems

Bilinear CNN models for fine-grained visual recognition

Convolutional two-stream network fusion for video action recognition

Multimodal compact bilinear pooling for visual question answering and visual grounding