Sequence Modeling with CTC

Introduction
------------

Consider speech recognition. We have a dataset of audio clips and
corresponding transcripts. Unfortunately, we don’t know how the characters in
the transcript align to the audio. This makes training a speech recognizer
harder than it might at first seem.

Without this alignment, the simple approaches aren’t available to us. We
could devise a rule like “one character corresponds to ten inputs”. But
people’s rates of speech vary, so this type of rule can always be broken.
Another alternative is to hand-align each character to its location in the
audio. From a modeling standpoint this works well — we’d know the ground truth
for each input time-step. However, for any reasonably sized dataset this is
prohibitively time consuming.

This problem doesn’t just turn up in speech recognition. We see it in many
other places. Handwriting recognition from images or sequences of pen strokes
is one example. Action labelling in videos is another.

![](assets/handwriting_recognition.svg)

**Handwriting recognition:** The input can be
 (x,y)(x,y)(x,y) coordinates of a pen stroke or
 pixels in an image.

![](assets/speech_recognition.svg)

**Speech recognition:** The input can be a spectrogram or some
 other frequency based feature extractor.

Connectionist Temporal Classification (CTC) is a way to get around not
knowing the alignment between the input and the output. As we’ll see, it’s
especially well suited to applications like speech and handwriting
recognition.

---

To be a bit more formal, let’s consider mapping input sequences
X=[x1,x2,…,xT]X = [x\_1, x\_2, \ldots, x\_T]X=[x1,x2,…,xT], such as audio, to corresponding output
sequences Y=[y1,y2,…,yU]Y = [y\_1, y\_2, \ldots, y\_U]Y=[y1,y2,…,yU], such as transcripts.
We want to find an accurate mapping from XXX’s to YYY’s.

There are challenges which get in the way of us
using simpler supervised learning algorithms. In particular:

* Both XXX and YYY
 can vary in length.
* The ratio of the lengths of XXX and YYY
 can vary.
* We don’t have an accurate alignment (correspondence of the elements) of
 XXX and Y.Y.Y.

The CTC algorithm overcomes these challenges. For a given XXX
it gives us an output distribution over all possible YYY’s. We
can use this distribution either to *infer* a likely output or to assess
the *probability* of a given output.

Not all ways of computing the loss function and performing inference are
tractable. We’ll require that CTC do both of these efficiently.

**Loss Function:** For a given input, we’d like to train our
model to maximize the probability it assigns to the right answer. To do this,
we’ll need to efficiently compute the conditional probability
p(Y∣X).p(Y \mid X).p(Y∣X). The function p(Y∣X)p(Y \mid X)p(Y∣X) should
also be differentiable, so we can use gradient descent.

**Inference:** Naturally, after we’ve trained the model, we
want to use it to infer a likely YYY given an X.X.X.
This means solving

Y∗=argmaxYp(Y∣X).
 Y^\* \enspace =\enspace {\mathop{\text{argmax}}\limits\_{Y}} \enspace p(Y \mid X).
Y∗=Yargmaxp(Y∣X).

Ideally Y∗Y^\*Y∗ can be found efficiently. With CTC we’ll settle
for an approximate solution that’s not too expensive to find.

The Algorithm
-------------

The CTC algorithm can assign a probability for any YYY
given an X.X.X. The key to computing this probability is how CTC
thinks about alignments between inputs and outputs. We’ll start by looking at
these alignments and then show how to use them to compute the loss function and
perform inference.

### Alignment

The CTC algorithm is *alignment-free* — it doesn’t require an
alignment between the input and the output. However, to get the probability of
an output given an input, CTC works by summing over the probability of all
possible alignments between the two. We need to understand what these
alignments are in order to understand how the loss function is ultimately
calculated.

To motivate the specific form of the CTC alignments, first consider a naive
approach. Let’s use an example. Assume the input has length six and Y=Y
=Y= [c, a, t]. One way to align XXX and YYY
is to assign an output character to each input step and collapse repeats.

![](assets/naive_alignment.svg)

This approach has two problems.

* Often, it doesn’t make sense to force every input step to align to
 some output. In speech recognition, for example, the input can have stretches
 of silence with no corresponding output.
* We have no way to produce outputs with multiple characters in a row.
 Consider the alignment [h, h, e, l, l, l, o]. Collapsing repeats will
 produce “helo” instead of “hello”.

To get around these problems, CTC introduces a new token to the set of
allowed outputs. This new token is sometimes called the *blank* token. We’ll
refer to it here as ϵ.\epsilon.ϵ. The
ϵ\epsilonϵ token doesn’t correspond to anything and is simply
removed from the output.

The alignments allowed by CTC are the same length as the input. We allow any
alignment which maps to YYY after merging repeats and removing
ϵ\epsilonϵ tokens:

![](assets/ctc_alignment_steps.svg)

If YYY has two of the same character in a row, then a valid
alignment must have an ϵ\epsilonϵ between them. With this rule
in place, we can differentiate between alignments which collapse to “hello” and
those which collapse to “helo”.

Let’s go back to the output [c, a, t] with an input of length six. Here are
a few more examples of valid and invalid alignments.

![](assets/valid_invalid_alignments.svg)

The CTC alignments have a few notable properties. First, the allowed
alignments between XXX and YYY are monotonic.
If we advance to the next input, we can keep the corresponding output the
same or advance to the next one. A second property is that the alignment of
XXX to YYY is many-to-one. One or more input
elements can align to a single output element but not vice-versa. This implies
a third property: the length of YYY cannot be greater than the
length of X.X.X.

### Loss Function

The CTC alignments give us a natural way to go from probabilities at each
time-step to the probability of an output sequence.

![](assets/full_collapse_from_audio.svg)

To be precise, the CTC objective
for a single (X,Y)(X, Y)(X,Y) pair is:

p(Y∣X)=p(Y \mid X) \;\; =p(Y∣X)=

∑A∈AX,Y\sum\_{A \in \mathcal{A}\_{X,Y}}A∈AX,Y∑

∏t=1Tpt(at∣X)\prod\_{t=1}^T \; p\_t(a\_t \mid X)t=1∏Tpt(at∣X)

 The CTC conditional **probability**

**marginalizes** over the set of valid alignments

 computing the **probability** for a single alignment step-by-step.

Models trained with CTC typically use a recurrent neural network (RNN) to
estimate the per time-step probabilities, pt(at∣X).p\_t(a\_t \mid X).pt(at∣X).
An RNN usually works well since it accounts for context in the input, but we’re
free to use any learning algorithm which produces a distribution over output
classes given a fixed-size slice of the input.

If we aren’t careful, the CTC loss can be very expensive to compute. We
could try the straightforward approach and compute the score for each alignment
summing them all up as we go. The problem is there can be a massive number of
alignments.

 For a YYY of length UUU without any repeat
 characters and an XXX of length TTT the size
 of the set is (T+UT−U).{T + U \choose T - U}.(T−UT+U). For T=100T=100T=100 and
 U=50U=50U=50 this number is almost 1040.10^{40}.1040.

For most problems this would be too slow.

Thankfully, we can compute the loss much faster with a dynamic programming
algorithm. The key insight is that if two alignments have reached the same
output at the same step, then we can merge them.

![](assets/all_alignments.svg)

 Summing over all alignments can be very expensive.

![](assets/merged_alignments.svg)

 Dynamic programming merges alignments, so it’s much faster.

Since we can have an ϵ\epsilonϵ before or after any token in
YYY, it’s easier to describe the algorithm
using a sequence which includes them. We’ll work with the sequence
Z=[ϵ, y1, ϵ, y2, …, ϵ, yU, ϵ]
Z \enspace =\enspace [\epsilon, ~y\_1, ~\epsilon, ~y\_2,~ \ldots, ~\epsilon, ~y\_U, ~\epsilon]
Z=[ϵ, y1, ϵ, y2, …, ϵ, yU, ϵ]
which is YYY with an ϵ\epsilonϵ at
the beginning, end, and between every character.

Let’s let α\alphaα be the score of the merged alignments at a
given node. More precisely, αs,t\alpha\_{s, t}αs,t is the CTC score of
the subsequence Z1:sZ\_{1:s}Z1:s after ttt input steps.
As we’ll see, we’ll compute the final CTC score, P(Y∣X)P(Y \mid X)P(Y∣X),
from the α\alphaα’s at the last time-step. As long as we know
the values of α\alphaα at the previous time-step, we can compute
αs,t.\alpha\_{s, t}.αs,t. There are two cases.

**Case 1:**

![](assets/cost_no_skip.svg)

In this case, we can’t jump over zs−1z\_{s-1}zs−1, the previous
token in Z.Z.Z. The first reason is that the previous token can
be an element of YYY, and we can’t skip elements of
Y.Y.Y. Since every element of YYY in
ZZZ is followed by an ϵ\epsilonϵ, we can
identify this when zs=ϵ. z\_{s} = \epsilon.zs=ϵ. The second reason is
that we must have an ϵ\epsilonϵ between repeat characters in
Y.Y.Y. We can identify this when
zs=zs−2.z\_s = z\_{s-2}.zs=zs−2.

To ensure we don’t skip zs−1z\_{s-1}zs−1, we can either be there
at the previous time-step or have already passed through at some earlier
time-step. As a result there are two positions we can transition from.

αs,t=\alpha\_{s, t} \; =αs,t=

(αs−1,t−1+αs,t−1)⋅(\alpha\_{s-1, t-1} + \alpha\_{s, t-1}) \quad\quad \cdot(αs−1,t−1+αs,t−1)⋅

 The CTC probability of the two valid subsequences after
 t−1t-1t−1 input steps.

pt(zs∣X)p\_t(z\_{s} \mid X)pt(zs∣X)

 The probability of the current character at input step t.t.t.

![](assets/cost_regular.svg)

**Case 2:**

In the second case, we’re allowed to skip the previous token in
Z.Z.Z. We have this case whenever zs−1z\_{s-1}zs−1 is
an ϵ\epsilonϵ between unique characters. As a result there are
three positions we could have come from at the previous step.

αs,t=\alpha\_{s, t} \; =αs,t=

(αs−2,t−1+αs−1,t−1+αs,t−1)⋅(\alpha\_{s-2, t-1} + \alpha\_{s-1, t-1} + \alpha\_{s, t-1}) \quad\quad \cdot(αs−2,t−1+αs−1,t−1+αs,t−1)⋅

 The CTC probability of the three valid subsequences after
 t−1t-1t−1 input steps.

pt(zs∣X)p\_t(z\_{s} \mid X)pt(zs∣X)

 The probability of the current character at input step t.t.t.

Below is an example of the computation performed by the dynamic programming
algorithm. Every valid alignment has a path in this graph.

 output

Y=Y =Y= [a, b]

 input, XXX

![](assets/ctc_cost.svg)

 Node (s,t)(s, t)(s,t) in the diagram represents
 αs,t\alpha\_{s, t}αs,t – the CTC score of
 the subsequence Z1:sZ\_{1:s}Z1:s after
 ttt input steps.

There are two valid starting nodes and two valid final nodes since the
ϵ\epsilonϵ at the beginning and end of the sequence is
optional. The complete probability is the sum of the two final nodes.

Now that we can efficiently compute the loss function, the next step is to
compute a gradient and train the model. The CTC loss function is differentiable
with respect to the per time-step output probabilities since it’s just sums and
products of them. Given this, we can analytically compute the gradient of the
loss function with respect to the (unnormalized) output probabilities and from
there run backpropagation as usual.

For a training set D\mathcal{D}D, the model’s parameters
are tuned to minimize the negative log-likelihood
∑(X,Y)∈D−logp(Y∣X)
\sum\_{(X, Y) \in \mathcal{D}} -\log\; p(Y \mid X)
(X,Y)∈D∑−logp(Y∣X)
instead of maximizing the likelihood directly.

### Inference

After we’ve trained the model, we’d like to use it to find a likely output
for a given input. More precisely, we need to solve:

Y∗=argmaxYp(Y∣X)
Y^\* \enspace = \enspace {\mathop{\text{argmax}}\limits\_{Y}} \enspace p(Y \mid X)
Y∗=Yargmaxp(Y∣X)
One heuristic is to take the most likely output at each time-step. This
gives us the alignment with the highest probability:

A∗=argmaxA∏t=1Tpt(at∣X)
A^\* \enspace = \enspace {\mathop{\text{argmax}}\limits\_{A}} \enspace \prod\_{t=1}^{T} \; p\_t(a\_t \mid X)
A∗=Aargmaxt=1∏Tpt(at∣X)
We can then collapse repeats and remove ϵ\epsilonϵ tokens to
get Y.Y.Y.

For many applications this heuristic works well, especially when most of the
probability mass is alloted to a single alignment. However, this approach can
sometimes miss easy to find outputs with much higher probability. The problem
is, it doesn’t take into account the fact that a single output can have many
alignments.

Here’s an example. Assume the alignments [a, a, ϵ\epsilonϵ]
and [a, a, a] individually have lower probability than [b, b, b]. But
the sum of their probabilities is actually greater than that of [b, b, b]. The
naive heuristic will incorrectly propose Y=Y =Y= [b] as
the most likely hypothesis. It should have chosen Y=Y =Y= [a].
To fix this, the algorithm needs to account for the fact that [a, a, a] and [a,
a, ϵ\epsilonϵ] collapse to the same output.

We can use a modified beam search to solve this. Given limited
computation, the modified beam search won’t necessarily find the
most likely Y.Y.Y. It does, at least, have
the nice property that we can trade-off more computation
(a larger beam-size) for an asymptotically better solution.

A regular beam search computes a new set of hypotheses at each input step.
The new set of hypotheses is generated from the previous set by extending each
hypothesis with all possible output characters and keeping only the top
candidates.

![](assets/beam_search.svg)

 A standard beam search algorithm with an alphabet of
 {ϵ,a,b}\{\epsilon, a, b\}{ϵ,a,b} and a beam size
 of three.

We can modify the vanilla beam search to handle multiple alignments mapping to
the same output. In this case instead of keeping a list of alignments in the
beam, we store the output prefixes after collapsing repeats and removing
ϵ\epsilonϵ characters. At each step of the search we accumulate
scores for a given prefix based on all the alignments which map to it.

 The CTC beam search algorithm with an output alphabet
 {ϵ,a,b}\{\epsilon, a, b\}{ϵ,a,b}
 and a beam size of three.

A proposed extension can map to two output prefixes if the character is a
repeat. This is shown at T=3T=3T=3 in the figure above
where ‘a’ is proposed as an extension to the prefix [a]. Both [a] and [a, a] are
valid outputs for this proposed extension.

When we extend [a] to produce [a,a], we only want include the part of the
previous score for alignments which end in ϵ.\epsilon.ϵ. Remember, the
ϵ\epsilonϵ is required between repeat characters. Similarly,
when we don’t extend the prefix and produce [a], we should only include the part
of the previous score for alignments which don’t end in ϵ.\epsilon.ϵ.

Given this, we have to keep track of two probabilities for each prefix
in the beam. The probability of all alignments which end in
ϵ\epsilonϵ and the probability of all alignments which don’t
end in ϵ.\epsilon.ϵ. When we rank the hypotheses at
each step before pruning the beam, we’ll use their combined scores.

The implementation of this algorithm doesn’t require much code, but it is
dense and tricky to get right. Checkout this
[gist](https://gist.github.com/awni/56369a90d03953e370f3964c826ed4b0)
for an example implementation in Python.

In some problems, such as speech recognition, incorporating a language model
over the outputs significantly improves accuracy. We can include the language
model as a factor in the inference problem.

Y∗=argmaxYY^\* \enspace = \enspace {\mathop{\text{argmax}}\limits\_{Y}}
 Y∗=Yargmax

p(Y∣X)⋅p(Y \mid X) \quad \cdotp(Y∣X)⋅

 The CTC conditional probability.

p(Y)α⋅p(Y)^\alpha \quad \cdotp(Y)α⋅

 The language model probability.

L(Y)βL(Y)^\betaL(Y)β

 The “word” insertion bonus.

The function L(Y)L(Y)L(Y) computes the length of
YYY in terms of the language model tokens and acts as a word
insertion bonus. With a word-based language model L(Y)L(Y)L(Y)
counts the number of words in Y.Y.Y. If we use a character-based
language model then L(Y)L(Y)L(Y) counts the number of characters
in Y.Y.Y. The language model scores are only included when a
prefix is extended by a character (or word) and not at every step of the
algorithm. This causes the search to favor shorter prefixes, as measured by
L(Y)L(Y)L(Y), since they don’t include as many language model
updates. The word insertion bonus helps with this. The parameters
α\alphaα and β\betaβ are usually set by
cross-validation.

The language model scores and word insertion term can be included in the
beam search. Whenever we propose to extend a prefix by a character, we can
include the language model score for the new character given the prefix so
far.

Properties of CTC
-----------------

We mentioned a few important properties of CTC so far. Here we’ll go
into more depth on what these properties are and what trade-offs they offer.

### Conditional Independence

One of the most commonly cited shortcomings of CTC is the conditional
independence assumption it makes.

![](assets/conditional_independence.svg)

 Graphical model for CTC.

The model assumes that every output is conditionally independent of
the other outputs given the input. This is a bad assumption for many
sequence to sequence problems.

Say we had an audio clip of someone saying “triple A”.
 Another valid transcription could
be “AAA”. If the first letter of the predicted transcription is ‘A’, then
the next letter should be ‘A’ with high probability and ‘r’ with low
probability. The conditional independence assumption does not allow for this.

![](assets/triple_a.svg)

 If we predict an ‘A’ as the first letter then the suffix ‘AA’ should get much
 more probability than ‘riple A’. If we predict ‘t’ first, the opposite
 should be true.

In fact speech recognizers using CTC don’t learn a language model over the
output nearly as well as models which are conditionally dependent.
 However, a separate language model can
be included and usually gives a good boost to accuracy.

The conditional independence assumption made by CTC isn’t always a bad
thing. Baking in strong beliefs over output interactions makes the model less
adaptable to new or altered domains. For example, we might want to use a speech
recognizer trained on phone conversations between friends to transcribe
customer support calls. The language in the two domains can be quite different
even if the acoustic model is similar. With a CTC acoustic model, we can easily
swap in a new language model as we change domains.

### Alignment Properties

The CTC algorithm is *alignment-free*. The objective function
marginalizes over all alignments. While CTC does make strong assumptions about
the form of alignments between XXX and YYY, the
model is agnostic as to how probability is distributed amongst them. In some
problems CTC ends up allocating most of the probability to a single alignment.
However, this isn’t guaranteed.

We could force the model to choose a single
alignment by replacing the sum with a max in the objective function,
p(Y∣X)=maxA∈AX,Y∏t=1Tp(at∣X).
p(Y \mid X) \enspace = \enspace \max\_{A \in \mathcal{A}\_{X,Y}} \enspace \prod\_{t=1}^T \; p(a\_t \mid X).
p(Y∣X)=A∈AX,Ymaxt=1∏Tp(at∣X).

As mentioned before, CTC only allows *monotonic* alignments. In
problems such as speech recognition this may be a valid assumption. For other
problems like machine translation where a future word in a target sentence
can align to an earlier part of the source sentence, this assumption is a
deal-breaker.

Another important property of CTC alignments is that they are
*many-to-one*. Multiple inputs can align to at most one output. In some
cases this may not be desirable. We might want to enforce a strict one-to-one
correspondence between elements of XXX and
Y.Y.Y. Alternatively, we may want to allow multiple output
elements to align to a single input element. For example, the characters
“th” might align to a single input step of audio. A character based CTC model
would not allow that.

The many-to-one property implies that the output can’t have more time-steps
than the input.

 If YYY has rrr consecutive
 repeats, then the length of YYY must be less than
 the length of XXX by 2r−1.2r - 1.2r−1.

This is usually not a problem for speech and handwriting recognition since the
input is much longer than the output. However, for other problems where
YYY is often longer than XXX, CTC just won’t
work.

CTC in Context
--------------

In this section we’ll discuss how CTC relates to other commonly used
algorithms for sequence modeling.

### HMMs

At a first glance, a Hidden Markov Model (HMM) seems quite different from
CTC. But, the two algorithms are actually quite similar. Understanding the
relationship between them will help us understand what advantages CTC has over
HMM sequence models and give us insight into how CTC could be changed for
various use cases.

Let’s use the same notation as before,
XXX is the input sequence and YYY
is the output sequence with lengths TTT and
UUU respectively. We’re interested in learning
p(Y∣X).p(Y \mid X).p(Y∣X). One way to simplify the problem is to apply
Bayes’ Rule:
p(Y∣X)∝p(X∣Y)p(Y).
p(Y \mid X) \; \propto \; p(X \mid Y) \; p(Y).
p(Y∣X)∝p(X∣Y)p(Y).

The p(Y)p(Y)p(Y) term can be any language model, so let’s focus on
p(X∣Y).p(X \mid Y).p(X∣Y). Like before we’ll let
A\mathcal{A}A be a set of allowed alignments between
XXX and Y.Y.Y. Members of
A\mathcal{A}A have length T.T.T.
Let’s otherwise leave A\mathcal{A}A unspecified for now. We’ll
come back to it later. We can marginalize over alignments to get
p(X∣Y)=∑A∈Ap(X,A∣Y).
p(X \mid Y)\; = \; \sum\_{A \in \mathcal{A}} \; p(X, A \mid Y).
p(X∣Y)=A∈A∑p(X,A∣Y).
To simplify notation, let’s remove the conditioning on YYY, it
will be present in every p(⋅).p(\cdot).p(⋅). With two assumptions we can
write down the standard HMM.

p(X)=p(X) \quad =p(X)=

 The probability of the input

∑A∈A∏t=1T\sum\_{A \in \mathcal{A}} \; \prod\_{t=1}^T∑A∈A∏t=1T

 Marginalizes over alignments

p(xt∣at)⋅p(x\_t \mid a\_t) \quad \cdotp(xt∣at)⋅

 The emission probability

p(at∣at−1)p(a\_t \mid a\_{t-1})p(at∣at−1)

 The transition probability

The first assumption is the usual Markov property. The state
ata\_tat is conditionally independent of all historic states given
the previous state at−1.a\_{t-1}.at−1. The second is that the observation
xtx\_txt is conditionally independent of everything given the
current state at.a\_t.at.

![](assets/hmm.svg)

 The graphical model for an HMM.

Now we can take just a few steps to transform the HMM into CTC and see how
the two models relate. First, let’s assume that the transition probabilities
p(at∣at−1)p(a\_t \mid a\_{t-1})p(at∣at−1) are uniform. This gives
p(X)∝∑A∈A∏t=1Tp(xt∣at).
p(X) \enspace \propto \enspace \sum\_{A \in \mathcal{A}} \enspace \prod\_{t=1}^T \; p(x\_t \mid a\_t).
p(X)∝A∈A∑t=1∏Tp(xt∣at).
There are only two differences from this equation and the CTC loss function.
The first is that we are learning a model of XXX given
YYY as opposed to YYY given X.X.X.
The second is how the set A\mathcal{A}A is produced. Let’s deal
with each in turn.

The HMM can be used with discriminative models which estimate p(a∣x).p(a \mid x).p(a∣x).
To do this, we apply Bayes’ rule and rewrite the model as
p(X)∝∑A∈A∏t=1Tp(at∣xt)p(xt)p(at)
p(X) \enspace \propto \enspace \sum\_{A \in \mathcal{A}} \enspace \prod\_{t=1}^T \; \frac{p(a\_t \mid x\_t)\; p(x\_t)}{p(a\_t)}
p(X)∝A∈A∑t=1∏Tp(at)p(at∣xt)p(xt)
∝∑A∈A∏t=1Tp(at∣xt)p(at).
\quad\quad\quad\propto \enspace \sum\_{A \in \mathcal{A}} \enspace \prod\_{t=1}^T \; \frac{p(a\_t \mid x\_t)}{p(a\_t)}.
∝A∈A∑t=1∏Tp(at)p(at∣xt).

If we assume a uniform prior over the states aaa and condition on all of
XXX instead of a single element at a time, we arrive at
p(X)∝∑A∈A∏t=1Tp(at∣X).
p(X) \enspace \propto \enspace \sum\_{A \in \mathcal{A}} \enspace \prod\_{t=1}^T \; p(a\_t \mid X).
p(X)∝A∈A∑t=1∏Tp(at∣X).

The above equation is essentially the CTC loss function, assuming the set
A\mathcal{A}A is the same. In fact, the HMM framework does not specify what
A\mathcal{A}A should consist of. This part of the model can be designed on a
per-problem basis. In many cases the model doesn’t condition on YYY and the
set A\mathcal{A}A consists of all possible length TTT sequences from the
output alphabet. In this case, the HMM can be drawn as an *ergodic* state
transition diagram in which every state connects to every other state. The
figure below shows this model with the alphabet or set of unique hidden states
as {a,b,c}.\{a, b, c\}.{a,b,c}.

In our case the transitions allowed by the model are strongly related to
Y.Y.Y. We want the HMM to reflect this. One possible model could
be a simple linear state transition diagram. The figure below shows this with
the same alphabet as before and Y=Y =Y= [a, b]. Another commonly
used model is the *Bakis* or left-right HMM. In this model any
transition which proceeds from the left to the right is allowed.

![](assets/ergodic_hmm.svg)

**Ergodic HMM:** Any node can be either a starting or
 final state.

![](assets/linear_hmm.svg)

**Linear HMM:** Start on the left, end on the right.

![](assets/ctc_hmm.svg)

**CTC HMM:** The first two nodes are the starting
 states and the last two nodes are the final states.

In CTC we augment the alphabet with ϵ\epsilonϵ and the HMM model allows a
subset of the left-right transitions. The CTC HMM has two start
states and two accepting states.

One possible source of confusion is that the HMM model differs for any unique
Y.Y.Y. This is in fact standard in applications such as speech recognition. The
state diagram changes based on the output Y.Y.Y. However, the functions which
estimate the observation and transition probabilities are shared.

Let’s discuss how CTC improves on the original HMM model. First, we can think
of the CTC state diagram as a special case HMM which works well for many
problems of interest. Incorporating the blank as a hidden state in the HMM
allows us to use the alphabet of YYY as the other hidden states. This model
also gives a set of allowed alignments which may be a good prior for some
problems.

Perhaps most importantly, CTC is discriminative. It models p(Y∣X)p(Y \mid
 X)p(Y∣X) directly, an idea that’s been important in the past with other
discriminative improvements to HMMs.
Discriminative training let’s us apply powerful learning algorithms like the
RNN directly towards solving the problem we care about.

### Encoder-Decoder Models

The encoder-decoder is perhaps the most commonly used framework for sequence
modeling with neural networks. These models have an encoder
and a decoder. The encoder maps the input sequence XXX into a
hidden representation. The decoder consumes the hidden representation and
produces a distribution over the outputs. We can write this as

H=encode(X)p(Y∣X)=decode(H).
\begin{aligned}
H\enspace &= \enspace\textsf{encode}(X) \\[.5em]
p(Y \mid X)\enspace &= \enspace \textsf{decode}(H).
\end{aligned}
Hp(Y∣X)=encode(X)=decode(H).

The encode(⋅)\textsf{encode}(\cdot)encode(⋅) and
decode(⋅)\textsf{decode}(\cdot)decode(⋅) functions are typically RNNs. The
decoder can optionally be equipped with an attention mechanism. The hidden
state sequence HHH has the same number of time-steps as the
input, T.T.T. Sometimes the encoder subsamples the input. If the
encoder subsamples the input by a factor sss then
HHH will have T/sT/sT/s time-steps.

We can interpret CTC in the encoder-decoder framework. This is helpful to
understand the developments in encoder-decoder models that are applicable to
CTC and to develop a common language for the properties of these
models.

**Encoder:** The encoder of a CTC model can be just about any
encoder we find in commonly used encoder-decoder models. For example the
encoder could be a multi-layer bidirectional RNN or a convolutional network.
There is a constraint on the CTC encoder that doesn’t apply to the others. The
input length cannot be sub-sampled so much that T/sT/sT/s
is less than the length of the output.

**Decoder:** We can view the decoder of a CTC model as a simple
linear transformation followed by a softmax normalization. This layer should
project all TTT steps of the encoder output
HHH into the dimensionality of the output alphabet.

We mentioned earlier that CTC makes a conditional independence assumption over
the characters in the output sequence. This is one of the big advantages that
other encoder-decoder models have over CTC — they can model the
dependence over the outputs. However in practice, CTC is still more commonly
used in tasks like speech recognition as we can partially overcome the
conditional independence assumption by including an external language model.

Practitioner’s Guide
--------------------

So far we’ve mostly developed a conceptual understanding of CTC. Here we’ll go
through a few implementation tips for practitioners.

**Software:** Even with a solid understanding of CTC, the
implementation is difficult. The algorithm has several edge cases and a fast
implementation should be written in a lower-level programming language.
Open-source software tools make it much easier to get started:

* Baidu Research has open-sourced
 [warp-ctc](https://github.com/baidu-research/warp-ctc). The
 package is written in C++ and CUDA. The CTC loss function runs on either
 the CPU or the GPU. Bindings are available for Torch, TensorFlow and
 [PyTorch](https://github.com/awni/warp-ctc).
* TensorFlow has built in
 [CTC loss](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss)
 and [CTC beam search](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_beam_search_decoder)
 functions for the CPU.
* Nvidia also provides a GPU implementation of CTC in
 [cuDNN](https://developer.nvidia.com/cudnn) versions 7 and up.

**Numerical Stability:** Computing the CTC loss naively is
numerically unstable. One method to avoid this is to normalize the
α\alphaα’s at each time-step. The original publication
has more detail on this including the adjustments to the gradient.
 In practice this works well enough
for medium length sequences but can still underflow for long sequences.
A better solution is to compute the loss function in log-space with the
log-sum-exp trick.

When computing the sum of two probabilities in log space use the identity
log(ea+eb)=max{a,b}+log(1+e−∣a−b∣)
\log(e^a + e^b) = \max\{a, b\} + \log(1 + e^{-|a-b|})
log(ea+eb)=max{a,b}+log(1+e−∣a−b∣)
Most programming languages have a stable function to compute
log(1+x)\log(1 + x)log(1+x) when
xxx is close to zero.

Inference should also be done in log-space using the log-sum-exp trick.

**Beam Search:** There are a couple of good tips to know about
when implementing and using the CTC beam search.

The correctness of the beam search can be tested as follows.

1. Run the beam search algorithm on an arbitrary input.
2. Save the inferred output Y¯\bar{Y}Y¯ and the corresponding
 score c¯.\bar{c}.c¯.
3. Compute the actual CTC score ccc for
 Y¯.\bar{Y}.Y¯.
4. Check that c¯≈c\bar{c} \approx cc¯≈c with the former being no
 greater than the latter. As the beam size increases the inferred output
 Y¯\bar{Y}Y¯ may change, but the two numbers should grow
 closer.

A common question when using a beam search decoder is the size of the beam
to use. There is a trade-off between accuracy and runtime. We can check if the
beam size is in a good range. To do this first compute the CTC score for the
inferred output ci.c\_i.ci. Then compute the CTC score for the ground
truth output cg.c\_g.cg. If the two outputs are not the same, we
should have cg<ci.c\_g \lt c\_i.cg<ci. If ci<<cgc\_i << c\_gci<<cg then
the ground truth output actually has a higher probability under the model and
the beam search failed to find it. In this case a large increase to the beam
size may be warranted.

A visual guide to Connectionist Temporal Classification, an algorithm used to train deep neural networks in speech recognition, handwriting recognition and other sequence problems.

Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition

Exploring Neural Transducers for End-to-End Speech Recognition

Large scale discriminative training of hidden Markov models for speech recognition

Connectionist Temporal Classification : Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Phone recognition on the TIMIT database

Supervised Sequence Labelling with Recurrent Neural Networks

Towards End-To-End Speech Recognition with Recurrent Neural Networks

Deep Speech: Scaling up end-to-end speech recognition

A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks

LipNet: End-to-End Sentence-level Lipreading

Connectionist Temporal Modeling for Weakly Supervised Action Labeling

An application of recurrent neural networks to discriminative keyword spotting

An End-to-End Architecture for Keyword Spotting and Voice Activity Detection

Sequence Transduction with Recurrent Neural Networks

Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling

Sequence Modeling via Segmentations

Segmental Recurrent Neural Networks

Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Sequence to Sequence Learning with Neural Networks

Attention and Augmented Recurrent Neural Networks


Feature Visualization

There is a growing sense that neural networks need to be interpretable to humans.
 The field of neural network interpretability has formed in response to these concerns.
 As it matures, two major threads of research have begun to coalesce: feature visualization and attribution.

![](images/objectives/neuron.png)
![](images/objectives/channel.png)

**Feature visualization** answers questions about what a network — or parts of a network — are looking for by generating examples.

![](images/attribution-1.png)
![](images/attribution-2.jpg)

**Attribution**

 As a young field, neural network interpretability does not yet have standardized terminology.
 Attribution has gone under many different names in the literature — including “feature visualization”! — but recent work seems to prefer terms like “attribution” and “saliency maps”.

 studies what part of an example is responsible for the network activating a particular way.

 This article focuses on feature visualization.
 While feature visualization is a powerful tool, actually getting it to work involves a number of details.
 In this article, we examine the major issues and explore common approaches to solving them.
 We find that remarkably simple methods can produce high-quality visualizations. Along the way we introduce a few tricks for exploring variation in what neurons react to, how they interact, and how to improve the optimization process.

---

Feature Visualization by Optimization
-------------------------------------

 Neural networks are, generally speaking, differentiable with respect to their inputs.
 If we want to find out what kind of input would cause a certain behavior — whether that’s an internal neuron firing or the final output behavior — we can use derivatives to iteratively tweak the input
 towards that goal .

Starting from random noise, we optimize an image to activate a particular neuron (layer mixed4a, unit 11).

 While conceptually simple, there are subtle challenges in getting the optimization to work. We will explore them, as well as common approaches to tackle them in the section ”[The Enemy of Feature Visualization](#enemy-of-feature-vis)″.

### Optimization Objectives

 What do we want examples of?
 This is the core question in working with examples, regardless of whether we’re searching through a dataset to find the examples, or optimizing images to create them from scratch.
 We have a wide variety of options in what we search for:

 #optimization-objectives .objectives {
 grid-column: text;
 grid-template-columns: repeat(1, 1fr);
 }

 #optimization-objectives .objective {
 display: grid;
 grid-template-columns: repeat(2, 1fr) 1.1fr;
 }

 #optimization-objectives .objectives figcaption {
 padding: 4px 8px;
 word-wrap: break-word;
 word-break: break-word;
 }

 #optimization-objectives .objective .objective-icon {
 padding: 8px;
 }

 @media (min-width: 512px) {
 #optimization-objectives .objectives {
 grid-template-columns: repeat(5, 1fr);
 grid-column-gap: 16px;
 }

 #optimization-objectives .objective {
 display: flex;
 flex-flow: column;
 }

 #optimization-objectives .objectives figcaption {
 padding: 0;
 padding-top: 4px;
 word-wrap: break-word;
 word-break: break-word;
 }
 }

Different **optimization objectives** show what different parts of a network are looking for.

`**n**` layer index

`**x,y**` spatial position

`**z**` channel index

`**k**` class index

![](images/objectives/neuron.png)
**Neuron**
`layern[x,y,z]`

![](images/objectives/channel.png)
**Channel**
`layern[:,:,z]`

![](images/objectives/layer.png)
**Layer**/DeepDream
`layern[:,:,:]2`

softmax

![](images/objectives/logits.png)
**Class Logits**
`pre_softmax[k]`

softmax

![](images/objectives/logits_post.png)
**Class Probability**
`softmax[k]`

 If we want to understand individual features, we can search for examples where they have high values — either for a *neuron* at an individual position, or for an entire *channel*.
 We used the channel objective to create most of the images in this article.

 If we want to understand a *layer* as a whole, we can use the DeepDream objective , searching for images the layer finds “interesting.”

 And if we want to create examples of output classes from a classifier, we have two options — optimizing *class logits* before the softmax or optimizing *class probabilities* after the softmax.
 One can see the logits as the evidence for each class, and the probabilities as the likelihood of each class given the evidence.
 Unfortunately, the easiest way to increase the probability softmax gives to a class is often to make the alternatives unlikely rather than to make the class of interest likely .
 From our experience, optimizing pre-softmax logits produces images of better visual quality.

 While the standard explanation is that maximizing probability doesn’t work very well because you can just push down evidence for other classes, an alternate hypothesis is that it’s just harder to optimize through the softmax function. We understand this has sometimes been an issue in adversarial examples, and the solution is to optimize the LogSumExp of the logits instead. This is equivalent to optimizing softmax but generally more tractable. Our experience was that the LogSumExp trick doesn’t seem better than dealing with the raw probabilities.

 Regardless of why that happens, it can be fixed by very strong regularization with generative models. In this case the probabilities can be a very principled thing to optimize.

 The objectives we’ve mentioned only scratch the surface of possible objectives — there are a lot more that one could try.
 Of particular note are the objectives used in style transfer , which can teach us about the kinds of style and content a network understands,
 and objectives used in optimization-based model inversion , which help us understand what information a model keeps and what it throws away.
 We are only at the beginning of understanding which objectives are interesting, and there is a lot of room for more work in this area.

### Why visualize by optimization?

 Optimization can give us an example input that causes the desired behavior — but why bother with that?
 Couldn’t we just look through the dataset for examples that cause the desired behavior?

 It turns out that optimization approach can be a powerful way to understand what a model is really looking for,
 because it separates the things causing behavior from things that merely correlate with the causes.
 For example, consider the following neurons visualized with dataset examples and optimization:

 Optimization also has the advantage of flexibility.
 For example, if we want to study how neurons jointly represent information,
 we can easily ask how a particular example would need to be different for an additional neuron to activate.
 This flexibility can also be helpful in visualizing how features evolve as the network trains.
 If we were limited to understanding the model on the fixed examples in our dataset, topics like these ones would be much harder to explore.

 On the other hand, there are also significant challenges to visualizing features with optimization.
 In the following sections we’ll examine techniques to get diverse visualizations, understand how neurons interact, and avoid high frequency artifacts.

---

Diversity
---------

 Do our examples show us the full picture?
 When we create examples by optimization, this is something we need to be very careful of.
 It’s entirely possible for genuine examples to still mislead us by only showing us one “facet” of what a feature represents.

 Dataset examples have a big advantage here.
 By looking through our dataset, we can find diverse examples.
 It doesn’t just give us ones activating a neuron intensely:
 we can look across a whole spectrum of activations to see what activates the neuron to different extents.

 In contrast, optimization generally gives us just one extremely positive example — and if we’re creative, a very negative example as well.
 Is there some way that optimization could also give us this diversity?

### Achieving Diversity with Optimization

 A given feature of a network may respond to a wide range of inputs.
 On the class level, for example, a classifier that has been trained to recognize dogs should recognize both closeups of their faces as well as wider profile images — even though those have quite different visual appearances.
 Early work by Wei *et al.* attempts to demonstrate this “intra-class” diversity by recording activations over the entire training set, clustering them and optimizing for the cluster centroids, revealing the different facets of a class that were learned.

 A different approach by Nguyen, Yosinski, and collaborators was to search through the dataset for diverse examples and use those as starting points for the optimization process .
 The idea is that this initiates optimization in different facets of the feature so that the resulting example from optimization will demonstrate that facet.
 In more recent work, they combine visualizing classes with a generative model, which they can sample for diverse examples .
 Their first approach had limited success, and while the generative model approach works very well — we’ll discuss it more in the section on regularization under [learned priors](#learned-priors) — it can be a bit tricky.

 We find there’s a very simple way to achieve diversity: adding a “diversity term”

 For this article we use an approach based on ideas from artistic style transfer. Following that work, we begin by computing the Gram matrix GGG of the channels, where Gi,jG\_{i,j}Gi,j is a the dot product between the (flattened) response of filter iii and filter jjj:
 Gi,j=∑x,ylayern[x, y, i]⋅layern[x, y, j]
 G\_{i,j} = \sum\_{x,y} \text{layer}\_n\text{[x, y, i]} \cdot \text{layer}\_n\text{[x, y, j]}
 Gi,j=x,y∑layern[x, y, i]⋅layern[x, y, j]
 From this, we compute the diversity term: the negative pairwise cosine similarity of pairs of visualizations.
 Cdiversity=−∑a∑b≠a vec(Ga)⋅vec(Gb)∣∣vec(Ga)∣∣ ∣∣vec(Gb)∣∣
 C\_{\text{diversity}} = - \sum\_{a} \sum\_{b\neq a} ~ \frac{\text{vec}(G\_a) \cdot \text{vec}(G\_b)}{||\text{vec}(G\_a)||~||\text{vec}(G\_b)||}
 Cdiversity=−a∑b≠a∑ ∣∣vec(Ga)∣∣ ∣∣vec(Gb)∣∣vec(Ga)⋅vec(Gb)
 We then maximize the diversity term jointly with the regular optimization objective.
  to one’s objective that pushes multiple examples to be different from each other.
 The diversity term can take a variety of forms, and we don’t have much understanding of their benefits yet.
 One possibility is to penalize the cosine similarity of different examples.
 Another is to use ideas from style transfer  to force the feature to be displayed in different styles.

 In lower level neurons, a diversity term can reveal the different facets a feature represents:

![](images/diversity/mixed4a_97_optimized.png)
Simple Optimization

![](images/diversity/mixed4a_97_diversity.png)

 Optimization with diversity reveals four different, curvy facets. *Layer mixed4a, Unit 97*

![](images/diversity/mixed4a_97_examples.jpg)
Dataset examples

 Diverse feature visualizations allow us to more closely pinpoint what activates a neuron, to the degree that we can make, and — by looking at dataset examples — *check* predictions about what inputs will activate the neuron.

 For example, let’s examine this simple optimization result.

![](images/diversity/mixed4a_143_optimized.png)
Simple optimization

 Looking at it in isolation one might infer that this neuron activates on the top of dog heads, as the optimization shows both eyes and only downward curved edges.
 Looking at the optimization with diversity however, we see optimization results which don’t include eyes, and also one which includes upward curved edges. We thus have to broaden our expectation of what this neuron activates on to be mostly about the fur texture. Checking this hypothesis against dataset examples shows that is broadly correct. Note the spoon with a texture and color similar enough to dog fur for the neuron to activate.

![](images/diversity/mixed4a_143_diversity.png)

 Optimization with diversity. *Layer mixed4a, Unit 143*

![](images/diversity/mixed4a_143_examples.jpg)
Dataset examples

 The effect of diversity can be even more striking in higher level neurons, where it can show us different types of objects that stimulate a neuron.
 For example, one neuron responds to different kinds of balls, even though they have a variety of appearances.

![](images/diversity/mixed5a_9_optimized.png)
Simple Optimization

![](images/diversity/mixed5a_9_diversity.png)
Optimization with diversity reveals multiple types of balls. *Layer mixed5a, Unit 9*

![](images/diversity/mixed5a_9_examples.jpg)
Dataset examples

 This simpler approach has a number of shortcomings:
 For one, the pressure to make examples different can cause unrelated artifacts (such as eyes) to appear.
 Additionally, the optimization may make examples be different in an unnatural way.
 For example, in the above example one might want to see examples of soccer balls clearly separated from other types of balls like golf or tennis balls.
 Dataset based approaches such as Wei *et al.* can split features apart more naturally — however they may not be as helpful in understanding how the model will behave on different data.

 Diversity also starts to brush on a more fundamental issue: while the examples above represent a mostly coherent idea, there are also neurons that represent strange mixtures of ideas.
 Below, a neuron responds to two types of animal faces, and also to car bodies.

![](images/diversity/mixed4e_55_optimized.png)
Simple Optimization

![](images/diversity/mixed4e_55_diversity.png)
Optimization with diversity show cats, foxes, but also cars. *Layer mixed4e, Unit 55*

![](images/diversity/mixed4e_55_examples.jpg)
Dataset examples

 Examples like these suggest that neurons are not necessarily the right semantic units for understanding neural nets.

---

Interaction between Neurons
---------------------------

 If neurons are not the right way to understand neural nets, what is?
 In real life, combinations of neurons work together to represent images in neural networks.
 A helpful way to think about these combinations is geometrically: let’s define *activation space* to be all possible combinations of neuron activations.
 We can then think of individual neuron activations as the *basis vectors* of this activation space. Conversely, a combination of neuron activations is then just a vector in this space.

 This framing unifies the concepts “neurons” and “combinations of neurons” as “vectors in activation space”. It allows us to ask: Should we expect the directions of the basis vectors to be any more interpretable than the directions of other vectors in this space?

 Szegedy *et al.* found that random directions seem just as meaningful as the directions of the basis vectors.
 More recently Bau, Zhou *et al.* found the directions of the basis vectors to be interpretable more often than random directions.
 Our experience is broadly consistent with both results; we find that random directions often seem interpretable, but at a lower rate than basis directions.

Dataset examples and optimized examples of **random directions** in activation space. The directions shown here were hand-picked for interpretability.

 We can also define interesting directions in activation space by doing arithmetic on neurons.
 For example, if we add a “black and white” neuron to a “mosaic” neuron, we obtain a black and white version of the mosaic.
 This is reminiscent of semantic arithmetic of word embeddings as seen in Word2Vec or generative models’ latent spaces.

By jointly optimizing two neurons we can get a sense of how they interact.

 These examples show us how neurons jointly represent images.
 To better understand how neurons interact, we can also interpolate between them.
 The optimization objective is a linear interpolation between the individual channel objectives. To get the interpolations to look better, we also add a small alignment objective that encourages lower layer activations to be similar. We additionally use a combination of separate and shared image parameterizations to make it easier for the optimization algorithm to cause objects to line up, while still giving it the freedom to create any image it needs to.
 This is similar to interpolating in the latent space of generative models.

 This is only starting to scratch the surface of how neurons interact.
 The truth is that we have almost no clue how to select meaningful directions, or whether there even exist particularly meaningful directions.
 Independent of finding directions, there are also questions on how directions interact — for example, interpolation can show us how a small number of directions interact, but in reality there are hundreds of directions interacting.

---

The Enemy of Feature Visualization
----------------------------------

 If you want to visualize features, you might just optimize an image to make neurons fire.
 Unfortunately, this doesn’t really work.
 Instead, you end up with a kind of neural network optical illusion — an image full of noise and nonsensical high-frequency patterns that the network responds strongly to.

Even if you carefully tune learning rate, you’ll get noise.

Optimization results are enlarged to show detail and artifacts.

 These patterns seem to be the images kind of cheating, finding ways to activate neurons that don’t occur in real life.
 If you optimize long enough, you’ll tend to see some of what the neuron genuinely detects as well,
 but the image is dominated by these high frequency patterns.
 These patterns seem to be closely related to the phenomenon of adversarial examples .

 We don’t fully understand why these high frequency patterns form,
 but an important part seems to be strided convolutions and pooling operations, which create high-frequency patterns in the gradient .

Each **strided convolution or pooling** creates checkerboard patterns in the gradient magnitudes when we backprop through it.

 These high-frequency patterns show us that, while optimization based visualization’s freedom from constraints is appealing, it’s a double-edged sword.
 Without any constraints on images, we end up with adversarial examples.
 These are certainly interesting, but if we want to understand how these models work in real life, we need to somehow move past them…

### The Spectrum of Regularization

 Dealing with this high frequency noise has been one of the primary challenges and overarching threads of feature visualization research.
 If you want to get useful visualizations, you need to impose a more natural structure using some kind of prior, regularizer, or constraint.

 In fact, if you look at most notable papers on feature visualization, one of their main points will usually be an approach to regularization.
 Researchers have tried a lot of different things!

 We can think of all of these approaches as living on a spectrum, based on how strongly they regularize the model.
 On one extreme, if we don’t regularize at all, we end up with adversarial examples.
 On the opposite end, we search over examples in our dataset and run into all the limitations we discussed earlier.
 In the middle we have three main families of regularization options.

 #feature-vis-history .row {
 }
 #feature-vis-history .line {
 width: 900px;
 }
 #feature-vis-history .row .info {
 display: inline-block;
 width: 320px;
 height: 56px;
 vertical-align: top;
 }
 #feature-vis-history .row .info img {
 width: 56px;
 height: 56px;
 border-radius: 5px;
 background: #EEE;
 }
 #feature-vis-history figcaption {
 line-height: 16px;
 vertical-align: top;
 }
 #feature-vis-history .row .info figcaption {
 width: 250px;
 margin-left: 8px;
 display: inline-block;
 }
 #feature-vis-history .row.header-row .info figcaption {
 margin-left: 0;
 }
 #feature-vis-history .header-row .sub-headers {
 display: inline-block;
 margin-top: 16px;
 margin-bottom: 10px;
 vertical-align: top;
 }
 #feature-vis-history .header-row .sub-headers figcaption {
 width: 100px; /\*104px;\*/
 display: inline-block;
 word-wrap: break-word;
 word-break: keep-all;
 }

 #feature-vis-history .row .spacer {
 display: inline-block;
 height: 48px;
 width: 45px;
 }

 #feature-vis-history .info p {
 margin-bottom: 4px;
 }

 #feature-vis-history .info .paper-title {
 font-weight: bold;
 }

 #feature-vis-history .info .paper-text {

 }

 @media (max-width: 1100px) {
 #feature-vis-history .row .spacer {
 display: inline-block;
 height: 48px;
 width: 24px;
 }
 #feature-vis-history .line {
 width: 750px;
 }
 }

 #feature-vis-history .row .category-check-container {
 /\*vertical-align: middle;\*/
 display: inline-block;
 margin: 8px;
 margin-left: 0px;
 margin-right: 70px;
 width: 32px;
 height: 32px;
 border-radius: 5px;
 border: 1px solid #CCC;
 }
 #feature-vis-history .row .category-check-container .category-check {
 margin: 6px;
 width: 20px;
 height: 20px;
 border-radius: 6px;
 }
 #feature-vis-history .row .category-check-container .set {
 background: #909092;
 }
 #feature-vis-history .line {
 margin-top: 4px;
 margin-bottom: 4px;
 margin-left: 67px;
 height: 1px;
 background: #EAEAEA;
 }

**Weak Regularization**
 avoids misleading correlations, but is less connected to real use.
**Unregularized**
**FrequencyPenalization**
**TransformationRobustness**

**Strong Regularization**
 gives more realistic examples at risk of misleading correlations.
**LearnedPrior**
**DatasetExamples**
![](images/prior_vis_icons/Erhan2009.png)
Erhan, *et al.*, 2009

Introduced core idea. Minimal regularization.

![](images/prior_vis_icons/Szegedy2013.png)
Szegedy, *et al.*, 2013

Adversarial examples. Visualizes with dataset examples.

![](images/prior_vis_icons/Mahendran2015.png)
Mahendran & Vedaldi, 2015

Introduces total variation regularizer. Reconstructs input from representation.

![](images/prior_vis_icons/Nguyen2015.png)
Nguyen, *et al.*, 2015

Explores counterexamples. Introduces image blurring.

![](images/prior_vis_icons/Mordvintsev2015.png)
Mordvintsev, *et al.*, 2015

Introduced jitter & multi-scale. Explored GMM priors for classes.

![](images/prior_vis_icons/Oygard2015.png)
Øygard, *et al.*, 2015

Introduces gradient blurring.
(Also uses jitter.)

![](images/prior_vis_icons/Tyka2016.png)
Tyka, *et al.*, 2016

Regularizes with bilateral filters.
(Also uses jitter.)

![](images/prior_vis_icons/Mordvintsev2016.png)
Mordvintsev, *et al.*, 2016

Normalizes gradient frequencies.
(Also uses jitter.)

![](images/prior_vis_icons/Nguyen2016a.png)
Nguyen, *et al.*, 2016

Paramaterizes images with GAN generator.

![](images/prior_vis_icons/Nguyen2016b.png)
Nguyen, *et al.*, 2016

Uses denoising autoencoder prior to make a generative model.

### Three Families of Regularization

 Let’s consider these three intermediate categories of regularization in more depth.

**Frequency penalization** directly targets the high frequency noise these methods suffer from.
 It may explicitly penalize variance between neighboring pixels (total variation) , or implicitly penalize high-frequency noise by blurring the image each optimization step .
 If we think about blurring in Fourier space, it is equivalent to adding a scaled L2 penalty to the objective, penalizing each Fourier-component based on its frequency.
 Unfortunately, these approaches also discourage legitimate high-frequency features like edges along with noise.
 This can be slightly improved by using a bilateral filter, which preserves edges, instead of blurring .

 (Some work uses similar techniques to reduce high frequencies in the gradient before they accumulate in the visualization .
 These techniques are in some ways very similar to the above and in some ways radically different — we’ll examine them in the next section, [Preconditioning and Parameterization](#preconditioning).)

Frequency penalization directly targets high frequency noise

**Transformation robustness** tries to find examples that still activate the optimization target highly even if we slightly transform them.
 Even a small amount seems to be very effective in the case of images ,
 especially when combined with a more general regularizer for high-frequencies .
 Concretely, this means that we stochastically jitter, rotate or scale the image before applying the optimization step.

Stochastically transforming the image before applying the optimization step suppresses noise

**Learned priors.**
 Our previous regularizers use very simple heuristics to keep examples reasonable.
 A natural next step is to actually learn a model of the real data and try to enforce that.
 With a strong model, this becomes similar to searching over the dataset.
 This approach produces the most photorealistic visualizations, but it may be unclear what came from the model being visualized and what came from the prior.

 One approach is to learn a generator that maps points in a latent space to examples of your data,
 such as a GAN or VAE,
 and optimize within that latent space .
 An alternative approach is to learn a prior that gives you access to the gradient of probability;
 this allows you to jointly optimize for the prior along with your objective .
 When one optimizes for the prior and the probability of a class, one recovers a generative model of the data conditioned on that particular class.
 Finally, Wei *et al.* approximate a generative model prior, at least for the color distribution, by penalizing distance between patches of the output and the nearest patches retrieved from a database of image patches collected from the training data.

---

Preconditioning and Parameterization
------------------------------------

 In the previous section, we saw a few methods  that reduced high frequencies *in the gradient* rather than the visualization itself.
 It’s not clear this is really a regularizer:
 it resists high frequencies, but still allows them to form when the gradient consistently pushes for it.
 If it isn’t a regularizer, what does transforming the gradient like this do?

 Transforming the gradient like this is actually quite a powerful tool — it’s called “preconditioning” in optimization.
 You can think of it as doing steepest descent to optimize the same objective,
 but in another parameterization of the space or under a different notion of distance.

 Gradient blurring is equivalent to gradient descent in a different parameterization of image space, where high frequency dimensions are stretched to make moving in those directions slower. Gradient Laplacian Pyramid normalization  is a kind of adaptive learning rate approach in the same space.

 This changes which direction of descent will be steepest, and how fast the optimization moves in each direction, but it does not change what the minimums are.
 If there are many local minima, it can stretch and shrink their basins of attraction, changing which ones the optimization process falls into.
 As a result, using the right preconditioner can make an optimization problem radically easier.

 How can we choose a preconditioner that will give us these benefits?
 A good first guess is one that makes your data decorrelated and whitened.
 In the case of images this means doing gradient descent in the Fourier basis,

 This points to a profound fact about the Fourier transform.
 As long as a correlation is consistent across spatial positions — such as the correlation between a pixel and its left neighbor being the same across all positions of an image — the Fourier coefficients will be independent variables.
 To see this, note that such a spatially consistent correlation can be expressed as a convolution, and by the convolution theorem becomes pointwise multiplication after the Fourier transform.

 with frequencies scaled so that they all have equal energy.

 Note that we have to be careful to get the colors to be decorrelated, too. The Fourier transforms decorrelates spatially, but a correlation will still exist between colors.
 To address this, we explicitly measure the correlation between colors in the training set and use a Cholesky decomposition to decorrelate them. Compare the directions of steepest decent before and after decorrelating colors:

![](images/correlated_colors.jpeg)

 Correlated Colors

![](images/decorrelated_colors.jpeg)

 Decorrelated Colors

 Let’s see how using different measures of distance changes the direction of steepest descent.
 The regular L2 gradient can be quite different from the directions of steepest descent in the L∞ metric or in the decorrelated space:

 Three directions of steepest descent under different notions of distance

 All of these directions are valid descent directions for the same objective,
 but we can see they’re radically different.
 Notice that optimizing in the decorrelated space reduces high frequencies,
 while using L∞ increases them.

 Using the decorrelated descent direction results in quite different visualizations.
 It’s hard to do really fair comparisons because of hyperparameters, but the
 resulting visualizations seem a lot better — and develop faster, too.

Combining the preconditioning and transformation robustness improves quality even further

 (Unless otherwise noted, the images in this article were optimizing in the decorrelated space and a suite of transformation robustness techniques.

 Images were optimized for 2560 steps in a color-decorrelated fourier-transformed space, using Adam at a learning rate of 0.05.
 We used each of following transformations in the given order at each step of the optimization:

 • Padding the input by 16 pixels to avoid edge artifacts

 • Jittering by up to 16 pixels

 • Scaling by a factor randomly selected from this list: 1, 0.975, 1.025, 0.95, 1.05

 • Rotating by an angle randomly selected from this list; in degrees: -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5

 • Jittering a second time by up to 8 pixels

 • Cropping the padding

)

 Is the preconditioner merely accelerating descent, bringing us to the same place
 normal gradient descent would have brought us if we were patient enough?
 Or is it also regularizing, changing which local minima we get attracted to?
 It’s hard to tell for sure.
 On the one hand, gradient descent seems to continue improving as you exponentially increase the number of optimization steps — it hasn’t converged, it’s just moving very slowly.
 On the other hand, if you turn off all other regularizers, the preconditioner seems to reduce high-frequency patterns.

---

Conclusion
----------

 Neural feature visualization has made great progress over the last few years.
 As a community, we’ve developed principled ways to create compelling visualizations.
 We’ve mapped out a number of important challenges and found ways of addressing them.

 In the quest to make neural networks interpretable, feature visualization
 stands out as one of the most promising and developed research directions.
 By itself, feature visualization will never give a completely satisfactory
 understanding. We see it as one of the fundamental building blocks that,
 combined with additional tools, will empower humans to understand these systems.

 There remains still a lot of important work to be done in improving feature visualization.
 Some issues that stand out include understanding neuron interaction, finding which units are most meaningful for understanding neural net activations, and giving a holistic view of the facets of a feature.

Chris Olah
Alexander Mordvintsev
Ludwig Schubert

How neural networks build up their understanding of images

Going deeper with convolutions

Imagenet: A large-scale hierarchical image database

Visualizing higher-layer features of a deep network

Inceptionism: Going deeper into neural networks

Deep inside convolutional networks: Visualising image classification models and saliency maps

A neural algorithm of artistic style

Understanding deep image representations by inverting them

Understanding Intra-Class Knowledge Inside {CNN}

Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks

Plug & play generative networks: Conditional iterative generation of images in latent space

Intriguing properties of neural networks

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Deconvolution and checkerboard artifacts

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Visualizing GoogLeNet Classes

Class visualization with bilateral filters

DeepDreaming with TensorFlow

Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

Tensorflow: Large-scale machine learning on heterogeneous distributed systems