Learn how to visualize your CNTK based neural network

When you're building complex neural networks you often want to visualize them. Visualization offers a great way to debug things like performance bottlenecks, because parts of your network get duplicated by CNTK. Luckely, it is quite easy to visualize a neural network in CNTK.

In this post I will show you how you can visualize complex neural networks in CNTK so that you get insights in how the layers of your network are connected.

Once upon a time there was a neural network

As part of an experiment I'm working on a neural network for counting people in the crowd. The neural network I use is a residual network. This kind of neural network is good at recognizing patterns in images.

It's a rather complex neural network. And to make matters worse, I've chosen to perform multiple tasks at once in my neural network. In short, it means that I have a neural network that has multiple output layers connected to the same set of hidden layers.

I know it should have a structure like this:

<input-layer>
<residual-block>
<residual-block>
<residual-block>
<pooling-layer>
<output 1><output 2><output 3>

I use the following python code to write my neural network.

input_var = C.input_variable((3,320,240), name='input')

def ResidualBlock(filter_size, num_filters):
    def generate_residual_block(input_layer):
        # The first bit of a residual block is a set of convolution filters.
        # We use padding here to keep the shape intact.
        conv_block = For(range(2), lambda: [
            BatchNormalization(),
            Convolution(filter_size, num_filters, pad=True, activation=None)
        ])(input_layer)
        
        # the output of a residual block is the sum of the convolution filter and the input layer.
        # a ReLU activation function is used for this part of the block.
        output = conv_block + input_layer
        output = relu(output)

        return output
    
    return generate_residual_block

# The core module with residual blocks
core_module = Sequential([
    Convolution((5,5), 64, activation=None, pad=True),
    For(range(3), lambda: [
        ResidualBlock((5,5), 64)
    ]),
])

# Pooling layer
intermediate_results = MaxPooling((3,3))(core_module)

# Output 1
people_count = Sequential([
    Dense(64),
    Dense(1, activation=relu)
])(intermediate_results)

# Output 2
violence = Sequential([
    Dense(64),
    Dense(2, activation=softmax)
])(intermediate_results)

# Output 3
density_level = Sequential([
    Dense(64),
    Dense(1, activation=relu)
])(intermediate_results)

# The final model
model = combine([density_level, people_count, violence])(input_var)

That's a lot of code, with a high risk of generating a crappy network. Especially the connection from the hidden layers to the output layers is rather complex.

If I get the connection between the intermediate_results layer and the rest of the outputs wrong, the Microsoft Cognitive Toolkit (CNTK) library will generate duplicate layers. Resulting not only in slower training times, but also generating bad results.

Visualize it

So how do you know that all layers connect in the right way and CNTK doesn't duplicate any of my layers? You need to visualize it.

CNTK includes a graph plot function in the logging package that you can use to visualize models. It is based on Graphviz. A free graph visualization tool.

Before you can visualize your model, make sure you have Graphviz installed. You can download a version here: https://graphviz.gitlab.io/_pages/Download/Download_windows.html

Next add the binaries folder in the Graphviz installation folder to your PATH variable. Mine is installed in C:\Program Files (x86)\Graphviz2.38\bin so I add that to my path.

Now you can visualize your model. For this you need to use the following code:

import cntk as C

C.logging.graph.plot(model, 'model.png')

First import the cntk package and then call the logging.graph.plot function with the instance of your model and a filename.

This generates a picture that looks like this:

Now you can check the neural network for any bad connections or weird names.
As you can see in the picture, my neural network looks as expected. Except there are a few naming issues with the output layers.

Try it yourself

The visualization logger of CNTK makes it easier to see what your neural network looks like. But that is not all. You can also use this to generate pretty pictures to explain the network to your colleagues or other people.

Give it a spin and let me know what you think!