Kareem Nassar

Coder and Observer

GitHub
AngelList
Linkedin
Android.mk

My first neural network using TensorFlow

I'm currently taking an online course called Creative Applications of Deep Learning with TensorFlow. For this class's second project, I am attempting to train a neural network using TensorFlow to draw this:


tf.reset_default_graph()
# Our input, will take row and column
X = tf.placeholder(tf.float32, shape=(None, 2))

# Our output: RGB vector
Y = tf.placeholder(tf.float32, shape=(None, 3))

Helper method for creating a single neural network layer:


def linear(x, n_output, name=None, activation=None, reuse=None):
    if len(x.get_shape()) != 2:
        x = flatten(x, reuse=reuse)

    n_input = x.get_shape().as_list()[1]

    with tf.variable_scope(name or "fc", reuse=reuse):
        W = tf.get_variable(
            name='W',
            shape=[n_input, n_output],
            dtype=tf.float32,
            initializer=tf.contrib.layers.xavier_initializer())

        b = tf.get_variable(
            name='b',
            shape=[n_output],
            dtype=tf.float32,
            initializer=tf.constant_initializer(0.0))

        h = tf.nn.bias_add(
            name='h',
            value=tf.matmul(x, W),
            bias=b)

        if activation:
            h = activation(h)
        return h, W

We'll create 6 hidden layers w/100 neurons each. In this example, I use the relu activation function.


n_neurons = 100
n_layers = 6
activation_func = tf.nn.relu

prev_input = X
for i in range(0, n_layers):
    H_i, W_i = utils.linear(
        x=prev_input, n_output=n_neurons,
        name='layer%d' % (i+1), activation=activation_func)
    prev_input = H_i

# last layer, and our predicted RGB values:
Y_pred, W_last = utils.linear(
    prev_input, 3, activation=None, name='pred')

Now, we set up our l2-norm cost function. This will penalize predictions that are off from our actual values more heavily.


error = tf.abs(tf.squared_difference(Y, Y_pred))
sum_error = tf.reduce_sum(error, reduction_indices=1)
cost = tf.reduce_mean(sum_error)
We'll perform gradient descent to search the parameter space for minimizing this cost function.

optimizer = tf.train.GradientDescentOptimizer(
    learning_rate=0.1).minimize(cost)
n_iterations = 1000
batch_size = 500
sess = tf.Session()
sess.run(tf.initialize_all_variables())

costs = []
gif_step = n_iterations // 10
step_i = 0

for it_i in range(n_iterations):
    
    # Get a random sampling of the dataset
    idxs = np.random.permutation(range(len(xs)))
    
    # The number of batches we have to iterate over
    n_batches = len(idxs) // batch_size
    
    # Now iterate over our stochastic minibatches:
    for batch_i in range(n_batches):
         
        # Get just minibatch amount of data
        idxs_i =
            idxs[batch_i * batch_size: (batch_i + 1) * batch_size]

        # And optimize, also returning the cost so we can monitor
        # how our optimization is doing.
        training_cost = sess.run(
            [cost, optimizer],
            feed_dict={X: xs[idxs_i], Y: ys[idxs_i]})[0]
After we've gone through our iterations we'll want to pull out our final values:

ys_pred = Y_pred.eval(feed_dict={X: xs}, session=sess)
img = np.clip(ys_pred.reshape(img.shape), 0, 1)
plt.imsave('final', img)

Here is final output of the neural network. Not bad for a 600-neuron brain.



Here is a visualization of the neural network's output during training. The graph shows the cost over time for each batch interation.