Kareem Nassar

Coder and Observer


My first neural network using TensorFlow

I'm currently taking an online course called Creative Applications of Deep Learning with TensorFlow. For this class's second project, I am attempting to train a neural network using TensorFlow to draw this:

# Our input, will take row and column
X = tf.placeholder(tf.float32, shape=(None, 2))

# Our output: RGB vector
Y = tf.placeholder(tf.float32, shape=(None, 3))

Helper method for creating a single neural network layer:

def linear(x, n_output, name=None, activation=None, reuse=None):
    if len(x.get_shape()) != 2:
        x = flatten(x, reuse=reuse)

    n_input = x.get_shape().as_list()[1]

    with tf.variable_scope(name or "fc", reuse=reuse):
        W = tf.get_variable(
            shape=[n_input, n_output],

        b = tf.get_variable(

        h = tf.nn.bias_add(
            value=tf.matmul(x, W),

        if activation:
            h = activation(h)
        return h, W

We'll create 6 hidden layers w/100 neurons each. In this example, I use the relu activation function.

n_neurons = 100
n_layers = 6
activation_func = tf.nn.relu

prev_input = X
for i in range(0, n_layers):
    H_i, W_i = utils.linear(
        x=prev_input, n_output=n_neurons,
        name='layer%d' % (i+1), activation=activation_func)
    prev_input = H_i

# last layer, and our predicted RGB values:
Y_pred, W_last = utils.linear(
    prev_input, 3, activation=None, name='pred')

Now, we set up our l2-norm cost function. This will penalize predictions that are off from our actual values more heavily.

error = tf.abs(tf.squared_difference(Y, Y_pred))
sum_error = tf.reduce_sum(error, reduction_indices=1)
cost = tf.reduce_mean(sum_error)
We'll perform gradient descent to search the parameter space for minimizing this cost function.

optimizer = tf.train.GradientDescentOptimizer(
n_iterations = 1000
batch_size = 500
sess = tf.Session()

costs = []
gif_step = n_iterations // 10
step_i = 0

for it_i in range(n_iterations):
    # Get a random sampling of the dataset
    idxs = np.random.permutation(range(len(xs)))
    # The number of batches we have to iterate over
    n_batches = len(idxs) // batch_size
    # Now iterate over our stochastic minibatches:
    for batch_i in range(n_batches):
        # Get just minibatch amount of data
        idxs_i =
            idxs[batch_i * batch_size: (batch_i + 1) * batch_size]

        # And optimize, also returning the cost so we can monitor
        # how our optimization is doing.
        training_cost = sess.run(
            [cost, optimizer],
            feed_dict={X: xs[idxs_i], Y: ys[idxs_i]})[0]
After we've gone through our iterations we'll want to pull out our final values:

ys_pred = Y_pred.eval(feed_dict={X: xs}, session=sess)
img = np.clip(ys_pred.reshape(img.shape), 0, 1)
plt.imsave('final', img)

Here is final output of the neural network. Not bad for a 600-neuron brain.

Here is a visualization of the neural network's output during training. The graph shows the cost over time for each batch interation.