In this tutorial, we demonstrate how to manually create a Convolutional Neural Network using the library, Tensorflow. The network will be trained using the Cifar-10 dataset. Before we continue, it is to note that Google has many wonderful tutorials using Tensorflow (including a great Cifar example), so please make sure you've at least looked into the basics of Tensorflow (Node, Graph, Session, Placeholder, One Hot Vector, etc) and Tensorflow Visualization (TensorBoard): Getting Started Tensorflow, Google (link), MNIST Tensorflow, Google (link), Cifar Tensorflow, Google (link), and TensoBoard Visualization, Google (link). Last, we import the Cifar-10 dataset from Matlab into python3, so please read the previous tutorial Matlab_to_Python_Tutorial.

Part 1: Preparing Data, Tensorflow Formatting

Before we get started, first import the Tensorflow library, the Numpy Library, and our custom python file to import our our dataset (Cifar-10: Training and Testing data created in Matlab, saved as .mat files, and then imported into python using prepareData.py). Afterwards, we need to set up our training and testing data. We can do this by calling the function load_data() from our prepareData.py and storing its return (class object) into a new variable. This varibale will hold all of our training and testing information.


import tensorflow as tf                                                         # Library for Tensorflow
import prepareData                                                              # Self-created module for loading .mat file
import numpy                                                                    # Libtary for matrix application/represenation/computation

data = prepareData.load_data()                                                  # Load in Dataset from .mat file


Next, we can go ahead and initalize our Interactive Session.


#---------------------------------------------------------------------
# InteractiveSession
#   - One can run variables without having to refer to session object
#   - Essentially the same as a normal Session (Execution of a graph)
#   - https://www.tensorflow.org/api_docs/python/tf/InteractiveSession
#---------------------------------------------------------------------

sess = tf.InteractiveSession()                                                  # Creating InteractiveSession Node

Now, we initialize the properties of our input data (e.g Cifar Dataset), by creating placeholder nodes on our computational graph. Each sample of the Cifar dataset is [32 x 32 x 3]. The flattened transform of each sample is 32 * 32 * 3= 3072. There are 10 labels of the cfiar data. By creating placeholders, each new batch will be able to pass their values to the corresponding placeholders via feeding: x(data), y_(actual labels), phase(True or False). Phase will be useful to determine how to use batch normalization for training (phase = True) and testing (phase = False). It is to note that a naming scope is used in the code below. As a reminder, a naming scope is for TensorBoard, and basically just groups similar information(operators) so it is easily referenced in our default graph.


#---------------------------------------------------------------------
# Create Placeholders, Reshape Input Sample
#   - Placeholder being a promise to fill its value later
#---------------------------------------------------------------------

with tf.name_scope('Inputs'):
    x = tf.placeholder(tf.float32, shape=[None, 3072])                          # None: *any* batch-size; 3072 pixels: single flattened 32 x 32 x 3 sample
    x_image = tf.reshape(x, [-1, 32, 32, 3])                                    # Reshape input image into 4D Tensor [1, H, W, D]
    y_ = tf.placeholder(tf.float32, shape=[None, 10])                           # Initialze node for target output classes, 10 classes specified
    phase = tf.placeholder(tf.bool)                                             # Placerholder for batch_norm, training (True) or not training (False)

Part 2: Create Convolutional Neural Network (CNN)

Next, we can initialize the architecture of our Convolutional Neural Network on our existing graph. In this example, each "block" can be thought of as a series of convolutional, relu, and batch normalization layers followed by a max pooling layer. After the blocks, the data will be passed through a fully-connected layer and then a dropout layer. Lastly the data will be finalized through a softmax regression layer. Here, batch normalization will be applied after each non-linear activation layer (ReLU) and before every pooling layer (Max Pooling). Since dropout is being used after the Fully-Connected layer, batch normalization will not be used there.


#---------------------------------------------------------------------
# Create CNN Architecture
#   - Dropout is used in training and turned off for testing
#   - Batch Normalization will be similar for training and testing
#---------------------------------------------------------------------

# Block 1
conv1 = conv_and_relu(x_image, [7, 7, 3, 64])									# Convolutional Layer and ReLU Layer
bn1 = batch_norm(conv1, phase)                                                  # Batch Normalization (input, True or False)
pool1 = max_pool2x2(bn1)														# Max Pooling Layer

# Block 2
conv2 = conv_and_relu(pool1, [7, 7, 64, 128])
bn2 = batch_norm(conv2, phase)
pool2 = max_pool2x2(bn2)

# Fully-Connected Layer (MLP)
fc1 = fully_connected(pool2, [8 * 8 * 128, 1024])								# Fully Connected Layer

# Dropout Layer - Reduce Overfitting, Scaling handeled automatically
keep_prob = tf.placeholder(tf.float32)                                          # Stores probability that neurons output is kept during dropout.
fc_drop = tf.nn.dropout(fc1, keep_prob)                                         # The placeholder allows us to train with dropout(keep_prob < 1.0) and test without it(keep_prob = 1.0)

# Readout Layer (Network Conclusion)
weights, biases = weight_and_bias([1024,10])                                    # Weights & Bias, Convolutional Filter [Input Neurons, Output Neurons]
y_conv = tf.matmul(fc_drop, weights) + biases                                   # Softmax Regression Layer



Before we continue, lets look at the details behind each layer of the CNN. We will start with the Convolutional and ReLU layer. The Convolutional layer is quickly defined function below, requiring an input (New batch, or Previous Layer Output), and a Convolutional filter [Height, Width, Depth, Neurons]. Once the requirements are met, the weights and bias are created for the layer using the Convolutional filter. Next, we can invoke Tensorflow's Convolution method by passing in our input, weights, stride, and a chosen type of padding (SAME: ceil(float(input - filter + 2( (input - 1)/2 ) + 1)/float(stride)); VALID: ceil(float(input - filter + 2(0) + 1)/float(stride)) ). Afterwards, we can invoke Tensorflow's ReLU method by passing in the recently created Convolutional Layer. It is to note that functions covered here and below will be how one invokes the corresponding layers (Convolutional, ReLU, Batch-Normalization, Max Pooling, Fully-Connected), but not the details of how they are programmatically defined.


#--------------------------------------------------------------------------
# Define Convolutional and Pooling Layers
#   - Initialize weights with noise to prevent 0 gradients
#   - Initialize slightly positive bias to avoid "dead neurons" (B/c RELU)
#   - Stride -> Batch, Height, Width, Channels
#--------------------------------------------------------------------------

def conv_and_relu(x, conv_filter, name = 'Conv and ReLU'):                      # Function, design convolutional layer. Needs 4D input, filter tensors
    weights, biases = weight_and_bias(conv_filter)                              # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
    conv = tf.nn.conv2d(x, weights, strides=[1, 1, 1, 1], padding='SAME')       # Convolutional Layer
    relu = tf.nn.relu(conv + biases)                                            # ReLU layer: Takes in output of Convolutional Layer, ReLU(w*x + b)
    return relu                                                                 # Return ReLU layer

For the Convolutional layer, the weights are initialized with a small amount of noise for symmetry breaking. The bias will be initialized slightly positive to avoid "dead neurons", due to using ReLU.


#--------------------------------------------------------------------------
# Define Weights and Bias
#   - Initialize slightly positive bias to avoid "dead neurons" (B/c RELU)
#   - Weights: normal distribution, specified mean and standard deviation
#       - |Values| > 2 * stddev's from the mean are dropped and re-picked.
#--------------------------------------------------------------------------

def weight_and_bias(myFilter):                                                  # Function, Initalize weights given shape from placeholder
    initial_w = tf.truncated_normal(myFilter, stddev=0.1)                       # Normal distribution Node, specified standard deviation value
    if(len(myFilter) == 2):                                                     # When len(shape) == 2, at end of CNN network
        initial_b = tf.constant(0.1, shape = [myFilter[1]])                     # Assign second element of filter to bias instead of last
    else:
        initial_b = tf.constant(0.1, shape = [myFilter[3]])                     # Positive value, shape being idcential the arugment.
    return tf.Variable(initial_w), tf.Variable(initial_b)                       # Return weights as Variable (So they can be updated via training)

Next, we can look at the max Pooling layer. This network uses standard [2 x 2] max pooling with stride [2 x 2]. Note that neurons and channels are unaffected by max pooling.


#--------------------------------------------------------------------------
# Define Max Pooling Layer
#   - Stride -> Neurons, Height, Width, Channels
#--------------------------------------------------------------------------

def max_pool_2x2(x, name = 'Max Pooling'):                                      # Function, design max pooling layer, need 4D input
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],                                # Specify pooling filters, stride, padding.
                        strides=[1, 2, 2, 1], padding='SAME')

Next, we can look at the fully connected layer. At this point, the image has been reduced spatially to a [8 x 8], so to cover the entire image, a filter of size [8 x 8] will be utilized, resulting in an output of [1 x 1]. It is to note that the filter passed into this function follows: input: (Height* Width* input_neurons, output neurons); flattened input: ([8 * 8 * 128, 1024]).


#--------------------------------------------------------------------------
# Define Fully-Connected Layer
#   - Stride -> Batch, Height, Width, Channels
#--------------------------------------------------------------------------

def fully_connected(x, conv_filter, name = 'Fully Connected'):                  # Function, design fully connected layer
    weights, biases = weight_and_bias(conv_filter)                              # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
    x_flat = tf.reshape(x, [-1, 7*7*64])                                        # Reshape pooling layer into batch of vectors. Flatten!
    fc = tf.nn.relu(tf.matmul(x_flat, weights) + biases)                        # Take ReLU. ** Note: matmul used instead of Convolution **
    return fc                                                                   # Return Fully Connected Layer

Lastly, we can look at the Batch Normalization Layer. For this layer, we will pass the input (previous layer) and the phase (Training = True or False). For more details behind the tensorflow technique, link found here: Batch Normalization Tensorflow, Google


#--------------------------------------------------------------------------
# Define Batch Normalization Layer
#   - Be sure to set Phase correctly
#--------------------------------------------------------------------------

def batch_norm(x, phase):
    return tf.contrib.layers.batch_norm(x,                                    	# Batch Normalization Layer
                        center = True, scale = True, is_training = phase)

Part 3: Training and Testing the CNN

After the network architecture has been created, we must include how to train our CNN on our existing graph. To do this, we must first implement a way to determine the loss of our model. Here, we will use the popular concept known as cross_entropy. Next, an optimization tool needs to be implemented to minimize the loss of our network. A popular option is Gradient Descent, but here we will use the ADAM optimizer. For most Tensorflow optimizers, we pass in the learning rate as an argument to the optimization tool. Here we use a learning rate of 1e-3. Next, we need to find a way to evaluate how well our CNN performs. To do this, we will check each prediction and use that to calculate the overall accuracy of the model. Lastly, keep in mind the different naming scopes (tf.name_scope) in the example below are for Tensorboard labeling. Now in regards to batch normalization, the biggest change to this section of code below are the first two lines. For training with batch normalization the moving mean and moving variance need to be updated. By default the update ops are placed in tf.GraphKeys.UPDATE_OPS, so they need to be added as a dependency to the train_op. We do this in the first two lines. After training the CNN for 40 epochs, training accuracy with batch normalization should be greater than 90% accuracy. However, in this example, without batch normalization, excpet to not find convergence and to have a training accuracy less than 15%.


#---------------------------------------------------------------------
# Training Convolutional Neural Network
#   - Learning rate is set to 1e-4 in "Train_step"
#   - Tensorflow offers MANY Optimization techniques: ADAM, SGD, etc
#   - tf.argmax() returns index of highest entry. **One hot Vector**
#       - tf.argmax(y_conv) - prediction output from model
#       - tf.argmax(y_) - True Output label
#   - keep_prob and feed_dict are used to control dropout for testing
#---------------------------------------------------------------------

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)                         # Add for batch Normalization
with tf.control_dependencies(update_ops):                                       # Add for batch Normalization

    with tf.name_scope('Cross_Entropy'):
        cross_entropy = tf.reduce_mean(                                         # Implement Cross_Entropy to compute the softmax activation
        tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))      # Cross Entropy: True Output Labels (y_), Softmax output (y_conv)
        tf.summary.scalar('cross_entropy', cross_entropy)						# TensorBoard scalar

    with tf.name_scope('train'):
        train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy)       # Train Network, Tensorflow minimizes cross_entropy via ADAM Optimization instead of SGD

    with tf.name_scope('Train_Results'):
        with tf.name_scope('Correct_Prediction'):
            correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))  # Check if prediction is wrong with tf.equal(CNN_result,True_result)
        with tf.name_scope('Accuracy'):
            accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))  # Find the percent accuracy, take mean of correct_prediction outputs
            tf.summary.scalar('accuracy', accuracy)								# TensorBoard scalar

Lastly, we begin our session and initialize all of our Tensorflow variables. We then specify how long we want to train for in iterations instead of epochs. Each iteration, a new mini-batch is pushed through the network. After all mini-batches, a single epoch is complete. We will train on cifar-10 for 40 epochs. In regards to batch noramlization, each time a new batch is ready to be inserted into the CNN, a phase = True is passed to the phase placeholder. This will allow the batch noramlization layer in the network to know that its in training mode and not testing mode. It is to note that unlike MNIST dataset, which was easily imported through a tensorflow command, our dataset is custom imported from matlab into python. Thus, we have to create our own batch function that can differentiate between training and testing, shuffle data when needed, and can also manipulate Tensorflows One-Hot-Vectors.


with tf.Session() as sess:                                                      # Make sure session is initalized

#---------------------------------------------------------------------
# Tf.summary.FileWriter:
# - Writes data from tensorflow to file system, read by TensorBoard
# - Writes any information I'm going to visualize in TensorBoard
#---------------------------------------------------------------------

merged_summary = tf.summary.merge_all()                                         # Merge all summaries together
train_writer = tf.summary.FileWriter('data/train_results', sess.graph)          # Initalize FileWriter for training, pass graph

#---------------------------------------------------------------------
# Training CNN
#---------------------------------------------------------------------

sess.run(tf.global_variables_initializer())                                     # Initalize all Tensorflow Variables
for i in range(20000):                                                          # 20,000 (default) training iterations
    batch = next_batch(100, data)                                               # Specify Batch size here
    batch_new = numpy.reshape(batch[0], (100, 3072))

    if i % 100 == 0:                                                            # Show progress ever 100 epochs
        train_accuracy = accuracy.eval(feed_dict={                              # Evaluate CNN for current accuracy.
            x: batch_new, y_: batch[1], phase: 0, keep_prob: 1.0})              # Turn Batch off (phase = 0) for accuracy evaluation
        print('step %d, training accuracy %g' % (i, train_accuracy))            # Display Evaluation Results while training

        s = sess.run(merged_summary, feed_dict={x: batch_new,					# Merge what we found
            y_: batch[1], phase: 1, keep_prob: 0.5})
        train_writer.add_summary(s,i)											# Write what we merged

    train_step.run(feed_dict={x: batch_new, y_: batch[1], phase: 1, keep_prob: 0.5})  # Feed Batch into the network, turn batch normalization (phase = True) on for training



#--------------------------------------------------------------------------
# Acquire New Batch From Dataset
#--------------------------------------------------------------------------

def next_batch(batch_size, data, shuffle = True, train = True):                 # Function, Batch Distribution

    start = data.index_in_epoch                                                 # Assign Start of indexing

    #---------------------------------------------------------
    # Update and Shuffle Data
    # - Case: Beginning of Epoch (Data Shuffling)
    # - Case: Middle of Epoch (Update mini-batch index)
    # - Case: End of Epoch (Finish-mini-batch, Shuffling)
    #---------------------------------------------------------

    # Beginning of Epoch
    if(data.epochs_completed == 0 and start == 0 and shuffle):                  # Is this the first mini batch? Beginning of Epoch
        perm = numpy.arange(data.num_examples)                                  # Acquire all possible sample indecies
        numpy.random.shuffle(perm)                                              # Shuffling all indicies
        if(train):
            data.train_data = data.train_data[perm]                             # Since data.train_data and perm are *numpy arrays*, can shuffle indicies!
            data.ohv_train = data.ohv_train[perm]                               # Since data.train_L and perm are *numpy arrays*. can match indicies to data.train_data
        else:
            if(shuffle):
                data.test_data = data.test_data[perm]                           # Since data.test_data and perm are *numpy arrays*, can shuffle indicies!
                data.ohv_test = data.ohv_test[perm]                             # Since data.test_L and perm are *numpy arrays*. can match indicies to data.test_data
            else:
                data.test_data = data.test_data
                data.ohv_test = data.ohv_test
    # End of Epoch
    if((start + batch_size) > data.num_examples):                               # Is this the last mini batch? End of Epoch
        data.epochs_completed += 1                                              # Increment Epoch: Complete pass through all training data
        print('Processing Epoch: ', data.epochs_completed)
        rest_num_examples = data.num_examples - start                           # How many training examples are left? Acquire the number of examples
        if(train):
            images_rest_part = data.train_data[start:data.num_examples]         # Assign the remainder of images, note that start is a very high index at this point
            labels_rest_part = data.ohv_train[start:data.num_examples]          # Assign the remainder of labels, note that start is a very high index at this point
        else:
            images_rest_part = data.test_data[start:data.num_examples]          # Assign the remainder of images, note that start is a very high index at this point
            labels_rest_part = data.ohv_test[start:data.num_examples]           # Assign the remainder of labels, note that start is a very high index at this point

        #---------------------------------------------------------
        # Shuffle Data *Again*
        # - An epoch has occured, therefore shuffle data
        # - If mini batch was not complete:
        #   - Current mini batch transitions to new epoch
        #---------------------------------------------------------

        if(shuffle):                                                            # Lets shuffle again, moving into next epoch
            perm = numpy.arange(data.num_examples)                              # First gather all indicies, ARRAY format
            numpy.random.shuffle(perm)                                          # Shuffle indicies
            if(train):
                data.train_data = data.train_data[perm]                         # Complete the shuffle training data, data.train_data = ARRAY[ARRAY]
                data.ohv_train = data.ohv_train[perm]                           # Complete the shuffle training labels, data.train_L = ARRAY[ARRAY]
            else:
                data.test_data = data.test_data[perm]                           # Complete the shuffle training data, data.test_data = ARRAY[ARRAY]
                data.ohv_test = data.ohv_test[perm]                             # Complete the shuffle training labels, data.test_L = ARRAY[ARRAY]

        start = 0                                                               # Staring over, so set start back to 0
        data.index_in_epoch = batch_size - rest_num_examples                    # Find how many examples needed from new epoch to fulfill mini batch size
        end = data.index_in_epoch                                               # Assign the end of indexing
        if(train):
            images_new_part = data.train_data[start:end]                        # Acquire needed training examples from new epoch
            labels_new_part = data.ohv_train[start:end]                         # Acquire needed training labels from new epoch
        else:
            images_new_part = data.test_data[start:end]                         # Acquire needed testing examples from new epoch
            labels_new_part = data.ohv_test[start:end]                          # Acquire needed testing labels from new epoch

        #---------------------------------------------------------
        # Return COMPLETE Mini Batch
        # - concatenate(old_epoch data, new_epoch data),
        # - concatenate(old_epoch_labels, new_epoch_labels)
        #---------------------------------------------------------

        return numpy.concatenate((images_rest_part, images_new_part), axis=0) , numpy.concatenate((labels_rest_part, labels_new_part), axis=0)

    # Middle of Epoch
    else:
        data.index_in_epoch += batch_size                                       # Update index in dataset
        end = data.index_in_epoch                                               # Assign the end of indexing
        if(train):
            return data.train_data[start:end], data.ohv_train[start:end]        # Return mini batch
        else:
            return data.test_data[start:end], data.ohv_test[start:end]          # Return mini batch


After training, we can use TensorBoard to visualize our Cross Entropy and our Training Accuracy. TensorBoard can easily keep track of scalar values, and will plot both accuracy and cross entropy vs number of iterations(e.g 20,000 iterations = 40 epochs). Here are the the plots:

Now that training is completed, a valuebale idea would be to save our trained CNN. We can do this in a few simple lines. We can then load our trained CNN back with another few simple lines.


saver = tf.train.Saver()                                                		# Save trained CNN
saver.save(sess, 'data/Saved_CNNs/Cifar_CNN')                           		# Store it into local directory path "data/Saved_CNNs/Cifar_CNN"

saver = tf.train.Saver()                                                		# Call to interact with stored CNN
CNN = tf.train.get_checkpoint_state('data/Saved_CNNs/')                 		# Load in trained CNN
saver.restore(sess, CNN.model_checkpoint_path)                          		# Restore all information of trained CNN

In regards to testing our CNN, its very similar to training. However, we need to specify on our created batch function that we are not training (train = False) We are also turning off dropout (keep_prob: 1.0) and batch normalization (phase = False). After training for 40 epochs with batch normalization, testing accuracy on the full test dataset should be around 60% (top1 accuracy). This is not spectacular, but neither is the basic CNN Architecture defined above in this network. With proper pre-processing techniques on the data, parameter tuning, and a more detailed approach at designing the CNN architecture, this accuracy will only increase. Without using batch normalization and with using the same basic CNN architecture, the testing accuracy will only be around 15%.


#---------------------------------------------------------------------
# Testing CNN
#---------------------------------------------------------------------

data.index_in_epoch = 0                                                     	# Reset index counter to position 0
data.epochs_completed = 0                                                   	# Reset epochs completed to 0
total = 0

for i in range(100):                                                        	# 200 (0->199) (default) testing iterations (1 epoch 200 * 50 = 10,000 samples)
    batch = next_batch(100, data, shuffle = False, train = False)           	# Specify Batch size here, default is 50, here it becomes a tuple
    batch_new = numpy.reshape(batch[0], (100, 3072))
    result = accuracy.eval(feed_dict={
          x: batch_new, y_: batch[1], phase: 0, keep_prob: 1.0})            	# For evaluation (testing) Keep batch normalization off (phase = False)
    print('Current batch: '+str(data.index_in_epoch)+' , Test Accuracy: ',result)
    total += result
print('Total Accuracy: ', total/i)


Below is our finalized code for training a CNN



	#--------------------------------------------------------------------------
	# Import Necessary Libraries
	#--------------------------------------------------------------------------

	import tensorflow as tf                                                         # Library for Tensorflow
	import prepareData                                                              # Self-created module for loading .mat file
	import numpy                                                                    # Libtary for matrix application/represenation/computation
	i
	#--------------------------------------------------------------------------
	# Acquire New Batch From Dataset
	#--------------------------------------------------------------------------

	def next_batch(batch_size, data, shuffle = True, train = True):                 # Function, Batch Distribution

	    start = data.index_in_epoch                                                 # Assign Start of indexing

	    #---------------------------------------------------------
	    # Update and Shuffle Data
	    # - Case: Beginning of Epoch (Data Shuffling)
	    # - Case: Middle of Epoch (Update mini-batch index)
	    # - Case: End of Epoch (Finish-mini-batch, Shuffling)
	    #---------------------------------------------------------

	    # Beginning of Epoch
	    if(data.epochs_completed == 0 and start == 0 and shuffle):                  # Is this the first mini batch? Beginning of Epoch
	        perm = numpy.arange(data.num_examples)                                  # Acquire all possible sample indecies
	        numpy.random.shuffle(perm)                                              # Shuffling all indicies
	        if(train):
	            data.train_data = data.train_data[perm]                             # Since data.train_data and perm are *numpy arrays*, can shuffle indicies!
	            data.ohv_train = data.ohv_train[perm]                               # Since data.train_L and perm are *numpy arrays*. can match indicies to data.train_data
	        else:
	            if(shuffle):
	                data.test_data = data.test_data[perm]                           # Since data.test_data and perm are *numpy arrays*, can shuffle indicies!
	                data.ohv_test = data.ohv_test[perm]                             # Since data.test_L and perm are *numpy arrays*. can match indicies to data.test_data
	            else:
	                data.test_data = data.test_data
	                data.ohv_test = data.ohv_test
	    # End of Epoch
	    if((start + batch_size) > data.num_examples):                               # Is this the last mini batch? End of Epoch
	        data.epochs_completed += 1                                              # Increment Epoch: Complete pass through all training data
	        print('Processing Epoch: ', data.epochs_completed)
	        rest_num_examples = data.num_examples - start                           # How many training examples are left? Acquire the number of examples
	        if(train):
	            images_rest_part = data.train_data[start:data.num_examples]         # Assign the remainder of images, note that start is a very high index at this point
	            labels_rest_part = data.ohv_train[start:data.num_examples]          # Assign the remainder of labels, note that start is a very high index at this point
	        else:
	            images_rest_part = data.test_data[start:data.num_examples]          # Assign the remainder of images, note that start is a very high index at this point
	            labels_rest_part = data.ohv_test[start:data.num_examples]           # Assign the remainder of labels, note that start is a very high index at this point

	        #---------------------------------------------------------
	        # Shuffle Data *Again*
	        # - An epoch has occured, therefore shuffle data
	        # - If mini batch was not complete:
	        #   - Current mini batch transitions to new epoch
	        #---------------------------------------------------------

	        if(shuffle):                                                            # Lets shuffle again, moving into next epoch
	            perm = numpy.arange(data.num_examples)                              # First gather all indicies, ARRAY format
	            numpy.random.shuffle(perm)                                          # Shuffle indicies
	            if(train):
	                data.train_data = data.train_data[perm]                         # Complete the shuffle training data, data.train_data = ARRAY[ARRAY]
	                data.ohv_train = data.ohv_train[perm]                           # Complete the shuffle training labels, data.train_L = ARRAY[ARRAY]
	            else:
	                data.test_data = data.test_data[perm]                           # Complete the shuffle training data, data.test_data = ARRAY[ARRAY]
	                data.ohv_test = data.ohv_test[perm]                             # Complete the shuffle training labels, data.test_L = ARRAY[ARRAY]

	        start = 0                                                               # Staring over, so set start back to 0
	        data.index_in_epoch = batch_size - rest_num_examples                    # Find how many examples needed from new epoch to fulfill mini batch size
	        end = data.index_in_epoch                                               # Assign the end of indexing
	        if(train):
	            images_new_part = data.train_data[start:end]                        # Acquire needed training examples from new epoch
	            labels_new_part = data.ohv_train[start:end]                         # Acquire needed training labels from new epoch
	        else:
	            images_new_part = data.test_data[start:end]                         # Acquire needed testing examples from new epoch
	            labels_new_part = data.ohv_test[start:end]                          # Acquire needed testing labels from new epoch

	        #---------------------------------------------------------
	        # Return COMPLETE Mini Batch
	        # - concatenate(old_epoch data, new_epoch data),
	        # - concatenate(old_epoch_labels, new_epoch_labels)
	        #---------------------------------------------------------

	        return numpy.concatenate((images_rest_part, images_new_part), axis=0) , numpy.concatenate((labels_rest_part, labels_new_part), axis=0)

	    # Middle of Epoch
	    else:
	        data.index_in_epoch += batch_size                                       # Update index in dataset
	        end = data.index_in_epoch                                               # Assign the end of indexing
	        if(train):
	            return data.train_data[start:end], data.ohv_train[start:end]        # Return mini batch
	        else:
	            return data.test_data[start:end], data.ohv_test[start:end]          # Return mini batch

	#--------------------------------------------------------------------------
	# Define Weights and Bias
	#   - Initialize slightly positive bias to avoid "dead neurons" (B/c RELU)
	#   - Weights: normal distribution, specified mean and standard deviation
	#       - |Values| > 2 * stddev's from the mean are dropped and re-picked.
	#--------------------------------------------------------------------------

	def weight_and_bias(myFilter):                                                  # Function, Initalize weights given shape from placeholder
	    initial_w = tf.truncated_normal(myFilter, stddev=0.1)                       # Normal distribution Node, specified standard deviation value
	    if(len(myFilter) == 2):                                                     # When len(shape) == 2, at end of CNN network
	        initial_b = tf.constant(0.1, shape = [myFilter[1]])                     # Assign second element of filter to bias instead of last
	    else:
	        initial_b = tf.constant(0.1, shape = [myFilter[3]])                     # Positive value, shape being idcential the arugment.
	    return tf.Variable(initial_w), tf.Variable(initial_b)                       # Return weights as Variable (So they can be updated via training)

	#--------------------------------------------------------------------------
	# Define Convolutional and Pooling Layers
	#   - Initialize weights with noise to prevent 0 gradients
	#   - Initialize slightly positive bias to avoid "dead neurons" (B/c RELU)
	#   - Default Conv format: NHWC -> Batch Size, Height, Width, Channels
	#   - Default Pool format: NHWC
	#--------------------------------------------------------------------------

	def conv_and_relu(x, conv_filter, name = 'Conv and ReLU'):                      # Function, design convolutional layer. Needs 4D input, filter tensors
	    weights, biases = weight_and_bias(conv_filter)                              # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
	    conv = tf.nn.conv2d(x, weights, strides=[1, 1, 1, 1], padding='SAME')       # Convolutional Layer
	    relu = tf.nn.relu(conv + biases)                                            # ReLU layer: Takes in output of Convolutional Layer, ReLU(w*x + b)
	    return relu                                                                 # Return Batch Normalization layer

	def max_pool2x2(x, name = 'Max Pooling'):                                       # Function, design max pooling layer, need 4D input
	    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],                                # Specify pooling filters, stride, padding.
	                        strides=[1, 2, 2, 1], padding='SAME')

	def fully_connected(x, conv_filter, name = 'Fully Connected'):                  # Function, design fully connected layer
	    weights, biases = weight_and_bias(conv_filter)                              # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
	    x_flat = tf.reshape(x, [-1, conv_filter[0]])                                # Reshape pooling layer into batch of vectors. Flatten!
	    fc = tf.nn.relu(tf.matmul(x_flat, weights) + biases)                        # Take ReLU. ** Note: matmul used instead of Convolution **
	    return fc

	def batch_norm(x, phase):
	    return tf.contrib.layers.batch_norm(x,                                      # Batch Normalization Layer
	                        center = True, scale = True, is_training = phase)
	#--------------------------------------------------------------------------
	# Load Dataset, Create CNN Network, Train and Test
	#--------------------------------------------------------------------------

	def main():

	    data = prepareData.load_data()                                              # Load in Dataset from .mat file
	    data.num_examples = len(data.train_L)                                       # Set number of training samples
	    sess = tf.InteractiveSession()                                              # Creating InteractiveSession Node

	    #---------------------------------------------------------------------
	    # Create Placeholders, Reshape Input Sample
	    #   - Placeholder being a promise to fill its value later
	    #---------------------------------------------------------------------

	    with tf.name_scope('Inputs'):                                               # Naming scope for basic operations, grouping all "Inputs" together
	        x = tf.placeholder(tf.float32, shape=[None, 3072])                      # None: *any* batch-size; 3072 pixels: single flattened 32 x 32 x 3 sample
	        x_image = tf.reshape(x, [-1, 32, 32, 3])                                # Reshape input image into 4D Tensor [1, H, W, D]
	        y_ = tf.placeholder(tf.float32, shape=[None, 10])                       # Initialze node for target output classes, 10 classes specified
	        phase = tf.placeholder(tf.bool)                                         # Placerholder for batch_norm, training (True) or not training (False)

	    #---------------------------------------------------------------------
	    # Create CNN Architecture
	    #   - Dropout is used in training and turned off for testing
	    #   - Batch Normalization will be similar for training and testing
	    #---------------------------------------------------------------------

	    # Block 1
	    conv1 = conv_and_relu(x_image, [7, 7, 3, 64])                               # Convolutional Layer and ReLU Layer
	    bn1 = batch_norm(conv1, phase)                                              # Batch Normalization (input, True or False)
	    pool1 = max_pool2x2(bn1)                                                    # Max Pooling Layer

	    # Block 2
	    conv2 = conv_and_relu(pool1, [7, 7, 64, 128])
	    bn2 = batch_norm(conv2, phase)
	    pool2 = max_pool2x2(bn2)

	    # Fully-Connected Layer (MLP)
	    fc1 = fully_connected(pool2, [8 * 8 * 128, 1024])

	    # Dropout Layer - Reduce Overfitting, Scaling handeled automatically
	    keep_prob = tf.placeholder(tf.float32)                                      # Stores probability that neurons output is kept during dropout.
	    fc_drop = tf.nn.dropout(fc1, keep_prob)                                     # The placeholder allows us to train with dropout and test without it

	    # Readout Layer (Network Conclusion)
	    weights, biases = weight_and_bias([1024,10])                                # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
	    y_conv = tf.matmul(fc_drop, weights) + biases                               # Softmax Regression Layer

	    print(pool1.shape)
	    print(pool2.shape)
	    print(fc1.shape)
	    print(y_conv.shape)

	    #---------------------------------------------------------------------
	    # Training Convolutional Neural Network
	    #   - Learning rate is set to 1e-4 in "Train_step"
	    #   - Tensorflow offers MANY Optimization techniques: ADAM, SGD, etc
	    #   - tf.argmax() returns index of highest entry. **One hot Vector**
	    #       - tf.argmax(y_conv) - prediction output from model
	    #       - tf.argmax(y_) - True Output label
	    #   - keep_prob and feed_dict are used to control dropout for testing
	    #---------------------------------------------------------------------

	    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)                     # Add for Batch Normalization
	    with tf.control_dependencies(update_ops):                                   # Add for Batch Normalization

	        with tf.name_scope('Cross_Entropy'):                                    # Naming scope for all "Cross Entropy" information
	            cross_entropy = tf.reduce_mean(                                     # Implement Cross_Entropy to compute the softmax activation
	            tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))  # Cross Entropy: True Output Labels (y_), Softmax output (y_conv)
	            tf.summary.scalar('cross_entropy', cross_entropy)                   # Add cross_entropy to our graph

	        with tf.name_scope('train'):                                            # Naming scope for our "Training Step"
	            train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy)   # Train Network, Tensorflow minimizes cross_entropy via ADAM Optimization instead of SGD

	        with tf.name_scope('Train_Results'):                                    # Naming Scope for "Training Results"
	            with tf.name_scope('Correct_Prediction'):                           # Naming Scope for "Correct_Prediction"
	                correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))  # Check if prediction is wrong with tf.equal(CNN_result,True_result)

	            with tf.name_scope('Accuracy'):                                     # Naming Scope for "Accuracy"
	                accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) # Find the percent accuracy, take mean of correct_prediction outputs
	                tf.summary.scalar('accuracy', accuracy)                         # Add Training Accuracy to graph

	    with tf.Session() as sess:                                                  # Make sure session is initalized

	        #---------------------------------------------------------------------
	        # Tf.summary.FileWriter:
	        # - Writes data from tensorflow to file system, read by TensorBoard
	        # - Writes any information I'm going to visualize in TensorBoard
	        #---------------------------------------------------------------------

	        merged_summary = tf.summary.merge_all()                                 # Important for TensorBoard, merge all summaries
	        train_writer = tf.summary.FileWriter('data/train_results', sess.graph)  # Initalize FileWriter for training, pass graph
	        test_writer = tf.summary.FileWriter('data/test_results')                # Initalize FileWriter for testing, pass graph

	        #---------------------------------------------------------------------
	        # Training CNN
	        #---------------------------------------------------------------------

	        sess.run(tf.global_variables_initializer())                             # Initalize all Tensorflow Variables
	        for i in range(20000):                                                  # 20,000 training iterations due to our dataset having 50,000 training examples
	            batch = next_batch(100, data)                                       # Specify Batch size here, default is 50, here it becomes a tuple
	            batch_new = numpy.reshape(batch[0], (100, 3072))

	            if i % 100 == 0:                                                    # Test Progress ever 100 epochs
	                train_accuracy = accuracy.eval(feed_dict={                      # Evaluate CNN for current accuracy.
	                    x: batch_new, y_: batch[1], phase: 0, keep_prob: 1.0})
	                print('step %d, training accuracy %g' % (i, train_accuracy))    # Display Evaluation Results while training

	                s = sess.run(merged_summary, feed_dict={x: batch_new,
	                    y_: batch[1], phase: 1, keep_prob: 0.5})                    # Important for TensorBoard
	                train_writer.add_summary(s,i)

	            train_step.run(feed_dict={x: batch_new, y_: batch[1], phase: 1, keep_prob: 0.5})  # Feed Batch into the network

	        saver = tf.train.Saver()                                                # Save trained CNN
	        saver.save(sess, 'data/Saved_CNNs/Cifar_CNN')                           # Store it into directory path "data/Saved_CNNs/Cifar_CNN"
	        pauseMe = input('Training Done, Press Enter To Continue...')

	main()


Here is our code for testing our CNN



	#--------------------------------------------------------------------------
	# Import Necessary Libraries
	#--------------------------------------------------------------------------

	import tensorflow as tf                                                         # Library for Tensorflow
	import prepareData                                                              # Self-created module for loading .mat file
	import numpy                                                                    # Libtary for matrix application/represenation/computation
	i
	#--------------------------------------------------------------------------
	# Acquire New Batch From Dataset
	#--------------------------------------------------------------------------

	def next_batch(batch_size, data, shuffle = True, train = True):                 # Function, Batch Distribution

	    start = data.index_in_epoch                                                 # Assign Start of indexing

	    #---------------------------------------------------------
	    # Update and Shuffle Data
	    # - Case: Beginning of Epoch (Data Shuffling)
	    # - Case: Middle of Epoch (Update mini-batch index)
	    # - Case: End of Epoch (Finish-mini-batch, Shuffling)
	    #---------------------------------------------------------

	    # Beginning of Epoch
	    if(data.epochs_completed == 0 and start == 0 and shuffle):                  # Is this the first mini batch? Beginning of Epoch
	        perm = numpy.arange(data.num_examples)                                  # Acquire all possible sample indecies
	        numpy.random.shuffle(perm)                                              # Shuffling all indicies
	        if(train):
	            data.train_data = data.train_data[perm]                             # Since data.train_data and perm are *numpy arrays*, can shuffle indicies!
	            data.ohv_train = data.ohv_train[perm]                               # Since data.train_L and perm are *numpy arrays*. can match indicies to data.train_data
	        else:
	            if(shuffle):
	                data.test_data = data.test_data[perm]                           # Since data.test_data and perm are *numpy arrays*, can shuffle indicies!
	                data.ohv_test = data.ohv_test[perm]                             # Since data.test_L and perm are *numpy arrays*. can match indicies to data.test_data
	            else:
	                data.test_data = data.test_data
	                data.ohv_test = data.ohv_test
	    # End of Epoch
	    if((start + batch_size) > data.num_examples):                               # Is this the last mini batch? End of Epoch
	        data.epochs_completed += 1                                              # Increment Epoch: Complete pass through all training data
	        print('Processing Epoch: ', data.epochs_completed)
	        rest_num_examples = data.num_examples - start                           # How many training examples are left? Acquire the number of examples
	        if(train):
	            images_rest_part = data.train_data[start:data.num_examples]         # Assign the remainder of images, note that start is a very high index at this point
	            labels_rest_part = data.ohv_train[start:data.num_examples]          # Assign the remainder of labels, note that start is a very high index at this point
	        else:
	            images_rest_part = data.test_data[start:data.num_examples]          # Assign the remainder of images, note that start is a very high index at this point
	            labels_rest_part = data.ohv_test[start:data.num_examples]           # Assign the remainder of labels, note that start is a very high index at this point

	        #---------------------------------------------------------
	        # Shuffle Data *Again*
	        # - An epoch has occured, therefore shuffle data
	        # - If mini batch was not complete:
	        #   - Current mini batch transitions to new epoch
	        #---------------------------------------------------------

	        if(shuffle):                                                            # Lets shuffle again, moving into next epoch
	            perm = numpy.arange(data.num_examples)                              # First gather all indicies, ARRAY format
	            numpy.random.shuffle(perm)                                          # Shuffle indicies
	            if(train):
	                data.train_data = data.train_data[perm]                         # Complete the shuffle training data, data.train_data = ARRAY[ARRAY]
	                data.ohv_train = data.ohv_train[perm]                           # Complete the shuffle training labels, data.train_L = ARRAY[ARRAY]
	            else:
	                data.test_data = data.test_data[perm]                           # Complete the shuffle training data, data.test_data = ARRAY[ARRAY]
	                data.ohv_test = data.ohv_test[perm]                             # Complete the shuffle training labels, data.test_L = ARRAY[ARRAY]

	        start = 0                                                               # Staring over, so set start back to 0
	        data.index_in_epoch = batch_size - rest_num_examples                    # Find how many examples needed from new epoch to fulfill mini batch size
	        end = data.index_in_epoch                                               # Assign the end of indexing
	        if(train):
	            images_new_part = data.train_data[start:end]                        # Acquire needed training examples from new epoch
	            labels_new_part = data.ohv_train[start:end]                         # Acquire needed training labels from new epoch
	        else:
	            images_new_part = data.test_data[start:end]                         # Acquire needed testing examples from new epoch
	            labels_new_part = data.ohv_test[start:end]                          # Acquire needed testing labels from new epoch

	        #---------------------------------------------------------
	        # Return COMPLETE Mini Batch
	        # - concatenate(old_epoch data, new_epoch data),
	        # - concatenate(old_epoch_labels, new_epoch_labels)
	        #---------------------------------------------------------

	        return numpy.concatenate((images_rest_part, images_new_part), axis=0) , numpy.concatenate((labels_rest_part, labels_new_part), axis=0)

	    # Middle of Epoch
	    else:
	        data.index_in_epoch += batch_size                                       # Update index in dataset
	        end = data.index_in_epoch                                               # Assign the end of indexing
	        if(train):
	            return data.train_data[start:end], data.ohv_train[start:end]        # Return mini batch
	        else:
	            return data.test_data[start:end], data.ohv_test[start:end]          # Return mini batch

	#--------------------------------------------------------------------------
	# Define Weights and Bias
	#   - Initialize slightly positive bias to avoid "dead neurons" (B/c RELU)
	#   - Weights: normal distribution, specified mean and standard deviation
	#       - |Values| > 2 * stddev's from the mean are dropped and re-picked.
	#--------------------------------------------------------------------------

	def weight_and_bias(myFilter):                                                  # Function, Initalize weights given shape from placeholder
	    initial_w = tf.truncated_normal(myFilter, stddev=0.1)                       # Normal distribution Node, specified standard deviation value
	    if(len(myFilter) == 2):                                                     # When len(shape) == 2, at end of CNN network
	        initial_b = tf.constant(0.1, shape = [myFilter[1]])                     # Assign second element of filter to bias instead of last
	    else:
	        initial_b = tf.constant(0.1, shape = [myFilter[3]])                     # Positive value, shape being idcential the arugment.
	    return tf.Variable(initial_w), tf.Variable(initial_b)                       # Return weights as Variable (So they can be updated via training)

	#--------------------------------------------------------------------------
	# Define Convolutional and Pooling Layers
	#   - Initialize weights with noise to prevent 0 gradients
	#   - Initialize slightly positive bias to avoid "dead neurons" (B/c RELU)
	#   - Default Conv format: NHWC -> Batch Size, Height, Width, Channels
	#   - Default Pool format: NHWC
	#--------------------------------------------------------------------------

	def conv_and_relu(x, conv_filter, name = 'Conv and ReLU'):                      # Function, design convolutional layer. Needs 4D input, filter tensors
	    weights, biases = weight_and_bias(conv_filter)                              # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
	    conv = tf.nn.conv2d(x, weights, strides=[1, 1, 1, 1], padding='SAME')       # Convolutional Layer
	    relu = tf.nn.relu(conv + biases)                                            # ReLU layer: Takes in output of Convolutional Layer, ReLU(w*x + b)
	    return relu                                                                 # Return Batch Normalization layer

	def max_pool2x2(x, name = 'Max Pooling'):                                       # Function, design max pooling layer, need 4D input
	    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],                                # Specify pooling filters, stride, padding.
	                        strides=[1, 2, 2, 1], padding='SAME')

	def fully_connected(x, conv_filter, name = 'Fully Connected'):                  # Function, design fully connected layer
	    weights, biases = weight_and_bias(conv_filter)                              # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
	    x_flat = tf.reshape(x, [-1, conv_filter[0]])                                # Reshape pooling layer into batch of vectors. Flatten!
	    fc = tf.nn.relu(tf.matmul(x_flat, weights) + biases)                        # Take ReLU. ** Note: matmul used instead of Convolution **
	    return fc

	def batch_norm(x, phase):
	    return tf.contrib.layers.batch_norm(x,                                      # Batch Normalization Layer
	                        center = True, scale = True, is_training = phase)
	#--------------------------------------------------------------------------
	# Load Dataset, Create CNN Network, Train and Test
	#--------------------------------------------------------------------------

	def main():

	    data = prepareData.load_data()                                              # Load in Dataset from .mat file
	    data.num_examples = len(data.train_L)                                       # Set number of training samples
	    sess = tf.InteractiveSession()                                              # Creating InteractiveSession Node

	    #---------------------------------------------------------------------
	    # Create Placeholders, Reshape Input Sample
	    #   - Placeholder being a promise to fill its value later
	    #---------------------------------------------------------------------

	    with tf.name_scope('Inputs'):                                               # Naming scope for basic operations, grouping all "Inputs" together
	        x = tf.placeholder(tf.float32, shape=[None, 3072])                      # None: *any* batch-size; 3072 pixels: single flattened 32 x 32 x 3 sample
	        x_image = tf.reshape(x, [-1, 32, 32, 3])                                # Reshape input image into 4D Tensor [1, H, W, D]
	        y_ = tf.placeholder(tf.float32, shape=[None, 10])                       # Initialze node for target output classes, 10 classes specified
	        phase = tf.placeholder(tf.bool)                                         # Placerholder for batch_norm, training (True) or not training (False)

	    #---------------------------------------------------------------------
	    # Create CNN Architecture
	    #   - Dropout is used in training and turned off for testing
	    #   - Batch Normalization will be similar for training and testing
	    #---------------------------------------------------------------------

	    # Block 1
	    conv1 = conv_and_relu(x_image, [7, 7, 3, 64])                               # Convolutional Layer and ReLU Layer
	    bn1 = batch_norm(conv1, phase)                                              # Batch Normalization (input, True or False)
	    pool1 = max_pool2x2(bn1)                                                    # Max Pooling Layer

	    # Block 2
	    conv2 = conv_and_relu(pool1, [7, 7, 64, 128])
	    bn2 = batch_norm(conv2, phase)
	    pool2 = max_pool2x2(bn2)

	    # Fully-Connected Layer (MLP)
	    fc1 = fully_connected(pool2, [8 * 8 * 128, 1024])

	    # Dropout Layer - Reduce Overfitting, Scaling handeled automatically
	    keep_prob = tf.placeholder(tf.float32)                                      # Stores probability that neurons output is kept during dropout.
	    fc_drop = tf.nn.dropout(fc1, keep_prob)                                     # The placeholder allows us to train with dropout and test without it

	    # Readout Layer (Network Conclusion)
	    weights, biases = weight_and_bias([1024,10])                                # Weights & Bias, Convolutional Filter [Height, Width, Depth, Feature Maps]
	    y_conv = tf.matmul(fc_drop, weights) + biases                               # Softmax Regression Layer

	    print(pool1.shape)
	    print(pool2.shape)
	    print(fc1.shape)
	    print(y_conv.shape)

	    #---------------------------------------------------------------------
	    # Training Convolutional Neural Network
	    #   - Learning rate is set to 1e-4 in "Train_step"
	    #   - Tensorflow offers MANY Optimization techniques: ADAM, SGD, etc
	    #   - tf.argmax() returns index of highest entry. **One hot Vector**
	    #       - tf.argmax(y_conv) - prediction output from model
	    #       - tf.argmax(y_) - True Output label
	    #   - keep_prob and feed_dict are used to control dropout for testing
	    #---------------------------------------------------------------------

	    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)                     # Add for Batch Normalization
	    with tf.control_dependencies(update_ops):                                   # Add for Batch Normalization

	        with tf.name_scope('Cross_Entropy'):                                    # Naming scope for all "Cross Entropy" information
	            cross_entropy = tf.reduce_mean(                                     # Implement Cross_Entropy to compute the softmax activation
	            tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))  # Cross Entropy: True Output Labels (y_), Softmax output (y_conv)
	            tf.summary.scalar('cross_entropy', cross_entropy)                   # Add cross_entropy to our graph

	        with tf.name_scope('train'):                                            # Naming scope for our "Training Step"
	            train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy)   # Train Network, Tensorflow minimizes cross_entropy via ADAM Optimization instead of SGD

	        with tf.name_scope('Train_Results'):                                    # Naming Scope for "Training Results"
	            with tf.name_scope('Correct_Prediction'):                           # Naming Scope for "Correct_Prediction"
	                correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))  # Check if prediction is wrong with tf.equal(CNN_result,True_result)

	            with tf.name_scope('Accuracy'):                                     # Naming Scope for "Accuracy"
	                accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) # Find the percent accuracy, take mean of correct_prediction outputs
	                tf.summary.scalar('accuracy', accuracy)                         # Add Training Accuracy to graph

	    with tf.Session() as sess:                                                  # Make sure session is initalized

	        #---------------------------------------------------------------------
	        # Testing CNN
	        #---------------------------------------------------------------------

	        saver = tf.train.Saver()                                                # Call to interact with stored CNN
	        CNN = tf.train.get_checkpoint_state('data/Saved_CNNs/')                 # Load in trained CNN
	        saver.restore(sess, CNN.model_checkpoint_path)                          # Restore all information of trained CNN

	        data.index_in_epoch = 0                                                 # Reset for testing
	        data.epochs_completed = 0                                               # Reset for testing
	        total = 0                                                               # Initialize final overall accuracy

	        for i in range(100):                                                    # 200 (0->199) (default) testing iterations (1 epoch 200 * 50 = 10,000 samples)
	            batch = next_batch(100, data, shuffle = False, train = False)       # Specify Batch size here, default is 50, here it becomes a tuple
	            batch_new = numpy.reshape(batch[0], (100, 3072))
	            result = accuracy.eval(feed_dict={
	                x: batch_new, y_: batch[1], phase: 0, keep_prob: 1.0})
	            print('Current batch: '+str(data.index_in_epoch)+' , Test Accuracy: ',result)
	            total += result
	        print('Total Accuracy: ', total/i)

	main()