Tensorflow — The Core Concepts

For a Tensor of 10×10, it makes more sense to create a constant Tensor.

But when working with huge sizes, one should prefer variables.

That is because variables are a lot more efficiently managed.

PlaceholdersConstant Tensors and Variable Tensors are intuitively similar to constants and variables in any programming language.

That does not take time to understand.

The placeholders define Tensors that would get a value just before the code runs.

In that sense, a placeholder can be compared to an input parameter.

This generates a Tensor x — with an assurance that its value will be provided just before the code actually runs.

Lazy ExecutionBy design, TensorFlow is based on lazy execution (though we can force eager execution).

That means, it does not actually process the data available till it has to.

It just gathers all the information that we feed into it.

It processes only when we finally ask it to process.

Such laziness (ironically) provides a huge improvement in the processing speed.

To understand how, we need to understand the Nodes and Graphs of TensorFlow.

Typical Dense Neural NetworkThis is the textbook view of a neural network.

As we can see, we have several inputs X1 — Xn.

These form the first layer of the network.

The second (hidden) layer is obtained as a dot product of each of these with the weight matrix, followed by the activation function like sigmoid or relu.

The third layer is just one value that is obtained as a dot product of its weight matrix with the output of the second layer.

NodeFor TensorFlow, each of these individual entities is a Node.

The the first layer has n+1 nodes (n inputs and 1 constant).

The second layer has k+1 nodes and the third layer has 1 node.

Each of these nodes is represented by a Tensor.

GraphWe can see that some nodes have a constant value (e.


the bias 1).

Some of them have variable values like the weight matrix — we start with a random initialization and tune it through the process.

And we have some nodes whose value is just based on some computation on the other nodes — these are dependent nodes — we cannot get their values until we have the values of the previous nodes.

In this network, we have k nodes in the middle layer and 1 node in the last layer that depend on other nodes — we have k+1 dependent nodes and k variables that we need to tune.

CompilationWhen we create individual Tensors, we just create individual nodes and assign the define the relations — these relations are yet to be implemented.

Once we are done with the definitions, we initiate the compile() method, that identifies this graph connecting the nodes.

This is an important step in the whole process.

If we have circular dependencies or any other reason that could break the graph, the error is identified at this point.

SessionThe TensorFlow computations are always executed in a “session”.

A Session is essentially an environment with a status of its own.

Session is not a thread, but if we have two independent computations that need to run together — without influencing each other, we can use sessions.

Here, A and C will run under session 1, and will see one environment and B and D will run in the session 2 — and see another environment.

Once we have defined the nodes and compiled the graph, we can finally run the command to get value of a particular node in the graph.

When we do so, TensorFlow looks back to check all the nodes that are required for this requested node.

Only those nodes are evaluated in the appropriate order.

Thus, a node in the graph is evaluated only when needed; only if it is needed.

This has a great impact on the processing speed and is a major advantage of TensorFlow.

Simple Code ExampleTo understand TensorFlow it is very important to understand the core concepts of Constants, Variables, Placeholders and Sessions.

Let us now work out an example that can display all of these concepts at once.

Ofcourse, we start by importing the TensorFlow moduleNow, let us define a few Tensors.

Here, t1 and t2 are constants, t3 is a placeholder and t4 is a variable.

Here, we define t1 as a Constant Tensor of size 4×5, with all the values set to 1.

t2 is a Constant Tensor of size 5×4, with random values.

t3 is a placeholder with 0 dimensions — a single number with float32.

Along with this, we define a variable t4 of shape 4×4.

The initializer is set to ones_initializer.

That means, whenever we initialize the variable, its values will be set to 1.

Note that this will happen only when we initialize the variable — not now.

Next, we can define the Tensor expressionThis code takes a dot product of t1 and t2, multiplies it with the scalar t3 and then adds it to t4.

The outcome of this is then assigned to t4.

Thus, the value of t4 changes on every execution of this expression.

Note that t3 is a placeholder, so we will have to provide the value of t3 when we want to process this expression.

Again, this code only defines the expression.

It is not executed right away.

With everything in place, we can now get the session and start working with the TensorsHere, we actually run the code.

We start with initiating the session.

We have to initialize the variables.

So far, t4 was just declared, not initialized.

Here, we actually initialize it.

The initialize code is executed.

In this case, it happens to be “tf.


So, t4 starts as a 4×4 Tensor with all values set to 1.

Next, we run the expression along with the feed_dict.

Remember that the expression has a placeholder t3.

It will not evaluate unless we give it a value for t3.

This value is passed through feed_dict.

Each run updates t4 and assigns a new value to it.

The above code generates this output:Note that t4 did not have a value till it was initialized and its value was altered only when the expression was run in a valid session.

One can evaluate the expression three times to assert the output is same as we expected.


. More details

Leave a Reply