Understanding Variance and Covariance

Consider I have three weights that are exactly the same and I want to balance them.

for that, I need to find the point of balance and that point of balance will be Mean.

Consider the below image- weights are placed at point 1,2 and 6.

The point of balance will be 3and it is Mean of points.

MeanLet’s consider below observations.

They have the same Mean but if you see the points on the top are a lot closer to each other and the ones at the bottom are a lot more spread out.

Mean can not interpret how widely data is spread out.

We need another measure to interpret this aspect of observation set and the term Variance comes into the picture here.

VarianceTo calculate Variance, take the distance from each point to the center and take the average of the square of the distance.

The distances are squared because to avoid canceling of negative numbers.

Calculation of VarianceEven if they have the same Mean the difference in Variance shows that they have not distributed over the same range.

Greater the values of variance larger the range over which values are spread.

But this is not true always.

why I am saying this because Variance not only describes how far observations from Mean but also judges each observations importance by its distance.

This means observations far away are judged more important.

confused?Let me put more light on this.

Consider two observation sets having an equal number of observation points and spread over the same range.

but if one observation set has higher Variance than others, it signifies that the observation points are spread far from the Mean.

And one with comparatively small Variance has observation points spread near to Mean.

Till now we saw how to calculate Variance for one-dimensional data.

In the case of two-dimensional observation points, we can find Variance by checking how it is spread out in the horizontal direction and how it is spread out in the vertical direction.

So basically we forget about the height and send everything to the horizontal or X-axis and then calculate the variance of those points and then do the same thing by forgetting the X co-ordinate sending everything to the vertical axis and now with those points we calculate the Y Variance and then we have two numbers the X Variance and the Y Variance .

Variance calculations for two-dimensional dataI hope the concept of Variance is clear by now.

Let’s move ahead.

If you observe below image there are two sets of observation each has three points.

If we calculate the Variance, both observation sets will have the same value.

But they are fundamentally different.

They are not exactly the same.

And variance fails to express that.

Here Covariance comes into the picture.

Understanding Variance and CovarianceIn the context of Machine LearningThis tutorial is a gentle introduction to variance and covariance concepts.

In machine learning, these terms used often.

These are simple terms and maybe that is the reason they considered trivial and nobody explains the intuition behind these terms.

This is my little contribution to explain these terms in an easy way:let’s understand Mean first.

What is the mean of a set of numbers?.Consider I have three weights that are exactly the same and I want to balance them.

for that, I need to find the point of balance and that point of balance will be Mean.

Consider the below image- weights are placed and 1,2 and 6 .

the point of balance will be 3.

And it is an average of points.

Meanlet’s consider below observations.

They have the same mean but if you can see the points on the top are a lot closer to each other and the ones at the bottom are a lot more spread out.

Mean can not interpret how widely data is spread out.

I need another measure which measures that and the term variance comes into the picture here.

VarianceTo calculate variance take the distance from each point to the center.

so the point in the left is a distance one from the center and the point of the right distance one from the center and the point in the middle is this distance zero here considering the distance from the center because it’s is mean where points balance and the same thing in the bottom.

The distances are squared because to avoid canceling of negative numbers.

Calculation of VarianceEven if they have the same mean the difference in variance shows that they are not spread over the same range.

Greater the values of variance larger the range over which values are spread.

but this is not true always.

why I am saying this because variance not only describes how far observations from Mean but also judges each observations importance by its distance.

This means observations far away are judged more important.

confused?Let me put more light on this.

Consider two observation sets having an equal number of observation points and spread over the same range.

It does not mean that they must have the same variance.

It depends on how observations points are spread around the Mean.

If one observation set has higher Variance than others, it signifies that the observation points are spread far from the Mean.

And one with comparatively small Variance has observation points spread near to Mean.

We saw how to calculate Variance for one-dimensional data.

In the case of two-dimensional observation points, we can find Variance by checking how to spread out it is in the horizontal direction and how is it spread out is in the vertical direction.

So basically we forget about the height and send everything to the horizontal or X-axis and then calculate the variance of those points and then do the same thing by forgetting the X co-ordinate sending everything to the vertical axis and now with those points we calculate the Y Variance and then we have two numbers the X Variance and the Y Variance .

Variance calculations for two-dimensional dataI hope the concept of Variance is clear by now.

Let’s move ahead.

If you see below image, there are two observation sets, each includes three points.

if we calculate Variance, both observation sets will have the same value.

But they are fundamentally different.

They are not exactly the same.

And Variance fails to express that.

We need another term which can measure this difference.

Here Covariance comes into the picture.

Covariance answers how these two observation sets are fundamentally different in spite of the same variance.

For that, it uses the product of coordinates.

let’s see above points on the plot for simplification.

Covariance tells us that these above observation sets are different.

let’s calculate the product of Co-ordinateCovariance CalculationIf we take the average of these products,The left observation set with -4/3 has negative Covariance.

It implies that the as X direction grows, the value of Y decreases.

The right observation set has 4/3, positive Covariance it implies that as X direction grows, Y also Grows.

The correlation and covariance are somewhat similar terms.

The correlation measure the strength of the relationship between two variables X and Y Correlation is between -1 and 1 and will be close to 1 in absolute value when the relationship is strong.

Covariance is just the correlation multiplied by the standard deviations of the two variables.

correlation is dimensionless, covariance is in the product of the units for variable X and variable Y.

let’s consider below observation sets to understand the significance of covariance :The first one has negative Covariance because the direction of the observation point shows that the as X grows, Y decreases.

The middle one has very small covariance or almost zero, implies there is no strong relation in X and Y.

and the right observation set has positive covariance because as X direction grows, Y grows.

I hope now we are clear with Variance and Covariance concepts.

For any feedback please use the comment section.

Thanks for reading !!!.

. More details

Leave a Reply