What are Eigenvalues and Eigenvectors?A must-know concept for Machine LearningFarhad MalikBlockedUnblockFollowFollowingJan 6Some concepts live in the heart of data science.
Eigenvectors and eigenvalues are one of those concepts.
This article will aim to explain what eigenvectors and eigenvalues are, how they are calculated and how we can use them.
Eigenvalues and eigenvectors form the basics of computing and mathematicsQuick IntroductionEigenvector —Every vector (list of numbers) has a direction when it is plotted on a XY chart.
Eigenvectors are those vectors when a linear transformation (such as multiplying it to a scalar) is performed on them then their direction does not change.
This attribute of Eigenvectors make them very valuable as I will explain in this article.
Eigenvalue— The scalar that is used to transform (stretch) an Eigenvector.
Eigenvectors and eigenvalues are used to reduce noise in data.
They can help us improve efficiency in computational intensive tasks.
They also eliminate features that have a strong correlation between them and also help in reducing over-fitting.
Photo by Helloquence on UnsplashArticle StructureFirstly I will give a brief introduction of eigenvectors and eigenvalues.
Then I will explain how eigenvectors and eigenvalues are calculated but I will explain them from the foundation of matrix multiplication and addition so that we all can understand them thoroughly.
I will then present a working example and we will calculate the eigenvectors and eigenvalues togetherLastly I will outline how we can compute the eigenvectors and eigenvalues in Python.
It will build our confidence in dimension reduction techniques which are crucial to understand in data scienceYou are probably wondering why eigenvalues and eigenvectorsIntroducing Eigenvalues and Eigenvectors — Where are they used?When we are building forecasting models that are trained on images, sound and/or textual contents then the input feature sets can end up having a large set of features.
It is also difficult to understand and visualize data with more than 3 dimensions.
As a result, we often use one-hot encoding to transform values in textual features to separate numerical columns which can end up taking a large amount of space on disk.
Eigenvalues and Eigenvectors are the key tools to use in those scenariosEigenvalues and Eigenvectors have their importance in linear differential equations where you want to find a rate of change or when you want to maintain relationships between two variables.
Think of eigenvalues and eigenvectors as providing Summary of a large matrixWe can represent a large set of information in a matrix.
One eigenvalue and eigenvector is used to capture key information that is stored in a large matrix.
Performing computations on a large matrix is a very slow process.
To elaborate, one of the key methodologies to improve efficiency in computational intensive tasks is to reduce the dimensions after ensuring most of the key information is maintained.
Component analysis is one of the key strategies that is utilised to reduce dimension space without losing valuable information.
The core of component analysis (PCA) is built on the concept of eigenvalues and eigenvectors.
Additionally, eigenvectors and eigenvalues are used in facial recognition techniques such as EigenFaces.
It is now apparent that Eigenvalues and Eigenvectors are one of core concepts to understand in data science.
Hence this article is dedicated to them.
Have a look at this article if you want to understand dimension reduction and PCA:What Is Dimension Reduction In Data Science?We have access to a large amounts of data now.
The large amount of data can lead us to create a forecasting model where…medium.
comPhoto by Joel Filipe on UnsplashMore usecases of Eigenvalues and EigenvectorsOccasionally we gather data that contains a large amount of noise.
Finding important or meaningful patterns within the data can be extremely difficult.
Eigenvectors and eigenvalues can be used to construct spectral clustering.
We can also use eigenvector to rank items in a dataset.
Lastly in non-linear motion dynamics, eigenvalues and eigenvectors can be used to help us understand the data better as they can be used to transform and represent data into manageable sets.
What are Eigenvalues and Eigenvectors?Eigenvectors are used to make linear transformation understandable.
Think of eigenvectors as stretching/compressing a X-Y line chart without changing its direction.
Eigenvectors and eigenvalues revolve around the concept of matrices.
Matrices are used in machine learning problems to represent a large set of information.
Eigenvalues and eigenvectors is about constructing one vector with one value to represent a large matrix.
Sounds very useful, right?Let’s quickly recap and refresh how matrix multiplication and addition works before we take a deep diveMatrix AdditionMatrix addition is simply achieved by taking each element of a matrix and adding it together as shown below:Multiplying Scalar With A MatrixMultiplying Matrix by a scalar is as straight forward as multiplying each element by the scalar:Matrices MultiplicationMatrices multiplication is achieved by multiplying and then summing matching members of the two matrices.
The image below illustrates how we can multiple a 3 by 3 and a 3 by 1 matrix together:That’s all the Maths which we need to know for the momentLet’s utilise the knowledge above to understand eigenvectors and eigenvalues:Although we don’t have to calculate the Eigenvalues and Eigenvectors by hand every time but it is important to understand the inner workings to be able to confidently use the algorithms.
Key Concepts: Let’s go over the following bullet points before we calculate Eigenvalues and EigenvectorsEigenvalues and Eigenvectors have following components:A matrix has a size of X rows and Y columns.
A square matrix is the one which has a size of n, implying that X and Y are equal.
Square matrix is represented as A.
This is an example of a square matrixEigenvector is an array with n entries where n is the number of rows (or columns) of a square matrix.
Eigenvector is represented as x.
Key Note: The direction of an eigenvector does not change when a linear transformation is applied to it.
Therefore, Eigenvector should be a non-null vectorNow Eigenvalues: We are required to find a number of values, known as eigenvalues such thatA * x – Lambda * x = 0Eigenvalues are represented as lambda.
The above equation states that we need to multiply a scalar lambda (eigenvalue) to the vector x such that it is equal to the linear transformation of matrix A once it is scaled by vector x (eigenvector).
Key note: The above equation should not be invertible.
Determinant of a matrix is a number that is computed from a square matrix.
It is basic arithmetic where the diagonal elements are multiplied by each other and then they are summed together.
As the above equation should not be invertible, we need to ensure that the determinant of the matrix is 0.
Last component: Identity Matrix.
A square matrix which has 1 as diagonal and all other elements are 0 is known as an identity matrix.
Identity matrix is represented as I:We can representA * x – Lambda * x = 0asx * (A – Lambda * I) = 0AdditionallyDeterminant(A – Lambda * I) = 0How do I calculate Eigenvalue?For a matrix A of size n, find Eigenvalues of size n.
The aim is to find: Eigenvector and Eigenvalues of A such that: A * Eigenvector — Eigenvalue * EigenVector = 0Find Lambda Such that Determinant(A — Lambda * I) = 0Based on the concepts learnt above:1.
Lambda * I is:If A is:2.
Then A — Lambda * I is:3.
Finally calculate the determinant of (A-Lambda*I) as:Once we solve the equation above, we will get the values of Lambda.
These values are the Eigenvalues.
I will present a working example below to ensure we understand the concepts thoroughly.
How do I calculate Eigenvector?Once we have calculated eigenvalues, we can calculate the Eigenvectors of the matrix A by using Gaussian Elimination.
Gaussian elimination is about converting the matrix to row echelon form.
Finally it is about solving the linear system by back substitution.
Detailed explanation of gaussian elimination is out of scope of this article so that we can concentrate on Eigenvalues and Eigenvectors.
Once we have the Eigenvalues, we can find Eigenvector for each of the Eigenvalues.
We can substitute the eigenvalue in the lambda and we will achieve an eigen vector.
x * (A – Lambda * I) = 0Therefore if a square matrix has a size n then we will get n eigenvalues and as a result, n eigenvectors will be computed to represent the matrix.
Now that we have the key, it is the time to compute the Eigenvalues and Eigenvectors together with mePhoto by CMDR Shane on UnsplashLet’s calculate Eigenvalue and Eigenvector togetherIf there are any doubts then do inform me.
Let’s find eigenvalue of following matrix:First multiply lambda to an identity matrix and then subtract the two matricesWe will then be left with a matrix which we need to compute a determinant of:Find determinant of the following matrix:Once we solve the quadratic equation above, we will be left with two Eigenvalues:Now that we have computed Eigenvalues, let’s calculate Eigenvectors togetherTake the first Eigenvalue (Lambda) and substitute the eigenvalue into following equation:x * (A – Lambda * I) = 0For the first eigenvalue, we will get following Eigenvector:This Eigenvector now represents the key information of matrix AI want you to find the second Eigenvector.
Message me your values and workings.
Calculate Eigenvalues and Eigenvectors In PythonAlthough we don’t have to calculate the Eigenvalues and Eigenvectors by hand but it is important to understand the inner workings to be able to confidently use the algorithms.
Furthermore, It is very straightforward to calculate eigenvalues and eigenvectors in Python.
We can use numpy.
linalg.
eig module.
It takes in a square matrix as the input and returns eigen values and eigen vectors.
It also raises an LinAlgError if the eigenvalue computation does not converge.
from numpy import linalg as LAinput = np.
array([[2,-1],[4,3]])w, v = LA.
eig(input)SummaryThis article explained one of the key areas of machine learning.
It started by giving a brief introduction of eigenvectors and eigenvalues.
Then it explained how eigenvectors and eigenvalues are calculated from the foundations of matrix multiplication and addition so that we can understand the key components thoroughly.
It then presented a working example.
Lastly it outlined how we can compute the eigenvectors and eigenvalues in Python.
Hope it helps.
Please let me know if you have any questions or feedback.
.. More details