Eigenvectors: The Stubborn Ones Who Refuse to Change
No, no. It’s not what you’re thinking. While we all have stubborn colleagues in our workplace who refuse to change, this blog isn’t a rant about them :-)
This is about a special set of vectors that don’t change direction when space is transformed.
Core Concept
Before we dive into some mathematical work, let’s first understand what eigenvectors and eigenvalues are.
In this blog, we saw how linear transformations change all the vectors in a space.
That transformation is represented by a matrix, which essentially shows where the basis vectors will land after transformation. Using this matrix, we can determine where any particular vector will end up in the transformed space.
Below, you can see how after transformation, 𝒗⃗, which was originally at [1,1], has now landed at [4, 2].
Pictorially, we can see that 𝒗⃗ has rotated and been knocked off its original path. 𝒗⃗ (on the red line) represents the original space, while 𝒗⃗₁ has been knocked off its path and landed on the blue line.
The Special Vectors
But there are certain vectors that don’t change their direction at all — they only get scaled or squished during the transformation. These vectors are Eigenvectors, and the value by which they scale or squish is the Eigenvalue.
For transformation A, we can find one such 𝒖⃗, which simply gets scaled by a factor of 2 (the eigenvalue) but remains on the same green line.
As you can see from the computation below:
By the way, for the same matrix, there exists another eigenvector: [1,0] with eigenvalue 3. You can try taking the matrix multiplication of matrix A and this vector to see that it doesn’t change direction either — it only gets scaled by a factor of 3.
How to Find Eigenvectors and Eigenvalues
From linear transformations, we know that we can find transformed vectors through matrix multiplication of the transformation matrix and the original vector, which gives us the transformed vector:
A·𝒗⃗ = 𝒗⃗₁
But here, 𝒗⃗ and 𝒗⃗₁ are essentially the same vector, just scaled or squished, so it can be written as:
A·𝒗⃗ = λ𝒗⃗₁, where λ is the scaling factor.
This equation looks unbalanced since we have a matrix on the left side but only a scalar on the right side. We can solve this using the Identity matrix to make both sides comparable:
A·𝒗⃗ — λI·𝒗⃗ = 0
(A — λI)·𝒗⃗ = 0
Finding Eigenvalues
Eigenvalues can be found by solving the characteristic equation:
Finding Eigenvectors
Once eigenvalues are found, they can be plugged into the equation below and solved for a non-zero vector:
(A — λI)·𝒗⃗ = 0
Let’s see this in an example:
Now plug the value of λ back into the equation to solve for v:
Similarly, the second eigenvector can be found by plugging λ = 3.
Does Every Matrix Have Eigenvectors?
The simple answer is “No.” Not all matrices have real eigenvectors that we can visualize.
Consider a rotation matrix that rotates space 90 degrees to the left. When this happens, all vectors in each quadrant move to the next quadrant, and the eigenvectors involve complex numbers rather than real vectors we can visualize.
The “Aha!” Moment
Great! We’ve found eigenvectors and eigenvalues. But why does this matter?
Natural Coordinate System
Since eigenvectors don’t change direction, they form the natural coordinate system of the transformation. They are like the “grain” of wood — working along the grain is easier than working against it.
Matrix Simplification Through Diagonalization
If a matrix has sufficient eigenvectors and eigenvalues, the matrix can be simplified and represented as:
Where:
- D is a diagonal matrix (eigenvalues on the diagonal)
- Columns of P are the eigenvectors of A
We’ll explore diagonalization in more detail later, but this formula helps reduce the matrix to a diagonal form where we only need to compute powers of individual numbers.
Computational Efficiency
Let’s say we need to compute A to the power of 10 in 4D space. This typically requires 1,008 operations, but if A can be reduced to a diagonal matrix, the same computation can be done in just 228 operations — a whopping 4.4 times faster!
For larger matrices (say 100×100), the difference becomes astronomical — millions versus thousands of operations.
Dimensionality Reduction
Principal Component Analysis uses eigenvectors to find the “most important” data directions, compressing high-dimensional data while preserving essential information. This forms the foundation of machine learning, image compression, and data visualization.
Understanding Long-term Behavior
The largest eigenvalue dominates system behavior over time. Google’s PageRank algorithm finds the “most important” web pages this way, and population models use this concept to predict which species will dominate.
Conclusion
Eigenvectors might seem abstract, but they’re the “stubborn” directions that reveal the true nature of any transformation. From speeding up computations to powering page ranking of Google’s search algorithm, they’re the mathematical backbone of countless real-world applications.
The next time you use Google search or see a photo recommendation, remember: eigenvectors are working behind the scenes, refusing to change direction and making our digital world more efficient. Sometimes being stubborn pays off!
