Linear Algebra ›Linear Systems ›Linear Dependence

Linear Dependence and Independence

A set of vectors is linearly dependent if one or more vectors in the set can be expressed as a linear combination of the others. If this is not possible, the vectors are said to be linearly independent.

The formal definition says that a set of vectors v⃗1,v⃗2,⋯ ,v⃗n\vec{v}_1, \vec{v}_2, \cdots, \vec{v}_n is linearly independent if the only solution to

c1v⃗1+c2v⃗2+⋯+cnv⃗n=0c_1 \vec{v}_1 + c_2 \vec{v}_2 + \cdots + c_n \vec{v}_n = 0

is

c1=c2=⋯=cn=0.c_1 = c_2 = \cdots = c_n = 0.

The idea is “can we combine the vectors to cancel each other out?”.

In this lesson I would like to build some intuition for this concept, as I think that - in general, but especially in this case - it’s the most important step to help you link this idea to other important concepts in linear algebra.

We first start with numerical examples and then move on to a more visual way to think of linear dependence.

Numerical Examples

First Example

Take the vectors

u⃗=[123],v⃗=[321],w⃗=[567].\vec{u} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}, \quad \vec{v} = \begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix}, \quad \vec{w} = \begin{bmatrix} 5 \\ 6 \\ 7 \end{bmatrix}.

Now, the vectors are linearly dependent, because

w⃗=2u⃗+1v⃗=2[123]+[321]=[246]+[321]=[567].\begin{align*} \vec{w} &= 2 \vec{u} + 1 \vec{v} \\ &= 2 \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} + \begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix} \\ &= \begin{bmatrix} 2 \\ 4 \\ 6 \end{bmatrix} + \begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix} \\ &= \begin{bmatrix} 5 \\ 6 \\ 7 \end{bmatrix}. \end{align*}

As you can see, w⃗\vec{w} can be expressed as a linear combination of u⃗\vec{u} and v⃗\vec{v}.

Second Example

There is an even simpler case. Take the vectors

v⃗=[4−3],w⃗=[−86].\vec{v} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}, \quad \vec{w} = \begin{bmatrix} -8 \\ 6 \end{bmatrix}.

A “linear combination” of v⃗\vec{v} is just v⃗\vec{v} multiplied by some scalar. But it’s still a trivial case of a linear combination.

And as you can see, you can rewrite w⃗\vec{w} as −2v⃗-2 \vec{v}:

w⃗=−2v⃗=−2[4−3]=[−86].\vec{w} = -2 \vec{v} = -2 \begin{bmatrix} 4 \\ -3 \end{bmatrix} = \begin{bmatrix} -8 \\ 6 \end{bmatrix}.

That means that w⃗\vec{w} is linearly dependent on v⃗\vec{v}.

Third Example

Let’s now look at an example where the vectors are linearly independent.

Take the vectors

u⃗=[−42−1],v⃗=[112],w⃗=[3−12].\vec{u} = \begin{bmatrix} -4 \\ 2 \\ -1 \end{bmatrix}, \quad \vec{v} = \begin{bmatrix} 1 \\ 1 \\ 2 \end{bmatrix}, \quad \vec{w} = \begin{bmatrix} 3 \\ -1 \\ 2 \end{bmatrix}.

To check whether these vectors are independent or not, we use the definition of linear dependence:

0⃗=c1u⃗+c2v⃗+c3w⃗.\vec{0} = c_1 \vec{u} + c_2 \vec{v} + c_3 \vec{w}.

If we can find a solution which is not setting all coefficients to 00, the vectors are dependent. If the only solution is

c1=c2=c3=0,c_1 = c_2 = c_3 = 0,

then the vectors are linearly independent.

So, let’s proceed:

0⃗=c1u⃗+c2v⃗+c3w⃗=c1[−42−1]+c2[112]+c3[3−12]. \begin{align*} \vec{0} &= c_1 \vec{u} + c_2 \vec{v} + c_3 \vec{w} \\ &= c_1 \begin{bmatrix} -4 \\ 2 \\ -1 \end{bmatrix} + c_2 \begin{bmatrix} 1 \\ 1 \\ 2 \end{bmatrix} + c_3 \begin{bmatrix} 3 \\ -1 \\ 2 \end{bmatrix}. \end{align*}

We can rewrite this as a system of equations:

{−4c1+c2+3c3=02c1+c2−c3=0−c1+2c2+2c3=0.\begin{cases} -4c_1 + c_2 + 3c_3 = 0 \\ 2c_1 + c_2 - c_3 = 0 \\ -c_1 + 2c_2 + 2c_3 = 0 \end{cases}.

You can solve this however you want, I will assume that you’re able to solve a linear system of equations. You will find that the only solution to the system is

c1=c2=c3=0.c_1 = c_2 = c_3 = 0.

Therefore, the vectors u⃗\vec{u}, v⃗\vec{v}, and w⃗\vec{w} are linearly independent.

Visual Intuition

The idea that I discuss in this section is extremely important, as we will build upon it a lot in later lessons.

Take the vectors

x⃗=[10],y⃗=[01].\vec{x} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad \vec{y} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}.

As you might have already noticed, these vectors are unit vectors which point in the same direction as the axes in a two-dimensional coordinate system: x⃗\vec{x} points in the direction of the xx-axis, and y⃗\vec{y} points in the same direction as the yy-axis.

With these two vectors, you can reach any point on the xyxy-plane. When you have some point in a coordinate system, say (3,2)(3,2), you are saying that to reach that point you have to “move in the positive xx direction by 33 units and move in the positive yy direction by 22 units”. You can express that same idea with the x⃗\vec{x} and y⃗\vec{y} vectors defined above. To reach the vector

p⃗=[32],\vec{p} = \begin{bmatrix} 3 \\ 2 \end{bmatrix},

you can use

3x⃗+2y⃗. 3 \vec{x} + 2 \vec{y}.

And just like with the point (3,2)(3,2), you can reach any point on this plane. Therefore, the linear combination of x⃗\vec{x} and y⃗\vec{y} describes the xyxy-plane.

The thing is, you don’t really need the vectors to point in the same direction as the axes to reach any point on the plane. Take the vectors

x⃗=[12],y⃗=[56].\vec{x} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \quad \vec{y} = \begin{bmatrix} 5 \\ 6 \end{bmatrix}.

To reach the point (3,2)(3,2), you can use

[32]=−2x⃗+y⃗=−2[12]+[56]=[−2−4]+[56]=[32].\begin{align*} \begin{bmatrix} 3 \\ 2 \end{bmatrix} &= -2 \vec{x} + \vec{y} \\ &= -2 \begin{bmatrix} 1 \\ 2 \end{bmatrix} + \begin{bmatrix} 5 \\ 6 \end{bmatrix} \\ &= \begin{bmatrix} -2 \\ -4 \end{bmatrix} + \begin{bmatrix} 5 \\ 6 \end{bmatrix} \\ &= \begin{bmatrix} 3 \\ 2 \end{bmatrix}. \end{align*}

Just like before, with some linear combination of x⃗\vec{x} and y⃗\vec{y}, it is possible to reach any point on the xyxy-plane.

But now consider the pair of vectors

x⃗=[11],y⃗=[22].\vec{x} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \quad \vec{y} = \begin{bmatrix} 2 \\ 2 \end{bmatrix}.

In this case, we are not able to reach every point on the xyxy-plane. But why? What makes this pair of vectors different from the other two?

The answer is that these two vectors are linearly dependent, because 2x⃗=y⃗2\vec{x} = \vec{y}.

But what does this mean visually? Well, in this simple case it’s easy: the two vectors point in the same direction. In fact, y⃗=2x⃗\vec{y} = 2 \vec{x} is the definition of a parallel vector.

The second vector y⃗\vec{y} does not “unlock” any new territory. Before, the second vector allowed us to go in a different direction compared to the first vector, but now both vectors point in the same direction. By summing up scaled versions of these vectors (aka linear combination) we can only move along a line. Most of the two-dimensional space cannot be reached.

This idea extends to higher dimensions. Take the vectors

x⃗=[101],y⃗=[010],z⃗=[111].\vec{x} = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}, \quad \vec{y} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \quad \vec{z} = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}.

These are three-dimensional vectors. The vectors x⃗\vec{x} and y⃗\vec{y} together allow us to move along a plane in three-dimensional space. That is, their linear combination allows us to reach all the points on a plane in this space.

Okay, so now we need a vector to move away from this plane and reach the rest of the three-dimensional space. Let’s look at z⃗\vec{z}. But wait, z⃗\vec{z} is inside the plane formed by the linear combinations of x⃗\vec{x} and y⃗\vec{y}. That is because

z⃗=x⃗+y⃗.\vec{z} = \vec{x} + \vec{y}.

They’re linearly dependent. The vector z⃗\vec{z} does not give us any “new information”. You cannot reach spots of the 3D space that you weren’t able to reach by combining only x⃗\vec{x} and y⃗\vec{y}.

Basically, each independent vector adds one new dimension. If a vector can be built from the others, it does not add new information.

If this concept is not entirely clear yet, don’t worry. We will cover this extensively in future lessons.