# Least squares approximation | Linear Algebra | Khan Academy

Suppose matrix A is known It is an n×k matrix Given equation Ax=b In this case, x belongs to Rk Because there are k columns And b belongs to Rn Let us suppose Ax=b has no solution what does this mean? We expand A I think you should know If A is written in this form a1 a2 I write it in the form of a column vector Until ak Then multiply it by Vector [x1,x2,…,xk]’ This is equivalent to this equation I just wrote the two matrices specifically This is equivalent to x1a1 plus x2a2 Until xk*ak is equal to vector b If this formula has no solution That means the column vector of A About result b There is no weight in other words There is no linear combination of vector a Make the result equal to b A more profound statement is b is not in the column space of A The linear combination of these terms cannot be equal to it Let’s consider intuitively I first draw the column space of A Column space of A like this I assume it is a plane in Rn It does not have to be flat Can be a very general space This is a column space This is the column space of A If this is column space And b is not in this column space We can draw b like this Assuming this is the origin b starts here This is a 0 vector This is vector b It is not in column space Not in this plane So far we have got this equation We build an augmented matrix Turn it into a simple stepped shape And get a straight line 0=1 So we know it can’t do anything without a solution But can we do better? Obviously we can’t find the solution But can we get What is the closest solution? If you want to find a certain x I call it x* where – I want to find some x* Where Ax is – this is a vector – As close as possible-I write it- As close to b as possible Look at it another way when I mention “near” Actually it’s length So it is the minimum required length- I write it down Find the minimum length of b-Ax* Maybe you already know What will you get When you take the difference between the two And take its length what’s the result? For Ax Ax belongs to the column space I call it v Is to multiply any vector in Rk by matrix A The result obtained belongs to the column space So any Ax is in the column space Maybe this is the vector v It is equal to Ax* We want this vector to be away from this vector As close as possible as long as it satisfies – It must be in column space We hope that the difference between these two vectors minimize Now i want to explain The origin of the term I haven’t given it a proper name If you take this vector Just take this vector v for simplicity – It is equal to the length of this vector Difference for each component Is b1-v1 b2-v2 until bn-vn If you take the length of this vector This is the same as this So it is equal to the root sign I square this length The square of its length is (b1-v1)? plus (b2-v2)? Add to (bn-vn)? I ask for the minimum To take a certain value Minimize this Or I ask for a least squares estimate This is why…… This is my last explanation I want to tell everyone Why is this called Least squares estimation Or the least square solution Or least squares approximation For the equation Ax=b There is no solution to this equation But we can find a certain x* So that if A is multiplied by x* The result is in the column space And make the resulting vector As close to b as possible We have learned in many classes A vector outside the distance space in any space What is the nearest vector? The nearest vector is its projection The vector closest to b in the subspace Is the projection of b in the column space This is the closest vector If the minimum value is required Just ask for x* Where x* is equal to vector b in the subspace That is, the projection in the column space of A Remember what we are doing now Knowing that Ax=b has no solution But we can find a certain x Make the result as close as possible to b So called the least square solution Or least squares approximation This item Obviously in column space Because we multiply A by some vector v What you get is these column vectors Linear combination So it belongs to the column space I want this vector As close as possible to this vector The closest to this vector in the column space Is its projection So Ax is equal to b projection in column space It needs to be equal to it But it is difficult to find You know how to do Multiply matrix A by Multiply the inverse of A’A by A’ It is difficult to find the transformation matrix So we look for Is there a least square solution or optimal solution Simple way It is not a real solution It is an optimal solution This is why we call it Least square solution or least square estimation We subtract b from both sides Will get something interesting If you subtract b from both sides What will happen? I do it on the right The left side of the equation gives Ax* It is difficult to write x and * together Because they are similar If you subtract b Subtract vector b It is equivalent to the projection of b in the column space minus b What i do is Subtract b from both sides of the equation So how much is the projection of b minus b? I draw here It is this vector – I use orange to write It is this vector Is it this vector? If we take the projection of b Is this item minus b And get this vector So there is b plus this vector Equal to the projection of b on the subspace This vector is vertical In fact, it can be seen from the definition of projection This term is orthogonal to Subspace or column space So this term is orthogonal to the column space I can write Ax*-b It is orthogonal to the column space Or it belongs to Orthogonal Complement of Column Space The so-called orthogonal complement is Orthogonal to the subspace In all column spaces Collection of vectors So here This vector perpendicular to the plane Obviously belongs to Orthogonal Complement of Column Space You may be familiar with it What is the orthogonal complement of the column space? Orthogonal Complement of Column Space Is equal to the zero space of A’ Or the left zero space of A We have done it in many classes So that A is multiplied by Least squares estimate of Ax=b-I wrote it x* is Ax=b Least square solution So Ax*-b belongs to A’zero space what does this mean? This shows that if you use A’ Multiply this term by Ax*―― I don’t want to omit the vector symbol above x This is a vector This must not be forgotten So there is Ax*-b If you multiply this term by A’ Both are the same What will you get? This term belongs to the null space of A’ So this term multiplied by A’is equal to 0 This term is the equation of A’x=0 A solution Now let’s see if we can simplify it We know Equation A’Ax*-A’b=0 If on both sides of the equation Add this item Get A’A Multiply the least squares solution of Ax=b Equal to A’b This is the result Why do we do this work? Pay attention to our starting point I said we asked for A solution of Ax=b But it has no solution I said We require at least x* to minimize b Minimize the distance between b and x* We call it the least square solution We call it the least squares solution because When taking the length Or when looking for the minimum length Just begging The minimum squared difference So call it the least squares solution Ask for this minimum We know that it is the vector closest to b in the subspace And the vector closest to b in the subspace Is the projection of b in the subspace Which is the projection in the column space of A We know that for A-I change a color We know that A is multiplied by the least squares solution Should be equal to The projection of b in the column space of A If you can find x that satisfies this condition in Rk Then it is the least square solution We learned before Finding the projection of b is easy to say but difficult to do A lot of work We can do it in a simpler way And this is a simple way If we find this We can get the solution of the equation Knowing that the equation Ax=b has no solution What i want to do is Put both sides of the equation And multiply by A’ If you multiply both sides of the equation by A’ There is A’Ax=A’―― I also use blue to write-A- No, this is not a kind of blue-A’b What I did was multiply both sides by this So the solution of this equation Is equal to the solution of this equation There is always a solution to this equation And this term is the least squares solution This is the least square solution Note that this is a matrix This is a vector As long as you can find a solution We found out A solution of the equation Ax=b We minimize the error We ask for Ax* And make the difference between Ax* and b Reach the minimum This is the least square solution This is a bit abstract But hope that in the next class We can realize This is a useful concept 