# Linear Algebra ## Basic Terminology Linear Algebra is equal to Vectors (V) and Matrices (M) and operation on them, V and M are objects in space with various dimensions. * Scalar - Only magnitude * Vector - Direction & Magnitude (1d array of numbers) * Matrices - 2d array of numbers ( can be represented transform/multiple vectors) * Tensors - N-dimensional array of numbers * Line - Equation of a line in two dimensions is : `y = mx + c` ![](/files/fofGnnkLuvBzldPoJCFi) * Plane - A line in three dimensions called a plane and its equation is given by: `ax+by+cz = d` * Hyperplane - A hyperplane is a subspace whose dimension is one less than that of its ambient space. If space is 3D, then hyperplane is 2d and its equation is given by $$ a\_1x\_1 + a\_2x\_2 + a\_3x\_3 ... + a\_n = d $$ ## Where you can apply linear algebra as a ML engineer * Formulation of Machine Learning problems * Image Processing * Dimensionality Reduction * Natural Language Processing * Recommendation System * Computer Vision ## Vectors Sybmol of vector: **→** Symbol of unit vector **^** ![](/files/m0LcnTWCNLcxymNEkMDx) The magnitude of Vector: The length of a vector is called the magnitude/norm of a vector. There are two norms L1 and L2. L1 Norm - Computes Manhattan distance $$ ||u|| = |x\_1| + |y\_1| $$ ``` # Linear algebra in Python import numpy as np import scipy.linalg as la u = np.array([1,2]) np.linalg.norm(u, ord=1) ``` L2 Norm: Computes Euclidean distance $$ ||u|| = \sqrt{x\_1^2 + y\_1^2} $$ ``` np.linalg.norm(u) # Euclidean Distance ``` **Vector Properties** * Associativity of addition `u + (v + w) = (u + v) + w` * Commutativity of addition `u + v = v + u` * Distributivity of scalar multiplication with respect to vector addition `a(u+v) = au + av` * Multiplicative identity is 1 ``` # Vector Arithmetic u = np.array([1,2]) v = np.array([3,4]) u + v #Addition u - v #Subtraction u * v #Multiplication u / v #Division ``` **Vector Operations** * Dot Product ``` a = np.array([[1, 2], [3, 4]]) b = np.array([[5, 6], [7, 8]]) np.dot(a, b) a@b ``` Scalar Projection: A scalar projection of u onto v is the length of the shadow that u casts on v. ![](/files/0oraIAbhWAlcjKS6OuAb) ``` u = np.array([6, 7]) v = np.array([5, -12]) sc_proj = np.dot(u, v) / np.linalg.norm(v) ``` Vector Projection: A vector projection of u onto v is the vector in the same direction as v, whose length is the scalar projection of u on v. The scalar projection is the magnitude of vector projection. ![](/files/kJSLvQT0Qdz0e5ZzRkzq) ``` vec_proj = (v * sc_proj)/ np.linalg.norm(v) ``` > Q. Find out the scalar projection, vector projection and orthogonal component of (6i + 7j) onto (5i -12j). $$ Sp = ( 6*5 + 7*(-12) ) /\sqrt{5^2 + (-12)^2} $$ $$ Vp = ( Sp \* (5i -12j) ) / \sqrt{5^2 + (-12)^2} $$ $$ Oc = (6i + 7j) - Vp $$ **Linear Independence** A set of vectors is linearly dependent, if one of the vectors can be defined as a combination of others vectors. Hence, for vectors to be linearly independent, the linear combination of the vectors results in zero vector, if and only if all the constants are zero. $$ c\_1v\_1 + c\_2v\_2 + ... + c\_nv\_n = 0 , c\_1 = c\_2 ...c\_n = 0 $$ * Direction of two linearly dependent vectors is the same. This shows that there can be only as many solutions as the no. of independent vectors. * Determines if the vector form a basis * Determines the no of independent equations * Determine the no. of independent vectors {% tabs %} {% tab title="First Tab" %} {% embed url="" %} {% endtab %} {% tab title="Second Tab" %} {% embed url="" %} {% endtab %} {% tab title="Third Tab" %} {% embed url="" %} {% endtab %} {% endtabs %} **Change of basis** Change of basis is a technique applied to finite-dimensional vector space in order to rewrite vectors in terms of a different set of basis elements. The basis is set of N vectors * that are linearly independent * spans the space with the N dimensions Orthonormal Basis: Two vectors are orthogonal when their dot product is zero. When the basis is orthogonal and normalized, the basis is orthonormal. Extra Resource: {% tabs %} {% tab title="First Tab" %} {% embed url="" %} {% endtab %} {% tab title="Second Tab" %} {% embed url="" %} {% endtab %} {% tab title="" %} {% embed url="" %} {% endtab %} {% tab title="" %} {% embed url="" %} {% endtab %} {% endtabs %} ## Matrix * Matrix transform Space/vector * Change of basis can be effected through a matrix transform * Properties * Associative `A + B = B + A` * Transpose `A = AT` * Identity Matrix * Upper triangular Matrix * Lower triangular Matrix * Diagonal Matrix * Orthogonal Matrix `A.AT = AT.A = I` ![](/files/amhUuaQUODrCYfrqK5Hy) ``` # matrix creation in python np.zeros((3,3)) np.ones((3,3)) np.full((3,3), 1) np.eye(3) M = np.array([ [1, 2, 3], [4, 5, 6], [7, 8, 9]]) lower = tril(M) # lower triangular matrix upper = triu(M) # upper triangular matrix d = diag(M) # extract diagonal vector V = inv(M) # inverse equivalence ``` ### Linear Transformations * **Scale Transformation** * **Reflection Transformation** * **Projection Transformation** * **Rotation Transformation** * **Identity Matrix Transformation** **Matrix Properties** * Not commutative `AB ≠ BA` * Dot/Inner Product `A(BC) = (AB)C` * `A*I = A` * Matrix Multiplication (A columns need to be equal to B rows) $$ C m*p = Am*n × Bn\*P $$ ### Matrix Inverse The Inverse of matrix A can be written as: ![](/files/QJVpZI5EcqRd77Zg30cF) ![](/files/csqmZYMlIB37zIK9YtK3) **Gaussian Elimination** to get Matrix Inverse: ![](/files/KJ1lI7HIrxRTgFoxhktN) ### Orthogonal Matrix A matrix with orthonormal row and column vectors is called an orthogonal matrix. Orthonormal means that the vectors in the basis are orthogonal(perpendicular)to each other, and they each have a length of one. * An orthogonal matrix Q is necessarily invertible * The determinant of an orthogonal matrix is either +1 or -1 ![](/files/GCMS25qXaOXO0Yna5XGO) **Gram-Schmidt** Process: Convert a matrix of column vector u1,u2...Un to an orthognal matrix v1,v2...Vn. ### Eigen Vector and Eigen Values {% tabs %} {% tab title="First Tab" %} {% embed url="" %} {% endtab %} {% tab title="Second Tab" %} {% embed url="" %} {% endtab %} {% endtabs %} ## Application of Linear Algebra: **Picture**: * Computers use a discrete form of pictures • * The process of transforming continuous space to discrete space is called digitization * Steps in digitization 1. Picture 2. Sampling + Quantization 3. Digital picture A (2D) picture P is a function defined on a (finite) rectangular subset G of a regular planar orthogonal array. G is called (2D) grid, and an element of G is called a pixel. P assigns a value of P(p) to each point. **Pixels**: PIXELS are ATOMIC ELEMENTS called PICTURE ELEMENTS of a digital image. * It is the smallest element of an image represented on the screen. * A pixel can have values ranging from 0 to 255. • Where 0 is black and 255 is white. * Images are stored as rectangular arrays of pixels. * Each pixel can be thought of as a single square point of colored light. **Channels** * Images can have different channels. Examples - RGB, BGR * R - Red, G - Green, and B -Blue * In the RGB model, the colors are stored as RGB values – red in layer 0, green in layer 1, blue in layer 2. * Grayscale image has just one channel. **Intensity** **Values** * If: R=255, G =255, B = 255, then that pixel is white * If R =0, G = 0, B= 0, then the pixel is black * If R, G, and B are any values in the range(1,255), then the pixel has a particular color shade based on the corresponding intensity values. * Spatial Domain - 2D - the physical 2D location of the pixels * Intensity domain - 3rd Dimension - the actual pixel value **Image Representation**: Image is represneted as n-dimensional array where each pixel is a vector(1d array(\[R,G,B]) of intensity values). * GrayScale Image * Color Image \[[LINK](https://ai.stanford.edu/~syyeung/cvweb/tutorial1.html)] ![](/files/96C7OEucUXM3Z9pbvRmE) ### **Image Processing** * **Intensity Transformation – Filtering \[**[LINK](https://setosa.io/ev/image-kernels/)**]** When the color of every pixel is changed, using a function that gets as input the original pixel, or in more complex cases, a submatrix of pixels (*adjustment of brightness, contrast, and colors,* grayscale conversion, color inversion (negative), gamma correction, *blur and noise reduction)*. **Filters/ Kernels:** An image kernel is a small matrix used to apply effects, such as blurring, sharpening, outlining, or embossing. They're also used in machine learning for 'feature extraction', a technique for determining the most important portions of an image. In this context, the process is referred to more generally as "convolution". ![](/files/ynzdPDO95q0lTveImqaT) **I \* W = ∑ ∑ I(K,L) W(i+K, j+L)** * Kernel is also called convolution matrix, mask or filter * Convolution with different kernels can be used for different image transformations/filtering * You can use different kernels for different Feature extraction like edge detection, Sharpen, blurring, etc. * **Spatial Affine Transformation** When the pixels change their position inside the image, or most precisely when every pixel in the matrix is build based on another pixel of the matrix, but without altering its color. (*Rotation, flips, scaling, skewing, and translation*). * A spatial transformation of an image is a geometric transformation of the image coordinate system. * An affine transformation is any transformation that preserves collinearity(i.e., all points lying on a line initially still lie on a line after transformation) and ratios of distances (e.g., the midpoint of a line segment remains the midpoint after transformation). ``` # Different ways of reading an image # Pillow img = Image.open('download.png') # Matplotlib from matplotlib.image import imread imgarr=plt.imread('download.png') # skimage from skimage import io myimage = io.imread('download.png') plt.imshow(myimage) ``` ## Linear Transformation * Identity * Scaling * Rotation * Translation * Horizontal Shear * Vertical Shear ## Functions, Derivatives, and Integrals ## **Principal Component Analysis** **PCA** is a dimensionality-reduction method that is often used to reduce the dimensionality of large data sets, by transforming a large set of variables into a smaller one that still contains most of the information in the large set or keeps its covariance structure as much as possible. In practice, this algorithm is used for data points that are not necessarily random and In statistics, PCA can be used for estimation. {% tabs %} {% tab title="First Tab" %} {% embed url="" %} {% endtab %} {% tab title="Second Tab" %} {% embed url="" %} {% endtab %} {% tab title="" %} {% embed url="" %} {% endtab %} {% endtabs %} ## Matrix Factorization: SVD Matrix Decomposition, also known as matrix factorization involves describing a given matrix using its constituent elements. The most widely used matrix decomposition method is SVD(**Singular Value Decomposition)**. As such, it is often used in a wide array of applications including compressing, denoising, and data reduction. Given a square or non-square matrix A, linear algebra theorem SVD specifies that: $$ A = U Σ V^t $$ * Where U and V are orthogonal matrices and Σ is a diagonal matrix.� * The matrices are orthogonal so: U^T.U = V^T.V = I * The columns of U are orthonormal eigenvectors of AA^T, the columns of V are orthonormal eigenvectors A^T.A * Σ is a diagonal matrix containing the square roots of eigenvalues from U or V in descending order. A is a matrix that can be seen as a linear transformation. This transformation can be decomposed into three sub-transformation 1. rotation, 2. re-scaling(stretch), and 3. rotation and these three steps correspond to the three matrices U, Σ, and V. The general rule is that the transformation associated with diagonal matrix (Σ) implies only a rescaling of each coordinate without rotation. Steps involved in forming SVD matrices: * Calculate Transpose of matrix A(which needs to be factorized) * Find the eigenvalues and eigenvector of A.A\_transpose * Form matrix U using columns of eigenvector * Form orthogonal matrix U by normalizing above U matrix * Calculate matrix using same above steps but based on A^T.A * Form matrix E by taking the square root of nonzero eigenvalues. SVD used in the recommendation System, Signal Processing, and Image processing. Example(Problem): {% tabs %} {% tab title="First " %} {% embed url="" %} {% endtab %} {% tab title="Second " %} {% embed url="" %} {% endtab %} {% tab title="Third" %} {% embed url="" %} {% endtab %} {% endtabs %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://59r.gitbook.io/ml-university/mathematics/linear-algebra.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.