Advanced Linear Algebra

Question 1: L2 Norm Calculation

What is the L2 Norm of the vector u = [4,3,0]?

import torch

# Step 1: Define the vector as a PyTorch vector
u = torch.tensor([4.0, 3.0, 0.0])  # float is important for norm ops

l2_norm = torch.norm(u, p=2)  # computes the l2 norm euclidean norm

print("L2 Norm:", l2_norm.item())  # is used to extract the scalar value from a 0D Tensor

QTCM

1. Why is p=2 in the L2 norm?
p in torch.norm(..., p=...) refers to p-norm:

p=1 → Manhattan norm
p=2 → Euclidean norm (L2 norm)
p=∞ → Max norm (largest absolute value)

2. What happens if you don't use .item()?
l2_norm stays a 0 dimensional tensor

print(l2_norm)
# tensor(5.)

print(type(l2_norm))
# <class 'torch.Tensor'>

Question 2: Another L2 Norm

What is the L2 Norm of the vector u = [-2,2,1]?

u = torch.tensor([-2.0, 2.0, 1.0])

l2_norm = torch.norm(u, p=2)
print(l2_norm.item())

Question 3: Cosine of Angle Between Vectors

What is the cosine of the angle between the following two vectors?
U = [1, 2, 2], v = [0, 4, 3]

# define the vectors
u = torch.tensor([1.0, 2.0, 2.0])
v = torch.tensor([0.0, 4.0, 3.0])

# dot product
dot_product = torch.dot(u, v)

# Compute l2 norms
u_l2norm = torch.norm(u, p=2)
v_l2norm = torch.norm(v, p=2)

cos_theta = dot_product / (u_l2norm * v_l2norm)

print(cos_theta.item())

Question 4: Matrix Multiplication

What is the matrix multiplication of the following two matrices?

\begin{pmatrix} 1 & 2 \\ 4 & 3 \\ \end{pmatrix} \times \begin{pmatrix} -1 & 0 \\ 5 & 1 \\ \end{pmatrix}

A = torch.tensor([[1.0, 2.0],
                  [4.0, 3.0]])

B = torch.tensor([[-1.0, 0.0],
                  [5.0, 1.0]])

result = A @ B

Question 5: Matrix Determinant

What is the determinant of the following matrix?

\begin{pmatrix} 1 & -2 & 2 \\ 2 & -4 & 1 \\ 4 & -8 & 1 \\ \end{pmatrix}

A = torch.tensor([[1.0, -2.0, 2.0],
                  [2.0, -4.0, 1.0],
                  [4.0, -8.0, 1.0]])

det_A = torch.linalg.det(A)
print(det_A.item())

QTCM: What is torch.linalg?

Linear Algebra Toolbox

Function	What it does
`torch.linalg.norm()`	Computes vector/matrix norms
`torch.linalg.det()`	Determinant of a square matrix
`torch.linalg.inv()`	Matrix inverse
`torch.linalg.eig()`	Eigenvalues and eigenvectors
`torch.linalg.svd()`	Singular Value Decomposition
`torch.linalg.matrix_rank()`	Rank of a matrix
`torch.linalg.qr()`	QR decomposition
`torch.linalg.solve()`	Solves linear systems $Ax = b$

Question 9: System Analysis

Is the following system of equations overdetermined or underdetermined?

\begin{pmatrix} 1 & 2 & -1 \\ 2 & 10 & 3 \\ \end{pmatrix} \begin{pmatrix} x \\ y \\ z \\ \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \\ \end{pmatrix}

dtype = torch.float32

# 1. Define the matrices
A = torch.tensor([
    [1.0, 2.0, -1.0],
    [2.0, 10.0, 3.0],
], dtype=dtype)

b = torch.tensor([1.0, 2.0], dtype=dtype)

# 2. Reshape b to be a column vector (2, 1) for matrix operations
b = b.reshape(2, 1)

# 3. Analyze the system
num_equations = A.shape[0]
num_unknowns = A.shape[1]
print(f"Number of equations: {num_equations}")
print(f"Number of unknowns: {num_unknowns}")

if num_unknowns > num_equations:
    print("Result: the system is underdetermined")
elif num_equations > num_unknowns:
    print("Result: the system is overdetermined")
else:
    print("Result: the system has an equal number of equations and unknowns")

# Find the minimum norm solution using torch.linalg.lstsq
try:
    solution_tuple = torch.linalg.lstsq(A, b)
    x = solution_tuple.solution
    print(f"Solution: {x}")

    # Verify solution: check if A @ x ≈ b
    reconstruction = torch.matmul(A, x)
    print(reconstruction)
    
    is_close = torch.allclose(reconstruction, b)
    print(f"Is A @ x = b? {is_close}")
    if is_close:
        print("Valid solution")
except torch.linalg.LinAlgError as e:
    print(f"Error: {e}")

Understanding Underdetermined Systems

Intuitive Explanation:

You and I are trying to find a location on a map — a flat grid. You must end up somewhere where the sum of your east and north steps is 5:

x + y = 5

This one rule doesn't lock in exactly where you are... but it does tell you all the places you could be.

In 3D Space:
Instead of walking on a flat map (2D), now you can walk east (x), north (y), and climb upstairs (z).

x + y + z = 6

This is just one equation with three unknowns. There are infinitely many combinations that satisfy this.

Our system:

1·x + 2·y - 1·z = 1 → equation 1
2·x + 10·y + 3·z = 2 → equation 2

A.shape = (2, 3) → 2 equations, 3 unknowns → underdetermined system

Question 10: Solving Linear Systems

Solve the following system of equations:

2x + 3y + 4z = 6
2x + 3y - 4z = 4
2x - 3y + 4z = 2

import torch

# It's best practice to use a floating-point dtype for these operations.
dtype = torch.float32

# 1. Define the coefficient matrix A and the constant vector b
A = torch.tensor([[2.,  3.,  4.],
                  [2.,  3., -4.],
                  [2., -3.,  4.]], dtype=dtype)

b = torch.tensor([6., 4., 2.], dtype=dtype)

print("Matrix A:")
print(A)
print("\nVector b:")
print(b)
print("-" * 25)

# 2. Solve for the vector x in the equation Ax = b
# torch.linalg.solve is the recommended function for this.
try:
    solution = torch.linalg.solve(A, b)

    print("The system has a unique solution.")
    print("Solution vector (x, y, z):")
    print(solution)
    print("-" * 25)

    # 3. Print the results in a more readable format
    x, y, z = solution[0].item(), solution[1].item(), solution[2].item()
    print(f"Solved values are:")
    print(f"x = {x:.4f}")
    print(f"y = {y:.4f}  (fractional value is 2/3)")
    print(f"z = {z:.4f}  (fractional value is 1/4)")
    print("-" * 25)

    # 4. (Optional but recommended) Verify the solution
    # We check if A multiplied by our solution vector gives us back b.
    print("Verification: Multiplying A by the solution...")
    reconstructed_b = torch.matmul(A, solution)
    print("Result of A @ x:")
    print(reconstructed_b)

    # Use torch.allclose to confirm the result is numerically close to the original b
    is_correct = torch.allclose(reconstructed_b, b)
    print(f"\nIs the solution correct? {is_correct}")

except torch.linalg.LinAlgError:
    print("The matrix A is singular (not invertible).")
    print("The system either has no solution or infinite solutions.")

Additional Questions (11-20)

Here are the remaining questions for further exploration:

Question 11: Orthogonal Matrix

Is matrix Q an orthogonal matrix?

\begin{bmatrix} -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \end{bmatrix}

Question 12: Dot Product

Calculate v · u where u = [5,-1,1,2], v = [1, 2, -1, 3]

Question 13: Another Dot Product

Calculate v · u where u = [1,-1,2], v = [1, 5, 1]

Question 14: Characteristic Equation

What is the characteristic equation of the following matrix?

\begin{pmatrix} 0 & 1 & -2 \\ 2 & 1 & 0 \\ 4 & -2 & 5 \\ \end{pmatrix}

Question 15: Eigenvalues

Calculate the eigenvalue(s) of the following matrix:

\begin{pmatrix} 3 & 1\\ 1 & 3\\ \end{pmatrix}

Questions 16-20

Q16: What is the projection of vector b = [1, 3, 1] onto vector a = [-1, -3, -1]?
Q17: Are vectors v = [0, 1, 0] and u = [4, 0, -1] linearly independent?
Q18: Are vectors v = [1, 1] and u = [-1, -1] linearly independent?
Q19: What is the rank of the 2x2 matrix with values [3,1] and [1,3]?
Q20: Is the set of vectors [0,1,0] and [4,0,-1] a basis for R³?