The Jacobian matrix of a vector-valued function compiles all the first-order partial derivatives. For , it helps track how each output changes in response to each input variable.
- J(x) is the Jacobian matrix of f at point x.
- It has m rows and n columns.
Example: How to Calculate the Jacobian Matrix
Find the Jacobian matrix at the point (1,2) of the following function:
First of all, we calculate all the first-order partial derivatives of the function:
Now we apply the formula of the Jacobian matrix. In this case, the function has two variables and two vector components, so the Jacobian matrix will be a 2×2 square matrix:
Once we have found the expression of the Jacobian matrix, we evaluate it at point (1,2):
And finally, we perform the operations:
The Hessian Matrix
The Hessian matrix is an square matrix composed of the second-order partial derivatives of a function of n variables.
The formula for the Hessian matrix is:
Therefore, the Hessian matrix will always be a square matrix whose dimension will be equal to the number of variables of the function. For example, if the function has 3 variables, the Hessian matrix will be a 3×3 dimension matrix.
Example: How to calculate the Hessian Matrix:
Calculate the Hessian matrix at the point (1,0) of the following multivariable function:
First of all we have to compute the first-order partial derivatives of the function:
Once we know the first derivatives, we calculate all the second-order partial derivatives of the function:
Now we can find the Hessian matrix using the formula for 2×2 matrices:
So the Hessian matrix evaluated at the point (1,0) is:
Applications
Applications of Jacobian Matrix
- Multivariate Function Sensitivity: It captures how small changes in inputs affect multiple outputs simultaneously. It is essential for understanding sensitivity and stability in systems.
- Change of Variables in Integrals: The Jacobian determinant is critical for transforming variables in multivariable integrals, including coordinate transformations like polar, cylindrical or spherical coordinates.
- Neural Network Backpropagation: It encodes gradients of vector-valued functions, important for computing derivatives of neural network layers efficiently through automatic differentiation.
- Feature Sensitivity & Explainability: In machine learning, Jacobians provide insight into how sensitive model outputs are to each input feature.
Applications of Hessian Matrix
- Optimization Algorithms: The Hessian provides second-order derivative information that speeds up convergence in optimization. Newton’s method and Quasi-Newton methods (like L-BFGS) use Hessians or approximations.
- Curvature Analysis: Hessians quantify how the gradient changes around a point, guiding step size and direction in gradient-based optimization.
- Natural Gradient Methods: Used in advanced optimization to consider the geometry of parameter space, improving training efficiency.
- Training Large-scale Neural Networks: Techniques leverage diagonal or low-rank Hessian approximations for scalable second-order optimization.
Practice Problems
Jacobian Matrix
- Compute the Jacobian matrix at the point (0, -2) of the following vector-valued function with 2 variables:
- Calculate the Jacobian matrix of the following 2-variable function at the point (2, -1):
- Calculate the Jacobian matrix of the following 2-variable function at the point (2, -1):
Hessian Matrix
- Find the Hessian matrix of the following 2 variable function at point (1,1):
- Calculate the Hessian matrix at the point (1,1) of the following function with two variables:
- Compute the Hessian matrix at the point (0,1,π) of the following 3 variable function:
