Weighted Least Squares Regression in Python

Weighted Least Squares (WLS) regression is a powerful extension of ordinary least squares regression, particularly useful when dealing with data that violates the assumption of constant variance.

In this guide, we will learn brief overview of Weighted Least Squares regression and demonstrate how to implement it in Python using the statsmodels library.

What is Least Squares Regression?

Least Squares Regression is a method used in statistics to find the best-fitting line or curve that summarizes the relationship between two or more variables. Imagine you're trying to draw a best-fitting line through a scatterplot of data points. This line summarizes the relationship between two variables. LSR, a fundamental statistical method, achieves exactly that. It calculates the line that minimizes the total squared difference between the observed data points and the values predicted by the line.

What is Weighted Least Squares Regression?

Weighted Least Squares (WLS) Regression is a type of statistical analysis used to fit a regression line to a set of data points. It's similar to the traditional Least Squares method, but it gives more importance (or "weight") to some data points over others. WLS regression assigns weights to each observation based on the variance of the error term, allowing for more accurate modeling of heteroscedastic data. Data points with lower variability or higher reliability get assigned higher weights. When fitting the regression line, WLS gives more importance to data points with higher weights, meaning they have a stronger influence on the final result. This helps to better account for variations in the data and can lead to a more accurate regression model, especially when there are unequal levels of variability in the data.

Formula:

Where,

• ​ is the vector of estimated coefficients.
• X is the matrix of independent variables (with each row representing an observation and each column a different variable).
• W is a diagonal matrix of weights, where larger weights indicate observations with greater importance or reliability.
• y is the vector of dependent variable observations.

Weighted Least Squares Regression Implementation in Python

In Python, the statsmodels library is commonly used for various statistical modeling tasks, including ordinary least squares (OLS) regression. For weighted least squares (WLS) regression implementation we will use statsmodels library.

Steps for Weighted Least Squares Regression Implementation in Python

• Define your sample data:
  • Create arrays for the independent variable(s) (X) and dependent variable (y).
  • Ensure that your dependent variable (y) has more variability or heteroscedasticity to justify the use of weighted least squares regression.
• Calculate weights:
  • Calculate the errors by subtracting the mean of the dependent variable (y) from each observed value.
  • Compute the variance of the errors.
  • Calculate the weights as the inverse of the error variance. This step ensures that observations with higher variance contribute less to the estimation.
• Add constant term:
  • Include a constant term in the independent variable(s) matrix (X) using sm.add_constant().
• Fit the model:
  • Use sm.WLS() to specify the weighted least squares regression model.
  • Use .fit() to estimate the parameters of the model.
• Inspect results

In the below code, the implementation is demonstrated using statsmodels library.

Output:

 WLS Regression Results 
==============================================================================
Dep. Variable: y R-squared: 0.975
Model: WLS Adj. R-squared: 0.967
Method: Least Squares F-statistic: 118.5
Date: Wed, 10 Apr 2024 Prob (F-statistic): 0.00166
Time: 12:48:35 Log-Likelihood: 0.72561
No. Observations: 5 AIC: 2.549
Df Residuals: 3 BIC: 1.768
Df Model: 1 
Covariance Type: nonrobust 
==============================================================================
 coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
const 1.7300 0.283 6.105 0.009 0.828 2.632
x1 0.9300 0.085 10.885 0.002 0.658 1.202
==============================================================================
Omnibus: nan Durbin-Watson: 3.395
Prob(Omnibus): nan Jarque-Bera (JB): 0.537
Skew: 0.600 Prob(JB): 0.765
Kurtosis: 1.935 Cond. No. 8.37
==============================================================================

• The R-squared value is 0.975, indicating that 97.5% of the variance in the dependent variable is explained by the independent variable(s). Adjusted R-squared is a modified version of R-squared that adjusts for the number of predictors in the model.The Adjusted R-squared value is 0.967.
• The F-statistic tests the overall significance of the regression model. F-statistic value is 118.5 with a low p-value 0.00166 indicating that the model is statistically significant.

Ordinary Least Squares Regression Vs Weighted Least Squares Regression

Advantages of Weighted Least Squares Regression

• Handles Varying Data Uncertainty: WLS regression accommodates data where the uncertainty (variance) changes across observations, providing more accurate results compared to OLS regression.
• Improved Parameter Estimates: By giving more weight to reliable data points, WLS regression offers more precise estimates of coefficients and standard errors, especially in the presence of heteroscedasticity.
• Robustness: WLS regression can yield more robust estimates, making it suitable for various fields where data exhibit heteroscedasticity.

Disadvantages of Weighted Least Squares Regression

• Need for Correct Weighting: Correctly specifying weights based on error variance is crucial; incorrect weights can lead to biased results.
• Complexity in Weight Determination: Determining appropriate weights, especially in complex datasets, can be challenging and may require careful consideration.
• Computational Overhead: Implementing WLS regression may involve additional computational complexity, especially with large datasets or complex weighting schemes.
• Sensitivity to Outliers: WLS regression, like OLS, can be sensitive to outliers, which may affect estimation accuracy if not properly addressed.

Conclusion

Weighted Least Squares (WLS) regression offers a valuable enhancement to traditional regression methods by accommodating data with varying levels of uncertainty. By assigning weights based on error variance, WLS regression provides more accurate parameter estimates, making it a powerful tool across diverse fields from finance to healthcare.