Law of Large Numbers

The Law of Large Numbers (LLN) is a mathematical theorem in probability and statistics that helps us understand how averages work over time. It states that if an experiment is repeated many times, the average result will approach the expected value more closely.

Law of Large Numbers states that the average is closer to the expected or theoretical value as the number of trials or observations increases.
For example: if you flip a fair coin many times, the proportion of heads and tails will get closer to 50% each.

In the graph below, we can see how the proportion of heads approaches 0.5 as the number of trials increases.

The law of large numbers is important because of the following traits:

• Predictability: It helps in predicting the average outcome of random events over time.
• Reliability: It shows that the average of results from a large number of trials is reliable.
• Practical Use: It is widely used in statistics, economics, finance, and other fields to make informed decisions based on data.

Example explaining the law of large numbers: Imagine your bag contains blue and red balls- 50% red, 50% blue.

• If you draw just one ball, it could be red or blue — there's no way to predict the exact outcome, and that single draw doesn’t tell you much about the overall mix in the bag.
• Now suppose you draw 10 balls one by one, noting the color each time. You might get 6 red and 4 blue, or maybe 7 red and 3 blue. The ratio likely won’t be exactly 50:50, but it’ll be close.
• Now imagine repeating this process hundreds or even thousands of times. As your sample size grows, the proportion of red to blue balls in your recorded results will get closer and closer to the actual 50:50 ratio in the bag.

That’s the Law of Large Numbers:

The more trials you perform, the closer your observed average (or proportion) gets to the true theoretical value.

There are two versions of the law of large numbers, which will be discussed below.

Types of Law of Large Numbers

Various types of the Law of Large Numbers are:

1. Weak Law of Large Numbers (WLLN)

The WLLN states that, for a series of independent and identically distributed random variables (such as coin flips), as the number of observations rises the sample average will converge to the theoretical mean.

Imagine you have a fair coin and flip it several times to determine the Weak Law of Large Numbers (WLLN). The Weak Law of Large Numbers indicates that the proportion of heads (or tails) seen will get closer and closer to 0.5 (since the coin is fair, and the probability of obtaining a head or a tail is 0.5), as you increase the number of coin flips.

2. Strong Law of Large Numbers (SLLN)

The Strong Law of Large Numbers (SLLN) states that, for a sequence of independent and identically distributed random variables with a finite expected value, the sample average almost surely (with probability 1) converges to the expected value as the number of observations approaches infinity.

Returning to the coin flipping example, the proportion of heads will almost surely (with probability 1) settle at exactly 0.5 in the long run.

Both versions essentially mean that with enough trials, the results will stabilize around the expected average.

Limitations of the Law of Large Numbers

Various limitations of the Law of Large Numbers are:

• Sample Size: When the sample size is genuinely huge, the Law of huge Numbers operates most effectively. A limited sample size could mean that the results do not represent the underlying population and that the law does not hold.

• Independence: The Law of Large Numbers presupposes that the events or observations are unrelated to one another. The law might not apply correctly if there is any dependence or correlation between the observations.

• Rate of Convergence: The Law of Large Numbers states that as the sample size grows, the sample mean will converge to the population mean, but it doesn't say how quickly this convergence will happen. Sometimes, the convergence could be sluggish, and a huge sample size might be needed to get the appropriate degree of precision.

• Outliers and Extreme Values: The existence of outliers or extreme values in the data can have a significant impact on the Law of Large Numbers. Even with a high sample size, a few extreme observations can considerably affect the sample mean.

• Observations Not Identically Distributed: The Law of Large Numbers presupposes that the data come from a single probability distribution. The law might not hold if the underlying distribution varies over time or if the data are drawn from disparate distributions.

• Biased Sampling: Law of Large Numbers may not apply, and the sample mean may not converge to the true population mean if the sampling procedure is biased or non-random.

• Finite Population: In general, an infinite population is used to state the Law of Large Numbers. The law might need to be changed or altered when working with a finite population to take into consideration the population's finite size.

Law of Iterated Logarithm (LIL)

The Law of the Iterated Logarithm (LIL) refines the results of the Law of Large Numbers by precisely describing the magnitude of fluctuations of the sample average around the expected value. It shows that although the sample average converges to the expected value (as stated in the Strong Law of Large Numbers), it still exhibits small, diminishing oscillations around that value.

In the coin flipping example, the LIL indicates that, as the number of coin flips rises, the sample average will not only converge to 0.5 but also offer a specific range within which it is most likely to fall.

Applications of LLN in Computer Science

The Law of Large Numbers (LLN) is widely used in Computer Science, especially in areas involving randomness, estimation, and simulation. Some of the key applications are:

• Monte Carlo Simulations: A technique that uses repeated random sampling to estimate numerical results used in graphics rendering, AI game engines, risk analysis, and optimization problems.
• Probabilistic Algorithms: Algorithms like Randomized QuickSort, Las Vegas use randomness.
• Distributed Systems: Random sampling is used for decisions like which server to assign a task to.
• Load Balancing: LLN helps predict that, over time, the load will distribute evenly, assuming fair randomness.
• Machine Learning: LLN supports the reliability of training data with more data, and sample statistics (like mean or variance) converge to true population values.
• Network Traffic Analysis: Used to estimate average latency, packet loss, or throughput over time.

Law of Large Numbers in Finance

Law of Large Numbers is a fundamental concept in probability theory and statistics that has significant applications in finance.

In finance, the Law of Large Numbers is particularly relevant in the context of portfolio management and risk analysis. Here are a few examples:

• Investing in a single stock might cause quite erratic and extremely fluctuating returns. On the other hand, if you have a diversified portfolio comprising several stocks, the total portfolio returns usually show better consistency and closer alignment with the average return of the market. This is so because, as the portfolio grows, the positive and negative deviations of individual equities tend to cancel each other out.

• Insurance firms decide their policy premiums using the Law of Large Numbers. Analyzing a sizable pool of policyholders helps one to more precisely estimate the average frequency and severity of claims. This lets them create equitable and sustainable rates for the whole pool.

• In finance, Monte Carlo Simulations help to predict and examine the possible results of several investments or financial plans. The Law of Large Numbers guarantees that the simulated results converge towards the genuine anticipated value or distribution of outcomes as the number of runs rises.

• Financial institutions project the possible losses or benefits connected with different risk factors using the Law of Large Numbers. Analyzing several historical data points or scenarios helps one better estimate and control hazards.

Solved Questions for Law of Large Numbers

Example 1: A fair six-sided die is rolled repeatedly. What is the expected average value of the outcomes as the number of rolls increases

Solution:

Expected value of rolling a fair six-sided die is the average of the numbers 1 through 6, which is (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5

According to the Law of Large Numbers, as the number of rolls increases, the average outcome will approach 3.5

Example 2: In a game, a player flips a fair coin. If it lands heads, the player wins $1; if it lands tails, the player loses $1. What is the expected average profit/loss for the player as the number of flips increases?

Solution:

Expected value of flipping a fair coin is 0.5 for heads and 0.5 for tails

Expected profit/loss for each flip is (0.5 × $1) + (0.5 × -$1) = $0

Therefore, the expected average profit/loss for the player as the number of flips increases approaches $0 by the Law of Large Numbers

Example 3: A bag contains 20 red balls and 30 blue balls. A ball is drawn from the bag, and the color is noted. The ball is then returned to the bag, and the process is repeated. What is the expected proportion of red balls drawn as the number of draws increases?

Solution:

Probability of drawing a red ball on any single draw is 20/(20+30) = 20/50 = 2/5

By the Law of Large Numbers, as the number of draws increases, the proportion of red balls drawn will approach 2/5

Example 4: A factory produces light bulbs, and historical data show that 5% of the bulbs are defective. If a random sample of bulbs is taken from the production line, what is the expected proportion of defective bulbs as the sample size increases?

Solution:

Probability of selecting a defective bulb from the production line is 0.05

As the sample size increases, by the Law of Large Numbers, the proportion of defective bulbs in the sample will approach 0.05

Related Articles:

• Sampling Errors in Statistics
• Bayes’ Theorem
• Binomial Distribution
• Normal Distribution
• Bernoulli Distribution
• Probability Theory