Probability and Statistics for Artificial Intelligence and Machine Learning Part-1

Probability and Statistics for Artificial Intelligence and Machine Learning

Part-1

Probability is primarily a theoretical branch of mathematics that is concerned with the analysis of random phenomena.

Probability deals with the quantification of uncertainty of an event. It measures and predicts the likelihood of occurrence of future events. These events may result in the occurrence of any one of the several possible outcomes.  Probability measure is quantified as a number that can be interval from 0 to 1 (where 0 indicates impossibility and 1 indicates absolute certainty).

Statistics is primarily an applied branch of mathematics, which tries to make sense of observations in the real world and involves the analysis of the frequency of past events.

In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens. In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.

This distinction will perhaps become clearer if we trace the thought process of a mathematician encountering the six sided dice game for the first time.

If this mathematician has learned only probability, he or she would see the dice and think

“Six-sided dice....? Presumably each face of the dice is equally likely to land face up. Now assuming that each face comes up with probability 1/6, the mathematician who knows only probability will figure out the chances of quitting the game.”

If instead a statistician wandered by, he/she would see the dice and think "Those dice may look balanced and unbiased, but how do I know that they are not truly unbiased or loaded?”

Therefore the statistician would watch the game a while, and keep track of how often each number comes up and whether the observations are consistent with the assumption of equal-probability faces. Once confident enough that the dice is fair, he/she will call his friend who is an expert in probability theory for an opinion on how to use probability to win the game.

Different views of probability

There are essentially two points of view which are the frequentist view and the Bayesian view. From the frequentist view probability is understood in terms of the frequency of occurrence of an event expressed as a fraction or a limit over large number or infinite number of observations (e.g. probability of Heads to Tail equals 0.5 for over infinite number of trials. From the Bayesian view probability is seen as a belief as when someone predicts that there is 80 percent chance for it to rain.

Three major sources of uncertainty in machine learning

Machine learning must deal with uncertain and non-deterministic (stochastic) quantities.  Uncertainty of an observation or a value can be caused due to any of the following reasons

1. Noisy data. This causes inherent stochasticity in the system being modeled
2.  Incomplete data due to lack of complete observability of the problem domain.
3.  Imperfect modelling due to inappropriate selection of learning model.

Why Probability and Statistics are important in AI and ML

Probability and Statistics are the foundational pillars of Data Science. In fact, the underlying principle of machine learning and artificial intelligence is nothing but statistical mathematics and linear algebra. Machine learning algorithms learn to predict using uncertain data.

Dealing with uncertain data is fundamental in the field of artificial intelligence and machine learning. Hence knowledge in probability is essential for programmers and developers in the field of AI/ML.

Often you will encounter situations, especially in Data Science, where you have to read some research paper which involves a lot of mathematics in order to understand a the topic and so if you want to get better at Data Science, it's imperative to have a strong mathematical understanding.  

Major mistakes made by beginners in AI and ML programming

1. Software engineering and computer science courses focus on deterministic programs, with inputs, outputs, and no randomness or noise. Software programmers in various areas of data science don’t need to know how to use probability in order to develop software. Therefore it is common for machine learning practitioners coming from the computer science or developer tradition to not know and not value probabilistic thinking.
2. Machine learning programmers read through a textbook on probability or work through the material for an undergraduate course on probabilistic methods to cover the breadth and depth of the theory on probability. This results in learning probability the wrong way, which is beyond the needs of the machine learning practitioner.
3. Courses on probability at undergraduate level is intended to pass exams, the material is focused on the math, theory, proofs, and derivations, which is not the focus for machine learning practitioners. AI/ML practitioners need methods and code examples that they can use immediately on their project.  They only need intuitions behind the complex equations.

Random Variable

A random variable or stochastic variable is a variable which can take several possible real values or numerical outcomes, observed from a random event or experiment. It is an  outcome from an underlying random process. There are three types of random variables, discrete, continuous and mixed. 

A random variable has a probability distribution, which specifies the probability of its values. Therefore its values are characterized by their probability distribution functions or cumulative distribution functions.  

Figure-1: Discrete and Continuous random variable

Discrete random variables

A discrete random variable is one which may take on only a countable number of distinct values and thus can be quantified. For example, you can define a random variable X to be the number which comes up when you roll a fair dice. X can take only take values: [1,2,3,4,5,6] and therefore is a discrete random variable.

Figure-2: Discrete random variable

Examples

1.      # of fruits in a basket

2.      # of students in a class

3.      # of likes in Facebook

4.      # of votes in an election

5.      and so on

Continuous Random variables

A continuous random variable is one which takes an infinite number of possible values. For example, you can define a random variable x to be the height of students in a class. Since the continuous random variable is defined over an interval of values, it is represented by the area under a curve (or the integral over the curve).

Figure-3: Probability density function of continuous r.v.

Examples

1.      Weight of gold ornaments

2.      Wind speed

3.      Amount of rainfall

4.      Body temperature

5.      Voltage and current readings

6.      and so on

Mixed Random variables

Sometimes a r.v. is neither purely discrete nor continuous. Such random variables are called mixed r.v.s. In particular a mixed random variable will have a discrete part and a continuous part. The probability distribution function of such a variable will be discontinuous as shown in figure below

Figure-4

Cumulative probability distribution of a mixed r.v.

Consider a random variable x and its cumulative probability distribution which is defined as follows

f(x) = 0 when x < 0, 

f(x) = 1/x^3 when 0 ≤ x ≤ 1/2, and 

f(x) = 1 when x > 1/2,

This distribution is not supported for x < 0, it is continuous during the interval [0, 1/2] and discrete for all values of x > 3.

Examples of applications

1. Meteorological models—instantaneous wind speed,  or daily rainfall, 
2. Insurance claims and actuarial science, 
3. Sports analytics and so on.

Actuarial science is the discipline that applies mathematical and statistical methods to assess risk in insurance, finance and other industries and professions (Wikipedia).

Figure Credits

Figure-1: Difference between continuous random variable and discrete random variable,  

               onlinemath4all.com

Figure-2: Study.com

Figure-3: Probability Distributions - Continuous Random Variable, Penn State Eberly College of Science, STATS 800

Figure-4: Introduction to Probability, Statistics and Random processes, Mixed Random Variable, probabilitycourse.com