{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Mean and Standard Deviation\n",
"\n",
"Welcome! This workshop is from [TrainingDataScience.com](https://trainingdatascience.com/?utm_source=trainingdatascience&utm_medium=notebook&utm_campaign=workshop&utm_term=individual). Sign up to receive more free workshops, training and videos.\n",
"\n",
"This workshop is about two fundamental measures of data. I want to you start thinking about how you can best describe or summarise data. How can we best take a set of data and describe that data in as few variables as possible? These are called _summary statistics_ because they summarise statistical data. In other words, this is your first model!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Mean\n",
"\n",
"The _mean_, also known as the average, is a measure of the tendency of the data. For example, if you were provided some data then you could say that, on average, is most likely best represented by the mean.\n",
"\n",
"The mean is calculated as:\n",
"\n",
"$$\\mu = \\frac{\\sum_{i=0}^{N-1}{ x_i }} {N}$$\n",
"\n",
"The sum of all observations divided by the number of observations."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"x = [6, 4, 6, 9, 4, 4, 9, 7, 3, 6];"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"μ = 5.8\n"
]
}
],
"source": [
"N = len(x)\n",
"x_sum = 0\n",
"for i in range(N):\n",
" x_sum = x_sum + x[i]\n",
"mu = x_sum / N\n",
"print(\"μ =\", mu)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Of course, we should be using libraries to reduce the amount of code we have to write. For low level tasks such as this, the most common library is called Numpy.\n",
"\n",
"We can rewrite the above as:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"μ = 5.8\n"
]
}
],
"source": [
"N = len(x)\n",
"x_sum = np.sum(x)\n",
"mu = x_sum / N\n",
"print(\"μ =\", mu)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can take this even further and just use Numpy's implementation of the mean:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"μ = 5.8\n"
]
}
],
"source": [
"print(\"μ =\", np.mean(x))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Standard Deviation\n",
"\n",
"To describe our data, the mean alone doesn't provide enough information. It tells us what value we should observe on average. But the values could be +/- 1 or +/- 100 of that value. (+/- is shorthand for \"plus or minus\", i.e. \"could be greater than or less than this value\").\n",
"\n",
"To provide this information we need a measure of \"spread\" around the mean. The most common measure of \"spread\" is the _standard deviation_.\n",
"\n",
"Read more about the standard deviation at: [TrainingDataScience.com - Why do we use Standard Deviation and is it Right?](https://trainingdatascience.com/tips/why-do-we-use-standard-deviation/?utm_source=trainingdatascience&utm_medium=notebook&utm_campaign=workshop&utm_term=individual).\n",
"\n",
"The standard deviation of a population is:\n",
"\n",
"$$\\sigma = \\sqrt{ \\frac{\\sum_{i=0}^{N-1}{ (x_i - \\mu )^2 }} {N} }$$"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"x = [6, 4, 6, 9, 4, 4, 9, 7, 3, 6];"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"μ = 5.8\n"
]
}
],
"source": [
"N = len(x)\n",
"mu = np.mean(x)\n",
"print(\"μ =\", mu)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Deviations from the mean: [ 0.2 -1.8 0.2 3.2 -1.8 -1.8 3.2 1.2 -2.8 0.2]\n",
"Squared deviations from the mean: [ 0.04 3.24 0.04 10.24 3.24 3.24 10.24 1.44 7.84 0.04]\n",
"Sum of squared deviations from the mean: 39.6\n",
"Mean of squared deviations from the mean: 3.96\n"
]
}
],
"source": [
"print(\"Deviations from the mean:\", x - mu)\n",
"print(\"Squared deviations from the mean:\", (x - mu)**2)\n",
"print(\"Sum of squared deviations from the mean:\", ((x - mu)**2).sum() )\n",
"print(\"Mean of squared deviations from the mean:\", ((x - mu)**2).sum() / N )"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"σ = 1.98997487421\n"
]
}
],
"source": [
"print(\"σ =\", np.sqrt(((x - mu)**2).sum() / N ))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, we don't need to code this all up. The Numpy equivalent is:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"σ = 1.98997487421\n"
]
}
],
"source": [
"print(\"σ =\", np.std(x))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What's the Catch?\n",
"\n",
"You knew they'd be a catch, right? ;-)\n",
"\n",
"I didn't mention it at the start, but the two previous measures of the central tendency and the spread are specific to a very special combination of data.\n",
"\n",
"If the observations are _distributed_ in a special way, then these metrics perfectly _model_ the underlying data. If not, then these metrics are invalid.\n",
"\n",
"You probably said \"huh?\" to a few of those new words, so let's go through them."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}