Independence
Definition 1 (Statistical independence) Random variables \(X_1, \ldots, X_n\) are statistically independent [1] if, for all sets of real numbers \(A_1, \ldots, A_n\), the probability that every \(X_i\) falls in its set is the product of the individual probabilities:
\[\Pr(X_1 \in A_1, \ldots, X_n \in A_n) = \prod_{i=1}^n{\Pr(X_i \in A_i)}\]
Remark. We write \(X \perp\!\!\!\perp Y\) for “\(X\) and \(Y\) are independent”. The symbol \(\perp\!\!\!\perp\) is essentially \(\prod\) upside-down, which can remind you of the definition.
Theorem 1 (Independence of discrete random variables: the joint PMF factors) Discrete random variables \(X_1, \ldots, X_n\) are statistically independent if and only if their joint PMF factors into the product of their PMFs: for all real numbers \(x_1, \ldots, x_n\),
\[\mathbb{P}(X_1=x_1, \ldots, X_n = x_n) = \prod_{i=1}^n{\mathbb{P}(X_i=x_i)}\]
Proof. Only if. Take \(A_i = {\mathopen{}\left\{x_i\right\}\mathclose{}}\) for each \(i\) in Definition 1.
If. Let \(A_1, \ldots, A_n\) be sets of real numbers, and let \(C\) be the set of tuples \((x_1, \ldots, x_n)\) with each \(x_i \in A_i \cap \mathcal{R}(X_i)\). Each range \(\mathcal{R}(X_i)\) is countable, so \(C\) is countable. Each \(X_i\) takes its values in \(\mathcal{R}(X_i)\), so the event \({\mathopen{}\left\{X_1 \in A_1, \ldots, X_n \in A_n\right\}\mathclose{}}\) is the disjoint union of the events \({\mathopen{}\left\{X_1=x_1, \ldots, X_n = x_n\right\}\mathclose{}}\) over \((x_1, \ldots, x_n) \in C\):
\[ \begin{aligned} \Pr(X_1 \in A_1, \ldots, X_n \in A_n) &= \sum_{(x_1, \ldots, x_n) \in C} \mathbb{P}(X_1=x_1, \ldots, X_n = x_n) && \text{(countable additivity over the disjoint events } {\mathopen{}\left\{X_1=x_1, \ldots, X_n = x_n\right\}\mathclose{}} \text{)} \\ &= \sum_{(x_1, \ldots, x_n) \in C} \prod_{i=1}^n{\mathbb{P}(X_i = x_i)} && \text{(the joint PMF factors)} \\ &= \prod_{i=1}^n{\sum_{x_i \in A_i \cap \mathcal{R}(X_i)} \mathbb{P}(X_i = x_i)} && \text{(a sum over the product set } C \text{ of non-negative products factors)} \\ &= \prod_{i=1}^n{\Pr(X_i \in A_i)} && \text{(countable additivity over the disjoint events } {\mathopen{}\left\{X_i = x_i\right\}\mathclose{}} \text{)} \end{aligned} \]
Example 1 (Two fair coin flips) Flip two fair coins, and let \(X_1\) and \(X_2\) indicate heads on the first and second flip. Each of the four outcomes has probability \(1/4\), so for example:
\[ \begin{aligned} \mathbb{P}(X_1 = 1, X_2 = 1) &= \tfrac{1}{4} \\ &= \tfrac{1}{2} \cdot\tfrac{1}{2} \\ &= \mathbb{P}(X_1 = 1)\,\mathbb{P}(X_2 = 1) \end{aligned} \]
and the same factorization holds for the other three pairs of values, so \(X_1 \perp\!\!\!\perp X_2\) by Theorem 1. In contrast, \(X_1\) and the total number of heads, \(X_1 + X_2\), are not independent: \(\mathbb{P}(X_1 = 0, X_1 + X_2 = 2) = 0\), but
\[ \begin{aligned} \mathbb{P}(X_1 = 0)\,\mathbb{P}(X_1 + X_2 = 2) &= \tfrac{1}{2} \cdot\tfrac{1}{4} \\ &= \tfrac{1}{8}. \end{aligned} \]
Theorem 2 (Independence with densities: the joint density factors) Continuous case. Continuous random variables \(X\) and \(Y\) with densities \(\mathbb{p}(X = x)\) and \(\mathbb{p}(Y = y)\) are statistically independent if and only if the product \(\mathbb{p}(X = x)\,\mathbb{p}(Y = y)\) is a joint density of \(X\) and \(Y\).
Mixed case. A discrete random variable \(X\) and a continuous random variable \(Y\) with density \(\mathbb{p}(Y = y)\) are statistically independent if and only if the product \(\mathbb{P}(X = x)\,\mathbb{p}(Y = y)\) is a joint density-mass function of \(X\) and \(Y\).
Remark. The proof is beyond these notes’ scope: it needs measure theory that the notes do not develop, to extend a density’s integrals from intervals to general sets and to exchange the order of integration [2], [3]. The continuous case extends to \(n\) random variables with a joint density of all \(n\). The statement says “a joint density” rather than “the joint density” because a density is not unique: changing it on a set of zero area (zero length, in the mixed case) changes none of its integrals.
Example 2 (The PMF form fails for continuous random variables) The factorization in Theorem 1 does not work as a definition of independence for continuous random variables: there, both sides are \(0\) at every point, so it would call every pair of continuous random variables independent. For instance, let \(X \sim \text{Uniform}(0, 1)\) (uniform distribution), whose density is \(1\) on \([0, 1]\), and let \(Y = X\). For all real numbers \(x\) and \(y\), the event \({\mathopen{}\left\{X = x,\, Y = y\right\}\mathclose{}}\) is \({\mathopen{}\left\{X = x\right\}\mathclose{}}\) if \(y = x\) and empty otherwise, so it has probability \(0\), because \(\Pr(X = x) = 0\) for the continuous \(X\). Likewise
\[ \begin{aligned} \Pr(X = x)\,\Pr(Y = y) &= 0 \cdot 0 \\ &= 0, \end{aligned} \]
so the PMF form holds. But \(X\) and \(Y\) are not independent:
\[ \begin{aligned} \Pr(X \in [0, \tfrac{1}{2}],\, Y \in [0, \tfrac{1}{2}]) &= \Pr(X \in [0, \tfrac{1}{2}]) && \text{(} Y = X \text{)} \\ &= \int_0^{1/2} 1\,dx && \text{(the density of } X \text{ is } 1 \text{ on } [0, 1] \text{)} \\ &= \tfrac{1}{2} && \text{(integrate)} \end{aligned} \]
while
\[ \begin{aligned} \Pr(X \in [0, \tfrac{1}{2}])\,\Pr(Y \in [0, \tfrac{1}{2}]) &= \tfrac{1}{2} \cdot\tfrac{1}{2} \\ &= \tfrac{1}{4}. \end{aligned} \]
Definition 2 (Conditional independence) Random variables \(Y_1, \ldots, Y_n\) are conditionally independent [4] given a discrete random variable (or vector) \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\) if, for every value \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with \(\Pr(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) > 0\) and all sets of real numbers \(A_1, \ldots, A_n\), the conditional probability that every \(Y_i\) falls in its set is the product of the individual conditional probabilities:
\[\Pr(Y_1 \in A_1, \ldots, Y_n \in A_n \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \prod_{i=1}^n{\Pr(Y_i \in A_i \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})}\]
When \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\) is continuous, every event \(\{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\}\) has probability 0, so the condition is stated with densities instead: for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with \(\mathbb{p}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) > 0\) and all \(y_1, \ldots, y_n\),
\[\frac{\mathbb{p}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, Y_1 = y_1, \ldots, Y_n = y_n)}{\mathbb{p}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})} = \prod_{i=1}^n{\frac{\mathbb{p}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, Y_i = y_i)}{\mathbb{p}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})}}\]
where each \(\mathbb{p}(\cdot)\) is a joint density (with probability masses in place of densities for any discrete \(Y_i\), as in a joint density-mass function).
Remark. For continuous \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\), each ratio in the density form is the density of the \(Y\)’s under their conditional distribution given \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), so both forms say the same thing: given \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), the \(Y_i\)’s joint distribution factors.
Example 3 (Two tests of the same patient) Let \(X\) indicate whether a patient has a disease, and let \(Y_1\) and \(Y_2\) indicate positive results on two tests whose errors are unrelated, so that \(Y_1\) and \(Y_2\) are conditionally independent given \(X\). Suppose \(\Pr(X = 1) = 0.1\), each test is positive with probability \(0.9\) if \(X = 1\) and with probability \(0.1\) if \(X = 0\). Then, by the law of total probability:
\[ \begin{aligned} \Pr(Y_1 = 1, Y_2 = 1) &= (0.9)(0.9)(0.1) + (0.1)(0.1)(0.9) && \text{(condition on } X \text{; factor given } X \text{)} \\ &= 0.081 + 0.009 && \text{(multiply)} \\ &= 0.09 && \text{(add)} \end{aligned} \]
but
\[ \begin{aligned} \Pr(Y_1 = 1) &= (0.9)(0.1) + (0.1)(0.9) \\ &= 0.18, \end{aligned} \]
so \(\Pr(Y_1 = 1)\,\Pr(Y_2 = 1) = 0.0324 \ne 0.09\): the tests are conditionally independent given \(X\), but not independent.
Example 4 (Two coins, given their total) Let \(Y_1\) and \(Y_2\) indicate heads on two fair coin flips, which are independent (Example 1), and let \(X = Y_1 + Y_2\) be the number of heads. Given \(X = 1\), which has probability \(1/2\), exactly one coin is heads, so:
\[ \begin{aligned} \Pr(Y_1 = 1, Y_2 = 1 \mid X = 1) &= \frac{\Pr(Y_1 = 1, Y_2 = 1, X = 1)}{\Pr(X = 1)} && \text{(definition of conditional probability)} \\ &= \frac{0}{1/2} && \text{(two heads make } X = 2 \text{, not } 1 \text{)} \\ &= 0 && \text{(divide)} \end{aligned} \]
but
\[ \begin{aligned} \Pr(Y_1 = 1 \mid X = 1) &= \frac{\Pr(Y_1 = 1, X = 1)}{\Pr(X = 1)} && \text{(definition of conditional probability)} \\ &= \frac{\Pr(Y_1 = 1, Y_2 = 0)}{\Pr(X = 1)} && \text{(} Y_1 = 1 \text{ and } X = 1 \text{ means } Y_2 = 0 \text{)} \\ &= \frac{1/4}{1/2} && \text{(substitute)} \\ &= \tfrac{1}{2} && \text{(divide)} \end{aligned} \]
and likewise \(\Pr(Y_2 = 1 \mid X = 1) = 1/2\), so the product of the conditional probabilities is \(1/4 \ne 0\): \(Y_1\) and \(Y_2\) are independent, but not conditionally independent given \(X\).
Proposition 1 (Neither independence nor conditional independence implies the other) There are random variables \(Y_1\), \(Y_2\), and \(X\) such that \(Y_1\) and \(Y_2\) are conditionally independent given \(X\) but not independent, and there are random variables \(Y_1\), \(Y_2\), and \(X\) such that \(Y_1\) and \(Y_2\) are independent but not conditionally independent given \(X\).
Example 5 (Wet roads, umbrellas, and rain) Consider Brian Hutchinson’s intuitive illustration of conditional independence [5]:
Let \(w\) denote wet roads, \(u\) denote people carrying umbrellas, and \(r\) denote rain.
- Marginally, wet roads and umbrellas are not independent (\(w \not\perp u\)). If you look out a window and see people holding open umbrellas, that increases the probability that the pavement outside is wet.
- Conditionally on rain, they are independent (\(w \perp u \mid r\)). Once you know with certainty whether it is currently raining, learning whether pedestrians are carrying umbrellas tells you nothing more about the state of the roads:
\[\Pr(w, u \mid r) = \Pr(w \mid r)\,\Pr(u \mid r), \qquad \Pr(w \mid u, r) = \Pr(w \mid r)\]
Proposition 2 (Conditional independence when each \(Y_i\) depends only on its own \(X_i\)) Let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= (X_1, \ldots, X_n)\) be a discrete random vector, and let \(Y_1, \ldots, Y_n\) be conditionally independent given \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\). Suppose also that each \(Y_i\) depends on \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\) only through \(X_i\): for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (x_1, \ldots, x_n)\) with \(\Pr(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) > 0\) and every set of real numbers \(A_i\),
\[\Pr(Y_i \in A_i \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \Pr(Y_i \in A_i \mid X_i = x_i)\]
Then, for every such \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and all sets of real numbers \(A_1, \ldots, A_n\):
\[\Pr(Y_1 \in A_1, \ldots, Y_n \in A_n \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \prod_{i=1}^n{\Pr(Y_i \in A_i \mid X_i = x_i)}\]
Proof. Each conditional probability given \(X_i = x_i\) is defined, because \(\Pr(X_i = x_i) \ge \Pr(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) > 0\). So:
\[ \begin{aligned} \Pr(Y_1 \in A_1, \ldots, Y_n \in A_n \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \prod_{i=1}^n{\Pr(Y_i \in A_i \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})} && \text{(conditional independence given } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\text{)} \\ &= \prod_{i=1}^n{\Pr(Y_i \in A_i \mid X_i = x_i)} && \text{(each } Y_i \text{ depends on } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\text{ only through } X_i \text{)} \end{aligned} \]
Remark. In regression, \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\) is usually the full set of covariates \((X_1, \ldots, X_n)\), and a model usually also assumes, as Proposition 2 does, that each \(Y_i\) depends on the covariates only through its own \(X_i\). That second assumption is a separate one, not part of conditional independence.
Definition 3 (Identically distributed) Random variables \(X_1, \ldots, X_n\) are identically distributed if they all have the same CDF:
\[\forall i\in {\mathopen{}\left\{1, \ldots, n\right\}\mathclose{}}, \forall x \in \mathbb{R}: \Pr(X_i \le x) = \Pr(X_1 \le x)\]
Theorem 3 (Identically distributed discrete random variables have the same PMF) Discrete random variables \(X_1, \ldots, X_n\) are identically distributed if and only if they all have the same PMF: for every \(i \in {\mathopen{}\left\{1, \ldots, n\right\}\mathclose{}}\) and every real number \(x\),
\[\mathbb{P}(X_i = x) = \mathbb{P}(X_1 = x)\]
Proof. Write \(F_i(t) \stackrel{\text{def}}{=}\Pr(X_i \le t)\) for the CDF of \(X_i\).
Only if: the CDF determines the PMF through its jumps. Choose an index \(i\) and a value \(x\) of \(X_i\). Every number below \(x\) lies in exactly one of the intervals \((-\infty, x - 1]\), \((x - 1, x - \tfrac{1}{2}]\), \((x - \tfrac{1}{2}, x - \tfrac{1}{3}]\), \(\ldots\), so the event \({\mathopen{}\left\{X_i < x\right\}\mathclose{}}\) is the disjoint union of the events \(B_1 \stackrel{\text{def}}{=}{\mathopen{}\left\{X_i \le x - 1\right\}\mathclose{}}\) and \(B_k \stackrel{\text{def}}{=}{\mathopen{}\left\{x - \tfrac{1}{k - 1} < X_i \le x - \tfrac{1}{k}\right\}\mathclose{}}\) for \(k \ge 2\), and for each \(K\), the events \(B_1, \ldots, B_K\) have union \({\mathopen{}\left\{X_i \le x - \tfrac{1}{K}\right\}\mathclose{}}\):
\[ \begin{aligned} \Pr(X_i < x) &= \sum_{k=1}^{\infty} \Pr(B_k) && \text{(countable additivity)} \\ &= \lim_{K \to \infty} \sum_{k=1}^{K} \Pr(B_k) && \text{(definition of an infinite series)} \\ &= \lim_{K \to \infty} \Pr(X_i \le x - \tfrac{1}{K}) && \text{(finite additivity)} \\ &= \lim_{K \to \infty} F_i(x - \tfrac{1}{K}) && \text{(definition of the CDF)} \end{aligned} \]
The event \({\mathopen{}\left\{X_i \le x\right\}\mathclose{}}\) is the disjoint union of \({\mathopen{}\left\{X_i < x\right\}\mathclose{}}\) and \({\mathopen{}\left\{X_i = x\right\}\mathclose{}}\), so:
\[ \begin{aligned} \mathbb{P}(X_i = x) &= \Pr(X_i \le x) - \Pr(X_i < x) && \text{(finite additivity)} \\ &= F_i(x) - \Pr(X_i < x) && \text{(definition of the CDF)} \\ &= F_i(x) - \lim_{K \to \infty} F_i(x - \tfrac{1}{K}) && \text{(the display above)} \\ &= F_1(x) - \lim_{K \to \infty} F_1(x - \tfrac{1}{K}) && \text{(} F_i = F_1 \text{)} \\ &= \mathbb{P}(X_1 = x) && \text{(the same two steps, for } X_1 \text{)} \end{aligned} \]
If: the PMF determines the CDF through sums. Let \(S\) be the set of values \(u\) with \(\mathbb{P}(X_1 = u) > 0\). Since the PMFs are equal, \(S\) is also the set of \(u\) with \(\mathbb{P}(X_i = u) > 0\), so \(S \subseteq \mathcal{R}(X_i)\) and \(S \subseteq \mathcal{R}(X_1)\). For every real \(t\), the event \({\mathopen{}\left\{X_i \le t\right\}\mathclose{}}\) is the disjoint union of the events \({\mathopen{}\left\{X_i = u\right\}\mathclose{}}\) over the countably many \(u \in \mathcal{R}(X_i)\) with \(u \le t\), so:
\[ \begin{aligned} F_i(t) &= \sum_{u \in \mathcal{R}(X_i),\, u \le t} \mathbb{P}(X_i = u) && \text{(countable additivity)} \\ &= \sum_{u \in S,\, u \le t} \mathbb{P}(X_i = u) && \text{(drop the terms that are } 0 \text{)} \\ &= \sum_{u \in S,\, u \le t} \mathbb{P}(X_1 = u) && \text{(the PMFs are equal)} \\ &= \sum_{u \in \mathcal{R}(X_1),\, u \le t} \mathbb{P}(X_1 = u) && \text{(restore the terms that are } 0 \text{)} \\ &= F_1(t) && \text{(countable additivity)} \end{aligned} \]
Remark. As with independence (Example 2), the PMF form would not work as a definition in general: every continuous random variable has \(\Pr(X_i = x) = 0\) at every \(x\).
Example 6 (Identically distributed but not independent) In Example 1, \(X_1\) and \(1 - X_1\) (the indicator of tails on the first flip) both take the values \(0\) and \(1\) with probability \(1/2\) each, so they are identically distributed (Theorem 3). They are not independent: knowing one determines the other.
Definition 4 (Conditionally identically distributed) Random variables \(Y_1, \ldots, Y_n\) are conditionally identically distributed given random variables \(X_1, \ldots, X_n\) if the conditional CDF of each \(Y_i\) given \(X_i = x\) is one shared function \(G(y \mid x)\) of \(y\) and \(x\): for every \(i \in {\mathopen{}\left\{1, \ldots, n\right\}\mathclose{}}\), every \(y\), and every \(x\) with \(\Pr(X_i = x) > 0\),
\[\Pr(Y_i \le y \mid X_i = x) = G(y \mid x)\]
When \(X_i\) is continuous, the condition applies at every \(x\) with \(\mathbb{p}(X_i = x) > 0\), with \(\Pr(Y_i \le y \mid X_i = x)\) computed from densities: the integral of \(\mathbb{p}(X_i = x, Y_i = u) / \mathbb{p}(X_i = x)\) over \(u \le y\) (a sum over values \(u \le y\) when \(Y_i\) is discrete).
Example 7 (A shared regression model) Suppose each \(Y_i\) is binary, with \(\Pr(Y_i = 1 \mid X_i = x) = \pi(x)\) for one function \(\pi\) shared by every \(i\) (for instance, \(\pi(x) = x / (1 + x)\) for a dose \(x \ge 0\)). Then \(Y_1, \ldots, Y_n\) are conditionally identically distributed given \(X_1, \ldots, X_n\), with \(G(y \mid x) = 1 - \pi(x)\) for \(0 \le y < 1\) (and \(0\) for \(y < 0\), \(1\) for \(y \ge 1\)). Their marginal distributions can still differ: if participant 1 always receives dose \(0\) and participant 2 always receives dose \(1\), then \(\Pr(Y_1 = 1) = 0\) but \(\Pr(Y_2 = 1) = 1/2\).
Definition 5 (Independent and identically distributed) Random variables \(X_1, \ldots, X_n\) are independent and identically distributed (shorthand: “\(X_i\ \operatorname{iid}\)”) [6] if they are:
Remark. The IID assumption is one of the most common assumptions in introductory statistics: it says a sample \(X_1, \ldots, X_n\) can be treated as \(n\) independent draws from a single shared distribution.
Example 8 (Repeated die rolls) The results of \(n\) rolls of the same fair die are IID: the rolls are independent, and each is uniform on \({\mathopen{}\left\{1, \ldots, 6\right\}\mathclose{}}\).
Definition 6 (Conditionally independent and identically distributed) Random variables \(Y_1, \ldots, Y_n\) are conditionally independent and identically distributed given \(X_1, \ldots, X_n\) (shorthand: “\(Y_i \mid X_i\ \operatorname{ciid}\)” or just “\(Y_i \mid X_i\ \operatorname{iid}\)”) if:
- \(Y_1, \ldots, Y_n\) are conditionally independent given \((X_1, \ldots, X_n)\),
- each \(Y_i\) depends on \((X_1, \ldots, X_n)\) only through \(X_i\), and
- \(Y_1, \ldots, Y_n\) are conditionally identically distributed given \(X_1, \ldots, X_n\).
Example 9 (The usual regression assumption) In Example 7, if the \(Y_i\) are also conditionally independent given all the doses, and each \(Y_i\) depends on the doses only through its own \(X_i\), then \(Y_i \mid X_i\ \operatorname{ciid}\), and the joint conditional PMF is \(\prod_{i=1}^n{\pi(x_i)^{y_i}\mathopen{}\left(1 - \pi(x_i)\right)\mathclose{}^{1 - y_i}}\): one shared function evaluated at each \((x_i, y_i)\).
Hutchinson’s Probability Refresher (27 min) covers independence [7]. The login for the video site is posted on Canvas.
5MinuteAI explains independence in five minutes, following the University of Washington’s CSE 312 [8].