## Bayesian statistics #

Something to formalise my understanding of a little more. Maybe write a few things.

### Notes on Bayesian Data Analysis by Gelman et al #

#### I. Fundamentals of Bayesian inference #

1. Probability and inference

A really nice (re)introduction to probability theory - one of those introductions from a high level that seems to get across the essence of an idea, but that probably only works if you have a basic understanding of what they’re talking about, otherwise it’s impossible to follow. Probably got tripped up by these kinds of resources a bunch in the past.

• Exchangeability, related to i.i.d but slightly different, hadn’t heard of it before.

• Lots of interesting stuff here about the philosophy of statistics, Bayesian vs. frequentist definitions of uncertainty and probability. Nice questions that again I hadn’t really given too much thought to.

• Great quote explaining why combinations of simple to create complexity is better:

Useful probability models often express the distribution of observables conditionally or hierarchically rather than through more complicated unconditional distributions. For example, suppose $$y$$ is the height of a university student selected at random. The marginal distribution $$p(y)$$ is (essentially) a mixture of two approximately normal distributions centered around 160 and 175 centimeters. A more useful description of the distribution of $$y$$ would be based on the joint distribution of height and sex: $$p(\textrm{male}) \approx p(\textrm{female}) \approx \frac12$$ , along with the conditional specifications that $$p(y\mid\textrm{female})$$ and $$p(y\mid\textrm{male})$$ are each approximately normal with means 160 and 175 cm, respectively. If the conditional variances are not too large, the marginal distribution of $$y$$ is bimodal. In general, we prefer to model complexity with a hierarchical structure using additional variables rather than with complicated marginal distributions, even when the additional variables are unobserved or even unobservable; this theme underlies mixture models.

• Idea: As a nice blog post on introductory Bayesian statistics, reproduce the Bayes / Laplace billiard ball example using Monte Carlo methods to estimate the integrals. Maybe in Stan.

1. Single-parameter models

This is a general feature of Bayesian inference: the posterior distribution is centered at a point that represents a compromise between the prior information and the data, and the compromise is controlled to a greater extent by the data as the sample size increases.

• PyMC3 looks like the state-of-the-art in Python for computational Bayesian statistics.
• A great and free book by Andrew Gelman and others that gives a really solid treatment to the whole subject.
• An interesting paper on automatic variational inference, implemented in Stan.
• This nice worked example of the beta-binomial conjugacy that is discussed in Chapter 2 of the above book in a very terse way that I struggled to follow.