Plain-language explanation.
Bayesian statistics is an approach to statistical inference where probability represents degree of belief rather than long-run frequency. You start with a prior belief about a parameter, update it with data using Bayes' theorem, and get a posterior belief. Bayesian methods are particularly powerful for incorporating domain knowledge, working with small samples, and building hierarchical models.
Core concepts and standard treatment.
Bayesian inference fundamentals: Bayes' theorem (posterior is proportional to likelihood times prior: P(theta|data) proportional to P(data|theta) * P(theta)); conjugate priors (prior and posterior in same family — Beta-Binomial, Normal-Normal, Gamma-Poisson — enabling analytical solutions); and non-informative/weakly informative priors (Jeffreys prior, reference priors, regularising priors like half-Cauchy for scale parameters). Parameter estimation: maximum a posteriori (MAP — mode of posterior), posterior mean and median, and credible intervals (the parameter lies in the interval with stated posterior probability — unlike frequentist confidence intervals).
Deeper theory, debates and edge cases.
Computational Bayesian methods: Markov Chain Monte Carlo (MCMC — Metropolis-Hastings, Gibbs sampling, Hamiltonian Monte Carlo as in Stan and PyMC); Variational Inference (mean-field VI, ADVI — faster approximation to posterior); and Sequential Monte Carlo (particle filters for time-series). Probabilistic programming languages: Stan (NUTS sampler, used by academics and scientists), PyMC (Python-native, NUTS, flexible model specification), and NumPyro (JAX-based, GPU-accelerated, scalable). Bayesian model comparison: WAIC (Watanabe-Akaike Information Criterion), LOO-CV (leave-one-out cross-validation using Pareto-smoothed importance sampling), and Bayes factors.
How it is applied in practice.
Bayesian hierarchical (multilevel) models: partial pooling of information across groups — shrinkage toward group mean prevents overfitting in small groups while retaining variation; applications in education research (school-level and student-level effects), sports analytics (estimating player ability from small sample sizes), and clinical trials (adaptive designs updating interim estimates). Bayesian A/B testing: early stopping rules, sequential testing without alpha inflation, and direct probability statements ("there is 93% probability that variant B is better than A by at least 5%"). Bayesian in industry: Thompson sampling for multi-armed bandit problems (contextual, contextual combinatorial); Bayesian optimisation for hyperparameter tuning (Gaussian process surrogate models, acquisition functions — EI, UCB); and Bayesian neural networks for uncertainty quantification in safety-critical AI applications.