How to set up a Bayesian hierarchical model in PyMC with Neutropic: from the model to the checks

Told to analyse your repeated measures "with PyMC"? Follow Jordan, a PhD student, from a one-line question to model code, a checklist of diagnostics and the traps to avoid — all in real Neutropic screens. The answer is code to run yourself, not a model sampled on the server, and we say so.

Hi, this is the Neutropic team!

Picture this: you have a dataset where every participant was measured several times, and your supervisor says, "That needs a hierarchical model. Try it Bayesian, in PyMC." You have run a repeated-measures ANOVA before, but hierarchical models are new, and Bayesian ones even newer. Open any example notebook and you are hit with priors, chains and divergences all at once. 🥲

The model itself is surprisingly short. The hard part comes after it. If you read numbers without checking that the chains mixed, why divergences appeared and whether the model can reproduce your data, you get a confident wrong answer.

In this post we follow Jordan, a psychology PhD student, setting up a first PyMC hierarchical model for repeated measures. Jordan is fictional, but every screen below comes from a real chat in Neutropic.

Jordan asked one short question

Jordan wanted the big picture before uploading any data, so the question was simply:

How should I set up and check a Bayesian hierarchical model in PyMC for repeated measures?

It is short, but it has what matters: the tool (PyMC), the data structure (repeated measures) and two verbs. "Set up" alone would have produced model code. Adding "and check" is what pulled the diagnostics into the answer.

The request and the steps underneath: finding and reading a method guide.
The request and the steps underneath: finding and reading a method guide.

It read a PyMC method guide before answering

As soon as Jordan sent it, steps appeared. Finding a method guide located a guide for hierarchical models and their diagnostics, and Reading a method guide opened the PyMC guide twice. There was no code-running step in this chat, so nothing was sampled.

Method guides are analysis procedures Neutropic has reviewed in advance. Instead of answering from memory, the model reads a vetted procedure first and follows it.

The model nests observations within participants

The answer opened with the idea: observations nested within participants, a population-level effect plus participant-specific deviations, and, if there is a within-person predictor, letting its effect vary by participant when the data can support it.

The example was a basic Gaussian model on long-format data, one row per measurement. The 47-line code gave each participant their own intercept and an optional slope for a within-person predictor, both written in a non-centred form, and drew prior predictive, posterior and posterior predictive samples.

The model explanation and the code card, with the prior scales marked "illustrative only".
The model explanation and the code card, with the prior scales marked "illustrative only".

Right under the code came a warning: the prior scales are illustrative only. Choose them from the units of your outcome and predictor, and look at prior predictive outcomes before fitting. The answer also noted that if the predictor is between-person rather than within-person, you should separate within-person and between-person effects.

The checks came as a list

The part Jordan liked most was the checklist, with three lines of code to start it.

  • Before sampling: check that the priors imply plausible outcomes, that the design identifies the effects you want, and that the model starts with a finite log probability.
  • Sampling: inspect rank R-hat, bulk and tail ESS, MCSE for the contrast you actually care about, divergences, energy/BFMI, and the traces.
  • Posterior predictive checks: compare replicated and observed data, and also look at residuals by participant and over time.
  • Robustness and prediction: check sensitivity to plausible priors, simulate data with known parameters to test recovery, and validate with splits that match your prediction target.
The three-line check code: summary table, trace plot and posterior predictive plot.
The three-line check code: summary table, trace plot and posterior predictive plot.

Jordan nearly took "more draws" as the fix

When a first Bayesian model shows divergences or low ESS, the instinct is to sample longer. Jordan was about to do just that. The answer warns against it: investigate divergences and poor mixing rather than treating more draws as an automatic fix, and consider reparameterising, stronger justified priors, or simplifying random effects the data cannot support.

Two more cautions stood out. A pooled posterior predictive plot can hide misfit at the participant level or over time. And holding out single observations answers a different question from holding out whole participants.

The checklist in full, including the warnings about mixing and validation splits.
The checklist in full, including the warnings about mixing and validation splits.

One more honest note: this chat did not run the model. The answer presents itself as a template for a continuous outcome with conditionally independent residuals, and the numbers in the code are settings, not results. Jordan still needs to install PyMC and run it locally.

It suggested what to do next

At the bottom were two suggestions:

  1. Extend the model to correlated participant-specific intercepts and slopes, including when that covariance is identifiable and how to diagnose it
  2. Compare the Gaussian model with suitable likelihoods for binary, ordinal and count outcomes, and explain what changes in the predictors and checks

Click one and the conversation continues in the same chat.

Next-step suggestions under the answer.
Next-step suggestions under the answer.

A vague "use PyMC" became code with a checklist

At the startNow
ModelOnly the phrase "hierarchical model"Participant intercepts and optional slopes, non-centred
CodeDigging through example notebooks47 lines of PyMC code plus 3 lines of checks
ChecksKnew R-hat, roughlyBefore sampling, sampling, PPC, robustness
TrapsNo idea what to watchMixing, pooled plots, validation splits, priors
NextStuckTwo suggestions to click

To be clear, this is not a sampled model. The code has to run on Jordan's own machine, and the prior scales need adjusting to the real data's units. If residuals stay correlated over time, the answer says to model that dependence rather than assume the participant effect removes it. Still, starting with a checklist in hand is very different from starting from a blank notebook.

This walkthrough used Pro. Lite (the free plan) gives a short model structure and the key diagnostics; Expert adds prior sensitivity and model comparison.

If repeated measures leave you stuck, one question gets you the model and the checks to run on it. 😎

This week, ask the way Jordan did: write "set up and check". Start with Neutropic for free

More from the blog