What is calibration?
Calibration is how well your confidence matches what you can actually do. MemGym measures it by asking how sure you are before each card, then comparing that with how it went.
Why you predict first
Before you see the answer, you pick Low, Medium or High. Afterwards you score Missed, Partial or Nailed.
A score on its own says whether you got the card. The prediction adds a second question: did you know you would? The two can differ. You can know a lot and still be more sure than you should be, or know more than you give yourself credit for.
The prediction has to come first. Once you've seen the answer, it's too late to say honestly how sure you were.
How it differs from spaced repetition
Anki and similar apps schedule each card by whether you remembered it, and they do that well. MemGym does that too: a card you got comes back later than one you missed.
Calibration is an extra measurement on top. It shows where your sense of what you know is off, which a recall score alone can't.
How the gap is worked out
Your prediction and your score sit on the same three-step scale:
| Before | After | Counts as |
|---|---|---|
| Low | Missed | 0% |
| Medium | Partial | 50% |
| High | Nailed | 100% |
The gap on one card is the distance between the two. Predict High and score Partial, and the gap is 50%. Predict High and score Missed, and it's 100%. Predict Medium and score Partial, and it's 0%: you were calibrated.
Your calibration error is the average of those gaps. Confidence bias averages the same gaps but keeps their direction, so overconfident and underconfident cards cancel each other out. It says overconfident, underconfident, or well calibrated when the two roughly balance.
Overconfident and underconfident
Overconfident means you were surer than your scores. Underconfident means you did better than you predicted.
Calibration research has found overconfidence more often than underconfidence, especially on hard questions. On easy questions people often lean the other way. This is known as the hard–easy effect (Lichtenstein & Fischhoff 1977; Moore & Healy 2008).
Neither is a judgement of you. Each just tells you which way your confidence is off.
Reading your Progress page
- Calibration error: the average gap, as above. 0% means every prediction matched.
- Confidence bias: which way the gap leans. The number underneath is the average gap with its direction kept: plus for overconfident, minus for underconfident.
- Predictions logged: how many predictions you've made.
The chart shows how your calibration error has changed. Each bar is the average of five predictions in a row. Where it's worst breaks the error down by context tag, most overconfident first, with the number of predictions in each.
With only a few predictions the numbers jump around. They settle as you make more.
An example
You have a plan card: "If someone challenges my reasoning in a meeting, then I answer the point before defending myself." You predict High. In the meeting you manage it partly, so you score Partial. That's a gap of 50%, on the overconfident side.
After a few weeks, Where it's worst might show your meetings cards at 40% and your writing cards at 10%. That tells you where your confidence is least reliable, and so where to look more closely.
What calibration isn't
- A score to beat. Predicting Low on everything doesn't help. Every card you then get right shows up as underconfidence.
- A measure of how much you know. You can know a lot and be poorly calibrated, or know a little and be well calibrated.
- A judgement of you. A large gap means your confidence is off in that area. It's something to look at, not a verdict.
What we don't know yet
Most calibration research uses quiz-style questions. MemGym also asks you to predict plans, habits and situations, which have been studied much less. We don't claim that practising here will improve your calibration. What MemGym does is measure it, so you can see for yourself.