Teaching an AI when not to make a claim

I wear an Oura ring and a Garmin watch, I also wear a Garmin chest strap for training.

Between them, I have quite a lot of data about myself: sleep, resting heart rate, HRV, workouts, training load, recovery, temperature and various scores derived from all of those things.

While I was training for my first Half Marathon, I started wondering if an AI can access all of this, can I just ask it what is going on?

Connecting the data was easy, but the interesting part was deciding how the AI was actually allowed to answer.

One run, two versions of reality

A simple example is a run.

I record it on Garmin, usually with a chest strap. Garmin has the workout, pace, heart rate and training data.

Oura may also notice that I was exercising and create its own record for roughly the same period.

To me, there was one run.

To a system looking at two APIs, there are two pieces of data.

A naive implementation can quite reasonably count both. Suddenly I have exercised twice, accumulated more active minutes than I really did, and possibly burned twice the calories.

Or the system might try to reconcile the disagreement by averaging things that should never have been averaged in the first place.

The worse part is that the answer can still look completely sensible.

So that became the problem I wanted to solve: how do I stop the AI from creating certainty that the data does not justify?

Wearable data is messier than the dashboard makes it look

Consumer health products are very good at turning complicated measurements into simple outputs.

A readiness score of 82 looks precise.

But underneath it may sit several different measurements, collected in different ways, with different levels of reliability and combined using a methodology I cannot necessarily inspect.

That simplification is useful. It is also very easy to over-interpret.

Then put a language model on top.

Language models are extremely good at taking incomplete information and constructing a coherent explanation from it. Usually, that is one of their strengths, but in this context, it's a problem.

If my HRV falls after a difficult training session, there may be a perfectly reasonable physiological explanation.

But there are also plenty of alternatives. Sleep, illness, stress, measurement variation and other factors can all contribute.

There is a large difference between:

Your HRV was lower following a hard training day.

and:

Your HRV was lower because of yesterday's workout.

The second sentence sounds only slightly stronger.

Epistemically, it is doing something quite different.

So I started researching and establishing rules

MetricBraid grew out of trying to make those distinctions explicit.

Rather than letting the model decide how to interpret every conflict, I started defining rules around the evidence first.

For example, if Garmin and Oura both represent the same workout, the system should identify that they refer to one real-world event rather than treating them as two independent observations.

If two devices provide competing measurements, the preferred source depends on what is being measured.

For passive overnight physiology, Oura may be the more appropriate source.

For a deliberately recorded training session using a Garmin and chest strap, Garmin may be the better source.

The important part is not whether those particular choices are universally correct.

It is that the choice is made according to an explicit rule rather than whichever number happens to fit the answer.

There are other rules.

Two observations do not become a trend simply because a model can describe them as one.

Association does not automatically become causation.

My own baseline is usually more useful than a generic population comparison for highly individual metrics such as HRV.

And conflicting information is sometimes something to preserve rather than something to resolve.

If I report that a run felt unusually easy while my physiological data suggests the opposite, that disagreement may itself be useful information.

The system should not necessarily choose which version is “true”.

It should tell me that the two signals disagree.

The rules are grounded in published independent research where that evidence exists. Where it doesn’t, MetricBraid makes that uncertainty explicit rather than dressing a judgement up as fact.

This turned out not to be mainly about wearables

What I find interesting about these rules is that very few of them are really health rules.

They are rules about evidence.

Don't double-count the same underlying event.

Decide how conflicting sources will be treated before seeing which answer you prefer.

Don't turn correlation into causation.

Be explicit when the sample is too small.

Compare against an appropriate baseline.

Preserve uncertainty where uncertainty exists.

Those problems show up everywhere.

A product team can see two analytics events and accidentally count one user behaviour twice.

A dashboard can show a change that looks important without showing whether it sits comfortably inside normal variation.

A piece of user research can become “users want X” after three people mention it.

And an AI placed on top of any of those systems can make the conclusion sound considerably more certain without adding any new evidence.

That is the part of AI products I keep coming back to.

We spend a lot of time thinking about what models are capable of saying.

I think an equally important product question is deciding what evidence they need before they are allowed to say it.

The useful answer is sometimes no answer

MetricBraid is still an experiment.

Technically, it connects data from my wearable systems and gives an AI enough context to reason across them.

But the integration is not really the thing I am building.

The more interesting layer is the set of constraints between the data and the answer: provenance, source preference, duplication, confidence, baselines and rules for what can reasonably be inferred.

The best output is therefore not always an insight.

Sometimes it is:

There isn't enough evidence to support that conclusion yet.

For a system designed to generate answers, I think that is a useful capability.