Accuracy vs Precision vs Resolution in Calibration

Key points

• Calibration calibration confirms measurements you can defend in an ISO 9001 / 13485 / AS9100 audit.
• Every certificate is NIST-traceable with a full measurement-uncertainty budget (as-found / as-left).
• Techmaster is ANAB-accredited (AC-1736) with five US labs, in-lab or nationwide on-site.
• Standard turnaround is 5 business days, with 1–2 business-day expedite available.

Accuracy, precision, and resolution describe different parts of measurement performance. Accuracy concerns closeness to a reference value, precision concerns agreement among repeated results, and resolution is the smallest change that produces a perceptible change in indication. A high-resolution display can show many digits while the measurements remain biased, variable, or too uncertain for the decision.

The three terms at a glance

The International Vocabulary of Metrology (VIM) entry for accuracy treats accuracy as a qualitative concept rather than a numeric quantity. The VIM defines measurement precision through agreement among replicate indications or measured values under specified conditions, and resolution through a perceptible change in indication.

Worked example: one gauge block, five readings

Illustrative scenario: a digital caliper with a 0.01 mm display increment measures a reference whose assigned value is 25.000 mm. The reference value and calibration setup have their own uncertainty; the numbers below are simplified for teaching.

The average indication is 25.036 mm. The readings cluster within 0.01 mm, so repeatability under these conditions appears good. Yet the average is about 0.036 mm above the assigned reference value. The instrument can therefore be precise in this short test while showing a systematic offset. Whether it is acceptable depends on the applicable specification, the measurement uncertainty, and the decision rule.

An original visual model

Reference:          |
Readings:          ...
Axis:      low-------------high

Reference:          |
Readings:               ...
Axis:      low-------------high

Reference:          |
Readings:     .     .     .
Axis:      low-------------high

Reference:          |
Readings:          .    .    .
Axis:      low-------------high

Resolution does not set the uncertainty

A 0.01 mm display increment only tells you something about how the instrument indicates changes. It does not prove ±0.01 mm accuracy and does not make total measurement uncertainty 0.01 mm. Uncertainty can include the reference standard, repeatability, resolution, alignment, contact force, temperature, method, and other influence quantities. Some components may be larger than the display increment.

Conversely, a coarse display may hide small changes even when other parts of the measurement system are stable. The number of digits is therefore an input to instrument selection, not a complete performance statement.

Where measurement uncertainty fits

Measurement error is the difference between a measured value and a reference quantity value. Measurement uncertainty describes the dispersion of values that could reasonably be attributed to the measurand based on the available information. Uncertainty is not a known error bar that can simply be subtracted from every result.

When a calibration report states a result such as 25.036 mm with expanded uncertainty 0.012 mm and coverage factor k = 2, the reported interval qualifies the measurement result. Under common assumptions, k = 2 often corresponds to approximately 95% coverage, but the exact interpretation depends on the evaluation. See NIST Technical Note 1297.

How to diagnose a measurement problem

1. Define the measurand and reference value, including units and conditions.
2. Take repeated readings without changing the setup more than the procedure allows.
3. Examine spread to assess repeatability under those conditions.
4. Compare the mean or fitted result with the reference, including uncertainty, to investigate bias.
5. Check multiple points across the working range. A single point can miss linearity or local error.
6. Do not adjust the instrument until as-found results are recorded.
7. Compare the result and uncertainty with the intended tolerance using an agreed decision rule.

For a dimensional example, see caliper calibration. For the reporting concepts behind the result, read measurement uncertainty and traceability explained.

Why it matters

Accredited calibration at Techmaster

Techmaster Electronics is an ANAB-accredited (Certificate AC-1736) ISO/IEC 17025:2017 calibration laboratory with five US labs, NIST-traceable standards and a full measurement-uncertainty budget on every certificate, in-lab or on-site nationwide. See our calibration services or request a quote.

Frequently asked questions