Understanding ADC Calibration and Accuracy

When working with precision analog-to-digital conversion (ADC) systems, especially in low-voltage applications, achieving accurate and repeatable results is critical. Customers often encounter discrepancies between expected and measured ADC values—particularly at voltages below 0.9V—leading to concerns about calibration validity and compliance with datasheet specifications. This article addresses common questions around ADC offset error, calibration procedures, and overall accuracy, drawing from real-world measurement scenarios and clarifying key technical points to help users achieve reliable results.

ADC Calibration Sequence: A Prerequisite for Accurate Results

One of the most frequent sources of unexpected ADC behavior lies in the timing and sequence of calibration and conversion operations. The ADC calibration process is not optional—it is a mandatory step to ensure that the conversion results fall within the specified accuracy limits. According to Figure 3.1 and Figure 3.2 of the Nexus ADC User Guide, the calibration ready signal (adc_calrdy_o) must be asserted before initiating any ADC conversion via the adc_soc_i signal.

If conversions are performed without completing calibration, the resulting data can exhibit significant offset errors, gain drift, or non-linearity, even if the device is otherwise functioning correctly. This is especially noticeable at low input voltages, where small deviations are magnified relative to the full-scale range. In the case described, the observed deviation of several millivolts(mV) below ideal values at voltages below 0.9V may stem from conversions being executed prior to calibration completion.

To resolve this, ensure that your firmware or control logic strictly follows the recommended sequence:

1. Initiate ADC calibration using adc_clk_i input signal.
2. Wait for adc_calrdy_o to go high, indicating calibration is complete.
3. Only then assert adc_soc_i to start a conversion.

Failure to adhere to this sequence invalidates the guaranteed performance metrics. Even if the device is operating within its electrical limits, the absence of proper calibration means the results cannot be trusted for accuracy-critical applications.

Interpreting ADC Offset Error: Clarifying the Datasheet Specification

A common point of confusion arises from the interpretation of the ADC offset error specification listed in the datasheet. The stated value of ±2 LSB ADC Offset Error on CertusPro-NX, CrossLink-NX and MachXO5-NX Family Datasheets has led some users to question its validity, particularly when measured results appear to exceed this limit. However, the root of the issue lies in the unit of measurement.

The correct unit for ADC offset error is ±2 %FS_ADC ADC Offset Error, not LSB. This is a documented internal error in the datasheet, and the specification should be understood as a percentage of the full-scale ADC range, not a fixed number of least significant bits. For a 12-bit ADC with a reference voltage of 1.8V, the full-scale range is 1.8V, and 1 LSB corresponds to approximately 0.439mV (1.8V / 4096). Therefore, ±2LSB would equate to about ±0.878mV.

However, when expressed as %FS_ADC, the offset error is calculated relative to the full-scale voltage. For example, if the offset error is ±0.1% of full scale, this translates to ±1.8mV, which is significantly larger than ±2LSB. This explains why some measurements appear to exceed the ±2LSB threshold—because the specification was misinterpreted.

This clarification is essential: the ±2LSB value is not a standalone metric but part of a broader accuracy model. Relying solely on offset error without considering other error sources leads to an incomplete and potentially misleading assessment of ADC performance.

Total Unadjusted Error (TUE): The Complete Picture of ADC Accuracy

While offset error is important, it is only one component of overall ADC accuracy. To fully understand the deviation between measured and ideal values — particularly in low-voltage regions where errors represent a larger proportion of the signal — it is necessary to consider the Total Unadjusted Error (TUE). TUE represents the worst-case deviation of the actual ADC transfer function from the ideal, before any software or hardware calibration is applied.

1. ADC Offset Error (ADC_Oe): The deviation of the ADC output from zero when the input is zero. This is a fixed shift applied uniformly across the entire transfer function and is most significant at low input voltages.
   
2. ADC Gain Error (ADC_Ge): The deviation in the slope of the transfer function from ideal, measured after offset has been removed. It causes the error to grow proportionally with input magnitude, becoming most noticeable near full scale.
   
3. ADC INL Error (ADC_INLe): Integral Non-Linearity measures the maximum deviation of the actual transfer function from a best-fit straight line, after offset and gain have been corrected. It reflects the inherent curvature or bow in the ADC response.
   
4. ADC DNL Error (ADC_DNLe): Differential Non-Linearity measures the deviation of individual step sizes from the ideal 1 LSB. A DNL of −1 LSB indicates a missing code, while a large positive DNL produces a wider-than-ideal step. Both degrade the effective resolution of the converter.
   

As illustrated in the figures above, these errors are more visually apparent in higher-resolution ADCs where the transfer function is represented as a continuous line. In lower-resolution converters such as the 3-bit example, the coarse staircase response means that even a single LSB of error represents a significant fraction of the full-scale range, making individual step deviations immediately visible. At 12-bit resolution, the steps are imperceptibly small and errors instead manifest as a smooth deviation from the ideal transfer line — which is why a continuous curve representation is used.

Note that 2^N should be used in place of the hardcoded value 4096 to keep the formula general across different ADC resolutions. For a 12-bit ADC, 2^N = 4096.

Worked Example 1 — Single Measurement Verification

For a single ADC measurement with the following conditions:

Parameter	Value
Reference Voltage (VREF)	1.8V
Measured (True) Voltage	0.05019V
ADC Output Voltage	0.04747V
LSB size (VREF/2^N)	1.8/4096 = 0.4395 mV

ADC_Oe(V)   = |0.04747 − 0.05019| = 0.00272 V

ADC_Oe(LSB) = 0.00272 / 0.0004395 = 6.19 LSB

%FS_ADC (Offset Error) = 6.19 / 4096 × 100   = 0.151%

A %FS_ADC (Offset Error) of 0.151%FS for a 12-bit ADC is within the acceptable range for most general-purpose and industrial ADC datasheets. This confirms that despite the apparent deviation of approximately 6 LSB, the measurement remains within expected device performance bounds.

Note: This analysis considers offset error only. In a full TUE assessment, gain error, INL, and DNL from the device datasheet should also be included in the RSS calculation to obtain the complete error budget.

Worked Example 2 - DNL and INL Calculation and Overall Solution
Given a voltage sweep from 0V to 0.5V with VREF = 1.80002493V

      VREF     = 1.80002493 V

      LSB size = 1.80002493 / 4096 = 0.43946 mV

1. Offset Error - the offset error is the deviation at zero input -- the first measurement where the input is closest to zero

Ideal code at DMM = 0.000525155 V:

= 0.000525155 / 0.00043946

= 1.2 LSB

ADC_Oe = Actual code − Ideal code

       = 0 − 1.2

ADC Offset Error = −1.2 LSB

2. Gain Error - the gain error is calculated from the slope difference between the actual and ideal transfer functions, using the first and last measurement points:

Ideal slope = 4096 / 1.80002493

             = 2275.5 codes/V

Actual slope = (1133 − 0) / (0.500234415 − 0.000525155)

             = 2267.32 codes/V

Gain Error   = ((Actual slope − Ideal slope) / Ideal slope) × 4096

             = ((2267.32 − 2275.52) / 2275.52) × 4096

ADC Gain Error = -14.77 LSB

3. DNL Error - DNL measures how wide each step is compared to the ideal 1 LSB step width

            DNL = (Actual step width − Ideal step width) / Ideal step width

For example, VREF = 1.80002493V

LSB size = 1.80002493 / 4096 = 0.439459 mV (ideal step width)

Calculating Actual Step Width Between Each Row
Step width here means how much the DMM voltage changes per ADC code change between consecutive measurements:

Step 1→2:

ΔDMM = 0.010205825 − 0.000525155 = 0.009680670 V

ΔCode = 18 − 0 = 18 codes

Step = 9.680670 mV / 18 = 0.53782 mV/code 

Step 2→3:

ΔDMM = 0.050181781 − 0.010205825 = 0.039975956 V

ΔCode = 108 − 18 = 90 codes

Step = 39.975956 mV / 90 = 0.44418 mV/code

Step 3→4: ΔDMM = 0.100581354 − 0.050181781 = 0.050399573 V

ΔCode = 223 − 108 = 115 codes

Step = 50.399573 mV / 115 = 0.43826 mV/code

Step 4→5: ΔDMM = 0.150132455 − 0.100581354 = 0.049551101 V

ΔCode = 336 − 223 = 113 codes

Step = 49.551101 mV / 113 = 0.43851 mV/code

The remaining segments are calculated using the same method. The full results are summarised in the table below.

Summary:

Segment	ΔDMM (mV)	ΔCode	Actual Step (mV/code)	DNL (LSB)
0→18	9.6807	18	0.53782	+0.224
18→108	39.9760	90	0.44418	+0.011
108→223	50.3996	115	0.43826	−0.003
223→336	49.5511	113	0.43851	−0.002
336→451	50.4071	115	0.43832	−0.003
451→565	50.3173	114	0.44138	+0.004
565→679	50.1323	114	0.43976	+0.001
679→777	43.1145	98	0.43994	+0.001
777→792	6.3307	15	0.42204	−0.040
792→906	49.9454	114	0.43812	−0.003
906→1020	50.3500	114	0.44167	+0.005
1020→1133	49.5046	113	0.43809	−0.003

Pick the maximum absolute DNL = 0.224 LSB

4. INL Error - INL measures the deviation of the actual ADC transfer function from a straight line — but not just any straight line. It must be measured against the best-fit line through the actual data, not the ideal transfer line from the origin.

Why not the ideal line?

The ideal line assumes the ADC has zero offset and zero gain error. But offset and gain error already exist as separate terms in the TUE formula. If the ideal line is used as the INL reference, the offset and gain contributions get absorbed into the INL value, inflating it and double-counting errors that are already accounted for elsewhere.

The best-fit line naturally accounts for both offset and gain by fitting itself to the actual data. What remains after subtracting the best-fit line from each actual code is the pure nonlinearity only — which is exactly what INL is supposed to measure.

Why exclude the zero point?

The zero input point (DMM = 0.000525155V, Code = 0) is already fully captured by the offset error term ADC_Oe. Including it in the best-fit line calculation pulls the line away from the true trend of the remaining data, inflating INL artificially. The zero point belongs to ADC_Oe, not INL.

How the best-fit line is calculated (least squares method):

The least squares method finds the straight line that minimises the total squared deviation from all 12 non-zero data points. This gives the most statistically representative line through the actual ADC transfer function.

Given n = 12 non-zero points:

Slope (m) = (n × Σ(xy) − Σx × Σy) / (n × Σ(x²) − (Σx)²) = 2276.1335 codes/V

Intercept (b) = (Σy − m × Σx) / n = −5.7782 LSB

Best-fit code(n) = (2276.1335 × DMM voltage) − 5.7782

Calculate INL at each point:

      INL(n) = Actual code(n) − Best-fit code(n)

Step 1: - Calculate the best-fit code at this point

Best-fit code = (Slope × DMM voltage) + Intercept

              = (2276.1335 × 0.010205825) + (−5.7782)

              = 23.2290 + (−5.7782)

              = 17.4508

              ≈ 17.45

Step 2: Calculate INL 

INL = Actual code − Best-fit code

    = 18 − 17.45

    = +0.55 LSB

The remaining segments are calculated using the same method. The full results are summarised in the table below.

DMM Voltage (V)	Actual Code	Best-fit Code	INL (LSB)
0.010205825	18	17.45	+0.55
0.050181780	108	108.44	-0.44
0.100581354	223	223.16	-0.16
0.150132455	336	335.94	+0.06
0.200539569	451	450.68	+0.32
0.250856857	565	565.21	-0.21
0.300989204	679	679.31	-0.31
0.344103698	777	777.45	-0.45
0.350434372	792	791.86	+0.14
0.400379788	906	905.54	+0.46
0.450729768	1020	1020.14	-0.14
0.500234415	1133	1132.82	+0.18

The INL values fluctuating both above and below zero confirms this is correct — true nonlinearity oscillates randomly around the best-fit line. A consistent drift in one direction would indicate offset or gain error is still embedded in the calculation.

Take the maximum absolute INL = 0.55 LSB

TUE Calculation:

TUE = sqrt(ADC_Oe² + ADC_Ge² + ADC_INLe² + ADC_DNLe²)

= sqrt(1.2^2 + 14.77^2 + 0.55^2 + 0.224^2)

TUE = 14.83 LSB

TUE (V) = 14.83 x 0.43946mV = 6.517 mV

%FS_ADC = TUE/2^N * 100 = 14.83/4096*100

%FS_ADC = 0.362 %FS

Individual %FS and Error Budget

Error Source	LSB	%FS_ADC	Status
Offset Error (ADC_Oe)	1.20	0.029%	✓ Within ±2% FS_ADC spec
Gain Error (ADC_Ge)	14.77	0.361%	✓ Within ±0.5% FS_ADC spec
INL Error (ADC_INLe)	0.55	0.013%	✓ Within ±2.21 LSB spec
DNL Error (ADC_DNLe)	0.224	0.005%	✓ Within ±1 LSB spec
TUE (RSS)	14.83	0.362%	✓ Within 2.5% spec

Conclusion

In summary, accurate evaluation of ADC performance depends on both proper calibration sequencing and correct interpretation of accuracy specifications, particularly the use of %FS_ADC as the primary accuracy metric. The ADC offset error specified as ±2%FS_ADC is defined relative to the full-scale ADC range, not in absolute LSB terms. This distinction is critical, as offsets measured at low input voltages can appear large in LSBs while still remaining fully compliant when evaluated against the full-scale range. Misinterpreting this specification as a fixed ±2 LSB limit can therefore lead to incorrect conclusions about ADC accuracy and device compliance.

Furthermore, ADC offset error should never be evaluated in isolation. The appropriate method for assessing ADC accuracy is through Total Unadjusted Error (TUE), which combines offset, gain, INL, and DNL errors using a Root-Sum-Square (RSS) approach and expresses the result as %FS_ADC. When calibration is performed correctly and each individual error — offset (±2%FS), gain (±0.5%FS), INL, and DNL (as specified in the device datasheet) — remains within its respective datasheet specification, the combined TUE confirms the ADC is operating as intended and within datasheet guarantees. As demonstrated in Worked Example 2, a real-world 12-bit ADC measurement yielded a combined TUE of 0.362%FS — well within the acceptable 0.5%FS limit — confirming that the device performs as intended despite individual measurements showing deviations of up to 6 LSB. Applying %FS_ADC consistently ensures a realistic, system-level understanding of ADC performance and avoids over-constraining designs based on incomplete or misinterpreted metrics.