ReFlex Library: How to integrate an alternative calibration procedure

Table of Contents

1. General approach
2. Calibration in the processing pipeline
3. Step 1: Define the data contract
4. Step 2: Collect corresponding points
5. Step 3: Submit and activate the result
6. Step 4: Verify the integration
7. Using a different transformation model
8. Further reading

General approach

An alternative calibration procedure can change how reference positions are measured without changing touch detection. For example, a guided measurement tool can collect several observations per displayed marker and return one representative sensor position for each marker. This recipe describes the integration contract, leaving the measurement algorithm to your application.

The example retains the existing three-point affine mapping and display frame. It adds a batch submission endpoint so that a complete set of measurements is validated before replacing the previous points. The snippets below are proposed additions, not APIs already available in ReFlex. No new sensor or interaction observer is needed.

Unlike interaction detection, the current Calibrator has no interchangeable strategy interface: its matrix generator is private and its methods are not virtual. Reuse it when your procedure can provide its expected point pairs. See Using a different transformation model when it cannot.

⬆ back to top

Calibration in the processing pipeline

The relevant path is: sensor → filtered point cloud → interaction observer → lateral calibration → client output. The observer produces X/Y as point-cloud array indices and already normalises deformation depth Z. CalibrationManager.Calibrate() maps lateral coordinates into the display frame, normalises them by the frame width and height, and clamps them to [0, 1]. The built-in mapping preserves Z through a scale factor of 1.

TimerLoop applies calibration before NetworkManager.Broadcast(). The TrackingServer’s /ReFlex WebSocket also calls the calibration manager. In contrast, the processing SignalR stream used by ProcessingService.getInteractions() exposes raw lateral positions. The calibration UI uses these as input and requests a calibrated preview through CalibrationService.computeCalibratedAbsolutePosition(). Its raw, normalized and absolute arrays represent sensor indices, normalised output and display pixels respectively. Only raw is suitable for measuring calibration points.

Responsibilities are split between:

Calculation and activation are separate: ComputeTransformation() raises CalibrationFinished, but the manager’s event handler only publishes calibration data. CalibrationService.FinishCalibration() assigns the result to the manager, saves Settings.CalibrationValues through ConfigurationManager.Update() and reinitialises the manager. The existing HTTP ApplyCalibration action only computes; SaveCalibration calls FinishCalibration().

⬆ back to top

Step 1: Define the data contract

Despite their names, SourceValues are display positions and TargetValues are measured sensor positions. The computed matrix maps target to source. Correspondence is determined by array index, not by touch ID.

Do not send measured world-space X/Y, normalised output or Z as point coordinates. If your procedure aggregates floating-point samples, explicitly round to integer sensor indices for this contract. Preserve the existing threshold values; depth detection is configured separately.

The stored configuration contains point data and the frame, rather than an arbitrary matrix. On loading, Calibrator.Load() keeps its existing source list and recomputes the matrix using the loaded targets. Consequently, this example deliberately uses the generated markers. Supplying custom source points also requires changing source generation and loading so those points survive frame changes and restarts.

⬆ back to top

Step 2: Collect corresponding points

In a new component under tools/ReFlex.TrackingServer/ClientApp/src/app/calibration/, or as an alternative mode in CalibrationComponent:

1. Set the frame with CalibrationService.updateFrameSize() and wait for completion. Then read getCalibrationSourcePoints().
2. Render those three markers in their returned order. Collect observations from ProcessingService.getInteractions() or another measurement source that returns the same sensor-index coordinates.
3. Produce one CalibrationPoint for each marker. Keep all three locally until measurement and validation are complete. Disable frame edits and other calibration submissions during this session; discard measurements if the frame or sensor configuration changes.

For example, after your procedure has selected one representative raw interaction per marker:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import { CalibrationPoint, Interaction } from '@reflex/shared-types';

// Array order must match the markers returned by getCalibrationSourcePoints().
function toCalibrationPoints(samples: Interaction[]): CalibrationPoint[] {
  if (samples.length !== 3 || samples.some(sample =>
    !Number.isFinite(sample.position.x) ||
    !Number.isFinite(sample.position.y))) {
    throw new Error('Three finite sensor positions are required.');
  }

  return samples.map(sample => ({
    positionX: Math.round(sample.position.x),
    positionY: Math.round(sample.position.y),
    touchId: sample.touchId
  }));
}

The existing updateCalibrationPoint(index, point) can update individual measurements, but may calculate using a mixture of old and new points and throttles recalculation for 500 ms per point. The batch path below replaces the full set before computing. It does not use Restart or the staged AddCalibrationPoint API.

⬆ back to top

Step 3: Submit and activate the result

Add this method to the server’s tools/ReFlex.TrackingServer/Model/CalibrationService.cs. Its existing imports include CalibrationPoint; the project also provides the standard System and LINQ imports. Validation uses the current camera resolution and checks both triangles before modifying stored lists.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
public void ApplyAlternativeCalibration(CalibrationPoint[] points)
{
    var camera = _configurationManager.Settings.CameraConfigurationValues;
    if (Frame.Width <= 0 || Frame.Height <= 0 ||
        points == null || points.Length != 3 ||
        points.Any(p => p == null ||
            p.PositionX < 0 || p.PositionX >= camera.Width ||
            p.PositionY < 0 || p.PositionY >= camera.Height))
        throw new ArgumentException("Provide a valid frame and three sensor positions.");

    var targets = points.Select(p => p.ToArray()).ToList();
    var values = _calibrator.CalibrationValues;

    // Cast before subtraction/multiplication to avoid integer overflow.
    static double TwiceArea(List<int[]> p) =>
        ((double)p[1][0] - p[0][0]) * ((double)p[2][1] - p[0][1]) -
        ((double)p[1][1] - p[0][1]) * ((double)p[2][0] - p[0][0]);

    if (values.SourceValues.Count != 3 ||
        Math.Abs(TwiceArea(values.SourceValues)) < 1 ||
        Math.Abs(TwiceArea(targets)) < 1)
        throw new ArgumentException("Reference points must form non-degenerate triangles.");

    // Calibration is a struct, but these lists refer to the calibrator's lists.
    values.TargetValues.Clear();
    values.TargetValues.AddRange(targets);
    values.LastUpdated.Clear();
    values.LastUpdated.AddRange(Enumerable.Repeat(DateTime.Now, 3));

    // Computes the result and raises CalibrationFinished for the service.
    _calibrator.ComputeTransformation();
    // Activates the matrix, persists the point data and reloads the manager.
    FinishCalibration();
}

Add the endpoint to tools/ReFlex.TrackingServer/Controllers/CalibrationController.cs:

1
2
3
4
5
6
7
8
9
10
11
12
13
[HttpPost("Alternative")]
public IActionResult ApplyAlternative([FromBody] CalibrationPoint[] points)
{
    try
    {
        _calibrationService.ApplyAlternativeCalibration(points);
        return Ok();
    }
    catch (ArgumentException error)
    {
        return BadRequest(error.Message);
    }
}

Add the matching method to the existing Angular service in ClientApp/src/shared/services/calibration.service.ts; its existing imports already provide these types:

1
2
3
4
public submitAlternativeCalibration(points: CalibrationPoint[]): Observable<void> {
  return this.http.post<void>(
    `${this.baseUrl}${this.calibrationRoute}Alternative`, points);
}

On confirmation in the alternative UI, call submitAlternativeCalibration(toCalibrationPoints(samples)) and subscribe to the returned observable. On success, refresh getCurrentCalibrationTargetPoints() and the calibrated preview; on error, keep the collected measurements and display the error. Angular HTTP requests only execute when subscribed. No separate applyCalibration() or saveCalibration() call is needed for this new endpoint.

The sample assumes one calibration session at a time. If multiple clients can edit calibration, serialise all calibration/frame writes on the server and include a frame/session revision in the request. For measurement quality, also reject nearly collinear or noisy samples using an application-specific tolerance; the area check above only rejects degenerate integer triangles.

⬆ back to top

Step 4: Verify the integration

• Coordinates: Check marker order and frame offsets. Measurements must remain raw sensor indices; preview positions must lie at the corresponding display markers.
• Validation: Reject missing points, duplicate/collinear points, out-of-range indices and a zero-sized frame before changing calibration data. Verify that an invalid request leaves the previous calibration active.
• Activation: After successful submission, inspect CalibratedInteractions and the normal client output, not just the matrix returned by the UI. Check independent positions between the markers, including corners; clamping can conceal incorrect mappings.
• Persistence: Restart the server and confirm that the frame, target points and resulting output are restored. Recalibrate after changing the sensor mode or display geometry.
• Depth and metadata: Confirm that Z, touch IDs, confidence and interaction type retain their expected meaning.

Current implementation limitation: CalibrationManager.Calibrate() calculates Y using X that has already been transformed, normalised and clamped. A mapping with a non-zero X contribution to Y (for example rotation or shear) therefore does not apply the intended affine transform. Before relying on such mappings, correct that method to calculate both output coordinates from saved original X/Y, then normalise and clamp each result. Include a rotated mapping in verification. This recipe documents the limitation; it does not change the processing implementation.

⬆ back to top

Using a different transformation model

If the alternative procedure returns more than three correspondences, floating-point coefficients, a projective mapping or a nonlinear warp, the three-point storage contract is insufficient. CalibrationManager.CalibrationMatrix is writable, but assigning it alone is not a persistent integration: FinishCalibration(), frame changes and startup can replace it with a matrix recomputed from the built-in points.

For a different model, implement ICalibrationManager and adapt the server’s calibration service and persisted settings to carry the model type, parameters and frame. In Startup.ConfigureServices(), replace the calibManager construction and pass the same instance to both TimerLoop and AddSingleton<ICalibrationManager>(). Define how loading, reset, preview, activation and saving work for the new model; the existing interface still exposes concrete Calibrator initialisation and legacy loading methods, so this requires an explicit adapter or interface refactoring.

The current matrix convention for an affine mapping is:

[ a  b  tx  0 ]   xDisplay = a*xSensor + b*ySensor + tx
[ c  d  ty  0 ]   yDisplay = c*xSensor + d*ySensor + ty
[ 0  0   1  0 ]   zOutput  = zInput
[ 0  0   0  1 ]

Translation occupies column 2 (zero-based), not column 3. These are display-pixel outputs; frame offsets and normalisation belong to the manager. The current Calibrate() reads only specific coefficients and performs no homogeneous division, so a general projective 4×4 matrix cannot simply be substituted. A replacement manager must implement that evaluation itself, return a new interaction preserving metadata, and retain the normalised lateral/deformation-depth contract expected by clients.

⬆ back to top

Further reading

• Processing pipeline and interaction format.
• Server configuration.
• Implementing an Interaction Processor.

⬆ back to top