Parsing Serial Data on the Fly: From Raw Sensor Rows to Engineering Units

Raw serial output is easy to print and hard to trust later. A line like A0=621 temp=24.7 looks readable during debugging, but it becomes awkward when you need a clean CSV, live chart, or repeatable calibration workflow.

The better pattern is to separate three jobs:

1. Firmware collects raw measurements and prints structured rows.
2. The workstation parses rows into named columns.
3. Calibration math converts raw values into engineering units while preserving the original data.

That keeps firmware simple and makes bench data easier to audit.

Start with a Parseable Serial Contract

Treat the serial line as a tiny data contract. Every row should have the same delimiter, field order, and meaning.

Good:

time_ms,raw_adc,pressure_counts,temp_counts
1000,621,18442,2031
2000,622,18445,2033
3000,623,18448,2034

Risky:

Time 1000 ms: raw=621 pressure is 18442 and temp is 2031

The risky version is nice for a human watching a terminal. The structured version is better for a logger, parser, graph, or spreadsheet.

Choose a Delimiter Deliberately

Delimiter	Good for	Watch out for
Comma	CSV and Excel workflows	Do not include commas inside labels.
Tab	Human-readable columns	Some terminal tools hide tabs poorly.
Semicolon	Locales where comma is decimal separator	Must configure import tools correctly.
JSON lines	Complex nested data	Heavier firmware and parser requirements.

For most Arduino and ESP32 sensor logging, comma-separated rows are the simplest choice.

Keep Raw Data in the Stream

It is tempting to print only final values:

Serial.println(tempC);

That is easy until you discover the calibration was wrong. If the raw ADC count or resistance was never logged, the run may be impossible to repair.

Prefer:

Serial.print(millis());
Serial.print(",");
Serial.print(rawAdc);
Serial.print(",");
Serial.println(resistanceOhm, 2);

Then calculate temperature on the workstation:

time_ms,raw_adc,resistance_ohm,temp_c
1000,621,15443.21,17.82
2000,622,15478.18,17.76
3000,623,15513.25,17.70

This gives you both the measurement and the interpretation.

Why Not Put All Calibration in Firmware?

Firmware calibration can be the right answer for production devices. For bench testing and sensor characterization, it creates friction.

Firmware calibration issue	Bench impact
Recompile for every coefficient change	Slows down iteration.
Floating-point math on small MCUs	Can affect loop timing.
Only final values are logged	Bad math can ruin a run.
Formula changes are buried in code history	Harder for teammates to reproduce.

A practical compromise is to log raw values and apply calibration live on the computer.

Live Parsing Workflow in DaqSense

DaqSense is designed around this workflow:

1. Connect to the serial port.
2. Set the delimiter, such as comma.
3. Watch incoming rows split into columns.
4. Name the columns.
5. Add derived or calibrated columns.
6. Record raw and calculated values to CSV.

Example input:

1000,621,15443.21

Parser columns:

Column	Meaning
time_ms	Device timestamp
raw_adc	ADC count from the microcontroller
resistance_ohm	Thermistor resistance

Derived output:

Column	Formula type
temp_c	Steinhart-Hart
voltage_v	Linear scale from ADC
pressure_psi	Sensor slope and offset

Handle Malformed Lines Explicitly

Real serial streams are messy. Boot messages, debug prints, and partial lines happen.

Common examples:

ets Jul 29 2019 12:21:46
1000,621,15443.21
debug: heater on
2000,622,15478.18

A good parser should make malformed rows visible and avoid mixing them into numeric columns. For firmware, keep debug messages separate when possible. For logging, preserve the raw stream so you can explain any skipped rows later.

A Simple Firmware Pattern

void setup() {
  Serial.begin(115200);
  Serial.println("time_ms,raw_adc,resistance_ohm");
}

void loop() {
  int rawAdc = analogRead(A0);
  float resistanceOhm = 10000.0 * rawAdc / (1023.0 - rawAdc);

  Serial.print(millis());
  Serial.print(",");
  Serial.print(rawAdc);
  Serial.print(",");
  Serial.println(resistanceOhm, 2);

  delay(1000);
}

That firmware is intentionally boring. Boring serial output is easier to parse, calibrate, record, and defend when someone asks where a number came from.