Field Test

The gates prove the code. The field test proves the instrument. Every gate in plesty check runs without hardware attached, so a module can be fully compliant and still be unable to reach the device it was written for.

A field test drives the module against the instrument itself and writes a report that says what was verified and how it performed. It runs after the module passes its gates, and before the version that ships is tagged.

Two tiers, and why both

No experiment reaches an instrument the way the host tier does. Every one of them goes through the client tier's path, and defects live in the layers between — ZMQ sockets losing messages across threads, client resends collapsing a serialised server. Those need real network latency and do not reproduce over loopback, which is why the client tier belongs on a second machine.

A green host tier does not mean a working module.

Configuring the test

uv run plesty init field-test

An interview about the instrument: which operations are safe to call unattended, which settings survive being written back, what is slow enough to be worth interrupting. It generates tests/field_test.py and tests/field_test_client.py from the answers, each decision carried into the file with the reason it was given.

Answer for the instrument, not for the code, and pick what your experiments actually call. A parameter no experiment writes proves nothing by round-tripping cleanly.

The generated files are rewritten from the answers, so change an answer and re-run the interview — never edit the generated test.

Running it

Name the address. With no address given, the test falls back to the simulator, passes, and publishes a report that looks exactly like a hardware run — same model, same tier, same gate list. Check the log names a real port, and that the run took a physically plausible time.

On the instrument machine:

uv run python tests/field_test.py --address COM4     # or /dev/ttyUSB0, or a serial number

Then start the device server there and drive the client tier from another machine:

uv run python tests/field_test_client.py --address tcp://<instrument-host>:5551

Running it through a bench

Where the instrument is managed by plesty-bench, both tiers run as jobs and neither needs a person at the machine:

plesty-bench -b <host:port> ping                  # agent version, bench home
plesty-bench -b <host:port> list                  # declared devices, versions, state
plesty-bench -b <host:port> field-test <name> --tier host --address <address> --no-wait
plesty-bench -b <host:port> jobs log <job-id>     # follow it

A real host tier runs for minutes while the client's reply timeout is a minute, so --no-wait and poll rather than waiting on the call. The host tier stops a running device server first, because it needs the port itself, and relaunches it when it finishes.

The client tier reads the host tier's report to learn which operations and parameters actually reached this instrument, so fetch that report before running it:

plesty-bench -b <host:port> fetch "repos/<package>/<ref>/reports/<MODEL>/field-test.json" \
  -o reports/<MODEL>/field-test.json

The reports

A run writes three files and commits one of them:

• reports/<MODEL>/field-test.json — the sanitised projection. What was verified, how it performed, what it found wrong. This is the one you commit, and the hub reads it.
• reports/<MODEL>/field-test-client.json — the same for the client tier. Also committed.
• the markdown and JSONL beside them — the full record, carrying the host name, the instrument address and its serial number. Git-ignored on purpose: module repositories are world-readable. Set PLESTY_REPORT_ARCHIVE to copy them somewhere colleagues can read them.

Check the JSON for a serial or a host name before committing it.

Skipped gates are findings

A skip is the test saying the configuration does not fit the instrument, and each has a different answer:

• "completed within 1 ms, so nothing was abandoned" — the operation named as the slow one is not slow, so the gates that interrupt it have nothing to interrupt. Watch for a module that skips redundant work: a move to a position the stage already holds is slow the first time and instant on every repeat.
• "no parameter reported usable hardware limits" — the instrument answers its own minimum and maximum with an empty range. That is the instrument, not the configuration.
• "the server runs on another machine" — expected in the attached client mode; comparing threading modes would mean restarting the server.

What the run finds becomes an issue on the module, labelled field-test, and is fixed before the next release. That loop is the point: a field test that surfaces nothing anybody acts on has only cost the bench an afternoon.

What a green run still does not cover

Operations the interview excluded are not exercised — homing, anything that leaves the instrument changed, anything needing an argument the test would have to invent. A field test reports on the operations it was told were safe to call unattended, and says nothing about the rest.