Recording class#

class pyneon.Recording(recording_dir: str | Path)#

Container of a multi-modal recording with streams, events, and videos.

The recording directory is expected to follow either the Pupil Cloud format (tested with data format version >= 2.3) or the native Pupil Labs format. (tested with data format version >= 2.5). In both cases, the directory must contain an info.json file.

Example Pupil Cloud recording directory structure:

recording_dir/
├── 3d_eye_states.csv
├── blinks.csv
├── events.csv
├── fixations.csv
├── gaze.csv
├── imu.csv
├── saccades.csv
├── info.json (REQUIRED)
├── labels.csv
├── saccades.csv
├── scene_camera.json
├── world_timestamps.csv
└── *.mp4

Example native Pupil Labs recording directory structure:

recording_dir/
├── blinks ps1.raw
├── blinks ps1.time
├── blinks.dtype
├── calibration.bin
├── event.time
├── event.txt
├── ...
├── gaze ps1.raw
├── gaze ps1.time
├── gaze.dtype
├── ...
├── info.json (REQUIRED)
├── Neon Scene Camera v1 ps1.mp4
├── Neon Scene Camera v1 ps1.time
├── Neon Sensor Module v1 ps1.mp4
├── Neon Sensor Module v1 ps1.time
├── ...
├── wearer.json
├── worn ps1.raw
└── worn.dtype

Streams, events, and scene video will be located but not loaded until accessed as properties such as gaze, imu, and eye_states.

Parameters:
recording_dirstr or pathlib.Path

Path to the directory containing the recording.

Attributes:
recording_idstr

Recording ID.

recording_dirpathlib.Path

Path to the recording directory.

format{‘cloud’, ‘native’}

Recording format, either “cloud” for Pupil Cloud format or “native” for native format.

infodict

Information about the recording. Read from info.json. For details, see https://docs.pupil-labs.com/neon/data-collection/data-format/#info-json.

data_format_versionstr | None

Data format version as in info.json.

Methods

clear_der_dir([include, exclude])

Clear the derived data directory by removing files and folders.

concat_events(event_names)

Concatenate different events.

concat_streams(stream_names[, ...])

Concatenate data from different streams under common timestamps.

detect_apriltags([tag_family, overwrite, ...])

Detect AprilTags in a video and return their positions per frame.

estimate_camera_pose(tag_locations_df[, ...])

Compute the camera pose (R|t) for every frame.

estimate_scanpath([sync_gaze, lk_params, ...])

Estimate scanpaths by propagating fixations across frames with Lucas-Kanade.

export_cloud_format(target_dir[, rebase])

Export native data to cloud-like format.

export_eye_bids(output_dir[, prefix, ...])

Export eye-tracking data to Eye-tracking-BIDS format.

export_motion_bids(motion_dir[, prefix, ...])

Export IMU data to Motion-BIDS format.

find_homographies(tag_info[, ...])

Compute and return homographies for each frame using AprilTag detections and reference marker layout.

fixations_on_surface([gaze_on_surface, ...])

Project fixation events into surface space by summarizing gaze samples.

gaze_on_surface([homographies, tag_info, ...])

Project gaze coordinates from eye space to surface space using homographies.

overlay_detections_and_pose(...[, ...])

Overlay AprilTag detections and camera poses on the video frames.

overlay_scanpath([scanpath, show_video, ...])

Render a video with the scan-path overlaid.

plot_distribution([heatmap_source, ...])

Plot a heatmap of gaze or fixation data on a matplotlib axis.

smooth_camera_pose([camera_pose_raw, ...])

Kalman-smooth camera positions and gate outliers.

sync_gaze_to_video([window_size, overwrite, ...])

Synchronize gaze data to video frames by applying windowed averaging around timestamps of each video frame.

Return a cached pyneon.Events object containing blink event data.

For Pupil Cloud recordings, the data is loaded from blinks.csv.

For native recordings, the data is loaded from blinks ps1.raw, along with the corresponding .time and .dtype files.

clear_der_dir(include: str | list[str] = ['all'], exclude: str | list[str] = [])#

Clear the derived data directory by removing files and folders.

Parameters:
includestr or list of str, optional

Files or folders to delete. If [“all”], delete everything in the directory. Supports full names or base names without extensions (e.g. “scanpath” matches “scanpath.pkl”).

excludestr or list of str, optional

Files or folders to exclude. Applies only if include is [“all”]. Also supports base names.

concat_events(event_names: str | list[str]) → Events#

Concatenate different events. All columns in the selected event type will be present in the final DataFrame. An additional “type” column denotes the event type. If events is selected, its “timestamp [ns]” column will be renamed to “start timestamp [ns]”, and the “name” and “type” columns will be renamed to “message name” and “message type” respectively to provide a more readable output.

Parameters:
event_nameslist of str

List of event names to concatenate. Event names must be in {"blinks", "fixations", "saccades", "events"} (singular forms are tolerated).

Returns:
Events

Events object containing concatenated data.

concat_streams(stream_names: str | list[str], sampling_freq: Number | str = 'min', float_kind: str | int = 'linear', other_kind: str | int = 'nearest', inplace: bool = False) → Stream#

Concatenate data from different streams under common timestamps. Since the streams may have different timestamps and sampling frequencies, resampling of all streams to a set of common timestamps is performed. The latest start timestamp and earliest last timestamp of the selected sreams are used to define the common timestamps.

Parameters:
stream_namesstr or list of str

Stream names to concatenate. If “all”, then all streams will be used. If a list, items must be in {"gaze", "imu", "eye_states"} (“3d_eye_states” is also tolerated as an alias for “eye_states”).

sampling_freqfloat or int or str, optional

Sampling frequency of the concatenated streams. If numeric, the streams will be interpolated to this frequency. If “min” (default), the lowest nominal sampling frequency of the selected streams will be used. If “max”, the highest nominal sampling frequency will be used.

float_kindstr, optional

Kind of interpolation applied on columns of float type, Defaults to “linear”. For details see scipy.interpolate.interp1d.

other_kindstr, optional

Kind of interpolation applied on columns of other types, Defaults to “nearest”. Only “nearest”, “previous”, and “next” are recommended.

inplacebool, optional

Replace selected stream data with interpolated data during concatenation if True. Defaults to False.

Returns:
Stream

Stream object containing concatenated data.

detect_apriltags(tag_family: str = 'tag36h11', overwrite: bool = False, output_path: Path | str | None = None, **kwargs) → Stream#

Detect AprilTags in a video and return their positions per frame.

Runs AprilTag detection on video frames using the pupil_apriltags backend. Uses saved results if available unless overwrite=True.

Parameters:
tag_familystr, optional

The AprilTag family to detect (e.g., “tag36h11”). Default is “tag36h11”.

overwritebool, optional

If True, reruns detection even if saved results exist.

output_pathstr or pathlib.Path, optional

Path to save the detection JSON file. Defaults to <der_dir>/apriltags.json.

**kwargskeyword arguments
Additional parameters for AprilTag detection, including:

  • ‘nthreads’: Number of threads to use for detection. Default is 4.

  • ‘quad_decimate’: Decimation factor for the quad detection. Default is 1.0, thus no decimation.

  • ‘skip_frames’: Number of frames to skip between detections. Default is 1, thus no skipping.

Returns:
Stream
Stream indexed by “timestamp [ns]” with one row per detected tag, including:

  • ‘frame_idx’: Index in the downsampled video sequence

  • ‘orig_frame_idx’: Original frame index in the full video

  • ‘tag_id’: ID of the detected AprilTag

  • ‘corners’: A 4x2 array of tag corner coordinates

  • ‘center’: A 1x2 array of the tag center

estimate_camera_pose(tag_locations_df: DataFrame, all_detections: Stream | None = None, output_path: Path | str | None = None, overwrite: bool = False) → Stream#

Compute the camera pose (R|t) for every frame.

Parameters:
tag_locations_dfpandas.DataFrame

3-D positions, normals and size for every AprilTag (columns: ‘tag_id’,’x’,’y’,’z’,’normal_x’,’normal_y’,’normal_z’,’size’).

all_detectionsStream, optional

Per-frame AprilTag detections. If None, they are produced by Recording.detect_apriltags().

output_pathstr or pathlib.Path, optional

Path to save the resulting camera pose data as a JSON file. Defaults to <der_dir>/camera_pose.json.

overwritebool, optional

If True, forces recomputation and overwrites any existing saved result. Default is False.

Returns:
Stream

Indexed by “timestamp [ns]” with columns 'frame_idx', 'translation_vector', 'rotation_vector', 'camera_pos'.

estimate_scanpath(sync_gaze: Stream | None = None, lk_params: dict | None = None, output_path: Path | str | None = None, overwrite: bool = False) → Stream#

Estimate scanpaths by propagating fixations across frames with Lucas-Kanade.

If a cached result exists and overwrite is False, it is loaded from disk.

Parameters:
sync_gazeStream, optional

Gaze data synchronised to video frames. If None, it is created with Recording.sync_gaze_to_video().

lk_paramsdict, optional

Parameters forwarded to the LK optical-flow call.

output_pathstr or pathlib.Path, optional

Where to save the pickle. Defaults to <der_dir>/scanpath.pkl.

overwritebool, optional

Force recomputation even if a pickle exists.

Returns:
Stream

Indexed by “timestamp [ns]” with one column “fixations” containing a nested DataFrame for each video frame.

property events: Events#

Returns a (cached) pyneon.Events object containing events data.

For Pupil Cloud recordings, the events data is loaded from events.csv.

For native recordings, the events data is loaded from event.txt and event.time.

export_cloud_format(target_dir: str | Path, rebase: bool = True)#

Export native data to cloud-like format.

Parameters:
target_dirstr or pathlib.Path

Output directory to save the Cloud-Format structured data.

rebasebool, optional

If True, re-initialize the recording on the target directory after export.

export_eye_bids(output_dir: str | Path, prefix: str = '', extra_metadata: dict = {})#

Export eye-tracking data to Eye-tracking-BIDS format. Continuous samples and events are saved to .tsv.gz files with accompanying .json metadata files. Users should later edit the metadata files according to the experiment.

Parameters:
output_dirstr or pathlib.Path

Output directory to save the Eye-tracking-BIDS formatted data.

prefixstr, optional

Prefix for the BIDS filenames, by default “sub-XX_recording-eye”. The format should be <matches>[_recording-<label>]_<physio|physioevents>.<tsv.gz|json> (Fields in [] are optional). Files will be saved as {prefix}_physio.<tsv.gz|json> and {prefix}_physioevents.<tsv.gz|json>.

Notes

Eye-tracking-BIDS is an extension to the Brain Imaging Data Structure (BIDS) to standardize the organization of eye-tracking data for reproducible research. The extension is still being finalized. This method follows the latest standards outlined in bids-standard/bids-specification#1128.

export_motion_bids(motion_dir: str | Path, prefix: str | None = None, extra_metadata: dict = {})#

Export IMU data to Motion-BIDS format. Continuous samples are saved to a .tsv file and metadata (with template fields) are saved to a .json file. Users should later edit the metadata file according to the experiment to make it BIDS-compliant.

Parameters:
motion_dirstr or pathlib.Path

Output directory to save the Motion-BIDS formatted data.

prefixstr, optional

Prefix for the BIDS filenames, by default “sub-wearer_name``_task-XXX_tracksys-NeonIMU". The format should be "sub-<label>[_ses-<label>]_task-<label>_tracksys-<label>[_acq-<label>][_run-<index>]" (Fields in [] are optional). Files will be saved as ``{prefix}_motion.<tsv|json>.

extra_metadatadict, optional

Extra metadata to include in the .json file. Keys must be valid BIDS fields.

Notes

Motion-BIDS is an extension to the Brain Imaging Data Structure (BIDS) to standardize the organization of motion data for reproducible research [1]. For more information, see https://bids-specification.readthedocs.io/en/stable/modality-specific-files/motion.html.

References

[1]

Jeung, S., Cockx, H., Appelhoff, S., Berg, T., Gramann, K., Grothkopp, S., … & Welzel, J. (2024). Motion-BIDS: an extension to the brain imaging data structure to organize motion data for reproducible research. Scientific Data, 11(1), 716.

property eye_states: Stream#

Returns a (cached) pyneon.Stream object containing eye states data.

For Pupil Cloud recordings, the data is loaded from 3d_eye_states.csv.

For native recordings, the data is loaded from eye_state ps1.raw, along with the corresponding .time and .dtype files.

property eye_video: Video#

Returns a (cached) pyneon.Video object containing eye video data.

Eye video is only available for native recordings and is loaded from the Neon Sensor Module*.mp4 file in the recording directory.

find_homographies(tag_info: DataFrame, all_detections: Stream | None = None, overwrite: bool = False, output_path: Path | str | None = None, **kwargs) → Stream#

Compute and return homographies for each frame using AprilTag detections and reference marker layout.

Parameters:
tag_infopandas.DataFrame
DataFrame containing AprilTag reference positions and orientations, with columns:

  • ‘tag_id’: ID of the tag

  • ‘x’, ‘y’, ‘z’: 3D coordinates of the tag’s center

  • ‘normal_x’, ‘normal_y’, ‘normal_z’: Normal vector of the tag surface

  • ‘size’: Side length of the tag in meters

all_detectionsStream, optional

Stream containing AprilTag detection results per frame. If None, detections are recomputed.

overwritebool, optional

Whether to force recomputation even if saved homographies exist.

output_pathstr or pathlib.Path, optional

Optional file path for saving the homographies as JSON. If None, defaults to <der_dir>/homographies.json.

**kwargskeyword arguments
Additional parameters for homography computation, including:

  • ‘coordinate_system’: Coordinate system for the homography (‘opencv’ or ‘psychopy’). Default is ‘opencv’.

  • ‘surface_size’: Size of the surface in pixels (width, height). Default is (1920, 1080).

  • ‘skip_frames’: Number of frames to skip between detections. Default is 1.

  • ‘settings’: Additional settings for the homography computation.

Returns:
Stream
A Stream object indexed by “timestamp [ns]” containing:

  • ‘frame_idx’: Video frame index

  • ‘homography’: 3x3 NumPy array representing the homography matrix for that frame

property fixations: Events#

Returns a (cached) pyneon.Events object containing fixations data.

For Pupil Cloud recordings, the data is loaded from fixations.csv.

For native recordings, the data is loaded from fixations ps1.raw, along with the corresponding .time and .dtype files.

fixations_on_surface(gaze_on_surface: Stream | None = None, overwrite: bool = False, output_path: Path | str | None = None) → Events#

Project fixation events into surface space by summarizing gaze samples.

This function maps each fixation to surface coordinates by averaging the surface-transformed gaze points (x_trans, y_trans) associated with that fixation. If saved data exists and overwrite is False, it is loaded from disk instead of being recomputed.

Parameters:
gaze_on_surfacepandas.DataFrame, optional

DataFrame of gaze coordinates already transformed to surface space. If None, will be computed via self.gaze_on_surface(). Must include ‘fixation id’, ‘x_trans’, and ‘y_trans’ columns.

overwritebool, optional

If True, forces recomputation and overwrites any existing output file. Default is False.

output_pathstr or pathlib.Path, optional

Custom path to save the resulting fixation data as a CSV. If None, defaults to self.der_dir / “fixations_on_surface.csv”.

Returns:
Events
An events object containing:

  • All columns from the raw fixations table

  • ‘gaze x [surface coord]’float

    Mean surface-space x-coordinate for the fixation.

  • ‘gaze y [surface coord]’float

    Mean surface-space y-coordinate for the fixation.

property gaze: Stream#

Return a cached pyneon.Stream object containing gaze data.

For Pupil Cloud recordings, the data is loaded from gaze.csv.

For native recordings, the data is loaded from gaze_200hz.raw (if present; otherwise from gaze ps1.raw) along with the corresponding .time and .dtype files.

gaze_on_surface(homographies: Stream | None = None, tag_info: DataFrame | None = None, synced_gaze: Stream | None = None, overwrite: bool = False, output_path: Path | str | None = None) → Stream#

Project gaze coordinates from eye space to surface space using homographies.

Computes or loads frame-wise homographies and applies them to the synchronized gaze data to transform it into surface coordinates. If a saved version exists and overwrite is False, the data is loaded from disk.

Parameters:
homographiesStream, optional

Stream containing precomputed homographies. If None, they are computed from tag_info.

tag_infopandas.DataFrame, optional

AprilTag marker info used to compute homographies. Required if homographies is None.

synced_gazeStream, optional

Gaze data aligned to video frames. If None, will be computed using sync_gaze_to_video().

overwritebool, optional

If True, recompute and overwrite any existing surface-transformed gaze data. Default is False.

output_pathstr or pathlib.Path, optional

File path to save the resulting CSV. Defaults to <der_dir>/gaze_on_surface.csv.

Returns:
Stream
A Stream containing gaze data with surface coordinates, including:

  • ‘frame_idx’: Frame index

  • ‘x_trans’, ‘y_trans’: Gaze coordinates in surface pixel space

  • Any additional columns from the synchronized gaze input

property imu: Stream#

Return a cached pyneon.Stream object containing IMU data.

For Pupil Cloud recordings, the data is loaded from imu.csv.

For native recordings, the data is loaded from imu ps1.raw, along with the corresponding .time and .dtype files.

overlay_detections_and_pose(april_detections: DataFrame, camera_positions: DataFrame, room_corners: ndarray = array([[0, 0], [0, 1], [1, 1], [1, 0]]), video_output_path: Path | str | None = None, graph_size: ndarray = array([300, 300]), show_video: bool = True)#

Overlay AprilTag detections and camera poses on the video frames. The resulting video can be displayed and/or saved.

Parameters:
april_detectionspandas.DataFrame

DataFrame containing the AprilTag detections.

camera_positionspandas.DataFrame

DataFrame containing the camera positions.

room_cornersnumpy.ndarray

Array containing the room corners coordinates. Defaults to a unit square.

video_output_path pathlib.Path or str

Path to save the video with detections and poses overlaid. Defaults to ‘detection_and_pose.mp4’.

graph_sizenumpy.ndarray

Size of the graph to overlay on the video. Defaults to [300, 300].

show_videobool

Whether to display the video with detections and poses overlaid. Defaults to True.

overlay_scanpath(scanpath: Stream | None = None, show_video: bool = False, video_output_path: Path | str | None = None, overwrite: bool = False, **kwargs) → None#

Render a video with the scan-path overlaid.

Parameters:
scanpathStream, optional

Nested scan-path table (as from Recording.estimate_scanpath()). If None, it is loaded or computed automatically.

show_videobool

Display the video live while rendering.

video_output_pathstr or pathlib.Path, optional

Target MP4 path. Defaults to <der_dir>/scanpath.mp4. If None, no file is written.

overwritebool, default False

Regenerate the overlay even if the MP4 already exists.

kwargsdict, optional
Optional arguments:
  • circle_radius: int, default 10

    Radius of the fixation circles.

  • line_thickness: int, default 2

    Thickness of the fixation lines.

  • text_size: int, default 1

    Size of the fixation text.

  • max_fixations: int, default 10

    Maximum number of fixations to display.

plot_distribution(heatmap_source: Literal['gaze', 'fixations', None] = 'gaze', scatter_source: Literal['gaze', 'fixations', None] = 'fixations', show: bool = True, **kwargs) → tuple[Figure, Axes]#

Plot a heatmap of gaze or fixation data on a matplotlib axis. Users can flexibly choose to generate a smoothed heatmap and/or scatter plot and the source of the data (gaze or fixation).

Parameters:
recRecording

Recording object containing the gaze and video data.

heatmap_source{‘gaze’, ‘fixations’, None}

Source of the data to plot as a heatmap. If None, no heatmap is plotted. Defaults to ‘gaze’.

scatter_source{‘gaze’, ‘fixations’, None}

Source of the data to plot as a scatter plot. If None, no scatter plot is plotted. Defaults to ‘fixations’. Gaze data is typically more dense and thus less suitable for scatter plots.

showbool

Show the figure if True. Defaults to True.

**kwargskeyword arguments
Additional parameters for the plot, including:

  • ‘step_size’: Step size for the heatmap grid. Default is 10.

  • ‘sigma’: Sigma value for Gaussian smoothing. Default is 2.

  • ‘width_height’: Width and height of the figure in pixels. Default is (1600, 1200).

  • ‘cmap’: Colormap for the heatmap. Default is ‘inferno’.

  • ‘ax’: Matplotlib axis to plot on. If None, a new figure and axis are created.

Returns:
figmatplotlib.figure.Figure

Figure object containing the plot.

axmatplotlib.axes.Axes

Axis object containing the plot.

property saccades: Events#

Returns a (cached) pyneon.Events object containing saccades data.

For Pupil Cloud recordings, the data is loaded from saccades.csv.

For native recordings, the data is loaded from fixations ps1.raw, along with the corresponding .time and .dtype files.

property scene_video: Video#

Returns a (cached) pyneon.Video object containing scene video data.

For Pupil Cloud recordings, the video is loaded from the only *.mp4 file in the recording directory.

For native recordings, the video is loaded from the Neon Scene Camera*.mp4 file in the recording directory.

smooth_camera_pose(camera_pose_raw: DataFrame | None = None, overwrite: bool = False, output_path: Path | str | None = None, **kwargs) → DataFrame#

Kalman-smooth camera positions and gate outliers.

Parameters:
camera_pose_rawpandas.DataFrame, optional

Raw camera-pose table with columns 'frame_idx' and 'camera_pos'. If None, tries to load camera_pose.json from self.der_dir or computes it via Recording.estimate_camera_pose().

overwritebool, default False

Recompute even if a smoothed file already exists.

output_pathstr or pathlib.Path, optional

Where to save the JSON (smoothed_camera_pose.json by default).

kwargsdict, optional
Optional arguments:

  • initial_state_noise: float (default=0.1)

  • process_noise: float (default=0.1)

  • measurement_noise: float (default=0.01)

  • gating_threshold: float (default=2.0)

  • bidirectional: bool (default=False)

Returns:
pd.DataFrame

Same rows as camera_pose_raw with the extra column 'smoothed_camera_pos' (3-vector).

property start_datetime: datetime#

Start time (datetime) of the recording as in info.json. May not match the start time of each data stream.

property start_time: int#

Start time (in ns) of the recording as in info.json. May not match the start time of each data stream.

sync_gaze_to_video(window_size: int | None = None, overwrite: bool = False, output_path: Path | str | None = None) → Stream#

Synchronize gaze data to video frames by applying windowed averaging around timestamps of each video frame.

Parameters:
window_sizeint, optional

Size of the time window in nanoseconds used for averaging. If None, defaults to the median interval between video frame timestamps.

overwritebool, optional

If True, force recomputation even if saved data exists. Default is False.

output_pathstr or pathlib.Path, optional

Path to save the resulting CSV file. Defaults to <der_dir>/gaze_synced.csv.

Returns:
Stream
A Stream indexed by “timestamp [ns]”, containing:

  • ‘gaze x [px]’: Gaze x-coordinate in pixels

  • ‘gaze y [px]’: Gaze y-coordinate in pixels

  • ‘frame_idx’: Index of the video frame corresponding to the gaze data