Recording class#

class pyneon.Recording(recording_dir: str | Path)#

Data from a single recording. The recording directory could be downloaded from either a single recording or a project on Pupil Cloud. In either case, the directory must contain an info.json file. For example, a recording directory could have the following structure:

recording_dir/
├── info.json (REQUIRED)
├── gaze.csv
├── 3d_eye_states.csv
├── imu.csv
├── blinks.csv
├── fixations.csv
├── saccades.csv
├── events.csv
├── labels.csv
├── world_timestamps.csv
├── scene_camera.json
├── <scene_video>.mp4
└── derivatives/ (PyNeon-generated derivatives)
    └── ...

Streams, events, (and scene video) will be located but not loaded until accessed as properties such as gaze, imu, and eye_states.

Parameters:: recording_dir (str or pathlib.Path) – Path to the directory containing the recording.

recording_id#

Recording ID.

Type:: str

recording_dir#

Path to the recording directory.

Type:: pathlib.Path

info#

Information about the recording. Read from info.json. For details, see https://docs.pupil-labs.com/neon/data-collection/data-format/#info-json.

Type:: dict

start_time#

Start time (in ns) of the recording as in info.json. May not match the start time of each data stream.

Type:: int

start_datetime#

Start time (datetime) of the recording as in info.json. May not match the start time of each data stream.

Type:: datetime.datetime

contents#

Contents of the recording directory. Each index is a stream or event name (e.g. gaze or imu) and columns are exist (bool), filename (str), and path (Path).

Type:: pandas.DataFrame

property gaze: Stream | None#: Returns a (cached) pyneon.Stream object containing gaze data or None if no gaze.csv is present.

property imu: Stream | None#: Returns a (cached) pyneon.Stream object containing IMU data or None if no imu.csv is present.

property eye_states: Stream | None#: Returns a (cached) pyneon.Stream object containing eye states data or None if no 3d_eye_states.csv is present.

property blinks: Events | None#: Returns a (cached) pyneon.Events object containing blinks data or None if no blinks.csv is present.

property fixations: Events | None#: Returns a (cached) pyneon.Events object containing fixations data or None if no fixations.csv is present.

property saccades: Events | None#: Returns a (cached) pyneon.Events object containing saccades data or None if no saccades.csv is present.

property events: Events | None#: Returns a (cached) pyneon.Events object containing events data or None if no events.csv is present.

property video: SceneVideo | None#: Returns a (cached) pyneon.SceneVideo object containing scene video data or None if no scene_video.mp4 is present.

concat_streams(stream_names: str | list[str], sampling_freq: Number | str = 'min', interp_float_kind: str = 'linear', interp_other_kind: str = 'nearest', inplace: bool = False) → Stream#

Concatenate data from different streams under common timestamps. Since the streams may have different timestamps and sampling frequencies, resampling of all streams to a set of common timestamps is performed. The latest start timestamp and earliest last timestamp of the selected sreams are used to define the common timestamps.

Parameters:

stream_names (str or list of str) – Stream names to concatenate. If “all”, then all streams will be used. If a list, items must be in {"gaze", "imu", "eye_states"} ("3d_eye_states") is also tolerated as an alias for "eye_states").
sampling_freq (float or int or str, optional) – Sampling frequency of the concatenated streams. If numeric, the streams will be interpolated to this frequency. If "min" (default), the lowest nominal sampling frequency of the selected streams will be used. If "max", the highest nominal sampling frequency will be used.
interp_float_kind (str, optional) – Kind of interpolation applied on columns of float type, Defaults to "linear". For details see scipy.interpolate.interp1d.
interp_other_kind (str, optional) – Kind of interpolation applied on columns of other types. Defaults to "nearest".
inplace (bool, optional) – Replace selected stream data with interpolated data during concatenation if True. Defaults to False.

Returns:

Concatenated data.

Return type:

pandas.DataFrame

concat_events(event_names: str | list[str]) → DataFrame#

Concatenate different events. All columns in the selected event type will be present in the final DataFrame. An additional "type" column denotes the event type. If events is selected, its "timestamp [ns]" column will be renamed to "start timestamp [ns]", and the "name and "type" columns will be renamed to "message name" and "message type" respectively to provide a more readable output.

Parameters:: event_names (list of str) – List of event names to concatenate. Event names must be in {"blinks", "fixations", "saccades", "events"} (singular forms are tolerated).
Returns:: Concatenated events.
Return type:: pandas.DataFrame

plot_distribution(heatmap_source: Literal['gaze', 'fixations', None] = 'gaze', scatter_source: Literal['gaze', 'fixations', None] = 'fixations', show: bool = True, **kwargs) → tuple[Figure, Axes]#

Plot a heatmap of gaze or fixation data on a matplotlib axis. Users can flexibly choose to generate a smoothed heatmap and/or scatter plot and the source of the data (gaze or fixation).

Parameters:

rec (Recording) – Recording object containing the gaze and video data.
heatmap_source ({'gaze', 'fixations', None}) – Source of the data to plot as a heatmap. If None, no heatmap is plotted. Defaults to ‘gaze’.
scatter_source ({'gaze', 'fixations', None}) – Source of the data to plot as a scatter plot. If None, no scatter plot is plotted. Defaults to ‘fixations’. Gaze data is typically more dense and thus less suitable for scatter plots.
show (bool) – Show the figure if True. Defaults to True.
**kwargs (keyword arguments) –
Additional parameters for the plot, including:
- ’step_size’: Step size for the heatmap grid. Default is 10.
- ’sigma’: Sigma value for Gaussian smoothing. Default is 2.
- ’width_height’: Width and height of the figure in pixels. Default is (1600, 1200).
- ’cmap’: Colormap for the heatmap. Default is ‘inferno’.
- ’ax’: Matplotlib axis to plot on. If None, a new figure and axis are created.

Returns:

fig (matplotlib.figure.Figure) – Figure object containing the plot.
ax (matplotlib.axes.Axes) – Axis object containing the plot.

sync_gaze_to_video(window_size: int | None = None, overwrite: bool = False, output_path: Path | str | None = None) → Stream#

Synchronize gaze data to video frames by applying windowed averaging around timestamps of each video frame.

Parameters:

window_size (int, optional) – Size of the time window in nanoseconds used for averaging. If None, defaults to the median interval between video frame timestamps.
overwrite (bool, optional) – If True, force recomputation even if saved data exists. Default is False.
output_path (str or pathlib.Path, optional) – Path to save the resulting CSV file. Defaults to <der_dir>/gaze_synced.csv.

Returns:

A Stream indexed by “timestamp [ns]”, containing:

’gaze x [px]’: Gaze x-coordinate in pixels
’gaze y [px]’: Gaze y-coordinate in pixels
’frame_idx’: Index of the video frame corresponding to the gaze data

Return type:

Stream

estimate_scanpath(sync_gaze: Stream | None = None, lk_params: dict | None = None, output_path: Path | str | None = None, overwrite: bool = False) → Stream#

Estimate scanpaths by propagating fixations across frames with Lucas-Kanade.

If a cached result exists and overwrite is False, it is loaded from disk.

Parameters:

sync_gaze (Stream, optional) – Gaze data synchronised to video frames. If None, it is created with Recording.sync_gaze_to_video().
lk_params (dict, optional) – Parameters forwarded to the LK optical-flow call.
output_path (str or pathlib.Path, optional) – Where to save the pickle. Defaults to <der_dir>/scanpath.pkl.
overwrite (bool, optional) – Force recomputation even if a pickle exists.

Returns:

Indexed by "timestamp [ns]" with one column "fixations" containing a nested DataFrame for each video frame.

Return type:

Stream

detect_apriltags(tag_family: str = 'tag36h11', overwrite: bool = False, output_path: Path | str | None = None, **kwargs) → Stream#

Detect AprilTags in a video and return their positions per frame.

Runs AprilTag detection on video frames using the pupil_apriltags backend. Uses saved results if available unless overwrite=True.

Parameters:

tag_family (str, optional) – The AprilTag family to detect (e.g., “tag36h11”). Default is “tag36h11”.
overwrite (bool, optional) – If True, reruns detection even if saved results exist.
output_path (str or pathlib.Path, optional) – Path to save the detection JSON file. Defaults to <der_dir>/apriltags.json.
**kwargs (keyword arguments) –
Additional parameters for AprilTag detection, including:
- ’nthreads’: Number of threads to use for detection. Default is 4.
- ’quad_decimate’: Decimation factor for the quad detection. Default is 1.0, thus no decimation.
- ’skip_frames’: Number of frames to skip between detections. Default is 1, thus no skipping.

Returns:

Stream indexed by “timestamp [ns]” with one row per detected tag, including:

’frame_idx’: Index in the downsampled video sequence
’orig_frame_idx’: Original frame index in the full video
’tag_id’: ID of the detected AprilTag
’corners’: A 4x2 array of tag corner coordinates
’center’: A 1x2 array of the tag center

Return type:

Stream

find_homographies(tag_info: DataFrame, all_detections: Stream | None = None, overwrite: bool = False, output_path: Path | str | None = None, **kwargs) → Stream#

Compute and return homographies for each frame using AprilTag detections and reference marker layout.

Parameters:

tag_info (pandas.DataFrame) –
DataFrame containing AprilTag reference positions and orientations, with columns:
- ’tag_id’: ID of the tag
- ’x’, ‘y’, ‘z’: 3D coordinates of the tag’s center
- ’normal_x’, ‘normal_y’, ‘normal_z’: Normal vector of the tag surface
- ’size’: Side length of the tag in meters
all_detections (Stream, optional) – Stream containing AprilTag detection results per frame. If None, detections are recomputed.
overwrite (bool, optional) – Whether to force recomputation even if saved homographies exist.
output_path (str or pathlib.Path, optional) – Optional file path for saving the homographies as JSON. If None, defaults to <der_dir>/homographies.json.
**kwargs (keyword arguments) –
Additional parameters for homography computation, including:
- ’coordinate_system’: Coordinate system for the homography (‘opencv’ or ‘psychopy’). Default is ‘opencv’.
- ’surface_size’: Size of the surface in pixels (width, height). Default is (1920, 1080).
- ’skip_frames’: Number of frames to skip between detections. Default is 1.
- ’settings’: Additional settings for the homography computation.

Returns:

A Stream object indexed by “timestamp [ns]” containing:

’frame_idx’: Video frame index
’homography’: 3x3 NumPy array representing the homography matrix for that frame

Return type:

Stream

Project gaze coordinates from eye space to surface space using homographies.

Computes or loads frame-wise homographies and applies them to the synchronized gaze data to transform it into surface coordinates. If a saved version exists and overwrite is False, the data is loaded from disk.

Parameters:

homographies (Stream, optional) – Stream containing precomputed homographies. If None, they are computed from tag_info.
tag_info (pandas.DataFrame, optional) – AprilTag marker info used to compute homographies. Required if homographies is None.
synced_gaze (Stream, optional) – Gaze data aligned to video frames. If None, will be computed using sync_gaze_to_video().
overwrite (bool, optional) – If True, recompute and overwrite any existing surface-transformed gaze data. Default is False.
output_path (str or pathlib.Path, optional) – File path to save the resulting CSV. Defaults to <der_dir>/gaze_on_surface.csv.

Returns:

A Stream containing gaze data with surface coordinates, including:

’frame_idx’: Frame index
’x_trans’, ‘y_trans’: Gaze coordinates in surface pixel space
Any additional columns from the synchronized gaze input

Return type:

Stream

fixations_on_surface(gaze_on_surface: Stream | None = None, overwrite: bool = False, output_path: Path | str | None = None) → Events#

Project fixation events into surface space by summarizing gaze samples.

This function maps each fixation to surface coordinates by averaging the surface-transformed gaze points (x_trans, y_trans) associated with that fixation. If saved data exists and overwrite is False, it is loaded from disk instead of being recomputed.

Parameters:

gaze_on_surface (pandas.DataFrame, optional) – DataFrame of gaze coordinates already transformed to surface space. If None, will be computed via self.gaze_on_surface(). Must include ‘fixation id’, ‘x_trans’, and ‘y_trans’ columns.
overwrite (bool, optional) – If True, forces recomputation and overwrites any existing output file. Default is False.
output_path (str or pathlib.Path, optional) – Custom path to save the resulting fixation data as a CSV. If None, defaults to self.der_dir / “fixations_on_surface.csv”.

Returns:

An events object containing:

All columns from the raw fixations table
’gaze x [surface coord]’float
Mean surface-space x-coordinate for the fixation.
’gaze y [surface coord]’float
Mean surface-space y-coordinate for the fixation.

Return type:

Events

estimate_camera_pose(tag_locations_df: DataFrame, all_detections: Stream | None = None, output_path: Path | str | None = None, overwrite: bool = False) → Stream#

Compute the camera pose (R|t) for every frame.

Parameters:

tag_locations_df (pandas.DataFrame) – 3-D positions, normals and size for every AprilTag (columns: ‘tag_id’,’x’,’y’,’z’,’normal_x’,’normal_y’,’normal_z’,’size’).
all_detections (Stream, optional) – Per-frame AprilTag detections. If None, they are produced by Recording.detect_apriltags().
output_path (str or pathlib.Path, optional) – Path to save the resulting camera pose data as a JSON file. Defaults to <der_dir>/camera_pose.json.
overwrite (bool, optional) – If True, forces recomputation and overwrites any existing saved result. Default is False.

Returns:

Indexed by "timestamp [ns]" with columns 'frame_idx', 'translation_vector', 'rotation_vector', 'camera_pos'.

Return type:

Stream

smooth_camera_pose(camera_pose_raw: DataFrame | None = None, overwrite: bool = False, output_path: Path | str | None = None, **kwargs) → DataFrame#

Kalman-smooth camera positions and gate outliers.

Parameters:

camera_pose_raw (pandas.DataFrame, optional) – Raw camera-pose table with columns 'frame_idx' and 'camera_pos'. If None, tries to load camera_pose.json from self.der_dir or computes it via Recording.estimate_camera_pose().
overwrite (bool, default False) – Recompute even if a smoothed file already exists.
output_path (str or pathlib.Path, optional) – Where to save the JSON (smoothed_camera_pose.json by default).
kwargs (dict, optional) –
Optional arguments:
- initial_state_noise: float (default=0.1)
- process_noise: float (default=0.1)
- measurement_noise: float (default=0.01)
- gating_threshold: float (default=2.0)
- bidirectional: bool (default=False)

Returns:

Same rows as camera_pose_raw with the extra column 'smoothed_camera_pos' (3-vector).

Return type:

pd.DataFrame

overlay_scanpath(scanpath: Stream | None = None, show_video: bool = False, video_output_path: Path | str | None = None, overwrite: bool = False, **kwargs) → None#

Render a video with the scan-path overlaid.

Parameters:

scanpath (Stream, optional) – Nested scan-path table (as from Recording.estimate_scanpath()). If None, it is loaded or computed automatically.
show_video (bool) – Display the video live while rendering.
video_output_path (str or pathlib.Path, optional) – Target MP4 path. Defaults to <der_dir>/scanpath.mp4. If None, no file is written.
overwrite (bool, default False) – Regenerate the overlay even if the MP4 already exists.
kwargs (dict, optional) –
Optional arguments:
- circle_radius: int, default 10
  Radius of the fixation circles.
- line_thickness: int, default 2
  Thickness of the fixation lines.
- text_size: int, default 1
  Size of the fixation text.
- max_fixations: int, default 10
  Maximum number of fixations to display.

overlay_detections_and_pose(april_detections: DataFrame, camera_positions: DataFrame, room_corners: ndarray = array([[0, 0], [0, 1], [1, 1], [1, 0]]), video_output_path: Path | str | None = None, graph_size: ndarray = array([300, 300]), show_video: bool = True)#

Overlay AprilTag detections and camera poses on the video frames. The resulting video can be displayed and/or saved.

Parameters:

april_detections (pandas.DataFrame) – DataFrame containing the AprilTag detections.
camera_positions (pandas.DataFrame) – DataFrame containing the camera positions.
room_corners (numpy.ndarray) – Array containing the room corners coordinates. Defaults to a unit square.
str (video_output_path pathlib.Path or) – Path to save the video with detections and poses overlaid. Defaults to ‘detection_and_pose.mp4’.
graph_size (numpy.ndarray) – Size of the graph to overlay on the video. Defaults to [300, 300].
show_video (bool) – Whether to display the video with detections and poses overlaid. Defaults to True.

clear_der_dir(include: str | list[str] = ['all'], exclude: str | list[str] = [])#

Clear the derived data directory by removing files and folders.

Parameters:

include (str or list of str, optional) – Files or folders to delete. If [“all”], delete everything in the directory. Supports full names or base names without extensions (e.g. “scanpath” matches “scanpath.pkl”).
exclude (str or list of str, optional) – Files or folders to exclude. Applies only if include is [“all”]. Also supports base names.

export_motion_bids(motion_dir: str | Path, prefix: str | None = None, extra_metadata: dict = {})#

Export IMU data to Motion-BIDS format. Continuous samples are saved to a .tsv file and metadata (with template fields) are saved to a .json file. Users should later edit the metadata file according to the experiment to make it BIDS-compliant.

Parameters:

motion_dir (str or pathlib.Path) – Output directory to save the Motion-BIDS formatted data.
prefix (str, optional) – Prefix for the BIDS filenames, by default “sub-wearer_name``_task-XXX_tracksys-NeonIMU". The format should be "sub-<label>[_ses-<label>]_task-<label>_tracksys-<label>[_acq-<label>][_run-<index>]" (Fields in [] are optional). Files will be saved as ``{prefix}_motion.<tsv|json>.
extra_metadata (dict, optional) – Extra metadata to include in the .json file. Keys must be valid BIDS fields.

Notes

Motion-BIDS is an extension to the Brain Imaging Data Structure (BIDS) to standardize the organization of motion data for reproducible research [1]. For more information, see https://bids-specification.readthedocs.io/en/stable/modality-specific-files/motion.html.

References

export_eye_bids(output_dir: str | Path, prefix: str = '', extra_metadata: dict = {})#

Export eye-tracking data to Eye-tracking-BIDS format. Continuous samples and events are saved to .tsv.gz files with accompanying .json metadata files. Users should later edit the metadata files according to the experiment.

Parameters:

output_dir (str or pathlib.Path) – Output directory to save the Eye-tracking-BIDS formatted data.
prefix (str, optional) – Prefix for the BIDS filenames, by default “sub-XX_recording-eye”. The format should be <matches>[_recording-<label>]_<physio|physioevents>.<tsv.gz|json> (Fields in [] are optional). Files will be saved as {prefix}_physio.<tsv.gz|json> and {prefix}_physioevents.<tsv.gz|json>.

Notes

Eye-tracking-BIDS is an extension to the Brain Imaging Data Structure (BIDS) to standardize the organization of eye-tracking data for reproducible research. The extension is still being finialized. This method follows the latest standards outlined in bids-standard/bids-specification#1128.

Recording class#

This Page