Scene video class#

class pyneon.video.NeonVideo(video_file: Path, timestamps_file: Path, info_file: Path)#

Bases: VideoCapture

Loaded video file with timestamps.

Parameters:

  • video_file (pathlib.Path) – Path to the video file.

  • timestamps_file (pathlib.Path) – Path to the timestamps file.

  • info_file (pathlib.Path) – Path to the scene camera info file.

timestamps#

Timestamps of the video frames in nanoseconds.

Type:

numpy.ndarray

ts#

Alias for timestamps.

Type:

numpy.ndarray

n_frames#

Number of frames in the video.

Type:

int

fps#

Frames per second of the video.

Type:

float

width#

Width of the video frames in pixels.

Type:

int

height#

Height of the video frames in pixels.

Type:

int

plot_frame(index: int = 0, ax: Axes | None = None, auto_title: bool = True, show: bool = True)#

Plot a frame from the video on a matplotlib axis.

Parameters:

  • index (int) – Index of the frame to plot.

  • ax (matplotlib.axes.Axes or None) – Axis to plot the frame on. If None, a new figure is created. Defaults to None.

  • auto_title (bool) – Whether to automatically set the title of the axis. The automatic title includes the video file name and the frame index. Defaults to True.

Returns:

  • fig (matplotlib.figure.Figure) – Figure object containing the plot.

  • ax (matplotlib.axes.Axes) – Axis object containing the plot.

detect_apriltags(tag_family: str = 'tag36h11') → DataFrame#

Detect AprilTags in the video frames.

Parameters:

tag_family (str, optional) – The AprilTag family to detect (default is ‘tag36h11’).

Returns:

A DataFrame containing AprilTag detections, with columns: - ‘timestamp [ns]’: The timestamp of the frame in nanoseconds, as an index - ‘frame_idx’: The frame number - ‘tag_id’: The ID of the detected AprilTag - ‘corners’: A 4x2 array of the tag corner coordinates, in the order TL, TR, BR, BL. (x, y) from top-left corner of the video - ‘center’: A 1x2 array with the tag center coordinates. (x, y) from top-left corner of the video.

Return type:

pd.DataFrame

overlay_scanpath(scanpath: DataFrame, circle_radius: int = 10, line_thickness: int = 2, max_fixations: int = 10, show_video: bool = False, video_output_path: Path | str = 'derivatives/scanpath.mp4') → None#

Plot scanpath on top of the video frames. The resulting video can be displayed and/or saved.

Parameters:

  • scanpath (pandas.DataFrame) – DataFrame containing the fixations and gaze data.

  • circle_radius (int) – Radius of the fixation circles in pixels. Defaults to 10.

  • line_thickness (int or None) – Thickness of the lines connecting fixations. If None, no lines are drawn. Defaults to 2.

  • max_fixations (int) – Maximum number of fixations to plot per frame. Defaults to 10.

  • show_video (bool) – Whether to display the video with fixations overlaid. Defaults to False.

  • video_output_path (pathlib.Path or str or None) – Path to save the video with fixations overlaid. If None, the video is not saved. Defaults to ‘scanpath.mp4’.

undistort(output_video_path: Path | str | None = 'undistorted_video.mp4') → None#

Undistort a video using the known camera matrix and distortion coefficients.

Parameters output_video_path : str

Path to save the undistorted output video.

pyneon.video.estimate_scanpath(video: NeonVideo, sync_gaze: NeonGaze, lk_params: dict | None = None) → DataFrame#

Map fixations to video frames using optical flow.

Parameters:

  • video (NeonVideo) – Video object containing the frames.

  • sync_gaze (NeonGaze) – Gaze data synchronized with the video frames.

  • lk_params (dict, optional) – Parameters for the Lucas-Kanade optical flow algorithm.

Returns:

DataFrame containing the scanpath with updated fixation points.

Return type:

pandas.DataFrame

pyneon.video.detect_apriltags(video: NeonVideo, tag_family: str = 'tag36h11', nthreads: int = 4, quad_decimate: float = 1.0, skip_frames: int = 1, random_access: bool = True) → DataFrame#

Detect AprilTags in a video and report their data for every processed frame, optionally using random access instead of sequential reading.

Parameters:

  • video (NeonVideo-like) – A video-like object with: - .read() -> returns (ret, frame) - .ts -> array/list of timestamps (in nanoseconds), same length as total frames - (Optional) .set(cv2.CAP_PROP_POS_FRAMES, frame_idx) for random access

  • tag_family (str, optional) – The AprilTag family to detect (default ‘tag36h11’).

  • nthreads (int, optional) – Number of CPU threads for the detector (default 4).

  • quad_decimate (float, optional) – Downsample input frames by this factor for detection (default 1.0). Larger values = faster detection, but might miss smaller tags.

  • skip_frames (int, optional) – If > 1, detect tags only in every Nth frame. E.g., skip_frames=5 will process frames 0, 5, 10, 15, etc.

  • random_access (bool, optional) – If True, jump directly to the frames you need (faster for large skip_frames). If False, read frames sequentially and skip in a loop (may be faster for small skip_frames or if random access is expensive).

Returns:

A DataFrame containing AprilTag detections, with columns: - ‘timestamp [ns]’ (index): The timestamp of the frame - ‘processed_frame_idx’: The count of processed frames (0-based) - ‘frame_idx’: The actual frame index in the video - ‘tag_id’: The ID of the detected AprilTag - ‘corners’: (4,2) array of tag corner coordinates - ‘center’: (1,2) array for the tag center

Return type:

pd.DataFrame

pyneon.video.estimate_camera_pose(video: NeonVideo, tag_locations_df: DataFrame, all_detections: DataFrame = None) → DataFrame#

Compute the camera pose for each frame using AprilTag detections stored in a DataFrame, handling arbitrary tag orientations.

Parameters:

  • video (NeonVideo) – Video object containing camera parameters.

  • tag_locations_df (pd.DataFrame) – DataFrame containing AprilTag 3D positions, normals, and sizes with columns: - ‘tag_id’: int, ID of the tag - ‘x’, ‘y’, ‘z’: float, coordinates of the tag’s center - ‘normal_x’, ‘normal_y’, ‘normal_z’: float, components of the tag’s normal vector - ‘size’: float, side length of the tag in meters

  • all_detections (pd.DataFrame) – DataFrame containing AprilTag detections with columns: - ‘frame_idx’: int, frame number - ‘tag_id’: int, ID of the detected tag - ‘corners’: np.ndarray of shape (4, 2), pixel coordinates of the tag’s corners - ‘center’: np.ndarray of shape (1, 2), pixel coordinates of the tag’s center (optional)

Returns:

A DataFrame with one row per frame containing: - ‘frame_idx’: int - ‘translation_vector’: np.ndarray of shape (3,) - ‘rotation_vector’: np.ndarray of shape (3,) - ‘camera_pos’: np.ndarray of shape (3,)

Return type:

pd.DataFrame

pyneon.video.transform_gaze_to_screen(gaze_df: DataFrame, homography_for_frame: dict) → DataFrame#

Apply per-frame homographies to gaze points to transform them into a new coordinate system.

Parameters:

  • gaze_df (pd.DataFrame) – DataFrame containing gaze points with columns: - ‘frame_idx’: int, the frame index - ‘x’, ‘y’: float, the gaze coordinates in the original coordinate system.

  • homography_for_frame (dict) – A dictionary mapping frame indices to 3x3 homography matrices.

Returns:

A copy of gaze_df with additional columns: - ‘x_trans’, ‘y_trans’: the transformed gaze coordinates.

Return type:

pd.DataFrame

pyneon.video.find_homographies(video, detection_df: DataFrame, marker_info: DataFrame, frame_size: tuple[int, int], coordinate_system: str = 'opencv', skip_frames: int = 1, undistort: bool = True, settings: dict | None = None) → dict#

Compute a homography for each frame using available AprilTag detections.

This function identifies all markers detected in a given frame, looks up their “ideal” (reference) positions from marker_info, and calls OpenCV’s cv2.findHomography to compute a 3x3 transformation matrix mapping from detected corners in the video image to the reference plane (e.g., screen coordinates).

If the coordinate system is “psychopy”, corners in both marker_info and detection_df are first converted to an OpenCV-like pixel coordinate system. If undistort=True and camera intrinsics are available in the video object, the marker corners are also undistorted.

The optional homography_settings dictionary allows customizing parameters like RANSAC thresholds and maximum iterations. The default is an OpenCV RANSAC method with moderate thresholds.

Parameters:

  • video (NeonVideo-like) – An object containing camera intrinsics (camera_matrix, dist_coeffs) and possibly timestamps. If undistort=True, these intrinsics are used to undistort marker corners.

  • detection_df (pd.DataFrame) – Must contain: - ‘frame_idx’: int - ‘tag_id’: int - ‘corners’: np.ndarray of shape (4, 2) in video or PsychoPy coordinates

  • marker_info (pd.DataFrame) –

    Must contain: - ‘marker_id’ (or ‘tag_id’): int - ‘marker_corners’: np.ndarray of shape (4, 2) giving the reference positions

    for each corner (e.g., on a screen plane)

  • frame_size ((width, height)) – The pixel resolution of the video frames. Used if coordinate_system=”psychopy” to convert from PsychoPy to OpenCV-style coordinates.

  • coordinate_system (str, optional) – One of {“opencv”, “psychopy”}. If “psychopy”, corners in detection_df and marker_info are converted to OpenCV pixel coords before the homography is computed.

  • undistort (bool, optional) – Whether to undistort marker corners using the camera intrinsics in video. Default is True.

  • settings (dict, optional) –

    A dictionary of parameters passed to cv2.findHomography. For example: {

    ”method”: cv2.RANSAC, “ransacReprojThreshold”: 2.0, “maxIters”: 500, “confidence”: 0.98,

    } The defaults are set to a moderate RANSAC approach.

Returns:

A dictionary mapping each frame index (frame_idx: int) to its corresponding homography matrix (3x3 NumPy array) or None if insufficient markers or points were available to compute a valid homography.

Return type:

dict

pyneon.video.detect_apriltags_parallel(video, tag_family='tag36h11', n_jobs=-1, nthreads_per_worker=1, quad_decimate=1.0, chunk_size=500) → DataFrame#

Parallel AprilTag detection by batching video frames across multiple CPU processes, with a tqdm progress bar to track overall progress.

Parameters:

  • video (OpenCV-like or custom object) – Must allow .read() -> (ret, frame) and have .ts or a known way to get timestamps. The length of video.ts is used for the total frame count.

  • tag_family (str, optional) – AprilTag family (default ‘tag36h11’).

  • n_jobs (int, optional) – Number of parallel workers (default -1 means ‘all cores’).

  • nthreads_per_worker (int, optional) – Number of CPU threads for each worker’s pupil_apriltags Detector. If you’re spawning multiple processes, it may be wise to keep this small (1 or 2).

  • quad_decimate (float, optional) – Downsampling factor for detection (default 1.0). Larger => faster detection, but can miss small tags.

  • chunk_size (int, optional) – Number of frames to read and process per chunk (default 500). Adjust for memory usage vs. efficiency.

Returns:

A DataFrame with columns: - ‘timestamp [ns]’ - ‘orig_frame_idx’ - ‘tag_id’ - ‘corners’ - ‘center’

If empty, no tags were detected or no frames were read.

Return type:

pd.DataFrame