Scene video class#

class pyneon.video.NeonVideo(video_file: Path, timestamps_file: Path, info_file: Path)#

Bases: VideoCapture

Loaded video file with timestamps.

Parameters:

  • video_file (pathlib.Path) – Path to the video file.

  • timestamps_file (pathlib.Path) – Path to the timestamps file.

  • info_file (pathlib.Path) – Path to the scene camera info file.

timestamps#

Timestamps of the video frames in nanoseconds.

Type:

numpy.ndarray

ts#

Alias for timestamps.

Type:

numpy.ndarray

n_frames#

Number of frames in the video.

Type:

int

fps#

Frames per second of the video.

Type:

float

width#

Width of the video frames in pixels.

Type:

int

height#

Height of the video frames in pixels.

Type:

int

plot_frame(index: int = 0, ax: Axes | None = None, auto_title: bool = True, show: bool = True)#

Plot a frame from the video on a matplotlib axis.

Parameters:

  • index (int) – Index of the frame to plot.

  • ax (matplotlib.axes.Axes or None) – Axis to plot the frame on. If None, a new figure is created. Defaults to None.

  • auto_title (bool) – Whether to automatically set the title of the axis. The automatic title includes the video file name and the frame index. Defaults to True.

Returns:

  • fig (matplotlib.figure.Figure) – Figure object containing the plot.

  • ax (matplotlib.axes.Axes) – Axis object containing the plot.

detect_apriltags(tag_family: str = 'tag36h11') → DataFrame#

Detect AprilTags in the video frames.

Parameters:

tag_family (str, optional) – The AprilTag family to detect (default is ‘tag36h11’).

Returns:

A DataFrame containing AprilTag detections, with columns: - ‘timestamp [ns]’: The timestamp of the frame in nanoseconds, as an index - ‘frame_idx’: The frame number - ‘tag_id’: The ID of the detected AprilTag - ‘corners’: A 4x2 array of the tag corner coordinates, in the order TL, TR, BR, BL. (x, y) from top-left corner of the video - ‘center’: A 1x2 array with the tag center coordinates. (x, y) from top-left corner of the video.

Return type:

pd.DataFrame

overlay_scanpath(scanpath: DataFrame, circle_radius: int = 10, line_thickness: int = 2, max_fixations: int = 10, show_video: bool = False, video_output_path: Path | str = 'derivatives/scanpath.mp4') → None#

Plot scanpath on top of the video frames. The resulting video can be displayed and/or saved.

Parameters:

  • scanpath (pandas.DataFrame) – DataFrame containing the fixations and gaze data.

  • circle_radius (int) – Radius of the fixation circles in pixels. Defaults to 10.

  • line_thickness (int or None) – Thickness of the lines connecting fixations. If None, no lines are drawn. Defaults to 2.

  • max_fixations (int) – Maximum number of fixations to plot per frame. Defaults to 10.

  • show_video (bool) – Whether to display the video with fixations overlaid. Defaults to False.

  • video_output_path (pathlib.Path or str or None) – Path to save the video with fixations overlaid. If None, the video is not saved. Defaults to ‘scanpath.mp4’.

undistort(output_video_path: Path | str | None = 'undistorted_video.mp4') → None#

Undistort a video using the known camera matrix and distortion coefficients.

Parameters output_video_path : str

Path to save the undistorted output video.

pyneon.video.estimate_scanpath(video: NeonVideo, sync_gaze: NeonGaze, lk_params: dict | None = None) → DataFrame#

Map fixations to video frames using optical flow.

Parameters:

  • video (NeonVideo) – Video object containing the frames.

  • sync_gaze (NeonGaze) – Gaze data synchronized with the video frames.

  • lk_params (dict, optional) – Parameters for the Lucas-Kanade optical flow algorithm.

Returns:

DataFrame containing the scanpath with updated fixation points.

Return type:

pandas.DataFrame

pyneon.video.detect_apriltags(video: NeonVideo, tag_family: str = 'tag36h11')#

Detect AprilTags in a video and report their data for every frame using the apriltag library.

Parameters:

  • video (cv2.VideoCapture or similar video object) – A video capture object from which frames can be read.

  • tag_family (str, optional) – The AprilTag family to detect (default is ‘tag36h11’).

Returns:

A DataFrame containing AprilTag detections, with columns: - ‘timestamp [ns]’: The timestamp of the frame in nanoseconds, as an index - ‘frame_idx’: The frame number - ‘tag_id’: The ID of the detected AprilTag - ‘corners’: A 4x2 array of the tag corner coordinates - ‘center’: A 1x2 array with the tag center coordinates

Return type:

pd.DataFrame

pyneon.video.estimate_camera_pose(video: NeonVideo, tag_locations_df: DataFrame, all_detections: DataFrame = None) → DataFrame#

Compute the camera pose for each frame using AprilTag detections stored in a DataFrame, handling arbitrary tag orientations.

Parameters:

  • video (NeonVideo) – Video object containing camera parameters.

  • tag_locations_df (pd.DataFrame) – DataFrame containing AprilTag 3D positions, normals, and sizes with columns: - ‘tag_id’: int, ID of the tag - ‘x’, ‘y’, ‘z’: float, coordinates of the tag’s center - ‘normal_x’, ‘normal_y’, ‘normal_z’: float, components of the tag’s normal vector - ‘size’: float, side length of the tag in meters

  • all_detections (pd.DataFrame) – DataFrame containing AprilTag detections with columns: - ‘frame_idx’: int, frame number - ‘tag_id’: int, ID of the detected tag - ‘corners’: np.ndarray of shape (4, 2), pixel coordinates of the tag’s corners - ‘center’: np.ndarray of shape (1, 2), pixel coordinates of the tag’s center (optional)

Returns:

A DataFrame with one row per frame containing: - ‘frame_idx’: int - ‘translation_vector’: np.ndarray of shape (3,) - ‘rotation_vector’: np.ndarray of shape (3,) - ‘camera_pos’: np.ndarray of shape (3,)

Return type:

pd.DataFrame