{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Scene Video and Scanpath Mapping\n", "\n", "In this tutorial, we will map gaze data from an eye-tracking recording to video frames, estimate a scanpath, and overlay the gaze fixations on the video. We will use the `pyneon` library to work with Neon eye-tracking recordings, which contain video and event data, including gaze information.\n", "\n", "---\n", "\n", "## Setup: Loading a Neon Recording\n", "\n", "First, we load the Neon recording, which contains video and gaze data. Ensure that you have installed the required libraries such as `pyneon` and have the recording dataset available." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Data format: cloud (version: 2.5)\n", "Recording ID: c17cd630-764e-4e61-87ee-95be3d6b8181\n", "Wearer ID: 028e4c69-f333-4751-af8c-84a09af079f5\n", "Wearer name: Pilot\n", "Recording start time: 2025-09-22 00:31:44.395000\n", "Recording duration: 35977000000 ns (35.977 s)\n", "\n" ] } ], "source": [ "# Import necessary libraries\n", "from pyneon import Dataset, get_sample_data, Stream\n", "\n", "# Load a sample recording\n", "dataset_dir = get_sample_data(\"markers\", format=\"cloud\")\n", "dataset = Dataset(dataset_dir)\n", "\n", "recording = dataset[1]\n", "print(recording)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Mapping Gaze Data to Video Frames\n", "\n", "In Neon recordings, gaze events are not naturally synchronized with the video. To map gaze data to specific video frames, we can use the `map_gaze_to_video` method. This method requires the `pyneon.video` object for determination of video timestamps, the `pyneon.fixations` object to make use of PupilLabs fixation detection pipeline and the `pyneon.gaze` object for improved time resolution of gaze estimation.\n", "\n", "By default, Neon reports fixations with a single coordinate. This is computed as average between all gaze coordinates over the interval denoted as a fixation. However, this clashes with the functional definition of a fixation as _tracking a fixed point in space_, used by Neon.\n", "\n", "Imagine looking at a fixed point, for example a street sign, while you are walking past it. Despite the movement of your body and the relative movement of the sign, the fixation will be stabilized. As such, taking an average gaze coordinate over the entire duration will not correspond to the location of the sign, or the fixation, at any given point in time. Feeding this point into an optical flow algorithm would, with high likelihood, lead to tracking anything but the sign.\n", "\n", "Therefore, we use partial averages of gaze locations around the respective frame's timestamp. As the video is sampled at 30Hz while the gaze output nominally reaches 200Hz, we expect to take the average over 6 subsequent gaze points. This achieves a trade-off between recency of the reported gaze position at the given frame and error minimization, by averaging over microsaccades around the actual fixation target as well as random errors." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Map gaze data to the video timestamps\n", "synced_gaze = recording.sync_gaze_to_video()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Above, we can see that each frame gets a current gaze position as well as a fixation status. Currently, three types of fixation status are used:\n", "\n", "1. `start` denoting the first frame corresponding to a fixation\n", "2. `during` corresponding to intermediate frames of the same fixation\n", "3. `end` denoting the last frame of the fixation\n", "\n", "This determination will become relevant for tracking the scanpath with optical flow. After all, while a fixation is still active, we get up-to-date gaze information. Only after its end, tracking becomes necessary." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Estimating the Scanpath\n", "\n", "Having matched every frame with a gaze coordinate, we can now get into the core of scanpath estimation. In dynamic scenes, the same object will not occupy the same scene-camera location over time. Therefore, we need to continuously map past fixation points as long as they are still visible in the frame.\n", "\n", "The `estimate_scanpath` method achieves this by feeding fixation points (those marked as `end`) into a Lucas-Kanade sparse optical flow algorithm. This algorithm compares the video region around each point with the subsequent frame, updating the point location according to its motion. While a point is tracked, its status is set to `tracked`. In practice, many scene frames will contain multiple past fixations; our implementation tracks them and repeatedly performs an optical flow estimation for each point. When a point can no longer be tracked it is marked `lost` and dropped for subsequent frames.\n", "\n", "Note: this algorithm is not optimized for performance and may take considerable time on limited hardware. On our machines it runs at roughly 0.5x real-time (about half the video length), but this benchmark depends heavily on the density of past fixation points and available computational resources.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Estimating scanpath: 98%|█████████▊| 1023/1044 [00:35<00:00, 29.25it/s]C:\\Users\\qian.chu\\Documents\\GitHub\\PyNeon\\pyneon\\video\\scanpath.py:149: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n", " curr_fixations = pd.concat(\n", "Estimating scanpath: 100%|██████████| 1044/1044 [00:35<00:00, 29.01it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ " fixations \\\n", "timestamp [ns] \n", "1758493904395000000 fixation id gaze x [px] gaze y [px] fixation... \n", "1758493904445000000 fixation id gaze x [px] gaze y [px] fixation... \n", "1758493904495000000 fixation id gaze x [px] gaze y [px] fixation... \n", "1758493904545000000 fixation id gaze x [px] gaze y [px] fixation... \n", "1758493904595000000 fixation id gaze x [px] gaze y [px] fixation... \n", "\n", " frame index \n", "timestamp [ns] \n", "1758493904395000000 0 \n", "1758493904445000000 1 \n", "1758493904495000000 2 \n", "1758493904545000000 3 \n", "1758493904595000000 4 \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "# Estimate the scanpath based on the mapped gaze data\n", "from pyneon.video import estimate_scanpath\n", "\n", "scanpath_df = estimate_scanpath(recording.scene_video, synced_gaze)\n", "scanpath_df.index.name = \"timestamp [ns]\"\n", "scanpath = Stream(scanpath_df)\n", "\n", "# Inspect the estimated scanpath\n", "print(scanpath.data.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We should take a moment to understand the format of the `scanpath.data`. To map a scanpath to every video frame, we create it as a dataframe of dataframes. Each row contains the timestamp and the frame index of the underlying video and stores a dataframe in the `fixations` cell. In that dataframe, every present fixation has an id, coordinates, and a fixation status. Treating it as a dataframe enables intuitive pandas indexing; for example, you can get the list of fixations at frame 2000.\n", "\n", "Because Neon can take some time to start, the first frames usually do not yield usable results. We keep them for consistency.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " fixation id gaze x [px] gaze y [px] fixation status\n", "0 32 794.401571 622.680714 during\n", "1 31 828.15155 648.225586 tracked\n", "2 20 319.503265 492.966797 tracked\n", "3 19 372.389526 775.100464 tracked\n", "4 18 1205.629761 745.42688 tracked\n", "5 17 1214.341553 401.155457 tracked\n", "6 16 1181.176392 460.626556 tracked\n", "7 10 415.88559 809.316284 tracked\n", "8 7 1187.457275 749.382629 tracked\n", "9 5 1210.753784 408.591949 tracked\n" ] } ], "source": [ "# print fixations when column frame_idx is 1334. Frame_idx is not the idx of the dataframe, but the index of the video frame.\n", "print(scanpath.data.loc[scanpath.data[\"frame index\"] == 500, \"fixations\"].values[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## 4. Understanding Fixation Status\n", "\n", "Each fixation is assigned a status that indicates its lifecycle:\n", "\n", "- **start**: first frame of fixation\n", "- **during**: intermediate frames of fixation\n", "- **end**: last frame of fixation\n", "- **tracked**: Optical flow algorithm tracks fixation\n", "- **lost**: Tracking is lost, fixation is no longer tracked and gets dropped\n", "\n", "---\n", "\n", "## 5. Overlaying Fixations on the Video\n", "\n", "Now that we have the scanpath, we can overlay the gaze fixations on the video. This creates a video output with overlaid fixations, where:\n", "\n", "- A **blue dot** represents the current gaze location.\n", "- **Green dots** represent tracked fixations.\n", "- A **red dot** indicates no fixation (saccades or blinks).\n", "\n", "Further, we draw connecting lines between past fixations to show the scanpath for the current video. The show_video option creates a live-output of the video rendering, but also increases the runtime." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Plotting scanpath on scene video: 100%|██████████| 1044/1044 [00:25<00:00, 40.55it/s]\n" ] } ], "source": [ "# Overlay the scanpath on the video and show the output\n", "from pyneon.vis import overlay_scanpath\n", "\n", "overlay_scanpath(\n", " recording.scene_video,\n", " scanpath.data,\n", " circle_radius=10,\n", " line_thickness=2,\n", " text_size=1,\n", " max_fixations=10,\n", " show_video=True,\n", " output_path=None,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Summary\n", "\n", "- **Mapping Gaze to Video**: We used the `map_gaze_to_video` method to match gaze data with video frames based on timestamps.\n", "- **Estimating Scanpath**: The scanpath was estimated using `pyneon.video.estimate_scanpath`, which tracks fixations and uses optical flow to follow past fixations across scene changes.\n", "- **Overlaying Fixations**: The fixations were visualized on the video by calling `pyneon.vis.overlay_scanpath`.\n", "\n", "This workflow can be used to process eye-tracking data, align it with video frames, and visualize gaze movements within video recordings.\n" ] } ], "metadata": { "kernelspec": { "display_name": "pyneon", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.11" } }, "nbformat": 4, "nbformat_minor": 2 }