{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exporting to BIDS Formats\n",
    "\n",
    "## What is BIDS and Why Use It?\n",
    "\n",
    "The [Brain Imaging Data Structure (BIDS)](https://bids.neuroimaging.io/index.html) is a comprehensive framework designed to systematically organize and share diverse types of data, including behavioral, physiological, and neuroimaging information. Converting datasets into BIDS format is a widely adopted methodology, particularly in the process of curating datasets that adhere to the principles of FAIR (Findable, Accessible, Interoperable, Reusable).\n",
    "\n",
    "**Key benefits of using BIDS:**\n",
    "- **Standardization**: Consistent naming conventions and directory structures across datasets\n",
    "- **Interoperability**: Enables automated analysis pipelines and data sharing\n",
    "- **Reproducibility**: Comprehensive metadata ensures experiments can be understood and replicated\n",
    "- **Community adoption**: Widely accepted format in neuroscience research\n",
    "\n",
    "The general framework of BIDS is described in the following publication:\n",
    "\n",
    "> <cite>Gorgolewski, K., Auer, T., Calhoun, V. et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci Data 3, 160044 (2016). https://doi.org/10.1038/sdata.2016.44<cite>\n",
    "\n",
    "## BIDS Extensions for Neon Data\n",
    "\n",
    "For datasets from Pupil Labs Neon eye-tracker, we utilize two BIDS extensions:\n",
    "\n",
    "1. **Motion-BIDS (BEP029)**: Organizes motion data including acceleration, angular velocity (gyroscope), and orientation from the IMU sensor\n",
    "2. **Eye-Tracking-BIDS (BEP020)**: Organizes gaze position, pupil size/diameter data, and eye-tracking events (fixations, saccades, blinks)\n",
    "\n",
    "In this tutorial, we demonstrate how to export Neon recordings to these BIDS formats using PyNeon's `export_motion_bids()` and `export_eye_bids()` methods. These functions handle all file naming, metadata generation, and formatting requirements automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "from seedir import seedir\n",
    "from pyneon import Dataset, get_sample_data\n",
    "\n",
    "# Load sample data\n",
    "dataset = Dataset(get_sample_data(\"markers\", format=\"cloud\"))\n",
    "rec = dataset.recordings[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting to Motion-BIDS\n",
    "\n",
    "The Motion-BIDS specification provides a standardized way to organize motion sensor data from devices like IMUs (Inertial Measurement Units):\n",
    "\n",
    "> <cite>Jeung, S., Cockx, H., Appelhoff, S. et al. Motion-BIDS: an extension to the brain imaging data structure to organize motion data for reproducible research. Sci Data 11, 716 (2024). https://doi.org/10.1038/s41597-024-03559-8<cite>\n",
    "\n",
    "### Understanding the BIDS Prefix\n",
    "\n",
    "The `export_motion_bids()` method requires a **prefix** string that specifies the experimental context. The prefix follows this standardized format (fields in brackets are optional):\n",
    "\n",
    "```text\n",
    "sub-<label>[_ses-<label>]_task-<label>_tracksys-<label>[_acq-<label>][_run-<index>]\n",
    "```\n",
    "\n",
    "**Required fields:**\n",
    "- `sub-<label>`: Subject/participant identifier (e.g., `sub-01`, `sub-Alice`)\n",
    "- `task-<label>`: Name of the experimental task (e.g., `task-Navigation`, `task-Reading`)\n",
    "- `tracksys-<label>`: Tracking system used (for Neon IMU: `tracksys-NeonIMU`)\n",
    "\n",
    "**Optional fields:**\n",
    "- `ses-<label>`: Session identifier for multi-session experiments\n",
    "- `acq-<label>`: Acquisition parameters or protocol\n",
    "- `run-<index>`: Run number for repeated acquisitions\n",
    "\n",
    "### Adding Custom Metadata\n",
    "\n",
    "You can include additional experiment-specific metadata by passing a dictionary to the `extra_metadata` argument. This information will be saved in the JSON metadata file and is crucial for documenting your experimental setup.\n",
    "\n",
    "Let's export the motion data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sub-01/\n",
      "└─ses-1/\n",
      "  ├─motion/\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_channels.json\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_channels.tsv\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_motion.json\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_motion.tsv\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physio.json\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physio.tsv.gz\n",
      "  │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physioevents.json\n",
      "  │ └─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physioevents.tsv.gz\n",
      "  └─sub-01_ses-1_scans.tsv\n"
     ]
    }
   ],
   "source": [
    "# Create a BIDS directory\n",
    "motion_dir = Path(\"export\") / \"BIDS\" / \"sub-01\" / \"ses-1\" / \"motion\"\n",
    "\n",
    "# Export the motion data to BIDS format\n",
    "prefix = \"sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1\"\n",
    "extra_metadata = {\n",
    "    \"TaskName\": \"LabMuse\",\n",
    "    \"TaskDescription\": \"Watching artworks on the screen\",\n",
    "    \"InstitutionName\": \"Streeling University\",\n",
    "    \"InstitutionAddress\": \"Trantor, Galactic Empire\",\n",
    "    \"InstitutionalDepartmentName\": \"Department of Psychohistory\",\n",
    "}\n",
    "\n",
    "rec.export_motion_bids(motion_dir, prefix=prefix, extra_metadata=extra_metadata)\n",
    "\n",
    "seedir(motion_dir.parent.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Motion-BIDS File Structure\n",
    "\n",
    "The export creates four files that work together to fully describe the IMU data:\n",
    "\n",
    "1. **`_motion.tsv`**: Tab-separated file containing the raw IMU time-series data (no header)\n",
    "2. **`_motion.json`**: Metadata describing the recording setup, device, and data characteristics\n",
    "3. **`_channels.tsv`**: Information about each data channel (type, units, sampling rate)\n",
    "4. **`_channels.json`**: Coordinate system information for the motion data\n",
    "\n",
    "Additionally, a **`_scans.tsv`** file is created in the parent directory to log all acquisitions for the subject/session.\n",
    "\n",
    "Let's examine each file in detail.\n",
    "\n",
    "#### 1. Motion Time-Series Data (`_motion.tsv`)\n",
    "\n",
    "This file contains the continuous IMU measurements. Each row is a sample, and each column is a sensor channel (13 total: 3 gyroscope + 3 accelerometer + 7 orientation quaternion):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Motion data shape: (26556, 13)\n",
      "          0         1          2         3         4         5         6   \\\n",
      "0 -51.717758 -2.668381  60.407639 -0.239258 -0.118164  0.899902  7.143754   \n",
      "1 -57.800365 -2.150263  66.800467 -0.232189 -0.139369  0.913207  7.205822   \n",
      "2 -63.838428 -1.759561  73.407542 -0.214823 -0.143753  0.923766  7.273488   \n",
      "3 -72.211448 -0.018978  84.477888 -0.208008 -0.153056  0.926946  7.357109   \n",
      "4 -77.977801  2.357307  92.820179 -0.196248 -0.166075  0.932501  7.462461   \n",
      "\n",
      "         7           8         9         10        11        12  \n",
      "0 -3.179328 -106.205219  0.597605  0.033180  0.059532 -0.798889  \n",
      "1 -3.592368 -105.606506  0.601491  0.031118  0.062889 -0.795788  \n",
      "2 -4.038350 -104.953976  0.605699  0.028853  0.066495 -0.792377  \n",
      "3 -4.546941 -104.196205  0.610548  0.026258  0.070614 -0.788375  \n",
      "4 -5.104590 -103.347745  0.615927  0.023435  0.075171 -0.783844  \n"
     ]
    }
   ],
   "source": [
    "physio_tsv_path = motion_dir / f\"{prefix}_motion.tsv\"\n",
    "physio_df = pd.read_csv(physio_tsv_path, sep=\"\\t\", header=None)\n",
    "print(f\"Motion data shape: {physio_df.shape}\")\n",
    "print(physio_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Motion Metadata (`_motion.json`)\n",
    "\n",
    "This file contains crucial metadata about the recording setup and data characteristics. Note how our custom metadata (TaskName, InstitutionName, etc.) has been included:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"TaskName\": \"LabMuse\",\n",
      "    \"TaskDescription\": \"Watching artworks on the screen\",\n",
      "    \"Instructions\": \"\",\n",
      "    \"DeviceSerialNumber\": \"114837\",\n",
      "    \"Manufacturer\": \"TDK InvenSense & Pupil Labs\",\n",
      "    \"ManufacturersModelName\": \"ICM-20948\",\n",
      "    \"SoftwareVersions\": \"App version: 2.9.26-prod; Pipeline version: 2.8.0\",\n",
      "    \"InstitutionName\": \"Streeling University\",\n",
      "    \"InstitutionAddress\": \"Trantor, Galactic Empire\",\n",
      "    \"InstitutionalDepartmentName\": \"Department of Psychohistory\",\n",
      "    \"SamplingFrequency\": 110,\n",
      "    \"ACCELChannelCount\": 3,\n",
      "    \"ANGACCELChannelCount\": 0,\n",
      "    \"GYROChannelCount\": 3,\n",
      "    \"JNTANGChannelCount\": 0,\n",
      "    \"LATENCYChannelCount\": 0,\n",
      "    \"MAGNChannelCount\": 0,\n",
      "    \"MISCChannelCount\": 0,\n",
      "    \"MissingValues\": \"n/a\",\n",
      "    \"MotionChannelCount\": 0,\n",
      "    \"ORNTChannelCount\": 7,\n",
      "    \"POSChannelCount\": 0,\n",
      "    \"SamplingFrequencyEffective\": 110.0000011,\n",
      "    \"SubjectArtefactDescription\": \"\",\n",
      "    \"TrackedPointsCount\": 0,\n",
      "    \"TrackingSystemName\": \"Neon IMU\",\n",
      "    \"VELChannelCount\": 0\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "motion_json = motion_dir / f\"{prefix}_motion.json\"\n",
    "with open(motion_json, \"r\") as f:\n",
    "    motion_metadata = json.load(f)\n",
    "print(json.dumps(motion_metadata, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Channel Information (`_channels.tsv`)\n",
    "\n",
    "This file provides detailed information about each channel in the motion data, including the sensor type, spatial component (x/y/z), units, and sampling frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              name component   type tracked_point      units  \\\n",
      "0           gyro x         x   GYRO          Head      deg/s   \n",
      "1           gyro y         y   GYRO          Head      deg/s   \n",
      "2           gyro z         z   GYRO          Head      deg/s   \n",
      "3   acceleration x         x  ACCEL          Head          g   \n",
      "4   acceleration y         y  ACCEL          Head          g   \n",
      "5   acceleration z         z  ACCEL          Head          g   \n",
      "6             roll         x   ORNT          Head        deg   \n",
      "7            pitch         y   ORNT          Head        deg   \n",
      "8              yaw         z   ORNT          Head        deg   \n",
      "9     quaternion w    quat_w   ORNT          Head  arbitrary   \n",
      "10    quaternion x    quat_x   ORNT          Head  arbitrary   \n",
      "11    quaternion y    quat_y   ORNT          Head  arbitrary   \n",
      "12    quaternion z    quat_z   ORNT          Head  arbitrary   \n",
      "\n",
      "                    placement  sampling_frequency status  status_description  \n",
      "0   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "1   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "2   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "3   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "4   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "5   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "6   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "7   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "8   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "9   Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "10  Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "11  Head-mounted Neon glasses          110.000001   good                 NaN  \n",
      "12  Head-mounted Neon glasses          110.000001   good                 NaN  \n"
     ]
    }
   ],
   "source": [
    "channels_tsv_path = motion_dir / f\"{prefix}_channels.tsv\"\n",
    "channels_df = pd.read_csv(channels_tsv_path, sep=\"\\t\")\n",
    "print(channels_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sensor types in Neon IMU:**\n",
    "- **GYRO**: Angular velocity (rotation rate) in degrees/second\n",
    "- **ACCEL**: Linear acceleration in g-force units\n",
    "- **ORNT**: Orientation quaternion (w, x, y, z) in arbitrary units\n",
    "\n",
    "#### 4. Coordinate System (`_channels.json`)\n",
    "\n",
    "This file defines the reference frame for interpreting the motion data. For Neon, the global reference frame is defined by the IMU axes (X=right, Y=anterior, Z=superior):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"reference_frame\": {\n",
      "        \"Levels\": {\n",
      "            \"global\": {\n",
      "                \"SpatialAxes\": \"RAS\",\n",
      "                \"RotationOrder\": \"ZXY\",\n",
      "                \"RotationRule\": \"right-hand\",\n",
      "                \"Description\": \"This global reference frame is defined by the IMU axes: X right, Y anterior, Z superior. The scene camera frame differs from this frame by a 102-degree rotation around the X-axis. All motion data are expressed relative to the IMU frame for consistency.\"\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "channels_json_path = motion_dir / f\"{prefix}_channels.json\"\n",
    "with open(channels_json_path, \"r\") as f:\n",
    "    channels_metadata = json.load(f)\n",
    "print(json.dumps(channels_metadata, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting to Eye-Tracking-BIDS\n",
    "\n",
    "The Eye-Tracking-BIDS specification standardizes how gaze position, pupil data, and eye-tracking events should be organized:\n",
    "\n",
    "> <cite>Szinte, M., Bach, D. R., Draschkow, D., Esteban, O., Gagl, B., Gau, R., Gregorova, K., Halchenko, Y. O., Huberty, S., Kling, S. M., Kulkarni, S., Maintainers, T. B., Markiewicz, C. J., Mikkelsen, M., Oostenveld, R., & Pfarr, J.-K. (2026). Eye-Tracking-BIDS: The Brain Imaging Data Structure extended to gaze position and pupil data. bioRxiv. https://doi.org/10.64898/2026.02.03.703514<cite>\n",
    "\n",
    "### Export Configuration\n",
    "\n",
    "The `export_eye_tracking_bids()` method has similar arguments to `export_motion_bids()`:\n",
    "- **output_dir**: Directory where files will be saved\n",
    "- **prefix**: BIDS naming prefix (must include `sub-<label>` and `task-<label>` at minimum). If not provided, the function will attempt to infer it from existing files in the output directory.\n",
    "- **extra_metadata**: Optional dictionary of additional metadata\n",
    "\n",
    "**Important**: When exporting eye-tracking data, it's best practice to use the same output directory as your motion (or other modality) data to link them together as part of the same recording session. The function will automatically detect and use the matching prefix from existing files, or you can explicitly provide one. Eye-tracking data is considered physiology data in BIDS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\qian.chu\\Documents\\GitHub\\PyNeon\\pyneon\\preprocess\\preprocess.py:67: UserWarning: 23 out of 48219 requested timestamps are outside the data time range and will have empty data.\n",
      "  warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "motion/\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_channels.json\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_channels.tsv\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_motion.json\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_motion.tsv\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physio.json\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physio.tsv.gz\n",
      "├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physioevents.json\n",
      "└─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_run-1_physioevents.tsv.gz\n"
     ]
    }
   ],
   "source": [
    "rec.export_eye_tracking_bids(motion_dir)\n",
    "seedir(motion_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Eye-Tracking-BIDS File Structure\n",
    "\n",
    "Eye-Tracking-BIDS creates four main files:\n",
    "\n",
    "1. **`_physio.tsv.gz`**: Compressed time-series data for gaze and pupil measurements\n",
    "2. **`_physio.json`**: Metadata describing the eye-tracking setup and data columns\n",
    "3. **`_physioevents.tsv.gz`**: Event data (fixations, saccades, blinks, custom messages)\n",
    "4. **`_physioevents.json`**: Metadata for the events file\n",
    "\n",
    "#### 1. Physiological Time-Series Data (`_physio.tsv.gz`)\n",
    "\n",
    "This compressed file contains continuous gaze and pupil data with 5 columns:\n",
    "- **timestamp**: Time in nanoseconds\n",
    "- **x_coordinate**: Horizontal gaze position in pixels\n",
    "- **y_coordinate**: Vertical gaze position in pixels\n",
    "- **left_pupil_diameter**: Left pupil diameter in millimeters\n",
    "- **right_pupil_diameter**: Right pupil diameter in millimeters\n",
    "\n",
    "Let's inspect the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eye-tracking data shape: (48219, 5)\n",
      "                     0        1         2       3       4\n",
      "0  1758493516475391023  535.360  1016.989  4.6517  4.5655\n",
      "1  1758493516480394023  519.555  1033.236  4.0452  4.5209\n",
      "2  1758493516485390023  521.541  1045.861  3.4693  4.5801\n",
      "3  1758493516490390023  503.323  1049.000  4.1283  4.7350\n",
      "4  1758493516495406023  498.538  1065.085  4.5646  4.8635\n"
     ]
    }
   ],
   "source": [
    "physio_tsv_path = motion_dir / f\"{prefix}_physio.tsv.gz\"\n",
    "physio_df = pd.read_csv(physio_tsv_path, sep=\"\\t\", compression=\"gzip\", header=None)\n",
    "print(f\"Eye-tracking data shape: {physio_df.shape}\")\n",
    "print(physio_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Physiological Data Metadata (`_physio.json`)\n",
    "\n",
    "This file provides comprehensive metadata about the eye-tracking data, including column definitions, sampling frequency, and device information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"SamplingFrequency\": 199.71919360432685,\n",
      "    \"StartTime\": 0,\n",
      "    \"Columns\": [\n",
      "        \"timestamp\",\n",
      "        \"x_coordinate\",\n",
      "        \"y_coordinate\",\n",
      "        \"left_pupil_diameter\",\n",
      "        \"right_pupil_diameter\"\n",
      "    ],\n",
      "    \"DeviceSerialNumber\": \"114837\",\n",
      "    \"Manufacturer\": \"Pupil Labs\",\n",
      "    \"ManufacturersModelName\": \"Neon\",\n",
      "    \"SoftwareVersions\": \"App version: 2.9.26-prod; Pipeline version: 2.8.0\",\n",
      "    \"PhysioType\": \"eyetrack\",\n",
      "    \"EnvironmentCoorinates\": \"top-left\",\n",
      "    \"RecordedEye\": \"cyclopean\",\n",
      "    \"SampleCoordinateSystem\": \"gaze-in-world\",\n",
      "    \"EyeTrackingMethod\": \"real-time neural network\",\n",
      "    \"timestamp\": {\n",
      "        \"Description\": \"UTC timestamp in nanoseconds of the sample\",\n",
      "        \"Units\": \"ns\"\n",
      "    },\n",
      "    \"x_coordinate\": {\n",
      "        \"Description\": \"X-coordinate of the mapped gaze point in world camera pixel coordinates.\",\n",
      "        \"Units\": \"pixel\"\n",
      "    },\n",
      "    \"y_coordinate\": {\n",
      "        \"Description\": \"Y-coordinate of the mapped gaze point in world camera pixel coordinates.\",\n",
      "        \"Units\": \"pixel\"\n",
      "    },\n",
      "    \"left_pupil_diameter\": {\n",
      "        \"Description\": \"Physical diameter of the pupil of the left eye\",\n",
      "        \"Units\": \"mm\"\n",
      "    },\n",
      "    \"right_pupil_diameter\": {\n",
      "        \"Description\": \"Physical diameter of the pupil of the right eye\",\n",
      "        \"Units\": \"mm\"\n",
      "    },\n",
      "    \"TaskName\": \"LabMuse\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "physio_json = motion_dir / f\"{prefix}_physio.json\"\n",
    "with open(physio_json, \"r\") as f:\n",
    "    physio_metadata = json.load(f)\n",
    "print(json.dumps(physio_metadata, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Eye-Tracking Events (`_physioevents.tsv.gz`)\n",
    "\n",
    "This file contains all detected eye-tracking events and custom messages. Each row represents one event with columns:\n",
    "- **onset**: Event start time in nanoseconds\n",
    "- **duration**: Event duration in seconds (for fixations, saccades, blinks)\n",
    "- **trial_type**: Type of event (`fixation`, `saccade`, `blink`)\n",
    "- **message**: Custom event messages/markers (when applicable)\n",
    "\n",
    "PyNeon automatically exports all available events from the recording, including algorithmically detected events (fixations, saccades, blinks) and user-defined messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total events: 1101\n",
      "                     0      1         2                3\n",
      "0  1758493514096000000    NaN       NaN  recording.begin\n",
      "1  1758493516475391023  0.105   saccade              NaN\n",
      "2  1758493516580513023  0.535  fixation              NaN\n",
      "3  1758493517116002023  0.110   saccade              NaN\n",
      "4  1758493517226130023  0.135  fixation              NaN\n",
      "5  1758493517326259023  0.180     blink              NaN\n",
      "6  1758493517361251023  0.115   saccade              NaN\n",
      "7  1758493517476369023  0.585  fixation              NaN\n",
      "8  1758493518061856023  0.055   saccade              NaN\n",
      "9  1758493518116984023  0.075  fixation              NaN\n"
     ]
    }
   ],
   "source": [
    "physioevents_tsv_path = motion_dir / f\"{prefix}_physioevents.tsv.gz\"\n",
    "physioevents_df = pd.read_csv(\n",
    "    physioevents_tsv_path, sep=\"\\t\", header=None, compression=\"gzip\"\n",
    ")\n",
    "print(f\"Total events: {physioevents_df.shape[0]}\")\n",
    "print(physioevents_df.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Events Metadata (`_physioevents.json`)\n",
    "\n",
    "This file describes the structure and meaning of the events data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"Columns\": [\n",
      "        \"onset\",\n",
      "        \"duration\",\n",
      "        \"trial_type\",\n",
      "        \"message\"\n",
      "    ],\n",
      "    \"Description\": \"Eye events and messages logged by Neon\",\n",
      "    \"OnsetSource\": \"timestamp\",\n",
      "    \"onset\": {\n",
      "        \"Description\": \"UTC timestamp in nanoseconds of the start of the event\",\n",
      "        \"Units\": \"ns\"\n",
      "    },\n",
      "    \"duration\": {\n",
      "        \"Description\": \"Event duration\",\n",
      "        \"Units\": \"s\"\n",
      "    },\n",
      "    \"trial_type\": {\n",
      "        \"Description\": \"Type of trial event\",\n",
      "        \"Levels\": {\n",
      "            \"fixation\": {\n",
      "                \"Description\": \"Fixation event\"\n",
      "            },\n",
      "            \"saccade\": {\n",
      "                \"Description\": \"Saccade event\"\n",
      "            },\n",
      "            \"blink\": {\n",
      "                \"Description\": \"Blink event\"\n",
      "            }\n",
      "        }\n",
      "    },\n",
      "    \"TaskName\": \"LabMuse\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "physioevents_json = motion_dir / f\"{prefix}_physioevents.json\"\n",
    "with open(physioevents_json, \"r\") as f:\n",
    "    physioevents_metadata = json.load(f)\n",
    "print(json.dumps(physioevents_metadata, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Exporting Multiple Recordings\n",
    "\n",
    "In real studies, a dataset often contains multiple recordings from the same participant (for example, repeated sessions or separate acquisitions). PyNeon makes this straightforward: iterate over `dataset.recordings`, generate a session-specific BIDS prefix, and export each recording into its own `ses-*` directory.\n",
    "\n",
    "This workflow:\n",
    "- Keeps all recordings grouped under one subject (`sub-01`)\n",
    "- Creates a separate BIDS session folder for each recording (`ses-1`, `ses-2`, ...)\n",
    "- Exports both Motion-BIDS and Eye-Tracking-BIDS outputs in a consistent structure\n",
    "\n",
    "A similar logic applies to multi-participant datasets: add an outer loop over participants and update the `sub-*` entity in both the prefix and output path.\n",
    "\n",
    "Let's perform a complete batch export for all recordings in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\qian.chu\\Documents\\GitHub\\PyNeon\\pyneon\\preprocess\\preprocess.py:67: UserWarning: 23 out of 48219 requested timestamps are outside the data time range and will have empty data.\n",
      "  warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BIDS_complete/\n",
      "└─sub-01/\n",
      "  ├─ses-1/\n",
      "  │ ├─motion/\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_channels.json\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_channels.tsv\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_motion.json\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_motion.tsv\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_physio.json\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_physio.tsv.gz\n",
      "  │ │ ├─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_physioevents.json\n",
      "  │ │ └─sub-01_ses-1_task-LabMuse_tracksys-NeonIMU_physioevents.tsv.gz\n",
      "  │ └─sub-01_ses-1_scans.tsv\n",
      "  └─ses-2/\n",
      "    ├─motion/\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_channels.json\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_channels.tsv\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_motion.json\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_motion.tsv\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_physio.json\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_physio.tsv.gz\n",
      "    │ ├─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_physioevents.json\n",
      "    │ └─sub-01_ses-2_task-LabMuse_tracksys-NeonIMU_physioevents.tsv.gz\n",
      "    └─sub-01_ses-2_scans.tsv\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\qian.chu\\Documents\\GitHub\\PyNeon\\pyneon\\preprocess\\preprocess.py:67: UserWarning: 23 out of 6496 requested timestamps are outside the data time range and will have empty data.\n",
      "  warn(\n"
     ]
    }
   ],
   "source": [
    "for i, rec in enumerate(dataset.recordings):\n",
    "    prefix = f\"sub-01_ses-{i + 1}_task-LabMuse_tracksys-NeonIMU\"\n",
    "    dir = Path(\"export\") / \"BIDS_complete\" / \"sub-01\" / f\"ses-{i + 1}\" / \"motion\"\n",
    "    rec.export_motion_bids(dir, prefix=prefix, extra_metadata=extra_metadata)\n",
    "    rec.export_eye_tracking_bids(dir, prefix=prefix)\n",
    "seedir(Path(\"export\") / \"BIDS_complete\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pyneon",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}