{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6fcaa20",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import lumicks.pylake as lk\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc507ff",
   "metadata": {},
   "source": [
    "# Files and channels\n",
    "\n",
    "[Download this page as a Jupyter notebook](_downloads/dcb2177831375e2e0d8e1c04e8057839/file.ipynb)\n",
    "\n",
    "Opening a Bluelake HDF5 file is very simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25056fec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lumicks.pylake as lk\n",
    "\n",
    "file = lk.File(\"example.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71566b3c",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "To see a textual representation of the contents of a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a09826bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b807329",
   "metadata": {},
   "source": [
    "For a listing of more specific timeline items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73ffb4a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "list(file.scans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd23524a",
   "metadata": {},
   "outputs": [],
   "source": [
    "list(file.kymos)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "555adfde",
   "metadata": {},
   "source": [
    "They can also be printed to get more information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01e1cbff",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file.scans)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5263a5fd",
   "metadata": {},
   "source": [
    "## Channels\n",
    "\n",
    "Just like the Bluelake timeline, exported HDF5 files contain multiple channels of data. They can be easily accessed as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23432c67",
   "metadata": {},
   "outputs": [],
   "source": [
    "file.force1x.plot()\n",
    "plt.savefig(\"force1x.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff4662f2",
   "metadata": {},
   "source": [
    "The channels have a few convenient methods, like `.plot()` which make it easy to preview the contents, but you can also always access the raw data directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb0d4cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1x_data = file.force1x.data\n",
    "f1x_timestamps = file.force1x.timestamps\n",
    "plt.plot(f1x_timestamps, f1x_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a62cb5",
   "metadata": {},
   "source": [
    "The `timestamps` attribute returns absolute values in nanoseconds. The relative time values in seconds can also be accessed directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "637348ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1x_seconds = file.force1x.seconds\n",
    "plt.plot(f1x_time, f1x_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12014f23",
   "metadata": {},
   "source": [
    "The above examples use the `force1x` channel. A full list of available channels can be found on the [`File`](tutorial/../_api/lumicks.pylake.File.html#lumicks.pylake.File) reference page.\n",
    "\n",
    "### Slicing\n",
    "\n",
    "By default, entire channels are returned from a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9fb82d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "everything = file.force1x\n",
    "everything.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ec3304",
   "metadata": {},
   "source": [
    "But channels can easily be sliced:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "790941b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the data between 1 and 1.5 seconds\n",
    "part = file.force1x['1s':'1.5s']\n",
    "part.plot()\n",
    "# Or manually\n",
    "f1x_data = part.data\n",
    "f1x_timestamps = part.timestamps\n",
    "plt.plot(f1x_timestamps, f1x_data)\n",
    "\n",
    "# More slicing examples\n",
    "a = file.force1x[:'-5s']  # everything except the last 5 seconds\n",
    "b = file.force1x['-1m':]  # take the last minute\n",
    "c = file.force1x['-1m':'-500ms']  # last minute except the last 0.5 seconds\n",
    "d = file.force1x['1.2s':'-4s']  # between 1.2 seconds and 4 seconds from the end\n",
    "e = file.force1x['5.7m':'1h 40m']  # 5.7 minutes to an hour and 40 minutes\n",
    "\n",
    "# Subslicing is also possible\n",
    "a = file.force1x['1s':]  # from 1 second to the end of the file\n",
    "b = a['1s':]  # 1 second relative to the start of slice `a`\n",
    "              # --> `b` starts at 2 seconds relative to the beginning of the file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d00d98",
   "metadata": {},
   "source": [
    "Note that channels are indexed in time units using numbers with suffixes. The possible suffixes are d, h, m, s, ms, us, ns, corresponding to day, hour, minute, second, millisecond, microsecond and nanosecond. This indexing only applies to channels slices. Once you access the raw data, those are regular arrays which use regular array indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8086feb",
   "metadata": {},
   "outputs": [],
   "source": [
    "channel_slice = file.force1x['1.5s':'20s']  # timestamps\n",
    "data_slice = file.force1x.data[20:40]  # indices into the array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b53956fb",
   "metadata": {},
   "source": [
    "Plotting is typically performed with the origin of the plot set to the timestamp of the start of the slice. Sometimes, you may want to plot two slices together that have different starting times. You can pass a custom reference timestamp to the plotting function to make sure they use the same time shift:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ca0dba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "first_slice = file.force1x['5s':'10s']\n",
    "second_slice = file.force1x['15s':'20s']\n",
    "first_slice.plot()\n",
    "second_slice.plot(start=first_slice.start)  # we want to use the start of first_slice as time point \"zero\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11358903",
   "metadata": {},
   "source": [
    "### Downsampling\n",
    "\n",
    "A slice can be downsampled using various methods.\n",
    "\n",
    "To downsample to a specific frequency use `downsampled_to` with the desired frequency in Hz:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15dfbcf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "channel = file.force1x # original frequency 78125 Hz\n",
    "timestep = np.diff(channel.timestamps[:2]) * 1e-9        # timestep 12.8 us\n",
    "\n",
    "ds_channel = channel.downsampled_to(3125)\n",
    "ds_timestep = np.diff(ds_channel.timestamps[:2]) * 1e-9  # timestep 320 us"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "012d09e4",
   "metadata": {},
   "source": [
    "By default, this method will take the mean of every N samples where N is defined as the ratio between the two sampling times. This can cause issues when N isn’t an integer, leading to an unequal number of points contributing to each point in the downsampled channel. To automatically find the nearest higher frequency that will fulfill this requirement, use the `method=\"ceil\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35479495",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_channel2 = channel.downsampled_to(3126, method=\"ceil\")\n",
    "ds_timestep2 = np.diff(ds_channel2.timestamps[:2]) * 1e-9  # timestep 307.2 us"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39ca35dc",
   "metadata": {},
   "source": [
    "For data that is recorded with variable sampling frequencies, it is usually not possible to downsample to a single sample rate, while maintaining an equal number of samples per downsampled sample. To force downsampling to a single frequency in the case of variable sample rates, use `method=\"force\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "764e119d",
   "metadata": {},
   "outputs": [],
   "source": [
    "variable_channel = file.downsampled_force1x\n",
    "variable_ds_channel = variable_channel.downsampled_to(3125, method=\"force\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f54477f",
   "metadata": {},
   "source": [
    "Note that this same flag can also be used to force a specific downsampling rate for non-integer downsampling rates.\n",
    "\n",
    "A slice can also be downsampled over arbitrary time segments by using `downsampled_over` and supplying a list of `(start, stop)` tuples indicating over which ranges to apply the function.\n",
    "\n",
    "A slice that contains equally spaced timestamps can be downsampled by a specific factor using `downsampled_by` (note that the ratio of the original/final sampling frequencies must be an integer.):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a8bd13f",
   "metadata": {},
   "outputs": [],
   "source": [
    "channel = file.force1x # original frequency 78125 Hz\n",
    "timestep = np.diff(channel.timestamps[:2]) * 1e-9        # timestep 12.8 us\n",
    "\n",
    "ds_channel = channel.downsampled_by(5)\n",
    "ds_timestep = np.diff(ds_channel.timestamps[:2]) * 1e-9  # timestep 64 us"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e77e93",
   "metadata": {},
   "source": [
    "Sometimes, one may want to downsample a high frequency channel in exactly the same way that a Bluelake low frequency channel is sampled. For this purpose you can use `downsampled_like`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bf90d85",
   "metadata": {},
   "outputs": [],
   "source": [
    "lf_data = file[\"Force LF\"][\"Force 1x\"]\n",
    "downsampled = file[\"Force HF\"][\"Force 1x\"].downsampled_like(lf_data)\n",
    "\n",
    "lf_data.plot()\n",
    "downsampled.plot(start=lf_data.start)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c731f6a9",
   "metadata": {},
   "source": [
    "## Calibrations\n",
    "\n",
    "Calibration information for force channels can be found by checking the calibration member. This gives a list of calibrations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd41783e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file.force1x.calibration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "378e99c6",
   "metadata": {},
   "source": [
    "The actual values can be obtained from the list as follows, where the index refers to the calibration entry and the name to the actual field value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08dfc742",
   "metadata": {},
   "outputs": [],
   "source": [
    "file.force1x.calibration[0][\"Offset (pN)\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e762067d",
   "metadata": {},
   "source": [
    "If we slice a force channel, we only obtain the calibrations relevant for the selected region.\n",
    "\n",
    "## Markers\n",
    "\n",
    "We can see that the file also contains markers. These can be accessed from the markers attribute which returns a dictionary of markers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6af344f",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file.markers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "934c88dc",
   "metadata": {},
   "source": [
    "The actual markers can be obtained from the dictionary as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e77fbbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "file.markers[\"FRAP 3\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8be1d87d",
   "metadata": {},
   "source": [
    "We can find the start and stop time with `.start` and `.stop`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2b140e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file.markers[\"FRAP 3\"].start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2385083a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(file.markers[\"FRAP 3\"].stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61665903",
   "metadata": {},
   "source": [
    "## Exporting h5 files\n",
    "\n",
    "We can save the Bluelake HDF5 file to a different filename by using [`save_as()`](tutorial/../_api/lumicks.pylake.File.html#lumicks.pylake.File.save_as). When transferring data, it can be beneficial to omit some channels from the h5 file, or use a higher compression ratio. In particular, high frequency channels tend to take up a lot of space, and aren’t always necessary for every analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a012a24c",
   "metadata": {},
   "outputs": [],
   "source": [
    "file.save_as(\"no_hf.h5\", omit_data={\"Force HF/*\"})  # Omit high frequency force data from export"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60ae4e8a",
   "metadata": {},
   "source": [
    "We use `fnmatch` patterns for specifying which fields to omit from the saved `h5` file."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}