PiVR has been developed by David Tadres and Matthieu Louis (`Louis Lab
<https://labs.mcdb.ucsb.edu/louis/matthieu/>`__).

.. _Benchmark:

PiVR Benchmark
***************

.. _Concepts:

Concepts
========

When creating a virtual reality two different measurements of time
are relevant:

#. The **loop time**: How often per second does the system take input
   from the subject? This is usually expressed in **Hz**.
#. The **latency**: How long between taking the input and presenting the
   subject with the updated stimulus that makes the virtual reality?
   This is usually expressed in milliseconds (**ms**).

Next, how can one measure time on a computer? Most operating systems
(Windows, MacOS and Linux) are *non-real time operating systems*.
This means that the OS can **not** guarantee that a given line of
code is executed at a fixed interval. If one asks the system to
simply record the time to measure *loop time* and *latency* on
introduces uncertainty, possibly in the order of milliseconds.

Luckily PiVR has a real-time operating system - on the GPU:
Digital cameras work by reading out lines on their sensor and the
camera used by PiVR is no different. (Read `here
<https://picamera.readthedocs.io/en/release-1.13/fov.html#>`__ for a
fantastic introduction).

For example at 90 frames per second at 640x480 the camera must read
out 480 lines in ~11ms. It must therefore spend no more than 2.3^-5s
or 23.1us per line. This means every 23.1us the camera is pushing one
frame line to the GPU (see `here
<https://picamera.readthedocs.io/en/release-1.13/fov.html#division-of-labor>`__
for detailed information).

The GPU records the timestamp of when the first line of a new frame
is received based on its real-time hardware clock. We
take advantage of this while doing the PiVR measurements. This timestamp
is called presentation timestamps (**PTS**). (More info `here
<https://picamera.readthedocs.io/en/release-1.13/_modules/picamera/frames.html?highlight=timestamp##>`__).

.. _Benchmark_loop_time:

PiVR loop time
==============

PiVR loop time has been tested by recording the PTS while doing
real-time tracking at 640x480 resolution and a variety of framerates
using a Raspberry Pi 3.

.. figure:: Figures/benchmarking/1_loop_time_640x480.jpg
  :width: 100 %
  :alt: 1_loop_time_640x480.jpg

For example at 30fps each camera timestamp should be 1/30 = 0.033s
which of course is 33ms.
The reported timestamps are perfect for 30 and 40fps. At 50, 60 and
70fps we start loosing a few frames (6/5,598, 25/7,198 and 16/8,398,
respectively) meaning PiVR could not keep up with the images coming
in therefore the loop time was larger than 1/fps.

For more detail please see `Timing Performance of PiVR
<https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000712#sec025>`__

.. _Benchmark_latency:

PiVR latency
=============

To measure latency we turned the LED on at a defined frame and asked
how long it takes for PiVR to detect the change in light intensity.

This was based on the excellent suggestion of `Reviewer #1
<https://journals.plos.org/plosbiology/article/peerReview?id=10.1371/journal.pbio.3000712>`__:

#. Run closed loop tracking at 70fps at 640x480 with a facsimile larva.
#. At each 50th frame of a trial (and multiples thereof), turn the red
   LED on.
#. Take the median of a set of 10 pixels in the search box and compare
   the average current intensity to the intensity of the previous frame.
#. If this difference is larger than 2, consider the LED “ON” and
   immediately turn the LED “OFF”.
#. If the LED has been turned “OFF”in the previous frame, turn the LED
   “ON”again.

The code used for this can be found on the Gitlab repository of PiVR
`Gitlab repository of PiVR <https://gitlab.com/LouisLab/pivr>`__ in
branch ‘LED_flash_test’ in the file ‘fast_tracking.py’ with
`relevant code going from line 1457 to 1584
<https://gitlab.com/LouisLab/pivr/-/blob/LED_flash_test/PiVR/fast_tracking.py>`__.

Using the methodology described above, we found that the detection of a
‘LED ON’ event was either detected during the earliest possible frame or
one frame later. This difference in latency was dependent on the region
of the image (region of interest, ROI) used to compare the pixel
intensity. The detection latency was systematically longer when the
ROI was in the top (green box) as compared to the bottom of the image
(magenta box).

.. figure:: Figures/benchmarking/2_latency_70fps_640x480.jpg
  :width: 100 %
  :alt: 2_latency_70fps_640x480.jpg


As illustrated in the figure below, this can be explained by the fact
that turning the LED “ON” while the camera is grabbing the frame through
the rolling shutter cannot lead to a successful detection of the
change in light intensity. During frame #50 the LED is still “OFF” in
the top part of the image whereas it is “ON” in the lower part of
the image.

.. figure:: Figures/benchmarking/3_rolling_shutter_effect.png
  :width: 100 %
  :alt: 3_rolling_shutter_effect.png

As shown above the number of frames between two consecutive events
corresponding to the LED being turned “ON” was 3 frames if the ROI was
located in the top part of the image (green box). The schematic
diagram below outlines our interpretation of this 3-frame
delay: The LED is being turned “ON” during frame (j) while the frame is
being recorded (vertical arrow in frame j + 1). If the PiVR algorithm
is set to monitor a ROI in the upper part of the image, this ROI is
read before the LED intensity has switched to the “ON” state. Therefore,
the LED “ON”event is not observed during frame j. Instead, it is
detected during processing of frame j + 1. This delay happens again
when the second LED “ON” event is detected in frame j + 4.

.. figure:: Figures/benchmarking/4_top_diagram.jpg
  :width: 100 %
  :alt: 4_top_diagram.jpg

By contrast, if PiVR is set to monitor a ROI in the lower part of the
image, the LED turn “ON” event can be detected while processing frame j
+ 1
(see below).

.. figure:: Figures/benchmarking/5_bottom_diagram.jpg
  :width: 100 %
  :alt: 5_bottom_diagram.jpg

Consequently, the time that elapses between the two LED “ON” events is
either 5 frames (or 71ms at 70 Hz) for a ROI in the top part of the
image or 2 frames for a ROI in the bottom part of the image.

Based on these observations, we conclude that the LED “ON” event must
be happening **while** the image is being captured, which corresponds
to a duration of **1.34ms**. The longest possible latency between the
action of an animal (in this example during frame j) and the actuation
of the update of the LED is therefore 2 frames + 1.34ms
(time window associated with frame j + 2), which is equivalent to less
than 30 ms (frame duration at 70 Hz is **14.29 ms**.

.. _PiVR_loop_time_high_res:

PiVR loop time (High Res)
==========================

PiVR v1.5.0  was released on 27th of March, 2021. It allows users to
perform :ref:`online tracking <TrackingLabel>` with resolutions other
than 640x480.

Higher resolutions impact the :ref:`closed loop times <Concepts>` as
a larger image is being analyzed to define the animal in each frame.

We tested the performance using both a
`Raspberry Pi 3 Model B Plus Rev 1.3
<https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus/>`__
and the newer and more powerful
`Raspberry Pi 4 Model B Rev 1.2
<https://www.raspberrypi.org/products/raspberry-pi-4-model-b/>`__

The **Raspberry Pi 3** was able to handle 30fps without dropping frames
at 1024x768. At 1296x972 it was only able to handle 15fps without
dropping frames.

.. figure:: Figures/benchmarking/6_RPi3_Image_processing_time.jpg
  :width: 100 %
  :alt: 6_RPi3_Image_processing_time.jpg

The `50% more powerful
<https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711/README.md>`__
**Raspberry Pi 4** on the other hand was able to handle 40fps at
1024x768 and 30fps at 1296x972:

.. figure:: Figures/benchmarking/7_RPi4_Image_processing_time.jpg
  :width: 100 %
  :alt: 7_RPi4_Image_processing_time.jpg