Real-time neural network-based fluorescence lifetime imaging with SPAD sensors

Abstract

Fluorescence lifetime imaging (FLI) is a powerful imaging technique in biological and medical research, which is receiving increased attention for clinical applications in recent years. However, existing FLI systems often suffer from the tradeoff among processing speed, accuracy, and robustness. Herein, we propose a recurrent neural network (RNN)-based FLI system that accurately estimates the fluorescence lifetime, from the raw timestamps instead of histograms, on the fly. We train variants of RNNs on a synthetic dataset and compare them to center-of-mass method (CMM) and least squares fitting (LSF). The results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LSF in terms of accuracy and outperform CMM and LSF by a large margin in the presence of background noise. Besides, the Cramer-Rao lower bound analysis shows that the RNN models are close to the theoretical optima. We build a FLI microscope setup for evaluation, utilizing Piccolo, a 32x32 SPAD sensor developed by our lab. Four quantized GRU cores, capable of processing 4 million photons per second, are deployed on a Xilinx Kintex-7 FPGA. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images up to 10 frames per second. With these advantages, the proposed FLI system is promising for many important biomedical applications ranging from biological imaging of fast-moving cells to fluorescence-assisted diagnosis and surgeries.

Yang Lin

Doctoral Assistant

My research interests include machine learning, fluorescence lifetime imaging, and single-photon avalanche diode.