People:

Collaborators

Concept

Facial weakness, limb weakness, limb ataxia, and abnormal eye movements are common signs of nervous system damage caused by stroke, traumatic brain injury, and neurodegenerative diseases. This project aims to develop an automated, accurate, quantitative, easily-deployable, and low-cost digital screening tool for neurological disease that would:

  1. identify neurological deficits quickly in the field;
  2. minimize inter-operator variability and operator error of standard neurological examination.

The expected outcomes of this project will augment standard neurology care in rural and underserved areas, expedit the diagnostics and treatment process, and provide more accurate diagnosis and triage of neurological patients.

To be specific, the system will evaluate for signs of neurological disease from a video of a standard patient examination. We aim to deploy the system on commonly available camera-enabled devices (e.g. laptop computers, tablets, phones, etc.), without the need for additional specialized hardware, and without the need to transfer image/video data from the device to servers (all processing to be performed on the device). Once developed, our system will provide real-time, clinically relevant, triaging information for cases when a neurological event happens and access to neurological expertise is limited, as illustrated below:

centered image

A user scenario.

Our pilot study demonstrates a prototype for facial deficit detection. The prototype has been implemented as a live, real time tool, able to run on modest hardware such as commonplace laptops. Demonstration of the BANDIT (brain attack and deficit identification tool) on a volunteer mimicking a normal examination, as well as left and right deficits are shown below:

Facial weakness detection demo.


Applications

Stroke

In the United States, stroke is a leading cause of death and disability with more than 795,000 people suffering a stroke every year. The mainstay treatment of acute ischemic stroke (AIS) is rapid reperfusion therapy with intravenous tissue plasminogen activator (tPA) and/or endovascular thrombectomy (EVT), which substantially decrease the odds of death or disability. Numerous studies have shown that reduction in time to treatment by as little as fifteen minutes significantly decreases mortality and disability in stroke patients. Recognition of stroke in the prehospital setting by emergency medical services (EMS) leads to earlier and more frequent treatment with acute stroke therapy. However, the identification of stroke by non-neurologists can be challenging, as illustrated by the finding that paramedics fail to detect stroke in high as 56% of patients when not using a diagnostic aid.

Traumatic brain injury

Traumatic Brain Injury (TBI) includes concussion, subconcussion, and most exposures to explosive blasts from improvised explosive devices. One study indicates that the prevalence of TBI among returning service members range from 15.2% to 22.8%, affecting as many as 320,000 troops . The delay in diagnosis increases the risk of developing Alzheimer's disease, Parkinson's disease, motor neuron disease, and stroke. For instance, a recent study demonstrated that TBI was associated with 10.21, 4.61, and 2.32 times greater risk of stroke at three months, one year and five years after the injury, respectively. Facial and limb weakness are symptoms of neurological deficit caused by traumatic brain injuries as well.

Other neurodegenerative diseases such as Multiple sclerosis and Bell’s Palsy are assoicated with common symptoms of limb weakness and facial weakness.


UVA Neruolgoical Deficit Dataset

Data acqusisition and verification

The database of stroke survivors includes the following neurological deficits: facial and limb weakness, limb ataxia, and abnormal ocular movements. The database also has healthy controls. Board-certified neurologists rated the dataset to provide a ground truth to the presence or absence of the neurological deficits previously mentioned. After verification, the performance of different rating groups and algorithms can be directly compared among each other.

Dataset examples

Our team is working on curating the acquired video examinations and creating the UVA Neruolgoical Deficit Dataset. Below shows some examples of facial weakness. Note that only the near-mouth region is kept based on the HIPAA compliance requirements.

The examples of cropped near-mouth region: left facial weakness, normal, and right facial weaknes from the dataset.


Preliminary results

We utilize the open source packages such as dlib and openpose to extract facial location or pose from examination movie frames. Once appropriately localized, features (e.g. optical flow, histogram of oriented gradients, local binary patterns, etc.) can be extracted and machine learning can be utilized to regress the correct label (diagnosis) of a given video examination, as shown below. The subsequent section presents our preliminary work on facial weakness, eye movement abnormalities, and limb weakness detection.

centered image

The overview of our system.

Facial weakness

Facial weakness is a common presenting sign of stroke. A central brain lesion such as a stroke will cause pathological, asymmetric weakness on the lower facial muscles on the contralateral side. To detect and quantify this neurological sign, we explored the shape-based features[4] , appearace-based features[5] or combined[6] for static images or videos. The live and real-time prototype has been implemented based on [7]. Below is the list of publications.


centered image

Camera-based Facial Weakness Analysis for Videos

Submit to IEEE Transactions on Biomedical Engineering

We propose a framework for facial weakness detection using a regular RGB camera. The framework models the temporal dynamics of both shape and appearance-based features of each target frame through a Bi-LSTM network. The system is evaluated on a "in-the-wild" video dataset that is verified by three board-certified neurologists. In addition, three emergency medical services personnels and three upper level residents also rated the dataset. We compare the evaluation of human raters with our algorithm. Experimental evaluation demonstrates that (1) the framework can provide the visualizable and interpretable results to illustrate how shape and appearance-based features are used; (2) the proposed algorithm achieves the accuracy, specificity, and sensitivity of 94.3%, 91.4%, and 95.7%, which are equivalent to paramedics. A prototype is implemented on a regular laptop to demonstrate the feasibility of our study as a proof-of-concept.


centered image

Facial Weakness Analysis and Quantification Of Static Images

To appear in IEEE Journal of Biomedical and Health Informatics

We experimentally evaluate the performance of multiple state-of-the-art landmark feature extraction methods for measuring facial weakness, showing that landmark-based methods can suffer from inaccuracies in face landmarks localization. Moreover, we demonstrated that the combination of landmarks and intensity features (specifically HoG-based features) produced the best results when compared to either landmarks or intensity features separately in our dataset. At end of the work, we visualize the maximum weight of the used linear SVM classifier and its corresponding orientation in each cell for the HoG features as well as the significant landmarks to show that the clinically meaningful pathology can be detected by our model.


centered image
centered image

F-DIT-V: An Automated Video Classification Tool for Facial Weakness Detection

2019 IEEE EMBS International Conference on Biomedical & Health Informatics

Assessing facial weakness relying solely on a single static image is somewhat limited such as dependence on the availability of a given static image. While a doctor, or trained technician, could potentially acquire such an image, it would be preferable if a simple video examination procedure could be used instead. Because the video examination contains more information (many frames) it could be potentially more robust in identifying the signs of deficit. Therefore, in this work we propose a framework to perform video classification for facial weakness detection. we propose an automated video classification detection tool, which exploits Histogram of Oriented Gradients (HOG) feature and a straightforward voting classifier to perform more accurate facial weakness detection for a given video than the LBPTOP method. Our current prototype for the proposed tool for facial deficit detection, called F-DIT-V, achieves a classification accuracy of 92.9%, precision of 93.6%, recall of 92.8%, and specificity of 94.2%, which is higher and more reliable compared with the LBPTOPand the LSTM, which are widely used in previous and current studies for activity recognition.

The prototype has been implemented as a live, real time tool, able to run on modest hardware such as commonplace laptops. Demonstration of the BANDIT (brain attack and deficit identification tool) on a volunteer mimicking a deficit are shown on the left.


centered image

Pathological facial weakness detection using computational image analysis

2018 IEEE 15th International Symposium on Biomedical Imaging

In our pilot study[4], we present an automatic pathological facial weakness detection tool based on a single RGB image. The proposed system was able to extract the facial landmarks and classify facial weakness using a learning method. The learning method projected the shape-based features onto a much lower-subspace. The low-dimensional representation not only greatly reduced the feature dimension and facilitate the computation but also produced the visualizable and interpretable results to understand facial weakness. However, this study also reported that the accuracy of facial landmark extraction approaches is insufficient in some cases, where the facial land-mark fails to delineate the shape of the mouth accurately.

Abnormal eye movement

centered image

BANDIT for Posterior Cerebral Artery Stroke

Identifying abnormal eye movement is another study of the Brain Attack Neurological Deficit Identification Tool for Posterior Cerebral Artery Stroke (BANDIT-PCS) funded by the American Heart Association (AHA). In this project we intend to develop a tool to detect neurological eye deficits by analyzing gaze movement. As the first step we have developed RoADIE (Rolling Apparatus to Detect Impairment of the Eyes). RoADIE is an apparatus that is equipped with various imaging and signal modalities to acquire patient data of a set of neuro-eye exams at the emergency room. RoADIE will be used primarily to collect gaze data related to eye movements while performing neuro-eye exams. Currently, we are in the midst of initiating the study to collect PCS data using our imaging and signal modalities.

Limb drift

centered image

Camera-based Limb Drift Detection

Pronator limb drift, or limb drift test, is one of the most sensitive signs of stroke indicative of muscle weakness in the arm. The acute stroke, focal cerebral hemisphere lesions, and multiple sclerosis, cervical spine injury will affect patient's ability to control the upper body, the balancing and coordination ability.

Coming soon...



Publications

[1] Uribe, Omar, et al. "Automated Detection of Facial Weakness Using Machine Learning." 2018 International Stroke Conference.
[2] Uribe, Omar, et al., American Neurological Association Annual Meeting, (2018).
[3] McDonald, Mark, et al. "Comparison of Human and Machine Learning Based Facial Weakness Detection." 2019 International Stroke Conference.
[4] Zhuang, Yan, et al. “Pathological facial weakness detection using computational image analysis,” 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI), IEEE, 2019.
[5] Zhuang, Yan, et al. "F-DIT-V: An Automated Video Classification Tool for Facial Weakness Detection." 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). IEEE, 2019.
[6] Zhuang, Yan, et al. "Facial Weakness Analysis and Quantification Of Static Images." IEEE Journal of Biomedical and Health Informatics (2020).
[7] Zhuang, Yan, et al. "Camera-based Facial Weakness Analysis for Videos." in press IEEE TBME (2021).