Facial weakness, limb weakness, limb ataxia, and abnormal eye movements are common signs of nervous system damage caused by stroke, traumatic brain injury, and neurodegenerative diseases. This project aims to develop an automated, accurate, quantitative, easily-deployable, and low-cost digital screening tool for neurological disease that would:
The expected outcomes of this project will augment standard neurology care in rural and underserved areas, expedit the diagnostics and treatment process, and provide more accurate diagnosis and triage of neurological patients.
To be specific, the system will evaluate for signs of neurological disease from a video of a standard patient examination. We aim to deploy the system on commonly available camera-enabled devices (e.g. laptop computers, tablets, phones, etc.), without the need for additional specialized hardware, and without the need to transfer image/video data from the device to servers (all processing to be performed on the device). Once developed, our system will provide real-time, clinically relevant, triaging information for cases when a neurological event happens and access to neurological expertise is limited, as illustrated below:
Our pilot study demonstrates a prototype for facial deficit detection. The prototype has been implemented as a live, real time tool, able to run on modest hardware such as commonplace laptops. Demonstration of the BANDIT (brain attack and deficit identification tool) on a volunteer mimicking a normal examination, as well as left and right deficits are shown below:
Facial weakness detection demo.
Other neurodegenerative diseases such as Multiple sclerosis and Bell’s Palsy are assoicated with common symptoms of limb weakness and facial weakness.
The database of stroke survivors includes the following neurological deficits: facial and limb weakness, limb ataxia, and abnormal ocular movements. The database also has healthy controls. Board-certified neurologists rated the dataset to provide a ground truth to the presence or absence of the neurological deficits previously mentioned. After verification, the performance of different rating groups and algorithms can be directly compared among each other.
Our team is working on curating the acquired video examinations and creating the UVA Neruolgoical Deficit Dataset. Below shows some examples of facial weakness. Note that only the near-mouth region is kept based on the HIPAA compliance requirements.
The examples of cropped near-mouth region: left facial weakness, normal, and right facial weaknes from the dataset.
We utilize the open source packages such as dlib and openpose to extract facial location or pose from examination movie frames. Once appropriately localized, features (e.g. optical flow, histogram of oriented gradients, local binary patterns, etc.) can be extracted and machine learning can be utilized to regress the correct label (diagnosis) of a given video examination, as shown below. The subsequent section presents our preliminary work on facial weakness, eye movement abnormalities, and limb weakness detection.
Facial weakness is a common presenting sign of stroke. A central brain lesion such as a stroke will cause pathological, asymmetric weakness on the lower facial muscles on the contralateral side. To detect and quantify this neurological sign, we explored the shape-based features , appearace-based features or combined for static images or videos. The live and real-time prototype has been implemented based on . Below is the list of publications.
We propose a framework for facial weakness detection using a regular RGB camera. The framework models the temporal dynamics of both shape and appearance-based features of each target frame through a Bi-LSTM network. The system is evaluated on a "in-the-wild" video dataset that is verified by three board-certified neurologists. In addition, three emergency medical services personnels and three upper level residents also rated the dataset. We compare the evaluation of human raters with our algorithm. Experimental evaluation demonstrates that (1) the framework can provide the visualizable and interpretable results to illustrate how shape and appearance-based features are used; (2) the proposed algorithm achieves the accuracy, specificity, and sensitivity of 94.3%, 91.4%, and 95.7%, which are equivalent to paramedics. A prototype is implemented on a regular laptop to demonstrate the feasibility of our study as a proof-of-concept.
We experimentally evaluate the performance of multiple state-of-the-art landmark feature extraction methods for measuring facial weakness, showing that landmark-based methods can suffer from inaccuracies in face landmarks localization. Moreover, we demonstrated that the combination of landmarks and intensity features (specifically HoG-based features) produced the best results when compared to either landmarks or intensity features separately in our dataset. At end of the work, we visualize the maximum weight of the used linear SVM classifier and its corresponding orientation in each cell for the HoG features as well as the significant landmarks to show that the clinically meaningful pathology can be detected by our model.
Assessing facial weakness relying solely on a single static image is somewhat limited such as dependence on the availability of a given static image. While a doctor, or trained technician, could potentially acquire such an image, it would be preferable if a simple video examination procedure could be used instead. Because the video examination contains more information (many frames) it could be potentially more robust in identifying the signs of deficit. Therefore, in this work we propose a framework to perform video classification for facial weakness detection. we propose an automated video classification detection tool, which exploits Histogram of Oriented Gradients (HOG) feature and a straightforward voting classifier to perform more accurate facial weakness detection for a given video than the LBPTOP method. Our current prototype for the proposed tool for facial deficit detection, called F-DIT-V, achieves a classification accuracy of 92.9%, precision of 93.6%, recall of 92.8%, and specificity of 94.2%, which is higher and more reliable compared with the LBPTOPand the LSTM, which are widely used in previous and current studies for activity recognition.
The prototype has been implemented as a live, real time tool, able to run on modest hardware such as commonplace laptops. Demonstration of the BANDIT (brain attack and deficit identification tool) on a volunteer mimicking a deficit are shown on the left.
In our pilot study, we present an automatic pathological facial weakness detection tool based on a single RGB image. The proposed system was able to extract the facial landmarks and classify facial weakness using a learning method. The learning method projected the shape-based features onto a much lower-subspace. The low-dimensional representation not only greatly reduced the feature dimension and facilitate the computation but also produced the visualizable and interpretable results to understand facial weakness. However, this study also reported that the accuracy of facial landmark extraction approaches is insufficient in some cases, where the facial land-mark fails to delineate the shape of the mouth accurately.
Identifying abnormal eye movement is another study of the Brain Attack Neurological Deficit Identification Tool for Posterior Cerebral Artery Stroke (BANDIT-PCS) funded by the American Heart Association (AHA). In this project we intend to develop a tool to detect neurological eye deficits by analyzing gaze movement. As the first step we have developed RoADIE (Rolling Apparatus to Detect Impairment of the Eyes). RoADIE is an apparatus that is equipped with various imaging and signal modalities to acquire patient data of a set of neuro-eye exams at the emergency room. RoADIE will be used primarily to collect gaze data related to eye movements while performing neuro-eye exams. Currently, we are in the midst of initiating the study to collect PCS data using our imaging and signal modalities.
Pronator limb drift, or limb drift test, is one of the most sensitive signs of stroke indicative of muscle weakness in the arm. The acute stroke, focal cerebral hemisphere lesions, and multiple sclerosis, cervical spine injury will affect patient's ability to control the upper body, the balancing and coordination ability.
 Uribe, Omar, et al. "Automated Detection of Facial Weakness Using Machine Learning." 2018 International Stroke Conference.
 Uribe, Omar, et al., American Neurological Association Annual Meeting, (2018).
 McDonald, Mark, et al. "Comparison of Human and Machine Learning Based Facial Weakness Detection." 2019 International Stroke Conference.
 Zhuang, Yan, et al. “Pathological facial weakness detection using computational image analysis,” 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI), IEEE, 2019.
 Zhuang, Yan, et al. "F-DIT-V: An Automated Video Classification Tool for Facial Weakness Detection." 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). IEEE, 2019.
 Zhuang, Yan, et al. "Facial Weakness Analysis and Quantification Of Static Images." IEEE Journal of Biomedical and Health Informatics (2020).
 Zhuang, Yan, et al. "Camera-based Facial Weakness Analysis for Videos." submitted IEEE TBME (2020).