This article is the second in our article series titled “Wearable Devices for Lifestyle and Medical Monitoring”. Readers can access the other articles here:

Section 1: “General Problems in Existing Health-Care System and How Personicle Addresses These Gaps”

Section 2: “Details of Device Ecosystem”

Section 3: “How Personicle Addresses Technical Challenges in Leveraging Data from Wearable Devices”

There is a large variety of sensors available for monitoring and recording user lifestyle, health, and behavior. These are ubiquitous in today’s world, ranging from commercially available consumer-grade wearable devices to specialized clinical monitoring devices employed in Intensive Care Units (ICUs) in hospitals. A complete lifestyle and health navigation system needs to leverage the data generated by all such devices to get a complete picture of users’ life and health. We will discuss what are the different categories of such devices, how these devices manage and communicate the data, and what architectural components we are employing to efficiently collect and utilize this data.

The sensor ecosystem consists of different major categories of devices. There are three integral components to these devices across the categories; 1. sensing and data collection hardware, 2. communication devices that convey data to a remote center, and 3. data analysis tools or techniques used in extracting and analyzing important health information for the purpose of health and wellness (source). Sensors, actuators, and software are connected to the cloud which allows for real-time collection, analysis, and transmission of personal health data. Cloud servers and infrastructure are forwarded data from wearable devices through wireless connectivity via gateways and a firewall. Cloud servers also offer data storage, processing, and analytical models. A cloud-based web application manages the sensory input data on the Cloud which hosts the business’s logic and control applications, as well as user interfaces accessible from the patients, medical staff, medical device technicians, and administrators. The user interfaces are a way for those who are generating the data or who require the data in their work to be collected and analyzed from wearables and sends instructions and prompts to the wearable device. This data can be integrated with the Electronic Health Record (EHR), thus giving users a sweeping view of patient medical history.

Different wearable devices serve different purposes including health condition monitoring, patient therapy delivery, patient rehabilitation, early disease diagnostics, and activity tracking. Health condition monitoring employs wearable spirometers, hemodynamic or pressure monitoring devices, and electrocardiogram (ECG) patches. Patient therapy delivery employs devices that work alongside or are implanted into the bodily system of the patient and include pacemakers, defibrillators, closed-looped pancreas systems, and insulin pumps. For patient rehabilitation, the use of smart gloves, range-of-motion assessment sensors, body temperature, and respiration sensors, and electromyography (EMG) sensors. Early disease diagnostics involves the use of wearables that can give insights into a patient’s health state and data trends that can alert the user or physicians to possible health challenges that may arise; these include skin temperature and perspiration sensors, heart rate monitors, glucose monitors, and Holter monitors which record the electrical activity of the heart continuously over a 24 hour period. Finally, devices for activity tracking include wearable activity trackers, smartwatches, perspiration sensors, and skin temperature sensors.

Types of sensors

Consumer wearable devices

Consumer wearable devices include devices that are commercially available to the public, activity tracking devices like Garmin and Fitbit, sleep tracking devices like Oura, and lifestyle tracking devices like Apple Watch and Amazon Halo. These primarily utilize on-device accelerometers, GPS, altitude sensors, and optical (PPG sensor) heart rate monitoring. Garmin fitness devices record each activity with summary data and a tracklog. Each point recorded in the track log includes details about your activity (e.g. heart rate, cadence, location, and other relevant data). All Fitbit devices tracks steps taken, active zone minutes, active minutes, and other statistics of choice (sleep, exercise, water and food intake, weight, heart rate, hourly activity, distance, floors, and menstrual health)(source).

Sleep tracking and stress monitoring

Sleep tracking and stress monitoring devices include Oura, specialized Garmin devices, and other watches that measure heart rate variability. The Oura ring uses data points along with an accelerometer to detect movement and track your sleep activity, transmitting your results to an accompanying app via Bluetooth. The ring also includes two sensors that measure your skin temperature(source). The downsides of Oura are mainly regarding the ability to detect certain sleep events accurately: 96% for sleep detection, 48% for wakefulness detection, 65% for light sleep detection, 51% for deep sleep detection, and 61% for REM sleep detection relative to PSG(source).

Lifestyle tracking

Lifestyle tracking is usually undertaken by devices such as Amazon Halo and Apple Watch. The Amazon Halo uses microphones that listen to the user’s voice and determine their emotional state. The microphone is not for voice commands, but the user must opt-in for the band to track their emotional state, or the Amazon Halo won’t do anything at all(source). The microphone uses machine learning to determine whether the user sounds positive, negative optimistic, energetic, fatigued, irritated, and a variety of other moods. The Amazon Halo tracks 4 main types of information: activity, body fat percentage, tone of voice, and sleep. Duration of movement, time spent being sedentary, and intensity of movement are all tracked automatically throughout the day. In contrast to the Amazon Halo, the Apple Watch tracks mostly sleep duration and actively engages in behavioral modification for better sleep. Things, like setting a bedtime and rising time and spending less time on smart devices, can drastically improve patient sleep and restfulness. It actively works with the user in creating a sleep schedule, tracking sleep, measuring sleeping respiratory rate, and showing visualizations of sleep trends. The Heart Rate app on Apple Watch can be enabled to send alerts about whether the user’s heart rate is too low or too high based on a user-set heart rate threshold. There is also a feature for irregular heart rhythmic notifications which can be used to identify Atrial fibrillation. The Mindfulness app also includes a Reflect feature that prompts the user to engage in mindful practices using written prompts and animations that relax the psyche. The Apple Watch app on iPhone also has a Handwashing feature that pushes the user to wash their hands for 20 seconds as well as informs the user if they haven’t washed their hands within minutes of returning home. Finally, women can track their menstrual cycles using the Cycle Tracking app which logs information about their menstrual cycle daily and uses the data collected to populate predictions on periods and fertility windows, further improving prediction accuracy using heart rate data.

Clinical medical devices

Clinical medical devices include continuous glucose monitors and electroencephalography (EEG)s used in sleep studies (polysomnography). Doctors prescribe these devices to patients and look at the data they generate to get insights into the progression of the disease. As defined by Diabetes Canada, a continuous glucose monitor (CGM) is “a wearable device that tracks blood glucose (sugar) every few minutes, throughout the day and night. The readings are relayed in real-time to a device which can be read by the patient, caregiver, or health-care provider, even remotely”(source). It can be worn by a patient in real-time and can help patients prevent patients from experiencing low glucose at night. The device, which uses a sensor to measure glucose levels continuously at minute intervals, sends this data to a wireless transmitter. It can then be connected to a smartphone or other devices.

An electroencephalogram (EEG) is a test that measures the electrical activity in the brain. Brain cell activity during both waking and sleeping hours allows continuous communication through electrical impulses. It is performed using small metal electrodes that are attached to the scalp (source). The results show varying brain waves on a recording. It is used to diagnose brain disorders. Doctors can order 2 distinct types of EEG: (1) routine, and (2) sleep deprived. Sleep deprived EEGs require patients to stay up the night prior to their EEG. EEGs are used for sleep tracking.

Additional wearable medical devices such as electrode patches, vital sign devices, drug delivery system, heart rate monitoring devices, and vision correction systems are employed by practitioners to gain insights on patient health, well-being, and chronic disease risk.

Electrode patches are used for electromyogram (EMG), electrocardiogram (ECG), and electrooculography (EOG). They work by sketching the outline of a patient’s vital signs and transmit that data to smartwatches and smartphones. Biometric feedback from smartwatches is then used to promote healthy lifestyle choices to patients.

Fitness devices include wristband monitors, chest strap monitors, and ear clip monitors. Wristband monitors contain a sensor that detects body movement which vibrates every time the patient exceeds their daily goals(source). The Apple watch is a great example of this, having on-screen displays for daily steps taken, notifications, and viewing time. Chest-strap monitors have the functionality of tracking heart rates by being worn around a patient’s chest. Ear clip monitors send biometric data to smartphones and other wearables through Bluetooth.

Vital Sign Devices are the types of wearables that can track and collect data regarding vital signs over a long period of time – metrics such as heart rate, body temperature, and blood pressure. A downside to these devices is that they lack the functionality of being able to transmit data to receivers. These devices are often used in hospitals because the use of the data necessitates that healthcare workers collect and analyze the biometric data manually. The wearable type monitor includes a flexible thermometer strip, an inflatable bladder on a wearable cuff, or a pulse oximeter sensor that measures the patient’s oxygen levels by focusing on skin color changes.

Portable drug delivery devices have been introduced to improve medication compliance for patients who need to take multiple medications at various times. Medical devices have an integrated pump that releases a specific dose of a specific drug from the skin at regular intervals. These intelligent drug delivery systems ensure accurate control of dose distribution and dosing time while tracking the number of doses a patient receives. They also prevent underdosing or overdosing by making sure medication is administered regularly and in specific amounts. Combing insulin pumps and continuous glucose monitors creates a sort of “artificial pancreas” that acts as a closed cybernetic loop for drug delivery. This highlights how user-generated data can be leveraged to provide continuous health navigation. Patients with Type 1 Diabetes use this system to mimic the natural system in our body which triggers the release of insulin or glycogen upon detection of glucose in the blood by pancreatic islet cells. The insulin pump is used for insulin delivery, the blood glucose monitor or CGM is used for assessing blood glucose level, and the calculator calculates the dose required and delivers it using an algorithm which adjusts basal insulin (as required) at minute intervals. Mealtime involves more user input, requiring users to input their carbohydrate consumption. Based on this information, the calculator determines the insulin necessary and gives instructions on dosage requirements to the insulin pump(source).

The heart rate monitor constantly monitors the patient’s heartbeat to prevent a heart attack. There are four main heart rate monitors: one-lead ECG, disposable ECG sensor, wireless ECGs, and proprietary wearables(source). One-lead ECG machines contain several ECG sensors that can be worn on the patient’s arms and chest. It records ECG waves on paper or digital reports and analyzes them. Disposable ECG sensors have 2 electrodes that adhere to the skin and a 3rd electrode that connects to a smartphone, tablet, laptop, or other device that is used to record the output. The recording device wirelessly transmits data to hospitals through Bluetooth, and the patient can be observed real-time during heart rate monitoring exercise. Finally, wireless ECGs send information to laptops, smartphones, or cloud storage systems so that professionals and family members can review it on demand.

By focusing an image at the desired point in front of the wearer’s eyes, contact lenses and glasses can effectively correct blurred vision. A new class of high-tech wearable glasses, a type of wearable device called Ocular Lens Displays (OLDs), can provide basic vision correction working with wearable medical device technologies. Ocular Lens Displays remove the need for eyeglasses by having in-built glasses that project images directly onto the retina.

Devices under research

There are also new devices that are currently under research that are being used to collect food-related data (nutrition recognition, eating moment recognition), mood and mental state, sleep-related data, and blood pressure or blood glucose data.

Food related

There are several data streams that are useful for monitoring food-related events such as image and video-based food recognition, sound-based eating moment recognition, accelerometer-based hand-to-mouth gesture recognition, and IoT tooth-mounted sensor. Many of body-worn sensors for monitoring of food intake rely on food intake detection through detection of hand-to-mouth gestures [source], chewing [source], [source] and swallowing [source], [source]. Use of inertial sensors for hand to mouth gesture recognition was proposed in [source]. For monitoring of chewing or mastication, acoustic [source], proximity [source], and strain sensors [source], [source] have been proposed in the literature. For monitoring of swallowing, similar acoustic [source], strain [source], and bio-impedance measurement [source] sensors have been proposed. The results obtained with these body-worn devices are promising in terms of automatic detection of food intake. However, one common limitation of these devices is the need for their attachment to the body at different regions of the head e.g. on the larynx [source], [source]; in ear [sourcev], [source]; or on the temporalis muscle [source], [source], [source].

There are several image and video-based food and nutrition recognition applications that are good for research applications but need to be refined for commercial applications. Food image recognition systems are used to estimate food quantity ingested as well as determining dietary constituents. The issue with these systems is that they are difficult to implement and are unsuccessful at accurately classifying different foods. To overcome this issue, deep convolution neural networks such as NutriNet and Food AI have been implemented. NutriNet was tuned on a recognition dataset containing 225,953 512 × 512 pixel images of 520 different dishes and beverages from different food groups, on which we achieved a classification accuracy of 86.72%, along with an accuracy of 94.47% on a detection dataset containing 130,517 images(source).

FoodAI uses deep-learning food recognition that can identify 756 different classes of food to build a smart food logging system. Within the classes include main courses, beverages, and snacks. This approach beats the constraints of traditional food logging procedures and ensures systematic and effective logging. It consists of an API services, web-interface, a deep neural network model within the inference engine, and database that houses user-captured food images and images collected by FoodAI from the internet through a process called” web data crawling.” Users can access the API for food recognition and write daily food logs in their own diet journal. During mealtime, the user snaps a picture of their food and logs it. This enables users to watch trends in their overall eating habits over time. Based on a trained validation set, it achieved over 83% top-1 accuracy and 95% top-5 accuracy(source).

Sound-based eating moment recognition typically uses microphones to infer chewing or mastication sounds. These microphones place sounds in categories and are recorded during mealtimes. One such microphone records sounds using non-invasive microphones that are positioned in the outer ear canal (source). A database involving 51 participants enjoying 7 types of food and 1 drink was developed for algorithm development and model training. By comparing the signal energy detected in the inner ear to sounds in the environment, food consumption is logged. Single chew or swallowing events are determined using Hidden Markov models. Results indicated a food classification accuracy (79%) and detection accuracy (83%) on a test set including 10% of all data collected. With this approach, there is currently no way of monitoring the time or the incident of eating.

Food recognition can also employ accelerometers for hand-to-mouth gesture recognition. One such system uses a wrist-worn tri-axial accelerometer to recognize ingestion of food and beverages. It works by using adaptive segmentation to categorize different gestures picked up by the accelerometer during mealtimes. The data picked up by the accelerometer is continuous. A type of study called a “posteriori”, which utilizes Convolutional Neural Networks to recognize different eating and drinking gestures is performed. To achieve this, 3 time-series to image encoding frameworks (the signal spectrogram, Markov Transition Field, and Gramian Angular Field) are used in combination with different multi-input multi-domain networks(source). The gestures are then classified into 3 categories: ”Eat”, ”Drink”, and ”Null”, where null includes all unidentifiable gestures included in the set following segmentation. A mean accuracy of 97.7% was achieved per class.

Upper limb mounted motion sensors, another system used to evaluate hand-to-mouth gestures, are advantageous as compared to sensors mounted to a user’s head or neck. These include a combination of inertial sensors (accelerometers and gyroscopes) as well as proximity sensors (RFID) mounted to the upper limb. These sensors are mainly advantageous when used alongside traditional methods for conducting dietary assessments and supporting dietary change interventions. Examples include asking for the user to take pictures of their meal upon eating moment detection and supporting certain hand-to-mouth gestures.

Sensors mounted on clothing or daily items are another type of accelerometer-based detection system which are used to determine food intake without reliance on self-reporting from patients. Integrating sensors into items used or worn daily can eliminate the limitation of having the accelerometers mounted on different areas of the head. Improper mounting to the required areas on the body, including the ear, the larynx, or the temporalis muscle can severely affect the data collected in a negative way. Eyeglass-mounted sensors are feasible alternatives to wearing sensors directly on the skin for 64% of the U.S. populace.

Mood or mental state monitoring devices

Mood or mental state-related devices (also known as mood-forecasting devices) are also in the works and gaining popularity in an emerging new field. Mood forecasting devices are concerned with detecting suicidal thoughts or ideation, anxiety, sadness, emotional distress, emotional arousal, and psychological brain disorders such as major depressive disorder, mania, or bipolar disorder. Informing caregivers, loved ones, or health care professionals about oscillations in mood indicating concerning behavior can bode well in preventing suicide. Mood forecasting devices work by exploiting mind-body links. Changes in one’s state of mind and mental well-being can have a clear and detectable effect on the body which can be monitored. Biomarkers such as changes in heart rate, movement, perspiration, or skin temperature. Heart rate is the best emotional biomarker and an elevated heart rate is indicative of stress. Reduced movement or sleep can also be indicative of depression. Finally, skin temperature and perspiration are indicative of emotional arousal and stress (source). Sensors that can be involved in the process of collecting such data include those that collect pulse, skin temperature, or movement as well as information about how users interact with their smartphones informing a ”digital phenotype” which notes the texting rate, calling rate or posting rate on social media – correlated with a change in mood. More activity is present when happier, and less activity is present when sad.

Sleep monitoring

Sleep-related wearable technology currently under development is attempting to integrate newer advances in sensor technology such as low power consumption, low cost, connectivity, miniaturization, connectivity, and functionality of bio-sensors. These advances allow newer wearable devices to continuously record a broad range of bio-signals(source). The newer technology is more accurate than motion-based approaches which were used to determine if the user was awake or asleep in previous sleep tracker technology; due to newer technologies being multisensory there is greater resultant accuracy. It should be noted that the empirical validity of the multisensory approach to sleep regulation requires further probing. Skepticism and caution must be used to deter largely inaccurate data sets which can be scientifically misleading and result in drawing erroneous conclusions.

Blood pressure and blood glucose monitoring

Preventing diabetes requires adjustment in medication strength and observing trends in hypoglycemia in patients through blood glucose monitoring. There are studies looking at the general adherence of patients in systematically monitoring their own blood glucose using non-invasive devices, which significantly relieve them of certain burdens (pain, potential infections) associated with the commonplace devices that are being currently used such as blood glucose monitors. Non-invasive devices differ from invasive devices in that the former uses various types of energy detection, whereas the latter uses biochemical analysis involving fluid samples taken from patients. Current blood glucose monitoring devices require invasive procedures that are often painful and expensive to acquire. Non-invasive devices that are currently being developed for accurately assessing blood glucose levels of individuals are breath (Acetone Detection), finger detection (Metabolic Heat Confirmation (MHC), near-infrared spectroscopy (NIR), mid-infrared spectroscopy (MIRS)), Ear Lobe (Ultrasound, Magnetic Sensing, Thermal Detection combined), Hand sensing (Millimeter and Microwave sensing), and Tears (Optical Polarimetry System, Retina Pigmentation, Enzymatic Detection Technology)(source).