Introduction
Following the initial surgery for hypoplastic left heart syndrome, home monitoring programmes are used to reduce interstage mortality. Several studies have shown that home monitoring programmes are associated with significant reduction in the risk of mortality, from 15 to 0%. Reference Ghanayem, Hoffman and Mussatto1–Reference Hansen, Furck and Petko4 The home monitoring programmes typically include close monitoring of weight and oxygen saturations. Pulse oximeters are used as a component of the home monitoring programmes to detect changes in oxygen saturations. Families measure the oxygen saturation daily, manually log the values, and then convey them to the nurse practitioner, who calls them once a week. Trends in the readings or major acute changes prompt medical evaluation and treatment. Inability to report this data on time can result in poor outcomes, including death. Reference Hansen, Furck and Petko4 In multiple other diseases, including diabetes and asthma, digital tools that can make automatic measurements have been shown to improve outcomes. Reference Kvedar, Fogel and Elenko5
As a part of the home monitoring programme, families are sent home with hospital-grade pulse oximeters, often determined by their health insurance. These devices are FDA-approved for use in adults and children with saturations between 70 and 100%. However, despite their widespread use, FDA-approved hospital-grade pulse oximeters are found to have significant inaccuracies in hypoxemic patients. Reference Ross, Newth and Khemani6 In addition they are relatively bulky and expensive.
As technologies continue to improve, pulse oximeters are becoming more compact, affordable, and portable, and FDA-approved devices are available to the public on the consumer market. These new devices also have the same standards as the current hospital-grade pulse oximeters used for monitoring hypoxemic infants. In addition they are often equipped with Bluetooth, enabling data to be remotely monitored. We hypothesized that these devices would be at least as accurate as current hospital-grade pulse oximeters and could be used for home monitoring. We believe the smaller devices will be more convenient for the families, improve family adherence to the home monitoring programme, and provide easier data accessibility for the clinical team.
The aim of this study was to evaluate the accuracy of an FDA-approved portable pulse oximeter, the WristOx2 3150 with infant sensors 8008J (Nonin Inc., Plymouth, Minnesota, United States of America) and the hospital-grade pulse oximeter used in our institution, the Masimo LCNS (Masimo Inc., Irvine, California, United States of America) compared to the gold standard of arterial oxygen saturation measured by co-oximetry. Equivalence was defined as an absolute difference of 5% or less between the pulse oximeter reading and the co-oximeter measurement. Reference Kemper, Mahle and Martin7,Reference Schroeder, Marmor and Pantell8
Materials and methods
Patients and data collection
This institutional review board– approved, investigator-initiated prospective, observational study was performed at Lucile Packard Children’s Hospital, Stanford, in the paediatric cardiovascular ICU and the neonatal ICU between April, 2014 and August, 2015. Following written parental consent, infants and children weighing 2–20 kg with saturations at or below 90% (measured by pulse oximetry at the time of assessment) with an arterial line in place were enrolled in the study. Exclusion criteria were unstable clinical status resulting in marked fluctuation of saturations or anatomy, which caused differential saturations between extremities (e.g. neonatal aortic coarctation, patent ductus arteriosus with right-to-left shunting).
All patients already had an indwelling arterial line that could be used to obtain an arterial blood sample and a hospital-grade pulse oximeter – Masimo LCNS saturation sensor connected to Philips monitor (hospital device) – in place as part of their clinical care. That pulse oximeter was left in the location it had been placed for clinical care. Study time was coordinated with the time of a blood draw needed for clinical care. The WristOx2 3150 with infant sensors 8008J (study device) was placed on a thumb, big toe, or mid-foot of an extremity that did not have the standard pulse oximeter (to avoid interference) nor an arterial line to avoid issues of hypo-perfusion-related artifact. Saturations were recorded continuously by the two devices, at a point of stability, for at least 2 minutes following which 0.2 mL of an arterial blood sample was obtained for bedside co-oximetry and the sample was analysed immediately. Study device recorded for a minimum of 2 minutes prior to when blood sample was obtained. After the blood sample was obtained, the study device was removed from the patient. Data from the hospital device were collected manually in an Excel file, and data from the study device were obtained using the N-vision software, which allowed for exporting data into an Excel file.
Statistical analysis
Pulse oximetry with WristOx2 3150 with infant sensors 8008J (study device) and Masimo LCNS saturation sensor connected to Philips monitor (hospital device) were measured simultaneously and compared to arterial oxyhemoglobin saturation measured by co-oximetry. Statistical analysis with Schuirmann’s TOST equivalence tests, consisting of two one-sided paired t-tests, was performed to evaluate the performance of both the hospital and study pulse oximeter devices compared to co-oximetry. Equivalence was defined as an absolute difference of 5% saturation or less. Based on past data and clinical relevance, 5% was chosen as an acceptable limit. Reference Harris, Char and Feinstein9
The difference between study sensor bias and hospital sensor bias was evaluated with a signed rank test, a non-parametric test used due to non-normally distributed data.
Accuracy root mean square was calculated for each device as compared to co-oximetry using the formula
${\rm{Accuracy\ root\ mean\ square}} = \sqrt {{{\mathop \sum \nolimits_{i = 1}^N {{(Sp{O_{2i}} - Sa{O_{2i}})}^2}} \over n}}$
, where SpO2i = estimated oxygen saturation (pulse oximetry) of infant i, SaO2i = arterial blood oxygen saturation using co-oximetry of infant i, i = 1 to 24 infants, and n = 24, as defined by Ross et al., 2014.
Reference Ross, Newth and Khemani6
As a guide, accuracy root mean square ≤ 3 is required by the FDA for approval of a pulse oximeter.
10
Modified Bland–Altman plots depict gold standard co-oximetry measurements plotted against the bias for each device. For each sensor (hospital and study sensor), association between equivalence with arterial co-oximetry (within 5% yes/no) and the following factors were tested: age, using Wilcoxon rank sum test due to non-normal distribution; Massey score, using t-test; and sensor placement (finger, toe, or thumb vs hand or foot), using Fisher’s exact test.
Results
Twenty-four infants were included in the analysis. Two patients were excluded from data analysis after the study was completed because the pulse oximeter had been placed on the same limb as the arterial line. The median age of the patients was 13 days (2–195 days), mean weight was 3.9 kg (2.08–6.85 kg), and the most common diagnosis was hypoplastic left heart syndrome (Table 1). Sensor placement for the hospital device was not changed for the purposes of the study as it was being used for clinical care; notably many sensors had been placed in a non-standard position on the hand or foot (Table 2). The mean saturation by co-oximetry was 72% (standard deviation (SD) ± 9%). Mean saturation measured by the study device was 76% (SD ± 8%). Mean saturation measured by the hospital device was 80% (SD ± 8%).
Table 1. Patient characteristics, n = 24.
SpO2 = oxygen saturation; HLHS = hypoplastic left heart syndrome; DORV = double outlet right ventricle; TGA = transposition of the great arteries; TAPVR = total anomalous venous return; TET/PA/MAPCAs = tetralogy of fallot/pulmonary atresia/major aortopulmonary collateral artery.
SD = standard deviation.
Table 2. Location of sensors.
Neither the study (p = 0.16) nor hospital device (p = 0.97) met the predefined standard for equivalence of readings within 5% of co-oximeter. The study device reading was on average 4.0% higher than the co-oximeter (90% confidence interval (CI) 2.3–5.7%). Note that even though the mean bias of 4% was in the equivalence range, the confidence bounds extended beyond 5%. On average the hospital device was 7.4% higher than the co-oximeter (90% CI 5.4–9.5%) and also did not meet the predefined standard for equivalence (Table 3). The hospital sensor bias was significantly greater than study sensor bias (p = 0.018). About 54% of the study device sensor readings and 38% of hospital device sensor values were within 5% points of the co-oximeter measurements. The accuracy root mean square of the study device was 6 and that for the hospital device was 9. Both devices had increased bias with decreasing saturation, as shown graphically in the Bland–Altman plots (Fig 1a and b).
Table 3. Mean paired differences between SpO2 and SaO2.
O2 values are arterial blood oxygen saturation (SaO2), when measured using co-oximetry, and estimated oxygen saturation (SpO2) when measured using pulse-oximtery. CI = 90% confidence interval. p-values from Schuirmann’s TOST test for equivalence.
Figure 1. Top and bottom reference lines at + /− 5 depict equivalence bounds. ( a ) Bland–Altman plot of arterial co-oximeter versus study device saturation ( b ) Bland–Altman plot of arterial co-oximeter versus LCNS (hospital) device saturation.
Age and Massey skin colour score were not associated with the accuracy of pulse oximetry measurements for either of the devices. The hospital device had improved accuracy when placed on a thumb, finger, or toe, as recommended, compared to when placed on the hand or foot (62% agreement with co-oximetry versus 9% agreement with co-oximetry, p = 0.013, Table 4). The study device was placed only on the hand or foot once, so no comparison could be made based on location.
Table 4. Hospital sensor bias in standard versus non-standard position with acceptable bias being <= 5%.
Discussion
We found that the smaller, more portable, Bluetooth-enabled pulse oximeter, the WristOx2 3150 with infant sensors 8008J, performed at least as well as the larger hospital-grade pulse oximeter and Masimo LCNS saturation sensor. The pulse oximetry values from both the devices were compared to the gold standard co-oximetry values in hypoxemic infants with arterial oxygen saturations lower than 90%. On average, both the pulse oximeters overestimated the oxygen saturation in this patient population when compared to co-oximetry. We found that the study device, WristOx2 3150, had less bias from co-oximetry compared to the hospital device with a mean difference of 4.0 and 7.4%, respectively. From this, we conclude that the study device when appropriately placed on the finger, toe, or thumb is at least as accurate as the hospital device as currently used. However, it is concerning that the 90% CI of arterial oxygen saturation minus pulse oximeter saturation for both sensors crossed the threshold of 5%, which has been previously described as the clinically acceptable range for managing patients with cyanotic heart disease. Reference Harris, Char and Feinstein9 It is also notable that both of these FDA-approved devices did not meet the FDA accuracy root mean square requirement, which is ≤ 3%, when being used in this clinical environment. Further study is required to determine whether devices that more accurately measure peripheral saturations will be associated with improved survival and better neurodevelopmental outcomes in cyanotic patients.
While age was not associated with accuracy in this study, studies have found foetal haemoglobin (which is present at higher concentrations at younger ages) to cause inaccuracies in pulse oximetry. Reference Chan, Chan and Chan11 Although some studies have also found that skin pigment does increase the bias of pulse oximetry, particularly at lower saturations Reference Fouzas, Priftis and Anthracopoulos12,Reference Bickler, Feiner and Severinghaus13 in this study there was not a significant effect from the Massey skin colour scores.
Standard pulse oximeter sensor placement is on thumb, finger, or toe, and except for the neonatal probes (intended only for <3 kg) those are the only locations that have FDA approval for use. Despite approval for use being limited to those standard locations, in the hospital setting pulse oximeter probes are commonly placed in other locations. Non-standard placement of the hospital sensor on the hand or foot was correlated with less accurate pulse oximetry reading, corroborating previous findings. Reference Sedaghat-Yazdi, Torres and Fortuna14 Since there has not been official testing in these non-standard locations, it is not surprising that they are less accurate. It is a limitation of our study that the hospital device sensors were not all in standard locations, which would be the best way to directly compare the devices. However, the hospital devices were tested as they are used in the clinical setting, which increases the external validity of the results. Overall, these results support the findings that correct placement of the sensor is important in establishing accurate measurements in the hospital sensor. Given the importance of sensor placement, parent or caregiver education about sensor placement is extremely important as a part of home monitoring in single ventricle patients to allow for the most accurate readings.
One limitation of this study is that it is a single sample from each patient. Reproducibility should be addressed in the future with additional cohorts to provide the evidence to determine if the study device is more accurate than the currently used hospital-grade pulse oximeters.
In addition, there are many other portable, FDA-approved devices that were not specifically tested. It is unclear whether these results would also hold true for other devices, but we would expect similar results as long as they are used within their specifications. The Masimo WristOx2 3150 device was chosen because it was FDA-approved, small (and therefore portable), cheaper than hospital-grade devices and Bluetooth enabled to allow for automatic data transmission. In this study, we did not survey parent or physician satisfaction with the study device as they were not involved in the use (either applying or interpreting results) of this device.
Conclusion
Our results suggest that one of the small, FDA-approved, consumer pulse oximeters – the WristOx2 3150 with infant sensor – is a suitable alternative to the large and expensive hospital-grade pulse oximeters currently used for the home monitoring programme at our institution. Our data suggest that not only is the WristOx device as accurate as the hospital-grade device, but that it may also be more accurate when placed in standard position on the finger, toe, or thumb. The study device has many advantages. It is smaller, portable, cheaper, and Bluetooth-enabled. It allows reliable, efficient data transmission for home monitoring, and it is easier for families to manage. We recommend that FDA-approved, smaller portable pulse oximeters be considered for use in home monitoring programmes.
Overall, we found that there is a significant bias when using both study and hospital pulse oximeters in hypoxemic infants. This highlights the need for further improvements in pulse oximetry design for children with cyanotic CHD. While trends may be clinically useful for surveillance, physicians should consider obtaining confirmatory arterial blood gas analysis with co-oximetry when there is a concern about a patient’s clinical status as pulse oximetry measurements may be an overestimation of true arterial saturation.
Acknowledgements
None.
Financial Support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of Interest
There are no conflicts of interest to disclose.
Ethical Standards
Research study was IRB approved. Informed consent was obtained from the parents of the study participants. The privacy rights of human subjects were observed.
