Digital health technologies (DHTs) are rapidly emerging as an important part of health services, and may revolutionize health care. DHTs range from digital assistants for diagnostic devices, for example, AQUA Radiology's software platform that unifies all medical device quality assurance (1), electronic health databases, for example, MEDITECH (2), wearable sensors, for example, a wearable biosensor for measuring vital signs (3), robotics, for example, a telerobotic ultrasound (4), artificial intelligence (AI) solutions, for example, a machine learning engine for Prostate Cancer and other conditions (5), portable diagnostic technologies, for example, a mobile x-ray and ultrasound equipment (6), and mobile health apps, for example, a discharge care app (7). Assessing in advance the possible merit of validating and adopting DHTs by healthcare providers has not been rigorously addressed, although such assessment may be cost-effective.
This article reports an implemented methodology for validation and adoption of DHTs by healthcare providers. This methodology was established and tried at Thunder Bay Regional Health Sciences Centre (TBRHSC) and its Research Institute (TBRHRI) in Ontario, Canada, and then adapted by the Council of Academic Hospitals of Ontario (CAHO). CAHO, which is one of Ontario's Ministry of Health and Long-Term Care's selected Innovation Brokers, developed a system with one point of access for digital health vendors, making it easier for them to connect with CAHO's twenty-three member hospitals (8).
Opportunity for Industry
Digital health care, including smart phone apps, online solutions, and AI decision support software, may help streamline and improve access to and experience of effective health care. This industry may have an important role to play in two key foci of many healthcare providers around the world, that is, efficiency and effectiveness (reducing healthcare costs while providing better care such as for elderly people, respectively).
Healthcare spending is increasing at an unsustainable pace in all countries. In 2017, total health expenditure in Canada was expected to be approximately $242 billion, or $6,604 per person (9). Healthcare costs can be reduced by DHTs and related healthcare technology. For example, the Ontario Telemedicine Network offers patients in Ontario's north a way to avoid travelling long distances to reach specialists and other providers, thus reducing cost of travel (10).
Increased life expectancy in the last century has given rise to an increasing aging population (Reference Majumder, Aghayi and Noferesti11). The World Health Organization (WHO) reports that between 2015 and 2050, the proportion of the world's population aged over 60 years will nearly double from 12 percent to 22 percent (12). DHTs may enable continuous and remote monitoring of elderly people's health and wellbeing at lower costs (Reference Majumder, Aghayi and Noferesti11).
Widespread adoption and access to cyber tools (such as internet, cellular phones and computers, and cloud storage) provide increasing opportunities for the digital health industry to promote the use of health technologies at Point of Care and at home. It is suggestive that the number of mobile phone internet users in Canada is expected to increase from 22.8 million in 2016 to approximately 28.6 million by 2021 (13).
DHTs have been shown to be effective as tools in clinical research (14), prevention and self-management of chronic disease (Reference Lee, Choi and Lee15), diagnosis (6), treatment (Reference Ngwatu, Nsengiyumva and Oxlade16), and follow-up (7). Wearable and household monitors can transmit data to integrated health portals. Such portals can assist with health related decision making by analyzing data to suggest an actionable response to the appropriate type of healthcare provider and trigger interventions (Reference Gambhir, Ge and Vermesh17).
Guidance of Healthcare Providers
Building an organization of the future (networked, devolved, collaborative, project-based, and mobile) is one of the most important challenges for healthcare leaders. One of the key trends that is expected to impact the future of work is that of the advanced digital industry, robotics, AI, sensors and cognitive technologies, which is expected to help redesign almost every job (18).
CAHO has been appointed as one of three innovation brokers (IB) for the Province of Ontario by the Ministry of Health and Long-Term Care. CAHO represents Ontario's twenty-three research hospitals, which are a significant entryway for new innovations into the healthcare system, covering much of the full spectrum of health services. CAHO's IB (CAHO IB) work is supported by an Innovation Broker Task Force (CAHO IB TF), comprised of representatives from a range of disciplines across CAHO member hospitals. The CAHO IB TF has published a list of critical problems requiring innovative solutions to provide innovators with market intelligence on the needs of CAHO hospitals and the healthcare system in Ontario and beyond (19). The CAHO IB TF has also published a request for applications from innovators who are interested to validate their health technologies with CAHO member hospitals (8).
Innovators (mostly digital health technology companies) are now responding to the request for applications published by the CAHO IB. The CAHO IB initially screens the proposals from innovators and then forward them as batches to all of CAHO's member hospitals. Additionally, TBRHRI is also being contacted by industry innovators directly, who propose that TBRHRI/TBRHSC provide a validation test site for their technologies. Hence, a new challenge arose: how to decide which of the dozens of proposals from these many industry vendors to pursue; as digital health is an emerging sector, decision-making processes regarding its validation are not as well established as in other sectors such as pharmaceutical clinical research.
All CAHO member hospitals have a decision-making process for long-term projects such as purchasing a PET scanner, but many hospitals struggled to screen so many DHTs applications with an expected turnaround time of four weeks. To address the challenge of fast screening and turnaround times, the authors developed a specialized algorithm to help make that decision (Figure 1). This algorithm has been adapted by the CAHO IB for other hospitals.

Fig. 1. Innovation validation studies: decision tree.
According to our algorithm, proposals for validation studies or less mature projects such as pilots of innovative health technologies such as DHTs are accepted for further discussion with the vendor, or rejected, based on the following decision criteria: (i) Relevance to the research Strategic Plan of TBRHRI/TBRHSC; (ii) Alignment with current infrastructure at TBRHRI/TBRHSC; (iii) Availability of funds from the vendor or external grants; (iv) Immediate revenues expected for TBRHRI/TBRHSC from a given validation opportunity; (v) Availability of relevant expertise and required resources at TBRHRI/TBRHSC; and (vi) Expected experience to be gained by TBRHRI's Clinical Research Services Department (CRSD).
This algorithm and its associated decision criteria have resulted in positive responses to 10 percent of proposals and negative/deferred responses to 90 percent of proposals of a total of 40 proposals submitted by means of CAHO or directly to TBRHRI/TBRHSC; this may be regarded as an acceptable ratio of positive screening, considering that no more than 10 percent of start-ups succeed commercially (20).
Considering that the focus of the decision algorithm (Figure 1) was to prescreen applications within a short time frame (such as 4 weeks suggested by CAHO) by involving relevant but limited resources, the overly time-consuming quantitative evaluation criteria were not used at this stage of decision making. Many applicants have not yet studied their products in a clinical context to an appreciable extent. Thus, in the absence of direct clinical evidence it was not possible to apply any numerical evaluation criteria or use rigorous (qualitative and/or quantitative) evidence as selection criteria. The emphasis was placed more on process factors such as funding required versus availability of funds from vendors or potential grants, and clear relevance to the hospital's and institute's strategic plans. For example, TBRHRI/TBRHSC decided to explore the feasibility of a pilot with one vendor because there was an opportunity to apply for a collaborative government grant. The decision criterion, availability of funds from the vendor or external grants, was critical among others to move forward with this proposal (sufficient strengths in Figure 1). As an example of “Significant Weaknesses” (Figure 1), a vendor presented a text-based solution to communicate remotely with patients but it lacked, at that time, a needed capability of voice recognition.
The shotgun approach listed in Figure 1 brought a couple of limitations to attention so far; one proposal was deferred during a prescreen but was considered an important solution by a respective department when the department was involved at a later stage. Another limitation involved the lack of availability of appropriate staff to evaluate the proposal within a required time frame on a few occasions (such as due to vacation or busy schedules). These limitations do not invalidate the decision algorithm because any deferred proposal can be brought to the attention of appropriate professionals once further information is provided by the vendor or professionals are available to evaluate the proposal.
DHT proposals to healthcare providers are increasing especially due to programs like the CAHO IB (8). This workload requires additional resources or workload management strategies within an organization. Healthcare providers are encouraged to create or amend internal policies so that appropriate resources are dedicated to conduct a thorough review of the proposals and educate the healthcare workforce about the benefits of clinical validation, pilots and ultimate adoptions of DHTs. The presumed benefits include improved economic performance, better patient–caregiver relationships and patient-centered care.
Guidance for Industry
Governments and health systems continue to implement cost containment measures for reducing clinical and administrative waste and improving operational efficiency. One of the common tactics involves technology-assisted service provision and delivery methods, such as robots for drug dispensing, e-prescriptions, novel payment cards, patient administrative systems, electronic health records (EHRs), personal health records (PHRs), online healthcare provision, and telemedicine. Stakeholders are advocating the shift from a “break-fix” model of health care to one focused on prevention and the overall holistic health of populations rather than merely episodic and transaction-based treatments (Reference Ngwatu, Nsengiyumva and Oxlade16).
Although most DHTs such as wearables directly target patients in helping them manage their conditions as well as the general public in helping people adopt a healthy lifestyle and prevent chronic disease, the involvement of healthcare providers is still key such as in terms of collecting and analyzing the data to better monitor health parameters and, based on that, improve health care.
Ideally, innovation should respond to an identified health problem and should be tested by healthcare providers and service users. Even if the innovation has already been developed, it should still follow a route to target a particular problem identified by the healthcare provider. For example, key challenges of TBRHSC/RI are associated with their fairly isolated location in Northern Canada, serving remote communities with diverse cultures and increased rates of chronic diseases. This is especially true for indigenous people whose health is a key priority for TBRHSC/RI.
It is important for industry to know the unmet needs and strategic directions of healthcare providers. For example, TBRHSC is seeking technology-driven solutions which are most suited to address patient care in widely dispersed populations with a catchment area the geographical size of France. Health in the North, a report on geography and the health of people in Ontario's two Northern regions (10), notes that people in Northern Ontario have a shorter life expectancy than the average person in all of Ontario. This can ultimately be attributed to a heavy burden of chronic diseases, infectious diseases, mental health, and addictions in remote and Indigenous populations.
The strategic directions of TBRHSC/RI focus on patient-centered research in these priority areas. DHTs can play a significant role in addressing this health inequity in Northern Ontario. Stronger links between primary care (family physicians, nurse practitioners) and acute care (the hospital) can help. Thus, for TBRHSC, the most immediate advantages will be in the area of distance health and particularly indigenous health, overcoming barriers to care, including geographical and cultural barriers. So the chances that DHTs addressing any of these issues be adopted at TBRHSC/RI are higher compared to other DHTs not addressing such strategic directions of TBRHSC/RI.
It is also important for industry to know the decision-making criteria of healthcare providers. For example, TBRHSC must weigh the needs of the patient population in Northwestern Ontario as well as other considerations such as alignment with the TBRHSC's and TBRHRI's strategic plans (which include these needs as well as other factors), and the current infrastructure to support validation or piloting of DHTs. Each DHTs proposal accepted for collaboration by TBRHSC/RI has met one or more decision criteria.
In conclusion, DHTs are rapidly emerging, and their industry sector can play a key role in person-centered, evidence-informed, and socially responsible health care (Reference Rudnick21). DHTs will significantly improve patient care in hospitals, at home and elsewhere, keeping patients closer to home. We strongly recommend that industry innovators review need statements and decision criteria of healthcare organizations and systems such as the list of critical problems of CAHO's twenty-three member hospitals (19) and the decision-making algorithm provided by CAHO to its member hospitals (the generalizability of our approach cannot be concluded about at this stage; data to be collected by CAHO are expected to provide clarity about generalizability). The decision criteria established by TBRHSC/RI about its strategic direction and the availability of resources is proven practical in screening proposals from vendors of DHTs in a short time frame. This information can provide market intelligence for industry to better align its proposed innovations with the needs of healthcare organizations and ultimately the health needs of the population. This is expected to better serve patients and society at large.
Conflicts of Interest
The authors have nothing to disclose.