Introduction
The Scottish Health Technologies Group (SHTG), which is part of Healthcare Improvement Scotland (HIS), provides evidence support and advice to the National Health Service in Scotland (NHS Scotland) on the use of new and existing health technologies, which, although not medicines, are likely to have significant implications for people's care. The main decision-making body of the SHTG is the committee that the staff team support. The role of the committee is to advise and support the best use of health technology interventions within NHS Scotland. The committee is made up of representatives from NHS health boards, clinical and professional networks, academia, NHS Scotland National Procurement, and the Scottish government. Out of the thirty members, it also includes four volunteer public partners who represent a public view. All committee members are treated equally with the same access to meeting papers and the same rights for decision making.
As part of a new strategic plan in 2016, the SHTG recognized the need for greater levels of patient involvement throughout the nonmedicine technology assessment process. It was recognized that a well-regarded process for involving patient organizations was already in place at the Scottish Medicines Consortium (the HIS body that assesses new medicines for their value to NHS Scotland) and that a similar approach could be put in place for nonmedicine technologies. Since then, significant progress has been made in ensuring that patient and public voice is reflected in the SHTG committee's advice. Key developments include dedicated staff resource, the inclusion of a “patient issues” section in SHTG publications, the introduction of a patient organization submission form, and inviting patient organizations to present their views on a health technology during SHTG committee meetings.
The purpose of this paper is to highlight the developments of SHTG work with patient organizations and to examine feedback received on the process up until 2019 from three case study examples.
Methods
In September 2017, a dedicated staff member with responsibility for patient and public involvement (PPI) was appointed for the first time to the SHTG. The role of the Public Involvement Advisor (PIA) is to engage proactively with patient organizations, work with the public partners of the SHTG committee to support and further develop their role, and to drive forward new ways of working to ensure that patient and public voice is heard effectively by the SHTG committee.
Patient voice is captured in the work of the SHTG in a number of different ways. At the very start of a technology assessment, a subgroup of the SHTG committee, of which the PIA is a member, makes a decision on the type of involvement that would be most appropriate for each technology being assessed. Options include, as set out in SHTG guidance documents (1), inviting patient organizations to peer-review an assessment, asking for a patient organization submission, and requesting attendance of patient organizations to the SHTG committee to present the patient view. In some cases where there is no clearly identifiable patient group, or when scoping shows that there is a body of published literature, a health services researcher will undertake a targeted patient issue search of the available literature and produce a synthesis of the research.
The information thus gathered is used to populate the "patient issues" section of the SHTG committee's advice to NHS Scotland.
The introduction of a dedicated PIA for the SHTG has enabled a more focused engagement with patient organizations, which, in turn, has led to the SHTG increasing its interaction with patient organizations and receiving patient organization submissions for the first time. The submissions—directly received from patient organizations, which were introduced in 2017 and used for the first time in 2018—elicit information on the experiences of those living with, and caring for, people with the condition for which the health technology is indicated. This can be from a variety of sources such as surveys and focus groups. The submission form and accompanying guidance was adapted from guidance produced by the Health Technology Assessment international (HTAi) Interest Group for Patient and Citizen Involvement in HTA (2) and from the "Guide for Patient Group Partners" developed by the Scottish Medicines Consortium (3).
Every patient organization that sends in a submission has the option to attend the relevant SHTG committee meeting, at which it can present its submission and answer any questions on it. This is done by a PowerPoint presentation, with organizations given 10 min to make a presentation if they are doing a solo presentation or up to 20 min if it is a joint presentation with other patient organizations.
Since 2018, out of four technologies that patient organization submissions have been requested for, the SHTG has received submissions for three. The reason that no submission was received for the fourth technology was a lack of available resources from the patient organizations to complete one.
Across the three assessments, five patient organizations submitted patient organization submission forms. From these five organizations, four attended the SHTG committee meeting to present the findings of their submissions. The organization that could not attend did not have a staff member available to travel to the meeting, but it did provide input to the presentation given by another patient organization during the meeting.
Case studies of each of the technology assessments are presented in the "results" section, to illustrate the processes involved.
In order to gain feedback on the new processes, the patient organizations that participated in three technology assessments were asked to fill in an online questionnaire. These technology assessments were the following:
(1) Freestyle Libre (4)
(2) Robot-assisted Surgery for Rectal Cancer (5)
(3) Autologous Hematopoietic Stem Cell Transplant (AHSCT) for Multiple Sclerosis (6)
For the Freestyle Libre case study, the patient organization filled in a general online survey for people who observe meetings because the patient organization survey had not been fully developed by that point. In addition, a telephone interview was conducted with each of the four public partners who sits on the SHTG committee. These interviews lasted for around 30 min with each participant. The questions that were asked are detailed in Table 1.
Table 1. Questions used to interview public partners via telephone
This was the only example where a telephone interview was carried out with public partners by the PIA. The intention was to get a snapshot of views on the first committee meeting at which a patient organization was present. The feedback from all four public partners was summarized and sent to them to ensure that their views had been captured fully.
For the second and third case studies, the patient organizations filled in a specific online feedback form as set out in Table 2.
Table 2. Online survey questions sent to patient organizations for feedback on the SHTG involvement process
Feedback from committee members was gathered via the general online survey questions that are sent to all members to get their views on how the meeting was conducted and to capture any learning for future meetings. Participation in the general survey is usually around 25 percent of committee members.
For the third case study on AHSCT, in addition to the general online survey, a specific survey was developed for committee members to get more detailed views on patient organization involvement (Table 3).
Table 3. Online survey questions for SHTG committee members to gain feedback on patient organization input to meetings
The number of patient organizations providing peer-review input to SHTG advice products was recorded to compare the quantity of peer reviews from patient organizations received before and after the employment of the PIA. Feedback was also gathered from two health service researchers who worked on the three case study examples. The researchers were asked for their views by e-mail by the SHTG PIA and, in particular, requested to give their reflections on having patient organizations involved in the SHTG process.
Results
Freestyle Libre
Freestyle Libre was the first SHTG assessment for which a patient organization submitted evidence and then subsequently attended an SHTG meeting to present its findings to the committee. This organization was given 10 min to make the presentation, before being asked to return to the public gallery. The feedback form completed by the patient organization showed that overall it was content with the process. It stated:
“We welcomed the opportunity to present the patient perspective and found the experience very positive.
It is always difficult when providing written and oral evidence, to know if the information resonates with the audience and is recognized. The experience of the meetings was that this was the case and that those in the room had taken on board the patient perspective to assist in a difficult task.”
The main area of improvement they identified was to ensure that in future, patient organizations could remain at the committee table throughout the discussion of a particular technology. This would enable the organizations not only to answer any questions after their presentation slot, but also to contribute to other discussions about patient experiences of using the technology. Further to this, there were issues raised about how their submission had been adapted to populate the "patient issues" section of the SHTG advice document. The adaptation was viewed as being too condensed and, due to the tight timescales involved, the patient organization was not given the opportunity to check the adaptation before it was included in the SHTG committee meeting paperwork. This valuable feedback was acted upon to improve the process for the next patient organization submission and presentation to the SHTG.
From the general meeting survey of committee members, specific comments (n = 2) that related to the patient organization involvement were positive. They welcomed the inclusion of patient organizations in the process. One respondent stated:
“The inclusion of the patient group submission and expert was a great advance and was excellent for clarity of the advice note [for NHS Scotland].”
The public partners (n = 4) of the committee, via telephone interviews (Table 1), agreed that the patient organization submission and presentation was a positive addition to the meeting. The themes from the feedback received were that it brought a new perspective and dynamic, while helping to balance economic and patient-related considerations. Comments about this included the following:
“The patient group submission brought a new dynamic to the meeting and shows the reality of what happens for someone who has diabetes and how Freestyle Libre® can help.”
“The patient group helps to balance the economic and patient data.”
Regarding areas for improvement, public partners noted that the display of presentation material could be clearer for committee members, and the theme from the general survey was a desire to see more patient organizations at future meetings, with one person stating:
“Real effort is being made to ensure that the process is inclusive.”
The learning from this first experience shaped the approach for the next two assessments that included patient organization presentations. Updated guidance was issued for those attending SHTG meetings, which made clear that organizations could stay at the committee table for the full discussion and answer questions—via the chairperson—about the technology under review. A standard slide presentation template was developed to make it easier for organizations to structure their presentations.
Robot-Assisted Surgery for Rectal Cancer
This was the second SHTG assessment that included a patient organization submission, followed by attendance and participation during the SHTG committee meeting. The patient organization representative was seated at the committee table for the full discussion about the topic and had been given sight of the draft advice papers before the meeting, with the opportunity to act as a peer reviewer on behalf of the organization they represented. This meant that the patient organizations could see in advance how their submission had been included within the SHTG documentation and had the opportunity to give any comments on the use of their material before the committee meeting. The representative gave a 10-minute PowerPoint presentation using the standard SHTG template and answered questions from committee members afterward. They were also able to input to the discussion through the chair where appropriate.
Following the meeting, the patient organization representative completed their online questionnaire (Table 2). The survey indicated that they were satisfied with the level of support given to take part in the SHTG process. They stated:
“I felt really well supported throughout the process.”
“Made to feel very welcome by the committee, which was really appreciated. I also felt that my contribution to the meeting was valued by the committee, which made all the hard work worthwhile!”
The patient organization representative went on to write a blog (7) about their experience of the SHTG process.
There was feedback from committee members through the general meeting survey (n = 3). Committee members commented directly about the patient organization involvement, with two positive comments and one negative. This feedback included the following:
“Feels that the SHTG is moving forward by having this type of presentation”
“The patient submission like last time was very good. Need to have more of these, it is changing how I view the evidence being presented.”
“I think too much weight is given in the new style approach to the SHTG to clinical and public/patient view and not enough to evidence.”
In an attempt to explore the one negative comment, it was decided that for the next assessment that included a patient organization submission, a separate feedback form (Table 3) would be developed to specifically ask for members’ views on patient organization involvement.
AHSCT for Multiple Sclerosis
This was the third technology that had a patient organization submission and subsequent presentation. Three charities submitted patient organization submissions. Two of them went on to do a joint presentation for the SHTG committee. The third was unable to attend the meeting, but its submission was part of the committee papers, and it inputted to the joint presentation of the other organizations that took part. The presentation format followed the previous examples, but with the addition of a video that highlighted the experience of one person who had undergone the treatment. The two patient organizations that presented completed the patient organization survey (Table 2) about their experience.
They were satisfied with the information and support provided by the SHTG during the submission process and noted no areas for improvement in support running up to the SHTG committee meeting.
Both organizations expressed their satisfaction with the opportunity given to contribute to the discussion about the technology. They felt that the 20 min given for joint submissions to present at the committee was enough time to share the views of the multiple sclerosis (MS) community.
For what could be improved during the meeting attendance, one organization stated:
“On a personal level we were not confident enough to question the comments and views of one of the clinical experts. If we were to have been directly asked our opinion, then we would have commented. This is not a negative comment, only providing further information with hindsight.”
For general comments about the process, one organization commented:
“It is a very inclusive process. Being involved throughout the process has been very welcome, especially having the chance to comment on the draft, peer-review documents, and draft advice before the meeting. One of the researchers was still asking me questions about information we provided throughout the process before the meeting started. I felt that we were a valued contributor to the process.”
A separate online questionnaire (Table 3) was sent to the SHTG committee members to get their views on the input from patient organizations during the AHSCT assessment. The findings from the committee members (n = 6) showed that the patient organization submissions were valued, with the common theme being that they give a good understanding of what life is like with MS and the patient perspective on the treatment.
The presentation was well received, and positive comments were given about using PowerPoint and video. There was clear benefit in hearing directly from patient organizations at the meeting.
For areas of improvement, it was felt that asking patient organizations directly if they had questions about the clinical presentations should be considered for future meetings so as to enhance their involvement in the discussions.
General comments included how to balance the messages from patient organization views with more "traditional" evidence (e.g., randomized controlled trials) and the importance of ensuring that the committee is open to listening to patient views to help inform advice.
One committee member commented in regard to the patient organization submission form: “It helped with understanding the reasons why people may make certain decisions about their treatment and it brought to life the reality of the impact of living with the disease.”
Regarding the use of video for the first time, one respondent stated: “I think all patient groups should be encouraged to consider video—this will allow the groups to be sure they have conveyed what they need to convey and will reduce the stress of the presentation for them.”
One comment regarding what could have been done to improve the patient organization input went like this: “It would have been good to hear whether the patient organizations had any questions or comments about the clinical presentations.”
This echoes the area for improvement that one of the patient organizations picked up in their comments.
In the general comments section of the questionnaire, there were two responses, with one respondent stating that they were concerned about the committee being “emotionally swayed” by the views given. The second comment called for the SHTG to ensure that it continues to carefully listen to patient views and ensure that this approach continues.
All feedback received on the three topics that had patient organization submissions and presentations was shared and noted by the SHTG committee to inform future process developments.
Two health services researchers who worked on the assessments for the three case studies were asked for their views on the impact of patient organization involvement. They stated:
“Patient organizations provide an important, unique perspective on the topics assessed by the SHTG. In particular, they have offered insight into the patient experience of technologies and clinical conditions, raised patient issues not identified in the published literature, and highlighted contextual factors relating to patients in SHTG advice.” Health Services Researcher One
“The facilitation of close and ongoing collaboration with the patient organizations right from the start allowed us to share perspectives and offer complete transparency around our processes including question setting and evidence retrieval. Mutual sharing of information sources helped stakeholders to be confident that all relevant material had been assessed and contributed to the formation of advice.” Health Services Researcher Two
Peer Review
Out of the twenty-two SHTG publications containing advice for NHS Scotland in 2018 and 2019, 86 percent had patient organization involvement through peer review. This is double the level of input achieved over the 2016–2017 period, when out of fourteen SHTG publications, 43 percent had been peer-reviewed by a patient organization.
Discussion
The experience of the SHTG engaging with patient organizations is still at an early stage. For patient organizations to commit resources to take part in an HTA process, it is important to ensure that patient input is sufficiently valued. Feedback that has been received so far from the small number of patient organizations that have taken part in SHTG processes is encouraging, supported by the initial feedback from SHTG committee members and staff members.
Feedback has not been obtained from patient organizations taking part in the peer-review process. However, the increase in the number of peer reviews shows the difference made by having a staff member in place who is responsible for relationships with patient organizations. To date there have been no concerns registered from patient organizations about the process, but the SHTG will be putting in place an online survey for patient organizations to feedback their experience of the peer-review process.
A question that has still to be considered is whether a decision made at an SHTG meeting would have been the same if it had not had any patient organization involvement. As yet, data have not been collected to enable this to be answered. In the meantime, it is reassuring to know from the information that has been captured so far from committee members that patient organization involvement has been generally well received. It is important, though, to reflect that the number of committee members who participated in the feedback surveys was low and, therefore, the information gathered cannot be considered to be the view of the whole committee but rather a small proportion of them. To gain further insight into views and the impact made, the online survey needs to be complemented with a series of one-to-one interviews. This is something that the SHTG is considering for future evaluations. Such a mixed methods approach is highlighted by Mason et al. (Reference Mason, Searle, Bombard, Rahmadian, Chambers and Mai8) in their scoping review on evaluating the impact of patient involvement in HTA.
The learning from SHTG work to date is being taken on board to refine its involvement process as exemplified by how the feedback questionnaires have been amended. Guidance for how patient organizations can present at meetings is being updated to allow for prerecorded patient stories to be played to the committee as part of a patient organization presentation. In addition, work is ongoing to look at how the SHTG can open up its public involvement work by allowing direct engagement with members of the public for relevant health technology appraisals where there is likely to be much interest. The SHTG is scoping how other agencies carry out work like this, while considering best practice tools including the Guideline International Network Public Toolkit (9) and the Healthcare Improvement Scotland—Participation Toolkit (10). It is important for the SHTG and other HTA agencies to take as broad an approach as possible to not only learn from their own practice but also look at what is happening elsewhere with the wealth of resources available.
Continual learning from evidence-informed PPI practice is key for all HTA agencies to not only show its impact but also to ensure that practice is updated where it needs to be in order to meet the requirements of stakeholders. Evaluation is important as part of this, and using robust recognized methods is vital to ensuring that PPI continues to have credibility in the appraisal process.
The lessons learned from the SHTG's experience so far show that when undertaking involvement work, it benefits from being adequately resourced with a dedicated staff member in place, as has been evidenced by the increase in patient organization involvement since 2017. It is also important that patient organizations know and understand exactly what is expected of them in any contribution that they make. Clear lines of communication and openness in any process is always key, and the three case studies demonstrate the importance of this.
A challenge that remains for the SHTG is to make an attempt to identify patient organizations for every review and encourage their involvement in the process.
Conclusion
Significant progress has been made in improving the SHTG's patient involvement processes. The increase in patient organizations being involved through peer review and taking part in SHTG committee meetings has been achieved by having dedicated staff resource, with time to implement and support engagement. This demonstrates the importance of investing in posts like this to enhance relationships with patient organizations and provide tailored support.
Committee members have been broadly positive about the increased role of patient organizations, although as highlighted, responses to feedback surveys have been low, with more work needed to address this. Innovations such as the addition of video to a presentation show the dynamic way in which the SHTG is moving forward to embrace new ways of working for health technology assessments. The impact on how the involvement of patient organizations influences the final content of advice statements is still to be fully assessed and will proceed as more examples of patient organization involvement with the SHTG are gathered.
The SHTG has acted on all feedback from patient organizations to improve its processes, and the feedback from patient organizations gathered so far in the contexts highlighted shows a high level of satisfaction with the involvement procedures that are in place.
Funding Statement
This research received no specific grant from any funding agency, commercial or not-for-profit sector.
You have
Access
Introduction
The Scottish Health Technologies Group (SHTG), which is part of Healthcare Improvement Scotland (HIS), provides evidence support and advice to the National Health Service in Scotland (NHS Scotland) on the use of new and existing health technologies, which, although not medicines, are likely to have significant implications for people's care. The main decision-making body of the SHTG is the committee that the staff team support. The role of the committee is to advise and support the best use of health technology interventions within NHS Scotland. The committee is made up of representatives from NHS health boards, clinical and professional networks, academia, NHS Scotland National Procurement, and the Scottish government. Out of the thirty members, it also includes four volunteer public partners who represent a public view. All committee members are treated equally with the same access to meeting papers and the same rights for decision making.
As part of a new strategic plan in 2016, the SHTG recognized the need for greater levels of patient involvement throughout the nonmedicine technology assessment process. It was recognized that a well-regarded process for involving patient organizations was already in place at the Scottish Medicines Consortium (the HIS body that assesses new medicines for their value to NHS Scotland) and that a similar approach could be put in place for nonmedicine technologies. Since then, significant progress has been made in ensuring that patient and public voice is reflected in the SHTG committee's advice. Key developments include dedicated staff resource, the inclusion of a “patient issues” section in SHTG publications, the introduction of a patient organization submission form, and inviting patient organizations to present their views on a health technology during SHTG committee meetings.
The purpose of this paper is to highlight the developments of SHTG work with patient organizations and to examine feedback received on the process up until 2019 from three case study examples.
Methods
In September 2017, a dedicated staff member with responsibility for patient and public involvement (PPI) was appointed for the first time to the SHTG. The role of the Public Involvement Advisor (PIA) is to engage proactively with patient organizations, work with the public partners of the SHTG committee to support and further develop their role, and to drive forward new ways of working to ensure that patient and public voice is heard effectively by the SHTG committee.
Patient voice is captured in the work of the SHTG in a number of different ways. At the very start of a technology assessment, a subgroup of the SHTG committee, of which the PIA is a member, makes a decision on the type of involvement that would be most appropriate for each technology being assessed. Options include, as set out in SHTG guidance documents (1), inviting patient organizations to peer-review an assessment, asking for a patient organization submission, and requesting attendance of patient organizations to the SHTG committee to present the patient view. In some cases where there is no clearly identifiable patient group, or when scoping shows that there is a body of published literature, a health services researcher will undertake a targeted patient issue search of the available literature and produce a synthesis of the research.
The information thus gathered is used to populate the "patient issues" section of the SHTG committee's advice to NHS Scotland.
The introduction of a dedicated PIA for the SHTG has enabled a more focused engagement with patient organizations, which, in turn, has led to the SHTG increasing its interaction with patient organizations and receiving patient organization submissions for the first time. The submissions—directly received from patient organizations, which were introduced in 2017 and used for the first time in 2018—elicit information on the experiences of those living with, and caring for, people with the condition for which the health technology is indicated. This can be from a variety of sources such as surveys and focus groups. The submission form and accompanying guidance was adapted from guidance produced by the Health Technology Assessment international (HTAi) Interest Group for Patient and Citizen Involvement in HTA (2) and from the "Guide for Patient Group Partners" developed by the Scottish Medicines Consortium (3).
Every patient organization that sends in a submission has the option to attend the relevant SHTG committee meeting, at which it can present its submission and answer any questions on it. This is done by a PowerPoint presentation, with organizations given 10 min to make a presentation if they are doing a solo presentation or up to 20 min if it is a joint presentation with other patient organizations.
Since 2018, out of four technologies that patient organization submissions have been requested for, the SHTG has received submissions for three. The reason that no submission was received for the fourth technology was a lack of available resources from the patient organizations to complete one.
Across the three assessments, five patient organizations submitted patient organization submission forms. From these five organizations, four attended the SHTG committee meeting to present the findings of their submissions. The organization that could not attend did not have a staff member available to travel to the meeting, but it did provide input to the presentation given by another patient organization during the meeting.
Case studies of each of the technology assessments are presented in the "results" section, to illustrate the processes involved.
In order to gain feedback on the new processes, the patient organizations that participated in three technology assessments were asked to fill in an online questionnaire. These technology assessments were the following:
(1) Freestyle Libre (4)
(2) Robot-assisted Surgery for Rectal Cancer (5)
(3) Autologous Hematopoietic Stem Cell Transplant (AHSCT) for Multiple Sclerosis (6)
For the Freestyle Libre case study, the patient organization filled in a general online survey for people who observe meetings because the patient organization survey had not been fully developed by that point. In addition, a telephone interview was conducted with each of the four public partners who sits on the SHTG committee. These interviews lasted for around 30 min with each participant. The questions that were asked are detailed in Table 1.
Table 1. Questions used to interview public partners via telephone
This was the only example where a telephone interview was carried out with public partners by the PIA. The intention was to get a snapshot of views on the first committee meeting at which a patient organization was present. The feedback from all four public partners was summarized and sent to them to ensure that their views had been captured fully.
For the second and third case studies, the patient organizations filled in a specific online feedback form as set out in Table 2.
Table 2. Online survey questions sent to patient organizations for feedback on the SHTG involvement process
Feedback from committee members was gathered via the general online survey questions that are sent to all members to get their views on how the meeting was conducted and to capture any learning for future meetings. Participation in the general survey is usually around 25 percent of committee members.
For the third case study on AHSCT, in addition to the general online survey, a specific survey was developed for committee members to get more detailed views on patient organization involvement (Table 3).
Table 3. Online survey questions for SHTG committee members to gain feedback on patient organization input to meetings
The number of patient organizations providing peer-review input to SHTG advice products was recorded to compare the quantity of peer reviews from patient organizations received before and after the employment of the PIA. Feedback was also gathered from two health service researchers who worked on the three case study examples. The researchers were asked for their views by e-mail by the SHTG PIA and, in particular, requested to give their reflections on having patient organizations involved in the SHTG process.
Results
Freestyle Libre
Freestyle Libre was the first SHTG assessment for which a patient organization submitted evidence and then subsequently attended an SHTG meeting to present its findings to the committee. This organization was given 10 min to make the presentation, before being asked to return to the public gallery. The feedback form completed by the patient organization showed that overall it was content with the process. It stated:
“We welcomed the opportunity to present the patient perspective and found the experience very positive.
It is always difficult when providing written and oral evidence, to know if the information resonates with the audience and is recognized. The experience of the meetings was that this was the case and that those in the room had taken on board the patient perspective to assist in a difficult task.”
The main area of improvement they identified was to ensure that in future, patient organizations could remain at the committee table throughout the discussion of a particular technology. This would enable the organizations not only to answer any questions after their presentation slot, but also to contribute to other discussions about patient experiences of using the technology. Further to this, there were issues raised about how their submission had been adapted to populate the "patient issues" section of the SHTG advice document. The adaptation was viewed as being too condensed and, due to the tight timescales involved, the patient organization was not given the opportunity to check the adaptation before it was included in the SHTG committee meeting paperwork. This valuable feedback was acted upon to improve the process for the next patient organization submission and presentation to the SHTG.
From the general meeting survey of committee members, specific comments (n = 2) that related to the patient organization involvement were positive. They welcomed the inclusion of patient organizations in the process. One respondent stated:
“The inclusion of the patient group submission and expert was a great advance and was excellent for clarity of the advice note [for NHS Scotland].”
The public partners (n = 4) of the committee, via telephone interviews (Table 1), agreed that the patient organization submission and presentation was a positive addition to the meeting. The themes from the feedback received were that it brought a new perspective and dynamic, while helping to balance economic and patient-related considerations. Comments about this included the following:
“The patient group submission brought a new dynamic to the meeting and shows the reality of what happens for someone who has diabetes and how Freestyle Libre® can help.”
“The patient group helps to balance the economic and patient data.”
Regarding areas for improvement, public partners noted that the display of presentation material could be clearer for committee members, and the theme from the general survey was a desire to see more patient organizations at future meetings, with one person stating:
“Real effort is being made to ensure that the process is inclusive.”
The learning from this first experience shaped the approach for the next two assessments that included patient organization presentations. Updated guidance was issued for those attending SHTG meetings, which made clear that organizations could stay at the committee table for the full discussion and answer questions—via the chairperson—about the technology under review. A standard slide presentation template was developed to make it easier for organizations to structure their presentations.
Robot-Assisted Surgery for Rectal Cancer
This was the second SHTG assessment that included a patient organization submission, followed by attendance and participation during the SHTG committee meeting. The patient organization representative was seated at the committee table for the full discussion about the topic and had been given sight of the draft advice papers before the meeting, with the opportunity to act as a peer reviewer on behalf of the organization they represented. This meant that the patient organizations could see in advance how their submission had been included within the SHTG documentation and had the opportunity to give any comments on the use of their material before the committee meeting. The representative gave a 10-minute PowerPoint presentation using the standard SHTG template and answered questions from committee members afterward. They were also able to input to the discussion through the chair where appropriate.
Following the meeting, the patient organization representative completed their online questionnaire (Table 2). The survey indicated that they were satisfied with the level of support given to take part in the SHTG process. They stated:
“I felt really well supported throughout the process.”
“Made to feel very welcome by the committee, which was really appreciated. I also felt that my contribution to the meeting was valued by the committee, which made all the hard work worthwhile!”
The patient organization representative went on to write a blog (7) about their experience of the SHTG process.
There was feedback from committee members through the general meeting survey (n = 3). Committee members commented directly about the patient organization involvement, with two positive comments and one negative. This feedback included the following:
“Feels that the SHTG is moving forward by having this type of presentation”
“The patient submission like last time was very good. Need to have more of these, it is changing how I view the evidence being presented.”
“I think too much weight is given in the new style approach to the SHTG to clinical and public/patient view and not enough to evidence.”
In an attempt to explore the one negative comment, it was decided that for the next assessment that included a patient organization submission, a separate feedback form (Table 3) would be developed to specifically ask for members’ views on patient organization involvement.
AHSCT for Multiple Sclerosis
This was the third technology that had a patient organization submission and subsequent presentation. Three charities submitted patient organization submissions. Two of them went on to do a joint presentation for the SHTG committee. The third was unable to attend the meeting, but its submission was part of the committee papers, and it inputted to the joint presentation of the other organizations that took part. The presentation format followed the previous examples, but with the addition of a video that highlighted the experience of one person who had undergone the treatment. The two patient organizations that presented completed the patient organization survey (Table 2) about their experience.
They were satisfied with the information and support provided by the SHTG during the submission process and noted no areas for improvement in support running up to the SHTG committee meeting.
Both organizations expressed their satisfaction with the opportunity given to contribute to the discussion about the technology. They felt that the 20 min given for joint submissions to present at the committee was enough time to share the views of the multiple sclerosis (MS) community.
For what could be improved during the meeting attendance, one organization stated:
“On a personal level we were not confident enough to question the comments and views of one of the clinical experts. If we were to have been directly asked our opinion, then we would have commented. This is not a negative comment, only providing further information with hindsight.”
For general comments about the process, one organization commented:
“It is a very inclusive process. Being involved throughout the process has been very welcome, especially having the chance to comment on the draft, peer-review documents, and draft advice before the meeting. One of the researchers was still asking me questions about information we provided throughout the process before the meeting started. I felt that we were a valued contributor to the process.”
A separate online questionnaire (Table 3) was sent to the SHTG committee members to get their views on the input from patient organizations during the AHSCT assessment. The findings from the committee members (n = 6) showed that the patient organization submissions were valued, with the common theme being that they give a good understanding of what life is like with MS and the patient perspective on the treatment.
The presentation was well received, and positive comments were given about using PowerPoint and video. There was clear benefit in hearing directly from patient organizations at the meeting.
For areas of improvement, it was felt that asking patient organizations directly if they had questions about the clinical presentations should be considered for future meetings so as to enhance their involvement in the discussions.
General comments included how to balance the messages from patient organization views with more "traditional" evidence (e.g., randomized controlled trials) and the importance of ensuring that the committee is open to listening to patient views to help inform advice.
One committee member commented in regard to the patient organization submission form: “It helped with understanding the reasons why people may make certain decisions about their treatment and it brought to life the reality of the impact of living with the disease.”
Regarding the use of video for the first time, one respondent stated: “I think all patient groups should be encouraged to consider video—this will allow the groups to be sure they have conveyed what they need to convey and will reduce the stress of the presentation for them.”
One comment regarding what could have been done to improve the patient organization input went like this: “It would have been good to hear whether the patient organizations had any questions or comments about the clinical presentations.”
This echoes the area for improvement that one of the patient organizations picked up in their comments.
In the general comments section of the questionnaire, there were two responses, with one respondent stating that they were concerned about the committee being “emotionally swayed” by the views given. The second comment called for the SHTG to ensure that it continues to carefully listen to patient views and ensure that this approach continues.
All feedback received on the three topics that had patient organization submissions and presentations was shared and noted by the SHTG committee to inform future process developments.
Two health services researchers who worked on the assessments for the three case studies were asked for their views on the impact of patient organization involvement. They stated:
“Patient organizations provide an important, unique perspective on the topics assessed by the SHTG. In particular, they have offered insight into the patient experience of technologies and clinical conditions, raised patient issues not identified in the published literature, and highlighted contextual factors relating to patients in SHTG advice.” Health Services Researcher One
“The facilitation of close and ongoing collaboration with the patient organizations right from the start allowed us to share perspectives and offer complete transparency around our processes including question setting and evidence retrieval. Mutual sharing of information sources helped stakeholders to be confident that all relevant material had been assessed and contributed to the formation of advice.” Health Services Researcher Two
Peer Review
Out of the twenty-two SHTG publications containing advice for NHS Scotland in 2018 and 2019, 86 percent had patient organization involvement through peer review. This is double the level of input achieved over the 2016–2017 period, when out of fourteen SHTG publications, 43 percent had been peer-reviewed by a patient organization.
Discussion
The experience of the SHTG engaging with patient organizations is still at an early stage. For patient organizations to commit resources to take part in an HTA process, it is important to ensure that patient input is sufficiently valued. Feedback that has been received so far from the small number of patient organizations that have taken part in SHTG processes is encouraging, supported by the initial feedback from SHTG committee members and staff members.
Feedback has not been obtained from patient organizations taking part in the peer-review process. However, the increase in the number of peer reviews shows the difference made by having a staff member in place who is responsible for relationships with patient organizations. To date there have been no concerns registered from patient organizations about the process, but the SHTG will be putting in place an online survey for patient organizations to feedback their experience of the peer-review process.
A question that has still to be considered is whether a decision made at an SHTG meeting would have been the same if it had not had any patient organization involvement. As yet, data have not been collected to enable this to be answered. In the meantime, it is reassuring to know from the information that has been captured so far from committee members that patient organization involvement has been generally well received. It is important, though, to reflect that the number of committee members who participated in the feedback surveys was low and, therefore, the information gathered cannot be considered to be the view of the whole committee but rather a small proportion of them. To gain further insight into views and the impact made, the online survey needs to be complemented with a series of one-to-one interviews. This is something that the SHTG is considering for future evaluations. Such a mixed methods approach is highlighted by Mason et al. (Reference Mason, Searle, Bombard, Rahmadian, Chambers and Mai8) in their scoping review on evaluating the impact of patient involvement in HTA.
The learning from SHTG work to date is being taken on board to refine its involvement process as exemplified by how the feedback questionnaires have been amended. Guidance for how patient organizations can present at meetings is being updated to allow for prerecorded patient stories to be played to the committee as part of a patient organization presentation. In addition, work is ongoing to look at how the SHTG can open up its public involvement work by allowing direct engagement with members of the public for relevant health technology appraisals where there is likely to be much interest. The SHTG is scoping how other agencies carry out work like this, while considering best practice tools including the Guideline International Network Public Toolkit (9) and the Healthcare Improvement Scotland—Participation Toolkit (10). It is important for the SHTG and other HTA agencies to take as broad an approach as possible to not only learn from their own practice but also look at what is happening elsewhere with the wealth of resources available.
Continual learning from evidence-informed PPI practice is key for all HTA agencies to not only show its impact but also to ensure that practice is updated where it needs to be in order to meet the requirements of stakeholders. Evaluation is important as part of this, and using robust recognized methods is vital to ensuring that PPI continues to have credibility in the appraisal process.
The lessons learned from the SHTG's experience so far show that when undertaking involvement work, it benefits from being adequately resourced with a dedicated staff member in place, as has been evidenced by the increase in patient organization involvement since 2017. It is also important that patient organizations know and understand exactly what is expected of them in any contribution that they make. Clear lines of communication and openness in any process is always key, and the three case studies demonstrate the importance of this.
A challenge that remains for the SHTG is to make an attempt to identify patient organizations for every review and encourage their involvement in the process.
Conclusion
Significant progress has been made in improving the SHTG's patient involvement processes. The increase in patient organizations being involved through peer review and taking part in SHTG committee meetings has been achieved by having dedicated staff resource, with time to implement and support engagement. This demonstrates the importance of investing in posts like this to enhance relationships with patient organizations and provide tailored support.
Committee members have been broadly positive about the increased role of patient organizations, although as highlighted, responses to feedback surveys have been low, with more work needed to address this. Innovations such as the addition of video to a presentation show the dynamic way in which the SHTG is moving forward to embrace new ways of working for health technology assessments. The impact on how the involvement of patient organizations influences the final content of advice statements is still to be fully assessed and will proceed as more examples of patient organization involvement with the SHTG are gathered.
The SHTG has acted on all feedback from patient organizations to improve its processes, and the feedback from patient organizations gathered so far in the contexts highlighted shows a high level of satisfaction with the involvement procedures that are in place.
Funding Statement
This research received no specific grant from any funding agency, commercial or not-for-profit sector.