Design of the qualitative longitudinal study

To be useful to policy makers, research must be relevant and timely. When pilots are set up, early findings can show how policy is being put into operation, and whether the outcomes desired are likely to be achieved. Service users’ views and experiences are important. If the policy is designed to bring about change, longitudinal perspectives can throw light on whether and how this happens (Duncan and Harrop, Reference Duncan and Harrop2006). The qualitative longitudinal study described here was designed with Pathways policy customers to meet their requirements. This section of the paper discusses the original study design.

The study was an example of collaborative working. The Social Policy Research Unit was joined by qualitative researchers at the National Centre for Social Research and the Policy Studies Institute which were part of the research consortium for the overall evaluation. The researchers concerned had worked together in research consortia on other large evaluations of government initiatives in the general area of employment and disability. In this longitudinal study, researchers from all three organisations contributed to the design, preparation of topic guides for interviews, fieldwork and data extraction.

The Department for Work and Pensions wanted to know how incapacity benefits recipients taking part in the Pathways Pilot experienced the processes and services involved, and what they thought of them. Tracing people during and after their involvement with the Pilot would, it was hoped, provide more understanding about how people with health problems and impairments make progress towards work and whether and how Pathways could be helpful. The purpose of the longitudinal study was thus to explore experiences of and views about taking part in Pathways, and to capture changes occurring in people's circumstances, perceptions, attitudes and expectations in relation to moving into work from benefit. Making repeated contacts with service users to trace developments in their lives as they move through processes in service provision has been used previously with some success. The authors had experience of this approach in research with people with health conditions and impairments in the client panel study that formed part of the evaluation of the Job Retention and Rehabilitation Pilot (Farrell et al., Reference Farrell, Nice, Lewis and Sainsbury2006) discussed by Lewis (this issue).

The appropriate number of contacts and the time intervals in this kind of study depend on the focus for enquiry and the time frame of the policy intervention (Molloy et al., Reference Molloy, Woodfield and Bacon2002). Transitions and trajectories in people's lives do not occur at the same rate (Thomson et al., Reference Thomson, Bell, Holland, Henderson, McGrellis and Sharpe2002). In setting a prospective time-frame researchers must decide the periods over which any impact of intervention might be expected (Lewis, this issue; Thomson and Holland, Reference Thomson and Holland2003). In longitudinal studies for evaluation of other kinds of employment programmes different time-frames have been appropriate. Evaluation of the Job Retention and Rehabilitation Pilot, in which service users were expected to have frequent contacts with service, involved research contacts with service users at monthly intervals over a period of six months (Farrell et al., Reference Farrell, Nice, Lewis and Sainsbury2006). In evaluation of the New Deal for Young People, service users were followed for up to 18 months, with three contacts timed to fit three distinct phases of the programme (as discussed in Molloy et al., Reference Molloy, Woodfield and Bacon2002).

In this study of Pathways participants the approach taken was to aim for three contacts with each participant. In order to provide some basis for comparison as people engaged with the Pilot, the first research interviews were timed to take place close to the time people were expected to attend a second work-focused interview in Jobcentre Plus. The plan was to re-contact participants at predetermined points, when it might be expected they would have had further contact with Pathways and perhaps engaged in interventions offered to them. Thus initial face-to-face depth interviews were to be followed by telephone interviews after three months and again after a further six months. The overall nine month time span would, it was hoped, provide some indication of any longer-term impact of taking part in Pathways.

It seemed important not to link later data collection points too closely with the Pilot processes, for example using further work-focused interviews as triggers for research interviews. It was possible that some people would not go on to take part in a series of work-focused interviews anyway, as Advisers had discretion to waive later interviews. For everybody, it was likely that a number of influences would be important in shaping views and behaviour in relation to work. Influences such as health trajectories and family circumstances might have chronologies quite independent of processes of engagement with Pathways.

In order to provide a flow of information the study was designed to be conducted in three cohorts, generating an overall study group of 105 people. The first cohort started in April 2004 and the final cohort ended in December 2005. The cohorts were timed so that they closely followed the Pilot's implementation, to enable findings to feed directly into the development of the Pilot, particularly the second phase pilot sites and any later expansion. To be particularly policy relevant, there were requirements to report and publish findings as soon as possible. This led to a fairly demanding schedule of an initial cross-sectional analysis from the first wave of interviews followed by separate reports from each of the longitudinal analyses of first, second and third cohorts (see Table 1).

Table 1 Longitudinal qualitative study fieldwork and reporting

Notes: ^aUnpublished interim report (Nice and Sainsbury, Reference Nice and Sainsbury2004).

^bReport on 1st full cohort (Corden et al., Reference Corden, Nice and Sainsbury2005).

^cReport on 2nd cohort (Corden and Nice, Reference Corden and Nice2006a).

^dReport on 3rd cohort (Corden and Nice, Reference Corden and Nice2006b).

In this rapidly developing policy area, it was possible that policy interest might shift during the lifetime of the study. There might be changes in the service or new elements brought in as the programme was implemented. The research design enabled some flexibility, should there prove to be a need to respond to such changes. Having separate topic guides for each interview in the series meant that new topics and new emphases in themes could be explored at each interview, as new interests arose. Phasing the study over three cohorts meant that there would be some scope for increasing or changing recruitment targets in the third cohort, if need arose, although this might have budget and timetable implications.

It was unknown at the design stage how easy or hard it would be to recruit people to take part in a series of three interviews, and what levels of attrition there would be. The target population was expected to include people with severe and chronic health problems and impairments, which might influence both willingness to agree in advance to take part in all three research interviews, and whether those recruited would stay with the study. Attrition is a potential problem in longitudinal research, both in terms of cost-effectiveness (Cotter et al., Reference Cotter, Burke, Stouthamer-Loeber and Loeber2005) and in relation to possible selectivity of findings (Ruspini, Reference Ruspini1999). It was hoped that retention might be helped by small incentive payments at the first and final interviews, and by the rapport established between participants and researchers at initial contacts and during the interview series.

Dealing with the data

The researchers expected to be dealing with transcripts and notes from more than 300 qualitative interviews, a large volume of data for systematic analysis. The data was cumulative over the entire panel, but reports were required quickly after each phase. Across the team of researchers there was expertise in several possible approaches to extraction and analysis of this kind of data. Some members of the team preferred approaches based on software packages for qualitative analysis. However, across the team there was greater shared experience of using Framework (Ritchie et al., Reference Ritchie, Spencer, O'Connor, Ritchie and Lewis2003), and this was the approach chosen. The main responsibility for the analysis lay with the authors.

The authors’ view is that Framework, discussed in detail in an earlier article (Lewis, this issue), provided a number of advantages for this analysis. Framework enabled both cross-sectional and longitudinal analysis (Molloy et al., Reference Molloy, Woodfield and Bacon2002). The thematic framework for classification and summary of data from the first research interviews was used for early cross-sectional analysis, and then expanded and developed for extracting data from further interviews. This expansion enabled both within- and between-case comparisons at different points in time. If lines of enquiry changed in response to shifts in interest, or new kinds of data were gathered in response to lessons learned, the framework could again be expanded accordingly. The Framework approach enabled the authors to look for change, or absences of change, from common ‘baselines’ against which comparisons might be made. Saldana Reference Saldana(2003) explains how setting ‘baselines’ of descriptive qualitative data is essential for assessing any future change. In this study, one such baseline was the date of application for benefit and another was the first face-to-face contact with the Pilot.

Using Framework for immediate data extraction and summary after each wave of interviews made it easy for researchers to read and reflect on the content and chronology of data collected thus far, in order to prepare for the next interview.

Key findings for policy makers

There were a number of substantive findings from the study which have been fully reported in the published reports (Corden et al., Reference Corden, Nice and Sainsbury2005; Corden and Nice, Reference Corden and Nice2006a, Reference Corden and Nice2006b). Here, we mention some of the findings for policy makers about aspects of change and temporality which might not have been so apparent without the qualitative longitudinal approach.

First, the general initial support for the principles of Pathways among people required to take part extended during and beyond the period of contact. Whether the intervention was experienced as helpful, however, depended on whether it came at a time perceived to be appropriate. Critical here were people's perceptions of their health and the impact on their lives of their health condition. Some people valued a series of regular contacts with the Adviser. For example, some who experienced improvement in health used the contacts and services offered and made progress towards work. Some people not able to take steps towards work but who still saw work as a possibility for the future valued the continuing support and encouragement from an Adviser in a series of contacts. When people perceived no change in health and personal circumstances and no gain from meeting with an Adviser, a series of interviews at Jobcentre Plus could become repetitive and seemed unnecessary. People who went to the first work-focused interview feeling it was unlikely they would work again generally continued to show little interest in looking at opportunities through Pathways, especially when they perceived no improvement in their health.

The study highlighted some of the influences on people's use of support and how their views changed. For example, initial interest in services offered and agreements to take part often did not transfer into trying them or continuing in a programme. Contacts with some of the new services, such as the Condition Management Programme developed jointly between Jobcentre Plus and local NHS providers, were especially fragile. People were easily put off by lack of understanding of or unfamiliarity with the processes involved, or by discovering that parts of a programme were delivered in group settings. Among people prepared to continue, such services were found helpful.

The study showed some of the different time parameters for the influence of the intervention. Over a period of months, people forgot some of the information offered at the first interview with the Adviser, especially when this had seemed irrelevant to their circumstances at the time they heard it. Thus, when some people's circumstances changed and they started to think about going back to work, they did not remember some of the opportunities for support and help that might have been useful to them. Among the best remembered and most valued information were details of ways in which financial support was made available to people who wanted to try work. Some people beyond the formal period of contact with Pathways went back to their Advisers for help with applications for financial support, when ready to try work.

As described, there was continuous dialogue between the researchers and policy makers throughout the analysis. Interesting issues arose and some challenges, discussed in the next section.

Emerging findings and longer-term perspectives

The importance attached to Pathways by policy makers meant they were keen to get early feedback. They asked for an interim report of ‘emerging findings’ as soon as possible after the first round of qualitative interviews.

This first report (Nice and Sainsbury, Reference Nice and Sainsbury2004) was based on cross-sectional analysis of data from the first research interview with the 24 recruits to the first cohort. An important finding for policy makers at this stage was that people's first meeting with a Personal Adviser was especially significant in setting the tone of discussions and establishing trust, and in providing new information. Also clear was that although most people had positive aspirations about work, and indeed some were taking steps towards working, many felt they were not yet able to work. They perceived their health condition or impairment as the most important barrier. People had generally favourable opinions about their Adviser, and there were positive views about the outcome of their meetings. Some people had been referred to services and others were expressing interest.

In this first report, it was too soon to assess the impact on people of taking part in Pathways, and the researchers drew no conclusions. Some of these early findings pointed positively to the potential and scope of Pathways in supporting people in moving into work. Great care was taken in presenting balance between findings which policy makers would find positive and encouraging and others which might seem disappointing or more negative. It was felt important that this early report did not raise over-positive expectations that might leave policy makers puzzled by more long-term findings. There were already indications from this early cross-sectional analysis that there was a group of people who did not expect to work again.

As the study progressed and the researchers looked towards the longitudinal analysis of data from the first cohort, due to be presented in the second report to policy makers (Corden et al., Reference Corden, Nice and Sainsbury2005), the issue of attrition arose. As already discussed, the researchers knew from the outset that it was unlikely that all the study recruits would complete the series of three research interviews. There was, however, no pre-determined strategy for dealing with this analytically because it was unknown how many or which people would drop out. The approach taken was decided at the completion of the first cohort, taking into consideration the amount of attrition, any indications about selectivity in attrition, and the kind of data available from people who did not go on to complete the full series of interviews.

Eighteen of the 24 people recruited to the first cohort went on to complete three interviews. The loss of six people from the panel was related mainly to problems in remaking contact, and telephone appointments not kept by respondents. The level of attrition was the same in each of the three initial pilot areas. The number of people involved was too small to look for any patterns in their characteristics. Full longitudinal analysis across the same chronological panel period was therefore only possible for 18 of the 24 people. We had to consider how to use data from those who dropped out.

One option was abandoning data from those six people. It is unethical not to use information from people who have agreed to take part in research and who expect their views to be taken into consideration. In addition to ethical issues, not using some of the data would be wasteful both in terms of resources expended already and, importantly, in terms of the loss of insights from the data collected thus far. For examination of the overall data quickly showed that changes that took place in some people's lives were already apparent by the first research interview. Developments in health, changes in personal and family circumstances, opportunities available or people's financial situations had time dimensions independent of the nine-month chronology of the study. Similarly, the nine-month time span often did not reflect temporal dimensions of people's participation in Pathways. Some people's participation had apparently ended by the time of the first research interview, whereas others were still expecting further contacts with their Adviser at the time of the last research interview.

The decision was therefore taken to use longitudinal analysis to look for change, or absence of change, and any association with taking part in Pathways across all the interviews achieved, including those with people who did not complete the series. We did this by taking, as a ‘baseline’ from which to seek change, the point of initial contact with Pathways. There was retrospective information from everybody about any changes in behaviour since then. For example, exploration of experiences and views of people who used Pathways services or support was based on all the data available from all the interviews, and looked at how people experienced support, the extent to which it was helpful, and how. There were useful findings here from people who did not complete the series of research interviews, as well as the 18 who did. Some of those who stayed with the study provided more or new information on these issues in later interviews, but not all did, for example people who had no more contact with Pathways. Each part of the written report therefore started by explaining which groups of participants generated the data for the analysis that followed.

Clear explanations were also necessary when a different kind of perspective was included in the second report. In their final research interview, the 18 people who did complete the series of interviews were invited to look back and think overall about what happened after they joined Pathways. Most were then at least 12 months from their first contact with Pathways. Asked to reflect on their current circumstances compared with those when they joined Pathways, and their expectations for the future, they talked about any difference Pathways had made in their lives. These kinds of perspective do not provide detailed information about contemporaneous experience of service, or development of views. Rather, they are reflective comments and suggestions from people looking overall at their lives, and balancing up from different distances the positive and negative impacts of all their experiences of Pathways. This is the picture of Pathways that people may take forward into the future, when details of interactions have become more blurred but general impressions and beliefs may still be important in influencing views and behaviours, or what they may tell other people about Pathways.

Thomson and Holland (Reference Thomson and Holland2003: 243) have discussed differences between ‘contemporaneous insight and retrospective hindsight’. In this study, views expressed after retrospective reflection at this stage sometimes appeared more polarised than views on the same topics expressed earlier. For example, a small group of men expressed strong negative feelings when they looked back on their overall experience, saying they had never expected anything useful, and their time had been wasted. In some cases, these men had talked in earlier interviews about some positive elements, for example learning new information, but this was absent from their retrospective summing up.

However, while some views appeared more polarised, others now seemed less extreme. Some of those who had strongly criticised aspects of service when they spoke contemporaneously, did not mention these again in their overall reflection. Bringing together in the same report longitudinal analyses of data collected at time intervals, with analysis of data from the same people looking back over the whole time span thus required careful explanation. It was important that policy makers understood the different analytical approach, and what it meant. Without such understanding, findings in this final chapter of the report might seem to contradict what had been reported in earlier chapters. This might serve to puzzle people reading the report or undermine the validity of the findings, rather than offer a different perspective that deepened understanding.

Only those who completed the series of interviews had the opportunity to reflect over the past year. This is an unresolved issue in the bringing together of the two approaches, because we do not know whether there was selective attrition. Whether people are prepared to continue in a series of interviews, and the way in which they share information with individual researchers at any one point in time, have implications for both the composition of the panel and the content and quality of data collected.

As the research progressed, the researchers used similar analytic approaches and ways of reporting in the two later reports, which followed the second and third cohorts. The work required in production of four separate analyses and full written reports was demanding, but probably also helpful for keeping the analysis on schedule. Also helpful in this respect was the discipline imposed by the researchers’ expectations that data from each interview would be extracted into Framework display before the next interview with the same person. This helped individual researchers focus on what was happening for each participant at each stage, and prepare for the next interview in the series. Keeping up to date in this way also meant that the research team was always broadly familiar with the general patterns developing, as the study went forward. This proved important in the final cohort, when it was decided to refocus the topic guide to be used for the last interview in the series, in the light of the balance of data emerging, as explained in the next section.

The ‘emerging findings’, reports from separate cohorts, and then the final overall picture provided a steady stream of information for policy makers. The flow of findings also helped build policy makers’ understanding of the longitudinal approach. Had there been no reporting between ‘emerging findings’ and the final report, policy makers might have been unprepared for the overall picture. For example, it was initially thought it might be possible to relate some people's experiences to differences in the way Pathways was implemented in different Jobcentre Plus districts. Such findings would have been useful to policy makers looking for best practice. It was clear from the second cohort that such analysis would not be appropriate, because in some of the newer pilot areas, relatively fewer people stayed with the panel. Explaining this to policy makers at the time removed the expectations for the final report. As another example, the picture of chronic ill health and the limitations imposed on people's activities and use of services, which was a general feature of the longitudinal analysis, was not a strong feature in the ‘emerging findings’. It was, however, becoming clear by the beginning of the second and larger cohort, and the implications of this were discussed at this stage with policy makers. Had there been no reports at the end of the first and second cohorts, there might have been greater expectations of evidence of use of services within Pathways. As policy makers received the reports from the first and second cohorts, reasons for relatively limited evidence of use of services were discussed with them, providing insights and understanding as soon as possible.

Shifts in focus of interest

As explained above, one of the opportunities perceived in policy related qualitative longitudinal work is that the approach is well suited to accommodate shifts in the focus of policy interest during the lifetime of the study.

In fact, there were relatively few adjustments in focus in the interviews in response to shifts in policy interest, and those that did happen were relatively small. When Enhanced In-Work SupportFootnote ² was added to the package of services offered in Pathways, asking people if they knew about this or had used it was dealt with easily in the interview prompts. During the study period, the policy decision was taken to extend Pathways to longer-term incapacity benefits recipients, but this was to be evaluated by new research, rather than incorporation into this study.

A more substantial adjustment than those made in response to shifts in policy interest was that suggested to policy makers by the researchers themselves, as follows.

As the overall study period drew to an end, the research team discussed how to approach the final interviews in the third and last cohort. The second cohort had generated rather limited data about use of services, which may have been related to selectivity in attrition, or the particular characteristics of those who did stay with the study. Looking at the first two interviews in the third cohort suggested that there was likely again to be rather limited detailed experiential data about use of services. Rather than persisting with the existing topic guide, designed with early focus on participation and service use, the researchers saw a possibility of generating richer data by refocusing the interview. With policy makers’ agreement, the final topic guide was redesigned, to introduce retrospective reflection as an earlier focus of the interview. People were encouraged to spend more time and at an earlier stage than in preceding interviews, looking at their experience as a whole, talking about what had been especially helpful at which stages, what might have been more helpful, what an ideal service might look like, and how and when they would like to receive it.

If interviewers prepared carefully, there seemed no particular risk of losing potentially useful contemporaneous data about circumstances or service by adopting this approach. The adjustment would also provide new stimulation for the researchers, who had come to the end of a long series of semi-structured and fairly focused interviews. When discussed in the research team, people were interested in taking this different approach, which would require new intellectual investment and energy.

This adjustment did work well, and generated rich data for inclusion in the final report. Again, care was taken to draw readers’ attention to the differences in perspective, which came from presenting data from retrospective reflection from an early stage in the research interview.