Research can be a tricky business. There are many difficulties: ethical considerations, sampling techniques, logistics and safety, and ensuring the whole endeavour answers important and under-researched questions. Many of these issues can be addressed before the research kicks off, particularly through a thorough literature review and stakeholder consultations. However, to really iron out the kinks in a proposed research project, a well-designed pilot phase is essential. Piloting means testing research instruments and approaches with people who are similar to research participants, but do not participate in the main study.
Despite a mountain of text-books promoting its benefits, all too often researchers fail to devote the time and resources piloting deserves – this is particularly true with qualitative or mixed-methods research. In addition, the results of pilot studies are rarely written up, leading sociologists Edwin R. van Teijlingen and Vanora Hundley to call for ‘more discussion amongst researchers of both the process and outcomes of pilot studies’.
The importance of piloting a research instrument was brought home to us last year when we tested a questionnaire in Papua New Guinea (PNG). The research, eventually conducted across four provinces, aimed at better understanding public servants’ perceptions and experiences of good governance and corruption. To do so, a semi-structured questionnaire was designed to gather quantitative and qualitative data. Clearly, this is a sensitive topic. We wanted to know just what types of questions we could ask, and which were too sensitive. So, we undertook a two-pronged approach to testing the questionnaire.
First, we spoke to PNG public servants undertaking studies at the Australian National University. This allowed for testing the questions in what we thought would be a safe environment – that is, public servants were away from their colleagues and were able to talk more freely than at their workplaces in PNG.
Second, we tested the research instrument in PNG with public servants working at provincial and national administrations.
Through this process we tidied up or removed questions that didn’t make sense and, perhaps more importantly, became more confident about what questions we could include without ruffling feathers. We included two questions in our pilot that appeared to be potentially sensitive. The first was, ‘During the last two years, did you personally find out about any case of corruption?’ Before testing the research instrument, we thought that respondents might be wary about answering this question. The pilot showed that respondents were less concerned about us asking this question than we were. Those who participated in the pilot also helped us frame the question and explain the research to mitigate any potential fallout.
Including this question in the four-province survey resulted in some fascinating findings, with a staggering 63 per cent of 136 respondents admitting that they had found out about corruption in the preceding two years.
The second question we thought could prove to be too sensitive was around reporting corruption. We asked those who had found out about a case of corruption, whether they had reported it or not. Feedback from our pilot phase suggested that asking for a yes or no response to this question was best. Participants in the pilot were concerned that probing further would make respondents uncomfortable, so we did not include questions that probed about public servants’ experiences with corruption.
However, when conducting the four-province survey we found that some respondents were willing and ready to talk about their experiences. In this case, those who participated in the pilot were more conservative than the actual respondents. On reflection, we could have included a carefully worded qualitative question that probed respondents’ experiences with reporting.
Even so, including the ‘yes/no’ option still produced meaningful insights. Table 1 shows that less than half of respondents who found out about corruption reported it. Those in Eastern Highlands were least likely to report (we were told this was because of concerns about payback), while those in Milne Bay were most likely. Importantly, there were significant differences between the genders and seniority. Men and senior respondents were more likely to say they reported corruption.
Table 1: Reporting corruption (by percentage) (n=85)
These findings point to the importance of targeting anti-corruption initiatives to women and junior public servants, and supporting them when they report.
The pilot phase also helped us modify other elements of the questionnaire. It showed that many were concerned about questions around salaries and benefits; these questions were replaced with ones around public service grade (which is linked to salary and benefits). We also discovered that our initial research instrument was far too long. In turn, we cut down the number of questions and prepared show cards displaying options to answer some questions (for example, ‘strongly agree’; ‘agree’; ‘disagree’) in large font – this cut down on the time to interview respondents.
These changes evolved over time. The piloting phase consisted of an iterative process of continual refinement – we removed or rephrased questions and then retested the instrument.
In sum, piloting questions with public servants in Canberra and PNG helped refine the survey instrument, and meant that questions were included that provided important insights into the nature of corruption in the public service. However, with hindsight after conducting the research across four provinces, it is possible that we could have probed respondents further in some instances. Still, the time we spent piloting our questionnaire was invaluable, and the insights gained remind us that piloting remains a critical – though unfortunately in many cases often a perfunctory – part of any research project.