Methods Matter Podcast – Surveys & Questionnaires

Dr Donncha Mullin:

Hello, and welcome to the Methods Matter Podcast from Dementia Researcher and the National Center for Research Methods, the show that tries to make sense of research methods, to help me and you to understand them. I’m Dr. Donncha Mullin. I’m a PhD clinical fellow at the University of Edinburgh and I research motoric cognitive risk or more simply I’m investigating slow walking speed in the presence of self-reported cognitive complaint in older people without dementia or mobility disability. You will all know by now that in these podcasts, we bring together a methodology expert and a dementia researcher who’s put method into practice to talk about the why, how, and benefits of different methodologies and provide tips on avoiding some of the pitfalls. Today, we’re talking about surveys, a go-to standard use widely in research and in business alike, perfect for when you need to question individuals, test concepts, reflect attitudes of people, and establish their levels of satisfaction. However, you don’t really want to hear that from me, and that’s why we have an expert with this.

Dr Donncha Mullin:

So let’s introduce our guests. Today’s expert is Dr. Petra Boynton. Petra is a social psychologist specializing in international health research from a critical perspective. Over the past 20 years, she has taught, supervised, and undertaken research across the social and health sciences and development within university and community settings, as well as writing extensively about academic life, including publishing a book called Being Well in Academia. Hi Petra. Great to meet you.

Dr Petra Boynton:

Hello. Thank you for having me.

Dr Donncha Mullin:

It’s our pleasure. Thanks so much for coming along. Can you tell us a little bit about yourself? You seem to have a fascinating career so far.

Dr Petra Boynton:

Thank you. I’ve been really lucky to go in lots of different directions I’ve been interested in. I’m predominantly interested in methods and different methods and working with communities, but most of my research has been on sensitive topics and that’s meant I’ve had to think very carefully about ethics and about sort of inclusion. And I apply my work by sort of doing advice giving and working as an agony answer advice columnist. So that’s given an extra level of trying to make work accessible to people.

Dr Donncha Mullin:

I bet. Amazing way of bringing research out of these ivory tower type situation. You mentioned sensitive topics. Can you give me an idea of what type of things have you talked about in the past?

Dr Petra Boynton:

Yes. My early research predominantly in surveys was sort of working around sexual reproductive health, and trying to find ways to get people to feel comfortable answering very personal topics. And more recently I’ve been working on pregnancy and baby loss. So that’s been thinking more about grief and memorializing and sort being able to make people feel safe and able to share quite sensitive topics with a wider audience that they want to reach.

Dr Donncha Mullin:

Wow. Yeah. Okay. Well, I look forward to talking to you more on that. Thank you so much today. Our researcher joins us all the way from Australia. I’m delighted to introduce Dr. Larissa Bartlett, who is an Island Research Fellow at the Wicking Dementia Research and Education Center at the University of Tasmania. Her PhD focused on the promises and challenges of workplace delivered mindfulness interventions for employee health and performance. Now Larissa leads the Island Study a large 10 year prospective public health cohort study with nested interventions targeting modifiable dementia, risk factors at population level in adults, age 50 plus. Hello, Larissa. Welcome to the show.

Dr Larissa Bartlett:

Thank you, Donncha. Well done. That’s such a big mouthful.

Dr Donncha Mullin:

I was a wee bit worried. Is it the Island Study or is it spell out? I-S-L-A-N-D Study or?

Dr Larissa Bartlett:

Yeah, no, we call it the Island Study. Yep.

Dr Donncha Mullin:

Good. Well, we are delighted to have you here and thank you so much for calling in with the time difference and everything, really appreciated.

Dr Larissa Bartlett:

My pleasure. Thanks for having me.

Dr Donncha Mullin:

Now, Larissa, I couldn’t help but notice in the bio you provided that your fun fact was that you were a quote “I’m a 10 pound pom of Birmingham stock”, which I assume means you made the trip south at some point. Is that correct?

Dr Larissa Bartlett:

Yeah, that’s right. I have a very big family in England, but I grew up pretty much my whole life in Australia. And I think had a wonderful childhood as a result of my parents deciding to head over the other side of the world, but it’s given me the opportunity to have footprints in both countries, which is really a great pleasure and yeah.

Dr Donncha Mullin:

Amazing.

Dr Larissa Bartlett:

I’m struggling to find a more interesting fun fact. I’m sure there’s others.

Dr Donncha Mullin:

It’s always that line of how interesting do we really want to be here with our fun fact? Was it always Tasmania where you were based or did you grow up in a different part of Australia?

Dr Larissa Bartlett:

Yeah, so we very first arrived in Sydney, but my father got a position as a social worker on the Northwest coast of Tasmania in the town called Penguin. So we moved around few different places in Tasmania, but I grew up on this little island, which is actually where the Island Study is located. I left, as Islanders, do when I was in my late teens, early twenties and went off and I came back in my thirties, had some kiddies, and discovered a research career. So I’m very excited to be working on this Island Project. Right at the bottom of the world.

Dr Donncha Mullin:

That’s ideal. That’s the dream to get back to a small place you grew up would have what sounds like a really challenging and interesting opportunity there. That’s awesome.

Dr Larissa Bartlett:

Oh yeah.

Dr Donncha Mullin:

Now I think we should crack on with the show. I don’t want to go into midnight in Australia, so we’ll start with our first segment, which is called, What do I know? And we begin each podcast with me summarizing what I understand to be the method we’re exploring, which of course today is surveys.

Dr Donncha Mullin:

Today I am feeling confident, although I suspect this is going to be more complicated than I first imagined. We’ve all completed polls and surveys. They seem to be a way to ask people what they think, but in a sort of more structured or focused way, essentially asking a series of questions, maybe survey questions, such as which flavor of ice cream best represents you, or would you rather fight 10 duck sized horses or one horse size duck? Do you think pineapple belongs on pizza? These are all very important questions, but Petra, could you please correct me and properly introduce the method for us?

Dr Petra Boynton:

Oh, I think you’ve done a great job, actually. I wouldn’t want to sort of change anything from the ducks and the horses really. Interestingly, whenever I teach this, I have to actually write it down because as you said, it’s not as simple as you think, so I’ve written it down and I’m going to just read it to you.

Dr Petra Boynton:

So a survey is really referring to collecting a lot of information. So collecting and analyzing, applying and interpreting information from big groups of people or small groups of people and you do it with structured questions. So that could be the, yes, no answer. Do you like ice cream? Yes. No. Or on a rating scale, one to five rate this flavor of ice cream, or you might have seen more recently, particularly with online surveys or if you’ve been in shops or at airports or stations, sort of emoji thing where you click the smiley or frowny face that shows where you best feel. So that’s still a rating scale, but it’s using sort of visuals within it. Or you can also use text responses. So people could answer prompts, but you are giving them quite specific prompts usually to respond to and you can do this work on or offline.

Dr Donncha Mullin:

Okay. And why would someone maybe choose surveys, do you think, or what’s your experience of that in the past? What motivated people to use surveys?

Dr Petra Boynton:

I think the main reason, and it’s always been true, but the pandemic has definitely made a difference to this, is that it’s seen as very quick and very easy, particularly with online surveys. And I think when we sort of hit a situation where it was harder to reach people, then it seemed the most logical way that you could put something online and send out a link and people could respond. It’s very familiar. We’re mostly are used to sort of filling in rating scales or quizzes, very popular people like doing those sort of quizzes and answers. So all of that I think are the reasons why people are drawn to it. But as you said, there’s a kind of twist in the tail with this bit is that all of those reasons are deceptive in that it’s actually complicated and challenging and difficult. So I think I would say to most people, I wouldn’t assume it’s quicker and easier just because we are so familiar with it.

Dr Donncha Mullin:

Okay. And there’s something that I was never that clear on. And what is the difference between polling and surveying or is it just terminology?

Dr Petra Boynton:

No, there is a difference. So a poll is usually a couple of very quick questions. So they’re normally done to gain a general answer on something quite simplistic. So what do you think of a prime minister or what do you think of a choice of food or something like that? A survey is more wide ranging. It tends to be not necessarily reaching more people, but it’s going into more issues in more depth and a questionnaire, which is another sort of term that comes up in this, is the tool that you’d use to do the measuring. So quite often there’s lots of words that are used interchangeably. Quite often people use survey to mean questionnaire and poll together. And I think as long as we understand what we mean that’s okay.

Dr Donncha Mullin:

Okay, brilliant. Thanks for clearing that up. Now, Larissa, I’d like to bring you into the conversation if I, could you tell us a little bit about your research?

Dr Larissa Bartlett:

Yes, absolutely. Thank you, Donncha. There’s a very good match with the Island Project and this topic of surveys really. So the Island Project is a population level, dementia risk reduction, health promotion program. That’s taking place in Tasmania, which is an island off the bottom of Australia. Australia’s only island state, it has a population of about half a million people and health profile that is a bit older and less well educated and a little bit less well off, I guess, than the rest of the other states and territories in Australia. So it’s quite a good environment to have a look at the health and lifestyle factors that we understand to be related to contributing to dementia. So given this is a dementia podcast, so I guess it’s very topical. And so we’ve established this population based study. The only way we can reach all of these people really is using online technologies and surveys.

Dr Larissa Bartlett:

So our goal is to have a really sort of, I guess our goal initial goal is to reach 10,000 people with the island messaging, which is as a health promotion component, and then to include a good proportion of those people in research associated with the effectiveness of a health promotion campaign. So the goal is to, I guess, understand what people know about the potential to reduce dementia risk, and then recruit them as part of a long term tenure health promotion study and increase what they know by providing them with all sorts of different ways of educating, I guess, and opportunities to participate in risk reduction activities. So it’s a big project. We have 8,000 baseline people who’ve provided us with our baseline and the baseline is survey.

Dr Larissa Bartlett:

So essentially what we ask people to do is come on board and tell us what their own health histories and demographics are. And tell us what they know about the potential to reduce your risk of dementia through a survey with series of structured questions. What their motivations might be to change their behaviors related to dementia risks, also through surveys. And then we have another survey which is asking them about their behaviors and what they do in relation to how they manage their blood pressure and cholesterol and their sort of cardiometabolic health, their physical activity, their diet. So we collect a lot of information from our participants about their health and their behavior and their lifestyle. And we’ve got 8,000 people. And I just don’t think there’s any way we could have achieved that without using the survey method and using our online approach.

Dr Larissa Bartlett:

I’ve got so much I could say, there’s so much about this study, but I guess one of the things that we need to be able to do is find out well, what our goal of this study is to find out what people know and what their behaviors are and see whether we can change them. So our baseline is really to characterize what Tasmanian adults dementia risk profile, or modifiable dementia risk profile might be, and then track them over this long period of time. But we also need to understand whether what they’re telling us is really related to their actual health. And so one of the things about surveys is we can collect what I’m saying about myself, but it’s really important to validate that information, particularly when you’re looking at health outcomes and wanting to inform some policy on the other side. So we use other forms of data as well in the island project.

Dr Donncha Mullin:

And Is it linkage to family medicine, GP data, or what are the other methods as well?

Dr Larissa Bartlett:

That’s down the track, I think with linkage. So we started the study in 2019. We’ve invited the participants when they come back to do their annual surveys to join other parts of the island. And so we’ve got, I think just on the 1700 people, who’ve provided us with a blood sample and we’ve got cognitive tests like functioning data, cognitive functioning data, using online testing. So the CANTAB online battery. So using those methods again to collect the cognitive functioning, and then we’ll use those functional data with the biomarker data and the behavior change over time to look at whether the changes that might be happening through a health promotion campaign are actually leading to a lower actual risk of dementia over the long term.

Dr Donncha Mullin:

Brilliant. It sounds like a really, yeah, you’re gathering the baseline evidence of first off, what people’s understanding is instead of teaching people what they already know, so you can focus your interventions and your modifiable risk factor interventions on what they may need to work on. And in that sense like a perfect public health policy, well, the right way of doing it and a really good use of surveys. And I have a few questions around it, and this is pure naivete, I haven’t done a survey, but how did you target the people who received the survey or did it go to everyone over the age of 18 say, or how did you know who to send them to?

Dr Larissa Bartlett:

So anybody who’s aged 50 or over and lives in Tasmania could join, so we just had a big campaign to, and it’s actually only just closing now. So we’ve had enrollment into the cohort open for three years and we’ve done a lot of community engagement, lots of talks in community, a lot of media and social media, and so encouraging people to come and join. Yeah, it is really cool. Because we use the survey method, we’re able to provide people with a thing when they’ve completed their surveys, which is a report that tells them kind of a traffic light sort of version of the report that tells them against nine of the domains, that they can do something about what the World Health Organization guidelines suggests they might do to shift into a lower risk category. And so they take that. And what we’ve found is that a lot of our participants are talking to other people about what they’re learning. So that’s actually something that we’re trying to find out a way of capturing the real effect, the flow and effect of that kind of thing.

Dr Donncha Mullin:

Amazing. And do you find that out of the 8,000 or so, am I right, 8,000 or so have completed the survey so far? So there are about a half a million people in Tasmania and if it’s 50 plus that’s quite a good proportion of the 50 plus probably is it 8,000 and do you know how representative it is of the overall population?

Dr Larissa Bartlett:

Yeah, that’s two really good questions. One is, so if we were to achieve 10,000, we would’ve had 5% of the population aged 50 and over, so that would’ve been kind of handy then if we had, sorry, the other question was?

Dr Donncha Mullin:

Oh, just around the national representativeness of it. Whether that was a goal or not?

Dr Larissa Bartlett:

Look, I think it needs to be, I think then we know it’s not entirely we have, because it’s an online research program run by university people. We have people who are prepared to engage with that environment. And there’s no doubt that we’ve actually done some looking at behavioral phenotypes and we have a much higher proportion of people in the cohort who have only one or two or maybe three risk factors. And then we’ve got a small proportion of the cohort that’s got quite a few of them. So we’re very aware that there is a skew and we just need to account for that in our analyses. And it also, I guess, gives us, well, what can we learn from these people that we can then transfer into community perhaps in a way that’s more accessible, but yeah. Achieving. Yeah. So we have to work with what we have that but understand that there are some limitations.

Dr Donncha Mullin:

Sure, sure. I’ve yet to see a longitudinal cohort that doesn’t. I don’t think it exists.

Dr Larissa Bartlett:

Indeed. It’d be a dream to have true representation, but again, for us to be able to do this in the long term, we’re trying to, part of what we’re doing is trying to create a sense of community in our cohort now. So that they’ll stay and then, yeah. So it’d be very hard to do longitudinal studies with full representative samples, I think.

Dr Donncha Mullin:

Yeah. But so many interesting things, threads that I could pull on from all of that. It’s fascinating. I think you’ve covered a lot of key things around surveys, even just that last point of maintaining your and reducing attrition by making it worthwhile to take part because they are time consuming.

Dr Donncha Mullin:

Cool. So that’s a really brilliant description of the method and a great example of how it’s being used from Petra and Larissa. So to get into the detail and provide some top tips for anyone new to using the method, I’m going to ask some straightforward questions of both guests on how to put the method into practice.

Dr Donncha Mullin:

Petra, the first ones are for you. Is survey a robust method to inform research, or do you think there are better approaches?

Dr Petra Boynton:

It’s going to depend on your research question, but I could listen to Larissa talk about that research all day. It was just so fascinating to hear about the whole approach and how to get people involved. And for me that’s the bit, I always find interesting about surveys. I think when people think about the survey or they’re told there’s a class on surveys, they think it’s going to be something maybe about statistics or maybe it’s something about design, but they’re not really sure. And they see it’s quite dry and boring, but that whole sense that Larissa was saying about building a community and getting representative people involved and the capacity to be able to get data, so much data online, but then who that might bring in or leave out those questions for me are the interesting ones. So I think any method can be robust if it’s the right method for the question and we are using it in the right way.

Dr Petra Boynton:

I think surveys get a bit of a bad reputation because again, when we were listening to Larissa’s talk, what’s hidden behind that is it’s taken three years to get those 8,000 people. It’s not going to be just Larissa doing this work, the volume of work to find people and get them involved and keep them there and make them feel included and ensure their data’s used. And that lovely sense of them having, not just the sort of involvement in the survey, but the fact that they get some answers about their own health back so that they feel really included. That to me, is part of the sort of robustness in that it’s making everybody feel part of the job, part of the research we’re sort of co-producing this project. But I think what most people think about is probably what they do in a lot of cases, which is I need to find out something it’s just crossed my mind that I’m interested in it. I’ll quickly type out a few questions tonight. I’ll send that out on a link and that’s my survey done.

Dr Petra Boynton:

And then they suddenly find the question was phrased badly or nobody wants to reply. So there’s an awful lot of stuff that has to be done very early on in surveys and people are, I think always surprised to make it robust, the amount of planning and designing and piloting the ethics, thinking really, really carefully about what you are going to do and who you’re going to do it with. And I think, again, crucially, it could be a really good design. It could be really robust. It could be the right thing to be using, but if the participants don’t want to use it, then you are kind of a bit stuck.

Dr Petra Boynton:

So that is also something I think people need to bear in mind that just because you would like to use a survey doesn’t mean your participants would. And I found in my own research, sometimes we want to avoid a survey because we feel it’s too constraining. We want to let people chat and talk, but actually they’d rather have the constraint of a survey, particularly on very emotional and sensitive topics. But equally sometimes when you give them a survey it’s in the wrong language or it’s designed in an inaccessible way. And we can maybe talk a bit about that in a minute around how to make it more accessible.

Dr Donncha Mullin:

Yeah. I’d love to think about that. And you’ve mentioned some key things there about when you’re preparing to produce the survey and before you just dive in and make your survey monkey and send it to everyone which I’m guilty of in the past, you mentioned piloting questions, ethics around the questions and then checking the design. Are there other key things that someone should think about before producing a survey?

Dr Petra Boynton:

Yeah. I think one thing that people really don’t think about often enough and I think it’s the way that we are taught research is that we’re always taught that it has to be new, that you’ve got to do something novel and people interpret novel as in everything has to be new. Whereas I think at the moment, if I was interested in trying to do a tracking piece of work and find out how people were coping and planning their health and their sort of health outcomes, it would make sense that I would look at what Larissa has done and see if I can use that research model and maybe even the same questionnaires or adapt them slightly rather than thinking I’m going to just start completely from scratch in health and development. In other areas, there’s quite a lot of standardized measures we could be using and that can make sense.

Dr Petra Boynton:

We can use them alongside additional questions that we add of our own, or it might be that we take some time to develop a questionnaire ourselves. But I think it’s not a quick method. I think people think it’s really quick and it’s easy and it’s about thinking of some questions, putting them either a paper questionnaire or an online questionnaire, and then off you go. Whereas actually you need to be thinking of kind of the end point. Why are you doing this? How long is it going to take? What do you want to happen from this research, who is going to be in it? How are you going to find them is a survey, the best method for them? And if it is in what format is it online? If it’s online, do you need video? Do you need audio? Do you need descriptions? Do you need text? Do you need visual scales? How are you going to instruct them what you want me to do? If it’s pen and paper, how are you going to do that? Is it postal old fashioned?

Dr Petra Boynton:

But in pandemic days, lots of people went back to that, right? Are you going to use a researcher? Is somebody going to be there on the phone or on the computer typing in the answers or sitting by the person, all of those things will make a difference to how many people join in and how included they feel. So I think a lot of the time people assume because it’s an easy method and they’ve done it a lot. They don’t really need training. They don’t really need much help. They can maybe read a book or just try it themselves and see how it works. So yeah, I think that would work in terms of thinking about what makes it so interesting for me is also all of the stuff that goes into making it robust.

Dr Donncha Mullin:

Brilliant, brilliant. One thing is so many good things to ask yourself before divine. I imagine most PhD students ask themselves those questions, answer them right and thoughtfully that they’ll be after a very good start. So another part of the start of a survey is the introduction. And you’ve both mentioned about keeping participants involved in a study are having them complete a survey once started. And I imagine the introduction is maybe where you get that across, the importance of it and things. So is there a certain approach you think is helpful when writing an introduction to a survey?

Dr Petra Boynton:

I think before you even write anything, again, it comes back to who are your participants? What are their specific and diverse needs? And there may be multiple needs. So this may not be just one approach. There might be lots of ways you’re going to need to reach people and in lots of different places. And then how are you going to reach them? So that’s your recruitment strategy. And again, I think a lot of people forget that you need to do that. And it’s probably why, if anyone’s on social media, you see endless calls each day of please complete my survey with the link. There’s no information on the survey, you don’t know what the link’s going to be about half the time it’s not in any way relevant to you, but people are just desperate for numbers. And I think you need to think about, are we getting the right people?

Dr Petra Boynton:

So, I mean, Larissa was really clear in describing who were the people that needed to be the age group, where they were located, what they needed to do. Those are the things to think first. And then you need to think about how can I make this interesting and inviting. I found quite often there’s a disconnect between what charities or universities or organizations think is a priority. And that’s lots of disclaimers and long, long winded explanations and loads of logos. And every partner in the study has got to have a logo on there. And participants can be less interested with the very long ethics committee number and the fact that you’ve got this approval and that approval, and you are this many ranked in the world in the university and there’s 57 different icons in your description. They want to know what’s in this for me. Why should I sign up? Why is it interesting?

Dr Petra Boynton:

So the bonus of this might tell you something about your health or you could help other people or that it might be something fun to do, but this is really why we want you. And because we really want you, you can join in different ways. You can join in different formats. Maybe you’ve got a video of a researcher chatting and saying, “Hello, here I am. Would you like to join in my study?” Certainly I think we need to think again about various ways. You’ve got the kind of advertising aspect. So across social media, maybe on mainstream media, through other networks, through friends, through word of mouth, through sort snowballing, all sorts of different ways. How do you want to get it out there? And have you made sure that’s accessible and ethical? Because quite often people will share a link to something that’s really sensitive.

Dr Petra Boynton:

I mean, even something say we’re thinking about dementia that, say if I’ve recently been diagnosed or I was worried about a relative and suddenly popping into my timeline is a dementia study that might make me feel very excited or happy that there’s something I can join in with and help with my worries. But equally I might feel really anxious that how do they know about that? Or I’m upset enough about this and now there’s this piece of research just dropping in as if it’s not even a big deal for me. So there’s a whole thing of thinking around, getting the information out there. And then when you are sharing information, is it done in the right languages that people speak with different people that are appealing to them, people that seem like them, that they can relate to in different ways that we all feel like if I wanted to join this, it’s not as simple as clicking a link.

Dr Petra Boynton:

I think that’s the big mistake people make. They think if I send a link, people will click on it. They’ll join in. There’s a connection, there’s a relationship, there’s a building. And that’s the thing that takes the time. That’s where it goes wrong. And I think when we teach surveys, we do a real disservice in not unpacking it. This to me, as you can see is the fascinating part. I love talking about this, but we don’t teach this routinely. And then when you do your survey and inevitably nobody joins in or people complain or they get angry or they don’t join in, or they say it’s boring, you are panicking that you’ve only got three people and you wanted 300. All of that’s making you feel like you failed as a researcher. Whereas actually, if someone had taught you to build this stuff as carefully as you would, if it was a trial or it was focus groups, or if it was any other method, I think we probably be much happier researchers.

Dr Donncha Mullin:

And there’s plenty of room for improving the happiness of researchers in the world. So that’s a nice way to finish on that. Okay. Thank you so much again, so much richness. Now I’m going to just ask a few quick fire questions then just to see if maybe there aren’t any, but do you recommend any particular platforms for developing a survey and sending, I guess I’m thinking online surveys?

Dr Petra Boynton:

I think the key thing to note is if you’re in a university, a charity or another organization, they may have a preferred platform you must use. So always check that first. I generally don’t recommend platforms because I think that actually you need to be looking at who are your participants? What’s the most accessible platform for them. And again, it may be that an online survey is not right for you. In Larissa’s case, I can’t think of any other way to do it, but in other places it might be an interviewer sitting there with a laptop is the better way. And they would be using an online platform, but participants wouldn’t. So it’s about choosing the platform, but also choosing what data is generated from it. Because I think we tend to think about front end, does the platform let me ask the questions, but you need to think about backend, which is what is it going to give you in terms of data, if it’s all scrambled and hard to follow or you’ve got to do loads of recoding, you might want to investigate others. So again, that’s another thing to think about.

Dr Donncha Mullin:

Brilliant. And you’ve sort of alluded to it there as well that sometimes online isn’t the way forward. Other times it’s ideal. And you mentioned earlier about during the pandemic, there were some more postal surveys being done, more old fashioned. Are there any particular considerations for a postal survey, such as text size or readability or things that online you can get around maybe by using some translating app or something, are there certain things you need to consider now with a postal one?

Dr Petra Boynton:

I think that the biggest thing with postal surveys is they are very, very unreliable in terms of response rates. So I would say that I would only use them, if it’s really what participants want and yes, you absolutely have to think about only printing on one side. So if you print on both sides, quite often, people miss the second page. So one side, big fonts, much bigger than you think you’re going to need. And very clear instructions with worked examples. So don’t say to somebody tick this answer, because they don’t know whether you tick at the side or in the box, you have to show them what you want them to do.

Dr Petra Boynton:

But it’s interesting you mentioned about font size and things for online surveys, because again, I think a lot of people don’t think about accessibility and they assume that everybody can expand text on their screens. A lot of people do these surveys on their phones. So actually using visuals, using one question per page, there’s lots of things you can do to make it much more accessible. It looks longer, but it actually goes quicker. And again, thinking about either video descriptions, so people are showing you what to do or if people are using screen readers, a lot of online surveys are terrible for screen readers. So you’ve got to think about that too.

Dr Donncha Mullin:

Okay. Amazing. Okay. Well thank you so much for those answers, so much to consider and so much good guidance. Larissa, it’s your turn. Are you ready for a few similar quickfire questions?

Dr Larissa Bartlett:

Sure. Hit me.

Dr Donncha Mullin:

Well, many, many of our listeners will be seeking the views of older people carers or are those living with dementia. What special considerations extra that haven’t been kind of discussed do you need to make, to get input from these people?

Dr Larissa Bartlett:

Such an important question. I think I’m not sure I can… I just need to reiterate what Petra’s been saying. You really need to know what your research question is and who it is you need to ask the questions of and then put in place a recruitment and the participant engagement and management strategy that suits both people. I would not be expecting anybody who has dementia to be participating in Island. Although I suspect that in time, over the 10 years of the study, we may find that some of our participants will. And so there’s second component to Island, which is a diagnostic clinic for dementia that’s associated with the prevention clinic study. And so we have a way of, we ask people each year how they feel about their brain health and we provide information about how they can access advice from the clinic.

Dr Larissa Bartlett:

If they’re interested in finding out more, it’s not within my capacity as a desk researcher to provide the kind of sensitivity and care that would be needed to administer a series of standardized questions for people who are living even with mild cognitive decline, really. But that’s what does happen in the clinic. And it’s a research clinic and those questionnaires, we have quite a lot of common questionnaires in the Island and in the clinic. And so the questionnaire is administered, but done so with a clinician and/or with a carer presence.

Dr Larissa Bartlett:

So I think that necessarily limits the scale of the kind of studies that you can conduct with people unless you’ve got a lot of money, but it provides you really just need to research, do no harm. Absolutely everything we do when we engage with our research participants is about, we are researching, we’re gathering their information, to inform knowledge and we need to make sure that what they get out of it is a sense of satisfaction and contribution. And yeah, I’m not sure what else I can say. I don’t think that’s the rule of thumb. I think you just have to be really aware and find out what needs to be done to make sure that you have that safety in place.

Dr Donncha Mullin:

That’s really thoughtful and considerate advice. Thank you. And a major challenge of surveying people and we’ve all sort of talked about it already is ensuring you get enough responses. Now you’ve had what seems like a fantastic response rate so far in Island. And what advice would you have for people to get more responses from survey participants?

Dr Larissa Bartlett:

Yeah, it’s big challenge for us. I think with every longitudinal study, we’re trying things out, we’re going into community to provide feedback in community and also for online through webinars to our participants to tell them what we’ve learned from the information that they’ve given us and what we are doing next and what we’ve planned to do with it. So we combine a fair bit of the data collection with telling people what it is we are learning and thanking them for being involved. Still in that context, we’ve still got about half of the sample hasn’t been back beyond baseline, which is a very high attrition. So we are working, we’ve still got all of their, well, the other thing is that our only mode of communicating with our participants is by email. So we know that there will be people who provided an email that’s no longer active.

Dr Larissa Bartlett:

So there’ll be a proportion of people we just actually can’t contact. We wish that we had also collected a mobile phone number at the time of it, of that baseline. Looking back on that, more than one mode of contact would’ve been a wise thing to have included in that baseline survey. We’ve actually just decided that we’re going to offer an incentive and just five lots of $100 vouchers to Coles Myer, which is very easy to use card here in Australia. And so they go into a draw if they complete the next wave of surveys coming up in October. So we’ll be putting that out there to see whether it’s stimulates a little bit more engagement by people who haven’t been coming back. We’ve got about 3000 people who are just ironed on. They do everything we ask, which is great, but we need the others in order for it to be really interesting. I mean more interesting.

Dr Donncha Mullin:

Sure. And you mentioned there collecting a second contact detail and I imagine it’s not just a case of, oh, we just asked that now. I imagine there’s ethics protocol and processes around that. But in your experience, what did the ethics committee look like? Or what would it look like for those reviewing survey study applications?

Dr Larissa Bartlett:

I think a lot of applications for research would be survey based. So I would imagine that people on ethics review boards would be familiar with the survey approach. So there may be some considerations about, well, why have you chosen this survey? But that would be because the protocol doesn’t defend the choice of survey sufficiently. I think responder burden is a really big consideration and it’s something certainly in our study group because we collect so much information from people. It takes an hour to do the annual survey. So it is a lot of time. And in conversation with ethics, we’ve structured it so that people can finish a survey and then have a break and come back later and they can start back again where they left off. I think that responder burden and making sure that there’s a good fit.

Dr Larissa Bartlett:

And also some questionnaires can, as Petra was saying earlier, they can particularly, if they’re get digging into a sensitive topic, they can be a little distressing. And so you need to wrap them up with support. And so again, it’s just that making sure that there’s the safety of the participant is front of mind and from an ethical perspective, if you’re not going to use the data, even if it’s interesting, don’t ask it. It’s not fair to collect data from people just because, oh, you might also want to know about that, but you’re not actually planning on using it. And it adds to the responder burden when the Island Study was first launched, it was launched as a series of surveys. So it was a string of surveys and it was at the Wicking Center and we have the preventing dementia and understanding dementia MOOCs. And so the learning management system that supports those MOOCs we used these surveys were kind of embedded in that learning management system, but it wasn’t the-

Dr Donncha Mullin:

A MOOC, is that the massive open online course? Is that the idea? The free course?

Dr Larissa Bartlett:

Thank you, acronym. Yep. Yep. Very fair pull up. But that learning management system is customized for people who are going in to do an educational program and that’s what they’re intending to do, but the surveys were not set up with the same intention. That didn’t fit. So people, when they came to do the surveys were presented with something that was, oh, I’m not doing a course. And so I think we lost a lot of people who might have been interested to start with, just that.

Dr Larissa Bartlett:

So one of the things we did in that very first year was very quickly we actually closed off for recruitment and we created an Island Home using the learning management system, but actually creating. So every participant has their own profile page where their dementia risk profile is stored for each wave of surveys where it’s familiar. So all of the study information that they want, plus the access to their newsletters and events that are coming up and things are all in the one place. So every time we assess, we get them to come back and log back into Island Home. And I really wish we’d had that in place when we first launched recruitment because we would’ve perhaps kept some of those people in the first [inaudible 00:43:44]

Dr Donncha Mullin:

Yeah. That sounds like really good learning and adapting and then benefiting from reflecting on it. It sounds ideal. You’ve mentioned that it can sometimes take about an hour to complete the surveys. I’m imagining there’s a lot of information coming in, a lot of data. And how do you approach analyzing the results?

Dr Larissa Bartlett:

I think I need a clone. Clear the data. So I’m very fortunate. I have work with some awesome data managers and, as Petra was saying before, the planning that goes on behind the collection of survey data is just so much work that goes on. So we are not just planning all of the experience out the front. We’re also planning how the data’s going be collected and managed particularly for a long term project over time and with a complicated rolling baseline. So what I do is I talk to Tim and Alex who do a fantastic job of documenting and securing the waves of data. And I ask them to give me access. So there’s nobody in the Island team that accesses the raw data from the surveys because of this complex nature of the way in which it’s collected. So it’s held under some fairly strong, strict data management protocols and we send a request in, and I want these variables for this reason, I’m going to do this with it.

Dr Larissa Bartlett:

And that way they know what shape I want. That way they can provide me some information about how to interpret the shape of the data. So we clean it. We have a look at is what I’m looking at following a normal curve, or is it skewed or whatever, so that we can understand what analytic method we would use to answer the questions that we’ve collected the data for. So you always intend to use this strategy when you get your data and it might not be quite that shape. So yeah, it’s working out, there’s quite a lot of play, not play, time involved in doing that prep work. And then yeah, you’d let it start telling its story. I love that bit.

Dr Donncha Mullin:

That must be so exciting when you just see the pattern and answering in real life. Okay, awesome. The last wee question is, and it’s maybe related to that planning how the data looks like at the end, but do you find that your survey in Island has more open ended or closed ended questions?

Dr Larissa Bartlett:

Most of the data that we generate through the surveys is quantitative. So mostly numeric. We do have short text responses to some variables, which we can then code and treat and use in analyses. We then also in almost it, yeah, wherever we feel like it might be, whether we want to get some more information about these experience or we feel that patients might want to tell us something more about what happened for them we provide free text boxes. I guess we’ve done some really great topic modeling with data that comes through from not so much through Island, but through the massive open online courses, which is looking at the feedback from participants and doing some wonderful establishing of themes and mapping of the kinds of content that people discuss in those online settings. But so doing the preventing dementia MOOC is one of the interventions within Island, but for Island itself, we don’t collect a lot of qualitative data under the main study protocol.

Dr Donncha Mullin:

Sure. Okay. Cool. Thank you so much.

Dr Donncha Mullin:

Okay. So I am going to try and recap some of that fascinating guidance, stories of putting it into practice and general advice into my few top points, five learning points. So for me, some of the areas that I learned most about is that surveys can be as robust as many other research methods, depending on the research question, they can be the most appropriate approach. But the important thing to bear mind is that a good, robust, relevant survey, they’re not quick to design. So although that’s what everyone probably says and thinks that the outset there can be a lot of work to ensure that it is actually a good survey. Sure. Anyone can produce a pretty rubbish survey in a coffee break, but to actually have the right questions to design appropriate, ethical, thoughtful questions that will capture what you’re looking for. You need to spend quite a lot of time and that involves trialing the questions, piloting it, and tweaking, as necessary.

Dr Donncha Mullin:

Along with that, so second point for me was about the introduction section. So I think I probably had in mind that you said, oh, please, thanks for taking the time to answer these short questions. And this is how long it’ll probably take you, but there’s so much more to it. If you wanted a big survey that takes a lot of time, you have to give something back to the participant and tell them what the pros and cons are of taking part things that they may benefit from, things that might worry them. Things that might be an ethical issue and explain that upfront. And then as one of the participants said to wrap it with a safety mechanism around that, a support mechanism around the survey. So that idea of a much more detailed third introduction section is really important.

Dr Donncha Mullin:

And then finally, everyone has talked about generating a sense of community among participants and how that can help prevent withdrawal from the study and probably give a much bigger sense of participation for those who take part in a sense of community. So that can take any form that sense of community, I guess. And it depends on what way it would work for the participants and the people you’re asking the questions of. Larissa gave the example of having a platform where each participant has their own individual profile, where they can track their dementia risk. And I imagine how interesting and how much ownership that will give each participant over that data, and this sense of this is a valuable thing. I’m not just giving my information away to someone, but it’s being organized for me. And that’s a really good service to provide for participants.

Dr Donncha Mullin:

I know I’ve said finally, but finally planning what the survey answers will look like afterwards. So the back end of the survey is also something that can take a lot of time at the outset and probably a lot of planning meetings with data scientists and with the people who are going to be using the data. But it sounds like it’s massively worth spending time on that beforehand. And that probably every minute you spend on that will save you an hour of pain and suffering afterwards when it comes to analyzing the data, it seems like, yeah, if you can ask questions in certain ways, it’ll make it easier and more logical, or even just possible to analyze at the end of the survey, then you’ll be doing yourself a lot of good service later on. So hopefully the information we’ve already shared will ensure you don’t face any issues. However, in this final part of the show, we’re going to discuss some common pitfalls, challenges and how to avoid them. Petra, what are the common pitfalls and how do you avoid them?

Dr Petra Boynton:

What I think we’ve probably covered an awful lot in this discussion, but I would sort of sum up by saying, one of the big pitfalls is trying to do this all yourself without any kind of testing or planning, jumping straight in thinking it’s easy. I hope we’ve demonstrated that even if you’re doing a very small questionnaire, not anything as large as Larissa’s taking on, which is phenomenal. Even if you were doing a few, maybe 50 people or something like that, you still need the quality, the planning, all the work, working with colleagues, people who know how to do data, all of that stuff together, people who are expert in design is important. I think I would just add on top of that, ensuring that things about accessibility is really crucial. I know I’ve said that, but it is missed so often in research and we’ve talked a lot about participant wellbeing and participant safety, but I think we also need to remember researcher wellbeing.

Dr Petra Boynton:

And that might be that the nature of the questions you are asking is quite draining or demanding or upsetting. So you do need help, even if you’re getting lots of quantitative data back, you are still thinking about what that quantitative data means, and that can be quite challenging. But separately, I think that if you’ve got low response rates or if you are struggling with your data or if it’s a very stressful time or that in a lot of these projects, you’re under considerable pressure to get lots of information in, is making sure you’re not coercing people in because you just need to get those numbers up. And you’re not being made to feel bad if you are not actually getting that information back yourself.

Dr Petra Boynton:

So I think all of it’s really about getting supported. It’s about being trained. It’s about being supervised. It’s having space to reflect and hopefully really enjoying the method. I hope we’ve shown through this discussion that it’s so interesting. And there’s so many angles to it. If you’ve got other people, you can share that with it can really help on the days when it’s a long, hard slog, it’s a hard method to use. It’s not an easy method.

Dr Donncha Mullin:

And Larissa, you have already covered a lot of really important learning points and things you’ve adapted and changed along the way to improve the study. Are there other things that you would like to impart on potential survey researchers that may help them?

Dr Larissa Bartlett:

Thank you. There is one thing Donncha. I would like to say, first of all, forgive the absence of my moving face. My internet connection is a little weak, but the use of survey data for researchers is really powerful. We can draw our conclusions from it. And then the benefit of using the same measures across multiple studies. So the same questionnaires to generate data from multiple studies and pull that data together is undermined if you change your surveys, if you change the questionnaires. So the best thing I remember doing a meta analysis in my PhD, where I went in and yeah, it was so frustrating because it’s quite common for people to adapt or change or adjust an established questionnaire with probably good reason to try and make it a little more relevant to their participants or to their research question. But actually what you’re doing is undermining the ability to answer what a lot of people say in response to these questions, which is the great benefit of pooled data.

Dr Larissa Bartlett:

So I guess there’s a lot of work that goes into validating psychometric scales, and questionnaires that are used in social and behavioral research. And yeah, if you’re actually going to adapt one it would be good to validate it. But yeah, it was a bug bear of mine doing a meta analysis of data generated from measures that had been somewhat adapted.

Dr Donncha Mullin:

Okay. Really valuable lesson. And it sounds like it would prevent people reinventing the wheel if there’s already a good survey there, use it. Okay. Well thank you.

Dr Donncha Mullin:

Folks. This has been really, really great. I’ve learned so much and I hope our listeners have too. It seems that clearly a well built survey can provide valuable information, but if you get it wrong, you could be wasting your time and maybe more importantly, your participant’s time. Hopefully listening to today’s podcast has saved all of you yourselves some time in your jobs and some frustration and will help in your research. Just before we finish up. I just wanted to ask Petra to give us a quick rundown, to tell our listeners in one minute, what this should go away and read to further their knowledge on this method.

Dr Petra Boynton:

I would say, go look at surveys in the wild. We’re blessed with so many of them now. So go and look at them. Think about what would you do differently? Who do they leave out? Who could they bring in? So that’s one thing. If you’ve got any opportunity to be trained in surveys and you want to use surveys, I would always recommend it. As we’ve heard here, all of us do a survey and at the end, think I wish I’d done this differently. So talk to people who’ve used surveys, get trained up.

Dr Petra Boynton:

And I would say, I wrote a guide years ago, a three part guide that was in the British medical journal called the Hands On Guide to Questionnaires. And although it’s a bit outdated now, the very basics of what we’ve been talking about are set out there. And also Helen Carra has written a lovely little book. It’s only $7.99 and it’s called Write a Questionnaire, a Little Quick Fix. And that’s quite a nice little guide book to just talk you through the basics. But as I said, I would never attempt doing a survey small or large, like Larissa’s doing without a lot of support and training too.

Dr Donncha Mullin:

Okay. Brilliant. Well, that is all we have time for today. Folks, I would like to say a huge thank you to our guests. We’ve had the wonderfully helpful Dr. Larissa Bartlett sharing your experiences and in Expert Corner, the incredible Dr. Petra Boynton. Thank you both. It has been a real pleasure.

Dr Donncha Mullin:

If listeners want to know more about survey methodology or more about the NCRM the National Center for Research Methods, dementia research or our guests today you will find all the links in the show notes. And remember, if you found this useful and learned some stuff, then please share this podcast with your friends or leave a review online and subscribe to the Dementia Researcher Podcast. I’m going to end the show by encouraging you all to complete our short survey, to share your feedback on the show. Only joking, have a great day.