Methods Matter Podcast – Multilevel Modelling

Leah Fullegar:

Hello, and welcome to the Methods Matter Podcast from Dementia Researcher and the National Center for Research Methods. The show for people who didn’t pay enough attention during the methods lectures, and for those who did but just love to consume knowledge while they’re running, cooking, driving or dodging whatever they should be doing, which for me is writing up my thesis. In this series, we’re looking at five different research methods with an expert from the field and a dementia researcher that has put that method into practice. And it’s all delivered to coincide with the fantastic NCRM Research Methods Festival.

Leah Fullegar:

I’m Leah Fullegar, I’m a PhD student at the University of Southampton and I researched dementia care and faecal incontinence. This podcast came about when I got to the research methods section of my PhD and realized this is going to be a lot harder than I thought. So now I’m searching for answers and rapidly realizing that there are many other ways, probably better ways I could have gone about my PhD.

Leah Fullegar:

Today we’re joining the dots, layering the data, putting away the curves and going linear to discuss multilevel modelling. And helping us today are two amazing guests from opposite sides of the Atlantic Ocean. In the X back corner from the University of Bristol, we have Professor Bill Browne. Bill is a professor of statistics who works across many disciplines, including education and animal welfare and behaviour. His research spans the area of statistical modelling from the development of statistical methods to fit realistically complex statistical models to describe real life problems. And the implementation of those models in statistical software, I’m going to really struggle to say the word statistical. But most importantly, he actually wrote the book on this. Hi, Bill. Thank you for joining us.

Professor Bill Browne:

Hi, Leah, lovely to be here.

Leah Fullegar:

In preparing for this discussion, I of course, Googled you and I was fascinated by your recent publication, Aerosol and Droplet Generation from Performing with Woodwind and Brass Instruments. That sounds like something no one would have ever considered. Did that involve an actual experiment?

Professor Bill Browne:

So Leah, one of the joys of being a statistician is that we get to work with loads of great collaborators from different disciplines. And during the pandemic, last year, I was approached by colleagues in our school of chemistry. And they were working with medical researchers on aerosol particle movement in terms of the size of the particles and the number that were being produced, and obviously that has a big impact when you think of COVID. So they research was clearly going to be useful. And initially they were looking at the differences between when you speak and when you sing. Before then thinking about well, what about in the UK, for example, concerts were stopped and everything else so they moved on to musical instrument playing. And they were even on our local news where they were showing experiment with participants who repeatedly had to sing happy birthday.

Professor Bill Browne:

And in fact, it’s really this repeating that is the art of repeat different activities, those are nested within each participant. And that gives their data multilevel structure. And so they emailed me for help with their analysis. So yes, there was an experiment. But sadly, due to COVID and what is often the case being the statistician in the team, I just get to deal with the numbers that come out at the end rather than take part of the experiment.

Leah Fullegar:

I’m sorry, you didn’t get to play any brass instruments.

Professor Bill Browne:

I used to play the bassoon many, many years ago. [inaudible 00:03:38]

Leah Fullegar:

So our second guest today is our hands on dementia researcher, hailing all the way from Penn State College of Health and Development in the US of A, I am delighted to welcome Dr. Jacqueline Mogle. Jacqueline is the co-director of the ReMind Lab, which focuses on promoting health and wellbeing in older adults and identifying early indicators of changes in psychological and cognitive health. Jacqueline’s current projects examine psychological and behavioral risk factors associated with the development of early cognitive decline. These projects are designed to uncover early inflation targets for older adults prior to precipitous declines in everyday cognitive functioning. So hello, Jacqueline, thank you for joining the show. Your work sounds amazing. What attracted you to dementia?

Dr Jacqueline Mogle:

First, thank you so much for having me. I’m really excited to be here, I’m a big fan of the podcast. So this is a bucket list. So I got interested in sort of cognitive health, particularly for older adults and dementia. When I was in undergrad doing my bachelor’s, I got to work with older adults in a memory care facility. And so I actually did a lot of hands on care and working with them. And just seeing the range of functioning. But also, even though these are individuals who had memory problems, they learned me, they were able to recognize me, they figured out my name, all of that, they knew I was a safe person, that I was there to provide care. And I just thought that was really fascinating, that they were losing function and other ways, but they were able to do some of the things that I think we commonly don’t associate with individuals who may be suffering, or individuals who are living with memory problems.

Dr Jacqueline Mogle:

So that was really interesting. And it brought me to graduate school to do this kind of work. But then, while I was in graduate school, I fell in love with statistics and methods, and aging and changes in cognitive functioning are a great place to play around with statistics, and do some of these different methods. So that brings together to two parts of my experiences and the things that I enjoy working on.

Leah Fullegar:

I feel very out of my depth, because statistics is a very, very, very scary dark cloud in the corner for me. And so what do I do, we begin each show with me having a stab at what I understand that this method, and today for some reason, the multilevel part has got me fixated on an Nintendo games from the ’90s. But let’s have a go with multilevel modelling.

Leah Fullegar:

So despite my earlier joke, I think perhaps this method is about connecting what might otherwise appear to be unrelated datasets or analysing data with a repeated measurement. So thinking of this in the context of dementia, this could be the time taken to complete cognition tests with different individuals or groups. It sounds complicated, and something which could perhaps because requires some coding skills. Bill, how did I do? Perhaps you could give us a proper introduction to the method.

Professor Bill Browne:

Thanks Leah, you ought to be thinking there of Pac-Man, but of course, shows my age where you’re connecting dots. And those dots are nested in game levels. So everywhere is a multilevel model. But really, actually, it’s best to start with thinking about data. And what makes that data multilevel in context. So when you go and collect data, and you’re going to analyze it, then many of the standard statistics that you might have been taught, those techniques tend to assume that the data you collect is some sort of random sample that you’ve collected from a big population. And then that data is kind of collected independently. So you could imagine the UK the National Lottery, where there’s the machine with the balls spinning round, and those balls come out at random.

Professor Bill Browne:

But in practice, when we go out and collect data for real, particularly that data is observational in nature, then that independence idea tends to go out the window. For example, if you wanted to answer questions, I work in education these days, and your questions were about, say school education, then you’d have to go into some data collection and construct a list of all the pupils in the UK like the phonebook, and then randomly choose pupils, travel hundreds of miles, just to get one pupil from this school and one pupil from that school. And in practice, what we’re more likely to do was to pick a selection of schools, then maximize the use of each of those schools, by sampling a large number of pupils in each. And this will then result into what we call a hierarchical or multilevel data structure, we’ve got two levels, with the pupils nested within schools.

Professor Bill Browne:

Now if you’re in a more medical context, if you thinking about dementia, I guess we may have, people with dementia who are maybe nested in care homes, for example, and you have a similar two level structure. So having collected the data, you kind of broken the assumption of independence, so we need to adjust our modelling. And this is where the multilevel modelling rather than data comes in. So if you think again, of the education context, if you’ve got pupils of any schools, then on average, two pupils that come from the same school, there’ll be much more alike in many ways than two randomly picked pupils in the whole population.

Professor Bill Browne:

They’ll share lots of things in common, there’re have the same teachers, the school might have the same curriculum, the same ethos, same catchment areas, all of these things might influence the outcomes for those pupils. So the correlation needs to be accounted for. Multilevel models in a nutshell are effectively extensions to those standard statistical models that make some sort of correction to account for the underlying data structures. They can do other fun things, they can answer research questions about the structure, as well as, for example how important the schools are, how much correlation there is in data.

Leah Fullegar:

Okay, I think I get it. So it’s sort of like, well, I’m going to repeat what you’ve just said back, but it’s different levels of data, isn’t it? So you’ve got the sort of the, I’m picturing concentric circles for some reason. One within the other, I don’t know.

Professor Bill Browne:

Yeah. And you could have more than two levels. And then you could have, if you really think of the dementia example, you could have the people nested within care homes, or maybe hospital settings, and those in more larger medical areas, districts, for example. And each of these levels, these structures actually influenced the responses within them, because there is this correlation built in because they share things.

Leah Fullegar:

Is this method is hard to use as it sounds? Because it sounds terrifying.

Professor Bill Browne:

Well, I mean, multilevel modelling, actually, it’s not so much hard for the person, it’s actually harder for the computer, because the mathematics that’s involved in actually getting estimates out to these multilevel models. And the model fitting is much more complicated. So if we go way back into the era of Pac-Man that we mentioned earlier, back in the 1980s, and ’90s, then you could only really fit these multilevel models in specialists statistical software. So teams of researchers in statistics departments, were developing those methodological techniques, and then developing very old fashioned bits of software, which will take a long time to run those models. These days life has moved on computers are faster, my mobile phone is way faster than any machine in the 1990s. So you can do some multilevel models in most statistical packages. The models are more complicated. So you do, thinking from the personal perspective, need a little bit more in effort in actually interpreting what comes out of these models? But generally, it isn’t that much harder than it sounds.

Leah Fullegar:

So what are the benefits of multilevel modelling?

Professor Bill Browne:

Okay, that’s a good question. I think maybe a better question is, what are you losing if you don’t do a multilevel modelling? So what are the disadvantages of not doing multilevel modelling. And historically, obviously, multilevel modelling hasn’t been around forever, people would use simpler procedures and just pretend that they had independence. And when you do that, you don’t account for the cluster. It’s as if you’d had them, got on your bike and travelled to every different school and just got one pupil from each school. So there isn’t any clustering. And so if you assume there isn’t clustering in your data, you are overconfident in what you find, so you can find significant results that are not in practice really they, okay. So you really haven’t properly adjusted for the data you’ve collected?

Leah Fullegar:

Well, what might be an appropriate place to use this method? When might you use it?

Professor Bill Browne:

So I guess, the key point really is, if your data has this nested structure, any sort of dependency in your data requires you to use a statistical model that accounts for that dependency. And in practice, when I teach multilevel workshops to people from lots of different disciplines, they find that actually, the world is full of these hierarchies. Most observational data will have some structure built into it in the way that it’s being recorded. And then multilevel modelling is really a must. So there has actually been something of a boom in articles that use multilevel modelling, to the degree that some journals will say, come back with referees reports saying you must use multilevel modelling here. And one caveat is usually you need a reasonably large amount of data for multilevel modelling. So sometimes smaller studies don’t need to do multilevel modelling. You have to tell the referees, no, I don’t need multilevel modelling. I haven’t got that much data.

Leah Fullegar:

That’s fascinating and I think I understand it. So Jacqueline, thank you again so much for joining us. So you’ve actually used this method. And I’m hoping that will make a bit more sense to me. Can you tell me a bit about that work and how it can really help in dementia research?

Dr Jacqueline Mogle:

Sure. So I have definitely used this method a bunch. My advisor was someone who push this in a big way when I was in graduate school. So I’ve had a lot of experience doing this in and around cognitive health research and specifically within dementia, largely because we aren’t necessarily interested in comparing different groups of individuals, people living with dementia versus people who aren’t, that categorizes people in ways that doesn’t tell us much about how they change over time. So Leah, you mentioned earlier repeated assessments. And that’s exactly the way to think, that’s exactly the way that I think about it in dementia research is that, we really want to follow an individual over time and see how they change. And in that case, just as Bill mentioned, this creates that dependency. So we have the same person who’s coming back to the clinic and completing measures multiple times, across years, there’s a dependency there, all the data is being generated by the same individual. So we would expect there to be this correlation across observations. And that means that you want to account for that correlational structure.

Dr Jacqueline Mogle:

So when we do this, in our projects, we like to look at how cognition is changing over time. And we want to account for the fact that different people start with different levels of cognition. So they’re going to join our studies at different places, in the trajectory of how their cognition is changing. And the multilevel model can accommodate those differences across individuals, so that each person can have their own starting point, as opposed to making everyone kind of start in the same place. And that is, I think, one of the, for me, because a lot of my models are longitudinally based MLM multilevel models. We’re really talking about watching people change over time. And we don’t want to constrain anyone to look like another person when they don’t, so that we can tease apart, why are people sort of starting in different places? Is it different educational attainment? Is it different lifestyle behaviours? We can see sort of that people start in different places.

Dr Jacqueline Mogle:

And to me, one of the biggest strengths of the modelling is that you can also allow people to have different change over time. So you can include one of the big things, and this is more of a technical term. But hopefully the audience can look it up if they’re interested, is this idea of random slopes. So you can allow people to have different trajectories of change over time. And the random slopes, you can do the same sort of idea. So not only will people sort of start in different places, but they’ll also change differently over time. And that could be because of lifestyle factors that could be because of education, that could be because of other things that are happening for them. So we can understand sort of these models can capture a bit better how people start sort of differently in terms of their cognition, but then also whether the recognition is sort of declining slowly or declining quickly, across time, and we can allow our models to accommodate all of those.

Dr Jacqueline Mogle:

So some of the other techniques that have been used in the past, like repeated measures analysis of variance constrain everyone to sort of change in the same way over time. And the multilevel model gives you this more flexibility so that you can look at these other types of trajectories in the data.

Leah Fullegar:

Okay. So what kind of data are you collecting when you’re doing multilevel modelling?

Dr Jacqueline Mogle:

That’s a really great question because it’s like everything, we kind of throw the kitchen sink in a lot of our studies. So we have a tendency to collect these large datasets. So when we get participants in our studies, we treat them as kind of a captive audience. And so we get lots of assessments of cognition, we get other things that we think might be going on. So we’re really interested in this idea of lifestyle factors that might make cognitive change faster or slower for some folks. We’re particularly interested in stress and thinking about, it may not be how old you are that make your cognition change faster, it might be how many stressors you’ve had. And so we’re able to incorporate a lot of those measures in our annual assessments that we can look at things like lifestyle factors change and how that may precipitate cognitive change. Or looking at sort of the accumulation of stress over time and how that might lead to cognitive change.

Dr Jacqueline Mogle:

So the nice thing about the multilevel models is that we can look at things like where people enter the study, what their baseline levels of different lifestyle factors, or previous life history, like education, if they’re not still going to school or something like that, we can look at those factors, but we can also look at what’s going on with this person right now. And is that related to how their cognition is doing? And so we can look at two different kinds of predictors really things that change over time within a person that might change with their cognition, like stress or other things that are going on, like social interactions, physical activity. And we can see at times when someone is having more social interactions, is their cognition better, or at times when they’ve had fewer social interactions is their cognition worse, and that can start to unpack a little bit, what’s going on for a person, not just us sort of at baseline, but then also how they’re changing across time.

Leah Fullegar:

And is there any other sort of specific methods that you would use alongside multilevel modelling?

Dr Jacqueline Mogle:

So I tend to be a multilevel modeler, that’s kind of my identity. You can do other things like regressions, structural equation modelling. So a newer area of work is combining structural equation modelling with multilevel modelling. So you can have these multilevel structural equation models, which are really exciting and new direction, that have a lot more power, they have a lot more data requirements and things like that. But you can do some really, really interesting work and answer some new questions with these more sophisticated types of models. So we run the gambit with all of the different types of models, although I would say multiple models are bread and butter over here.

Leah Fullegar:

And are there any sort of special considerations for using this method in relation to dementia or neurodegenerative diseases?

Dr Jacqueline Mogle:

So I would say, one of the biggest benefits, but also one of the drawbacks to a lot of statistical analysis, when you’re looking at longitudinal data is lost to follow up. So particularly when we’re thinking about dementia or other other types of health conditions more generally, the folks who are sicker are more likely to not come back for follow up, they have a lot more going on. And obviously, and so they’re going to drop out sooner, and they could potentially die or have other problems where they can’t access the assessment. So that’s obviously a big weakness to any analysis, I will say, on the plus side, again, the push for multilevel modelling is that multilevel modelling can incorporate people who have different amounts of data. So even if an individual does drop out of the study, or is lost to follow up at some point, you can still incorporate their earlier data in the model, so that you can actually make your estimates more robust.

Dr Jacqueline Mogle:

So you’re not in a place where the sickest people end up getting dropped out of the model, because they couldn’t complete all of the eight assessments or five assessments, whatever your follow up is. And that’s really important for understanding how these different groups of individuals might be changing over time. The other consideration, and one that I think, just very gently, I think we could do better at, is the sensitivity of measures to change. So in our studies of cognition, we can rely on measures that maybe don’t pick up on change over time as well as they could. And so in those cases, your multilevel model, any variable that you want to analyse with the multilevel model needs to have variability across time. And if you’re using a measure that doesn’t have a ton of variability or doesn’t pick up on change very well, then your multilevel model might not work. But it’s not because people aren’t changing, it’s because your measure is not sensitive enough to pick it up. And I think that is an area that we, again, very gently could do better with, in sort of the cognitive health space.

Leah Fullegar:

That was very gentle. It sounds like that, it can be a lot more inclusive in multilevel modelling in sort of allowing for everything that happens just as part of life during research.

Dr Jacqueline Mogle:

Yeah, and you want to be planful about that. Again, we focus on stress. So we tend to collect samples and think about our samples focusing on stress as one of the big predictors of this. In other studies, when we’re thinking about other lifestyle factors and want to include more covariates and more of these variables that change over time, then you need more people, you need more observations per person. So these studies are incredibly intensive. And so you do have to think about what process is going on or at least you think is sort of theorize so that you can focus your measurement, focus your resources on those aspects of aging or lifestyle or whatever it is you’re interested in.

Leah Fullegar:

Thank you. So now we have a description of what the method is and an example of how it’s been used. Let’s talk guidance and help anyone who thinks this method could be useful for them. In this segment, I’m going to ask some quick, straightforward questions to both guests on how to put this method into practice.

Leah Fullegar:

Bill, you’re the lucky one, you get to go first. How should someone prepare to use this method? I assume a good head for maths is important.

Professor Bill Browne:

And thanks for that. I mean, it was interesting listening to Jacqueline there, the longitudinal data is one of one obviously, aspect where multilevel modelling comes in. And I think really, the first thing one has to check is that your data really requires the multilevel modelling, as sometimes what happens, people hear about multilevel modelling and assume they have to use it always and every time. And if you think of the longitudinal modelling, I mean, you may have to be studying a specific subset of dementia sufferers, which is quite rare. So you’ve got lots of data on three or four individuals, then it’d be quite hard to do multilevel modelling, because you haven’t really got enough data for that. So as long as you’ve got enough data, then multilevel modelling isn’t a problem. And it isn’t really necessary to know that much mathematics. I mean, the computer software should be doing the hard work for you.

Professor Bill Browne:

The challenge comes in a way, in actually interpreting what the computer spits out at you, in terms of the results. So in terms of actual preparation, it’s really is quite useful to revise your more standard statistics, having some knowledge of regression modelling is useful. Once you’ve got some data attending a training course is often a good idea. And our research centre, we actually also produced online training resources, we’ve got one called the LEMMA. And that’s been used by because it’s free, it’s been used by 1000s of people, and it’s felt to be useful. So there are about loads and loads of resources around the world. There are also sites that give lots of materials and I think doing a bit of homework before you start is always good.

Leah Fullegar:

Thank you, and what software would you recommend for multilevel modelling because I’ve heard so much about SPSS, is that one of them?

Professor Bill Browne:

That’s an interesting one. I’ve just finished teaching with SPSS this morning. So that’s kind of a tricky one for me, because we develop software specifically for doing multilevel modelling. So it’d be rude of me to say, like you should use our software, we developed something called Emeraldwin, which kind of just does multilevel modelling.

Professor Bill Browne:

But generally, the most standard statistical software packages, like R and Stata have got very good multilevel modelling functionality. And if you already use those packages for your other statistical work, then you’re probably best sticking with what you know. And in fact, we’ve written little packages within those packages, so that you can use our software within each of those packages as well. The LEMMA course that we produce has got training material and different software packages. So my suggestion always is use something you’re familiar with. The dreaded SPSS, well SPSS we use for more basic stats, I think mainly because universities have got licenses over across the piece. It has got some multilevel functionality, even SPSS, but it is rather limited. So if you are familiar with SPSS, you could go there, but I would probably suggest one of the other packages personally.

Leah Fullegar:

And what considerations should you give when considering the results.

Professor Bill Browne:

So from a multilevel modelling model, you’ll often get sort of coefficients for different things terms in the model, and standard errors, seeing how accurate those coefficients are, just like in regression model. So you can use those to check for significant relationships. And as with all statistical modelling, the models make lots of assumptions and lots of these packages are point and click type packages. So people just point and click, and not really realize there’s lots of assumptions going on in the background. So it can be useful to test those assumptions. So you can plot what we call residuals, how well the model fits in a way, and check if you’ve got any unusual observations, and that they follow the required distributions that are underlying the model that you’re fitting.

Leah Fullegar:

So I’ve had lots of mentions of regression models and I have to ask, how do multilevel models differ from regression models?

Professor Bill Browne:

So I guess most people, many people are familiar with a regression model and it’s like fitting a straight line for a set of points, will be the easiest way to describe it and really multiple models are just an extension of those regression models. So, one thing that we generally would suggest is that when you’re fitting a multi-dimensional model, you can test whether you need it by comparing it with a regression model, which doesn’t account for the clustering in your data. And if it’s no better in terms of its fit, then you might just use a regression model.

Professor Bill Browne:

So that’s one thing that can be seen as like multilevel models can be seen as an extension of regression models, but they can answer other questions, they can identify really the importance of different clusters. If you find some relationship. And you could say, well, does that vary from hospital to hospital from care home to care home? How important is the context that the data is in? So it can be useful to do that and looking to see if the relationships vary across the different clusters? Maybe in an education example, we might find that there’s a gender gap, for example, in exam results. Does that gap vary from school to school, and some gaps maybe unusually boys do better than girls in multiple schools tend to do better than boys unfortunately, for me, but there we are. I think there are lots of examples where we could use a multilevel model to answer research questions.

Leah Fullegar:

Brilliant. Thank you. And Jacqueline, it’s your turn. Are you ready?

Dr Jacqueline Mogle:

Mm-hmm (affirmative).

Leah Fullegar:

So you kindly told us about your research earlier. And how much work did you have to do on the data in advance?

Dr Jacqueline Mogle:

I would say on some level, no more than you would do for any statistical analysis, it feels like more, usually, because you have more data. So you usually have longitudinal assessments. And that means more data per person. And again, because we do throw the kitchen sink at our participants, there’s lots of variables that you could be working with, there are some additional steps like looking at loss to follow up, looking at where is your missing data? Are there any patterns there that you should be accounting for? So that you aren’t biasing the analysis in anyway by excluding people, like maybe you had a particularly difficult cognitive test, and very few people were able to finish it, then that might not be your best outcome measure. And you’ll want to be screening for that. So that you’re not using only the people who could potentially complete a particular test, or something like that. Going back to your question about packages, Leah, I would say that one of the problems with multilevel modelling is that different packages use different structures. So there can be some restructuring issues, depending on which statistical software you decide to use. So I would look for other examples of people doing it in the software that you’re deciding to use.

Dr Jacqueline Mogle:

I’m a SAS user, SAS uses a slightly different setup than other packages. And there are a lot of packages like bills out there that are specialized and do multilevel modelling really well. But again, you need to know the structure so that you’re setting up the data and the way that you want so that you’re not getting lots of errors that you don’t understand. And it’s simply because the data is not quite structured, right.

Dr Jacqueline Mogle:

The other I think, sort of initial step in the data is always looking at that variability question. Like Bill was saying that if you don’t have variability in your outcomes, if you don’t have variability in your risk measures, do you need multilevel modelling, I would argue in longitudinal data, you will almost always have that correlation. So you’ll always need to do some sort of multilevel modelling, but you still might not have enough variability going back to the measure sensitivity issue earlier. So doing some initial steps to make sure that there’s variability there, that you’re able to sort of detected that it’s not sort of tiny, and being able to make sure that the models are going to eventually converge so that you can get an answer to your question.

Leah Fullegar:

And if someone is mining data, or using an existing data set to perform research, are there any particular considerations for that?

Dr Jacqueline Mogle:

I’m going to sound a bit like a broken record. Sensitivity of measures. I think that is a big one, is making sure that your secondary data set has measures they’re going to pick up on change, and that you think are going to sort of perform well in this context. The other ones that I would think about are, how much follow up is there and how closely spaced are your follow up. So sometimes we’ll refer to this as the granularity of follow up. So how are people followed every year, every two years, different studies will have different spacing. And that can lead to different issues. For example, one of the data sets that I work on has a 10 year interval in between observations. So we have this really great aging perspective, because we can follow people over now we have two decades of data, but we only have three observations, really. So even though have a lot of information about aging per se, we don’t have this really closely time data that you might have in something like the English longitudinal study of aging where they follow people every year or so.

Dr Jacqueline Mogle:

So things like that I think are really important. The other thing that I again want to say, again, gently is the age of the data set. And by age, I don’t mean the age of the participants. I mean, how long ago was it collected? There are a lot of longitudinal data sets out there that started a really long time ago, and are still being mined, which I think is fabulous, and is a tribute to the participants who put in that time and energy to give us that data. But we also do know that there are cohort effects. And so we want to think about these different longitudinal trajectories and how people’s cognition is changing, I think in the context of when the data were actually collected. So I think that those are my caveats for secondary data analysis.

Leah Fullegar:

So that’s really helpful, thank you. And data and access to massive datasets has become huge in dementia research as we consider different risk factors and how they contrast and compare different populations. And with some really great platforms like dementia platforms UK, they have high birth and dementia specific cohorts with long histories of cognition test scores, that could be looked at. Or I can even see how things like a supermarket loyalty card or shopping data could be perhaps overlay to look at something like purchases over time maybe, to see if that’s a result of deteriorating condition. Whether I must have met with my loyalty card at Sainsbury’s, they always just send me vouchers for their cheese twists, and it gets a bit much.

Leah Fullegar:

So what have we learned so far, we have learned that multilevel modelling is something I find very confusing. We have thought about words of the different variables that are accounted for and the ways in which it can help us really be quite inclusive with research and make or draw conclusions based on context, and what’s happening around the participants at that time. And I can see Jacqueline nodding, so I’m hoping that makes sense. In this final part of the show, we’re going to discuss the common pitfalls, challenges and how to avoid them, I get the thing, this is going to be one of those methods that someone could grasp really quickly. And once you have a handle on it, it’s easy to use. Or if you’re like me, and you’re more qualitative, it’s completely hard to fathom. And it’s just sort of sat in the corner, and I don’t want to go near it very much. Jacqueline, can you tell us what you did come across in delivering your research and what you do differently if you could.

Dr Jacqueline Mogle:

So I do a lot of secondary data. So I want to be clear that like I love the available datasets that are out there, it’s really amazing to have access to that because we do want to follow people over time. And that means that we have to wait if we’re starting a new study and that can take a very long time. If we want to watch people and watch their cognition change. I think, one of the challenges I have now though, is that there are different types of contexts that I’m interested in, that I kind of wish I had longitudinal data on that we don’t have. And I think we’re getting a bigger perspective and a broader perspective on the types of lifestyle factors, the types of contextual factors, like you said, Leah, that we want to know more about, and that might be impacting cognitive health.

Dr Jacqueline Mogle:

And some of the older studies don’t have that. And so even though those are factors that we think are really important, we don’t have comprehensive data longitudinally on that, yet, I mean, we’re getting there, there are absolutely studies in the field doing this, but I do wish that I had not been so afraid of primary data collection earlier on, so that we could actually have some of that data and have our cohort sort of in the field and collecting that data, because we’re just learning so much from these data sets that I wish we could sort of jump in there and get some more data on some of these things.

Leah Fullegar:

Well, we always love opportunities for further research.

Dr Jacqueline Mogle:

Here we are.

Leah Fullegar:

Yeah. Bill, so you teach this method. What would you say are the common pitfalls? What would you warn your students about?

Professor Bill Browne:

I think Leah, just like all statistical modelling, one of the pitfalls is that people try to get as much information as they can from their data. And sometimes they’re trying to get information that isn’t really there. They haven’t collected enough data. They’ve got a really difficult problem that it’s difficult to collect lots of data in. So for example, in education, it may be true there’s an impact of clustering of pupils within schools and schools are important, maybe on exam results. But if your data set only has three schools, it’s going to be really hard to sort of pull apart school effects from that data, you really need more data to answer that. And then to generalize those findings to all pupils, not just in the free schools that you’ve collected data on.

Professor Bill Browne:

And I think sometimes people struggle with that. They also struggle with working out what is a level in a multilevel marketing, for example, like the schools we’re talking about here. Whereas if you’ve got say a categorical predictor, like say, gender or ethnicity, people think well, it’s another level and not really, gender and ethnicities, it’s categorical, but it doesn’t have, it’s got a very much pre described set of categories. And it’s not like a collection that you’re sampling from. So though those are things that people sometimes struggle to grasp.

Leah Fullegar:

I mean, I think this method could really benefit from some graphic examples. But to both of you, is there any advice you would like to share that hasn’t already been covered? I’ll nominate Bill first.

Professor Bill Browne:

Okay. So, I’m using the word graphic pop ups in a different context here. But when you’re doing statistics figures, and plots are always important. When you look at data, and certainly multilevel modelling, it’s no different. We really advise people to plot their data and look at patterns that exist before they turn the handle and fix the model in the computer package. And certainly in more complex models, plots of what the model is actually showing can explain to the researcher what’s going on, whether they do graphs, and I think Jacqueline mentioned this earlier, random slopes, versus random intercept models, the random intercept model will have lots of parallel lines showing you the relationship for the different clusters. Whereas a random slopes model will show you different relationships going on with each cluster. And there’s also the infamous caterpillar plots that we use in multilevel modeling, which if you squint your eyes enough may resemble the caterpillar, they can show differences between clusters. I’m not sure if that was what you meant by graphing examples. I think figures are important. I like how Jacqueline answered probably properly.

Leah Fullegar:

I’m just thinking of the hungry caterpillar now, sorry.

Dr Jacqueline Mogle:

Yeah, I mean, I guess the advice that I would give is like, don’t be afraid of it. This is a really sophisticated statistical tool, but there are people like us out here who are happy to share our knowledge and think the world of these types of models and think that they are really, really important for us to be using to truly understand what’s going on with folks. Some of the other types of data that I collect are actually ecological momentary assessment data, where we have individuals complete multiple surveys per day for multiple days. And then we do that again in like a year. So we actually get this really intense longitudinal data on folks. And that has three or four levels, depending on how you want to think about that data that can be really intimidating. And I would just say that, like I said, there are lots of experts out there who are willing and eager to see this type of work going on, and who are just really big proponents of these models, and can point you to resources, can point to other experts. And we want people using these models. We think they’re the best and so we really want to get them out there, and they’re not as scary as they first appear.

Leah Fullegar:

I find them terrifying. I must admit, after talking to you both, they don’t seem as scary, it seems like something I could wrap my head around. I can see how it’s a great method to have in your toolkit, I mean that even if this isn’t the primary method you might use, having the knowledge and the skills available to use it presents all kinds of new opportunities, particularly when you factor in the massive amounts of data now available to dementia researchers. Right, so it’s time for our final segment, I’m going to give our expert Bill one minute, exactly one minute to tell our listeners what they should go away and read to further their knowledge on this method. Bill, over to you, I’m starting the clock now.

Professor Bill Browne:

It would be remiss I guess, with me given the NCRM part of this podcast, not to plug their own websites because they have lots and lots of training resources including stuff on multilevel modelling. Lots of videos there as well. Our LEMMA course, I plugged early and certainly we’ve got lots of other resources in the centre for multilevel modelling work website.

Professor Bill Browne:

In terms of books, there are lots of good ones, in the Netherlands there’re lots of Dutch academics they have been more quantitative in their social science tradition. So, there are books by people like Tom Snyder’s and Roel Bosker, and also by Hugh Pox, and some of his co-authors, they’re good introductions, I would say. Our former colleague, Harvey Goldstein, he’s got a very cited book. But I would say it’s a bit mathematical for the beginner. So if you want scary equations, there’s lots Harvey’s book. And then also lots of handbooks out there that focus on particular application areas, and particular software packages, so it’s good to look around, these days there’s loads of stuff on the web as well, I’m sure if you Google multilevel model, you’ll come up with lots of resources, I can’t vouch for every one of them.

Leah Fullegar:

Well, thank you so much, it’s time for a trip to the library, or I’ll probably just YouTube it and watch a video. I’m afraid that’s all we have time for today. So let me say a huge thank you to our brilliant guests who have both opened my eyes to the potential of multilevel modelling. From Penn State, we have the inspirational Dr. Jacqueline Mogle, and of course, our very expert, expert from the University of Bristol, Professor Bill Browne. Thank you both so much for coming.

Leah Fullegar:

So join me again tomorrow for our fifth and final show in the series, which is perfect because I really, really need to do some writing and I can procrastinate no longer. Tomorrow we’ll be discussing a method I have used in my own work qualitative secondary analysis. If you’ve only just found our podcast or are catching up the series, remember all this week the NCRM Methods Festival is taking place and there is still time to take a look. So head over to ncrm.ac.uk for more information, where you will find many of the sessions available on catch up. Finally, of course, I’d also love to encourage you to visit the dementia researcher website for all things dementia and research from career support, research discussion, events, jobs, funding pools, and so much more. Thank you all for listening.

END