The importance of incorporating researcher beliefs into statistical models

Abstract

In this thesis I consider how statistical assumptions are driven by the assumptions the researcher makes about the data. I focus specifically on assumptions surrounding data generation, namely: a) the shape of distribution expected, b) the process by which data were obtained, c) the shape of the outcome distribution, and d) inferring information about missing data. Each chapter of this thesis will focus on one of these assumptions using a combination of tools. I use existing methods and propose new models before exploring from a cognitive perspective the types of inference people make. This allows us to explore the concept of researcher assumptions, and to consider where building them in to the statistical model might be beneficial. In three of the four main chapters of this thesis, I use simulation methods to compare models. The models I consider are both Bayesian and frequentist in framework. The aim of these simulations is not to compare frameworks, but to compare different model structures to ascertain the structure that allows the most accurate claims about the data to be made. There are four main arguments presented in this thesis. First I argue that it is very rare to conduct statistical tests without making some sort of assumption about the data. Second, I demonstrate that for distributional assumptions in a particular type of data, models where the assumptions are not violated can improve the accuracy of the claims made. Thirdly, I present two models that match the assumed generative process of two types of data; contaminated data and data with a heterogeneous effect. I demonstrate that these models are not only more accurate, they also allow the researcher to make richer claims about their data. Finally I experimentally investigate a well-known finding in cognitive psychology|a dislike for ambiguous or missing data. I replicate this preference whilst demonstrating that people are still sensitive to underlying distributional information. Together these findings suggest that the researcher is both sensitive to and makes assumptions about the data. Creating and using statistical models that do not violate the assumptions the researcher makes is important, but building more complicated assumptions into the model can give a richer and more accurate understanding of the data.