Common methodological problems in management research
Increasing methodological sophistication of quantitative research has prompted many journals to introduce specialized methods review procedures. In this presentation we take a look at how this was implemented in Journal of Operations Management, where the presenter managed the methods review process as a department editor. We analyze the first year and a half of editor decision letters (2018-2019) from methods review to understand what are the most common methodological problems in quantitative research articles submitted to the journal (Study 1). Thereafter, we apply these findings to the three years preceding the introduction of methods review (2016-2018) to assess how common these problems are in studies that have only passed a traditional review process (Study 2). We find that the traditional review process is effective at filtering out research design issues, but not as effective in eliminating suboptimal or even erroneous data analyses, justifications for analysis and design decisions, or results interpretation and reporting. Based on the two studies, we present a set of recommendations that authors using quantitative research methods should follow. We provide video lectures explaining these issues as online supplementary material. While the empirical context of the presentation is Journal of Operations Management, the findings are applicable more broadly in other areas of management as well.