1  Reproducible experiments

We use different technologies to develop experiments. Some examples are Psychopy, Qualtrics, Limesurvey, jsPsych, Gorilla, etc. Each of these has advantages and disadvantages and, in general, there are pragmatic aspects to take into account when adopting one or the other: cost, type of experiment (EEG or behavioral, lab or online), lab history and available resources, …

In our lab, we opted for jsPsych to run behavioral experiments because it is an open source javascript library, based on standard web technologies, and can be used online and offline.

In the last years, we started working on a set of tools to help people without coding expertise to create jsPsych experimental protocols (jsPsychMaker), simulate participants (jspsychMonkeys) and standardize and automatize the data preparation and analysis (jsPsychHelpeR).

Our final goal is to have a big catalog of tasks available to use in the jsPsychMaker repo. Each of the tasks should run with jspsychMonkeys to create virtual participants. And each task will have a sister script in jsPsychHelpeR to fully automate data preparation (re-coding, reversing items, calculating dimensions, etc.).

1.1 Open and reproducible pipeline

To replicate an experimental protocol from a publication is not trivial. Obels et al. (2020) checked the computational reproducibility of Registered Reports in Psychology. From 62 articles meeting the inclusion criteria, only 21 had both data and code, and could be computationally reproduced. One of the main goals of jsPsychR is to be able to create, share and reproduce an experiment, its data, and data preparation and analysis without any extra effort. If recent calls for Journals to assess computational reproducibility are successful (Lindsay 2023), this should be an unavoidable aspect of researcher’s work soon enough.

Furthermore, all the components of the pipeline are be Open Source, which allows reviewers, collaborators, etc. to check and run the code. This also makes it accessible to anyone with a computer connected to the internet, eliminating cost constrains.

With this system you can create a paradigm, simulate data and prepare data and analysis almost automatically.

The system output is standardized, so names of variables and the structure of the data are predictable. Finally, the plots, tables, reports and analysis are reproducible, so you can get everything ready with simulated data, preregister or even better, go for a registered report and just relaunch the data preparation and analysis when the participant’s responses arrive, with a single command.

And if you want to share the final data preparation and analysis project in a Docker container to make sure the future generations will be able to run it without dependency issues, we got you covered.

1.2 Automatization

We tried to get a few basic things right, but this is an evolving project, and some things are more complex than one would want. Please do report the issues you find:

• jsPsychMaker issues
• jsPsychMonkeys issues
• jsPsychHelpeR issues

SOURCE: https://xkcd.com/1425/

Lindsay, D. Stephen. 2023. “A Plea to Psychology Professional Societies That Publish Journals: Assess Computational Reproducibility.” Meta-Psychology 7 (September). https://doi.org/10.15626/MP.2023.4020.

Obels, Pepijn, Daniël Lakens, Nicholas A Coles, Jaroslav Gottfried, and Seth A Green. 2020. “Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology.” Advances in Methods and Practices in Psychological Science 3 (2). https://doi.org/10.1177/2515245920918872.