Analysis

Data preparation

This is an automatically generated report using the data from the survey about experiment issues.

In the tabs below you can see the frequency and types of issues reported.

Descriptive tables

Table 1: Frequency of issues

Table 1. Frequency of issues and GodLike behaviour

Characteristic	N = 2041
Frecuency of issues
Never	35 (17%)
Rarely	44 (22%)
Sometimes	50 (25%)
Often	39 (19%)
Always	36 (18%)
Who is God
Me	15 (43%)
My research assistant	20 (57%)
?	169
Experience
Student	44 (22%)
Postdoc	58 (29%)
Early Career	61 (30%)
Senior	38 (19%)
?	3
1 n (%)

Table 2: Types of issues

Table 2. Frequency of types of issues. 10 most common issues

Issues in experiments	N = 9531
Issue
Other	94 (9.9%)
Errors so bad that my friend had to start over	90 (9.4%)
Important content errors	90 (9.4%)
Data preparation was harder than it could have been	88 (9.2%)
Some variables or questions asked to participants but not used in the analysis/paper	88 (9.2%)
Match between hypotheses and data not clear	86 (9.0%)
My friends do not have errors in their protocols	85 (8.9%)
Errors in the coding of items	84 (8.8%)
Data not what we expected	83 (8.7%)
Spelling errors	83 (8.7%)
Errors in participants balancing to conditions	82 (8.6%)
1 n (%)

Table 3: Experience & Frequency

Table 3. Frequency of types of issues by Expertise.

Issues in experiments	1, N = 441	2, N = 581	3, N = 611	4, N = 381
Frecuency of issues
Never	6 (14%)	14 (24%)	10 (16%)	3 (7.9%)
Rarely	11 (25%)	12 (21%)	12 (20%)	9 (24%)
Sometimes	12 (27%)	10 (17%)	15 (25%)	12 (32%)
Often	9 (20%)	11 (19%)	10 (16%)	9 (24%)
Always	6 (14%)	11 (19%)	14 (23%)	5 (13%)
1 n (%)

Plots

Plot 1: Frequency of issues

Plot 2: Types of issues

Figure 2: Types of issues

Plot 3: Frequency by expertise

Figure 2: Types of issues

Analysis

Results

We fitted a linear model (estimated using OLS) to predict Number of Issues with
Frequency of Issues and Experience (formula: `Number of Issues` ~ `Frequency of
Issues` + Experience). The model explains a statistically significant and weak
proportion of variance (R2 = 0.12, F(2, 198) = 13.45, p < .001, adj. R2 =
0.11). The model's intercept, corresponding to Frequency of Issues = 0 and
Experience = 0, is at 2.76 (95% CI [1.38, 4.14], t(198) = 3.94, p < .001).
Within this model:

  - The effect of Frequency of Issues is statistically significant and positive
(beta = 0.93, 95% CI [0.57, 1.28], t(198) = 5.16, p < .001; Std. beta = 0.34,
95% CI [0.21, 0.48])
  - The effect of Experience is statistically non-significant and positive (beta
= 0.05, 95% CI [-0.41, 0.51], t(198) = 0.20, p = 0.839; Std. beta = 0.01, 95%
CI [-0.12, 0.15])

Standardized parameters were obtained by fitting the model on a standardized
version of the dataset. 95% Confidence Intervals (CIs) and p-values were
computed using a Wald t-distribution approximation.

Table

Characteristic	Beta	95% CI1	p-value
(Intercept)	2.8	1.4, 4.1	<0.001
Frequency of Issues	0.93	0.57, 1.3	<0.001
Experience	0.05	-0.41, 0.51	0.8
R²	0.120
Adjusted R²	0.111
Residual df	198
No. Obs.	201
1 CI = Confidence Interval

Plot model

Figure 3: Model plot

Descriptive plot

Figure 4: Descriptive plot

Surface plot

Anything else

• Automatic reports per participant
• Automatic emails to participants
• Fully reproducible pdf version of article using journal templates
• Etc.

---