Experimental Design and Reporting

The document has two major sections: one assisting with the design of an experiment, and one describing the subsequent process of reporting on the experiment. Planning the experiment and reporting on it are two separate activities, which require separate but related discussions of the experiment.

In planning stage, you develop and submit an experiment design that outlines your plans for the experiment. After finalizing the design, you run the experiment and produce a report that again describes the experiment but additionally reports on the results from the experiment.

Use the Experiment Planning Stage outline (below) to design an experiment. Follow this outline to provide a thorough description of the experimental design. The intent of the planning stage is to “preregister” your experimental design and your hypotheses. You should explicitly design each experiment. Be sure to think hard about the later reporting stage when planning your experiments.

After completing and submitting an Experiment Planning Stage outline, run the experiment. Then make use of the Experiment Reporting Stage outline. This outline describes how to produce a full report on the experimental design and results. Please include section labels, using the header styles available in your word processor.

Experiment Planning Stage

Your planning-stage document is a research proposal. It describes research that you plan to do but have not yet done. It states a research question and at least one hypothesis, and proposals an experimental design. Your discussion of the experimental design should reflect your careful mastery of the required readings on the design of experiments.

Your research proposal preregisters an experimental design. Preregistration specifies a research plan in advance of an experiment by submitting it to a registry of some sort. Preregistration separates hypothesis-generating from hypothesis-testing research. (Hypothesis-generating research is sometimes called exploratory; hypothesis-testing research is sometimes called confirmatory.) Preregistration improves the quality and transparency of your research. As you plan an experiment, be sure to anticipate your needs in the subsequent Experiment Reporting Stage.

Details

The research proposal should be around 3 to 5 pages in length, double-spaced. Please use the section headers provided in the Experimental Design Outline, and use the header styles available in your word processor.

NetLogo users should implement their design in BehaviorSpace, and the focal parameter must appear in the Interface tab. (It should not be set in the setup procedure.) In NetLogo models, a startup procedure must match the baseline parameterization table in your report.

NetLogo users must submit a screenshot of the completed BehaviorSpace dialog for the experiment. This must exactly match the code committed to the repository (for the reported the revision number). Make sure your parameter sweep shows in the screenshot; it must match the treatment table in your outline.

Each individual submits a unique experimental design. This design should include at least one hypothesis that you will test, a description of the parameter variation (e.g., parameter sweep) that you will use to generate your data, and a discussion of how you will use this data to test the hypothesis. (For example, if you will create a diagnostic chart or statistical test, what data will you use, and what will constitute rejection of your hypothesis?) Rely on the Experimental Design Outline as you plan your experiment. This initial experiment will usually use the original (unmodified) model chosen for your term project. For this initial experiment, consider the effects of a single parameter on a single output, and consider only end-of-simulation output (not a time-series). So your hypothesis should predict how a change in a parameter will change and end-of-simulation outcome. The hypothesis must specify a direction of response of an outcome measure, not just "something changes".

Design: Goals

For context, the pedagogical goals of this initial experimental design are to support students in achieving the following learning objectives.

  • Understand the concept of experimental design.

  • Understand the concept of pre-registration.

  • Gain familiarity with their group’s base model.

  • Gain experience with the use of Subversion that is required for the group project.

  • Gain experience with the use of BehaviorSpace that is required for the group project.

Experiment Reporting Stage

The experiment report begins with some repetition of the language in your research proposal. Feel free to reuse exactly the proposal language and headers wherever this is appropriate.

Although the report on your initial experiment will restate much that is in your registered research plan, it will also add much detail, show how you implemented the plan, and test your hypotheses. (Make sure that your preregistered experiment and submitted discussion coincide.)

Modified Sections

Make the following changes, now that you have actually run and analyzed the experiment.

Header

In addition to the date of your report, include the date that you ran experiment. (The experiment date is the date you ran the simulation that generated the data that you use in your report.)

Input Data

If any data are used as exogenous inputs into your simulation, describe them in enough detail to enable replication. (Often there is no input data.)

Output Data

  • Describe the values collected from your simulation experiment (i.e., the simulation data that you use to conduct your analyses).

  • Describe how you collect this data from your simulation (e.g., NetLogo’s BehaviorSpace or custom data export)?

  • Fully document your data by providing access to the CSV file generated by your experiment (or an equivalent spreadsheet file).

Data Analysis Tools and Methods

  • Do you transform your output data prior to analysis? (E.g., do you scale them or combine them, or does the raw exported data suffice?)

  • Specify the tools used to analyze your data (e.g., Excel spreadsheet or Stata). Specify the methods used to analyzed your data (e.g., descriptive statistics or linear regression).

  • Fully document your methods by providing access to the all files that do data manipulation. (E.g., if you use Stata, you must provide a file to produce your results, such as a .do file or notebook.)

New Sections

Beyond the sections in your research proposal, your research report contains the following new sections.

Abstract

The abstract should very briefly state your research question and your key findings. Put the Header and Abstract together on a separate title page.

Replicability

What steps are you taking to ensure replicability? Be explicit about how you set a different seed for each replicate. Be explicit about the stopping condition for each run.

Results and Interpretation

Aside from the model exposition, this is the bulk of the report. An earlier research proposal designed an experiment. This section presents the results from conducting that experiment. Present and discuss carefully formatted tables and charts. Careful box plots and scatter plots are a minimum requirement, but work seriously to analyze the behavior of your model. Use the most sophisticated tools in your toolkit. The presentation and interpration of results should be detailed and substantial.

Conclusion and Next Steps

  • Restate your research question.

  • Based on the results and interpretation provided in the previous section, what conclusions can you draw from your experiment?

  • Does this experiment help answer your research question? Does it shed light on your hypotheses? Explain.

  • Did this research suggest any further questions that deserve exploration?

References

Be sure your References section is complete. It should cite all the software used for data analysis (including version number).

Experiment: Partial Checklist for the Research Report

The report on the initial experiment will have some overlap with the preregistered experimental design.

  • Specify the baseline model (usually the original model chosen for the term project).

  • Include a table for baseline parameterization (i.e., original model parameterization). This should match the startup procedure.

  • Specify the changes from the baseline model, with a table for the parameter sweep (it is OK to have only one changing parameter for now). This table should include only focal parameters.

  • State a specific hypothesis to test, with an explanation of why you hold this hypothesis.

  • Specify the data exported, with a statement how it was exported. (This is usually a collection of reporters for BehaviorSpace.) Recall that you should have at least 10 replicates for each scenario you consider, with at least \(100\) runs overall.

  • Describe your data. Include at least one box-chart with discussion. (This is also called a box and whisker diagram, and is covered by a Spreadsheet exercise.)

  • Test your hypothesis. (Grad students should use an explicit statistical test.)

  • Conform to formatting checklist.

References and Resources

On Confirmatory versus Exploratory Research

Jaeger, R., & Halliday, T. (1998). Herpetologica, 54, S64--S66.

Exploratory and Confirmatory Research in the Open Science Era

Nilsen, E., Bowler, D., & Linnell, J. (2020). Journal of Applied Ecology, 57, 842--847.

[Cox.Reid-2000-ChapmanHall]

Cox, D.R., and N. Reid. (2000) The Theory of the Design of Experiments. : Chapman and Hall/CRC.