Term Project Details and Guidelines

Basic Considerations

For your course project, you will modify and experiment with an agent-based simulation. Plan to spend at least 30 hours on this project, spread throughout the term. Make sure that your final paper contains all of the core components, listed below. You will produce a research paper that uses the ODD framework to introduce your model and then presents your analysis of the results generated by your model.

Getting Started

Start to learn about the topic of your term paper, here. Keep your project simple enough to finish in one term! Most students will make modest modifications to an existing NetLogo model. You will experiment with your chosen model to see how your modifications affect it.

The biggest problem students have with their projects is making them overly ambitious. Finding a project that is simple enough to be doable but complex enough to hold some interest is an art. You should not expect the result to be beautiful, any more than you would expect that of your first serious attempt to paint.

Group Work and Individual Work

The term paper has some group components. The Project Documentation, and Code are group work. This section provides a brief summary of expectations for the term project. Be sure to carefully master the details provided later on.

A short proposal and a more detailed project documentation are due early in the course. The code is due one week before the last day of class. The paper is due on the last day of class. Pay careful attention to the due dates.

Code Development

Most students will implement their term project in NetLogo. Students are expected to work in groups of three or four. (I will assign groups early in the term.) Groups work together on the project proposal, documentation, and code. However, each student works alone on a term paper and produces their own individual experiments with the group model. (Name each experiment to identify the group member it belongs to.)

Students must use version control software while developing their project code. (Early in this course, I teaches the basic use of Subversion for version control.) Each group will have its own folder in an online class repository. The group folder should contain two (2) files. The first is an unchanged copy of the original model, which serves only as a reference. The second is a modification of the reference model, which should contain the only code that group members are working on. To repeat: there should be no other copies of the group’s project code. Both the reference code and the project code must be committed to the repository, but the file containing the original code should never be changed in any way. It is purely for reference.

Students seeking honors or graduate credit can consider running their simulation on AU's high performance computing facility, but this is not required or graded. More detailed instructions can be found below.

[↑Back To Top↑]

Experimental Design: Pedagogical Goals

This assignment has three core goals. First, it introduces you to the concept of developing a research question, formulating a hypothesis, designing an experiment, and producing a research proposal. Second, it introduces your to the important research practice of preregistration: you will design and pre-register a simulation experiment. Third, it helps you gain familiarity with the base model for your project. The base model is the model before your group changes it. Therefore, your initial experimental design involves systematically changing (at least) one existing model parameter. (Correspondingly, your sweep table may have only a single entry.)

[↑Back To Top↑]

Summary

This assignment is group work. Each group will produced detailed documentation of their term-project simulation model, following the ODD protocal and using UML where appropriate.

The term-project documentation is a collaborative document. Use a collaborative authoring environment that includes a commenting function, such as Google Docs or Microsoft Teams. (Use of a collaborative authoring tool is required.) Comment on each other’s prose and analysis until you produce well-written and coherent documentation. In a group, each individual should submit an identical PDF, which should list all contributors as authors. Non-contributing group members must not be listed and will receive zero credit for this assignment.

Project Documentation: Partial Checklist

• State specific research questions in the Research Question component.

• Identify SMART goals and stretch goals related to the research questions.

• Identify the baseline model. (Include a detailed parameterization table.)

• Identify changes from the baseline model.

• State how the model changes relate to the research questions.

• Provide a correct UML class diagram for each entity, with discussion.

• Provide a correct UML activity diagram for simulation schedule, with discussion.

• Present the model according to the ODD protocol, supported by appropriate discussion. (Go step by step; expect this to take at least 3 pages.)

• Provide a table for the baseline parameterization (typically, the original model parameterization), that lists each parameter, its baseline value. and (briefly) its interpretation in the model.

• Discuss (and correctly cite) at least two clearly relevant journal articles. For each paper, briefly indicate how it bears on your project (as you currently understand it, at this preliminary stage). Your discussion should not be at an "I read the abstracts" level but rather at an "I mastered the papers" level.

• Provide a full citation of the original model’s source code, so that a reader can retrieve this source code.

• Review spelling and grammar, and conform to formatting checklist.

Documentation Details

Each group collaborates to develop the core documentation for the term-project. (Each individual in the group should submit an identical file.) This document should be as specific as possible about the research goals of the group.

Key components of the documentation are an ODD-based discussion of the model along with some UML diagrams and specification of the baseline parameterization. The project documentation should demonstrate your understanding of the core concepts of the ODD protocol. (See the required reading on the ODD topic.) Not all parts of the ODD protocol are relevant to all projects; indicate which parts do not apply to your project and state why.

Create a UML class box representation of each key agent type in the base model. (You do not need to draw the relationships between agent types.) You should match the actual model code, as much as possible, when naming variables or procedures in the UML diagrams (and in the text). If agents will change from the base model, create a UML class box representation of each key agent type in the extended model model. You may optionally use a more elaborate UML class diagram to show the relationship between your modified agents and the agents in the base model. Include these UML diagrams along with the text, which should provide supporting discussion. (See the required reading on the UML topic.)

Create a UML activity diagram for the core simulation schedule. (So you do not need to represent iteration of this schedule.) You may add diagrams for other model activities, if these will be helpful to the reader. Accompany these UML diagrams with discussion in the text, which should help a new reader to understand the model. (See the required reading on the UML topic.)

The Project Documentation should clearly distinguish between the base model (the model you start with, taken from the literature or from a models library) and the modified model (which you will create by modifying the base model). Be sure to focus on how your modifications will allow you to achieve your goals with the modified model—goals that would not be achievable with the base model.

The new model should encompass the old model, in the sense that running it with the specified baseline parameter values replicates the original model. Specify these baseline parameter values in a table. As best as you can at this point, specify the deviations from this baseline that you plan to consider.

The details section of the ODD protocol will be somewhat truncated for this Project Documentation. In particular, you are not expected at this point to give a full description of every submodel (e.g., procedure). Procedures that you do discuss will typically be specified with a UML activity diagram or with a procedure outline (like those in my notes). Actual code (or pseudocode) is acceptable for particularly short and simple procedures.

This Project Documentation must describe some specific research questions. Propose modifications of the base model in pursuit of these goals. At this point, you may not have fully implemented these modifications, but be as explicit as you can. In particular, be very explicit about why these extensions to the model are needed in response to your research questions. This project documentation must explicitly cite the model underpinning your project as well as any paper that presents the model. Additionally, cite and discuss at least two other papers that are relevant to your project, including any papers that I flagged as important. You discussion of these papers should be extensive enough that it demonstrates a mastery of them.

[↑Back To Top↑]

Term Project Code: Detailed Checklist

The group should work together to make sure the final code satisfies the following criteria.

Group work

Each group will produce a single program file, reflecting a team effort. Each individual in the group should submit an identical file to Canvas. Individuals will primarily receive credit based on individual code contributions, but additionally have responsibility for the group product.

Header

Each code file should include (as comments) a header with the following information.

• Group members, course identifier, and date.

• Attribution of the underlying model to the original creator.

• A one paragraph summary of your key additions to the original model, along with a brief explanation of the purpose of these additions.

• You should also fully attribute the original model and make it clear how you will indicate your extensions in the code.

Use of Subversion:

Each group will actively maintain their code in our Subversion repository. Use of collaborative version control is required. Each group member must work on a personal version-controlled working copy of the project code. Each group member must regularly commit changes to the repository copy, as will be described in class. (Active and ongoing contributions will count towards your participation grade; do not wait until the end of the semester to make your contributions!)

To reiterate: this is a required part of the course. Every group member must learn to use Subversion in order to commit contributions to our Subversion repository. There should be only one program file (in the repository) for each group, which all group members must contribute to on a regular basis.

Please note that svn log displays the contributions of each contributor, and Subversion provides many details about what these contributions are. So although there is a single code file for each group, each group member will receive a separate grade reflecting his or her contributions.

Code Submission:

Although the program file is available to me in the repository, each group member must upload that code file to Canvas. The same file must be submitted by each group member. If a group has three members, then I will see three identical submissions on Canvas, which in turn are identical to what I see in the repository.

Each student must include a new experiment in the submitted code file, and this experiment must be committed to Subversion. Ideally a code submission will include all planned experiments. Nevertheless, you may add experiments subsequent to your code submission. Just be sure to commit to the code repository any experiments that you discuss in your paper.

Programming Language

Most groups will work in NetLogo. (Talk to me if your group all know another language they would like to use. Sometimes I will consider this possibility.)

Formatting

Adopt a consistent and readable formatting style for your code.

Comments:

At the very top of your code file should be a header comment. (See above for details.) In addition, the code should include many detailed, helpful, explanatory comments. Do not add completely obvious comments that simply repeat what each line of code says. Do add comments that will be helpful to the reader of your code, especially conceptual comments that explain what the code is doing.

Each member should uniquely label the beginning of each contributed comment (with initials or last name). Format comments nicely. Comments should not “spill out” to the edge; longer comments should go on their own line(s). A useful rule of thumb is that any line should not be more than 80 characters in length.

Procedures:

Each procedure should include a comment giving a pithy description of what it does and how. If your procedure just moves code from the original model (e.g., out of the go procedure), then include a comment to that effect. If it is an extension, then include a comment to that effect. Each procedure should include a comment specifying its context. (In NetLogo, this is who can run the procedure.)

Strive for modularity. Long procedures should be broken up into short, easily understood procedures.

Modifications:

Clearly attribute the code you borrowed. Clearly mark your modifications (initialed; with helpful comments). Ideally, you will be able to run the original model as a base case for your extended model, making for easy comparison of results.

Modifications can be destructive, elaborative, or cleansing. Destructive changes remove existing functionality so that the model is no longer able to do some things that the model was previously able to do. It is best to avoid destructive changes. Elaborative changes extend the model in a particular direction. Group members may disagree on the best direction for elaborative changes; involve me in this case. Cleansing changes include the addition of comments, refactoring of existing procedures for readability or efficiency, and improvements of the Interface. Interface improvement is generally a low priority. The other two cleansing changes are important: I would like to see you using the good coding practices you have learned in this course to produce fully commented, easy to read code.

Tests

Each student should contribute at least one test to the code. Ideally, this will test a reporter procedure that student introduced. However, tests of the procedures in the original model are also welcome. In NetLogo models, please start test names with test and place all tests at the bottom of the Code tab.

A good set of tests is required. Each group should try to ensure full test coverage. This means that there is a test procedure for each important model procedure. It is easiest to write tests for pure functions, so use these when possible. You may need to refactor code to make it easier to test.

A test will usually confirm that some procedure is working correctly. You have written some some code that changes things somehow. You know something about the "state" of the model before you run this code. You expect something about how the "state" of the model should change when you run the code. So you write a test that helps demonstrate that your expectations are being met.

It is easiest to write tests for reporters, especially if they are pure functions. However, we commonly write tests that test whether our setup is correct and whether things are are correct at the end of each iteration. Typically, our setup procedure will end with a testSetup procedure. It is also nice to end our go procedure with a testIteration procedure.

GUI

I do not ask you to exert much effort on the model aesthetics (shapes, colors, etc.). Nevertheless, you may find it useful to use color or labels or shapes to distinguish agents by their attribute values or breed. Focus on making the model as useful to users (including yourself) as possible. Add visual features only when they promote model understanding.

Ideally, you will separate the GUI considerations from the rest of your model, putting them in setupGUI and updateGUI procedures.

Freeze

Once the code is turned in, the model is frozen. When you turn in your code, report the Subversion revision number for the frozen model. There should be no more changes to the model or tests after submission. However, you can continue to add BehaviorSpace experiments. (If you do, please be sure to commit them.)

NetLogo Details

BehaviorSpace

Each member will add their own BehaviorSpace experiments to the single group code file. The model experiments should be individual. Each experiment should be clearly named, and the experiment’s name should clearly indicate who created it (e.g., by prepending initials, as in ai01, ai02, etc.) All of these experiments must be present in the single group code file in the repository.

While you are certainly permitted to collect data at every step, most students will find it useful to collect data only at the end of each run.

Pure functions

Remember that a pure function in NetLogo is a reporter (or a reporter task) that does not use or modify any global variables. It is easiest to write tests for pure functions.

Startup

Be sure to provide the user of your model with a simple way to set all parameters to your baseline values.

Documentation

Include appropriate model documentation in your Info tab. If the model you are modifying already includes such documentation, put your additions at the top, and refer freely to the original documentation (that will remain below yours).

Style Guide for Code Submissions

Term Paper Core Components

I am rather flexible: I care most that you produce a thoughtful and carefully written paper that reflects your hard work to master your model and the course content. As a guideline, however, your research paper should include the core components listed below. Undergrads should write at least 15 pages. Students seeking honors or graduate credit should write at least 20 pages. Page limits include the title page, citations, figures, and tables but not appendices.

Except for the title page, each core component should have its own section. Format the section headers as described in the formatting checklist.

Title Page

The first page of your paper should be a title page. This includes

• the title of your paper,

• your name,

• the date,

• name and number of the course and the name of the professor, and

• an abstract of around 200 words, which should briefly state your research question and key results.

Introduction:

Briefly describe your goals for this paper. (In terms of the ODD protocol, this is essentially the “purpose” section.) If you are modifying an existing model, but sure to be very specific about the model and about the purpose of your modifications. Be sure to state your research question and hypotheses.

Literature Review:

The object of the literature review is to provide background and context for the model and its concerns. (See the Literature Review overview for more detail.) You are not required to have an extensive literature review. However you should provide an in-depth discussion at least two (2) journal articles related to your work.

If your work is a replication or extension of a published paper, one of these articles will usually be the article that originally presented the model. If there is no supporting publication, you may comment on the model’s documentation (including the Info tab of a NetLogo model).

You may incorporate the literature review into your introduction, if you wish. If you prefer to fit your literature review into your ODD framework, I suggest that any discussions of the literature come in the Overview section. You may use a subsection entitled e.g. Review of Some Related Literature.

Model Description:

This description should be understandable by anyone who has taken this course, not just your group members. Do not assume your reader has looked at your code.

Use UML class diagrams when discussing your agents (in your discussion of entities). Use UML activity diagrams when discussing your simulation schedule (in your process overview). Your UML should show use the correct basic notation, as presented in class. However, you may adapt the diagrams as needed; you are not expected to master advanced UML concepts. Your diagrams do not need to be elaborate, nor entirely rigorous. Just use the correct graphical notation for the basic diagrams. These diagrams belong in the paper, alongside your discussion, not in a separate appendix.

You must use the ODD protocol to guide your model description. (This is required and should be much more polished than in the earlier project documentation.) The ODD protocol specifies the sections that you need to have in this part of your paper, and it suggests an order in which to put them. You need not adhere rigidly to the ODD protocol. For example, you will probably want to add a “limitations and future extensions” subsection to your Conclusion. In addition, you need to clearly distinguish between the original model and your extensions. (Focus on explicating important changes, and clearly state the reasoning behind them.)

Baseline Parameterization

List all model parameters in a table, along with baseline values in this table. Be sure to explain the role in the model of each parameter. Clearly indicate the focal parameters (i.e., the one's you will vary across scenarios; the treatment variables). Remember that not all globals are parameters. Some for example are variables used to store information as the model runs. A parameter remains constant during a simulation run.

A baseline is just a convenient point of reference. In this course, it is typically the original model before your changes. The baseline specification provides one value for each parameter. Depending on the amount of guidance provided by the original model, the baseline parameterization may be somewhat arbitrary. Typically you will extend a model in a way that encompasses the old model, so that the baseline is equivalent to the encompassed model.

Discussion of Experimental Design

This discussion should be smoothly integreated into you paper. (It should not just be your design outline, which you will submit separately.) Each student must create at least one unique experimental design that systematically explores the term-project model. This section can be a short but self-contained discussion of your preregistered term-project experimental design outline which you should submit with your paper. Be sure to show clear understanding of our discussion of the design of experiments by using the appropriate vocabulary when discussing your experiments. (For example, clearly distinguish between ‘experiment’, ‘scenario’, and ‘replicate’.) Each experiment should test an explicitly stated hypothesis. Explain the reasoning you used to develop each hypothesis, and be very clear about how you intend to test it. (This means that you will include a discussion of what you will measure and how you will measure it in your presentation of your hypotheses.)

Each group member must individually design and discuss experiments. Experimental design and analysis are individual activities, not group activities. Your discussion of your experimental design should demonstrate that you mastered the required reading for Design of Experiments. Additionally, to reiterate, submit a filled in Experimental Design Outline for each experiment. This is separate from the written description in the body of your paper, which should be self contained and should not depend on the presence of this material.

Results

Be sure to describe the outcomes under the baseline parameterization. For example, provide box-plots for your key outcome variables under the baseline parameterization.

Present your analysis of the results produced by your experimental design. Present appropriate summary statistics. (This is required.) Present helpful charts and tables. (This is required.) Regression analysis is desirable but not necessarily required. However, students seeking honors or graduate credit must additionally present and discuss results from a multiple regression. (This is recommended but optional for other students.) Use carefully formatted and annotated tables to present your regression results. Each table should be fully discussed in the text. Always specify the software you used to produce your tables, charts, or regression results. (The software should be cited and included in the paper’s references.)

Always provide at least a paragraph of discussion of each chart, table, or regression that you present. Tables and charts do not speak for themselves; some require extended discussion. Do your results support or contradict your hypotheses? Can you explain why you see these results in your model? It is up to you as the experimenter to demonstrate to the reader the extent to which the results support or conflict with the hypotheses.

Conclusion

Your conclusion should be a separate section of your paper. This is where you get to summarize what you have accomplished. Assess the model's capabilities and what you achieved with it. Be sure you restate your hypotheses and whether or not they appear to be supported by the model. Propose directions for future research.

Given the limited time for this project, you should try to implement only a small number of ideas. However, your project should produce many ideas for future research. Your conclusion is a good place to describe things left undone that you would like to do. These are your directions for future research.

Citations

Use inline author-date citation references, such as Smith (1999, p.99). (Do not use footnoted citation styles.) Be sure to provide supporting page numbers when appropriate.

You should provide a complete citation for each citation reference in your paper. Put your citation details on a separate page, even if you have only two citations. Provide full citations (not just links) in a final References section. Citations should be complete (including access dates for URLs) and should be consistently formatted. Sort citations by the last name of the first author. Be sure your citation formatting is consistent across citations. (You may use any consistent citation style, including APA or Chicago.)

Be sure to cite the model you are modifying. If you develop extensions of the model jointly with others, you may also cite the modified model, listing all contributors.

Appendices (may be submitted separately)

Appendices do not count against the page limits. These appendices may be uploaded with the paper instead of being actually attached to it.

1. Design Appendix:

   Attach an experimental design outline for each experiment that you report on. (Only one is required: your pre-registered design. additional experiments are optional.) Include a screenshot with each experimental design (e.g., the BehaviorSpace experiment details). If you take a BehaviorSpace screenshot, be sure your parameter sweep is visible. (You can move it to ensure this.)

   Your paper should be self-contained, so you may duplicate as much of this design content in your paper as you wish. (Appendices do not count toward page limits.) For each design, include a screen shot of the corresponding BehaviorSpace experiment. For credit, the designs must match the experiments that commit to our on-campus repository.

2. Robustness Appendix: If you wish, you may add an appendix with robustness check for your experiment and its results. (This is not required or graded.) Do not put key figures or tables for your paper in an appendix.

3. Data Analysis Supplement (submitted separately): Separately submit a data analysis supplement. This may be no more than a spreadsheet file that produces your charts and tables, such as an Excel file or Calc File. Or it may have two parts: a CSV file plus a separate code file, where the latter contains the code used to produce your charts and tables from the CSV file. Any charts or tables that you produce need to be copied into the body of your report.

Additional Considerations:

Before turning it in, please check your paper against the formatting checklist for this course. Here are some additional considerations.

Back Up

You are expected to continually back up your work. Back up at least once per hour to a separate storage location, such as an external drive or a cloud service (such as Google Drive, Dropbox, or OneDrive). Lost work due to failure to back up does not excuse non-submission.

Honors or Graduate Credit

Papers for honors or graduate credit are expected to be more detailed and more sophisticated, to develop more interesting and sophisticated hypotheses, to review and cite multiple related research publications, to include a more thorough data analysis (generally this should include a simple regression analysis), and to display mastery of the “advanced” required readings. Graduate students are strongly encouraged to use LaTeX or Mathematica when writing their papers, but this is not required.

[↑Back To Top↑]