All students will complete an extensive research project. This project will be ongoing during the entire course. Expect to fail if you put off your work until the last week of class.
This section provides a brief summary of expectations for the term project. Be sure to see the details below. The Initial Proposal, Detailed Proposal, and Code are group work. The Initial Experimental Design and Paper are individual work.
A short proposal and a detailed proposal are due early in the course. The code is due one week before the last day of class. The paper is due on the last day of class. Pay careful attention to the due dates. Pay particular attention to the formatting checklist.
A successful project will demonstrate an ability to substantially modify and experiment with a simple agent-based model. (Undergraduates may add modest extensions to the model and do limited hypothesis testing. Graduate students should add substantial extensions to the model and do sophisticated hypothesis testing.)
Most students will implement their term project in NetLogo. Students are expected to work in groups of two or three. (I will assign groups early in the term.) Groups work together on the project proposal, outline, and code. However, each student works alone on a term paper and produces their own individual experiments with the group model. (Name each experiment to identify the group member it belongs to.)
Students must use version control software while developing their project code. (You will learn how to do this early in the course.) Each group will have its own folder an online class repository. The folder should contain an unchanged copy of the original model, which serves only a reference function. The folder should additionally contain a modified model, which should be the only code file that group members are working on. Again, both files should be committed to the repository, but the file containing the original code should never be changed in any way. Graduate students can consider running their simulation on AU's high performance computing facility, but this is not required or graded. More detailed instructions can be found below.
For your course project, you will modify and experiment with an agent-based simulation. Plan to spend at least 35 hours on this project, spread throughout the term. Make sure that your final paper contains all of the core components, listed below.
You will use NetLogo's BehaviorSpace (or some equivalent) to export data from your simulation, which you will analyze (e.g., in a spreadsheet). You will produce a research paper that uses the ODD framework to introduce your model and then presents your analysis of the results generated by your model. I expect that you will share your paper with the class. (Contact me if you have cogent objections to sharing your paper.)
Pick a topic from the list of possible topics below. Try to choose a model in an area that you find substantively interesting (e.g., environmental policy, social hierarchy, economic development, transmission of inequality, historical dynamics, market dynamics). Keep your project simple enough to finish in one term! Clear your choice of model with me.
Most students will make modest modifications to an existing NetLogo model. You will experiment with your chosen model to see how your modifications affect it.
The biggest problem students have with their projects is making them overly ambitious. Finding a project that is simple enough to be doable but complex enough to hold some interest is an art. You should not expect the result to be beautiful, any more than you would expect that of your first serious attempt to paint.
Note
Some students (permission required) may attempt to replicate a published study, without having access to supporting NetLogo code. In that case, you would ideally verify that your implementation produces the same results as the original. However, many replication projects prove very ambitious for one term. In such cases, I will be perfectly happy if you implement a subset of the model features.
Turn in a project proposal based on the list of possible topics below. Each group develops one common initial proposal for the group project. (Each individual in the group should submit an identical file.)
This initial proposal is very limited in scope. It does the following:
This is an early-stage proposal. Nevertheless, be as specific about your goals as possible. (Deleting the adverbs and subjective adjectives from your proposal will help you make it more specific.) Indicate which of your goals you will attempt first, and which will be accomplished if time allows. Experiments are a required part of the project, so ideally you will indicate how you hope to experiment with your model.
Page Limits: 1-2 pages (including citations).
Formatting Details: Use the formatting checklist! In particular, put the project title, group number, all names, course number, and date at top. Use author-date citation references (not footnotes), and provide full citations (not just links) in a final References section. Citations should be complete (including access dates for URLs) and should be consistently formatted. The paper should be double-spaced, with 12 point serif font and one-inch margins. Submit in the PDF file format.
Page Limits: 4-7 pages (including citations and any code).
Formatting Details: Use the formatting checklist!
Turn in a detailed proposal for your Term Project. Each group works together to develop a single detailed project proposal. (Each individual in the group should submit an identical file.) Be as specific as possible about your goals.
The detailed proposal must follow the ODD protocol. Your detailed proposal should demonstrate that you understand the core concepts of the ODD protocol. (See chapters 3 & 4 of Railsback and Grimm.) Not all parts of the ODD protocol are relevant to all projects; indicate which parts do not apply to your project and why. In addition, you may find it useful to create a UML class box representation of each agent type. (You may also want to use a UML class diagram to show the relationship between your agents and the agents in the original model.) You may find it useful to use a UML activity diagram to characterize the simulation schedule.
In addition, your detailed proposal must clearly distinguish between the base model (the model you start with, taken from the literature or from a models library) and the modified model (which you will create by modifying the base model). Be sure to focus on how your proposed modifications will allow you to achieve your goals with the modified model—goals that would not be achievable with the base model.
The details section will be somewhat truncated for this proposal. Specifically, you are not expected at this point to give a full description of every procedure. However, you should try to specify the details of a baseline scenario and the deviations from it that you will consider. Ordinarily, you will include a table showing the initializations of the baseline scenario.
This detailed project proposal must reference a specific model that you will change, explain your goals in selecting and changing this model, and propose how you will modify the model the model in pursuit of these goals. Be very explicit about why these extensions to the model are needed in response to your research questions. This project proposal must explicitly cite the model underpinning your project and additionally discuss and cite one paper (at the very least) that is relevant to your project, beyond any paper presenting the model. Follow the formatting checklist for instructions on the formatting of citations and citation references.
The detailed proposal is not a contract. It is just a structured overview of what you hope to accomplish. (In particular, the structure is provided by the ODD protocol.) There are many reasons you might want to make changes as you work on the project. You might find an idea that sounded simple in fact proves infeasible. Or, you might discover a more interesting question as your get more familiar with your model. That said, the earlier you can finalize your project goals, the better.
Each individual submits a unique experimental design. This is not group work.
Page Limits: 2-3 pages. Rely on the Experimental Design Outline for guidance.
Use the outline to help you plan your experiment (parameter sweep description and hypotheses). You are not expected to have run your experiment at this point, so you will not have any results or conclusions (or experiement date). Your discussion of your design should reflect your careful mastery of the required readings.
Formatting Details: Use the formatting checklist!
Each code file should include as comments a header with the following information.
Each group will actively maintain their code in our Subversion repository. Use of collaborative version control is required. Each group member must work on a personal version-controlled working copy of the project code. Each group member must regularly commit changes to the repository copy, as will be described in class. (Active and ongoing contributions will count towards your participation grade; do not wait until the end of the semester to make your contributions!)
To reiterate: this is a required part of the course. Every group member must learn to use Subversion in order to commit contributions to our Subversion repository. There should be only one program file (in the repository) for each group, which all group members must contribute to on a regular basis.
Please note that svn log displays the contributions of each contributor, and Subversion provides many details about what these contributions are. So although there is a single code file for each group, each group member will receive a separate grade reflecting his or her contributions.
Although the program file is available to me in the repository, each group member must upload that code file to Blackboard. The same file must be submitted by each group member. If a group has three members, then I will see three identical submissions on Blackboard, which are in turn identical to what I see in the repository.
Ideally your code submission will include all of the experiments for your papers. However, you may add experiments subsequent to your code submission. Just make sure you commit to our repository any experiments that you discuss in your papers.
Make sure your code includes many detailed, helpful, explanatory comments. Each member should uniquely label his/her comments (with initials or last name). Do not add completely obvious comments that simply repeat what each line of code says. Do add comments that will be helpful to the reader of your code, especially conceptual comments that explain what the code is doing.
At the top of your code file should be a comment that lists the group members, the course, and the date. You should also fully attribute the original model and make it clear how you will indicate your extensions in the code.
Comments should not “spill out” to the edge. Format them nicely. Longer comments should go on their own line(s). A useful rule of thumb is that any line should not be more than 80 characters in length.
Each procedure should include a comment giving a pithy description of what it does and how. If your procedure just moves code from the original model (e.g., out of the go procedure), then include a comment to that effect. If it is an extension, then include a comment to that effect. Each procedure should include a comment specifying its context. (In NetLogo, this is who can run the procedure.)
Strive for modularity. Long procedures should be broken up into short, easily understood procedures.
Clearly attribute the code you borrowed. Clearly mark your modifications (initialed; with helpful comments). Ideally, you will be able to run the original model as a base case for your extended model, making for easy comparison of results.
Modifications can be destructive, elaborative, or cleansing. Destructive changes remove existing functionality so that the model is no longer able to do some things that the model was previously able to do. It is best to avoid destructive changes. Elaborative changes extend the model in a particular direction. Group members may disagree on the best direction for elaborative changes; involve me in this case. Cleansing changes include the addition of comments, refactoring of existing procedures for readability or efficiency, and improvements of the Interface. Interface improvement is generally a low priority. The other two cleansing changes are important: I would like to see you using the good coding practices you have learned in this course to produce fully commented, easy to read code.
Try to write a test procedure for each important model procedure. It is easiest to write tests for pure functions, so use these when possible. A good set of tests is required.
A test will usually confirm that some procedure is working correctly. You have written some some code that changes things somehow. You know something about the "state" of the model before you run this code. You expect something about how the "state" of the model should change when you run the code. So you write a test that helps demonstrate that your expectations are being met.
It is easiest to write tests for reporters, especially if they are pure functions. However, we commonly write tests that test whether our setup is correct and whether things are are correct at the end of each iteration. Typically, our setup procedure will end with a testSetup procedure. It is also nice to end our go procedure with a testIteration procedure.
I am not asking you to sink much energy into the model aesthetics (shapes, colors, etc.). However you may find it useful to use color or labels or shapes to distinguish agents by their attribute values or breed. Focus on making the model as useful to you as possible. Add visual features only when that helps you understand your model.
Ideally, you will separate the GUI considerations from the rest of your model,
putting them in setupGUI and updateGUI procedures.
Each member will add their own BehaviorSpace experiments to the single group code file. The model experiments should be individual. Each experiment should be clearly named, and the experiment’s name should clearly indicate who created it (e.g., by prepending initials, as in ai01, ai02, etc.) All of these experiments must be present in the single group code file in the repository.
While you are certainly permitted to collect data at every step, most students will find it useful to collect data only at the end of each run.
Each student must submit an individual research paper along with documentation of the experimental data (e.g., the .csv file produced). You may draw on the detailed proposal produced by your group, as long as you cite it, but all parts of this research paper are to be individual work. There are three key check lists before your submit your paper:
You will be graded on the quality of your writing, in addition to conformity with the checklists. Write your paper so that you expect that classmates who are not in your group could readily understand it. (Do not assume that they will look at your code!) Write your paper so that exact replication of your project should be possible for someone having access only to your paper (but not to your code).
Page Limits: 15-25 pages (including citations, figure, and tables). Undergrads should write about 15 pages. Graduate students and undergrads seeking honors credit should write about 20 pages. This length includes the title page and references, as well as all charts and tables.
I am rather flexible: I care most that you produce a thoughtful and carefully written paper that reflects your hard work. But as a rough “rubric”, your research paper should include the sections below. (Each section should start with a section header in a larger font. You may also number your sections; this is recommended but ungraded.)
The first page of your paper should be a title page. This includes
The object of the literature review is to provide background and context for the model and its concerns. (See the Literature Review section for more detail.) You are not required to have an extensive literature review. However you should provide an in-depth discussion at least two journal articles related to your work.
If your work is a replication or extension of a published paper, one of these articles will usually be the article that originally presented the model. If there is no supporting publication, the "article" may be no more than an Info tab.
If you want to fit your literature review into your ODD framework, I suggest that any discussions of the literature come in the Overview section. You may use a subsection entitled e.g. Review of Some Related Literature.
Use the ODD protocol to guide your model description. (This is required and should be much more polished than the outline you submitted.) The ODD protocol specifies the sections that you need to have in the first part of your paper, and it suggests an order in which to put them. You need not adhere rigidly to the ODD protocol. For example, you will probably want to add a “limitations and future extensions” subsection to your Conclusion. In addition, you need to clearly distinguish between the original model and your extensions. (Focus on explicating important changes, and clearly state the reasoning behind them.)
The Railsback and Grimm book presents a number of models using the ODD protocol, and you can look at those for good examples of how to do this. For example, see the presentation of the Telemarketer model of chapter 13. Additionally, be sure you have done the required reading for the ODD protocol.
Ordinarily you will use a UML class diagram when discussing your agents (in your discussion of entities). Ordinarily you will use a UML activity diagram when discussing your simulation schedule (in your process overview). Recall our discussions of how to adapt these diagrams to various languages.
Do not assume your reader has looked at your code. This description should be understandable by anyone who has taken this course, not just your group members.
List all model parameters in a table, along with baseline values in this table. Be sure to explain the role of each parameters. Clearly indicate the focal parameters (i.e., the one's you will vary across scenarios; the treatment variables).
A baseline is just a convenient point of reference; which is often the original model before your changes. The baseline documentation will provide one value for each parameter. If you extend a model in a way that encompasses the old model, the baseline will usually be equivalent to the encompassed model.
Remember that not all globals are parameters. Some for example are variables used to store information as the model runs. A parameter remains constant during a simulation run.
NetLogo users usually set the baseline case in a startup procedure. (However, group members may each set up a different baseline case.) NetLogo users usually introduce GUI widgets (e.g., sliders or choosers) for each focal paramter.
You need to explore your model systematically. Present at least one experimental design for your model. (Be sure to show clear understanding of our discussion of the design of experiments by using the approprite vocabulary when discussing your experiments. For example, clearly distinguish between ‘experiment’, ‘scenario’, and ‘replicate’.) Each experiment you design should test a hypothesis. Explicitly state each hypotheses. Explain the reasoning you used to develop each hypothesis. and be very clear about how you intend to test it. (This means that you will include a discussion of what you will measure and how you will measure it in your presentation of your hypotheses.)
Each group member must individually design and discuss experiments. The experiments are an individual activity, not a group activity. Be sure you have done the required reading for Design of Experiments. Be sure to attach (as an appendix) a filled in Experimental Design Outline form for each experiment, but your written description should be self contained and not depend on the presence of this appendix.
Present your analysis of the results produced by your experimental design. Present appropriate summary statistics. (This is required.) Present helpful charts and tables. (This is required.) You may also want to present regression results. (This is required for graduate students.) Always specify the software you used to produce your tables, charts, or regression results.
Always provide at least a paragraph of discussion of each chart, table, or regression that you present. Do your results support or contradict your hypotheses? Can you explain why you see these results in your model?
Your conclusion should be a separate section of your paper. This is where you get to summarize what you have accomplished. Assess the model's capabilities and what you achieved with it. Be sure you restate your hypotheses and whether or not they appear to be supported by the model. Propose directions for future research.
In a short course like this one, you should try to implement only a small number of ideas, but your project should produce many ideas for future research. Your conclusion is a good place to describe things left undone that you would like to do. These are your directions for future research.
Put your citation details on a separate page, even if you have only two citations. (Do not use footnoted citation styles.) You should provide a complete citation for each citation reference in your paper. Sort citations by the last name of the first author. Be sure your citation formatting is consistent across citations. (You may use any consistent citation style, including APA or Chicago.)
Be sure to cite the model you are modifying. If you develop model jointly with others, you may also cite the modified model, listing all contributors.
Before turning it in, please check your paper against the formatting checklist for this course. Here are some additional considerations.
It is expected that your tables will include a table of baseline parameter values. You may also want a table specifying each scenario considered. You should include a table of descriptive statistics. Tables should minimize the use of vertical lines. See the examples at https://www.inf.ethz.ch/personal/markusp/teaching/guides/guide-tables.pdf
It is expected that your figures will include at least the following: a rough UML classifier diagram of the model agent types, a rough UML diagram of the model schedule, and a scatter plot for each correlation considered. Construct your figures carefully to be maximally communicative about your model and results.
Figures and tables should appear next to the related text in your paper, not relegated to the end of the paper. Be wise in your choice of presentations. Do not use tables when figures would be more helpful. Do not use 3D figures when 2D figures are less distracting. Do not use grouped histograms when line charts would be easier to read. And so on.
Carefully read this outline to plan your experimental design. You will discuss your design twice: once briefly before as you plan your experiments, and once in detail in your term paper.
You need to extend and experiment with an existing model. (Most students will not attempt an original model; instead you will replicate and extend an extant agent-based model.) In this section, I provide some suggestions. You can choose models not mentioned in this, but be sure to discuss them with me first. Most students should resist the temptation to implement a model from scratch: make sure you have access to NetLogo code for the model you want to extend.
models from the NetLogo Models Library, including:
models from Agents, Games, and Evolution, including:
http://opim.wharton.upenn.edu/~sok/AGEbook/nlogo/
models from the OpenABM Models Library, including:
other models I can provide you with, including:
An annotated bibliography briefly summarizes each article that you have reviewed. A literature review is different: in addition to summaries, it is a critical analysis. This critical analysis should show how the literature you review is inter-related and relates to your research question.
For detailed guidance, see Galvan (2006). Here we limit discussion to some key points and add some course-relevant details.
Once you have identified and located the articles for your review, you need to analyze them and organize them before you begin writing:
Decide on the format in which you will take notes as you read the articles
use a small number of quotes to illustrate key concepts, terms, and statistics
be sure to note the page number so you can provide a complete citation in your review.
Galvan (2006) recommends building tables as a key way to help you overview, organize, and summarize your findings, and suggests that including one or more of the tables that you create may be helpful in your literature review. If you do include tables as part of your review each must be accompanied by an analysis that summarizes, interprets and synthesizes the literature that you have charted in the table.
Examples of tables that may be relevant to your review:
Using the notes that you have taken and summary tables, develop an outline of your final review. The following are the key steps as outlined by Galvan (2006: 71-79)
Reorganize your notes according to the path of your argument
Galvan, J. (2006). Writing literature reviews: a guide for students of the behavioral sciences (3rd ed.). Glendale, CA: Pyrczak Publishing.
Review of Literature: University of Wisconsin - Madison The Writing Center. https://writing.wisc.edu/Handbook/ReviewofLiterature.html
Write a Literature Review: University of California, Santa Cruz University Library. http://guides.library.ucsc.edu/write-a-literature-review
Information Fluency and Quantitative Analysis: Literature Review by Washington & Lee University http://info.wlu.edu/literature_review/literature_review.html
Mongan-Rallis, Helen (2014). Guidelines for writing a literature review http://www.duluth.umn.edu/~hrallis/guides/researching/litreview.html
You are expected to spell check your document before submitting it. Also, please use a grammar checker if one is available. (Most word processors include some kind of grammar checker.) I recommend that periods be followed by two spaces, in accord with the 6th edition of the APA style guide, but this is not required.
I expect the paper to be well organized and polished. Try to avoid passive voice, run-on paragraphs, and any pretence to know things you do not (see [oppenheimer-2006-acp]). Avoid both colloquialism and pointlessly elevated diction. If you are not a native English speaker, ask an English speaker to read over your paper. Unless you are a skilled writer, you should bring a draft of your paper to the Writing Center.
A widespread example of an inappropriate attempt to elevate diction is using ‘impact’ instead of ‘affect’. Save ‘impact’ for describing meteors hitting the earth or other actual impacts.
Use a consistent style throughout the paper. If you are using a word processing programming, you should achieve style consistency by actively using styles.
Resources:
| [klabunde.willekens-2016-ejp] | Klabunde, Anna, and Frans Willekens. 2016. Decision-Making in Agent-Based Models of Migration: State of the Art and Challenges. European Journal of Population 32, 73--97. |
| [oppenheimer-2006-acp] | Oppenheimer, Daniel M. 2006. Consequences of Erudite Vernacular Utilized Irrespective of Necessity: Problems with Using Long Words Needlessly. Applied Cognitive Psychology 20, 139--156. |
Social Dilemmas