Introduction to Social Science Simulation

Simulation: Basic Questions

• What is a theory?

• How does a model implement a theory?

• What is a simulation?

• What is the role of the computer in simulation?

• How does a conceptual model differ from an implemented model?

• What is an experiment?

• What does it mean to use simulations to experiment with a model?

• How do we choose a theory (or model, or simulation, or experiment) for a given investigation?

• How do we verify that a simulation implements a model?

• How do we test the real-world validity of a theory? A model? A simulation?

What Is a Theory?

We will mean more than just an idea or hunch, but we will not be very specific.

For us, a theory will roughly be a system of ideas that helps us organize our interpretations of something we observe. A theory is essentially an explanatory framework for a domain of experiences.

In contrast to common practice in the physical sciences, social scientists often use the term ‘theory’ just to mean a hypothesis. (Physical scientists often reserve the term ‘theory’ for hypotheses that are widely accepted due to the weight of the accumulated evidence.)

What Is a Scientific Theory?

With our usage, not all theories are scientific. A scientific theory generates scientific hypotheses. For a theoretical claim to qualify as a scientific hypothesis, our observations must be able to affect the credibility of the theory.

A scientific theory is one where experience (observations) can force revisions to beliefs (about the accuracy of the theory).

If it is logically impossible for observations to force us to revise our theory, it is not generating scientific hypotheses.

If we are committed to a theory regardless of the evidence, we are not doing science. A scientific theory is vulnerable to the evidence.

Questions about Modeling

• What constitutes a “good” model? (simplicity, or realism?)

• Why create models?

A model is good to the extent that it helps us achieve the goals we had when creating it.

We create models to help us understand, predict, or control something we observe or can imagine observing.

We can have several models of the same thing, depending on which aspects we want to emphasise, and how we will use the model.

Examples:

• classical mechanics vs. special relativity

• New Classical vs. New Keynesian macroeconomic models

What Makes a Model Useful?

A useful model will

• be informed by theory (i.e., by an explanatory framework)

• simplify some aspect of the modeled system.

• help explain (aid understanding), help predict (support decision making or policy fomulation), or help control the system.

The usefulness of a model should be assessed in terms of its goals. A pedagogical model may aid understanding without being useful for prediction. A statistical model may help predict without adding much understanding.

A more realistic model need not be a more useful model.

Related question: what makes a map useful? [Kitcher-2001-OUP]

RAD vs. KISS

[Coen-2009-CMOT] divides practitioners of computational simulation into two camps: “realist”, and “keep it simple”.

Realism, Accuracy, and Detail (RAD):

• realism is a virtue

• create accurate and detailed representations of relationships

• closer resemblance to nitty-gritty empirical investigations

Keep it stylized and simple (KISS):

• simplicity is a virtue

• create highly stylized representations of relationships

• closer resemblance to formal mathematical theory building

The KISS camp has demonstrated that even very simple rules can generate complex outcomes

RAD vs. KISS ...

This course will hew closer to the KISS camp, while recognizing that concrete applications call for greater realism.

“Both the researcher and the audience have limited cognitive ability. When a surprising result occurs, it is very helpful to be confident that one can understand everything that went into the model.”

—[axelrod-1997-iccsss]_

From the point of view of economists, the KISS camp is more closely aligned with traditional economic theory.

Epstein on Why We Model

[Epstein-2008-JASSS] offers 16 reasons other than prediction.

1. Explain (very distinct from predict)

   plate tectonics and earthquakes; electrostatics and lightning; evolution and speciation

2. Guide data collection

3. Illuminate core dynamics

4. Suggest dynamical analogies

5. Discover new questions

6. Promote a scientific habit of mind

7. Bound (bracket) outcomes to plausible ranges

8. Illuminate core uncertainties.

Epstein on Why We Model (cont.)

9. Offer crisis options in near-real time

10. Demonstrate tradeoffs / suggest efficiencies

11. Challenge the robustness of prevailing theory through perturbations

12. Expose prevailing wisdom as incompatible with available data

13. Train practitioners

14. Discipline the policy dialogue

15. Educate the general public

16. Reveal the apparently simple (complex) to be complex (simple)

Why Use Simulation Models?

Mathematical economists have been criticized for giving precise answers to the wrong questions.

Simulation (numerical methods) is sometimes characterized by practitioners as giving less precise answers to the right questions.

• complex dynamics

• easier examination of out-of-equilibrium adjustments

• emergence

• useful for docking and generalizing analytical models

• allows exploration of problems that are analytically intractable

Anecdote: economist looking for her lost car keys ... http://en.wikipedia.org/wiki/Streetlight_effect

Functions of Simulations

[Hartmann-1996_Hegselmann.etal], [Carley-1999-WP]

• technique: investigate the details of complex system dynamics

• Heuristic Tool: develop models, and theories.

• hypothesis generation (computational experiments suggest real world experiments)

• response to impossible experiments:

• tool for experimentalists: to support experiments.

• pedagogy: help students gain understanding of a complex process.

Heuristic Tool

Simulation is useful where the theory is not well developed, and the causal relationships are not well understood.

aid to theory development:

look for simulations that imitate the empirical process

Substitute for Experiment

When actual experiments are perhaps:

• pragmatically impossible (e.g., scale, time, need to explore a very large number of scenarios (parameter configurations)

• theoretically impossible (e.g., the exploration of counterfactuals)

• ethically impossible (removal of minimum wage?)

Perform numerical experiments instead!

Also: can complement lab experiments.

Tool for Experimentalists

it is easy enough to conduct experiments, it is far from easy to conduct irreproachable ones

—Louis Pasteur, 1864

We can use simulations to:

• inspire experiments

• preselect possible systems & set-ups

• analyse experiments (statistical adjustment of data)

Tool for Pedagogues

A pedagogic device through play ...

Example: Mitchell Resnick. Turtles, termites, and traffic jams: Explorations in massively parallel microworlds. MIT Press, 1994.

NetLogo Models Library:

Play with NetLogo models, and experience emergence: (famous examples: Conway’s Life; Schelling segregation model)

Role of Assumptions

Q: do more realistic assumptions → more accurate prediction?

“A simulation is no better than the assumptions built into it.”

—Herbert Simon

But: what makes a “good” assumption for modeling purposes?

Robust Predictions from Simple Theory

Latané (1996) offers four conceptions of simulation as a tool for doing social science:

1. As a scientific tool: theory + simulation + experimentation

2. As a language for expressing theory:

   • natural language (expressive but not precise and not transparent)

   • mathematical equations (precise and transparent, but closed form solutions may not exist)

   • computer programs (precise and expressive, but not transparent)

3. As an “easy” alternative to deductive model analysis: robust coding

4. a tool for discovering theory consequence:

   properties → consequences (i.e. characterize sufficiency)

A Third Way of Doing Science

[axelrod-1997-iccsss]_ proposes that simulation is a “third way of doing science”, along with deduction & induction.

• Deduction: derive theorems from explicit assumptions

• Induction: discover persistent patterns in empirical data

• Simulation: assumptions → data for inductive analysis

Simulation

• has explicit assumptions, like deductive analysis, but does not focus on proving theorems

• generates data that can be used in inductive analysis, but the data is generated rather than gathered

• differs from deduction & induction in its implementation

• differs from deduction & induction in its goals.

• permits increased understanding of systems through controlled computer experiments

Modeling Skills

A modeler needs to ability to

• usefully abstract from reality

• discover the model’s implications

• evaluate the model (not just to defend it!):

  • be willing to be proved wrong

  • acknowledge errors, flaws, shortcomings

  • be skeptical of oneself

  • search for alternative models and explanations

Why Use Agent-Based Models?

[axelrod-1997-iccsss]_, [axtell-2000-csed17]_

• equivalent to other simulation methods in many circumstances

• can be computationally more efficient than other simulation methods

• allows discovery of macro patterns that are robust to micro variation

• readily incorporates spatial considerations

• easily scaled

• allows the exploration of “emergent” properties

Emergence: Lewes (1875)

Use of the term emergence is often traced to the end of the 19th century. For example, [Mill-1843-LGRD] famously proposed that heteropathic laws govern aggregates (such as living beings) in ways that supplement the more fundamental physical laws. The classic statement is that of [Lewes-1875v2-KPTT].

The emergent is unlike it components insofar as these are incommensurable, and it cannot be reduced to their sum or their difference.

The key idea taken over in the agent-based literature is that emergent phenomena cannot simply be reduced to or deduced from the system components. This is a tricky and potentially suspicious idea.

Section

Abstract 1. Does emer

Emergence: Railsback and Grimm (2011)

Railsback and Grimm (ch. 8) contrast emergent outcomes with imposed outcomes. Imposed outcomes are “forced to occur in direct and predictable ways”. In contrast, some outcomes “are difficult or impossible to predict just by thinking”.

They offer three criteria for a outcome to be emergent:

• it is not simply the sum or the properties of the model’s individuals

• it is a different type of result than individual-level properties

• it cannot easily be predicted from the properties of individuals

Emergence: Wilensky and Rand (2015)

Wilensky and Rand (2015, p.6) give a more modern formulation. They define emergence as

the arising of novel and coherent structures, patterns, and properties through the interactions of multiple distributed elements.

This is better, but the meaning of “novel” is this context is obscure. They continue:

Emergent structures cannot be deduced solely from the properties of the elements, but rather, also arise from the interactions of the elements.

Here the meaning of “deduced” is obscure: are they referring to a logical impossibility, or a human limitation? For us the key is in their subsequent elaboration:

Structure (or rules) at the micro-level leads to ordered pattern at the macro-level. ... In complex systems, order can emerge without any design or designer.

Emergence (Humphreys 2009)

“It is a common, although not universal, feature of agent based models that emergent macro-level features appear as a result of running the simulation, that these features would not appear without running the simulation, that new macro-level descriptions must be introduced to capture these features, and that the details of the process between the model and its output are inaccessible to human scientists.”

—[humphreys-2009-synthese]

A Test Case

Consider the following set of rules for agent creation and movement. These agents are visually represented on a 2d canvas, with the origin at the center.

• each agent spawns at the origin, (0,0)

• each agent chooses a random heading (i.e., direction for movement)

• each agent moves a distance of 1 unit (in the direction of its heading)

Do this with 100 agents. We will detect a pattern: the agents are arranged in a circle. Is this emergent?

Note that our agents do not interact with each other. Their interaction with their spatial environment might be considered trivial.

Emergence

emergent properties

properties of a system that exist at a higher level of aggregation than the original description of the system.

Not from superposition, but from interaction at the micro level.

To produce emergent properties, we need a model with more than one “level”.

• sometimes emergent properties can be deduced (Pareto efficiency of competitive equilibrium), but often they are more readily “discovered”

Examples:

• ice, magnetism, money, markets, civil society

• Adam Smith's Invisible Hand → prices

• gambler’s ruin experiment → distribution of wealth

• Schelling’s residential “tipping” model → segregation

Emergence and Reductionism

[Epstein-2007-PrincetonUP] distinguishes classical and contemporary emergentism.

Classical emergentism:

• macro phenomena arise that cannot be explained in terms of the properties of the components

• E.g.: no description of the individual bee can ever explain the emergent phenomenon of the hive.

Contemporary emergentism:

• stable macroscopic patterns arise from the local interaction of agents [Epstein.Axtell-1996-MIT]

• E.g.: a bee’s rules for interacting with other bees are part of the description of the bee; we explain the emergent phenomenon of the hive by finding the right rules for the bees

• reductionist: we consider microstructures that are sufficient to generate macro patterns

Emergence and Reductionism

“Classical emergentism seeks to preserve a “mystery gap” between micro and macro; agent-based modeling seeks to demystify this alleged gap by identifying microspecifications that are sufficient to generate---robustly and replicably---the macro (whole).”

—[Epstein-2007-PrincetonUP]

Generativist Motto: not generated implies not explained

—[epstein-2007-pup]_

Verification & Empirical Validation

Verification (or internal validity)

• is the simulation working correctly

• also called “internal validation”

Empirical Validation

• is the correct model used in the simulation?

• does the model shed useful light on the real world?

“Far better an approximate answer to the right question ... than an exact answer to the wrong question.”

—John Tukey, 1962.

Verification

• use lots of assertions

• create a suite of tests (e.g., unittest module)

• run your tests every time you change the simulation code.

• perhaps ... code using a different platform, or “dock”.

Validation

Ideally ... compare the simulation inputs and outputs with the real world.

But ... what constitutes a comparison?

1. stochastic → complete accord is unlikely, and the distribution of differences is usually unknown

2. path-dependence: output may be sensitive to initial conditions/parameters

3. how to test for “retrodiction” (e.g., evolution of civilization). Calibrate with history?

4. what if the model is correct, but the input data are bad?

5. what if the model is highly sensitive to initial conditions and parameters?

Sensitivity Analysis

• determine whether the model is robust to assumptions made

• determine which are the crucial initial conditions and parameters

• key tool: parameter sweep (requires many simulation runs)

Sources of Error

[Judd-1998-MITPress] notes that model choice often involves a trade-off:

• the greater numerical errors of computational work

• the greater specification errors of analytically tractable models.

Replication

[Wilensky.Rand-2007-JASSS]

original model:

the initial (executable) implementation of a conceptual model

replication:

• a new (executable) implementation of a conceptual model

• written after the orginal model.

• may diverge in

  • hardware

  • programming language and/or toolkits

  • algorithms

  • authors

succesful replication:

output “sufficiently similar” to original model

replication standard:

criterion for success

Replication Standards

[Axtell.Axelrod.Epstein.Cohen-1996-CMOT]

Three kinds of replication (in decreasing closeness):

• numerical identity

• distributional equivalence

• relational alignment: qualitatively similar dependence of outputs on inputs

Docking

[Axtell.Axelrod.Epstein.Cohen-1996-CMOT] discuss “alignment of computational models” or “docking”.

Docking

• production of equivalent results from two different models

• a simulation model written for one purpose is “aligned” with another model by producing equivalent results

  • may be a model written for a different purpose, or may be a general purpose simulation system

• replication & generalisation: “docking” by replicating on a different platform or language

• but lack of standard software an issue.

Docking

1. Which null hypothesis? And sample size.

2. Minor procedural differences (e.g. sampling with or without replacement) can block replication.

Nevertheless, [Axtell.Axelrod.Epstein.Cohen-1996-CMOT] decide “docking” is not necessarily so hard.

Reasons for Errors in Model Docking

1. Ambiguity in published model descriptions.

2. Gaps in published model descriptions.

3. Errors in published model descriptions.

4. Software and/or hardware subtleties. (E.g. different floating-point number representation; see Axelrod 2006.)

Model of the Model-Building Process

1. observe (gather facts)

2. theorize (propose processes that might underlie your observations)

3. model (build a formalization of the theory)

4. deduce or simulate (explore the model’s implications) (Are the implications true? If not, consider model revision.)

5. predict (Are the predictions fulfilled? If not, consider model revision.)

6. generalize (if appropriate, change the model to describe more situations, which may can suggest a greater the variety of tests.)

How to Model Agent Architecture?

Early approach to modelling cognitive abilities (symbolic paradigm) was fragile, complex, and lacked common sense. Since then, five approaches:

1. Production Systems

2. Object Orientation

3. Language Parsing & Generation

4. Machine-Learning Techniques, and (most recently)

5. Probabilistic Robotics (Thrun et al. 2005).

We will focus on 1. and 2.

Production Systems

1. a set of rules (a condition + an action),

2. a working memory, and

3. a rule interpreter (is the condition satisfied? if so, act)

No prespecified order of rules: contingent.

The agent's designer specifies how to break ties among rules.

Production Systems

production system (PS)

• A set of variables each with its own associated possible value set. A set of bindings that associates a value with each variable is a state.

• A set of rules. Each rule has in the form set of variable/value bindings => single variable/value binding. where => denotes consequence in time. Given a state, the rules generate a successor state.

• An execution procedure that matches all rules to the existing state executes (possibly in parallel) all matching rules, subject to conflict resolution procedures or avoidance of conflicting rules

  The execution procedure is invoked repeatedly, and a production system thus generates a sequence of states.

Object Orientation

In “object-oriented” programming languages: “objects” are program structures containing data + procedures for operating on those data;

• the data are stored in “slots” inside the object;

• the procedures are called “methods”

• objects created from templates called “classes”

• classes are ranked in a hierarchy, with subordinate classes more specialised.

Example: Modeling pedestrian flow

Pedestrian flow in a shopping mall

class Pedestrian

• Data attributes: location, direction, gait

GroupWalker extends Pedestrian

• New Data attribute: list of companions

• New method attribute: interact with other

Method attributes (rules) are set at the class level: all agents share the rules.

Data attributes are set at the instance level: each agent has its own data.

OO computer languages: Python, C#, C++, Java. etc.

Probabilistic Robotics

18. Marks offers the following example:

2004 DARPA Grand Challenge

• robots used Production Systems architecture.

Results

• The most successful entrant in the 2004 race completed just 7.4 miles of the 150-mile off-road (desert) course, and only six of the fifteen cars competing travelled even 1.3 miles. In the 2005 Grand Challenge, many robots used Probabilistic (or Bayesian or fuzzy-logic) architecture. (5.) Results: #Stanley,# Stanford's robotic Volkswagen Touareg beat the field, completing the 132-mile race with a winning time of 6 hours 53 minutes 58 seconds (an average speed of 19.1 mph). Four other vehicles successfully completed the race. All but one of the 23 finalists in the 2005 race surpassed the 7.36 mile distance completed by the best vehicle in the 2004 race.

Grand Challenge Rules

• The vehicle must travel autonomously on the ground in under ten hours.

• The vehicle must stay within the course boundaries as defined by a data file provided by DARPA.

• The vehicle may use GPS and other public signals.

• No control commands may be sent to the vehicle while en route.

• The vehicle must not intentionally touch any other competing vehicle.

• An autonomous service station is permitted at a checkpoint area approximately halfway between start and finish.

The Stanford team won the first prize of US $2,000,000 in 2005, with "Stanley."

The DARPA Urban Challenge 2007

In November 2007, Carnegie Mellon's robot, #Boss,# pipped Stanford's #Junior,# to win $2,000,000. Stanford won the $1,000,000 second prize. #Unlike the 2005 desert race, not only had entrants to keep to the tarmac and obey the rules of the road, they had also to avoid colliding with a number of other cars being steered round the base by stunt drivers. The desert vehicles relied on radar, laser range-finders and speedy, cleverly programmed computers to avoid meddlesome objects while racing from point to point. The urban robots used similar technology to accomplish much more difficult tasks. In effect, they were taking the examination to receive a driving licence by demonstrating the ability to park in narrow spaces, slow down and indicate appropriately at junctions, and so on---as well, of course, as avoiding collisions.# --- The Economist, Nov 1, 2007.

Modeling the Environment

Definition of the environment depends on what is being modelled.

For individuals:

• move in a space, or on a network;

• use sensors to perceive the environment, including other agents;

• perhaps be able to affect the environment directly;

• perhaps receive and send signals in the environment.

For computer agents, the order of agents running can be crucial (“synchronicity”).

(We will return to this.)

G & T use NetLogo to build multi-agent simulations:

Majority: pp.158 http://cress.soc.surrey.ac.uk/s4ss/code/NetLogo/majority.html

SitSim: pp.163 http://cress.soc.surrey.ac.uk/s4ss/code/NetLogo/sitsim.html

Shopping Agents: pp.182 http://cress.soc.surrey.ac.uk/s4ss/code/NetLogo/shopping-agents.html

Crowds: pp.202 http://cress.soc.surrey.ac.uk/s4ss/code/NetLogo/crowds.html

Other popular ABM platforms: Swarm, RePast, and Mason.

Railsback & Grimm

A Course in Individual- and Agent-Based Modeling

• provides guidelines to build NetLogo models.

The models they build and modify including the following:

1. Flocking (Chapter 7)

2. Investment decisions (Chapters 9, 10, 11, 14)

3. Telemarketing (Chapters 12, 13)

see: http://www.railsback-grimm-abm-book.com/index.html as well as their Individual-Based Modeling and Ecology, (Princeton U.P., 2005).

Sensitivity Analysis

[edmonds.hales-2003-jasss]_

• find in a model of [Riolo.Cohen.Axelrod-2001-Nature] that simply changing the comparison in an if statemetn from >= to > dramatically reduces the emergence of cooperation

[galan.izquierdo-2005-jasss]_

• find in a model of [axelrod-1986-apsr]_ that changing arbitrary assumptions and running the simulation for longer generates the opposite results to those reported.

Replication Difficulties

• some authors report that replication required a great deal of communication between the original developers and the replicaters

  E.g.: Axtell, Axelrod, Epstein and Cohen (1995), Bigbee, Cioffi-Revilla and Luke (2005), [Wilensky.Rand-2007-JASSS]

[Polhill.Parker.Brown.Grimm-2008-JASSS] conclude that better model descriptions are needed.

See [richiardi.beombruni.saam.sonnessa-2006-jasss]_ and [Grimm.etal-2006-EcolModel] for such efforts.

Economic Journal June 2005

• issue focuses on Complex Adaptive Systems (CAS) in economics

• appeared just after Leombruni & Richiardi asked: “Why are economists sceptical about agent-based simulations?” (Physica A 355: 103-109, 2005.)

• included 4 papers: introduced by Markose, with papers by Axtell, Robson, and Durlauf,

  1. markets as complex adaptive systems,

  2. formal complexity issues,

  3. the co-evolutionary Red Queen effect and novelty,

  4. the empirical and testable manifestations of CAS in economic phenomena.

Markose (2005 EJ)

Many “anomalies” not understood or modeled using conventional optimisation economics:

• innovation

• competitive co-evolution

• persistent heterogeneity

• increasing returns

• “the error-driven processes behind market equilibrium”

• herding

• stock-market crashes and extreme events such as October 1987

→ need the methods of ACE simulation

Axtell (2005 EJ)

• the decentralised market as a whole can be seen as a collective computing device

• the parallel distributed agent-based models of k-lateral exchange

• the specific level of complexity (polynomial) in calculations of equilibrium prices and allocations.

Simon's Bounded Rationality

Agent-based models, following Simon (1982), also assume “bounded rationality”.

Indeed, in the absence of Turing machine (universal calculator), it is difficult not to.

Epstein (2006) reflects:

“One wonders how the core concerns and history of economics would have developed if, instead of being inspired by continuum physics ... blissfully unconcerned as it is with effective computability --- it had been founded on Turing. Finitistic issues of computability, learnability, attainment of equilibrium (rather than mere existence), problem complexity, and undecidability, would then have been central from the start. Their foundational importance is only now being recognized.”

Epstein on the virtues of boundedly rational agents ...

Duncan Foley summarizes:

“The theory of computability and computational complexity suggest that there are two inherent limitations to the rational choice paradigm. One limitation stems from the possibility that the agent's problem is in fact undecidable, so that no computational procedure exists which for all inputs will give her the needed answer in finite time. A second limitation is posed by computational complexity in that even if her problem is decidable, the computational cost of solving it may in many situations be so large as to overwhelm any possible gains, from the optimal choice of action.' (See Albin 1998, 46).”

ABM → Generative Explanation:

Generative explanation (Epstein 2006):

“If you haven't grown it, you haven't explained its emergence.”

To answer: how could the autonomous, local interactions of heterogeneous boundely rational agents generate the observed regularity (that emerges)?

• Generative sufficiency is a necessary but not sufficient condition for explanation. Each realisation is a strict deduction.

See also Miller & Page (2007) pp. 86-87. Grüne-Yanoff (2006) argues to distinguish functional explanations (easier for simulators) from causal explanations (much less achievable for social scientists).

Truth and Beauty

Josh Epstein (2006, p.64) suggests mathematical social scientists dislike AB modeling is that it lacks “beauty”.

Epstein quotes Bertrand Russell (1957):

“Mathematics, rightly viewed, possesses not only truth, but supreme beauty---a beauty cold and austere ... the premises achieve more than would have been thought possible, by means which appear natural and inevitable.”

Truth and Beauty ...

Few consider computer simulation to have cold and austere beauty, although there are also aesthetics in programming. but the second can occur

But the phenomenon of emergence in AB modeling does achieve more than would have been though possible.

Schelling’s (1971) segregation model is important not because it’s right in all details (which id doesn’t purport to be), and it’s beautiful not becuase it’ visually appealing (which it happens to be). It’s important because---even though highly idealized---it offers a powerful and counter-intuitive insight. And it’s beautiful because it does so with startling Russellian parsomy. --- Josh Epstein (2006, p.64)

Formalisation of Agent-Based Models

[Epstein-2007-PrincetonUP] p.55

• Every agent-based model is a computer program → Turing computable

• every Turing machine has a unique corresponding and equivalent partial recursive function (Hodel 1995)

• → “in principle, once could cast any agent-based computational model as an explicit set of mathematical formulas (recursive functions)

• the issue is ... which representation ... is most illuminating

Three Approaches to Validation

[Windrum.Fagiolo.Moneta-2007-JASSS] discuss

• indirect calibration,

• Werker-Brenner calibration,

• and history-friendly validation.

[Moss-2008-JASSS] argues that these all “depend on theories or techniques that are selected independently of the evidence and prior to designing and implementing any specific model.” In particular, economists tend to rely on neoclassical theories of individual behavior.

Indirect Calibration

1. Base agent behaviors and networks on "empirical and experimental evidence about microeconomic behaviour and interactions."

2. Target macro-level "stylised facts" (e.g., firm-size distributions or employment-growth relations)

3. Narrow the parameter space to fit the "empirical evidence on stylised facts"

Werker-Brenner Calibration

1. Calibrate initial conditions and limit the parameter space based on existing empirics.

2. Explore the parameter space with simulation experiments.

3. Retain only the parameter sets "associated to the highest likelihood by the current known facts."

4. Further constrain the parameter space by relying on with domain experts.

History-Friendly Calibration

1. Base agent behaviors and networks on evidence, whether empirical, historical, or even anecdotal.

2. Calibrate initial conditions and limit the parameter space based on this evidence.

3. Validate the model by "comparing its output ... with the 'actual' history of the industry."

Validation of Agent-Based Models

Axtell & Epstein (1994)

Level 0: Qualitatively similar at the micro level of individuals (agents) Level 1: Qualitatively similar at a higher, macro, level Level 2: Quantitative agreement of macro structures eg. means, moments, distributions, statistical tests Level 3: Quantitative agreement at the micro level eg. agents behave exactly the same.

Sufficiency vs. Necessity

Sufficiency

if a model produces the desired target behaviour, it is sufficient for that behavior: demonstrates one set of conditions that produce that behavior.

But there may be several sufficient models:

• how can we choose among them?

• how can we discover them all?

Necessity

Are there necessary conditions for a model to produce the desired target behavior?

This is a difficult question.

“Only when the set N of necessary models is known to be small (such as in the case of DNA structure by the time Watson & Crick were searching for it) is it relatively easy to use simulation to derive necessity.”

—Bob Mark

Resources

[OConnor-2020-StanfordEncycPhil] provides a useful and accessible overview of the research on emergence.

References