Virtual Experiments and Parameter Sweeps

Winter 2017

Psychology 120

The essential feature of any scientific experiment is comparisons to detect effects In the simplest experiments, there is a minimum of two conditions: manipulation and control.  A manipulation condition involves something we systematically do to a system of interest (e.g., a drug administered to mice) and a control condition, which attempts to mimic the manipulation condition but without the manipulation (e.g., a placebo administered to mice). The aim is to determine if the manipulation produced a change when compared to the control group, for example, the drug in the manipulation condition extended the average lifespan of mice by 10% when compared to the placebo-control group. An experimental effect is the difference between the manipulation condition and the control condition.  We typically use statistical analyses (e.g., t-tests) to determine whether any differences observed are statistically significant.  Scientific experiments are by no means limited to just a manipulation condition and a control condition, there can be multiple manipulation and control conditions.  For example, a drug may be administered in different doses.  The only limits on the number of conditions are practical: time, resources, and subjects.

Virtual experiments are like physical experiments in that different conditions can be compared.  However, unlike physical experiments, the only constraints on multiple conditions are computational: time and memory.  Moreover, unlike physical experiments, statistical analyses are almost never needed.  This is because we can replicate an experiment many times, which often eliminates the need for statistical analysis.  Because many more conditions can be simulated with virtual experiments, we can systematically compare the effects of different parameter values.  This allows us to systematically analyze our model for different parameter values by conducting virtual experiments.

To design a virtual experiment with manipulations and controls under multiple conditions of interest (i.e., different parameter values), we must specify values of the manipulation and control conditions (i.e., experimental conditions) and the number of replications for each condition. Consider the KH-Model with replacement.  A virtual experiment comparing replacement (manipulation) with non-replacement (control) for multiple conditions of interest might look like this:

• replacement: true, false
• similarity rule: true, false
• choosiness exponent: 1, 2, 3, 4
• Hold all other parameters constant, and specify these values

The conditions above would generate 2 x 2 x 4 = 16 experimental conditions. Ten replications per experimental condition is reasonable (and we can always investigate how many are needed for a given model), which yields a total of 16 x 10 = 160 simulations required for this virtual experiment.

The data we collect are the pair correlations when each simulated experiment ends.

In the Table above, there are eight comparisons for our manipulation and control conditions. First, there are comparisons for the two rules and nested within these rules are the “choosiness” exponents. When we graph these comparisons, we see the effects of replacement.

Notice that for the similarity rule, there is very little effect of the manipulation condition (replacement) on intra-pair correlations.  There is, however, a small effect of replacement increasing the intra-pair correlation of about 0.03 for the “choosiness” exponent > 1.

The choose “the best” rule shows a clear nonlinear effect in the opposite direction.  For a “choosiness” exponent of 1, the difference in correlation values is 0.17 and the difference decreases until it disappears at a “choosiness” exponent value of 4.

From these experimental results we can conclude that (1) replacement has small nonlinear effects for both rules with a little more of an effect for the “choose the best” rule, (2) that the effects are the opposite for the two rules, and (3) that for the original “choosiness” exponent value of 3 used by Kalick and Hamilton, replacement has only a very small effect, which is not likely important for their overall conclusions.

To conduct virtual experiments like the above with the models we have created, we need to be able to systematically sweep parameters and save data from different conditions, and to do so we will need to add the relevant functionality to our models.

To begin, we can download ABMToolsForMASONv1a.zip, BasicMovement.zip, and KH-Model.zip and install all three. Go to the Java build path of BasicMovement and the KH-Model and make sure ABMToolsForMASONv1a is in the build path of both.  Make sure the BasicMovement is in the build path of the KH-Model.

To implement parameter sweeps we need to first extend our most basic simulation environment with SimStateSweep instead of SimState.  For example, to turn the KH-Model with movement into a parameter sweeper, we should extend the BasicEnvironment class with SimStateSweep instead of SimState as illustrated below.

public class BasicEnvironment extends SimState

becomes

public class BasicEnvironment extends SimStateSweep

We also must add a new constructor method that we will come back to in a bit:

	public BasicEnvironment(long seed, Class observer) {
		super(seed, observer);
	}

And remove

	public AgentsWithGUI gui;

from the Environment in the KH-Model project.

Next, we need to introduce an agent that collects and saves the date for each experiment in a file.  The basic agent that can do this is an Observer. We will create our own Observer for these simulations defined as Experimenter, which will extend Observer. In preparation for creating an Experimenter as a subclass of Observer, we will add some code to the start method of the Environment to fully initialize the Observer.

	public void start(){
		super.start();
		space = new SparseGrid2D(gridWidth,gridHeight);
		correlation = new Correlation();
		placeAgents();
		if(observer != null){
			observer.init(space);
		}
		else{
			System.out.println("Observer is null");
		}
	}

Observer

To implement an Experimenter as a subclass of Observer to collect and save the data and to perform additional manipulations, we start with the code below.

package khMateChoice;

import sim.engine.SimState;
import sweep.Observer;
import sweep.ParameterSweeper;
import sweep.SimStateSweep;

public class Experimenter extends Observer {

	public Experimenter(String fileName, String folderName, SimStateSweep state, 
                   ParameterSweeper sweeper, String precision, String[] headers) {
		super(fileName, folderName, state, sweeper, precision, headers);
		// TODO Auto-generated constructor stub
	}
	
	public void step(SimState state){
		super.step(state);
	}
}

For our experimenter to record and save the data we need to give it access to the correlation data and then specify how the data will be recored.

package khMateChoice;

import sim.engine.SimState;
import sweep.Observer;
import sweep.ParameterSweeper;
import sweep.SimStateSweep;

public class Experimenter extends Observer {
	Correlation correlation = null; //variable for correlation probe

	public Correlation getCorrelation() {
		return correlation;
	}

	public void setCorrelation(Correlation correlation) {
		this.correlation = correlation;
	}
        
        /**
         We add data in the order we wish it to appear our saved data file.
        **/
	public boolean nextInterval(){
		if(correlation.n > 1){
			data.add(correlation.n);
			data.add(correlation.sX/correlation.n);
			data.add(correlation.sY/correlation.n);
			data.add(correlation.correlation());
		}
		else{
			data.add(0);
			data.add(0);
			data.add(0);
			data.add(0);
		}
		
		return true;
	}
	public Experimenter(String fileName, String folderName, SimStateSweep state, ParameterSweeper sweeper,
			String precision, String[] headers) {
		super(fileName, folderName, state, sweeper, precision, headers);
		// TODO Auto-generated constructor stub
	}
	
	public void step(SimState state){
		super.step(state);
		if(getdata){ //Get data is true at each collection interval
			nextInterval();
		}
	}

}

Calls to the data.add method above, stores the data for the specified time steps in Java arrayLists that calculate the means and standard deviations for these data.  Thus, for example, when we store the intra-pair correlation for a simulation at time step 10, the data will be used to calculate the overall mean and standard deviations for the replicate experiments within a condition. The code for this is in ABMToolsForMASON and in the package Sweep: class DataMeanSD.

Next, we must pass the correlation probe to the experimenter at start time using setCorrelation.

	public void start(){
		space = new SparseGrid2D(gridWidth,gridHeight);
		super.start();
		correlation = new Correlation();
		placeAgents();
		if(observer != null){
			observer.init(space);
			((Experimenter)observer).setCorrelation(correlation);
		}
		else{
			System.out.println("Observer is null");
		}
	}

Finally, we must modify the AgentsWithGUI as illustrated below, passing Environment constructor the Experimenter.class.

	public static void main(String[] args) {
		//Create the environment in the main method and pass it the Experimenter class for creation:
		Environment e = new Environment(System.currentTimeMillis(), Experimenter.class);
		//Create the AgentsWithGUI state and pass it the environment
		AgentsWithGUI ex = new AgentsWithGUI(e);
		e.setGui(ex); //pass the AgentsWithGUI to the Environment
		Console c = new Console(ex);
		c.setVisible(true);
		System.out.println("Start Simulation");
	}

	public AgentsWithGUI() {
		super(new Environment(System.currentTimeMillis()));
	}
	
	
	public AgentsWithGUI (SimState state){ //Constructor method for handling Environments
		super(state);
		Environment e = (Environment)state;
		((Environment)state).agentsPortrayal = agentsPortrayal;
	}

One Date per Time Step

When we implemented the KH-model we did not include any mechanism that restricted agents to only one date per time step. In the current model, each agent gets a chance to initiate a date. This implies that each agent dates two times on average with some dating only once and others many times by chance. So, to control this aspect of simulated experiments, we will keep track of dates and include the option to allow agents to date only once. To do this, lets add a boolean parameter to the Environment: public boolean oneDate = false. We then add a getter-setter set. To agent we add public boolean oneDate and boolean dated = false; When an agent dates on a round and oneDate is true, dated = true. The problem is how do we set dated = false for the next round?

First for Agent

	public void findDate(Environment state){
		Bag dates;
		if(findLocalDate){
			dates = state.space.getMooreNeighbors(x, y, dateSearchRadius, mode, false);
		}
		else{
			dates = state.space.getAllObjects();
		}
		
		if(dates.numObjs == 0){
			return; //stop method since there are no possible agents to date.
		}
		
		int r = state.random.nextInt(dates.numObjs);//start search
		Agent other = null;
		for(int i = r;i< dates.numObjs;i++){
			other = (Agent)dates.objs[i];
                        if(oneDate && !dated && !other.dated && other.female != this.female){
                                break;
                        }
			else if(other.female != this.female){
				break;
			}
			else{
				other = null;
			}
		}
		if(other == null){
			for(int i = 0;i< r;i++){
				other = (Agent)dates.objs[i];
				if(oneDate && !dated && !other.dated && other.female != this.female){
					break;
				}
                                else if(other.female != this.female){
                                        break;
                                {
				else{
					other = null;
				}
			}
		}
		double p1 = 0; //choosing agent
		double p2 = 0; //selected agent
                if(other == null){
                    return;
                }
		this.dated = true; //dated this time step
		other.dated = true;
		if(other != null){
			if(similar){
				p1 = chooseSimilar(other);
				p2 = other.chooseSimilar(this);
			}
			else{
				p1 = chooseTheBest(other);
				p2 = other.chooseTheBest(this);
			}
			p1 = closingTimeRule(p1); //correct for closing time rule
			p2 = other.closingTimeRule(p2);
			d++;// increment d
			other.d++;// increment d
			
			//make joint decision
			double p = p1 * p2; //joint probability
			if(state.random.nextBoolean(p)){
				if(this.female){ //(female, male)
					state.correlation.getData(this.attractivenessA, other.attractivenessA);
				}
				else{
					state.correlation.getData(other.attractivenessA, this.attractivenessA);
				}
				this.removeSelf(state);
				other.removeSelf(state);
				if(replacement){
					replicate(state,this.female);
					replicate(state,other.female);
				}
				//state.correlation.printData();
			}
		}

The problem is how to set dated back to false for the next time step.  To do this, we use our experimenter’s step method and add a for loop in which each .

	public void step(SimState state){
		super.step(state);
		if(getdata){ //Get data is true at each collection interval
			nextInterval();
		}
		for(int i=0;i<agents.numObjs;i++){
			Agent a = (Agent)agents.objs[i];
			a.dated = false;//set them to false
		}
	}

RunTimeFile and script file

Now, we have implemented the ability to do parameter sweeps and thus virtual experiments.  There are two files necessary for conducting parameter sweeps: a “runTimeFile.txt” and a “Script.txt” file.  The latter can have different names as specified in the runTimeFile.txt.  If these files do not exist, they will be created.  The first time you run your simulation program, it will create a runTimeFile.txt that looks like this (to see it, you will have to refresh your project):

/* This runtime file allows the user to specify a simulation script file for simulation.
It also allows the user to specify the data output folder and file name. The column headers
for the data can also be specified. The form of the script is specified below:*/


Scriptname: script.txt // The name of the file for parameter sweeps
DataFolder: data //Folder holding simulation results
DataFile: results.txt //Name of results file.
CheckSweepOptions: false //Determines whether parameter sweeps is automatically checked.
ColumnHeaders: N, item1, item2 //The column headers for the data file, 'steps' and 'rep' are prepended to this list
Precision: 3  //Numerical precision for data, must be an integer.

You will need to edit this file for your simulations.  The “script.txt” will contain a list of your publicly declared variables in your Environment (and super classes) that consist of the classes: short, int, char, long, double, float, boolean, String.  All other are ignored.  The data folder will hold your simulation results.txt files (you can create different names for different runs).  if CheckSweepOptions is true, then parameter sweeps will run automatically when you start the simulation.  The ColumnHeaders are the names of the columns in your results file.  Precision is the floating point precision with which your data is recorded in the results file.

For the KH-Model, I modified the runTimeFile.txt to look like this:

/* This runtime file allows the user to specify a simulation script file for simulation.
It also allows the user to specify the data output folder and file name. The column headers
for the data can also be specified. The form of the script is specified below:*/



Scriptname: script.txt // The name of the file for parameter sweeps
DataFolder: data //Folder holding simulation results
DataFile: results.txt //Name of results file.
CheckSweepOptions: true //Determines whether parameter sweeps is automatically checked.
ColumnHeaders: pairs, FA, MA, Corr //The column headers for the data file, 'steps' and 'rep' are prepended to this list
Precision: 3  //Numerical precision for data, must be an integer.

The script.txt file generated should look like this:

/* This script allows the user to perform parameter sweep of up to 3 parameters in a simulation
session. To sweep a parameter for two or more values, simply list the values after the parameter
as illustrated below:

public double x = 2, 3.1, 4.2, 5;

As mentioned aboveUp to 3 parameters can be swept in a single session, e.g.:

public double x = 2.7, 3.1;
public int y = 1, 2, 3;
public boolean z = true, false;

An x X y X z Cartisian crossproduct table is generated for conducting the parameter sweep:

public double x = 2.7,  2.7,  2.7,  2.7,  2.7,  2,7,  3.1,  3.1,  3.1,  3.1,  3.1,  3.1;
public int y =    1,    1,    2,    2,    3,    3,    1,    1,    2,    2,    3,    3;
public boolean z =true, false,true, false,true, false,true, false,true, false,true, false;

The table is generated from the first three parameters encountered with more than one value.
After 3 parameters are encountered with more than one value, subsequent parameters with more than
one value are ignored.*/


public int females = 1000;
public int males = 1000;
public boolean similar = true;
public double scaleK = 10.0;
public double exponentN = 3.0;
public double maxD = 50.0;
public boolean findLocalDate = false;
public int dateSearchRadius = 1;
public boolean replacement = false;
public boolean oneDate = false;
public int gridWidth = 50;
public int gridHeight = 50;
public boolean bounded = false;
public boolean uniqueLocation = false;
public boolean movement = true;
public boolean aggregate = false;
public boolean coordinate = false;
public boolean hyperAggCoor = false;
public boolean collisions = false;
public double randomMovement = 1.0;
public int searchRadius = 1;
public boolean paramSweeps = true;
public long simLength = 101;
public int simNumber = 5;
public String fileDataName = "results.txt";
public String folderDataName = "data";
public String scriptName = "script.txt";
public String simulationTitle = "Parameter Sweeps";
public int dataSamplingInterval = 10;

To run the virtual experiment described earlier, you can edit the script file as illustrated below:

/* This script allows the user to perform parameter sweep of up to 3 parameters in a simulation
session. To sweep a parameter for two or more values, simply list the values after the parameter
as illustrated below:

public double x = 2, 3.1, 4.2, 5;

As mentioned aboveUp to 3 parameters can be swept in a single session, e.g.:

public double x = 2.7, 3.1;
public int y = 1, 2, 3;
public boolean z = true, false;

An x X y X z Cartisian crossproduct table is generated for conducting the parameter sweep:

public double x = 2.7,  2.7,  2.7,  2.7,  2.7,  2,7,  3.1,  3.1,  3.1,  3.1,  3.1,  3.1;
public int y =    1,    1,    2,    2,    3,    3,    1,    1,    2,    2,    3,    3;
public boolean z =true, false,true, false,true, false,true, false,true, false,true, false;

The table is generated from the first three parameters encountered with more than one value.
After 3 parameters are encountered with more than one value, subsequent parameters with more than
one value are ignored.*/


public int females = 1000;
public int males = 1000;
public boolean replacement = false,true;
public boolean similar = true, false; //2 values
public double scaleK = 10.0;
public double exponentN = 1, 2, 3, 4; //4 values
public double maxD = 50.0;
public boolean findLocalDate = false;
public boolean oneDate = false;
public int dateSearchRadius = 1;
public int gridWidth = 50;
public int gridHeight = 50;
public boolean bounded = false;
public boolean uniqueLocation = false;
public boolean movement = true;
public boolean aggregate = false;
public boolean coordinate = false;
public boolean hyperAggCoor = false;
public boolean collisions = false;
public double randomMovement = 1.0;
public int searchRadius = 1;
public boolean paramSweeps = true;
public long simLength = 201; //Increase the length of simulations
public int simNumber = 10; //Replications
public String fileDataName = "results.txt";
public String folderDataName = "data";
public String scriptName = "script.txt";
public String simulationTitle = "Parameter Sweeps";
public int dataSamplingInterval = 10;

We are now ready to run a virtual experiment and collect the data.

Lab

The lab/assignment will consist of putting this altogether, conducting virtual experiments, and writing up the results as an assignment.

1.  Rename or delete your previous projects: KH-Model and BasicEnvironment.

2.  Download ABMToolsForMASONv1a.zip, BasicMovement.zip, and KH-Model.zip. Install all three.

3.   Go to the Java build path of BasicMovement and the KH-Model and make sure ABMToolsForMASONv1a is in the build path of both.  Make sure the BasicMovement is in the build path of the KH-Model.

4.   Follow the description above to change the inheritance of the BasicEnvironment, the start method of the KH-Model, create and Experimenter, and change the AgentsWithGUI class.

5.   Add the option for one date per time step as described above.

6.   Show us that you have it working and proceed to the Assignment.

Assignment

Conduct virtual experiments with the KH-sweep model.

Write a one page introduction to your study using the paper Human mate choice is a complex system as a reference.  Your introduction should include the following components:

A.  The Problem: What is the question or problem that the authors in Human mate choice is a complex system are trying to answer?  What is the evidence for this problem?

B. Agent-Based Simulation: Outline the agent-based simulation the authors developed including a description of the decision rules for choosing another agent for a “date”.

C. Main Results: What were the main results?

D. Your Investigation: What were their conclusions that are relevant to your proposed research?  Describe at least two virtual experiments (with manipulation and control conditions) that you will conduct.  Your virtual experiments could be replications of virtual experiments by the authors of Human mate choice is a complex system to see if their results hold up.  Or you could run virtual experiments not conducted by these authors such as the effects of aggregation or date-search radius.  What virtual experiments you run are up to you, but whatever you decide to do, briefly explain why you think they are interesting.

E. Methods: Describe the model and simulations methods.

F. Write up Your Virtual Experiments: Create graphs to compare manipulation and control conditions.  Describe what these graphs illustrate in the results section of your paper. An example of importing a tab-delimited text file into Excel and creating figures is in results.xlsx.

G. Discussion: Discuss your results.  Are they what you expected?  Are they related to  Human mate choice is a complex system paper?

H. Submission: Write up your study and submit it to us.  Two or three people can work on this assignment together and turn in one paper with all of your names on it. Jointly authored papers often receive high grades, but since two or three people worked on them, they should be very polished.  Reports with graphs are typically 4-7 pages single spaced (graphs typically take up a page or two).  Reports will be due March 9.