View
2
Download
0
Category
Preview:
Citation preview
COMPARATIVE ANALYSIS OF METAMODELING TECHNIQUES BASED
ON AN AGENT-BASED SUPPLY CHAIN MODEL
Mert Edali Gonenc Yucel
Department of Industrial Engineering Department of Industrial Engineering
Yildiz Technical University Bogazici University
34349, Besiktas, Istanbul, Turkey 34342, Bebek, Istanbul, Turkey
E-mail: medali@yildiz.edu.tr E-mail: gonenc.yucel@boun.edu.tr
KEYWORDS
Agent-Based Modeling, Beer Game, Metamodeling,
Input Sampling.
ABSTRACT
Agent-based models comprise interacting autonomous
entities, and generate both individual and emergent
system-level outputs. These models generally have a
large set of free parameters, whose impact on output
needs to be explored. Considering also the need for
replication due to stochasticity, a proper analysis
requires a very large set of simulation runs. Therefore,
obtaining a simpler representation of a simulation model
(e.g., metamodel) can prove useful. We primarily focus
on the potential utilization of various metamodeling
approaches, namely Decision Trees, Random Forests, k-
Nearest Neighbor Regression, and Support Vector
Regression in predicting the two different types of
outputs of an agent-based model. Results show that
system-level outputs are predicted with higher accuracy
compared to individual-level outputs under equal
sample sizes. Although there is no single metamodeling
technique performing best in all cases, we observe that
support vector regression is more robust to the increase
in the dimension of the problem.
INTRODUCTION
Agent-based modeling is a modeling approach used to
analyze complex adaptive systems from various
domains such as archaeology (Axtell et al. 2002),
sociology (Edmonds and Hales 2005), and economics
(Tesfatsion 2002). The main building block of these
models is agents; individual entities acting
autonomously based on their objectives, preferences,
internal states and perceptions about their environments.
The interactions of these micro-level autonomous
entities drive the macro-level system dynamics in agent-
based simulation models (Gilbert 2008; Wilensky and
Rand 2015).
Agent-based modeling is a “bottom-up” approach since
the system of interest is modeled by defining the
decision rules of individual entities (i.e., agents), and the
system-level outcome is a result of the interactions of
these agents at the individual-level (Macal 2010).
Therefore, agent-based models generate both individual
and system-level outputs (Wilensky and Rand 2015).
For example, in the Sugarscape model (Epstein and
Axtell 1996), the wealth of an individual is an important
indicator to check how metabolism and vision attributes
of an agent affect its wealth. Besides, wealth
distribution of agents is of interest to monitor the
economic inequality among the members of the society.
After verification and validation steps, the analyst can
use the model as an experimental platform for
policy/scenario analysis/testing and model exploration.
While the execution time of a single run of an agent-
based model is generally measured in seconds, if not in
minutes, many replications for each parameter
combination are needed due to stochasticity.
Additionally, these models are notoriously known for
the large number of parameters to be specified, which
expands the set of parameter combinations to be
explored. As a result, policy/scenario analysis/testing
and model exploration turn out to be time-consuming
tasks. Therefore, obtaining a simpler representation of a
simulation model can prove useful. In this study, we
primarily focus on the potential utilization of various
metamodeling approaches in the context of agent-based
modeling studies.
A metamodel is as an approximate representation of the
input-output relationship of a simulation model
(Kleijnen and Sargent 2000; Kleijnen et al. 2005).
Instead of running a complex simulation model, it can
be time-saving to generate estimates of simulation
outputs from a simplified representation of the model.
Therefore, metamodels are extensively used in
simulation literature (Kleijnen and Sargent 2000). The
use of metamodels comes into prominence especially
when the required time to run a simulation model takes
days instead of minutes (Kleijnen and van Beers 2004).
Besides time-saving benefits, metamodels also give the
analyst more insights into the model. For example, the
analysis of a linear regression metamodel may reveal
the interactions between simulation model parameters,
or the significance of the effect of simulation model
parameters on model output.
Fitting a metamodel requires input and output data
obtained from the simulation model for training.
Therefore, an appropriate input sampling method which
helps the analyst capture the behavioral richness of the
model outputs should be selected. However, performing
input sampling for agent-based models becomes a major
challenge since sampling methods offered by classical
Proceedings 32nd European Conference on Modelling and Simulation ©ECMS Lars Nolle, Alexandra Burger, Christoph Tholen, Jens Werner, Jens Wellhausen (Editors) ISBN: 978-0-9932440-6-3/ ISBN: 978-0-9932440-7-0 (CD)
design of experiments (DoE) literature may not be
applicable due to underlying assumptions of these
methods such as linear relationship between inputs
(model parameters) and output, normally distributed
errors, and small number of inputs (Sanchez and Lucas
2002; Kleijnen and van Beers 2004; Lee et al. 2015).
Besides, agent-based simulation models have some
special characteristics which may render classical DoE-
based sampling techniques inapplicable for metamodel
fitting. For example, tipping point behavior and other
emergent properties of agent-based models such as
adaptation and path dependency may imply potential
nonlinear relationships between model inputs and
outputs (Wilensky and Rand 2015; ten Broeke et al.
2016). Kleijnen et al. (2005) emphasize that matching
metamodeling methods and sampling techniques to
simulation approaches will be a significant contribution
to the existing literature.
Although metamodeling approaches coupled with
appropriate input sampling methods (for training
purposes) has the potential to be valuable additions to
the toolbox of agent-based modelers in principle, very
limited work has been done to explore this potential. In
that respect, we aim to present the results of our
experimental evaluation of several metamodeling
approaches in predicting the output of an agent-based
simulation model. In that context, we provide a
conceptual classification for output types that one can
get as a consequence of a simulation run with an agent-
based model. Then, we compare the performances of a
set of well-known metamodeling approaches in
predicting two different output types.
The remainder of this paper is organized as follows:
Section 2 summarizes the sampling and metamodeling
techniques used in this study. Section 3 presents
experimental setting. Section 4 contains results and
discussions. Finally, Section 5 concludes the study.
BACKGROUND
As mentioned earlier, one requires a training set that
includes tuples of model inputs and resulting outputs in
order to develop a metamodel for a simulation model.
Determination of this training set is of primary
importance for the performance of the resulting
metamodel. One can employ various sampling
approaches to identify the input combinations to be
included in the training set. In this study, we consider
four different sampling techniques to generate input and
output data for metamodel training: (i) full factorial
design (FFD), (ii) random latin hypercube sampling
(RLHS), (iii) maximin latin hypercube sampling
(MLHS), and (iv) random sampling (RS). To
approximate the input-output relationship of a
simulation model, we consider four well-known
machine learning techniques for metamodel
development: (i) decision trees (DT), (ii) random forests
(RF), (iii) support vector regression (SVR), and (iv) k-
nearest neighbor regression (k-NN R).
Sampling Techniques
In full factorial designs, all combinations of predefined
parameter values (i.e. levels) are considered. For
example, in a two-factor factorial design with
parameters X and Y, there will be x × y number of
design points if there are x different values of X and y
different values of Y. One of the mostly utilized factorial
design is 2k when there are k model parameters (factors)
each having two different levels (parameter values)
denoted by - (low) and + (high) (Montgomery 2013). If
the analyst assumes a more complex metamodel, it is
possible to use 3k or even more detailed factorial design
in the form of mk where m denotes the number of factor
levels (Kleijnen et al. 2005).
Although full factorial designs are easy to construct,
number of sample points exponentially increases as the
number of factors, k, increases. To overcome this
problem, the analyst can utilize random sampling to
generate samples with desired number of points, where
samples are generated from the joint probability
distribution of the factors. The drawback of random
sampling strategy is that it does not ensure evenly
distributed sample points, especially in the presence of
small sample sizes (Crombecq et al. 2009). Therefore,
the analyst can utilize random latin hypercube sampling
(RLHS), which is a sampling strategy with space-filling
property and aims to ensure that all parts of the input
space are sampled (McKay et al. 1979; Montgomery
2013). This is achieved by dividing each factor into n
distinct intervals of equal probability and samples are
randomly drawn so that there is only one sample in each
interval for each factor (Keane and Nair 2005). RLHS
can be used to develop complex metamodels involving
many quantitative factors (Sanchez and Lucas 2002;
Kleijnen et al. 2005).
Although RLHS has space-filling property, it does not
always guarantee that parameter space is evenly
sampled. Therefore, there are many extensions to the
generic RLHS algorithm to ensure the space-filling
property. Maximin LHS (MLHS) is one of these
extensions and aims to maximize the minimum distance
between sample points. The reader is referred to
Beachkofski and Grandhi (2002) and Deutsch and
Deutsch (2012) for the details of MLHS algorithm.
Metamodeling Techniques
A decision tree (DT) is a hierarchical classification and
regression technique (Alpaydin 2014). The method
generates axis-parallel binary splits on the input space
(model parameter space) by using only one input
variable at each iteration (Bishop 2006). The number of
splits is generally determined by node impurity, a
measure of heterogeneity of a node of the tree (Loh
2011). In regression, split decisions are based on the
comparison of sum-of-squared errors (SSE) and a
threshold value which represents the maximum
allowable SSE. Each terminal (leaf) node calculates the
mean (or median) of the outputs it includes. Several
algorithms such as C4.5 (Quinlan 1993) and CART
(Breiman et al. 1984) are proposed to construct decision
trees for classification and regression. Another
important attribute of the decision tree approach is that
it allows to capture the most important input variables
(e.g., simulation model parameters) (Breiman et al.
1984). Decision trees can also handle variable
interactions. In addition, Sanchez and Lucas (2002) and
Kleijnen et al. (2005) state that regression trees are more
interpretable compared to traditional regression models
since predictions for model outcomes can be easily
obtained by simply following the discriminating rules at
each node.
A random forest (RF) is an ensemble of individual
decision trees each using a random subset of training set
or input variables. In regression problems, the mean of
the outputs obtained from each tree is returned as
predictions. The random selection procedure and
combining the predictions of trees lead to significant
accuracy improvements in regression problems
(Breiman 2001; Alpaydin 2014). Compared to an
individual decision tree, results obtained from a random
forest are not directly interpretable since random forests
combine outputs obtained from a high number of trees
(e.g., 1000 trees). However, it is possible to visualize
the individual trees of a forest to observe the
discriminating rules. In addition, it is possible to
determine the most important variables (in our case,
input parameters of a simulation model) for prediction
(Breiman 2001; Chen et al. 2011).
Support vector regression (SVR) is an extension of
support vector machine (SVM) classification technique
to the problems with continuous outputs. Contrary to
traditional regression models aiming to minimize SSE,
SVR model incorporates ε-insensitive loss function,
which yields regression models robust to noise
(Alpaydin 2014). SVR is formulated as a quadratic
optimization problem. In its simple form, one can use
SVR for linear regression. However, the dual
formulation allows one to conduct nonlinear regression
by employing kernel functions such as polynomial
kernels and radial-basis functions. Besides these well-
known kernels, one can also develop specific kernels for
different applications (Alpaydin 2014). SVR technique
has been successfully implemented as metamodels to
approximate highly nonlinear systems (Zhu et al. 2009,
2012). Since agent-based models capture nonlinearities
embedded in a system, SVR stands as a promising tool
for metamodeling.
k-nearest neighbor regression (k-NN R) method simply
predicts the output of a test instance by averaging the
outputs of k-nearest neighbors of that test instance.
Contrary to abovementioned regression methods, k-NN
method does not require an explicit training step; we
only store the training set to determine k closest
neighbors and their output averages (Chen et al. 2009;
Hu et al. 2014). To quantify closeness, Euclidean and
Manhattan distance can be used (Juutilainen and Röning
2007).
For a more detailed background on the abovementioned
machine learning methods, the reader is referred to
Hastie et al. (2009).
EXPERIMENTAL DESIGN
Beer Game
In this paper, we use the Beer Game to conduct
experiments (Sterman 1989; Edali and Yasarcan 2014).
Beer Game is a four-agent supply chain simulation. In
this game, each agent controls the inventory level of one
of the four echelons, which can be listed as a retailer, a
wholesaler, a distributor, and a factory. Although the
agents are not allowed to communicate and share
information, they aim to minimize the team total cost at
the end of the game. Team total cost is the sum of the
individual costs of echelons. The individual cost of an
echelon is calculated by summing up inventory holding
and backlog costs generated at each simulated week.
We run the model for 520 simulated weeks. The main
outcome of interest of the Beer Game is a terminal
value, team total cost. Ordering behaviors of agents can
be represented by the anchoring and adjustment
heuristic (Tversky and Kahneman 1974). This heuristic
has two main parameters, namely stock adjustment
fraction (α) and weight of supply line (β) and each
agent implicitly uses these two parameters in ordering
decisions (Sterman 1989).
Beer Game is a suitable model as an experimental
platform since (i) it is a simple model with four agents
and eight parameters in total, (ii) it is highly nonlinear,
(iii) it can produce a rich set of outputs including
(unpredictable) chaotic behavior. In other words, it is a
simple model with complex behavior capabilities that
has the potential to challenge the predictive capabilities
of a metamodel to be trained based on limited input-
output tuples from this model. In that respect, it stands
as a very good experimental ground for our comparative
analysis on metamodeling approaches.
Output Type Categorization
Independent of the selected model, we can categorize
outputs of an agent-based simulation model. First
criterion is based on whether the output is related to a
single agent (individual-level) or to the population
(system-level). Second criterion considers the temporal
aspect of an output: (i) the output can be measured at a
single time point in the simulation horizon or (ii) it can
be a function of a (large) set of instantaneous
measurements over the simulation horizon (e.g, average
over time, total over time, maximum/minimum value
during a simulation run). This kind of output
categorization will give an idea about the difficulty of
predicting model outputs; we claim that predicting over-
time values are easier than predicting instantaneous
values and predicting system-level outputs are easier
than predicting individual-level outputs.
Table 1: Output Type Categorization for the Beer Game
Instantaneous
Values
Over-Time
Values
Individual-Level
Output
Inventory level
of the retailer at
week t
Maximum
inventory
level of the
retailer
System-Level
Output
Team total cost
at week t
Team total
cost at final
time
In this study, we generate metamodels for the prediction
of two different outputs of the Beer Game: The first one
is a system-level output and is an over-time value, team
total cost. The second output is individual-level and an
over-time value, maximum inventory level of the
retailer. We consider two different sets of model input
parameters: (i) αR (stock adjustment fraction of the
retailer), (ii) αR and βR (stock adjustment fraction and
weight of supply line of the retailer) as metamodel (and
simulation model) inputs. All of the parameter values of
remaining echelons are set to the average values of
these parameters used by the participants in the board
version of the game (Sterman 1989).
Datasets
We use two different datasets in the context of this
study: a training set and a test set. In each dataset,
sample points have two main components; the first one
is parameter values and the second component is the
simulation model output obtained by running the model
with these parameters. Training set is used to fit a
metamodel. We employ the sampling techniques
mentioned in the “sampling techniques” section for
training set generation. Since RLHS, MLHS, and RS
techniques generate different samples due to
randomness, we generate 30 sets of samples by using
each technique and fit a metamodel with each one of
these 30 sets. For the experiments where the input
parameter is only αR, we generate 21 sample points with
each sampling technique for metamodel fitting. In the
two-parameter case (i.e., αR and βR), we generate 25
sample points. For one- and two-parameter cases, we
use test sets each having 5,000 instances to assess the
prediction performance of the metamodels.
Hyperparameter Optimization
The metamodeling techniques that are used in this study
have some hyperparameters to be optimized. These
hyperparameters are C (penalty factor), ε (parameter of
the epsilon-insensitive loss function), and γ (spread
parameter of the Gaussian kernel) in SVR; ntree
(number of trees in the forest) and mtry (number of
randomly selected candidate variables at each split) in
RF; k (number of neighbors) in k-NN R; minsplit
(minimum number of instances in a node for splitting),
minbucket (minimum number of instances in a terminal
node), and cp (complexity parameter) in DT.
To optimize the hyperparameters of SVR, RF, and k-NN
R, we perform a grid search on the selected subset of
hyperparameter space of each technique. For each
hyperparameter combination, we perform leave-one-out
cross-validation on the training set. Then, the
metamodel with the hyperparameter combination
yielding the minimum leave-one-out cross-validation
error is selected. Finally, the metamodel with the
selected hyperparameters is used to predict the instances
on the test set. Hyperparameter subsets of each
metamodeling technique considered in optimization are
given in Table 2. However, in the DT method, we
follow a different procedure: We first fully grow a tree
by setting minsplit = 2, minbucket = 1, and cp = 0. Then,
we prune the tree. For tree pruning, the reader is
referred to Breiman et al. (1984).
Table 2: Hyperparameter subset of each metamodeling
technique
Metamodeling
Technique Hyperparameters
Support
Vector
Regression
C ∈ {10-3
, 10-2
, 10-1
, 1, 101, 10
2,
103}
ε ∈ {10-3
, 10-2
, 10-1
, 1}
γ ∈ {10-3
, 10-2
, 10-1
, 1}
Random
Forest
ntree ∈ {50, 100, 150, 200, 500,
1000, 2000}
mtry ∈ {1} (one-parameter case),
mtry ∈ {1, 2} (two-parameter case)
k-NN
Regression k ∈ {1, 3, 5, 7, 9}
As we mentioned, we generate 30 sample sets for
RLHS, MLHS, and RS. We perform hyperparameter
optimization on each sample set individually.
Metamodel Performance Evaluation Criteria
The main performance criteria for metamodel
evaluation is Mean Absolute Percentage Error (MAPE),
which is given as MAPE = (1 / N) × Σ |y i − yi| / yi, where
y i and yi are metamodel prediction and simulation
model output, respectively. Besides MAPE, we also
report Percentage Distribution of Relative Prediction
Error (PDRPEx%) (Alam et al. 2004), which calculates
the percentage of the metamodel outputs whose Relative
Prediction Error (RPEi = y i / yi) are within ±x% error.
RESULTS AND DISCUSSIONS
In this section, we present the results of the experiments
and some discussion on the results. In each table,
PDRPE10% and MAPE values show the performance on
the test set. MT stands for Metamodeling Technique and
ST stands for Sampling Technique. Total Time (TT) is
the sum of simulation time (for running the simulation
model with the parameter values obtained from
sampling), training time (for training the metamodel and
hyperparameter optimization) and test time (runtime of
the metamodel for the prediction of the instances in the
test set). All the reported times are in seconds. For
RLHS, MLHS, and RS, we give the averages of the
performance measures since they are replicated 30
times.
Case 1: One Parameter – System-Level Output
Case 1 consists of experiments where the model output
is team total cost and the only model input is αR.
Detailed results are given in Table 3. The first
observation is that all the methods yield similar MAPE
values around 11%, which is a satisfactory result with a
considerably small training set size. The lowest MAPE,
which is 9.96%, is achieved when we use k-NN
regression with full factorial sampling design. Besides,
regardless of the metamodeling and sampling technique,
83% of the test set instances are within ±10% error on
average. Another clear observation is that random
sampling method yields slightly higher MAPE values
for each metamodeling technique compared to the other
sampling techniques. Random forest is the most time-
consuming method since we take 30 replications due to
the random training and parameter subset selection of
the method. Decision tree stands as the fastest technique
compared to the other techniques.
Table 3: Results of experiments when model output is
team total cost and model input is αR (Case 1)
MT ST PDRPE10% MAPE TT
SVR
FFD 85.66 11.66 7.79
RLHS 87.14 11.22 7.66
MLHS 84.99 11.59 7.71
RS 85.03 12.03 7.54
DT
FFD 85.28 12.36 2.41
RLHS 78.95 12.26 2.22
MLHS 78.98 12.30 2.33
RS 77.46 13.17 2.23
RF
FFD 84.74 10.38 31.24
RLHS 83.58 11.38 28.48
MLHS 82.61 11.44 27.38
RS 80.08 12.26 27.37
k-NN
R
FFD 85.96 9.96 4.44
RLHS 81.16 11.70 4.26
MLHS 81.62 11.84 4.37
RS 79.39 12.70 4.27
Case 2: Two Parameters – System-Level Output
In Case 2, the model output is team total cost and the
model input parameters are αR and βR. We observe that
performance of each method significantly deteriorates
(with an average 28% increase in MAPE) compared to
one-parameter case (Case 1). However, support vector
regression is the least affected method (with an average
20% increase in MAPE) by the increase in the
dimension and performs best in all sampling techniques.
The lowest MAPE, which is 29.3%, is achieved when
we use support vector regression with full factorial
sampling design. Besides, this metamodeling and
sampling technique combination yields significantly
high PDRPE10% (65.10%) value compared to the other
results in this case. k-NN regression is the second best
method (Table 4).
Table 4: Results of experiments when model output is
team total cost and model inputs are αR and βR (Case 2)
MT ST PDRPE10% MAPE TT
SVR
FFD 65.10 29.30 10.45
RLHS 49.15 33.03 10.05
MLHS 55.54 31.15 10.23
RS 50.39 34.02 10.09
DT
FFD 46.88 37.43 2.90
RLHS 34.23 46.48 2.63
MLHS 28.93 56.78 3.64
RS 31.82 49.01 2.63
RF
FFD 38.38 47.62 79.36
RLHS 41.76 40.70 89.02
MLHS 43.71 41.68 89.61
RS 39.70 42.20 89.37
k-NN
R
FFD 49.40 36.60 5.42
RLHS 41.80 35.01 5.12
MLHS 45.63 34.97 5.17
RS 41.03 36.59 5.13
Case 3: One Parameter – Individual-Level Output
In Case 3, the model output is maximum inventory level
of the retailer and the model input is αR. Detailed
experimental results are given in Table 5.
Table 5: Results of experiments when model output is
maximum inventory level of the retailer and model
input is αR (Case 3).
MT ST PDRPE10% MAPE TT
SVR
FFD 51.40 48.00 7.70
RLHS 42.49 49.28 7.51
MLHS 43.99 48.61 7.69
R 38.53 52.98 7.46
DT
FFD 46.16 53.31 2.30
RLHS 41.34 51.92 2.21
MLHS 40.62 51.01 2.40
RS 36.84 55.14 2.21
RF
FFD 46.62 46.96 30.56
RLHS 44.61 48.27 28.70
MLHS 43.68 48.69 28.56
RS 39.67 51.52 28.02
k-NN
R
FFD 48.14 48.85 4.27
RLHS 43.55 51.81 4.21
MLHS 42.38 50.62 4.41
RS 38.73 55.75 4.22
Compared to Case 1, MAPE values much higher since
the coefficient of variation of the outputs is larger in
Case 3. All of the methods perform similar in terms of
MAPE. However, differences between MAPE values
are much higher compared to Case 1. The minimum
MAPE (46.96%) is achieved when we use random
forest with full factorial design. Random sampling
method gives slightly worse results in terms of MAPE
compared to the other sampling techniques for each
metamodeling technique.
Case 4: Two Parameters – Individual-Level Output
In Case 4, the model output is maximum inventory level
of the retailer and the model input parameters are αR
and βR. In this case, we obtain very high MAPE values
(see Table 6), even larger than 100% (DT, RF, and k-
NN R). The results indicate that the output is very hard
to predict with a limited number of training points. The
minimum MAPE, which is 66.10%, is achieved when
we use support vector regression with maximin LHS.
SVR gives the minimum MAPE values for all of the
sampling techniques.
Table 6: Results of experiments when model output is
maximum inventory level of the retailer and model
inputs are αR and βR (Case 4).
MT ST PDRPE10% MAPE TT
SVR
FFD 28.00 78.46 10.54
RLHS 25.28 74.71 10.10
MLHS 26.74 66.10 10.15
RS 24.66 78.32 10.03
DT
FFD 13.62 131.38 2.89
RLHS 14.22 123.38 2.64
MLHS 13.43 141.18 2.65
RS 15.30 116.53 2.64
RF
FFD 16.76 114.14 81.42
RLHS 19.15 100.37 88.80
MLHS 21.93 93.76 90.58
RS 17.62 106.91 87.11
k-NN
R
FFD 14.96 131.08 5.44
RLHS 19.76 80.97 5.15
MLHS 19.94 82.22 5.17
RS 18.12 93.78 5.15
CONCLUSIONS AND FUTURE WORK
In this study, we employ four different regression
techniques from machine learning domain for
metamodeling. For each metamodeling technique, we
consider four different sampling techniques for training
purposes. Results show that the analyst can obtain
predictions using a metamodel trained with a small
training set instead of running the simulation model in a
relatively short time (e.g., 80% shorter than running the
simulation model). However, to increase the prediction
accuracy of a metamodel, the analyst should expand the
training set, which naturally increases training time.
Although the metamodeling techniques used in this
study are time-saving, the analyst should use them with
caution since metamodel accuracies depend on output
types. Results show that team total cost, which is a
system-level output, is predicted with higher accuracy
compared to maximum inventory level of the retailer
under equal sample sizes. We can conclude that system-
level outputs are easier to predict compared to
individual-level outputs. However, this claim should be
validated by further experimenting with other agent-
based models. We also observe that we should increase
the training set size when the model output is
individual-level or system-level with high dimensions to
obtain better metamodel predictions.
Experimental results show that there is no single
metamodeling or sampling technique performing best in
all cases. However, we observe that support vector
regression is more robust to the increase in the
dimension of the problem. Besides, in all of the four
cases, highest proportion of instances predicted with
maximum 10% error are realized when we use support
vector regression method.
Although the error values are very high in some cases,
metamodels can guide the analyst to explore and focus
on the parameter subspaces where model output
deviates from the regular form captured by the
metamodel. These subspaces will potentially be in the
neighborhood of the sample points where the error
values obtained by leave-one-out cross-validation
process are high. In that respect, the added value of
metamodels may be more about guiding the model
exploration process, rather than substituting the model.
A metamodel that is trained with a small training set
(e.g., 25 model runs) may narrow down the parameter
space that needs to be explored significantly, and reduce
the time and effort required in exploring the behavior
space of an agent-based model.
As a continuation of this study, we are planning to
increase the input parameter set gradually up to eight
with the Beer Game, and observe the performance
deterioration as well as the required increase in the
training set to compensate that. Furthermore, we plan to
expand the study by following a similar procedure with
other agent-based models.
ACKNOWLEDGEMENTS
This research is supported by Bogazici University
Research Fund (Grant No: 12560 - 17A03D1).
REFERENCES
Alam, F. M., K. R. McNaught, and T. J. Ringrose. 2004. “A
Comparison of Experimental Designs in the Development
of a Neural Network Simulation Metamodel”. Simulation
Modelling Practice and Theory 12 (7): 559–578.
Alpaydin, E. 2014. Introduction to Machine Learning. MIT
Press.
Axtell, R. L., J. M. Epstein, J. S. Dean, G. J. Gumerman, A. C.
Swedlund, J. Harburger, S. Chakravarty, R. Hammond, J.
Parker, and M. Parker. 2002. “Population Growth and
Collapse in a Multiagent Model of the Kayenta Anasazi in
Long House Valley”. Proceedings of the National
Academy of Sciences 99 (suppl 3): 7275–7279.
Beachkofski, B., and R. Grandhi. 2002. “Improved Distributed
Hypercube Sampling”. In Proceedings of the 43rd
AIAA/ASME/ASCE/AHS/ASC Structures, Structural
Dynamics, and Materials Conference, 1274.
Bishop, C. M. 2006. Pattern Recognition and Machine
Learning. Springer.
Breiman, L. 2001. “Random Forests”. Machine Learning 45
(1): 5–32.
Breiman, L., J. Friedman, C. J. Stone, and R. A. Olshen. 1984.
Classification and Regression Trees. CRC Press.
Chen, X., M. Wang, and H. Zhang. 2011. “The Use of
Classification Trees for Bioinformatics”. Wiley
Interdisciplinary Reviews: Data Mining and Knowledge
Discovery 1 (1): 55–63.
Chen, Y., E. K. Garcia, M. R. Gupta, A. Rahimi, and L.
Cazzanti. 2009. “Similarity-Based Classification:
Concepts and Algorithms”. Journal of Machine Learning
Research 10 (Mar): 747–776.
Crombecq, K., L. De Tommasi, D. Gorissen, and T. Dhaene.
2009. “A Novel Sequential Design Strategy for Global
Surrogate Modeling”. In Proceedings of the 2009 Winter
Simulation Conference, 731–742. IEEE, Piscataway, N.J.
Deutsch, J. L., and C. V. Deutsch. 2012. “Latin Hypercube
Sampling with Multidimensional Uniformity”. Journal of
Statistical Planning and Inference 142 (3): 763–772.
Edali, M., and H. Yasarcan. 2014. “A Mathematical Model of
the Beer Game”. Journal of Artificial Societies and Social
Simulation 17 (4).
Edmonds, B., and D. Hales. 2005. “Computational Simulation
as Theoretical Experiment”. Journal of Mathematical
Sociology 29 (3): 209–232.
Epstein, J. M., and R. Axtell. 1996. Growing Artificial
Societies: Social Science from the Bottom Up. Brookings
Institution Press.
Gilbert, N. 2008. Agent-Based Models. Number 153. Sage.
Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements
of Statistical Learning: Data Mining, Inference, and
Prediction. Springer-Verlag.
Hu, C., G. Jain, P. Zhang, C. Schmidt, P. Gomadam, and T.
Gorka. 2014. “Data-Driven Method Based on Particle
Swarm Optimization and k-Nearest Neighbor Regression
for Estimating Capacity of Lithium-Ion Battery”. Applied
Energy 129:49–55.
Juutilainen, I., and J. Röning. 2007. “A Method for Measuring
Distance from a Training Data Set”. Communications in
Statistics–Theory and Methods 36 (14): 2625–2639.
Keane, A., and P. Nair. 2005. Computational Approaches for
Aerospace Design: The Pursuit of Excellence. John Wiley
& Sons.
Kleijnen, J. P., S. M. Sanchez, T.W. Lucas, and T. M. Cioppa.
2005. “State-of-the-Art Review: A Users Guide to the
Brave New World of Designing Simulation Experiments”.
INFORMS Journal on Computing 17 (3): 263–289.
Kleijnen, J. P., and R. G. Sargent. 2000. “A Methodology for
Fitting and Validating Metamodels in Simulation”.
European Journal of Operational Research 120 (1): 14–29.
Kleijnen, J. P., and W. C. van Beers. 2004. “Application-
Driven Sequential Designs for Simulation Experiments:
Kriging Metamodelling”. Journal of the Operational
Research Society 55 (8): 876–883.
Lee, J.-S., T. Filatova, A. Ligmann-Zielinska, B. Hassani-
Mahmooei, F. Stonedahl, I. Lorscheid, A. Voinov, J. G.
Polhill, Z. Sun, and D. C. Parker. 2015. “The Complexities
of Agent-Based Modeling Output Analysis”. Journal of
Artificial Societies and Social Simulation 18 (4).
Loh, W.-Y. 2011. “Classification and Regression Trees”.
Wiley Interdisciplinary Reviews: Data Mining and
Knowledge Discovery 1 (1): 14–23.
Macal, C. M. 2010. “To Agent-Based Simulation from System
Dynamics”. In Proceedings of the 2010 Winter Simulation
Conference, 371–382. IEEE, Piscataway, N.J.
McKay, M. D., R. J. Beckman, and W. J. Conover. 1979. “A
Comparison of Three Methods for Selecting Values of
Input Variables in the Analysis of Output from a
Computer Code”. Technometrics 21 (2): 239–245.
Montgomery, D. C. 2013. Design and Analysis of
Experiments. John Wiley & Sons.
Quinlan, J. R. 1993. C4.5: Programming for Machine
Learning. Morgan Kauffmann.
Sanchez, S. M., and T.W. Lucas. 2002. “Exploring theWorld
of Agent-Based Simulations: Simple Models, Complex
Analyses”. In Proceedings of the 2002 Winter Simulation
Conference, 116–126. IEEE, Piscataway, N.J.
Sterman, J. D. 1989. “Modeling Managerial Behavior:
Misperceptions of Feedback in a Dynamic Decision
Making Experiment”. Management Science 35 (3): 321–
339.
ten Broeke, G., G. van Voorn, and A. Ligtenberg. 2016.
“Which Sensitivity Analysis Method Should I Use for My
Agent-Based Model?”. Journal of Artificial Societies and
Social Simulation 19 (1).
Tesfatsion, L. 2002. “Agent-Based Computational Economics:
Growing Economies from the Bottom Up”. Artificial Life
8 (1): 55–82.
Tversky, A., and D. Kahneman. 1974. “Judgment Under
Uncertainty: Heuristics and Biases”. Science 185 (4157):
1124–1131.
Zhu, P., F. Pan, W. Chen, and S. Zhang. 2012. “Use of
Support Vector Regression in Structural Optimization:
Application to Vehicle Crashworthiness Design”.
Mathematics and Computers in Simulation 86: 21–31.
Zhu, P., Y. Zhang, and G. Chen. 2009. “Metamodel-Based
Lightweight Design of an Automotive Front-Body
Structure using Robust Optimization”. Proceedings of the
Institution of Mechanical Engineers, Part D: Journal of
Automobile Engineering 223 (9): 1133–1147.
Wilensky, U., and W. Rand. 2015. An Introduction to Agent-
Based Modeling: Modeling Natural, Social, and
Engineered Complex Systems with NetLogo. MIT Press.
AUTHOR BIOGRAPHIES
MERT EDALI is a Research and Teaching Assistant in
Industrial Engineering Department at Yildiz Technical
University. He earned his B.S. degree from Yildiz
Technical University, Istanbul, Turkey, in 2011. He
earned his M.S. degree in Industrial Engineering from
Bogazici University, Istanbul, Turkey, where he
continues his studies as a PhD student. His e-mail
address is medali@yildiz.edu.tr.
GONENC YUCEL is an Associate Professor in
Industrial Engineering Department at Bogazici
University. He received his B.S. and M.S. degrees in
Industrial Engineering from Bogazici University in
2000 and 2004. He earned his PhD degree in Policy
Analysis from Delft University of Technology. He has
been focusing on simulation-supported policy and
strategy analysis in his research, utilizing agent-based,
as well as system dynamics models. His email address is
gonenc.yucel@boun.edu.tr.
Recommended