Prior results

Heuristica can be used to reproduce results from prior research, making sure to have the same data set and heuristics. Consider the city population results on page 103 of Simple Heuristics That Make Us Smart (citation below). It reported the following percent correct for models fitting city populations– that is, when the models are given all 83 cities to fit and then are asked to estimate the population of the same 83 cities:

Getting the same data and models

Simulation

Now fit the models to the data. Regression will determine its beta weights, ttb will determine cue order, and unit weight linear and minimalist will determine cue direction (whether having a soccer team is associate with higher or lower population).

data_set <- city_population_original
criterion_col <- 3    # Population
cols_to_fit <- 4:ncol(data_set) # The 9 cues

reg <- regInterceptModel(data_set, criterion_col, cols_to_fit)
ttb <- ttbModel(data_set, criterion_col, cols_to_fit)
unit <- unitWeightModel(data_set, criterion_col, cols_to_fit)
min <- minModel(data_set, criterion_col, cols_to_fit)

To determine the percent of correct inference in fitting, pass the exact same data set into the percent correct function with the fitted models.

results <- percentCorrectList(data_set, list(reg, ttb, unit, min))
# Round values to make comparison easier.
round(results)

##   regInterceptModel ttbModel unitWeightModel minModel
## 1                74       74              74       70

The results match the results reported in the book– 74% correct for all models except minimalist, which got only 70% correct.

Prediction

The code above simulates fitting. To simulate prediction, fit the models with a subset of the cities and run percentCorrect with the other cities. Then repeat for various ways to split the data and run summary statistics on the results. Below is a function to do this from the cross-validation vignette.

# Cross-validate the data over the vector of models, suing training-proportion
# of the data to train and the rest as a test set.  Outputs the mean percent
# correct for each model in fitting and in prediction.
crossV <- function(vec_of_models, criterion_col, cols_to_fit, data, reps, training_proportion){
  fitting <- vector()
  prediction <- vector()
  for(i in 1:reps){

    #randomly sample training and test row indexes
    train <- sample(1:nrow(data), nrow(data)*training_proportion)
    test <- setdiff(1:nrow(data), train)

    #create training and test set
    training_set <- data[train,]
    test_set <- data[test,]

    # If a regression is overdetermined (e.g. has too many columns(), it will
    # drop the right-most columns.  To instead make it drop random columns,
    # we shuffle the column order.
    shuffled_cols_to_fit <- sample(cols_to_fit)

    models<-list()
    y <- 0
    for (mod in vec_of_models) { #fit the models to the training_set
      y <- y+1
      models[[y]] <- mod(training_set, criterion_col, shuffled_cols_to_fit)
    }

    #calculate percentage of correct predictions
    fittingAccuracy <- percentCorrectList(training_set, models)
    predictionAccuracy <- percentCorrectList(test_set, models)
    fitting <- rbind(fitting,fittingAccuracy)
    prediction <- rbind(prediction,predictionAccuracy)
  }

  results <- (rbind(colMeans(fitting),colMeans(prediction)))
  rownames(results) <- c("Fitting","Prediction")
  results
}

Now use the function with the models and data defined above.

set.seed(0) # Use the same seed if you want to reproduce same results.
reps <- 200 # The book used 10,000 but that takes minutes.
training_proportion <- 0.5

results <- crossV(c(regInterceptModel, ttbModel, unitWeightModel, minModel), criterion_col, cols_to_fit, data_set, reps,training_proportion)
# Round values to make comparison easier.
round(results, 1)

##            regInterceptModel ttbModel unitWeightModel minModel
## Fitting                 75.2     74.9            73.6     70.4
## Prediction              70.8     71.0            70.2     67.0

Notice that

• Accuracy is always lower for prediction compared to fitting. (Predicting out of sample is harder than fitting a known sample.)
• ttbModel is more accurate than regression in prediction on the cities data set.

This result differs from some others for several reasons

• The book used a regression model with intercept, so that is what was used here. However, the example on the cross-validation page uses a non-intercept regression, so slightly different answers are expected.
• A few hundred reps are not enough for stable results. The book used 10,000.

The book does not actually report the prediction results of the cities data set individually, but the results are similar to the average results. Some of the individual plots in book also show the large accuracy drop for regression between fitting and prediction, since this data represents the point at 9.21 objects per cue (83 cities / 9 cues).

Below is a fitting vs. prediction plot but using just the cities data. The book shows this type of plot across all 20 data sets.

library(ggplot2)
library(reshape)
p <- melt(results)

## Warning in type.convert.default(X[[i]], ...): 'as.is' should be specified by
## the caller; using TRUE
## Warning in type.convert.default(X[[i]], ...): 'as.is' should be specified by
## the caller; using TRUE

colnames(p) <- c("condition","model","value")
ggplot(p, aes(x=condition, y=value, colour=model,group=model)) + geom_point() + geom_line() + ylab("Proportion correct")

Citations

Gigerenzer, G., Todd, P. M., & the ABC Group (1999). Simple heuristics that make us smart. New York: Oxford University Press.