Question

2. Based on the enacted laws rate of 1% (169/15702), identify the possible types of misclassifications, and comment on the use of overall accuracy as a metric if a company claims they can predict if a bill will pass with 94% accuracy.

Answer 1

Due to the complexity of law-making and the aleatory uncertainty in the underlying social systems, we predict enactment probabilistically. It’s important to make probabilistic predictions for high consequence events because even small changes in probabilities for events with extreme implications can have large expected values. For instance, the 2009 stimulus bill cost $831 billion so even a 0.1 change in the predicted probability of this bill corresponds to a $83.1 billion dollar change in the expected value (the probability of an event multiplied by its consequences). Probabilities provide much more information than a simple “enact” or “not enact” prediction. Model performance metrics that don’t use probabilities, such as accuracy, are not suitable measures of rare event predictive ability. For instance, a blunt “never enact” model has a seemingly impressive 96% accuracy rate on this data but incorrectly classifies all the enacted bills with incalculable effects on society.