Can we trust computer models to accurately represent reality?
Computer models are widely used in science. But when can we trust them?
Computer models are often used in science to aid in understanding the world. Computer models are simulations of parts of the real world based on known scientific principles. I use computer models of radiation transport everyday in my work as a physicist. Astronomers use computer models to predict solar eclipses, comet paths, and satellite orbits.
The thing about computer models is that they are not the real world. Science depends on empirical observations in order to progress. Computer models are only useful when they accurately depict reality.
There is a very high bar to jump over to validate a computer model before its results can be considered to be valid science. When working with computer models of physical systems, we have to be careful when to trust them. This should primarily be the responsibility of the scientists who work with them. They are after all the ones most familiar with how computer models are constructed, and they have an ethical obligation to the public to be transparent about the strengths and weaknesses of a computer model.
Although scientists are good, honest people working hard to help us all understand our world, we have to accept that there are political and financial forces out there that may bias a scientist to skew results in a certain way, especially when models are so complex it is easy to hide monkey business. As well, we unfortunately have to acknowledge that there are bad actors in the media and government to further bias the communication of results from computer models.
This is worrisome because computer models can steer public policy in dramatic ways. For example, climate change models are used to predict future global warming and downstream effects. If these climate models are overblown in their negative assessments, then governments may place limits on cheap, reliable energy which if unreplaced may negatively impact the lives of billions around the world. Alternatively, if climate models are overly optimistic in their projections and we don’t react proportionally, well, that’s clearly not good either.
It is therefore crucial that the general public be able to detect computer model non-science. The public ultimately decides on public policy in democratic nations. A well-informed public should be able to call out computer models that are used to justify drastic government intervention in people’s lives.
Fortunately, there are simple ways to estimate the credibility of a computer model. First off, it is instructive to know when computer models succeed.
When computer models succeed
Science is the knowledge acquired from the repeatable application of the scientific method: generate a falsifiable hypothesis, test predictions from the hypothesis with appropriate experiments, observe the results and draw conclusions whether the hypothesis should be accepted, modified, or discarded.
Most importantly, science is inherently empirical. Real-world measurement is king. This makes computer modeling inherently suspect. The modeler’s claim is that the computer can accurately simulate and predict reality and is therefore scientific.
This is an incredibly tall claim. We can program computers to give any fiction that we want. A computer model spouting fiction is known as JIJO (junk in, junk out). Computer models must be rigorously validated before put to use, especially when used for serious applications such as medicine or public policy formation.
The scientific method can aid us in developing a general checklist for determining if a computer model has a shot at predicting reality. The scientific method demands empiricism, repeatability, and falsifiability. The following checklist is a probe as to whether a computer model adheres to good scientific practice.
Computer models succeed when the following are true:
All of the critical inputs to the system are accurately known.
There is a well-tested theory governing the computer simulation.
The system being modeled is not contaminated by chaos and undefinable complexity.
The computer model outputs can be directly tested against real-world observations.
The model is not extrapolated beyond its tested boundaries.
Consider a computer model of a rocket ship accelerating in space. All of the critical inputs--thrust force, mass, gravitational forces from other heavenly bodies--are known accurately enough to produce a reliable estimate of its trajectory. We have a rock-solid theory of gravity thanks to standing on the shoulders of Newton and Einstein. This theory is governed by deterministic equations, meaning that you can throw in all the input quantities into the equations and produce a single answer. The computer model can be directly tested in the most severe way possible: if it is wrong, then spaceships are lost and astronauts die. Fortunately, such computer models of rocket trajectories in space have been thoroughly worked out, as evidenced by the general success of space programs around the world.
When to become suspicious of computer models
This is when you should become suspicious. These are the converse of the five points above, and form a test of the credibility of a computer model.
All of the critical inputs are not known. It is extra suspicious if there is One Factor that drives the model’s response.
Some of the rules are not based on known scientific theory. Extra skepticism is warranted if fudge factors are used.
The modeled system is a complex, chaotic process. This is always the case when attempting to model human behavior.
There is inadequate ability to test the model with real-world observations.
The model is used to extrapolate beyond its tested boundaries.
Once again, this test is derived directly from the scientific method, which demands real-world observation. A computer model already somewhat violates this rule, since the data it produces is not real-world data. The only way a computer model is justified is if it produces accurate predictions of the physical system that it models. It is unlikely to accurately model reality when any or all of the above five points are true.
The five-point test to determine computer model credibility is built on extremely basic science: computer models are only valid insofar as they can accurately predict the response of real-world physical systems. The general public needs to understand when and how computer models are limited in order to stay informed on how these models can impact all of our lives.