France began construction of the Maginot Line in the 1930s as a line of defenses against possible German invasion. The most heavily fortified section of the line ended just east of the Ardennes Forest, which most experts considered to have terrain that was too difficult to traverse for an invasion. German military planners exploited this hole, and the results were Dunkirk and the fall of France.

Human behavior also stymies less historically significant policies from water-conserving showerheads to company IT security protocols. Workarounds, hacks, and other manifestations of human ingenuity undermine the best plans. Despite these challenges, policymakers and private foundations are increasingly determined to use science, evidence, and data to guide their decisions.

Daniel Sarewitz, who heads the Consortium for Science, Policy and Outcomes, describes the limits of science as a guide to policy in the current issue of The New Atlantis. Policy asks questions about people and complex systems that cannot be reduced to a small number of variables the way a chemistry experiment can. Autonomous people are less predictable than hydrogen atoms in their interactions, which puts policy questions in the realm of what Alvin Weinberg called “trans-science” to describe “questions that can be asked of science and yet which cannot be answered by science.”

These are not questions of morality, which cannot be asked of science in the first place, but questions of the best approach to a problem. Sarewitz explains,

Our common belief is that scientific truth is a unitary thing — there is one fact of the matter, that’s why the light always goes on when I flip the switch. But trans-scientific questions often reveal multiple truths, depending in part on what aspects of an issue scientists decide to research and how they go about doing that research. There is no point in asking who is right in such cases; the best we can hope for is to understand why experts are disagreeing, which often hinges on highly technical choices about how to carry out research and analyze data, not to mention the hidden biases that help determine those choices.

The NC Government Efficiency and Reform (NC GEAR) initiative, which I led for nearly two years, took an evidence-based approach to education, public safety, transportation, and business subsidies. Evidence did reduce expectations for huge savings in some cases. For example, it would seem that reducing the number of people who return to prison would generate large savings for state government based on the average cost of $28,000 per year, but the cost of incarcerating one more person is closer to $4,000 per year because the facility, guards, and most other resources cannot be adjusted whether Central Prison has 1,094 inmates or 1,104.

On larger questions, however, there was sharp disagreement on the value of Medicaid according to the Oregon Experiment, the effects of school choice in post-Katrina New Orleans, and the public health and safety justification to license certain occupations. Different notions about what would work resulted in asking different questions, making different interpretations of the data, and reaching different conclusions.

Some of our policy disagreements come down to subtle differences. Do we place more emphasis on ameliorating need or on eliminating the systemic failures that create need?

If we choose to focus on the effects, is government or charity the best approach? How concerned should we be that our efforts may hinder the ability or willingness of those we help to change their circumstances?

If we focus on systemic failures, how do we know which ones are most salient? Do we ignore the current situation entirely and simply reassure the person that his or her grandchildren will be better off, we think, because of our efforts?

Any answer will mean a diversion of resources from one area to another, so these are fundamental to good budgets. Budget writers and policymakers may not be able to find better answers or better measures, but they can ask better questions, recognize the limits of knowledge, and constrain the potential harm from bad policy decisions by reining in the scale and scope of government.