While I worked for Vulcan, Paul Allen asked the provacative question, "How good are we at AI and is there any evidence we're getting any better?" Project Halo was my attempt to answer. We put out an RFP for companies to write software that could take the AP Chemistry exam.

I chose an AP exam because a lot of effort is put into creating tests every year of roughly uniform difficulty. I chose Chemistry because it is a "textbook science" meaning the answers to the questions rely almost exclusively on material in the textbook and a small number of precursors. (In contrast to say English literature where the answers rely on an uncountable number of hard to define lessons about human nature, love, etc.) Physics and Math were eliminated because they relied too much on symbol manipulation and diagrams and I thought we'd spend all our time writing software to read force diagrams or math equations and never get to the actual reasoning part.

A secondary goal was to stimulate progress in the field and so a condition of acceptance was that all the projects be released under a GPL license on the theory that results could be reproduced and extended at low incremental cost.

Ultimately we hired three teams to perform in the context. We released a number of questions to them ahead of time they could build systems against. We then "sequestered" their systems and introduced a new set of questions. The theory was that the experiement could be run repeatedly over the years to measure progress but to my knowledge that did not occur. I left the company before the ultimate publication of the results but you can see those publications and some of the related publicity and artifacts on this page.