Project Aristotle

While I worked for Vulcan, Paul Allen asked the provacative question, "How good are we at AI and is there any evidence we're getting any better?" Project Halo was my attempt to answer. We put out an RFP for companies to write software that could take the AP Chemistry exam.

I chose an AP exam because a lot of effort is put into creating tests every year of roughly uniform difficulty. I chose Chemistry because it is a "textbook science" meaning the answers to the questions rely almost exclusively on material in the textbook and a small number of precursors. (In contrast to say English literature where the answers rely on an uncountable number of hard to define lessons about human nature, love, etc.) Physics and Math were eliminated because they relied too much on symbol manipulation and diagrams and I thought we'd spend all our time writing software to read force diagrams or math equations and never get to the actual reasoning part.

A secondary goal was to stimulate progress in the field and so a condition of acceptance was that all the projects be released under a GPL license on the theory that results could be reproduced and extended at low incremental cost.

Ultimately we hired three teams to perform in the context. We released a number of questions to them ahead of time they could build systems against. We then "sequestered" their systems and introduced a new set of questions. The theory was that the experiement could be run repeatedly over the years to measure progress but to my knowledge that did not occur. I left the company before the ultimate publication of the results but you can see those publications and some of the related publicity and artifacts on this page.


Vulcan Inc. - The Halo Pilot: Towards A Digital Aristotle

"Project Halo" Exceeds Expectations for Automated Reasoning by Artificial Intelligent Systems

Vulcan Inc. - Vulcan Inc Completes First Step Toward Digital Aristotle

"Project Halo" Exceeds Expectations for Automated Reasoning by Artificial Intelligent Systems

Seattle PI - Allen claims success in work on computers that can reason

Project Halo aims to develop a 'Digital Aristotle'

AI Magazine - Project Halo: Towards a Digital Aristotle - Towards a Quantitative, Platform-Independent Analysis of Knowledge Systems

AI Magazine - Project Halo Update: Progress Toward Digital Aristotle