My Ph.D. is in Systems Science, which deals with the ways that things work together. As part of my studies I got to work with some really neat tools. The dissertation was about building a neural net supply chain management system using a neural net AI control concept originally developed as an autopilot for a single-stage-to-orbit hypersonic vehicle. It doesn’t get any cooler than that.
Neural net controllers work best when they’ve been properly trained and to properly train them you need input/output data for the “plant” they are controlling. Where to get the data? In particular, where to get the data when the system you are trying to control doesn’t exist yet? Enter simulation.
I built a discrete event simulation of a multi-layer supply chain, including both the inventory and the transport costs. In order to get a wide range of values, and to help design the system structure, I used another AI technique called a genetic algorithm to optimize the whole operation. See what I mean by cool? Here’s a picture
Playing with all my toys at one time
Unfortunately, I made an error. The GA was developing policies for re-ordering products and passing them back to the simulation. The simulation would run by applying those policies, and report out the total cost at the end of the run. Buried in the simulation code was a set of decision rules about how to apply the reorder policies. But there was an error in the decision rules, one that rewarded inappropriate actions. The GA, having no sense of ethics, or even common sense, noted that error and began to exploit it. Policy sets that were obviously bad were coming out with high scores and taking over the population. What to do?
Well, the what to do is, go into your simulator code and track down and fix the error. This let the system operate in an appropriate fashion and come out with usable results. There’s a lesson here — if you leave a loophole in your programming, your AI will find it and exploit it to achieve the goal you gave it.
People who talk about runaway AI often think of killer bots wandering the streets, shooting people. That makes for exciting graphics, but it’s not the real problem.
Who would you like to kill today?
The real problem comes when an AI has the ability to interact with the world, and modifies the world to suit its own goals. Elon Musk talks about a strawberry-picking AI that works to cover the world with strawberry fields. Or think of a stock fund AI that decides the best way to make money is to buy defense stocks, and then start a war. Or an AI that shorts a bunch of stocks, and then causes a market crash.
Nobody knows why the market crashed that day…
So, twenty years ago, my little AI project stumbled upon a problem that is likely to be at the center of AI research for the next twenty.
FoundOnWeb 