Math Riff - The Supercomputer Brain Part 1

There are all sorts of interesting supercomputer news out there recently. Los Alamos National Laboratory’s Roadrunner supercomputer narrowly edged out Oak Ridge National Laboratory’s Jaguar to take the title of the world’s most powerful with a score of 1.1 petaflops of computational power. Not to be outdone, Lawrence Livermore National Laboratory announced they were contracting with IBM to deliver a 20 petaflop machine in 2009. That would actually create a machine more powerful than the 500 fastest supercomputers on the planet and represents a significant leap forward. Naturally, all of us are looking for faster computers these days. Ray Kurzweil’s hope is for computers fast enough to deliver full blown supra-human artificial intelligence. As in brain-in-a-jar, download your head and live forever stuff. You can read all about it in his book The Singularity Is Near. Be forewarned… If you’re the least bit apprehensive about computers inside your skull, Ray’s train of thought may have you pricing cabins in Montana. Ray pegs the arrival of the Singularity as circa 2040. To hit that goal, Ray suggests we need computers that can perform on the order of 10 quadrillion floating point calculations per second. In supercomputer-speak, that’s 10 petaflops. That’s actually right in the realm of where supercomputers are running today, and IBM’s 20 petaflop supercomputer-beast is scheduled to be turned on later this year. So did the Singularity already deplane and we just missed it? Probably not. With post-graduate work in AI, and a day job involving some fairly large distributed transaction processing systems, the computational aspects of the problem here is interesting to me personally. Certainly, the power and capability of computers will continue to rise over the years and more “intelligent” software systems are coming. We’ll leave the ethical dimensions of whether you want to replace your brain with something running Microsoft Windows 9.0 to another blog, but for now the comparison between computers and brains make for an interesting math riff. In this two-part riff, we’ll look at a brain by-the-numbers to see how big it is, discuss how that relates to computer hardware and then reflect on what that means for near-term computer-based intelligence comparable to human capability. A Brain by the Numbers Nominally, we can reduce a human brain down to a very large collection of cells called neurons that are all interconnected. A human brain has roughly 100 billion (1011) neurons. That sounds like a lot, but in the computer world we’re used to dealing with large numbers and 100 billion bytes would be 100GB. Your hard disk is probably bigger than that, and you’d find the equivalent amount of data on roughly two Blu-Ray disks. A neuron modeled in a computer program is going to be bigger than a byte obviously, but even still we’re starting out with something that superficially seems to be of similar scale. So, why isn’t my computer doing all my work for me? Well, there’s a lot more complexity between your ears. Neurons need to “talk” to each other to solve problems, store memories or perform all the other tasks your brain ultimately does for you. In the brain these connections are created using biological wires called axons, and where an axon touches another neuron you get a connection called a synapse. To put it another way for modeling purposes, imagine your brain is a map and all of the neurons are cities on the map. The axons are the roads between individual cities. From any given city you can drive directly to some number of cities, but in most cases you may have to go through multiple towns and cities to get from any spot on the map to another. In the case of neurons and axons, it turns out there are on the order of 10,000 or so connections between any given neuron. In our map analogy, that’s like 10,000 roads leading in or out of a given city. It becomes clear that the complexity of our computer brain model isn’t so much dictated by the number of neurons as it is dictated by the number of axons. If we’re looking at 100 billion neurons, our model suddenly needs to account for 100 billion x 10,000 axons, or one quadrillion (1016) connections. If we could model an axon as a byte, that’s roughly 1,000 TB of data. That’s 500 of the latest and greatest 2TB hard disks on the market, or a number called a petabyte in the disk storage industry. Of course, like neurons, we can’t model an axon connection using a single byte. Realistically, we’re probably looking at structures with multiple pointers, threshold potential values or other similar bits of data. A good lower bound on the size of a computer data structure to model an axon and its connection between two neurons is probably around 32 bytes, so our total brain model storage is around 32 petabytes, or about 16,000 of those shiny new 2TB hard disks. I’m starting to feel a little prouder of my brain’s storage capacity. So how long before we can fit our brain design on a single disk drive? In terms of raw disk storage, 2TB drives are the current state-of-the-art. We need a 16,000x increase, and we know disk storage density is doubling right now roughly ever year. 16,000 is close to a power of two… 214 is 16,384. So, if we don’t run into any significant technical obstacles, we should have some storage device capable of holding 32PB of data in around 14 years. Not bad if the rate of progress holds out. Except storage devices are terribly slow. In fact, disk drives are thousands of times slower than the actual microprocessor that would need to look at our brain model. In fact, given a modern 320MB/s SATA disk interface, just reading the data off a 32PB disk drive would take three years. Disk interfaces will improve over time as well, but clearly we can’t use our brain model efficiently when it’s on a storage device. The key is getting the brain model into faster silicon memory so that we can work on it quickly. And that, unfortunately, starts to tell us where Ray’s dream machine breaks down. Read the rest of the article in Part 2