Ari Vasvaria

Simulated neurons learning games

A network of 2,700 simulated neurons that plays Pong, Connect 4 and Doom, and a smaller agent that learns to aim in Doom.

C / OpenMP / Python / ViZDoom

Learning to aim in Doom

The task is the basic Doom scenario: a monster appears somewhere along the far wall, and the agent can turn left, turn right or shoot. It gets 101 points for a kill, loses 5 for each missed shot and loses 1 for every moment that passes.

The trained agent turning toward the monster and shooting.
An agent pressing random buttons.
Left: the agent after 3,000 episodes of training. Right: random button presses.

I trained two versions. One is told where the monster is. The other has to find it through its own retina, which is sharp in the centre and blurry at the edges like an eye. The chart shows the average score as training goes on.

0-50-100-1505001,0001,5002,0002,5003,000Episodes playedAverage reward, last 500 episodes
Trained, given the monster's positionTrained, seeing through its own retinaRandom actions
Two runs of each. The agent that is given the monster's position goes from about −110 to about −2. The one using its own retina goes from −144 to about −90. Random play stays near −150.

The agent is an actor-critic with a tonic average-reward baseline. The retina has a one-pixel foveal strip and coarser peripheral and motion pathways. Each configuration was run with two seeds for 3,000 episodes, with a frozen-weights control and a random-action control. The frozen controls score the same as random.

A network of 2,700 neurons

The voltage of four simulated neurons, each driven harder than the one above.

This is a separate, larger model. Each neuron is simulated with the Hodgkin–Huxley equations, which describe how the voltage across a real nerve cell rises and falls as channels in its membrane open and close.

The neurons are arranged in three parts: a retina that looks at a 32×32 picture of the game, a sheet of cells that responds to it, and a group that picks the button to press. Connections change strength based on the activity of the two neurons they join and on one signal that says whether something felt good or bad. It does not use backpropagation.

The same network runs Pong, Connect 4 and Doom, keeping its connections between games. It runs at about 38 game frames per second on a laptop CPU. It does not play well yet: in its last long Pong run it lost 387 to 0 against a simple scripted opponent. Getting it to learn the games properly is what I am working on.

Written in C with OpenMP. Plasticity is calcium-based, with calcium-activated potassium currents for adaptation, synaptic scaling to hold firing rates in range, and dopamine-gated synaptic tagging. Each game frame advances 10 ms of simulated time. After 30,000 Pong frames, excitatory cells in the sheet fire at about 10 Hz.