ProjectsFlyBrain

FlyBrain

FlyBrain

1st Place 路 SteelHacks XIII

Team

Kuldeep Debnath

Ryan Rana

Ronit Chandresh Virwani

Hilary Chan

Date
September 2026
Event
SteelHacks XIII

A robot pilot made from a real fruit fly's brain wiring. FlyBrain runs a spiking neural network on Drosophila connectome data, so visual motion drives steering, thrust, and a giant-fiber escape reflex, starting with collision avoidance.

Overview

Robotics is not software. A drone or a small legged robot has a tight power and compute budget, and the reflexes it most needs (dodge the thing coming at you, steer toward the target) have to run fast on cheap hardware. A fruit fly already does this with about a hundred thousand neurons and almost no energy. FlyBrain borrows that solution directly: instead of training a controller, it uses the fly's actual wiring diagram as the controller.

The Python package loads the full MaleCNS connectome (about 166k neurons, released by HHMI Janelia FlyEM and collaborators) and simulates it as a leaky integrate-and-fire network using the published parameters from Shiu et al., Nature 2024. Camera frames go through a software retina that computes optic flow and looming per grid cell and drives the matching sensory cell types: T4/T5 for direction of motion, LPLC2 and LC4 for something expanding toward the eye. On the motor side, descending neurons are read out as flight commands. DNg02 left and right set thrust and yaw, DNp03 triggers evasive turns, and DNp01 (the giant fiber) fires the escape. A safety governor clamps every command before it reaches a drone backend (simulator, Tello, Crazyflie, MAVLink, or a UDP bridge).

For the demo, the whole engine is ported line by line to JavaScript and runs in the browser as SpyFly, a Three.js scene with an 850-neuron MiniFly circuit flying an articulated fly. You can drag a burger around the room and watch the attraction vector steer it, poke any neuron group from the brain panel, or swat at it with your own hand through MediaPipe webcam tracking. The hand shows up to the fly's camera as a looming object and sets off the giant-fiber escape. Every session can stream spike rates, motor output, and 3D positions to a TimescaleDB hypertable on Tiger Cloud for live analytics.

Challenge

The hard part was turning a connectome into something that flies. A wiring diagram has no inputs or outputs. It is a signed matrix of synapse counts, and nothing in it says which neurons a camera should drive or which firing rates mean "turn left." We had to map image features onto the right sensory cell types and decode descending-neuron activity into flight commands. Then the result had to run in real time on a laptop and in a browser tab, while keeping clear which parts are published biology and which are engineering shortcuts.

Approach

We grounded each side of the loop in the literature and wrote down every shortcut. Inputs follow known cell-type roles: T4/T5 subtypes get directional optic flow and LPLC2/LC4 get looming size and speed, because both synapse directly onto the giant fiber. Outputs follow published descending-neuron studies, so thrust is proportional to total DNg02 activity, yaw to the left/right difference, and escape fires when the smoothed DNp01 rate crosses a threshold. The read-out is a linear map that can be refit by ridge regression against a stimulus battery, and the fit only changes the read-out, never the connectome. To keep the simulator fast, spikes propagate through a CSC synapse matrix in one vectorized gather per step, so cost scales with spikes times out-degree rather than total synapses. The browser port is checked in CI against the Python version so both climb, hold, escape, and descend the same way.

Obstacles

  • A quiet scene produced a silent motor. With no visual input the descending neurons never fired and the fly just dropped.

    Added a small tonic bias current on DNg02 to stand in for the "I am flying" state that real flies get from many sources, and documented it as an engineering choice rather than biology.

  • Simulating every synapse every step was too slow for real-time control on a laptop, let alone in a browser.

    Event-driven propagation: only neurons that spike have their outgoing columns gathered, delayed input sits in a ring buffer, and dense bursts fall back to a sparse matrix-vector product.

  • A live demo needed a way for judges to interact with the brain, not just watch numbers.

    Turned the webcam into a stimulus. MediaPipe hand tracking drives a 6-DoF 3D hand that the fly's simulated camera actually sees, so a swat expands on the retina and fires the real looming-to-giant-fiber pathway. A scored swat game logs dodges, hits, and giant-fiber lead time.

Results

Won the No Wrapper prize at SteelHacks XIII. The live browser demo flies a fly that forages toward a target, brakes and dodges looming obstacles, and escapes a swatting hand with a multi-axis giant-fiber maneuver, with spike rates and escape efficiency charted live from Tiger Cloud. Next steps are validating collision avoidance on physical hardware and measuring reaction time and energy use against conventional controllers.

Features

  • Leaky integrate-and-fire simulation of the MaleCNS Drosophila connectome (~166k neurons), plus an 850-neuron MiniFly circuit for demos and tests.
  • Software retina with per-cell optic flow and affine looming estimates driving T4/T5 and LPLC2/LC4.
  • Motor decoder mapping DNg02, DNp03, and DNp01 activity to thrust, yaw, evasive turns, and escape.
  • Safety governor with clamps, slew limits, geofence, watchdog, and battery checks.
  • Drone backends for simulation, Tello, Crazyflie, MAVLink, and a UDP bridge.
  • Browser simulator (SpyFly) in Three.js with neuron poking, burger foraging, and a swat game.
  • MediaPipe webcam hand tracking that drives real looming stimuli.
  • Telemetry streaming to a TimescaleDB hypertable on Tiger Cloud with time_bucket() analytics.
  • Spatial Web Audio buzz that follows wing thrust and chirps on saccades and escapes.

Tech stack

  • Python
  • NumPy
  • JavaScript
  • Three.js
  • MediaPipe
  • FastAPI
  • TimescaleDB (Tiger Cloud)
  • asyncpg
  • C99
  • ONNX
  • NVIDIA Nemotron