The Uropatagium
| Virtual Tour
| Robot Images and Movies
| Art & Science |
| Scientific Visualization
| Interactive 3D
| Scientific Animations |
Photos |
For optimal appearance of static images, turn off automatic image resizing. Internet Explorer: Tools, Internet Options, Advanced Tab, Multimedia: Uncheck `Enable automatic image resizing.' Firefox: 1) Enter `about:config' in address bar; 2) Enter `browser.enable_automatic_image_resizing' in Filter field; 3) You will get a single result in the bottom pane. Double click it to change value to false.
Lab Virtual Tour
The MacIver Lab Virtual Tour is a good way to quickly peruse some of our research themes.Robot Images and Movies
Robotic fish
Video of the Northwestern robotic ribbon fin (MPG, 640x480, 4.4 MB). Note penny in image for scale. This robot was designed for studies of the neural control of the ribbon fin of weakly electric fish, which undulates in a very similar way to what you can see in the video. We are also interested in highly maneuverable underwater vehicles based on this type of propulsion. On the real fish, the fin consists of a long (most of the length of the body), ribbon-like fin along the bottom edge of the body, with around 100 fin rays. The fish passes traveling waves along the fin to move through the dark and muddy rivers of the Amazon Basin, typically hunting for prey at night using its unique electric sense. The fish often switches direction, and swims in reverse, by changing the direction of this traveling wave along the fin. It can also hover and move directly up with the fin. We are currently investigating the fluid dynamics of this structure to better understand the relationship between sensory constraints and locomotion in animals. Here are stills and video:- Video of Northwestern robotic ribbon fin showing vortices shed on alternate sides of the fin, slow motion: QuickTime (825 kB)
- Video of Northwestern robotic ribbon showing test bed, slow motion: QuickTime (503 kB)
- Renderings of Northwestern robotic ribbon fin (html)
Built in 2004-5, from work with Ed Colgate, Michael Peshkin, and Kevin Lynch. The mechanism has eight rays and is about 23 cm long.
Videos of the earlier (and much larger) Caltech robotic ribbon fin. Built in 2002, from work with Joel Burdick. The mechanism has 13 rays and is 53 cm long.
The ElectroSenster: A robotic perception system based on fish-like sensing with electric fields
Image of the ElectroSenster. Movie of an electrolocation sequence (6 MB QuickTime).
Article in the New Scientist on
the ElectroSenster
Our model system, the weakly electric fish, hunts in the dark and muddy rivers of the Amazon where vision is useless. It emits a weak electric field (just a thousandth of what a flashlight battery has) and uses this as a kind of radar system, similar to how bats use sound. We have been investigating how they are able to identify and localize the things they are interested in with this unique sensory system. One approach we are using is to build robots that have similar capabilities. The movie above shows both the robot we built and an animation that displays the robot's "belief" about the position of the object it is trying to localize, a metal sphere. In the "particle filter" approach we use, the robot's belief about the location of the object is represented by particles (blue dots). Since the robot has no notion as to where the object is initially, the blue dots are evenly distributed over the space it can search at the start of a sequence. As the robot moves (CONTROL), it is moving to improve its belief as quickly as possible by taking a measurement at each new location and using a Bayesian approach to fusing the data with prior measurements (SENSOR FUSION). Details can be found here. Sensing range is roughly the distance between the electric field source, here just a few centimeters; to sense further away the field emitters would need to be further apart.
Using this robot we have been testing previously developed theory regarding how electric fish localize objects. For simple objects (spheres) we are obtaining excellent agreement. Here is an electric image of a small metal object obtained with the ElectroSenster side-by-side with theoretical prediction. We are also using it to develop algorithms for sensor-based motion control with uncertainty. From work with Kevin Lynch.
Art and Science
BODY ELECTRIC, 2003.
Malcolm MacIver and Simon Penny
Sponsored and funded by Caltech and the National Science Foundation. Exhibited at the
Williamson Art Gallery in the Art Center College
of Design in Pasadena CA, April 15-June 29 2003.
Video documentary on Body Electric (7 minutes long): Small QuickTime (12 MB, 240x180 pixels, 15 frames/s); Large Quicktime (37 MB, 640x480 pixels, 15 frames/s).
Web site on the exhibit: at the
Art Center College of Design.
Images: Construction 1; Construction 2; Art gallery opening; Early version of the real-time avatar.
Reviews: 1) Artweek (pdf, 384 kB); 2) Los Angeles Weekly (pdf, 44 kB); 3) Los Angeles Times preview (pdf, 37 kB); 4) Los Angeles Weekly Pick of the Week (pdf, 65 kB).
Computers, projectors, speakers, infrared lights, cameras, custom software
Exhibit statement:
Until recently, the prevailing orthodoxy concerning perception was rooted in an enlightenment-objectivist model which proposed that sensory systems take in the world passively, with information funneling into our brain to form rich internal representations of the world for guiding our behavior. David Marr began his influential book on vision by saying "vision is the process of discovering from images what is present in the world, and where it is." Andrew Blake called this approach "a prescription for the seeing couch potato." The passive view of perception is embedded in scientific culture and as a result, structures the hardware and software of our technologies, with its characteristic lack of sensory intelligence and dynamic engagement with the world. A more recent view of perception is that behavior is tightly coupled to the way we sense the world, that perception is a temporally extended process of active, embodied engagement with the world. Rather than develop rich internal models, animals, including people, maintain close sensorimotor contact with the world, allowing the world to be its own best model. Diverse embodied human cultural practices, including various art forms, are also incompatible with the objectivist view. In sensory neuroscience, the role of behavior in perception is especially clear for certain sensory champions, studied to address questions about how all animal sensing occurs. One of these is the weakly electric fish. These fish hunt at night, in muddy rivers where vision is useless. They utilize a weak self-generated field to sense their surroundings. Thousands of sensors covering the body surface become stimulated by nearby objects, and their nervous system decodes this to analyze the type and position of the object. Body Electric combines active sensing in interactive cultural experience with the study of such systems in neuroscience. In Body Electric, we simulate the electrosensory system of these fish through a custom multimodal real-time sensing and display system. The 'spectactor' engages in a complex sensorimotoric exploration of a novel environment.
Scientific Visualization
(Vol 202, Issue 10, May 1999) for a publication with Mark Nelson.
Interactive 3D Visualizations
Interactive 3D visualizations of the prey sensing (blue) and immediate movement volumes (green) of a weakly electric fish. These interactive volumes illustrate the relationship between where the fish can sense small prey, and where it can move to over three different time intervals: at 117 ms, around the neuromotor delay time for sensory signals to move from the surface of the body to the motor center and out to the muscles for movement; 432 ms, the time when the movement and sensory volumes maximally overlap; and 700 ms. Active sensing systems that work at a distance generate an energy field that drops in signal strength as the fourth power of the distance. We hypothesize that these creatures will therefore have conservative sensing volumes which allow them to sense far enough to come to a stop before colliding with the smallest objects of behavioral relevance such as prey. Creatures, such as ourselves, which utilize ambient energy for perception are not as strongly constrained to have a conservative sensing volume. From work done with Joel Burdick and Mark Nelson.
Scientific Animations
Numerical simulation of freely swimming fish (640x360, 29 MB QuickTime). Other portions of the lab's work concerns modeling and measuring what information an animal obtains from its environment as it does essential tasks such as detecting and striking at prey. In this work, we are working toward an accurate model of the mechanical abilities of the animal. By obtaining this, we expect to gain insight into what kinds of information needs to be extracted from the many thousands of channels of sensory data the animal has. Since our model system is a fish, with
Neelesh Patankar we
are working on novel approaches for simulating the free swimming behavior
of fish in a physically accurate manner.
We use full 3D Navier-Stokes, an immersed boundary method,
and a novel approach to implementing conservation of momentum to accurately
predict forward movement as a result of prescribed movement of the fish's
body. This video, produced with Bruce Gooch, shows
a simulation of the free swimming of an eel and a black ghost knifefish.
A short blurb on the science behind
this effort.
Measured fish prey capture strike versus mechanical optimality. This animation shows a 3D reconstruction of the black ghost weakly electric fish (Apteronotus albifrons, shown in green) detecting and attacking a water flea (Daphnia magna, small sphere) overlaid with an ellipsoid that is following a mechanically optimal trajectory (one that minimizes the mechanical effort to reach the prey). For details on the optimal control approach used for generating the optimal trajectory, check out our publication from IEEE Journal of Oceanic Engineering. From work done with Joel Burdick. Created with MATLAB and Mark Milam's nonlinear trajectory generator (NTG). Formats: QuickTime (720x480, 3 MB).
Three-dimensional reconstruction of electric fish attacking prey in the dark. This is a 3D reconstruction of the black ghost weakly electric fish (Apteronotus albifrons) detecting and attacking a water flea (Daphnia magna). The original behavior is recorded without visible light using two infrared cameras; using a custom software system, the camera images are utilized for obtaining a full 3D reconstruction of the behavior at high precision (±0.5 mm). The technique is described in one of our publications. The movie shows the position of the fish (absent of fins, which we do not track as they do not contain sensory receptors), the position of the prey (filled dot) and the line representing the shortest distance from the prey to the fish sensory surface (and therefore where the sensory signal on the surface of the fish will be maximal). The prey and time indicator changes color from red to green at the estimated time of detection, which we base on the onset of the fish's deceleration to reverse its movement for the backwards strike. From work done with Mark Nelson. Created with MATLAB. Formats: QuickTime (666x500, 776 kB), Windows Media Player (666x500, 472 kB).
Electrosensory signals that arise during prey attack. Weakly electric fish emit a weak oscillating electric field which gives them an ability to perceive objects in the dark, analogous to how bats use sound to perceive in the dark. This video shows a reconstruction of the active electrosensory input to the fish during the same prey capture sequence reconstructed above, using an empirically tested model of electrosensory image formation. The logarithmic color bar shows the change in the voltage across the skin due to the prey. Note that at the time of dectection, when the time indicator goes from red to green, the maximum signal is about 1 µV. This represents about an 0.1% change of voltage across the skin, since with no prey present the fish has approximately 1 mV across the skin due to its self-generated electric field. Not only is the signal extremely weak at the time of detection, it is also very broad and diffuse, while later in the sequence it becomes very focused. There is therefore a great deal of variation both in the strength and in the spatial properties of the signal over time. This may be the reason why the sensory information is split three ways at the brain, and gets sent to three different processing areas, each with a particular spatial and temporal filtering capability. From work done with Mark Nelson. Created with MATLAB. Formats: QuickTime (666x500, 782 kB), Windows Media Player (666x500, 456 kB).
Mechanosensory signals that arise during prey attack. Weakly electric fish, like all fish, possess a highly sensitive lateral line system. This sensory system is thought to have eventually led to the auditory system in mammals. When you buy fish at the market, the line you see along each side of the body is this sensory system (to be exact, it is the canal part of the mechanosensory lateral line; there are also sensors scattered over the body surface which are part of this system as well, but we do not consider those here). It is composed of a series of small pores, interconnected by a gel-filled canal. Within these canals there are receptors that are incredibly sensitive to pressure differences between the pores. This system allows fish to detect predators approaching, or the walls of a tank, or prey. Some fish, such as the blind cave fish, have a highly developed mechanosensory systems and may even be capable of imaging objects by detection of an array of pressure changes that they can induce by swimming by an object. In collaboration with
Sheryl Coombs, we
undertook a modeling study
to determine how significant these signals may be in guiding
the behavior of black ghost knifefish. Here is a reconstruction of those signals for
the trial considered above, with a dot between each pore (where the sensory receptor
is located) indicating the change in pressure due to the prey's movements.
The color bar shows the pressure in dB relative to 1 mm/s².
From work done with Mark Nelson.
Created with MATLAB.
Formats:
QuickTime (666x500, 782 kB),
Windows Media Player (666x500, 503 kB).
Multimodal fish ballet, with individual sensor rendering. An animation showing side-by-side reconstructions of the active electrosensory input, the passive electrosensory input, and the mechanosensory input over the course of a prey capture. Each sensor and its estimated activation level is represented for each of these three sensory modalities. The bioelectric field of the prey (Daphnia magna) used for estimating the activation levels of the passive electrosensory system (ampullary receptors) is from measurements from other researchers (see the modeling study). Three populations of sensors are shown: From left to right in the animation these are
- 13,857 sensors that make up the active electrosensory system (tuberous electroreceptors), at measured sensor densities. These sensors detect fluctuations in the animal's own self-generated electric field, caused by things like prey entering the field.
- 720 sensors that make up the passive electrosensory system (ampullary receptors) at measured sensor densities. These sensors detect extrinsic electrical fields, such as the bioelectric fields that exist around any living organism in water.
- 204 sensors that make up the canal subsystem of the mechanosensory lateral line (canal neuromasts), based on measurements in this publication. These sensors detect minute variations in the water flow as caused by prey swimming nearby.
For a publication with Mark Nelson and Sheryl Coombs. Created with MATLAB. Formats: QuickTime (1020x744, 1.5 MB), Windows Media Player (1020x744, 2 MB).
Computational fluid dynamics simulations of the weakly electric fish ribbon fin. An animation showing some early results of an immersed boundary method approach to simulating the fluid structures around the ribbon fin. From work with Neelesh Patankar. Generated with custom FORTRAN code, visualization in MATLAB. Format: QuickTime (1024x668, 24.6 MB).
Simulation of electrosensory multisensor data fusion using a Bayesian filter. This animation shows a field of possible target locations (pink dots) around an ellipsoidal approximation to an electrosensory organism after an initial sensor reading. Subsequently, the number of locations that the object could be shrinks as additional sensors are brought in (indicated by the yellow star) and fused to the prior readings in a Bayesian framework. From work with Kevin Lynch. Generated with MATLAB. Format: QuickTime (1004x648, 11 MB).
Photographs
From real to digital electric fish with a 3D tactile scanner. For a publication in 2000, we developed a 3D model of a fish based on a 3D scan of a urethane cast we made of Apteronotus albifrons. The photograph, taken by Ben Grosser, shows the scanner, the Microscribe, in front of a rendering of a computer model of the fish made from data points collected through the scanner. From work with Mark Nelson.
