Research can make for strange partnerships. Edgar Lobaton is an electrical and computer engineering researcher at NC State University who specializes in robotics, sensors and computer vision. Tom Marchittoc is a geological sciences researcher at CU Boulder who studies prehistoric oceans, and Ritayan Mitra was a post-doc at NC State and is currently a research affiliate at CU Boulder. Divided by geography and discipline, they are nevertheless partners in an attempt to solve a tricky engineering challenge and advance our understanding of Earth’s oceans.
At the heart of their collaboration are foraminifera, or forams – organisms that have been abundant in marine environments for more than a million centuries. If you haven’t noticed them, that’s probably because they are very small; most are only a fraction of a millimeter wide, meaning you could fit about a hundred of them next to each other in a single inch.
These forams are protists – neither plant nor animal – but they create small shells for themselves. And when they die, they leave those shells behind.
By studying these shells, or their fossils, scientists can learn a great deal about the ocean that those forams lived in. For example, different types of foram species thrive in different kinds of ocean environments, and chemical measurements can tell scientists about everything from the ocean’s chemistry to its temperature when the shell was being formed. This is the sort of work Marchitto’s lab is interested in.
But there’s a problem: all of those little shells are, well, little.
Trained students or staff have to spend a long time staring through a microscope to identify and sort these microscopic shells and fossils. And that’s where Lobaton’s lab comes in.
Lobaton, Marchitto and Mitra are working together to find a way to automate the foram sorting process – saving time, effort and (hopefully) speeding up the process of deciphering these clues to our planet’s history.
The team recently received a grant from the National Science Foundation to pursue the work, and The Abstract, NC State’s research blog, had some questions for the researchers about their project.
The Abstract: How did you three end up partnering on this, Ritayan?
Ritayan Mitra: Probably, I am to ‘blame’ for this strange partnership! Early last year I was visiting my wife, who was a postdoc at UCLA at that time. Her lab works in paleoceanographic processes and I saw her and others in the lab toil under the microscope for long hours picking forams from ocean mud samples. As an earth scientist with an interest in interdisciplinary problem solving, I got curious.
The process of picking forams from ocean sediments forms the foundation of major oceanographic discoveries, yet it remains extremely laborious and repetitive. As any repetitive process can (and should) be automated, I was interested to find out if anyone had attempted such automation. A literature survey revealed some preliminary studies toward such automation but most of them had looked at classification of several hundreds of species using neural networks. The training of neural networks for such a huge dataset had proved to be extremely difficult and, as a result, the studies had met with limited success. However, a large majority of oceanography studies often require only a small number of foraminiferal species and I realized that if we could automate the identification of a limited number of species, and subsequently their picking from mud samples (which has never been tried before), then that by itself would be a huge step forward in reducing the time and effort behind such studies.
This approach would bypass the bottleneck of using neural network for en masse species identification. Instead, leveraging advanced computer vision and pattern recognition techniques to identify a limited set of morphological features of a few important species would do the trick. If successful, we could then expand to include other species as well. At this point, I knew I had the seed of an interesting project in hand.
I studied the manual foram picking process in detail and came up with a rough design of the various components that would make up such an automaton. However, my robotics knowledge went only so far and I knew I had to get in touch with an expert to discuss viability of such a project. Most importantly, I was not sure if micromanipulation at this scale was even possible. I was in North Carolina State University at that time and I got in touch with Edgar and pitched him my idea.
He was enthusiastic and he discussed some projects he was already involved in that used robotic manipulation at microscales. The projects he discussed blew my mind and I knew I had gotten in touch with just the right person. We decided to work on a proposal together. Soon after, I got in touch with several labs in the earth science community who I thought would be interested on the application side of such an automation. When I moved to Boulder, Colorado, as a research affiliate, I discussed this idea further with Tom, who runs a thriving paleooceanography lab. He showed a keen interest in what we were developing. With Tom’s involvement we got the disciplinary expertise and context that our proposal needed.
The Abstract: Why are these forams interesting? What can we learn from them?
Tom Marchitto: We humans are in the midst of an unintentional, global-scale climate ‘experiment’ due to our emission of greenhouse gases. To predict the outcomes of that experiment we need a better understanding of how Earth’s climate behaves when its energy balance is altered. Fortunately there are myriad examples in the geologic past of times when the climate responded to some perturbation, be it greenhouse gases, changes in Earth’s orbit, solar luminosity or massive volcanism. Our records of these past events usually come from cores driven into ice sheets, or sediments at the bottoms of lakes and oceans. Since the oceans cover about 70 percent of Earth’s surface and play such an enormous role in its climate, the ocean record is a very important piece of the puzzle. We use forams because they are ubiquitous and because the chemistry of their shells records the physical and chemical characteristics of the waters that they grew in. These tiny critters bear witness to such past properties as temperature, salinity, acidity and nutrient concentrations. In turn we use those properties to reconstruct ocean circulation and heat transport during past climate events.
The Abstract: How and where do you collect samples of forams? And how many forams are in a typical sample?
Marchitto: Ocean sediment cores are taken all around the world using oceanographic research vessels. Collecting very short cores is pretty low-tech, basically involving a PVC pipe with a weight at the top; while very long cores require seafloor drilling. Nearly every core will contain forams. A typical 5 cubic centimeter sample from one of those cores might contain tens of thousands of shells.
The Abstract: How long does it take to process a sample of forams using conventional methods?
Marchitto: This depends on the objective. If we need, say, 30 individuals of an abundant species, picking them might take tens of minutes. If the species is rare, as is often the case for benthic (bottom-dwelling) forams, it might take hours to find enough.
The Abstract: What’s your overall goal for this particular project?
Edgar Lobaton: Our overall goal is to develop a portable visual system capable of identifying different species of forams. If we are successful with this process, we plan to use this system as a building block for a robotic platform that will be able to identify, pick and sort forams automatically. The visual system will incorporate controlled directional lighting in order to capture appearance as well as 3-D shape information that will be used to perform the desired species identification.
The Abstract: Does this project present particularly interesting engineering challenges?
Lobaton: One of the key challenges for this problem is the design of features – which are essentially properties extracted from the images and shape models – that are useful for identifying forams. These features will be obtained based on the knowledge of experts (such as our collaborators), but we will also learn some features automatically from the data. That is, we will have to search through the datasets of images that we will be creating and find the piece of information that makes it possible to discriminate between the different species of forams. Another challenge is the integration of the information coming from images taken using various directional lighting sources. We are aiming to fuse all that information by building a 3-D representation of the forams.
The Abstract: Do you have any ideas for how you might get technology to differentiate between the different types of forams? What might this process look like?
Mitra: The overarching goal of this project is to a) automate identification of different types of forams and b) engage a robotic manipulation to pick the relevant species. At present, we are working on the identification part which involves pattern recognition. Each species has a unique set of structural feature and under proper lighting and orientation we can identify these species under the microscope (magnification 20X to 40X). The idea is to use a camera fixed to a microscope to take pictures of a sample containing several species and then use pattern recognition techniques to identify the target species by recognizing their uniquely identifiable structural features. Initially, we will only look at the most relevant (in terms of oceanographic abundance and scholarly interest) 5-6 species.
Lobaton: From an algorithmic perspective, the system will require human users to first train the system. Experts will have to provide samples of labelled species. The system will then make use of features specified by experts as well as automatically learn how to differentiate between the different species. Once enough samples have been observed, the system will be able to tell us what species it belongs to and measure how confident it is that it is that species.
The Abstract: What’s your goal for how long it would take to sort forams using an automated approach?
Lobaton: Based on our discussions there is a lot of variability on the recognitions and picking time for these species ranging from a few seconds to possibly even an hour. Hence, we do not feel comfortable giving an answer to this question at this point. This is something that we will be able to quantify more accurately later on as well. One of the advantages of the robotic system (even if it takes just as long as the humans for picking) is that this system would be able to continuously and consistently perform the task without stopping.
The Abstract: Could the work you do here be applied to other challenges or problems?
Lobaton: From the engineering side, the techniques that we will develop will include the robust segmentation and classification. These approaches can be used for a number of applications including medical imaging (e.g., identification of cancerous tissues), and autonomous driving (e.g., identification of pedestrians and obstacles on the road).
The Abstract: What’s your timeframe for this project?
Lobaton: Our current award officially starts on Aug. 1, 2016 and lasts until July 31, 2018. During that time, we are planning to develop and validate our autonomous visual recognition system. After that, we are hoping to pursue further funding for developing the autonomous robotic system for picking and sorting of forams.