If you ever wander around wanting to know the names of plants and animals around you, Seek, a newly rebuilt app from the iNaturalist team at the California Academy of Sciences, now offers instant identifications through the camera view on a phone.
Point your phone’s camera at something alive and Seek will name it at the top of the screen. The app offers the most precise identification it can, backing up as far as it needs to go on the taxonomic ladder to make (pretty) sure it’s correct. Point it at a clover from waist height, for example, and it tends toward an ID of “group: dicot.” As you get closer, a series of dots at the top of the screen fills in the tree toward the “species” level. When it clearly identifies a species, the camera button turns green, and you can take pictures to “collect” and earn various digital rewards.
Seek fuzzes the location of observations, making it more secure for use by kids. An update to the app released in June lets you connect Seek and iNaturalist accounts, so observations can be shared across the two platforms.
The app also does all of this without needing a connection to the internet. The processing is done internally, so you can use it anywhere. That ability comes at a small cost to the storage size of the app — the computer vision takes up 30-40 MB and it’s a 150-MB download on iPhone, 70 MB on Android.
“[Computer vision] is innovative technology,” Seek’s lead developer Amanda Bullington said. “It works well on the server, and it’s great if you’re on the internet, but that’s not possible in nature.”
The app’s designers built the live taxonomic tree as a sort of teaching tool. Earlier versions of Seek, like iNaturalist, worked with already-taken photos from the camera to return an identification. The person, though, wouldn’t get any clues about how the machine had come up with its result. The new augmented reality version reveals more about what the computer sees, and allows the user to move the camera around to potentially help it see more.
“The black box gets frustrating when it won’t tell you what you want,” said Abhas Misraraj, Seek’s user experience designer. “Making it clear what it is, what it’s thinking, the seven segments it’s processing, can help people be more understanding.”
The app’s most common frustrations have roots more in nature than technology. There are things that are hard to identify from far away (birds), or to identify conclusively from even a good close-up photo (spiders, beetles, flies, banana slugs). You might really want to know what a particular plant is and find that it’s simply not possible to figure it out by just looking at it. That’s life, iNaturalist community and support coordinator Tony Iwane said. It’s not all easy.
“When I first started going out with naturalists, you expect that they can tell you every species,” Iwane said. “Well a lot of times that’s just not possible. It’s a good thing for a naturalist to learn it’s fine to know what a genus or family is. And it’s good not to overreach. To teach people the different levels of certainty in science.”
Assuming it’s identifiable, and you can get the thing you want identified into the camera frame and clear, Seek tends to be accurate. The identifications come from an artificial intelligence that has learned how to recognize wildlife by studying photos in the iNaturalist database, which now holds 21 million verified observations from around the world, 16 million of which have been identified to the species level. Seek’s accuracy increases with additional observations to iNaturalist, so the more commonly observed a species, the easier time the computer will have. The converse is true, too: if you stumble across something that people haven’t recorded often in iNaturalist, Seek often won’t know what to make of it. iNaturalist developer Alex Shepard told me the artificial intelligence trains only on species that have at least 100 observations in iNaturalist, at least 50 of which have had more than one person agree on the identification.
Testing it on a few hikes around the Bay Area, the app excelled at California native plants, which are particularly well-described and photographed in iNaturalist. It also nailed a few cultivated plants — often harder to identify since the location can’t be used as a clue. (Thanks to Seek I now know that a vibrant population of florist’s cineraria, Pericallis x hybrida, a British creation from two Canary Islands species, blooms in the open spaces in San Francisco’s McLaren Park.)
The app made it correctly to the genus level but not beyond with a handful of photos that iNaturalist’s human identifiers identified pretty quickly to the species — butterflies, nudibranchs, anemones, and shore crabs. It struggled to suggest anything at all for the colonial sponge and tunicate life of the San Francisco Bay; no surprise there, as so does just about everyone else on the planet.
As an example of seeing what the computer sees, though, it went with a conservative genus identification of Vanessa for an American lady butterfly from a photo of the forewings, but when I changed the photo to the more clearly distinguishing hindwings, it recorded the full species:
The app plays it conservative about some things that can be seemingly overconfidently identified — bumble bees with yellow faces, for example, which in coastal regions of the Bay Area are not all the “yellow-faced bumble bee” Bombus vosnesenskii. (The difference between Bombus vosnesenskii and Bombus caliginosus, according to the insect reference BugGuide: “Females with malar space relatively short and S4 usually black as compared with caliginosus, and males with different antennal proportions. These characters are rarely visible in photos so those taken at coastal sites where both could occur are rarely identifiable.”)
Seek will also offer identifications when you point your phone at photos and even illustrations. It will, as one user pointed out to me, make an honest attempt at identifying stuffed animals.
Exploring with it in the Bay Nature office the app correctly identified an indigo bunting on the cover of The Field Guide to Birds of North America, a rockfish — to the genus Sebastes — painting on a postcard, and — to the appropriate family — most of the painted butterflies on the cover of the spring issue of Bay Nature magazine. I mostly enjoyed this; if you’re curious about what something is, you’re curious about what it is, whether it’s in the wild or on the cover of a book.
“There’s a lot of interesting ways people can misuse the camera,” Bullington said. “But the goal is just getting people to learn about nature.”
