In order to protect animals that live in remote environments, researchers must be able to find them to understand the movements of their populations over time. As long-term passive acoustic monitoring capabilities have grown more technologically sophisticated, automatic animal species identification tools built on large datasets from these recorded soundscapes have become an increasingly vital tool for conservation and ecological research. While models such as Google Perch have emerged that can classify thousands of bird vocalizations, similar models that can classify vocalizations from several whale species at once have proven more challenging to develop.

The acoustic range of whale species is incredibly broad, ranging from as low as 10 Hz for blue whales to above 120kHz for odontocetes (toothed whales), and recordings also vary dramatically by location and with time, which can make model development difficult. Additionally, researchers often don’t know what types of vocalizations are made by some especially elusive whale species, which complicates identifying those animals in the soundscapes. This is illustrated in the decades-old mystery surrounding a sound, called a “Biotwang”, that was first recorded almost a decade ago in the depths of the Mariana Trench. The sound has a “metallic” or “chime-like” quality, quite unlike the tonal moans more typical of whale vocalizations. In a recent paper, our collaborators at the U.S. National Oceanic and Atmospheric Administration (NOAA) determined that the Biotwang sound is uniquely produced by the elusive Bryde’s whales (pronounced “broodus”).

Today we are delighted to share Google’s latest whale bioacoustics model, which can identify eight distinct species, as well as multiple calls for two of those species. Following on our collaborator’s discovery connecting Biotwangs to the Bryde’s whale and in the same paper, we expanded the model to include Biotwangs and used it to label more than 200,000 hours of underwater recordings. Here we describe the model and discuss some of the new insights into the ecology of whale species it is helping researchers to unlock. The model is now available for download via Kaggle Models.

Project background

Google Research’s journey with whale vocalization classification started in 2018 when we developed a novel classification model for detecting humpback whales in partnership with the Pacific Islands Fisheries Science Center (PIFSC) of NOAA. The model was used to identify humpback calls from over 187,000 hours of audio collected by NOAA, confirming spatio-temporal patterns of humpback songs and uncovering a new location at Kingman Reef where humpback songs had not been previously observed. We made further “splashes” with this model in collaboration with Google Creative Lab when we released Pattern Radio, an interactive visualization of a full year of underwater audio collected near Hawaii, labeled by the model, and peppered with additional expert insights on sections of the data. We released our humpback model publicly following Google’s AI Principles to understand and minimize the potential for misuse of the model.

These efforts led to a partnership with the Department of Fisheries and Oceans, Canada (DFO), especially with their Marine Mammal Response Program, whose operations in the Salish Sea include stewardship of the critically endangered Southern Resident Killer Whale population. Together, we published an orca (killer whale) detection model, which DFO also deployed in their hydrophone monitoring network, enabling real-time alerts.

A new whale bioacoustics model

We developed our new multi-species whale model to score and classify underwater audio for eight distinct species. Two of the species are further broken down by vocalization type, yielding a total of twelve classes. The model is multi-label, so scores are independent and not restricted to the top class or classes.

The following is the list of species for which the model can provide classification scores:

• Humpback whales (Megaptera novaeangliae)
• Killer whales (Orcinius orca)
• Blue whales (Balaenoptera musculus)
• Fin whales (Balaenoptera physalus)
• Minke whales (Balaenoptera acutorostrata)
• Bryde’s whales (Balaenoptera edeni)
• North Atlantic Right Whales (Eubalaena glacialis)
• North Pacific Right Whales (Eubalaena japonica)

Audio examples of each of the species are in this repository.

[…]

New insights

Bryde’s whales are baleen whales in the same genus as blue and fin whales. While sightings of these animals have been reported around the world, relatively little is known about their movements or population structure. Recordings collected in the Mariana Trench in 2014 and 2015 captured a unique vocalization called the “Biotwang”. This complex, 5-part call lasts approximately 3.5 seconds, starting with a low-frequency downsweeping moan from approximately 44 Hz to 30 Hz and followed by a “metallic” sound that goes up to 8000 Hz. Because the vocalizations were not associated with sightings of the animals, the researchers originally attributed them to an undetermined baleen species.Photo of a Bryde’s whale spotted during a NOAA cetacean survey in the Mariana Archipelago in 2010 (Credit: NOAA Fisheries/Adam Ü, NMFS MMPA-ESA Permit #14097).

Audio and spectrogram of the Biotwang call of a Bryde’s whale. The highest intensity downsweep is at low frequencies, while the additional lines directly above are harmonics that aren’t always present. The part that looks like quotation marks is the higher frequency “metallic” sound.

[…]