Spotify is the home of audio.

It's known for its personalized playlists, made with its recommendation algorithm.

Think about users as this raw material, and then, on top of the data layer, we're able to build shared models.

But relying so much on artificial intelligence has also drawn criticism from some industry experts worried about algorithmic bias.

Here's how Spotify uses AI to personalize users' experiences on the platform.

This is the tech behind Spotify.

In the early 2000s, many people found music recommendations through top charts and early streaming platforms like Pandora and Last.fm.

With the Last.fm app from the App Store, you can listen to great bands...

So, when Spotify entered the scene in 2008...

It's not so much that they were the first people to start using analytics to recommend music,

but it was the way in which they combined various computational techniques in order to make their recommendations feel more lifelike.

Thomas Hodgson studies algorithms and artificial intelligence, with a focus on how new technology from music streaming companies impact artists.

Fans who listen to discover weekly and daily mix.

The way that they talk about them is in very human-like terms.

Discover weekly, you magnificent ****; you've done it again.

In 2014, Spotify acquired music analytics firm, the Echo Nest,

which blended machine learning and natural language processing to build a database of songs and artists.

Spotify says this technology marked an important step in the evolution of its recommendation system.

So, how does that system work?

It starts with a process called "collaborative filtering".

Collaborative filtering looks at the pattern across all of this data and tries to understand: when do tracks happen to be playlisted together very often?

You can think of it as building a map of music and podcast.

That map looks something like this.

Each point represents a different track in Spotify's catalog,

and the location of each point is determined by collaborative filtering.

Which means that these tracks go together according to the way users have playlisted them and listened to them.

So, if these two songs are frequently playlisted together, they will be close to each other in this map.

Whereas if these songs are never playlisted together, they will be farther apart in the map.

But recommendations based purely on collaborative filtering aren't perfect.

For example, during the holidays, Mariah Carey's "All I Want for Christmas Is You" might get playlisted more frequently with "Silent Night",

even though this sounds like a pop song, and this sounds like a Christmas carol.

If Spotify only generated recommendations based on proximity,

then users who like Mariah Carey might get recommended "Silent Night" when they aren't interested in Christmas carols.

To prevent this, Spotify adds another layer of analysis called "content-based filtering".

This algorithm gathers metadata, like the release date and label, and executes a raw audio analysis.

It uses metrics like danceability and loudness to describe the sonic characteristics of the track.

These are the results for "Uptown Funk", which sounds like this...

and has a danceability score of 0.856 on a scale of 0 to 1.

The algorithm also dissects each track's temporal structure.

Here's a visual representation of that for "Anti-Hero" by Taylor Swift.

These are the beats, the bars, and the sections.

Content-based filtering also takes into account a track's cultural context,

which means studying the lyrics and analyzing the adjectives used to describe the track in articles and blogs.

These filtering techniques are not unique to Spotify,

but industry experts say what sets the platform apart is the amount of user data it has and the products it creates from it.

Spotify says its content-based filtering technology has evolved over the years, and now includes more advanced proprietary-facing features.

But Hodgson says the danger with algorithms is that they could reinforce existing biases.

This could mean that a particular catalog of music has more male artists than female artists.

One of the dangers with machine learning is that,

as listeners start to engage with that catalog, those biases become magnified, and that this creates what's called a, kind of, "feedback loop".

Spotify says its research teams evaluate and mitigate against potential algorithmic inequities and harms, and strive for transparency about its impact.

Another criticism is that the algorithm isn't optimized for new artists because there's no user data.

This is known as the "cold-start problem".

Sultan says this is where human editors play a significant role in delivering recommendations.

They're possibly some of the best people in the world that's trying to understand new releases and culture and what's relevant.

But Hodgson says the bigger concern is that certain metrics used in the platform's audio analysis might be culturally biased

In other parts of the world, they have musical systems and musical cultures that are entirely different.

Like this North Indian classical track, for example.

Spotify's algorithm labels its key signature as E minor, which Hodgson says is inappropriate for this musical tradition.

However, it's still the case that

the music that is emerging from South Asia is being understood algorithmically under, you know, the Western equal temperament scale.

Spotify says the audio analysis is one small part of the overall system, which takes into account many factors before making a recommendation.

Some industry experts also point to issues with how the system understands metadata for classical music.

For example, the metadata for a Tchaikovsky track can include not just the name of the work and the artist, but also the movement, opus number, and conductor.

Spotify's algorithm isn't optimized for that.

Apple Music, which has emerged in recent years as a competitor to Spotify,

released a new app in March that the company says is designed to solve this problem.

Spotify says it doesn't comment on a competitor's marketing campaigns.

In February, the streaming service joined the recent buzz around generative AI.

I'm X, and from this moment on, I'm gonna be your own personal AI DJ on Spotify.

The DJ gives the algorithm a human voice and offers listeners additional context around a recommendation.

Up next, I know you've been on a summer song kick lately.

Sultan says the company is also exploring reinforcement learning,

a technique that would allow the recommendation system to learn automatically based on feedback.

It will help with the diversity of their recommendation, it will help with the longer-term retention.

And we're trying to push the state-of-the-art in each of those,

introducing new technologies, new capabilities, and bringing new experiences.