How can I get an objective measure of song similarity?

Question

I was browsing through ML project ideas and found an interesting one (just the problem statement ) which was: detecting if two songs are similar using lyrics. I found it to be an interesting idea but Im not quite sure how I'd go about getting a score of similarity for the songs. For my dataset, I have features of genre, artist and lyrics. What is a potential method of scoring the similarity considering there is no such 'training data' to begin with.

I have come across word embeddings and stuff but their working isnt completely clear to me. Moreover, I think they dont take the song-like features that are available into account: things like type-token ratio, sentiment rating, word density(average number of words per sentence)etc.

Could an approach where first the songs were first clustered based on "high level features" such as type-token ratio, sentiment etc followed by a semantic similarity measure i.e. something like a cosine similarity metric on the word embeddings of songs in the same cluster make sense? How would I validate the usefulness of such an approach?

Answer 1

Two songs are two separated documents which have characteristics that make them similar or non-similar. There are plenty of techniques to determine similarness between documents:

How to compute the similarity between two text documents

Answer 2

For quick proof of concept you could take pretrained embedder i.e. LASER. Here is unofficial pypi package. It works just fine. Though, keep in mind, embedders are meant for rather shorter chunks of texts. It makes little sense to assign single semantic meaning to more than few sentences. Embedder produces numerical vector. Once you've embedded lyrics from two songs, you can calculate distance metric between them i.e. euclidean. It should kinda work out of the box, but don't expect something groundbreaking.

Example

from laserembeddings import Laser
from scipy.spatial.distance import euclidean

laser = Laser()

beatles_lyrics = """
I love you, 'cause you tell me things I want to know
And it's true that it really only goes to show
That I know that I, I, I, I
Should never, never, never be blue
"""

joy_division_lyrics = """
I've been waiting for a guide to come and take me by the hand,
Could these sensations make me feel the pleasures of a normal man?
These sensations barely interest me for another day,
I've got the spirit, lose the feeling, take the shock away.
"""

beatles_lyrics_embedded = laser.embed_sentences([beatles_lyrics], lang='en')[0]

joy_division_lyrics_embedded = laser.embed_sentences(
    [joy_division_lyrics],
    lang='en'
)[0]


similarity = euclidean(beatles_lyrics_embedded, joy_division_lyrics_embedded)

print(similarity)

Here you can find more distance metrics: https://docs.scipy.org/doc/scipy-0.14.0/reference/spatial.distance.html

Answers

I have come across word embeddings and stuff but their working isnt completely clear to me.

Embeddings encode hyper-dimensional text to lower-dimensional numerical space. It's trained in a way, that semantically similar sentences from different languages are closer to each other.

Moreover, I think they dont take the song-like features that are available into account: things like type-token ratio, sentiment rating, word density(average number of words per sentence)etc.

Embedder was pretrained to learn feature representation itself. They're high-level features but completely black box.

That's what deep learning is all about. It's hard for us to manually engineer features in such space. Instead we train deep model in a way it learns representation of those features itself.

If you want to learn sentiment from text, embedders are definitely the way to go. You just need to define sentiment for every lyrics.