I'm a composer and programmer. I'd like to use ML for composing music. There's already research on the general problem of composing by machine in known musical styles. I'm more interested in the specific problem of using a computer to assist myself in composing... to speed up certain repetitive tasks, or to delight me with unexpected ideas.

First, I want to design ML that can make a good/bad categorization on a single short passage. (Not looking to generate music at first.)

It's pretty easier to decompose such a passage into significant features. I can say things like

  • maybe there are particular "intervals" (harmony) present, and others absent
  • maybe notes are spread out evenly from low to high, or maybe not
  • maybe notes are clustered around a certain register rather than spread out
  • maybe the rhythm is particularly uneven
  • maybe a particular note occurs a lot

I can easily write algorithms that produce a list of such features for any given passage, and I can even be pretty sure that I'm including all of the relevant features. For an "art form," music is surprisingly amenable to analysis.

The trick is, first of all, there will be hundreds of observations, and second, that no one of these observations is good or bad in itself. Say we have three factors A, B, and C. A is not bad alone, but perhaps when A and B occur together, that's bad. But then maybe if A, B, and C all occur together, it becomes good again! And then maybe a fourth factor D, changes the context so that A, B, and C no longer matter much.

This seems like an ML problem, but I wonder what the best approach is?

I can construct or compose hundreds of examples of music labeled good and bad, and identify hundreds of "features" in each example. Some of the features are "present/not present" categories, while others are numbers that can be scaled from 0 to 1.

Is there a general and simple learning algorithm that even an ML novice such as myself (but I'm an experienced programmer) can implement?

I have to generate my own training examples and would be limited to around 300 to 1000, as I'm training it on simple passages in my style. This may actually be good problem to have... my job is not that complex compared to general musical intelligence.

I will contrast this task with image recognition. An image has millions of pixels and the features are spread out, occurring in endless permutations. It makes my head hurt to try to figure out what's going on in a deep neural net.

However, in music, there will be at most a few hundred features. The patterns will not occur in endless permutations. In fact, in stripped-down situations my wild guess is that you could make an 80% accurate classifier by looking at 10 features (assuming you have the weights and combinations well-trained). I'm aiming somewhat higher than that (I need more features and a bit more accurate).

To give you some idea of my data set: I'm in the early stages of writing the software to extract the features, but I have a pretty good mental idea of what I'm aiming for (been composing by hand for years and years). To make this more concrete I created a web page showing what the data will look like. This is very simple and just totally made up.


This shows four rows. Each row is an example passage of music that I will compose or generate automatically. Each row is classified "good" or "bad" by my own ear. The rest of the row is a long list of features, each of which is either "yes/no" or a single number scaled from 0.0 to 1.0.

I don't think a Naive Bayes classifier will work for this (if I understand correctly) because features are neither good nor bad in and of themselves... it is only combinations of them (context) that matter. In fact, the very same feature can contribute to "goodness" in one context, and be an immediate strong indication of "bad" in another context.

The data could possibly be structured differently. In practice, this spreadsheet would be "sparse" and maybe features should be grouped in some way, or arranged in a tree.

  • $\begingroup$ To do good/bad categorization you will need labeled data. Do you have access this kind of data? How many instances do you have? How many features do you have? $\endgroup$
    – JahKnows
    May 23, 2018 at 3:54
  • $\begingroup$ Edited to make this clear. I can make hundreds of examples labeled good/bad, maybe thousands by making slight variations, and I can compute the values of maybe 100 to 300 features in each example (depending on what proves to be useful or practical). $\endgroup$ May 23, 2018 at 4:04
  • $\begingroup$ Ideally you want to maximize the information entropy of your features, that means the smaller the number of features, but the most information in those features. For a problem this complex you will need in the tens of thousands of instances. Can you not get more data from publicly available datasets? $\endgroup$
    – JahKnows
    May 23, 2018 at 4:07
  • $\begingroup$ No public data because I'm training it on my own style, and also because I'm working with a simplified representation of music. The problem may be less complex than you anticipate.. there aren't that many patterns that make music good or bad, especially in my simplified representation. But no features are meaningful alone--I need some way to recognize the ways that simultaneous occurrence of features is significant. $\endgroup$ May 23, 2018 at 4:34
  • $\begingroup$ Also even a less audacious first goal, such as identifying the "top ten" features and experimenting with that (rather than aiming for a fully accurate classifier), would be helpful as a way of assisting me with composition. $\endgroup$ May 23, 2018 at 4:37

2 Answers 2


Let us formulate this problem in such a way that it can be understood from a machine learning perspective. You have a set of instances $X$ where each instance $x_i \in \mathbb{R}^m$ where $m$ is the dimensionality of the instance. In other words $m$ is the number of features that describe the instance. Your problem intends to go from a set of features to a class label good or bad. Thus, this is a mapping from $\mathbb{R}^m$ to $y \in \{0, 1\}$.

How to achieve this mapping?

This is when we will use the machine learning algorithms. We will train a model to effectively approximate the function which gives the output label from a set of inputs. It is evident that sparse features (low information entropy) will complicate the mapping function and will thus provide worse results. This is why feature engineering is of upmost importance for machine learning. It is probably the hardest part of the machine learning pipeline, however it is the lead factor in dictating your results.

You can use some feature reduction techniques in order to remove features which are uninformative with respect to the output label. Some techniques that I use frequently are principle component analysis (PCA), linear discriminant analysis (LDA). Alternatively, you can use some projection methods to reduce the dimensionality of the data whilst maintaining separation between the classes. Such techniques are Isomap, MDS, Spectral Embeddings and TSNE. You can check to see which is best suited for your type of data.

How to choose a model?

Firstly, your problem is a supervised classification problem. This already narrows the types of models you can use. Furthermore, model selection is based on some key factors such as: the number of instances you have, the number of features per instance and the number of output nodes. You should also keep in consideration that the separability of the probability distribution between the output classes will impact the performance of the model directly. For example discriminating between cars and oranges is much easier than oranges and clementines.

In your case, you have 1,000 instances and around 13 features. This means that deep learning based techniques are possible but discouraged. You do not have enough data. You can then attempt the following popular classification models

  • Support Vector Classifier
  • Naive Bayes
  • K-Nearest Neighbors
  • Decision Trees
  • Random Forests

To evaluate which model performs the best you will use the accuracy attained with a trained model on a test set. This set should be drawn independently from the training set as to catch overfitting. This is when the model cannot generalize to new data.

In code

Assuming matrix $X$ contains the data where rows are the instances and columns are the features, and matrix $Y$ contains the labels.

First we split our data into a training and testing set

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
print('Score: ', neigh.score(X_test, y_test))

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.ensemble import RandomForestClassifier 
forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))

This should be a starting point. Let us know if you fall into any problems, and let us know what accuracy you are getting we can then look deeper into these models and better suit them to your data source.

  • $\begingroup$ Thanks! It will be a lot of work now, to write the code that generates my examples and extracts the features, so I'll get crackin. $\endgroup$ May 23, 2018 at 7:45
  • $\begingroup$ Good luck! Let us know when questions arise! We'll be happy to give some pointers. $\endgroup$
    – JahKnows
    May 23, 2018 at 7:55
  • $\begingroup$ I had one thought. In some ways it's easier to write an algorithm that compares a test instance with a training instance. It would be an expensive computation until I optimize it, but it would be, in some ways, easier to methodically check similarity and rack up a score, by looping or recursing over tiny similarities and never having to design a fixed feature extraction system. Then I categorize the test instance as the same as the closest training example. Is this kind of the "gestalt" of the K-Nearest Neighbors method, even if inefficient? $\endgroup$ May 23, 2018 at 8:22
  • $\begingroup$ Yes this is very much how K-NN works. K-NN is a computationally expensive method. $\endgroup$
    – JahKnows
    May 23, 2018 at 8:24

I'll offer up a few suggestions to get the creative juices flowing.

First, since you have two tasks, you could consider two models configured as a Generative Adversarial Network where

  • One model generates musical passages
  • One model classifies musical passages

The output from the generative model is then evaluated by the classifier. The output from the classifier can then be used to improve (train) the generative model.

I imagine these would be trained on the same labeled data set. I would use a Recurrent Neural Network to train the classifier and tune the snot out of it (so to speak). I would use a Generative Neural Network for the generative model.

Maybe you were looking for more direction. If so, sorry to disappoint. I figured you may have been looking for better ways to search for help.

Best of luck.

  • $\begingroup$ This answer does not address the question. Why would you use a GAN for classifying passages as good/bad? $\endgroup$
    – JahKnows
    May 23, 2018 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.