I am trying to analyze vehicular mobility models, where I am trying to learn how a particular vehicle moves and then detect similar patterns from the testing data.

Here's what I have done for now: I have mobility data for 9 users, collected over 5 minutes for each of them at a frequency of 2 seconds. I have a userID value against each point, and the features that I used are speed and distance travelled 2 points as features.

For training, I combined details of all 9 users into a single dataframe.

When I use a similar structure for testing data (5 minute data without the userIDs for each user combined into a single dataframe), I get pretty weird results:

The accuracies I got are:

Random Forest - 98.00,

Decision Tree - 98.00,

Naive Bayes - 39.66,

KNN - 60.74,

SVM - 34.05,

Linear SVC - 30.85,

Logistic Regression - 30.69,

Perceptron - 19.95,

Stochastic GD - 19.95

I do not much about the mathematics behind most of the ML algorithms, so I am not able to figure out if there is any mistake in the dataset or pre-processing the data.

Coming to my questions:

  1. Is it better to train for each vehicle separately, and then also test for each vehicle separately (without combining them into a single dataframe)?
  2. What might be the reason for such huge differences in the accuracies in the different approaches as given in the table.
  3. I think the important information in the dataset for this particular problem is the relationship between the different points (how much distance is covered every 2 seconds or how the speed varies every 2 seconds), is there a way the testing gives me a result for a whole cluster of values; and not for every testing value.

If I understand you correctly, you are trying to determine which data points in your testing set correspond to which user ids ie you are attempting to categorize each of your testing set data points as belonging to one of your 9 users. My answers to your question are as follows.

  1. If this is the case, it seems that it would be wisest to continue as you are, training on all your data and testing on all your data. If you were to train on each vehicle individually, you would convert your problem into a yes/no classification problem. This could potentially lead you to incorrectly classifying large portions of your testing set (and thus getting a low accuracy) if two of your users have similar driving patterns.

  2. I am also not very familiar with the mathematics behind some of these algorithms, but my gut instinct would be that Random Forest and Decision Tree have much higher accuracy ratings since they are more typically used for this type of classification problem. Some things you might examine before weeding out some of your less well performing models might be the number of branches you have used in Random Forest and Decision Tree, just to make sure you aren't over fitting.

  3. I am having a little trouble understanding what you are asking in your third question, could you please rephrase it? If you are talking about the difference between the time series points on the features you already have, you could simply subtract them user id wise from each other and add that calculation as a feature. If you are talking about data points that are similar to each other in various ways, then I would suggest running some clustering algorithms against your data set as a whole and against the individual user id data. This will allow you to see how the data set is distributed, as well as how each user influenced that distribution. This data understanding will also help you analyze and interpret the later model results you get.

I'm new to Stack Exchange, but I think you might fare better if you broke this post up into a couple different questions. You also might consider posting some sample code or examples from the data set you are using.

  • $\begingroup$ For testing, I am planning to run the testing algorithm for each user separately. In my 3rd point: the individual points are instantaneous values of speed and distance covered from the previous point: so when the testing is running, I want it to consider all 150 points together and give me a single output on which user the data might belong to. So for that, is there any particular algorithm that gives a single output for all 150 points, or do I have to check every point and decide on the basis of the user that the majority of the 150 testing points correspond to? $\endgroup$ – user3656142 Jul 30 '19 at 19:27
  • $\begingroup$ Your algorithm will examine each point in the test set individually and return its prediction as to which user corresponds to that data point. Your accuracy will be derived from how many of those were predicted correctly. $\endgroup$ – Abram Moats Jul 30 '19 at 19:35

The good performance of Random Forest and even Decision Tree clearly indicates that there are some variables that help solving this problem, whereas the majority of variables is useless and "swamping" this for the other methods.

Study the variables used by the decision tree. There is a good chance that these are bad (e.g., identifiers). For the variables that don't play a major role there, extract better features.

While people like to pretend that AI is a universal magic weapon that can solve any problem if you just throw the magic AI sauce on it, that doesn't just work. Features and getting the right data is important, and that is why so many AI projects fail to deliver any ROI.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.