2
$\begingroup$

I am new to data science and have been doing research to familiarize myself and try to find a solution to my problem but I have not come across anything that seems to fit. I am looking to learn and use Python for this process.

My situation is as follows: I have multiple error logs for the cars I am testing on. Every time there is an error (there are multiple types) I have a log that has many different characteristics of the car at that time of the error. These include things like time of day, speed, location, temperature, and many more. EDIT: If needed I could also have data for normal operation as well.

My goal is to find patterns in the characteristics that have a correlation with and could be contributing to the error. I am not looking to develop a model that will predict future errors. I am only focused on seeing the patterns themselves and understanding them, and having it become more accurate the more data I give it when we encounter the error again.

For example, I want to feed it all my data for one type of error, and have it tell me something along the lines of "When speed is above x you will likely get this error", "When car is in reverse and running at night you will likely get this error", "There is no correlation between outside temperature and this error", etc. And then proceed to do this for every type of error.

Some of the variables could be dependant on each other. Some of the variables might not be relevant/meaningful.

How can I do this in Python? Any guidance is appreciated. I have tried to be as specific as I can and will update my post if further details are needed.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ A big problem I see is that you only have data at the moment of an error, and no data when things are operating normally. Do you plan to use the logs from all the other errors to diagnose the things that correlate with a particular error? This might work if there's no correlation between the error states, but this feels like rather a big assumption to me. $\endgroup$
    – John Rauser
    Jan 10, 2019 at 16:34
  • $\begingroup$ Would I need data for normal operation? Let's say I am just looking at one type of error, I want to see what is common for that one type of error. After I will proceed to analyze myself to see if it is actually a cause. This is mostly to point me in the right direction or in places it has found a correlation that I have not thought to look at yet. Does this help? $\endgroup$
    – J. V.
    Jan 10, 2019 at 16:46
  • $\begingroup$ Suppose certain error (e.g. thermostat failing) only happens when the car is in drive. But the car is in drive during 98% of it's normal operation. It is crucial that the car is in drive to see this error or not? Probably not. You need a baseline to compare against. $\endgroup$
    – John Rauser
    Jan 10, 2019 at 17:50
  • $\begingroup$ A reasonable place to start would be to just plot the distribution of each of your variables for every type of error. Then you could bring your domain expertise to bear. "Ah, this error seems to happen when temperature is abnormally high." You're leveraging your knowledge of typical temperature ranges. This is straight-ahead exploratory analysis, and not fancy machine learning, but might give you a start on your problem. $\endgroup$
    – John Rauser
    Jan 10, 2019 at 17:58
  • $\begingroup$ Ok I understand the issue. Could I just work with the error data and manually comb through to see if it is actually relevant or would it be easier to come up with a solution using all data (error and normal operation). I could get both, I just don't know where to go after this. Thank you for your input. $\endgroup$
    – J. V.
    Jan 10, 2019 at 17:59

1 Answer 1

Reset to default
1
$\begingroup$

You need data for when the error occurred and for normal operation as well, so that you can gain insights from comparing these two groups of log data in analyses and visualizations, e.g. box plots.

You also should consider to regard a time span of log data from before the error happened as possibly related to the occurance of the error. So, you should group your log data into the following two groups by adding a boolean feature to each log entry or data set row: (1) a specific error happened within the next x seconds / minutes / hours / days and (0) a specific error did not happen within the next x seconds / minutes / hours / days.

It is always a good idea to visualize your data, because it helps you to understand your data, e.g. to see if your data is "dirty" or if your assumptions about a specific feature behavior are correct or not as well as related behavior of two or three features. In chapter 4 of the "Python Data Science Handbook" by Jake VanderPlas, the author shows how plot data using python and matplotlib.

In a next step, you can do preprocessing tasks like e.g. to clean your data so that you can later apply statistical methods or machine learning to it to find feature behavior related to error occurrances.

And here is a link to the free online course at edX "Analyzing Data with Python" which you could take.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the info! This is a great place to start. Going a little bit further, what "statistical methods or machine learning" should I be looking towards to find the behavior after I clean the data, which is my end goal. $\endgroup$
    – J. V.
    Jan 11, 2019 at 14:16
  • $\begingroup$ You can try multiple classification methods with python using scikit-learn, but I think you should start with the decision tree classifier, because you can visualize it and see, what the model has learned. You can measure the performance of the classifier using the holdout method or cross-validation. A statistical method you could use is unpaired two-sample t-Test. Or calculate Pearson correlation coefficient for time until next error occurrence vs. each feature and look for negative correlations. $\endgroup$
    – Franziska W.
    Jan 11, 2019 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.