i have a dataset of work period from a lot of people during multiple years and i would like to find the overlapping date between these periods, the data would look something like this:

John Doe;2015-01-01;2017-01-01;Yes
Jonh Doe;2016-02-03;2016-06-04;No
Jane Doe;1980-01-01;2018-01-01;Yes

As you can see i can have multiple lines for the same person and are mapped as valid or not, there are a few more rules like if the person is a professor he can have overlapping work periods but the core is to find these wrong periods created by human error.

Since i am a beginner in machine learning i would to try to implement this as a learning project although i intend to use and expand in the future if all goes well, at least to my eyes seems like a somewhat easy task to do.

I would like your opinion in what kind of algorithm i should use, or what framework i should or not use, or even if this is not very cost effective to implement on ML.

I thought of somekind of supervised learning using tensorflow, maybe is too much?

Also any tips, tricks or tutorials would be much appreciate it.

EDIT: More Information At first i am providing the answer, which would be the "Yes" or "No" on Valid, but when i provide something like:

1;1;John Doe;2015-01-01;2017-01-01;
2;1;Jonh Doe;2016-02-03;2016-06-04;
3;1;Jonh Doe;2017-06-02;2018-04-01;
4;1;Jonh Doe;1990-01-01;2017-07-01;

So i would like it to point out, hey, John Doe has 2 invalid periods, row 2 and 4, why should it give me that? because 2016-02-03 to 2016-06-04 is already covered by row 1 and row 4 overlap with row 1,2,3

At first i thought on making this with regular programming to check for everything, but there are some cases where human error make this quite a difficult task to do, thats why ml got in to my mind, the algorithm does not need to have really high accuracy, this is mostly a learning project.

  • $\begingroup$ Hi Pettrus, Can you shed some light what do you exactly mean by overlapping date between periods?(probably by adding the answer to your sample dataset) $\endgroup$ – Aditya Apr 10 '18 at 3:30
  • $\begingroup$ There is not a proper answer but more like it pointing out to me invalid periods, i tried my best to give you more information on what i want $\endgroup$ – Pettrus Sherlock Apr 11 '18 at 3:25

Sorting and cleaning this dataset from overlapping periods is a task you can do with plain python, perhaps a bit more comfortable if you use pandas. Machine learning would be an additional overkill, it adds more complexity and is of no help for that task. Although I understand you are looking for a learning project, I do not recommend to use this one, and rather recommend to pick a task that is more often solved with machine learning.

| improve this answer | |
  • $\begingroup$ I see, i guess i will find another project more related to machine learning then, thanks for the answer $\endgroup$ – Pettrus Sherlock Apr 12 '18 at 11:42

This is quite easy and you don't need machine learning at all.

Order by Person and Start date. Then simply iterate over each data line for a person and see if the start dates are earlier than the end dates for the previous entry. Add rules as necessary.

| improve this answer | |
  • $\begingroup$ Yes but i like to use real problems to learn something new, but as @Frankstr said this is overkill, i am going to find another project to get started, thanks for the answer. $\endgroup$ – Pettrus Sherlock Apr 12 '18 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.