0
$\begingroup$

I'm a regular user of the StackOverflow forum, but as this question is about a recommendation for libraries, which is supported at StackOverflow, and as my question is about data science libraries, I've decided to take my chance here (my original question has been kicked from the "Artificial Intelligence forum").

I'm developping a program that takes a table of a database and predicts the next (limited list of) tupple(s), e.g. for the following table:

A    B    C    D
================
1   12    0    A
2   14    0    A
3   16    0    A
4   18    0    A
5   20    0    A

I would like to predict the following tupple:

A    B    C    D
================
6   22    0    A

For that, I have created two SQL queries:

  1. SELECT COUNT DISTINCT ...
    => if the result is 1, then all values are equal, easy peasy :-)
  2. SELECT MAX(col1-lag_col1) FROM ( SELECT Col1, LAG(Col1) ... (take the difference between subsequent tupples)
    => If both maximum and minimum are equal and not zero, there's an arithmetic progression (more information here). I thought I was on a good track, but then the first problems started arising:

In a list of +800 tupples, the arithmetic progression was disturbed by one single erroneous entry, messing up my prediction. I have seen values like AB123, AB124, AB125, ...: although this is easy for a person to predict, it's not that straightforward for a computer (and what in case of ABC12x or AB12, AC12, AD12, ...?). In both cases, the key solution is pattern recognition: if I can feed the values to a pattern recogniser, it might find the pattern (and maybe the values, messing it up) and I might use that for predicting the next tupple(s). What's the programmers term for pattern recognition? Right: it's A.I., artificial intelligence, more exactly data science. But as most of you know, that's an enormously broad term.

Therefore I'm risking my chances here: does anybody know about a library (or whatever) I might use in my C# program in order to find patterns and predict values (numbers, strings and their combinations)?

$\endgroup$

1 Answer 1

0
+100
$\begingroup$

So in the example that you have given it looks like a linear regression can just be mad between "a" and "b", but if the entire tuple is being predicted, assuming varying values of "c" and "d", you could attempt to simply use "a" as the independent value and the values you want to predict as "b", "c" ,"d". Then you could attempt to create a linear regression based on this, but you would have experiment with results to see if this or another model would work. The only thing I would have to ask is, "Do we know in advance if "a" is going to keep on increasing at the same increment"?

EDIT

So I found an article using stock price data that could be relevant to your example. This article uses a LSTM model to predict the price of an asset and, when it comes to sequences of numbers that change patterns frequently, I can't think of a better example than the financial markets.

Here is a more general introduction to a LSTM models.

Give it a try and let me know what you think.

Second Edit

I found the following links. The first one is an actual library that implements LSTM and other neural networks. The second one is of a gui tool that implements it as well with the third one being the code used for LSTM in this gui project. The fourth link is more of an explanation of the code in the third link.

  1. https://github.com/zhongkaifu/Seq2SeqSharp
  2. https://github.com/bhrnjica/anndotnet
  3. https://github.com/bhrnjica/anndotnet/blob/master/src/core/nnetwork.core/network/Recurrence.cs
  4. https://www.codeproject.com/Articles/5165357/In-Depth-LSTM-Implementation-using-CNTK-on-NET-Pla
$\endgroup$
7
  • $\begingroup$ No, the increment can change: yesterday I've found a column, with values 1, 2, 3 and 4, and again 1, 2, 3 and 4, always the same, for a total amount of hundreds of tupples. I have also seen tuppled that start with "AA01", "AA02" till "AA5", then it restarts with "AB01", "AB02" and "AB03", and then again it restarts with "CC01", "CC02", "CC03", up to "CC89". For a person it would be obvious to put "CC90" as the next, but for a computer it's not, hence my idea to "feed" the data to a pattern recognition system. But how? $\endgroup$
    – Dominique
    Commented Aug 26, 2021 at 12:18
  • $\begingroup$ Does the pattern seem to change within a table or just between different tables? $\endgroup$ Commented Aug 26, 2021 at 12:33
  • $\begingroup$ All changes I have been describing are happening within one single column of a table: there are columns who only contain the same value always, there are columns whose values increase in a linear progression, some columns only have four values who keep getting back, some columns contain string values who behave in the way I described. $\endgroup$
    – Dominique
    Commented Aug 26, 2021 at 12:36
  • $\begingroup$ Just added an edit to your my answer. Hope this helps. $\endgroup$ Commented Aug 26, 2021 at 13:27
  • $\begingroup$ Long Short Term Memory neural networks might indeed be helpful. Do you know .Net libraries who implement these networks, so that I can include them in my C# project? $\endgroup$
    – Dominique
    Commented Aug 26, 2021 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.