0
$\begingroup$

How can one use unstructured data for forecasting purposes? (by unstructured, I mean unstructured in the database sense).

I have a forecasting system that uses historical data and a set of additional explanatory features to forecast demand. In terms of databases, this is as structured as data can get.

I've heard mentioned in several places that big data and unstructured data can be used to improve demand forecasts.

But when I dig into the details, it seems that they always end up reducing the unstructured data to some structured format first, then feed it to whatever forecasting method they are using. They aren't using "true unstructured data".

How can one use true unstructured data in a forecasting algorithm? Any publications, references on this?

Improve this question
$\endgroup$
3
  • $\begingroup$ What do you mean exactly by unstructured? Neural networks for instance have many parameters and when you fit your data, they find the best match. Could it be that you are looking for "non-parametric" models? All parametric models have a shape (parameters). $\endgroup$
    – PM0087
    Jun 6, 2016 at 13:50
  • $\begingroup$ @PeyM87 By unstructured I mean from an unstructured database. that is data that doesn't always have the same number of fields or data format. See here $\endgroup$
    – Alex S Kinman
    Jun 13, 2016 at 17:49
  • $\begingroup$ You have to extract some features some way to learn you model. I can imagine universal feature extractor from anything - texts, images and so on and universal model chooser and tuner, but it will be very slow and inefficient in comparison with manual data science process based on some knowledge about the data you are trying to deal with. $\endgroup$
    – CrazyElf
    Dec 28, 2017 at 15:00

2 Answers 2

Reset to default
1
$\begingroup$

How you use data depends mostly on the domain problem you're trying to address.

In the case you mention, it would depend on what exactly you're trying to forecast. For example, if you're trying to forecast sales or renewalls, there are some immediate use cases where you can leverage "unstructured" data to increase the number and/or the quality of the signals (features) that you are feeding into your model: 1) process audio data from contact centre interactions to determine the overall sentiment of a customer (i.e. "is he up for a sales call?"); 2) process text from customer reviews and use sentiment/keywords as a preditor for a churn model; etc!

it seems that they always end up reducing the unstructured data to some structured format first, then feed it to whatever forecasting method they are using.

Probably because their forecasting algorithm or method requires that data is inputted in a specific format, like most forecasting or classification algorithms do; different algorithms will have different tolerances to data issues like missing values or outliers. So you process a piece of data that is unstructured with the goal of using it, usually by adding it to a data model or inputting it as data for a report or as a feature for a predictive model.

PS: I'm curious if anyone knows a case where you just throw unstructured data at a predictive algorithm (e.g. a neural network) and get something meaningful or useful out of it.

Improve this answer
$\endgroup$
0
$\begingroup$

A recurrent neural network can take unstructured data, such as video or raw text, to make a prediction.

For example, the model can use as input the text from a large set of emails, and from there try to forecast something.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.