0
$\begingroup$

I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance knowledge on this subject can enlighten me. Here is where I am getting confused:

How can a statistical model (such as linear regression model) built on Hadoop where data is distributed across many nodes, and where work is done only on subset of original data-set residing in each node have same or better accuracy than linear regression model built on single server where all data resides locally. Would the error-rate (accuracy) of a model build on hadoop be same, better or worse than model build on local system. I was thinking that since all data reside in local system, algorithm can be tuned to optimize error in a linear regression model. However, in case of Hadoop where data is distributed, what if only local optimization lead to drastically bad algorithm overall? Are the algorithms written separately for distributed system to take into account data distribution?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

There is nothing special about building something across many machines, or Hadoop, so maybe we can take that out of the question.

One general reason is that, even if the model building process were worse, building on more data could result in a better result.

You're asking why building many models on subsets of data could be better, but this is precisely why ensemble models like random forests can outperform single instances.

But lastly, it's not true that distributed algorithms always work by making an ensemble of small models, so the premise is not strictly true.

Improve this answer
$\endgroup$
2
  • $\begingroup$ My question was about building one model (like linear regression) on many subsets not building many different models on subsets. But, I suppose your ensemble approach is valid. When you are "building on more data", essentially you are building on many small subsets on each node, and I wasn't sure whether that would result in most optimal model....I think as some other posters (William and Ryan Zotti) suggests that either there are no guarantees of optimal model or depending upon what learning algorithm you use accuracy maybe same $\endgroup$
    – buy_sell1
    Jul 1, 2016 at 21:14
  • $\begingroup$ It's not necessarily true that you are building models on subsets. For example, you may perform SGD in a distributed way, computing partial gradients across subsets, but this isn't the same as many regressions. SGD doesn't guarantee an optimal model, but this isn't a function of being distributed. You can solve a linear regression with the normal equation in a distributed way too -- no multiple models there either. That's optimal. $\endgroup$
    – Sean Owen
    Jul 2, 2016 at 9:39
0
$\begingroup$

The answer to the question depends on the what is done on distributed environment.

For example, if the goal is to estimate the mean of the data. Each machine can get one mean value and the number of samples, then a reducer machine could get a weighted average of the means, which is the true mean of all data.

While for some algorithm to heuristically merge results from distributed machines, there is no guarantee on quality of the results.

Improve this answer
$\endgroup$
0
$\begingroup$

The accuracy should be the same. Yes, the data is distributed, and yes on each node there is only a subset of the data, but that does not mean that the learning algorithm (for example gradient descent for linear regression) happens on only one node for only one of the subset/local copies of the data.

One common technique is to perform gradient descent on each node's local subset of the data and then pass the updates back to a master node. The master aggregates incoming updates from across its workers and then sends out the updated weights to workers for continued gradient descent. When repeated numerous times, the error aggregated across all nodes eventually reaches a global minimum.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.