1
$\begingroup$

Detailed Question Explanation:

Suppose say our application X is processing huge logs (size varying from MBs to GBs) and giving insight results in these logs(NOT A Social Data logs or Security Logs)

now this logs are in format say log.y with different variety, using C++ as Engine to process these huge logs.(It generates imp. insights about data but need to be processed using our application X only and we don't want to change core way processing of application X)

If this processing happens on some server it under or over utilizes resources (That I already know).

If we use cloud computing for this processing we get that processing power with optimum usage.

How do we see help of BIG data analytics in this particular sort of usage? Any help or suggestion is very deeply appreciated

$\endgroup$
  • 1
    $\begingroup$ Can you please clarify what your problem is exactly? Is it poor utilization of server resources? Are you unable to process the data? $\endgroup$ – sheldonkreger Jun 18 '15 at 20:48
  • $\begingroup$ Poor utilization of resource so cloud solution is needed. But I don't know if I can use Big Data Analytics I can use in this case or not . $\endgroup$ – spt025 Jun 22 '15 at 4:11
0
$\begingroup$

If the size of your logs are still growing then a distributed data system is definitely the right way to go. I have been using Mesos in production for almost a year now and it solved the problem of if this processing happens on some server it under or over utilizes resources.

I probably would look into some stacks like this:

  • Mesos as your fault-tolerant and elastic distributed systems
  • Spark or some Hadoop-based solutions for log processing and store the output in a DFS like HDFS
  • Have your applications consume data stored in HDFS as final steps
| improve this answer | |
$\endgroup$
  • $\begingroup$ I need an suggestion, if we use this logs processing as web service for end user like my employee wants to give 100-200 gb for processing so Can I provide web service through which he/she can access this ?? $\endgroup$ – spt025 Jul 10 '15 at 10:13
  • $\begingroup$ It sounds like you need to develop some sort of web service by yourself. Otherwise if you have budget, try some SaaS based data platform like Databricks, they are the new kid in town but I have heard great things about them. $\endgroup$ – lumengxi Jul 11 '15 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.