0
$\begingroup$

Total Newbie question here :

5 years ago, I wrote a web application in LAMP stack not using a framework, targeting end users, which now has over 200,000 rows where each row has 50+ columns and the MySQL database size is over a GB.

How do I analyze the stats of the data including visually ?

Is there some tool to pass along this data to it and it magically generates stats ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

There are multiple options if you want to extract insights from your data, some of which are:

  1. Python or R : If you are comfortable in either of the 2 languages, you can easily generate visualizations in them. I cannot speak for R as I don't use it but in python there is a library called Dtale which automates exactly what you want. Just import the data into the function and it will generate multiple stats and visualizations for you.
  2. PowerBI or Tableau : No code solution for visualizations. ALthough I don't know how these can magically generate stats or visualizations! But it is extremely easy for you to do so as these tools require a drag and drop functionality.

However the problem is not the tool but rather the large dataset size. If you have sufficient memory then there's no issue but if you are limited by your RAM then you would have to use batches to perform your analysis. Also there are some python BigData libraries that can be used. Either way what you want can easily be done.

Cheers!

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Thanks for your response. But I'm surprised that 1GB is considered "large dataset size" - normally medium sized web apps have db sizes in multiples of GBs. $\endgroup$
    – anjanesh
    Jun 16 at 3:27
  • 1
    $\begingroup$ @anjanesh 1GB can be large for a not so powerful setup. If you have sufficient RAM then there should be no problem analysing your data in a single go or without using some Big Data libraries. Depends on your hardware IMO. $\endgroup$
    – spectre
    Jun 16 at 3:30
  • 1
    $\begingroup$ @anjanesh If you are running some kind of ML algorithm then definately you might fave OOM issues. Even for a simple regression analysis OOM occurs. So it depends on your task and hardware! $\endgroup$
    – spectre
    Jun 16 at 3:31
  • $\begingroup$ Thanks - I was wondering if this d-tale / jupyter could be on my application's server for other stakeholders to view, than being on my laptop alone - the server is on Azure or DO with sufficient RAM and only 1-2 users max. $\endgroup$
    – anjanesh
    Jun 16 at 4:20
  • 2
    $\begingroup$ Yes you can install jupyter-lab/notebook and dtale on your azure server or whichever cloud platform you prefer. $\endgroup$
    – spectre
    Jun 16 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.