2

First of all I just want to say that I am not a data engineer and there is definitely someone out there that can answer this better than me. I do think that there is a lot of theory behind data engineering. It is also very interesting. I too thought that it was boring and I was more interested in just data science/ machine learning. I am not sure if I can ...


2

Dataquest is a great place to start. It will help you bridge the gap from just learning to actually coding as a data analyst, data engineer, or data scientist. Unlike most online courses, there are no videos, and they provide an interactive coding and learning environment that makes it very easy to learn the practical techniques you'll actually need to work ...


2

As you said, since the Koalas is aiming for processing the big data, there is no such overhead like collecting data into a single partition when ks.DataFrame(df). However, the overheads are occurred when creating a default columns for creating the _InternalFrame which internally manages the metadata between pandas and PySpark. Koalas is internally using ...


1

For batch training i have been utilizing sagemaker though it's a bit expensive then ec2 but it's easy to setup and get started. Make a docker container and push it to ecr then start the training and track the metrics using any monitoring tool like wandb If your use case don't require any custom packages then you can also utilize HuggingFace DLC it which can ...


1

You probably should conduct a missing values analysis to see what is the percentage of missing per column (figure below, from dataprep package) Decide a threshold according to which you may want to completely drop a column or not (depending on how your analysis or model treats nans as well) For the columns that are not dropped, you should impute the missing ...


1

I believe Kolas is the Databricks DF equivalent of a Python DF and the equivalent of a Spark DF (I think Kolas is very,very new; released just a few months ago). I don't know what you mean by cost, but you can easily switch between Spark DF and Pandas DF. See the examples below. # Convert Koala dataframe to Spark dataframe df = kdf.to_spark(kdf) # Create ...


1

In most cases, you would use a file-storage solution such as Amazon S3 or Google Cloud and many others, which provide designated solutions for large object storage and retrieval. You would then ideally want to update your code to retrieve the model directly from the file storage. Whether this download needs be done on every run or only once (storing the ...


1

First of all, Congrats that you have started your journey of becoming a Data engineer/analyst. According to me there is no clear path on how to become a Data analyst. Applied data science with Python course is great but i would urge you to start working on problem statement of any domain (NLP,CV) and take part in competition where you will learn how to ...


1

There a three dimensions of knowledge you need to be a generally competent data scientist: Statistical / Mathematical knowledge Programming / CS skills Domain knowledge I think studying and courses help in 1) and to a degree in 2) but not at all in 3. If you feel confident enough in 1/2 that you can solve some real life cases it would be best to start ...


1

There certainly is theory, or at least competing methodologies, behind ETL and Data Warehousing, for a start look at the Inmon vs Kimball methodologies. In a nutshell (I could talk for days on this subject), Bruce Inmon's (the Father of Data Warehousing) methodology revolved around building a large, loosely 3rd normalized data warehouse from multiple sources,...


1

Your example is a star schema, it's just a star with three points (dimensions). It's OK to have star schemas with more dimensions. Some large OLAP schemas can have tens of dimensions. The star schema holds the underlying data. The cube is a convenient set of pre-aggregated values that make our run-time faster. The "cube" name is a handy visual and ...


1

You can do this, waaay to easier to what you are currently doing. For the data scraping, use whatever makes you happy. In my case I will use Uipath or just python, depending on the complexity. But this is up to you, you just want some dataset in a format that suits you. Once you have your data, you want to visualize that. This is a classical data science ...


1

There is no value in this. local[*] master will already use all cores and you can use as much local memory as you like already. Starting a worker just adds overhead.


1

Yes you definitely can. Here's an example: Using Convolutional Neural Networks to Classify Hate-Speech The authors used classic embeddings concatenated with a vector of size 28 representing the presence or not (in a tweet) of each letter of the alphabet (26) plus any digit (1) plus any other symbol (1). So basically for a tweet like 'I love NLP!' the ...


Only top voted, non community-wiki answers of a minimum length are eligible