Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [processing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
2answers
20 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
0answers
11 views

Setting up Zeppelin to work with a Spark Cluster

I have made a spark cluster containing 2 workers and 1 master. I followed the following link to set up the spark clusters. After successfully setting up the spark clusters I wanted to connect it to ...
0
votes
1answer
39 views

Specifying number of threads using XGBoost.train

When using the xgboost.train() function, all the threads are used. I would like to use a specific amount. Unfortunately, this function does not accept the ...
0
votes
0answers
10 views

Not able to sentence encode a list of sentences using multiprocessing technique - pool.map() function in python

I am trying to embed a text data which is in the form of list, since its a huge data I wanted to embed it using the multiprocessing Pool map() function. The embedding technique I'm using is google's ...
1
vote
0answers
12 views

Can data aggregation (and statistics) be done using an ETL solution?

I am doing a stock trend prediction model. I use MySQL and I have a table that has the following colums: date | open | high | low | close | volume For every day ...
1
vote
1answer
450 views

Gunicorn workers timeout

I'm using Flask where i load some pre-trained machine learning models once. I'm also using Gunicorn usually with 2 or 4 workers to handle parallel requests. Every request contains some texts that i ...
5
votes
1answer
158 views

How to split natural language script into segments?

I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
2
votes
1answer
38 views

Meltdown patch impact on data processing speeds

The patch for the Meltdown vulnerability disables speculative execution, which will impact all processing activities. The degree of impact is highly dependent on the type of processing being done. Is ...
4
votes
1answer
38 views

Alignment of square nonorientable images/data

Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better. Say I have a 2D ...
3
votes
2answers
965 views

Cheat Sheet of UNIX commands for Data Science

I was looking for a cheat sheet of UNIX commands, which are specifically usable for data science. I mean an introduction to very basic commands (starting from cd, <...
0
votes
2answers
67 views

SAS PROC means (two variants together)

PROC means data=d mean; var a; class b; var a; run; I want to perform the "PROC means" for continuous "var a": 1) in general and 2) by classes. But it performed by the classes only. How to make ...
2
votes
3answers
884 views

OCR / Text Recognition and Recovery Problem

I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. ...
3
votes
1answer
945 views

Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?

I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few ...
0
votes
2answers
1k views

Running huge datasets with R

I'm trying to run some analysis with some big datasets (eg 400k rows vs. 400 columns) with R (e.g. using neural networks and recommendation systems). But, it's taking too long to process the data (...
2
votes
1answer
83 views

Is it possible to automate generating reproducibility documentation?

First, think it's worth me stating what I mean by replication & reproducibility: Replication of analysis A results in an exact copy of all inputs and processes that are supply and result in ...
15
votes
2answers
3k views

What is the difference between Hadoop and noSQL

I heard about many tools / frameworks for helping people to process their data (big data environment). One is called Hadoop and the other is the noSQL concept. What is the difference in point of ...