Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

0
votes
0answers
3 views

How can I unzip my downloaded dataset from kaggle on google colab?

I want to unzip a datset I have downloaded on my colaboratory from kaggle competition but everytime I run the command unzip I get the error no such file foud, whereas I have downloaded the dataset. ...
0
votes
0answers
6 views

How to efficiently store large amounts of speech to text data

I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task. Most datasets I am using are coming ...
1
vote
0answers
15 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
2
votes
2answers
48 views

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
0
votes
0answers
18 views

Rate quality of sources

Sorry if the title is misleading, I don't know how to describe the issue correctly, which also makes it hard to google. Say we have historical weather predictions from three different sources. All ...
0
votes
0answers
6 views

Need suggestion on picking fast aggregation framework or solution

To state the problem, this is what I have in the MariaDB/MySQL. I have the follow table structure, scores ...
0
votes
0answers
5 views

Enrichment analysis Boostrapping fails VS hypergeometric test, why?

I found something that bugs me! Here is the problem: I have a set of ~500 objects, that have tags on them. After the selection of X members of that set, I want to know if any of the tags are ...
1
vote
0answers
10 views

Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)

We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or ...
1
vote
1answer
9 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
0
votes
2answers
39 views

Trying to find the correlation between inputs and output

I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code: ...
0
votes
0answers
11 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
0
votes
1answer
18 views

What is the concept of error bars in barplots in seaborn library?

I am using seaborn library to perform EDA on a dataset having categorical variables. I am using barplot function and in that there are some black bars which I guess are error bars as I read in ...
0
votes
0answers
3 views

Game Data Analysis (Stats)

Analyzing some game data and wanted some help to know how a specific column affects the game economy dynamics. Game Scenario: • This game data comes from a video game in which players earn items ...
0
votes
0answers
16 views

Is there a name for this phenomena (Asking a question that is obviously not covered by a case study)

I came across this situation when presenting conference papers as part of my PhD (Data Modelling, and Fault prediction). The set up is as follows: I would provide a 10 - 15 minute pitch of my case ...
-1
votes
0answers
15 views

Data preprocessing step

I have a question concern data preprocessing step. I have 5 tables (csv files) , should i clean them one by one before merging them into 1 dataframe? Or i merge them into 1 dataframe, then I clean the ...
0
votes
0answers
12 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
20 views

Behavioural data required to predict churn

I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all ...
0
votes
0answers
28 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
0answers
9 views

Which is the best way to generate synthetic data for data science development

I have a data pipeline in place for streaming and batch data, which collects data from various sources and land it into hadoop's hdfs storage. I want to generate synthetic data with the same schema of ...
0
votes
0answers
5 views

operations on multiple entries in one column based on conditions meet from multiple column entries

I have a data set where I need to find the difference between the first time entry and last time entry based on the following conditions: Day column entries are equal BusOne[0] = BusOne1 ... BusOne[n]...
1
vote
0answers
6 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
2
votes
2answers
21 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
0
votes
1answer
19 views

Want suggestions on choosing open source embedded BI tool [closed]

I need some suggestions from all group members regarding the open source reporting/dash-boarding tool which fulfills below specific requirements apart from some basic BI functionalities: 1. Can be ...
1
vote
0answers
14 views

Data matching between different columns when multiple values in single row

Can someone please provide input on this issue ? It seems so doable but we are novices in data manipulation. There are 4 columns in the reduced data set below. The problem is to find the values in ...
0
votes
0answers
10 views

Excessive Data checking

This probably too opinion-based. But, I am still a newbie to data science, and I often do not know how far is far enough when it comes to my analyses or ensuring data quality. A silly example: ...
0
votes
0answers
14 views

How to read a LISP-readable form data into python?

I have been trying to read this data into python. I tried various techniques but I am not able to organise it in a format that can be used for training a model. Has anybody got any idea how to read ...
0
votes
0answers
6 views

How to smoothen array of samples?

I am implementing a Genetic Algorithm library. However, on Roulette Wheel Selection algorithm sometimes one or two chromosomes dominate he roulette wheel with their relatively large fitness scores. I ...
2
votes
3answers
42 views

Does adding new features that contain information derivable from current features help performance?

So say you have some data that consists of some values: 1.3, 0.9, 1.1 You introduce a new feature which is the average of these values: 3.3 In this example lets say that you know the average of ...
-1
votes
0answers
16 views

Stacked machine learning models train - test split on time series data

I have a question about the best approach to a problem I am facing. I have time series data, say from day 1 to day 100. I Split the data into a training and test set, 80:20 and apply 3 machine ...
0
votes
1answer
45 views

When to stop collecting data?

I have developed an iterative process via which I can collect data in batches. The data are points in a predefined 3D space. I am trying to explore and locate clusters in that 3D space based on my ...
0
votes
1answer
10 views

Website for Datasets - Miles and Shevlin book, “Applying Regression & Correlation”

I've very much enjoyed Miles and Shevlin (2015) book, "Applying regression & correlation : a guide for students and researchers." The book mentions availability of datasets, but the website ...
0
votes
3answers
27 views

examples of semi structured data ?

i read that Json or XML are unstructured data; Are Json or XML data or are they tools to tag the data? I understand from this wikipedia page (https://en.wikipedia.org/wiki/Semi-structured_data) that ...
0
votes
1answer
30 views

What should be the requirement for training data in order to obtain a good regression model using neural network?

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet. Data:- I have 3000 samples ...
1
vote
0answers
55 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
0
votes
1answer
26 views

distribution data

I am considering to use a single number to represent the age distribution across different countires. But, some of the countries are long tailed but some of them are short tailed. I can think of ...
1
vote
1answer
25 views

Does this type of classification exist?

Im fairly new to data science and trying to see if a type of classification exists for my needs. I understand that a classification into 2 categories will look something like this: You have 2 ...
1
vote
1answer
41 views

What algorithms are good to predict next numbers?

Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are ...
0
votes
3answers
24 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
44 views

Solving correlation and covariance with MySQL

I am new to data science taking my first course in Data analysis. I am trying to figure out where to start with a problem solving the correlation and covariance with my sql. ...
0
votes
1answer
8 views

How to obtain and load a good initial data set for object localization?

I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data). What is a good initial data set to use? How can ...
0
votes
1answer
310 views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
0
votes
1answer
13 views

Do I need a datalake in my use case?

My web application stores usage data, for example: tickets opened an closed tasks executed user scores etc. I need to show dashboards and reports for usage and performance trends, like: How many ...
0
votes
1answer
17 views

Weather data of Texas by premise zip

I need historic data of Texas' weather of the past ten years(2010-2018) by zip codes on monthly basis. is there an api or a website from where i can scrape the required data?
1
vote
3answers
139 views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
0
votes
1answer
25 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
0
votes
1answer
87 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
0
votes
0answers
25 views

Data science short courses in Europe

Maybe my question is broad and even "bad", so excuse me for that! I am looking for some good and intensive short courses in data science/machine learning etc. around Europe. By short i mean a week ...
0
votes
1answer
27 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...