Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
1
vote
0answers
26 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
2
votes
2answers
29 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
0
votes
1answer
33 views

Want suggestions on choosing open source embedded BI tool [closed]

I need some suggestions from all group members regarding the open source reporting/dash-boarding tool which fulfills below specific requirements apart from some basic BI functionalities: 1. Can be ...
1
vote
0answers
16 views

Data matching between different columns when multiple values in single row

Can someone please provide input on this issue ? It seems so doable but we are novices in data manipulation. There are 4 columns in the reduced data set below. The problem is to find the values in ...
2
votes
3answers
50 views

Does adding new features that contain information derivable from current features help performance?

So say you have some data that consists of some values: 1.3, 0.9, 1.1 You introduce a new feature which is the average of these values: 3.3 In this example lets say that you know the average of ...
0
votes
1answer
55 views

When to stop collecting data?

I have developed an iterative process via which I can collect data in batches. The data are points in a predefined 3D space. I am trying to explore and locate clusters in that 3D space based on my ...
0
votes
1answer
24 views

Website for Datasets - Miles and Shevlin book, “Applying Regression & Correlation”

I've very much enjoyed Miles and Shevlin (2015) book, "Applying regression & correlation : a guide for students and researchers." The book mentions availability of datasets, but the website ...
0
votes
3answers
749 views

examples of semi structured data ?

i read that Json or XML are unstructured data; Are Json or XML data or are they tools to tag the data? I understand from this wikipedia page (https://en.wikipedia.org/wiki/Semi-structured_data) that ...
0
votes
1answer
42 views

What should be the requirement for training data in order to obtain a good regression model using neural network?

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet. Data:- I have 3000 samples ...
1
vote
0answers
469 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
0
votes
1answer
34 views

distribution data

I am considering to use a single number to represent the age distribution across different countires. But, some of the countries are long tailed but some of them are short tailed. I can think of ...
1
vote
1answer
27 views

Does this type of classification exist?

Im fairly new to data science and trying to see if a type of classification exists for my needs. I understand that a classification into 2 categories will look something like this: You have 2 ...
1
vote
2answers
894 views

What algorithms are good to predict next numbers?

Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are ...
1
vote
3answers
61 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
233 views

Solving correlation and covariance with MySQL

I am new to data science taking my first course in Data analysis. I am trying to figure out where to start with a problem solving the correlation and covariance with my sql. ...
0
votes
1answer
29 views

How to obtain and load a good initial data set for object localization?

I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data). What is a good initial data set to use? How can ...
0
votes
1answer
3k views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
1
vote
1answer
31 views

Do I need a datalake in my use case?

My web application stores usage data, for example: tickets opened an closed tasks executed user scores etc. I need to show dashboards and reports for usage and performance trends, like: How many ...
0
votes
1answer
22 views

Weather data of Texas by premise zip

I need historic data of Texas' weather of the past ten years(2010-2018) by zip codes on monthly basis. is there an api or a website from where i can scrape the required data?
1
vote
3answers
1k views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
0
votes
1answer
33 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
2
votes
1answer
376 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
0
votes
1answer
37 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
49 views

generate a report in R where the value in each Column is to be derived from different tables in sybase databasea

I work in a stress testing team. my boss has asked me to generate an interactive report every day using different tables in a Sybase database. I know the table structure and sql queries, but can't get ...
2
votes
1answer
47 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
1
vote
2answers
16 views

Different types of analysis which can be done with Webdata of users

I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are: Product Usage details Geographic details Session data: Capture what all they did ...
1
vote
2answers
34 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?
0
votes
2answers
32 views

Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

I have several marketing files that I need to download on a weekly basis. Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
0
votes
1answer
36 views

Dataset with features as SNOWMED codes and outputs as names of medications prescribed.What are some ML algorithms that suit this case?

So, i am constructing a dataset (from doctor prescription notes and diagnosis) where X (input features) represent the various snowmed codes for illnesses/surgeries performed/chronic conditions etc. ...
2
votes
2answers
3k views

Create a binary-classification dataset (python: sklearn.datasets.make_classification)

I would like to create a dataset, however I need a little help. The dataset is completely fictional - everything is something I just made up. Since the dataset is for a school project, it should be ...
2
votes
0answers
53 views

Beginner Question Related To Data Science Course [closed]

I am Third Year B.Tech student from 3-tier college of India ,Here is no one fellow or collegous who has somebit knowledge about ML or Data science and I am purely sure that I have strong background of ...
7
votes
6answers
281 views

Is it advisable to combine two dataset?

I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based ...
1
vote
1answer
35 views

Why not higher accuracy in Otto data?

On this site of Otto Group Product Classification Challenge, it is shown that best accuracy was possible with RandomForest method, but it was relatively low at 0.83. Accuracy with ANN and with Naive ...
17
votes
1answer
26k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
1
vote
1answer
365 views

How can I sort a data frame by groups? [closed]

I am using R. I have a data frame with "year , price, mileage" columns. I want to group the df by year first and then sort each group by mileage. How can I do this?
2
votes
1answer
38 views

Convert a csv file so that all attributes will become columns with corresponding element [closed]

so I have this dataset in a csv file. I would like to convert it into a matrix form. The data currently looks like this: ...
0
votes
1answer
58 views

Image data augmentation with this function

Following function is from http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification-py ...
0
votes
1answer
39 views

Handling imbalanced data by deleting over represented rows vs. adding under represented rows

I am currently working with a very imbalanced data set (frauded credit card data from kaggle, which has 492 rows of frauded cards and over 280,000 rows of non-frauded cards). As much as I know, there ...
1
vote
0answers
94 views

Transposing a dataframe (time-series)

I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting): ...
3
votes
2answers
41 views

How to generate data if algo itself is involved in the process with a feedback loop?

I have an algorithm which would be a rather easy classification task with a set of features and a class output which I would like to solve with a machine learning algo. But I am having issues and ...
2
votes
1answer
33 views

Algortihm for making predictions from minimal data

I am working with classification problem. I have a dataset with a lot of features. A lot of them can easily determine class. On production, I want to ask the user to provide me with only part of the ...
6
votes
5answers
2k views

Interactive labeling/annotating of time series data

I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
1
vote
0answers
18 views

Having the target and data row - how to find the correct model

I am coming from a web developer background, and having no data science background. I am looking to skill-up and get some answers for a current task: find a model that returns a predicted value same ...
4
votes
4answers
505 views

How to know for sure if we can learn from a given data or not?

I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?
1
vote
1answer
33 views

Prepare data : many products per row

I would like to find how to change my data structure to make it compatible with a ML model, with the following constraints : A Product is defined by {id, price, continuousVariable1, ..., ...
1
vote
1answer
211 views

Is there an API to extract questions and labels from yahoo answers?

I want to work on a ML project which involves the family and relationship category from yahoo answers. I want to extract questions from yahoo answers (only the title) with the label as the category it ...
1
vote
0answers
46 views

Time Series Autocorrelation Estimation

How do I find joint PDF f(xi,xj) needed for estimating autocorrelation from 1-D time series data X, using the kernel windowing technique? I am well aware of kernel windowing technique for estimation ...
4
votes
3answers
164 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
1
vote
0answers
10 views

Where to find Facial Visual Object Class

I am trying to make a face/head detection model using YOLO and darkflow. In order to do this I need a set of images peoples face annotated in Visual Object Class, VOC format. I would prefer data set ...
3
votes
0answers
45 views

Deep Learning: Does starting the training on a smaller subset of the data make sense?

I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the ...