Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
2
votes
1answer
516 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
1
vote
1answer
315 views

How to determine the weights of features when calculating the weighted average?

I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
0
votes
0answers
14 views

How to apply curve fitting on both completed and active data?

Suppose we have a set of part failure times on which a specific curve (e.g. gamma distribution) is already fitted. Then another set of times are given where the part is still active (not failed yet). ...
0
votes
0answers
12 views

Best Technique for find Similar Elements in Sets with Widely Varying Numbers of Attributes

I have a data set containing millions of items collected from many disparate sources. Each item contains a list of anywhere from fifty to a thousand attributes. The specific attributes available vary ...
1
vote
0answers
23 views

Calculating unaffected data [closed]

Some data sets values are missing. Missing values are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected in %. Why?...
0
votes
0answers
65 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
1
vote
0answers
10 views

To what extent can one link data in a structured or rigorous way, given only partial knowledge base?

I seem to have stumbled on an interesting problem: suppose I am given partial information in two tables, say imports and exports between different countries, and I can query these, but e.g. the ...
0
votes
0answers
21 views

String Matching columns based on weight

I have the following 2 DataFrames and want to match skills to the key_skills column based on weights. for eg: suppose student 1-{python, Java} job - {python, Ruby, Perl, etc } weight = 1 In the end,...
1
vote
1answer
390 views

How to feed data to Machine Learning Model?

I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression ...
1
vote
1answer
121 views

Python equivalent of Wolfram Language Query for JSON?

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
1
vote
1answer
34 views

Why is correlation between my independent variables helping my linear regression model?

I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three ...
1
vote
0answers
37 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
3
votes
2answers
178 views

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
1
vote
0answers
34 views

Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)

We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or ...
2
votes
2answers
192 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
0
votes
2answers
226 views

Trying to find the correlation between inputs and output

I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code: ...
2
votes
0answers
54 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
0
votes
1answer
900 views

What is the concept of error bars in barplots in seaborn library?

I am using seaborn library to perform EDA on a dataset having categorical variables. I am using barplot function and in that there are some black bars which I guess are error bars as I read in ...
0
votes
1answer
71 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
56 views

Behavioural data required to predict churn

I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all ...
0
votes
1answer
266 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
1
vote
0answers
28 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
2
votes
2answers
30 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
0
votes
1answer
33 views

Want suggestions on choosing open source embedded BI tool [closed]

I need some suggestions from all group members regarding the open source reporting/dash-boarding tool which fulfills below specific requirements apart from some basic BI functionalities: 1. Can be ...
1
vote
0answers
16 views

Data matching between different columns when multiple values in single row

Can someone please provide input on this issue ? It seems so doable but we are novices in data manipulation. There are 4 columns in the reduced data set below. The problem is to find the values in ...
2
votes
3answers
50 views

Does adding new features that contain information derivable from current features help performance?

So say you have some data that consists of some values: 1.3, 0.9, 1.1 You introduce a new feature which is the average of these values: 3.3 In this example lets say that you know the average of ...
0
votes
1answer
64 views

When to stop collecting data?

I have developed an iterative process via which I can collect data in batches. The data are points in a predefined 3D space. I am trying to explore and locate clusters in that 3D space based on my ...
0
votes
1answer
26 views

Website for Datasets - Miles and Shevlin book, “Applying Regression & Correlation”

I've very much enjoyed Miles and Shevlin (2015) book, "Applying regression & correlation : a guide for students and researchers." The book mentions availability of datasets, but the website ...
0
votes
3answers
859 views

examples of semi structured data ?

i read that Json or XML are unstructured data; Are Json or XML data or are they tools to tag the data? I understand from this wikipedia page (https://en.wikipedia.org/wiki/Semi-structured_data) that ...
0
votes
1answer
46 views

What should be the requirement for training data in order to obtain a good regression model using neural network?

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet. Data:- I have 3000 samples ...
1
vote
0answers
519 views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition - https://www.kaggle.com/c/ga-customer-revenue-prediction Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back ...
1
vote
1answer
27 views

Does this type of classification exist?

Im fairly new to data science and trying to see if a type of classification exists for my needs. I understand that a classification into 2 categories will look something like this: You have 2 ...
1
vote
2answers
1k views

What algorithms are good to predict next numbers?

Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are ...
2
votes
3answers
64 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
263 views

Solving correlation and covariance with MySQL

I am new to data science taking my first course in Data analysis. I am trying to figure out where to start with a problem solving the correlation and covariance with my sql. ...
0
votes
1answer
37 views

How to obtain and load a good initial data set for object localization?

I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data). What is a good initial data set to use? How can ...
0
votes
1answer
3k views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
1
vote
1answer
32 views

Do I need a datalake in my use case?

My web application stores usage data, for example: tickets opened an closed tasks executed user scores etc. I need to show dashboards and reports for usage and performance trends, like: How many ...
1
vote
3answers
2k views

Data Cleaning without pandas [closed]

How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the ...
0
votes
1answer
37 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
2
votes
1answer
413 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
0
votes
1answer
38 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
1
vote
1answer
58 views

generate a report in R where the value in each Column is to be derived from different tables in sybase databasea

I work in a stress testing team. my boss has asked me to generate an interactive report every day using different tables in a Sybase database. I know the table structure and sql queries, but can't get ...
2
votes
1answer
48 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
1
vote
2answers
17 views

Different types of analysis which can be done with Webdata of users

I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are: Product Usage details Geographic details Session data: Capture what all they did ...
1
vote
2answers
37 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?
0
votes
2answers
35 views

Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

I have several marketing files that I need to download on a weekly basis. Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
0
votes
1answer
38 views

Dataset with features as SNOWMED codes and outputs as names of medications prescribed.What are some ML algorithms that suit this case?

So, i am constructing a dataset (from doctor prescription notes and diagnosis) where X (input features) represent the various snowmed codes for illnesses/surgeries performed/chronic conditions etc. ...
2
votes
2answers
4k views

Create a binary-classification dataset (python: sklearn.datasets.make_classification)

I would like to create a dataset, however I need a little help. The dataset is completely fictional - everything is something I just made up. Since the dataset is for a school project, it should be ...
2
votes
0answers
54 views

Beginner Question Related To Data Science Course [closed]

I am Third Year B.Tech student from 3-tier college of India ,Here is no one fellow or collegous who has somebit knowledge about ML or Data science and I am purely sure that I have strong background of ...