Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
7
votes
5answers
2k views

Interactive labeling/annotating of time series data

I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
1
vote
1answer
144 views

can I use z-score normalization even if it doesn't make sense for my data to be negative?

I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
0answers
13 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
2answers
193 views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
2
votes
2answers
843 views

Multi-touch Attribution Model

I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling ...
2
votes
1answer
63 views

How to reduce position changes after dimensionality reduction?

Disclaimer: I'm a machine learning beginner. I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
0
votes
0answers
26 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
0answers
20 views

Dataset creation software

Hello I am looking for a dataset for printed english characters but I am not able to find anything concrete. Is there any software which can generate multiple images from one image. I can build my ...
1
vote
1answer
30 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
0
votes
1answer
12 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
2answers
40 views

Enormous dataset, how to proceed?

So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level ...
1
vote
1answer
29 views

Computing Jaccard Similarity between two documents

Data Mining: Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) = D1 = the quick brown fox jumps over the lazy dog D2 = jeff typed the quick brown dog jumps over the lazy fox ...
0
votes
1answer
54 views

Behavioural data required to predict churn

I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all ...
1
vote
2answers
157 views

What is the difference between a data-driven model and an empirical model?

Are they the same? Empirical models, per Wikipedia, are any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system ...
-1
votes
1answer
26 views

How to interact two variables in python?

I have about 8 features as my predictors in a logistic model I am trying to fit in python. One of the features is TotalAward (Financial Aid) and the second is NEED. I am attempting to predict the ...
1
vote
1answer
34 views

Growth function of a 6-dimensional linear classifier

In our course, we are dealing with a d-dimensional classification problem ($\chi = \mathbb{R}^{d}$ as our input space, and $y = \{-1,+1\}$). Our hypothesis class $H$ consists of all hypotheses of the ...
0
votes
0answers
21 views

real-valued data normalization for regression

We already know that non-real-valued fields of a data set to be modeled by neural network or some other machine learning approach must be transformed into real value. Some categorical or nominal type ...
1
vote
1answer
25 views

Meaningful and Non-Meaningful Data?

I can understand what meaningful data is like its important information that can be used to evaluate something but I don't get what non-meaningful data is? Is it less important data?
0
votes
0answers
15 views

Averaging over Irregular Intervals

Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation? Specifically, suppose we are ...
2
votes
1answer
71 views

How to correct mislabeled data in dataset?

I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly ...
1
vote
1answer
30 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
4
votes
0answers
60 views

Model for Differing Number of Rows per Observation

Looking to build a response model (click or no click) on marketing data which displays varying number of offers to a person. I don't want to model which offer they click but do they click any of the ...
1
vote
0answers
18 views

How to use SOFT dataset in Orange

I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?
1
vote
1answer
21 views

How has data generation evolved over the years (and centuries)?

I would like to know how data generation has evolved through the years. My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does ...
0
votes
1answer
64 views

Data Visualisation Techniques for Multi Labelled data

I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: ...
1
vote
0answers
32 views

Clustering data by multiple values

I am trying to find a way to cluster/group students by their knowledge of different subjects. Given following as an example: ...
3
votes
1answer
41 views

How to choose tools for web dashboard?

I am working in a small sales company. Started with making relatively easy, small, static reports in Excel, but then it was required to make a lot of highly-customizable and adaptable reports, that ...
1
vote
1answer
27 views

Where to find height dataset, or datasets in General

Hi there smart people, I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest. For example, I ...
1
vote
0answers
12 views

How to create trade flows-map chart?

I would like to know if there is a relatively easy way to create trade flows-map charts(preferably without coding), showing i.e. exports from Country A to Countries B,C,D. I have tried doing it so far ...
0
votes
0answers
22 views

Are application assignments in Datascience common practice?

this might seem off topic, but I hope this question will be tolerated. I was recently asked to help a friend with an assignment in the process of applying for a new job as data scientist. And ...
0
votes
0answers
27 views

Study resources for data manipulation interview questions

A colleague of mine is studying for tech roles, and they're asked to solve a consistent type of problem during the phone screenings: practicing manipulating data (sets, hash tables/dictionaries, ...
0
votes
2answers
158 views

Conditional Statement to update columns based on range

I have a dataframe called clients that has 5000+ rows with the following: ...
7
votes
3answers
150 views

Docker for data science

I recently started to read articles about Docker. To me, in data science, Docker is useful because: 1) You have a totally different environment, which protect you against libraries and dependencies ...
1
vote
0answers
15 views

Plotting ROC curve after re-sampling the data set

I am a beginner in Machine Learning. I am working on a project to compare the performance of three ML algorithms which are: LR, SVM and ANN on a dataset containing medical records. After exploring ...
0
votes
0answers
51 views

Self Union All in Tableau Public for Sankey Diagram

I'm following this 2017 youtube tutorial from-which I hope to learn Sankey diagrams. At 2:28 he instructs us to select Convert to Custom SQL, after having opened ...
2
votes
1answer
412 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
-1
votes
1answer
43 views

Inputting Data for Machine Learning in PyTorch

I have scoured the internet and documentation, but have not yet been able to find a simple explanation of how to load data and/or simply type data into pytorch. I am just learning this library, and I ...
0
votes
1answer
65 views

How to input a 3d model into ML algorithm?

I have a machine learning model that uses csv with measured data about buildings: width, length, height etc. I use it to predict some features and it works properly. I would like to drop csv with ...
0
votes
3answers
92 views

what are effects of working with categorical dataset

I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over ...
1
vote
1answer
45 views

Alternatives to Heat Maps for Communicating Daily Data

I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's ...
0
votes
0answers
23 views

Temperature in Farenheit : Interval or ratio data

According to the Level of Measurement classification, there are four types of data: nominal, ordinal, interval and ratio. The temperature in Fahrenheit and Celsius is often classified as interval ...
1
vote
2answers
17 views

Different types of analysis which can be done with Webdata of users

I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are: Product Usage details Geographic details Session data: Capture what all they did ...
-1
votes
1answer
31 views

Extracting information from online job postings

Okay, so I'm trying to build a data set about data science job openings. I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
2
votes
1answer
43 views

Data science degree or Computer Science degree? [closed]

I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for ...
0
votes
0answers
34 views

reverse engineer console game FPS sensitivity for replication

Would it be possible to reverse engineer a FPS games sensitivity/dead zone/acceleration curve/ and other data by recording the games screen and running a controller through a pc to record joystick ...
0
votes
0answers
10 views

Filter a signal at a frequency, which is proportional to another signal

I want to filter a signal at a frequency, which is proportional to a frequency of another signal. This other signal is perfectly measureable and changes its frequency from time to time, so this filter ...
2
votes
1answer
36 views

Can a Logsitic Regression model continue making predictions after removing predictions from the data set?

I have a logistic regression model that predicts churn (0 vs. 1). I was asked to use the model to predict on a historical group of non-churners, remove anyone who was marked as a churner, and then ...
2
votes
0answers
10k views

How to Process Large JSON Files with Python

I've been using Python for quite some time now, scripting command line tools, etc. I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files. ...
2
votes
2answers
44 views

Training vs test data set for supervised learning in real life scenario

In the tutorials, I have noticed only similar data has been used with models training and prediction. I was wondering how cases where you can't find training data that is similar to your final use ...