Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
1answer
27 views

how to know what my data are uninformative and that machine learning will not work with it?

following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I ...
2
votes
2answers
93 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
1
vote
0answers
14 views

How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?

I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine ...
0
votes
0answers
12 views

Is syslog/sysmon data considered 'raw data'

I apologize if this is considered a topic of 'discussion,' but I would hope it can be proven factually. Is sysmon data considered raw data? I understand that the data is formatted on output, but in my ...
0
votes
0answers
15 views

football score data, goal events

I am looking for event level football/soccer data. I want to know when a goal happened (minute) - other events are not important. Do you know where I can get this data. Preferably without much ...
1
vote
1answer
63 views

generate a report in R where the value in each Column is to be derived from different tables in sybase databasea

I work in a stress testing team. my boss has asked me to generate an interactive report every day using different tables in a Sybase database. I know the table structure and sql queries, but can't get ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...
0
votes
0answers
10 views

Rank impact of individual metrics on overall sales performance

I’m interested in learning about different methods to analyse sales data - at both store level and corporate. Essentially I have lots of performance metrics (~20) that break down the overall sales ...
1
vote
0answers
13 views
0
votes
0answers
30 views

How can I compare continuous and non-binary categorical data in Excel?

I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
0
votes
2answers
46 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
22 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
1
vote
2answers
38 views

Cross validation vs leave one out

I have found the following definitions, but I don't really see the difference. cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (...
1
vote
0answers
38 views

How to avoid bias when optimizing time-series model?

I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
0
votes
0answers
136 views

Anonymizing the CSV data using Python Faker

To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations. Is it possible to anonymize the ...
1
vote
2answers
670 views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
23 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views

Space and time fourier transform of data

I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
26 views

Free API for historical weather US data?

I am trying to retrieve a free R-Python API that provides historical weather data in US. In fact wunderground API is no longer available. Any suggestion?
11
votes
2answers
9k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
2
votes
1answer
52 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
7
votes
2answers
1k views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
0
votes
0answers
19 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
35 views

How to sort of “average” data without smoothing?

I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time. The problem ...
2
votes
1answer
75 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
2
votes
1answer
27 views

Is there a way to aggregate frequency distribution for many runs?

In the following plot I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
0
votes
0answers
570 views

Connecting to IEX with Pandas Datareader

My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
1
vote
1answer
27 views

Best file format for transfer of EHR data

I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
0
votes
0answers
13 views

Most reliable source for Literacy data

I'm searching for the most reliable source for granular American literacy population data. I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
1
vote
0answers
18 views

How can I measure data anonymity when present summary statistics

I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below: ...
0
votes
0answers
10 views

Reflections about types of bias in comparative analysis

I am working on an analysis on proposal rejections across two different periods in time and I was wondering what reflections I would have to make about the risk of bias. The reason for this is ...
1
vote
2answers
661 views

Class Imbalance and Cost-Sensitive Learning XGBoost

I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is ...
1
vote
1answer
154 views

can I use z-score normalization even if it doesn't make sense for my data to be negative?

I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
0answers
14 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
2
votes
2answers
861 views

Multi-touch Attribution Model

I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling ...
0
votes
0answers
46 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
0answers
29 views

Dataset creation software

Hello I am looking for a dataset for printed english characters but I am not able to find anything concrete. Is there any software which can generate multiple images from one image. I can build my ...
0
votes
1answer
12 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
2answers
45 views

Enormous dataset, how to proceed?

So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level ...
1
vote
1answer
59 views

Computing Jaccard Similarity between two documents

Data Mining: Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) = D1 = the quick brown fox jumps over the lazy dog D2 = jeff typed the quick brown dog jumps over the lazy fox ...
1
vote
2answers
466 views

What is the difference between a data-driven model and an empirical model?

Are they the same? Empirical models, per Wikipedia, are any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system ...
-1
votes
1answer
42 views

How to interact two variables in python?

I have about 8 features as my predictors in a logistic model I am trying to fit in python. One of the features is TotalAward (Financial Aid) and the second is NEED. I am attempting to predict the ...
1
vote
1answer
49 views

Growth function of a 6-dimensional linear classifier

In our course, we are dealing with a d-dimensional classification problem ($\chi = \mathbb{R}^{d}$ as our input space, and $y = \{-1,+1\}$). Our hypothesis class $H$ consists of all hypotheses of the ...
0
votes
0answers
29 views

real-valued data normalization for regression

We already know that non-real-valued fields of a data set to be modeled by neural network or some other machine learning approach must be transformed into real value. Some categorical or nominal type ...
1
vote
1answer
42 views

Meaningful and Non-Meaningful Data?

I can understand what meaningful data is like its important information that can be used to evaluate something but I don't get what non-meaningful data is? Is it less important data?
0
votes
0answers
26 views

Averaging over Irregular Intervals

Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation? Specifically, suppose we are ...
2
votes
1answer
124 views

How to correct mislabeled data in dataset?

I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly ...
1
vote
1answer
91 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
1
vote
0answers
23 views

How to use SOFT dataset in Orange

I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?
1
vote
1answer
23 views

How has data generation evolved over the years (and centuries)?

I would like to know how data generation has evolved through the years. My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does ...

1
3 4
5
6 7
12