Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
2
votes
1answer
35 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
0answers
10 views

access to the didi ditech2016 dataset

a bit off-topic, but I'm trying to locate a famous dataset used in the DiDi DiTech2016 supply-demand challenge. All sources I found point to a single url: http://research.xiaojukeji.com/competition/...
2
votes
2answers
203 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
1
vote
1answer
57 views

What's the best way to train a NER model?

I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used… To do that I want to ...
2
votes
1answer
22 views

Are parquet files compressed? [closed]

Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
1
vote
0answers
18 views

PointCloud value prediction - regression

I have point clouds for 400 objects. I have also some other features of these objects. For example their weights. How I can now train model to predict the weight based on the shape(point cloud)? I ...
1
vote
0answers
38 views

Is it possible retrive hourly data from daily weather data?

I'm a beginner, so sorry if my question could be basic. Reading on internet I've found example written in python that makes the reverse of my questions (covert from daily to monthly). My problem is ...
1
vote
2answers
45 views

Clearest way to visualise temporal data?

I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
5
votes
3answers
444 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
0
votes
1answer
21 views

how to know what my data are uninformative and that machine learning will not work with it?

following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I ...
2
votes
2answers
81 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
0
votes
1answer
31 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
1
vote
0answers
12 views

How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?

I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine ...
0
votes
0answers
12 views

Is syslog/sysmon data considered 'raw data'

I apologize if this is considered a topic of 'discussion,' but I would hope it can be proven factually. Is sysmon data considered raw data? I understand that the data is formatted on output, but in my ...
0
votes
1answer
38 views

Software for automated database processing

I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this. I have a dataset, for example: (brand-id, brand-name, product-class-name;) ...
0
votes
2answers
2k views

How to plot using facet_wrap, over multiple pages as a .pdf files in r cran

I am using ggplot, to compare 114 unique studies for a particular variable I'm interested in. This is what I have used. ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = ...
0
votes
0answers
12 views

football score data, goal events

I am looking for event level football/soccer data. I want to know when a goal happened (minute) - other events are not important. Do you know where I can get this data. Preferably without much ...
1
vote
1answer
59 views

generate a report in R where the value in each Column is to be derived from different tables in sybase databasea

I work in a stress testing team. my boss has asked me to generate an interactive report every day using different tables in a Sybase database. I know the table structure and sql queries, but can't get ...
2
votes
3answers
66 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
1answer
23 views
0
votes
0answers
10 views

Rank impact of individual metrics on overall sales performance

I’m interested in learning about different methods to analyse sales data - at both store level and corporate. Essentially I have lots of performance metrics (~20) that break down the overall sales ...
0
votes
2answers
76 views

Dataset Merging

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
2answers
1k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
0answers
12 views
17
votes
1answer
30k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
0
votes
2answers
39 views
2
votes
1answer
48 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
0
votes
1answer
286 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
0answers
17 views

How can I compare continuous and non-binary categorical data in Excel?

I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
0
votes
2answers
31 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
22 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
0
votes
2answers
35 views

Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

I have several marketing files that I need to download on a weekly basis. Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
1
vote
2answers
31 views

Cross validation vs leave one out

I have found the following definitions, but I don't really see the difference. cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (...
1
vote
1answer
340 views

pd.qcut bins error!

Hi Data Science community! I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error. ...
1
vote
0answers
31 views

How to avoid bias when optimizing time-series model?

I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
0
votes
1answer
68 views

PCA in visual Analytics

I m studying visual analytics and i have a theoretical question about this topic. My professor introduced this schema in him slide For connect data to visualisation. Some topic is very easy to ...
0
votes
0answers
44 views

Anonymizing the CSV data using Python Faker

To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations. Is it possible to anonymize the ...
0
votes
0answers
15 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
2
votes
1answer
567 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
0
votes
1answer
76 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
1
vote
2answers
99 views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
22 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views

Space and time fourier transform of data

I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
25 views

Free API for historical weather US data?

I am trying to retrieve a free R-Python API that provides historical weather data in US. In fact wunderground API is no longer available. Any suggestion?
11
votes
2answers
8k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
2
votes
1answer
49 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
1
vote
1answer
33 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below. <...
6
votes
2answers
463 views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
0
votes
0answers
16 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
32 views

How to sort of “average” data without smoothing?

I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time. The problem ...

1 2 3 4 5 10