Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
2answers
116 views

Date transformation for KNN

I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful ...
1
vote
1answer
151 views

Importing Excel format data into R/R Studio and using glmnet package?

I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
0
votes
1answer
15 views

Is it acceptable to make new independent variables out of old ones?

I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
0
votes
2answers
39 views

How to make a dataframe with lists or vectors as its elements

This is something I have been wondering for ages but I am never able to get an answer. I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
0
votes
0answers
27 views

Comparing the two feature sets

I am working on a classification of two feature sets derived from a dataset. We first obtain two feature matrices derived from two feature extraction methods. Now, I need to compare them. However, the ...
1
vote
1answer
50 views

Randomizing selection in R

I am trying to create a comparison group. So far this group contains 45 data points and I need to populate the remaining 55 (for a total of 100 data points). These remaining 55 need to be a ...
1
vote
2answers
39 views

What's my target variable?

I am beginner in data science. I have this "aids" dataset from "mdhglm" package in R. dataset = aids, info = Repatead Measures on AIDS Data ...
0
votes
1answer
10 views

Getting the earliest date (duplicates due to several call ids and agents)

I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates: ...
0
votes
1answer
572 views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
0
votes
0answers
28 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
2answers
2k views

What algorithms are good to predict next numbers?

Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are ...
0
votes
1answer
27 views

Is it appropriate to run correlation on continuous y value with discrete x value?

I am dealing with temperature data that are measured in different distances away from the forest edge (0m, 10m,30m,50m,80m) and I would like to find out whether the temperature will drop as the ...
0
votes
1answer
74 views

How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have ...
0
votes
0answers
15 views

Using Google Trends data for comparability across states/regions

I am interested in using Google Trends data in order to analyse some searches across say U.S. States. Say I wanted to compare search intensity for the keyword "Donald Trump" but across different ...
7
votes
2answers
111 views

How important is advanced SQL for data science?

Is advanced level SQL required to be competitive as a data scientist? Is it more important for a data analyst to be good in SQL? Is it enough to be able to extract data using simple SQL queries? I ...
0
votes
1answer
99 views

How to obtain and load a good initial data set for object localization?

I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data). What is a good initial data set to use? How can ...
1
vote
2answers
34 views

How to define tidy data if there is repeated measures?

Some time ago I read R for Data Sciece and there is the following definition of tidy data: There are three interrelated rules which make a dataset tidy: Each variable must have its own column. Each ...
0
votes
1answer
25 views

Standardizing features by one specific feature

I am working on a project with a dataset that looks something like the following: ...
1
vote
1answer
39 views

Logistic Regression: Is it viable to use data that is outdated?

TLDR: Want to predict who makes the playoffs (1,0), but there are more playoff spots now than there were in the past, is it okay to use that past data? I want to use binary logistic regression on MLB ...
0
votes
0answers
7 views

tensorflow feature columns

I am a beginner in TensorFlow. So while going through an article I came across tf.feature_columns being used. I understand the general use of it,but what I am confused about something.There were some ...
2
votes
1answer
22 views

When training NN, how do data loaders work on large datasets?

How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples ...
0
votes
2answers
26 views

CNN training on Different Datasets

I currently want to train a CNN but I have two small datasets that are slightly different because of the camera setup that captured the images. I'm interested in ultimately tuning the neural network ...
0
votes
1answer
33 views

Algorithm for user profiling without distinct profiles

I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
1
vote
0answers
14 views

Tableau: Dealing with Date values [closed]

Im a noob at tableau. I have a quick question I have data that doesn't explicitly list the date. (There is no column labelled "date" ) However there are columns with the following headers: "Income ...
0
votes
0answers
29 views

Machine learning model for record linkage of unstructured data

I have a list of string. Each string has position based distinct information. On the other hand, I have a totally different structure data ( say xml ). How do I match the string and the XML based on ...
0
votes
1answer
35 views

Building a content-based music recommendation system

I am trying to build a recommendation system in Python that recommends songs based on a playlist. What I have is two datasets: 1. One dataset consists of 350 songs from my playlist and 13 acoustic ...
6
votes
1answer
63 views

Un-learning a single training example from a trained model

I was going through the paper "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" by google on which suggests best practices for models in production. In a section ...
4
votes
1answer
600 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
0
votes
1answer
51 views

Append and replacing data in r

I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs. Here is my data ...
0
votes
0answers
17 views

How to use data provided in EEM20 forecasting competition?

I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...
2
votes
1answer
46 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
1answer
79 views

Convert categorical data in numeric preserve euclidean distance

I m looking how to preserve Euclidean distance with categorical attribute. Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "...
2
votes
1answer
63 views

Organizing complex analysis project

Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
0
votes
0answers
12 views

Converting elements in a list to the index value of an element in list of list

I have a list, l, where the list contains random integer numbers and there are 60000 elements in the list. I also have a list of lists, myListofLists, containing 60 lists and in total 2000 elements (...
1
vote
0answers
17 views

Fit model function out defined data range

I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
0
votes
0answers
8 views
0
votes
0answers
16 views

How can insert lots of data into MySQL database?

I have created a MySQL database with four tables, which are linked with each other (via references). I now want to insert lots of data from Excel into my database. Is there a possibility of inserting ...
0
votes
1answer
23 views

Call routing using AI

I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can ...
3
votes
1answer
41 views

Combining time-series data from different devices

I am using four different sensor models (four different devices) to gather some biometric data(eg. heart rate) for the same user, for the same time period. In an ideal scenario, all the devices are ...
4
votes
2answers
141 views

What kind of research can be done with genomic data?

It is well known that science has given us large amounts of free accessible data, such as https://www.1000genomes.org and https://www.ncbi.nlm.nih.gov/genbank. How can we play around with the data and ...
0
votes
0answers
34 views

How can I define a function in r to flag holidays by week?

I have a field of dates that mark the end of weeks. I am trying to create a new field using a function that flags (1 or 0) whether that week included any of 6 holidays specified using the timeDate ...
3
votes
5answers
238 views

How can I handle a column with list data?

I have a dataset which I processed and created six features: ['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList', 'totalProducts'] And the ...
0
votes
0answers
17 views

GEO Choropleth not found

I'm Trying create, from Geo add-on, a Chropleth map in Orange version 3.24.1 for windows . But, this widget just don't appear... some tip?
0
votes
0answers
23 views

Forecasting binary time series

I am working on the next event occurrence prediction task and the data is binary time series with 1 if the event occured and 0 if not. I want to predict whether the event will occur or not on the day ...
0
votes
0answers
14 views

Modeling of changing data for analysia using cube.js

I want to store users object and store all historical changes of their addresses so that I can calculate the count of users in each state at specific date or date range using cube.js Can someone ...
0
votes
1answer
52 views

Prepare data for an LSTM

-I want to make a python program using the LSTM model to predict an output value that is 1 or 0. -My data is stored in a .csv file of the form: (Example of line) Date time ...
0
votes
0answers
51 views

np.loadtxt function showing error, could not convert string to float

This is my data file : I am trying to read this file with loadtxt() and the geometry points get loaded into one NumPy array, so I separate the data into two arrays:...
1
vote
2answers
44 views

1 2
3
4 5
12