Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
696
questions
-1
votes
0answers
5 views
Web scraping ratings and review content of fast and furious 9
I am trying to scrap Fast and Furious 9 reviews and rating on imdb but I am stuck on getting the rating
...
1
vote
0answers
16 views
How to handle insufficient information about the collected data in the dataset?
I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures.
...
2
votes
0answers
24 views
What are some real world examples of a 2d data set where agglomerative methods work and K means doesnt? (for two real groups of data)
What is 2D dataset with two real groups of data where an agglomerative clustering method will be able to cluster the data correctly, and K-means will not?
I thought about a data set that would look ...
2
votes
1answer
26 views
Data Art/ Data Visualization Art/ Information Art
Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
1
vote
2answers
37 views
data preprocessing: missing labels
I have a classification task to predict {0,1} labels, but in my dataset I only have data with 1 labels. Class indicates that the buyer has made a purchase of some product.
Features: user id, product ...
0
votes
0answers
31 views
how to scrape images from facebook group
I want to scrape images from several Facebook groups some of them are public and some or not
I am new to web scraping but I tried to look for solutions with selenium or BeautifulSoup or scrappy but I ...
1
vote
0answers
16 views
Should I reshuffle the training set when benchmarking neural networks?
I'm trying to set up a fair benchmarking between various RNN models, where each of them is trained until convergence with a fixed random seed. Because the task is very costly, I am only able to run ...
0
votes
0answers
6 views
Can we consider Meta-features of a datasets as its embeddings?
While reading some works on meta-learning. I had this doubt.
Can we consider meta-features of a dataset as it's embedding ?
Given the meta-feature is a lower dimensional representation which also try ...
0
votes
0answers
22 views
How do I create a parameter for bining a Value for high's and low's
Create a new parameter 'kgl' for discretizing the 'kernel_groove_length' feature, which can take the values 'high' and 'low'
kernel_groove_length is the data of 3 different types of seeds which are ...
1
vote
0answers
20 views
How to Scrape Website Usage/User Data for a Website that Does Not Have an API [closed]
I am an aspiring Data Analyst and I was wondering how do I web scrape website usage/user data such as:
total people who visited that website,
total visitors per page,
how long did each visitor stay ...
2
votes
1answer
81 views
Loading medical imaging data from multiple folders
I have a fairly basic mathematical and implementational understanding of ML algorithms and CNNs, and I am trying to think of an approach for this task: https://www.kaggle.com/c/rsna-miccai-brain-tumor-...
0
votes
0answers
9 views
Can I select both MSc in Data Science and MSc in Artificial Intelligence programs at the same time while applying for Chevening Scholarship?
I know that Data Science is a subset of Artificial Intelligence. So, the two Masters programs are pretty similar, as both of them teach Machine Learning. However, Artificial Intelligence program ...
3
votes
0answers
21 views
What are descriptors
I am very new to data science, and r programming specifically. Could some explain what a descriptor in data science is? In particular, what are the differences between a 2d and 3d descriptor?
Thank ...
2
votes
1answer
40 views
How create a representative small subset from a huge dataset, for local development?
I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
0
votes
0answers
23 views
How to mathematically quantify the quality of a corpus?
I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
19 views
ML for data processing. What are the options?
Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity.
This entails that in many cases ...
0
votes
0answers
13 views
Best way to store large amounts of time series data? Relational database (SQL) or NoSQL route. Also additional python/pandas question inside
so I'm working on a project and I'm sort of stuck as to how to store my data. I have a concept I want to propose but am unsure whether it is possible, if it is not I would appreciate any help ...
0
votes
0answers
9 views
Find the formula for these data series to create model
Thank you for taking the time.
I'm working on creating a model my team can use to see the maximum number of miles that can be added to a delivery trip in a measured area based on the number of stops (...
0
votes
0answers
11 views
How to find the lag that shows highest correlation between two time series variables?
I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
1
vote
0answers
10 views
Is this set of vectors representing direction in a sequence trainable across LSTM? if not, what are the drawbacks?
I am starting my Neural Network Project for the first time, I have a lot of expectations:
The main idea is primarily analyzing the pattern of my writings or drawings when using the mouse given a ...
0
votes
0answers
5 views
How to model multinomail logistic regression to get desired outcome?
I want to run multinomial logistic regression in SPSS.
Dependent variable:
Code=0 Cognitively normal(CN)
Code=1 Mild cognitive normal(MCI)
Code=3 Alzheimer’s(AD)
Independent variable:
Genotype.1=0
...
1
vote
0answers
19 views
Multivariate regression - not enough data?
I have a table with data about 10 agriculture parcels. Each parcel has data in time regard the number of nutrients each parcel has received in each day and in the end I have the total number of ...
0
votes
1answer
13 views
Encoding when different number of records for each month-One hot or different type?
I am working on a dataset which is pretty small: 1169 records.
There is a column called month and it takes the values 'Jan','Feb', or 'March'.
The number of records for each month is different. I have ...
0
votes
1answer
37 views
Preprocessing , EDA , and Feature Engineering
What is the difference between EDA, Feature Engineering, and Preprocessing?
The main purpose is to make the raw data suitable for modeling. In EDA, we are cleaning the data and so does the ...
0
votes
0answers
12 views
Can you POST to a rest API from databricks?
I am looking to post a JSON file to a rest api via databricks. I am having trouble trying to figure out how to do this.
0
votes
1answer
23 views
Creating a complex featureset for regression modeling
I am currently working a on project that requires me to convert all of the categorical variables to continuous (or binary) variables to build a regression model. The problem is that I have more than ...
0
votes
1answer
19 views
Machine Learning A-Z: Help connecting with datasets
I'm in Machine Learning A-Z lecture 19 where we access our first dataset. I've connected to the Google Colab folder in my gmail account, but when I try to access the the file I get the below error ...
0
votes
0answers
18 views
How to generate a positive skewed array?
As the title suggests, I would like to specify the task that I really need help with:
Given three possible distributions of an array:
I know that given a skewed dataset (the left and the right), then ...
1
vote
0answers
13 views
UI-based Tool for Qualitative Evaluation of Data Quality
Dear DS StackExchange community,
I'm currently searching the interwebs for a (near-)ready-to-use solution, to perform a qualitative evaluation of extracted features from video data. In my head the ...
0
votes
0answers
19 views
What analytics model/classifier should be used to predict price?
I've go dataset with more than 10 features - football skills describing players. There is also price value in each row.
I would like to predict price by specifying ...
0
votes
1answer
30 views
How to interpret 3 outputs of Precision?
Say im running a classification Machine learning algorithm,
of 2 classes 0 & 1.
A 0 label is detecting a visitor/row did not Convert. while 1 label is detecting a visitor/row did Convert.
When the ...
0
votes
0answers
97 views
Why am I getting a different answer in Principal Component Analysis dimensional reduction?
Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8).
Compute the principal component using PCA Algorithm.
Use PCA Algorithm to transform the pattern (2, 1) ...
1
vote
2answers
21 views
What are some examples of ordered data?
Tan,Steinbech,Kumar's book says
For some types of data, attributes have relationships that involve order in time or space.
I want some real life examples of such data, can you provide me. I couldn't ...
0
votes
0answers
7 views
How to clean/analyze relative time series data
I'm relatively new to data science, and I decided to stretch a bit and try a project with time series data. I downloaded the Human Activity Recognition from Continuous Ambient Sensor Data Data Set, ...
0
votes
0answers
12 views
Dictionary of life sciences or medical terminologies
I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
1
vote
0answers
14 views
Clustering dataset with and without estimating means (no EM algorithm)
Given a dataset $D$ of the form
$$
D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n)
$$
sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
0
votes
1answer
13 views
How do you normalise the train+validation sets together?
This question is somewhat related to: Is it correct to join training and validation set before inferring on test-set?
As far as I understand, normalisation in general is done in the following way:
...
0
votes
1answer
45 views
Is it a good idea to use parallel coordinates for visualising outliers? [closed]
I tried using parallel coordinates to visualize outliers. Is it fundamentally correct?
0
votes
0answers
25 views
Training data for image digit recognition
I am creating a digit image recognition algorithm. For this I would like to collect sample data to train the model. Does python have any inbuilt libraries for this? Where could I get good training ...
0
votes
0answers
11 views
Most Hot Freelance Data Science Project Niches
I almost finish my final year in my information system major for computer science's bachelor degree, i also take big data and machine learning as my specialization. And then while i finding and apply ...
0
votes
2answers
30 views
Assign value of existing column to new columns in pandas based on conditions [closed]
I am trying to create a new column in pandas data frame based on existing columns.
column1
column2
column3
y1
y2
y3
100
200
300
2020
2021
2022
100
200
300
2021
2022
2023
100
200
300
2019
2020
2021
...
0
votes
0answers
23 views
How to do exploratory data analysis on a database
In all courses, the focus is mainly on building models from an already clean dataset that is in a csv file.
If I have a database, Postgres, with a table that has ~10 million rows, what is the best way ...
0
votes
3answers
213 views
Convert .data file to .csv
I'm using a data called 'adults.data', I need to work with that data as a '.csv' file. Can anyone guide me on how can I change the format?
I tried opening the file in excel and then save it as csv, ...
1
vote
1answer
12 views
How to deal with mismatch of timesteps in target variable and features in forecasting problem?
Background Info:
I am working with some climate data where I want to predict crop yields with my dataset containing climate- and satellite-derived features.
This is a time series regression ...
2
votes
1answer
32 views
0
votes
0answers
10 views
Why do images in Open CV template matching get matched to the templates more than once?
The following code reads the set of input tiff files under tiffile and template tiff files under files and performs template matching on them. After the image gets matched, opencv contour extracts the ...
2
votes
1answer
32 views
Visualization suggestion:-
I have a data frame like below:-
Here I have 194 countries and the columns are fan_out values which is in percent of the total population.
Like for country AD, the total fan_out value is 2.24 -06 % of ...
2
votes
2answers
26 views
How to input a list into my model and not have it care about order
I'm trying to predict a list of numbers, e.g:
[23,55,198,200,64]
The data I have includes multiple things, along with:
The numbers from the previous run (These ...
0
votes
0answers
16 views
When training a cGAN on a X (B/W image) to Y (RGB image) "paired" dataset, how much will slight differences in the images matter in terms of results?
Right now I am working on a colorization project with GANs, and had originally settled on using a CycleGAN because I considered structuring my dataset in an unpaired manner. I've since been able to ...
0
votes
1answer
48 views
Does the abstraction of a class affects the performance of neural networks?
For example, if I have 3 audio classes including
Ambulance Siren
Police Car Siren
Firetruck Siren
assuming these 3 classes could be distinguished by humans. If I just want the model to classify all ...