Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
711
questions
0
votes
0answers
2 views
Sample covariance between a quantitative variable and a binary encoded variable
I have the following question: What does the sample covariance between a quantitative variable and a binary encoded variable tells us? And would it make sense to include the binary variable in a ...
1
vote
0answers
15 views
Tokenizer returning incorrect values and losing a lot of data
(cross posted from main stackoverflow) This is a weird situation so I hope I can explain it correctly. My partner and I are working on a ML project where we create a model that predicts whether a ...
1
vote
1answer
17 views
Defining a function in Pandas [closed]
Hello Data Scientists,
Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things.
I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0answers
13 views
Why KNN output predicts all zeros rather one-hot label?
I have trained in supervised way several ML algorithms such as GNB,SVM, KNN. I have multi-class classification model (not multi-label). The input format has ~22 features and the output is one-hot ...
0
votes
0answers
8 views
Visualization of multidimensional trajectories data stored as an array
I have 14-dimensional trajectories data in such form (an output of data.shape):
(10000, 200, 14)
and it looks like this:
...
0
votes
0answers
29 views
Is it possible to derive anything useful from this piece of data?
Let's say you have online Profile A. Profile A is present on 3 websites: X, Y, Z.
...
0
votes
0answers
18 views
How do you do data management?
I have about a few million records (small CSV / JSON file) from different sources, with about 50k added everyday. All on my local host.
Until now, I have been using simple file structure to manage ...
0
votes
0answers
27 views
Identify the missing value in a table
How to compute the missing values x in the below table. which formula can be used to obtain the values of x. The values in the last row and last column are the sums of the rows and columns. What is ...
0
votes
0answers
10 views
Perform PCA on images
I'm actually studying how to perform a PCA-based image compression. In Hernandez & Mendez's Application of Principal Component Analysis to Image Compression, an approach to compute the Principal ...
2
votes
1answer
25 views
Best way to preprocess data
I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well:
...
0
votes
0answers
12 views
High dimensional data: Tensorboard Alternatives
Tensorboars has a nice tool for the Visualisation of the high dimensional data. Is there any other libraries that allow to do fast clustering and Visualisation without thr need to install PyTorch or ...
-1
votes
0answers
11 views
Accessing the Same Blob From Azure ML and Databricks
I am a bit new in this field so I thought I might ask the experts. Quite simply, is it possible to save to a blob storage from Azure Databricks and then read from it in Azure ML?
I have a project ...
1
vote
1answer
15 views
data.iterrows() + plt.imread() alternative, really exhausting
Currently, I'm loading my images for the training model with TensorFlow, like this:
...
0
votes
0answers
16 views
Google analytics custom dimensions into data studio
I'm looking at tracking custom dimensions that have been set up in google analytics within data studio.
I can find the custom dimensions that in analytics but I cannot find how to get them in data ...
0
votes
0answers
30 views
Data science/Dataframe question concerning an approach to a problem
I want to answer which kind of school type (mixed gender, only for boys or only for girls) is more efficient in training its students based on the result of a normalized exam score. And my model ...
1
vote
0answers
26 views
Can dummy variables be treated as continuous in statistical models? [closed]
Can dummy variables be treated as continuous in statistical models?
2
votes
1answer
14 views
cbind doesn't attribute the name of the column
I'm trying to create a new data frame that contains the row names and the first column of an existing data :
i tried this
...
1
vote
1answer
23 views
Removing outliers from a multi-dimensional dataset & Data augmentation
Removing the outliers of a single-dimensional data can be easily done by removing the points that are outside of the IQR range. But how should the process of detecting and removing outliers be done if ...
0
votes
1answer
30 views
Deleting a Column in a csv or Excel file using Pandas
I am trying to delete a column from my csv file (column 'A' called "Film Number") but have tried numerous variations of code and while it deletes the column in the dataFrame it doesn't do so ...
1
vote
0answers
22 views
How to handle insufficient information about the collected data in the dataset?
I am trying to create a ML model to predict foodborne diseases. I found online a few datasets with the informations I need. The problem is that none of them explains how they got the medical measures.
...
2
votes
0answers
30 views
What are some real world examples of a 2d data set where agglomerative methods work and K means doesnt? (for two real groups of data)
What is 2D dataset with two real groups of data where an agglomerative clustering method will be able to cluster the data correctly, and K-means will not?
I thought about a data set that would look ...
2
votes
1answer
31 views
Data Art/ Data Visualization Art/ Information Art
Few days ago, I learned about data art/ data visualization art/ information art. I think I have interest in it. I want to see how I can use my data science skills in this area. However, I don't know ...
1
vote
2answers
59 views
data preprocessing: missing labels
I have a classification task to predict {0,1} labels, but in my dataset I only have data with 1 labels. Class indicates that the buyer has made a purchase of some product.
Features: user id, product ...
0
votes
0answers
37 views
how to scrape images from facebook group
I want to scrape images from several Facebook groups some of them are public and some or not
I am new to web scraping but I tried to look for solutions with selenium or BeautifulSoup or scrappy but I ...
1
vote
0answers
19 views
Should I reshuffle the training set when benchmarking neural networks?
I'm trying to set up a fair benchmarking between various RNN models, where each of them is trained until convergence with a fixed random seed. Because the task is very costly, I am only able to run ...
0
votes
0answers
6 views
Can we consider Meta-features of a datasets as its embeddings?
While reading some works on meta-learning. I had this doubt.
Can we consider meta-features of a dataset as it's embedding ?
Given the meta-feature is a lower dimensional representation which also try ...
1
vote
0answers
20 views
How to Scrape Website Usage/User Data for a Website that Does Not Have an API [closed]
I am an aspiring Data Analyst and I was wondering how do I web scrape website usage/user data such as:
total people who visited that website,
total visitors per page,
how long did each visitor stay ...
2
votes
1answer
84 views
Loading medical imaging data from multiple folders
I have a fairly basic mathematical and implementational understanding of ML algorithms and CNNs, and I am trying to think of an approach for this task: https://www.kaggle.com/c/rsna-miccai-brain-tumor-...
0
votes
0answers
9 views
Can I select both MSc in Data Science and MSc in Artificial Intelligence programs at the same time while applying for Chevening Scholarship?
I know that Data Science is a subset of Artificial Intelligence. So, the two Masters programs are pretty similar, as both of them teach Machine Learning. However, Artificial Intelligence program ...
3
votes
0answers
21 views
What are descriptors
I am very new to data science, and r programming specifically. Could some explain what a descriptor in data science is? In particular, what are the differences between a 2d and 3d descriptor?
Thank ...
2
votes
1answer
41 views
How create a representative small subset from a huge dataset, for local development?
I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
0
votes
0answers
23 views
How to mathematically quantify the quality of a corpus?
I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
19 views
ML for data processing. What are the options?
Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity.
This entails that in many cases ...
0
votes
0answers
16 views
Best way to store large amounts of time series data? Relational database (SQL) or NoSQL route. Also additional python/pandas question inside
so I'm working on a project and I'm sort of stuck as to how to store my data. I have a concept I want to propose but am unsure whether it is possible, if it is not I would appreciate any help ...
0
votes
0answers
9 views
Find the formula for these data series to create model
Thank you for taking the time.
I'm working on creating a model my team can use to see the maximum number of miles that can be added to a delivery trip in a measured area based on the number of stops (...
0
votes
0answers
12 views
How to find the lag that shows highest correlation between two time series variables?
I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
1
vote
0answers
10 views
Is this set of vectors representing direction in a sequence trainable across LSTM? if not, what are the drawbacks?
I am starting my Neural Network Project for the first time, I have a lot of expectations:
The main idea is primarily analyzing the pattern of my writings or drawings when using the mouse given a ...
0
votes
0answers
6 views
How to model multinomail logistic regression to get desired outcome?
I want to run multinomial logistic regression in SPSS.
Dependent variable:
Code=0 Cognitively normal(CN)
Code=1 Mild cognitive normal(MCI)
Code=3 Alzheimer’s(AD)
Independent variable:
Genotype.1=0
...
1
vote
0answers
19 views
Multivariate regression - not enough data?
I have a table with data about 10 agriculture parcels. Each parcel has data in time regard the number of nutrients each parcel has received in each day and in the end I have the total number of ...
0
votes
1answer
14 views
Encoding when different number of records for each month-One hot or different type?
I am working on a dataset which is pretty small: 1169 records.
There is a column called month and it takes the values 'Jan','Feb', or 'March'.
The number of records for each month is different. I have ...
0
votes
1answer
45 views
Preprocessing , EDA , and Feature Engineering
What is the difference between EDA, Feature Engineering, and Preprocessing?
The main purpose is to make the raw data suitable for modeling. In EDA, we are cleaning the data and so does the ...
0
votes
0answers
13 views
Can you POST to a rest API from databricks?
I am looking to post a JSON file to a rest api via databricks. I am having trouble trying to figure out how to do this.
0
votes
1answer
23 views
Creating a complex featureset for regression modeling
I am currently working a on project that requires me to convert all of the categorical variables to continuous (or binary) variables to build a regression model. The problem is that I have more than ...
0
votes
1answer
23 views
Machine Learning A-Z: Help connecting with datasets
I'm in Machine Learning A-Z lecture 19 where we access our first dataset. I've connected to the Google Colab folder in my gmail account, but when I try to access the the file I get the below error ...
0
votes
0answers
21 views
How to generate a positive skewed array?
As the title suggests, I would like to specify the task that I really need help with:
Given three possible distributions of an array:
I know that given a skewed dataset (the left and the right), then ...
1
vote
0answers
13 views
UI-based Tool for Qualitative Evaluation of Data Quality
Dear DS StackExchange community,
I'm currently searching the interwebs for a (near-)ready-to-use solution, to perform a qualitative evaluation of extracted features from video data. In my head the ...
0
votes
0answers
19 views
What analytics model/classifier should be used to predict price?
I've go dataset with more than 10 features - football skills describing players. There is also price value in each row.
I would like to predict price by specifying ...
0
votes
1answer
30 views
How to interpret 3 outputs of Precision?
Say im running a classification Machine learning algorithm,
of 2 classes 0 & 1.
A 0 label is detecting a visitor/row did not Convert. while 1 label is detecting a visitor/row did Convert.
When the ...
0
votes
0answers
208 views
Why am I getting a different answer in Principal Component Analysis dimensional reduction?
Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8).
Compute the principal component using PCA Algorithm.
Use PCA Algorithm to transform the pattern (2, 1) ...
1
vote
2answers
22 views
What are some examples of ordered data?
Tan,Steinbech,Kumar's book says
For some types of data, attributes have relationships that involve order in time or space.
I want some real life examples of such data, can you provide me. I couldn't ...