Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
640
questions
0
votes
0answers
8 views
Finding the wiggles pattern in the original dataset. (Wiggles appear after performing division by another dataset)
I have multiple measurements regarding scientific observations. The problem is that there is a subtle noise pattern caused by the instrument - the wiggles. These wiggles are invisible when looking at ...
0
votes
0answers
8 views
Data generation ( Low KMO index)
I tried to generate a dataset with a respect to a correlation matrix R using cholesky decomposition. The problem is that i always get an KMO index that is very low ( smaller than 0.5 ).
Is there any ...
1
vote
1answer
10 views
How do I assign specific values to categorical variables
I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
0
votes
1answer
14 views
Data Analytics how to read ECDF graph
Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be ...
1
vote
0answers
28 views
Is scraping job posts and gathering skills tags illegal? [closed]
I want to scrape a freelancer site (freelancer.com) and gather their public projects skill tags. Is this illegal to do? And is publishing it illegal?
0
votes
0answers
6 views
Database best-practice when sunsetting a feature
Say, your product team decided to sunset a feature of your product, and there are columns in your database related to that feature. Is it a good practice to keep the columns (so you don't lose this ...
0
votes
1answer
33 views
How to aggregate data inserted by users to avoid outliers?
I'm developing a new application based on machine learning. In this application users can insert new data to improve the prediction system.
As you may guess, users could insert data that doesn't make ...
-1
votes
0answers
22 views
Documentation For Data Science Graduation project [closed]
is there any suggestion for an excellent Documentation Template used to write an Data Science Graduation projects
Thanks alot
0
votes
1answer
11 views
how can I convert tensor shape to numpy array?
This is what my data looks like:
I want it to be numpy array, instead of a tensor, so that i can convert it to a dataframe. Is there a way to do so?
thanks in advance!
0
votes
2answers
65 views
+50
Public dataset for account to account payments
I'm looking for a dataset which contains account to account payments (bank transfers). Ideally, this dataset would contain labeled data for transactions or accounts known to be victims of phishing ...
0
votes
0answers
18 views
is padding input images better than resizing image?
I am working with images and was thinking of pre-training VGG19 and EffecientNETB0 model on my dataset. However, a query:-
Is ...
0
votes
0answers
22 views
How can I cluster 3-dimensional data?
For a project in my data mining class, I am to perform fuzzy-c means clustering on a data set, where each data point has 3 axes.I'm not exactly sure how I would do so, especially given the clustering ...
0
votes
1answer
46 views
Data Selection according to Feature Values
I am given a dataset consisting of 10 million molecules. Each row contains:
The average value predicted by an ensemble of regression models trained to predict a
certain property about chemicals (...
1
vote
0answers
9 views
0
votes
1answer
19 views
Why are ordered lists considered one dimensional?
At Uni I am currently taking data visualization as a course.
While reading thru the script I stumbled upon two examples for one dimensional data.
Example one:
Bus stops with their name and order (...
0
votes
0answers
11 views
How to organize data in workflows with user input between steps?
I am building a multistep workflow.
User provides input files input.csv, parameters_global.yml, ...
0
votes
0answers
19 views
Data visualization multiple choices multiple condition conditions
Say I have a element that base on a certain condition (1, 2, 3 or 4), it chooses a decision (A,B or C). What diagrams can be used to present this data if I have more than one thousand elements so that:...
0
votes
1answer
21 views
What kind of data analysis problem is this?
This might be a dumb question, but I am an absolute beginner.
What kind of data analysis problem is the following?
Is it an Exploratory Data Analysis problem/Data Mining problem/data analytics problem?...
2
votes
1answer
29 views
Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?
The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed
...
0
votes
0answers
13 views
Manually labeled data quality check methods?
In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation ...
0
votes
0answers
22 views
Load data from SQL for Machine Learning
I have an SQL DB which has millions of records(10 years of data) and wanted to implement some ML Models. Most of the courses/tutorials explains using a local file or downloaded CSV. But in my case I ...
0
votes
0answers
21 views
What should be the architecture design for the dataset to be used for a machine learning API
I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
1
vote
1answer
36 views
Hi, im currently working for a company that has some inventory control problems
First, I was asked by the manager to make a plot showing produced vs received items, its a multistage process so we are only in charge of one of the steps which is designing, I made the plot comparing ...
1
vote
0answers
6 views
Available data sets on basic income pilot studies? [closed]
I'm interested in exploring data relating to any UBI/basic income pilot studies that have been conducted but have been unable to find any so far?
thanks
0
votes
0answers
6 views
Question on What I would use for a Piece of Data
I have 3 categories that we can call A, B, and C. In almost every case we B/A = C. B and A are given in the Dataset and C is what I am find an trying to use elsewhere. The higher number C is the ...
0
votes
1answer
39 views
Which data science framework should I start learning? [closed]
I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of ...
0
votes
1answer
18 views
Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series')
Several terms are used to describe the structure of instances of a dataset. For example, an instance can be 'univariate' if it depends on only one variable or 'multivariate' if it depends on 2 or more ...
1
vote
1answer
52 views
Why does an imbalanced data set badly effect distance measures like Mahalanobis?
I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
0
votes
1answer
38 views
How to specify different columns stacked vertically within CSV using pandas?
I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
0
votes
0answers
9 views
How to Combine Oracle and Snowflake/Redshift data on the fly
Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift.
Most ...
1
vote
1answer
19 views
Where can I find a dataset that contains criminal case sentencing data? [closed]
I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as:
Type of crime
Defendant Age/Sex/Race
Plea
Verdict
Sentence
Is there a dataset ...
0
votes
0answers
23 views
Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?
I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
0
votes
1answer
25 views
Problem with binning
I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements.
My questions are:
Which type of ...
0
votes
0answers
17 views
1
vote
0answers
17 views
MCMC algorithm — understanding some paremeters
I am trying to understand an MCMC program. I manage to run it, but I am trying to understand the meaning of the some parameters in the analysis.
The code is something like this
...
0
votes
0answers
19 views
Download weather data from a http directory
This link contains weather data with a lot of folders and subfolders. I need to download as a whole without visiting every file inside the folders.
https://www.ncei.noaa.gov/data/global-forecast-...
0
votes
0answers
19 views
LSTM Train accuracy decreases in training time during the epoch
I'm training an LSTM model for sentiment analysis on a text corpus. There's a thing that I believe is not normal because I never have seen it in training my models. At the start of the epoch, the ...
1
vote
1answer
13 views
Structuring extensive medical histories + demographic information for prediciting future medical outcomes
I'm looking for advice structuring extensive medical histories for predicting future outcomes, specifically hospital admissions.
Let's say I want to predict the whether or not someone will be admitted ...
0
votes
1answer
17 views
How to calculate an average cliff date?
Please let me know if this question belongs elsewhere.
In my simple data set focused on sales pursuit opportunities, I have the following columns available.
Pursuit Name
Status(Open, Won, Lost)
Date ...
1
vote
0answers
40 views
why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies
Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only ...
0
votes
0answers
54 views
Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`
I want to train a Sequential Neural Net (NN) with Tensorflow.
...
0
votes
2answers
27 views
How match output (pred value) to input value
I'm working with data(with 4 columns which are p(product), M(name of the store)), I want predict the demand of store for that I sued SVR on the data by theses formulation:
...
0
votes
0answers
24 views
R: saving/printing all intermediate results
currently, we have developed a model at work and it is at the verification step. the goal is for another person to receive output from each and every step of calculation to verify.
The issue is that ...
-2
votes
3answers
51 views
How to decrease $R^2$ value and change it to positive value [closed]
I'm working on a data, and use regression , as you see bellow:
from sklearn.svm import SVR
regressor = SVR(kernel = 'linear')
regressor.fit(trainX,trainY)
above ...
0
votes
1answer
593 views
How to use inverse_transform in MinMaxScaler for pred answer in a matrix
I am working on a data, for preding output, I used SVR by bellow code:
...
0
votes
0answers
18 views
How to find the relationship between one dataset and two others?
From a STEM problem, I vary a variable x within a range and calculate quantities $U(x)$, $V(x)$ and $W(x)$. I want to figure out an analytical relationship between unknown $U(x)$ and the other two ...
0
votes
1answer
17 views
How to Impute Top-Censored Data
What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?
1
vote
0answers
9 views
Is there enough datapoints to make a reasonable predcitction?
I am planning to use this dataset (for a school project) in order to determine the most important features in predicting whether a student will receive a placement.
Further I would like to create a ...
3
votes
2answers
50 views
Is it possible to detect drift with real time predictions?
I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or ...
1
vote
0answers
11 views
Comments vs. Files (& how to import w/o packages) for Data Science [closed]
I am trying to settle on an efficient & pythonic workflow for developing scripts in data science work that are scalable and don't turn to spaghetti once things get complicated and parameters are ...
