Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
511
questions
0
votes
0answers
18 views
How to take certain percent of weights from certain featurs in machine learning model?
I am trying to build a ML model that predicts the winner, 2nd, and 3rd place. The dataset has 15 features. However, I would like to have 80% influence of 9 features and the 20% from the rest of the ...
-1
votes
0answers
20 views
How can practically become data scientist [closed]
I am in the IT fields for 15 years and want to get a step change to become data scientist. I am taking the dataquest course online but i am still struggle will people consider me to be data scientist. ...
1
vote
2answers
21 views
How to define tidy data if there is repeated measures?
Some time ago I read R for Data Sciece and there is the following definition of tidy data:
There are three interrelated rules which make a dataset tidy:
Each variable must have its own column.
...
0
votes
0answers
10 views
0
votes
0answers
13 views
np.loadtxt function showing error, could not convert string to float
This is my data file :
I am trying to read this file with loadtxt() and the geometry points get loaded into one NumPy array, so I separate the data into two arrays:...
0
votes
1answer
17 views
Prepare data for an LSTM
-I want to make a python program using the LSTM model to predict an output value that is 1 or 0.
-My data is stored in a .csv file of the form: (Example of line)
Date time ...
0
votes
1answer
23 views
Is it a Problem if Training Data and Evaluation Data are Very Similar?
I'm working on a group project where we have decided to use this Microsoft WIP repository as our starting point. It's a project which compares different frameworks and models and their ability to do ...
0
votes
1answer
16 views
Standardizing features by one specific feature
I am working on a project with a dataset that looks something like the following:
...
0
votes
0answers
17 views
Best practice - memory management when data wrangling
I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:
...
-1
votes
0answers
9 views
Is there books for for efficient data management? [closed]
This is a general question regarding how to manage a large dataset in principle. I am familiar with my database (I primarily work with SQL and MongoDB) thus I am looking for principles, not technical ...
-1
votes
0answers
7 views
How to formulate my data for an LSTM, classification problem?
I have several input data that I have ordered chronologically (temperature, wind, etc.) and which cause me to output the output value (1or0).
I have a csv file which contains chronologically ordered ...
0
votes
0answers
8 views
Are there any object detection datasets with the same format as PASCAL VOC?
I was wondering if there are any object detection datasets that have the same format as the PASCAL VOC dataset. Furthermore I would be specifically looking for satellite imagery datasets in that ...
1
vote
0answers
16 views
Find the mode value and frequency in R
I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
0
votes
0answers
11 views
Guidance on sequencing the data science professional courses
I am planning to start data science online professional courses at Harvard University, but I don't which course should I begin with . I request for help in sequencing these courses below so that I can ...
0
votes
2answers
37 views
Data Cleaning, refining etc
I have data as below:
Carpanter
Carepnter
Carpentor
Labourer
Labor
Labour
Housewife
House Wife
housewife.
I want to clean data and rectify the spelling mistakes ...
0
votes
0answers
10 views
What is the state of the art/research metric to compare ellipses but jaccard coefficient?
Im looking for the, if there is one, metric to compare ellipses with each other. Last time a had a similar dataset (malaria cells, now its pupiles) i used jaccard coefficient but that was more because ...
0
votes
1answer
14 views
License Plate OCR
Are there any pretrained models or annotated data available for license plate detection?
I've seen some sets but most of them are little in size. I'm more interested in prod-like quality.
Are there ...
0
votes
1answer
15 views
Call routing using AI
I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can ...
1
vote
1answer
28 views
Why there is very large difference between cross validation scores?
I have a very simple regression model and I am doing the cross validation. When cv=10 the highest score i got is 60.3 and lowest is -9.7 which is useless.
Average will be 30.
No of row data set= ...
0
votes
1answer
23 views
Dimensionality reduction without select components
I would like to use dimensionality reduction algorithm in my pipeline.
I have 2k features and I'm using xgboost.
My model is rebuilding each day (there are new records that should be involve to ...
0
votes
1answer
15 views
Why normalize function has a different result on a matrix vs single value?
I have a matrix like:
B=[ 1.5035; 1.5728; 1.6485; 1.5369; 1.5467; 1.572; 1.5374; 1.787; 1.5825; 1.6905];
Using normalize function like normalize(B,'range') ...
1
vote
0answers
28 views
ValueError: Found input variables with inconsistent numbers of samples: [2, 44] [closed]
Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned"
Then i create a ML model using the above as one of my features.
...
0
votes
1answer
48 views
What to do if a specific label of a category appears only a few times?
Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
0
votes
1answer
14 views
Help with breaking up a column in PowerBI to make a bar chart
My data set is a list of documents. I have a column in my data set - 'Independent Variable'. This column can contain nothing, i.e. 'NA', or one variable, i.e. 'var1', or multiple, i.e. 'var1;var2;var3'...
1
vote
2answers
58 views
Is there a possibility that there is no relationship between some inputs and outputs?
I have a general question that comes to my mind, I'm doing machine learning projects and I took a look at many datasets and worked with, mostly there are already famous datasets that everyone uses.
...
0
votes
0answers
20 views
Python Function to Incorporate Data Transform
I am new to python and I have developed a model and like to create a function for API use.
For example: predict(1,2,1) <-- call the function
if the first argument is 1 and it has 7 categories, ...
0
votes
0answers
20 views
How does scatter plot with multiple features work?
Plotting one independent(100 rows) and one dependent(100 rows) feature is straightforward. But how plotting multiple independent(100 rows, 10 columns) with one dependent(100 rows, 1 column) feature ...
0
votes
0answers
17 views
How to label sequential data for LSTM usage
I want to process data to feed it to a LSTM later, each 100 rows correspond to single category how should I label the data? Should I concatenate the 100 rows into a single row?
Data Sample:-
...
1
vote
0answers
14 views
NearestNeighbors testing
I have used
nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features)
distances, indices = nbrs.kneighbors(item_features)
to find some ...
2
votes
1answer
21 views
Combining time-series data from different devices
I am using four different sensor models (four different devices) to gather some biometric data(eg. heart rate) for the same user, for the same time period. In an ideal scenario, all the devices are ...
0
votes
2answers
33 views
How to cluster categorical and numerical data in the same dataset?
I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some ...
0
votes
1answer
112 views
How to preprocess data for Word2Vec?
I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?
0
votes
0answers
7 views
Virtual Generation of Synthetic Ancient and Dirty English Documents
I have a collection of dirty background image, below is the sample:
I have also a collection of an actual image of a dirty document with text on it, just like below:
My problem with my actual image ...
2
votes
1answer
62 views
When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?
I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould.
This completely changed my ...
0
votes
0answers
42 views
How do I read a dat file, for which I don't know its structure?
Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it.
How do I figure out the contents in my dat file. ...
3
votes
1answer
22 views
Can numerical discrete finite data be always treated also as categorical?
In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or ...
1
vote
1answer
21 views
Random Parameters to fix in ML to perform controlled experiments
Many algorithms and methods in modern Machine Learning techniques contain randomness, and because of that, running the same ML script several times can result in different outputs, therefore accuracy ...
0
votes
0answers
12 views
Using a GUI and ML to parse semi structured data from pdfs?
I have an issue where I have several different types of (PDF) documents. These documents are what I would call semi structured, in that there are often lines and borders around data, but the data ...
2
votes
2answers
308 views
Ways to simulate weather data over several periods (Python or R)?
I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding ...
2
votes
1answer
36 views
How to estimate and/or determine how much data is enough to train a model?
To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
2
votes
1answer
22 views
How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?
I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
2
votes
1answer
60 views
Fixed-Width vs Adaptive Binning
I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
0
votes
1answer
40 views
difference btw data distribution and frquency distribution
I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?
2
votes
1answer
41 views
How to separate words that are together in a large data set
in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'
i tried this but this is slow
...
1
vote
0answers
14 views
SEM (Structural Equation Modelling) with Exploratory Factor Analysis
Problem Statement: I need to do some Structural Equation Modelling at work to get the main factors in a marketing survey data-set.
There are no assumed equations to perform SEM on so what would be ...
5
votes
1answer
97 views
0
votes
1answer
26 views
How can I extract the reason of the legal compensation from a court report?
I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report)
Order mister X to ...
2
votes
1answer
22 views
How do I best visualize this voltage data for a science project
I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
1
vote
0answers
32 views
Dataset for musical Instruments recognition
I am looking for a "rich" Dataset to Teach my model to seperate Music intruments .. any Suggestion ? I found the URMP Dataset but it doesn't have many files.. I appreciate your help
thanks and happy ...
1
vote
2answers
42 views
Should I use MongoDB instead of storing data in CSV in python?
I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
