Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
1 vote
0 answers
8 views

Are there any public medical image databases?

I'm not sure if I'm asking this in the right place, please direct me to the right place for me to ask this if I am not. I'm working on a project to look at images of the inner ear from an otoscope and ...
Skymero's user avatar
  • 11
0 votes
0 answers
11 views

Building a dataset for classification

I'm thinking of building a powershell script classifier using different architectures of neural networks. I have approximately 6k powershell scripts (3k malicious, 3k benign). My questions are: How ...
freaks's user avatar
  • 1
0 votes
0 answers
8 views

Missing Value in a dataset

i'm curently cleaning a dataset and theres a column called " Institution", it is needed for encoding and training the classification model later so it needs to be cleaned. In that column, ...
Harry lou's user avatar
2 votes
0 answers
23 views

Why is it called 'Test' dataset and not 'Testing' Dataset?

Training data is named as such as it is used for 'training'. Then why not the same for 'Testing' dataset? I apologise if this is not very relevant, but its something I believe only the community can ...
Sameervdtc's user avatar
0 votes
0 answers
15 views

how to eal with a column name having comma space or and

I am taking the Coursera assignments on SQL and came across to a question where one of the tables has a column with this name: Elementary, Middle, or High School I do not know how to deal with this ...
Baghban's user avatar
  • 101
0 votes
0 answers
19 views

Using Reinforcement learning for minimisation

I would like to use reinforcement learning for the optimisation of a given function under some contraints. Take for example the following problems: ...
Ach Raf's user avatar
0 votes
0 answers
11 views

Dataset for access control policies?

I'm working on a dynamic access control policy generation model using machine learning and I need a real dataset to evaluate the performance. Is there any access control policy dataset consisting ...
Binod Karunanayake's user avatar
0 votes
0 answers
20 views

Continuous Function from Binned Data with Consistent Integrals

I have binned energy production data (in Wh) in 5-minute intervals. This means that, at the timestamp of t + 5 minutes, the value represents the amount of energy generated in the interval from t to t +...
Benjamin K's user avatar
0 votes
0 answers
12 views

Implementing Fuzzy Matching and NLP for Transaction Classification

I’m a trainee at a fintech startup, and I’m working on a project that involves classifying transactions using Natural Language Processing (NLP) and fuzzy matching techniques. The main goal is to ...
RAN's user avatar
  • 1
0 votes
1 answer
11 views

issue with data sample

...
lyel's user avatar
  • 1
0 votes
0 answers
18 views

What is the best library for time-series analysis?

Recently, I've been using statsmodels, but I would like to know if others work fine for you.
L1rola's user avatar
  • 13
0 votes
0 answers
18 views

Permanent account number

Permanent account number PAN is a unique 10 digit alphanumeric. Will there be requirement in future to be changed to a 11 digit alphanumeric taking into consideration the growing population of India? ...
Prashant Akerkar's user avatar
1 vote
1 answer
21 views

Treat Age as Categorical or Continuous

I am in the middle of processing data for feature reduction (haven't decided the method yet). The data consists of a row of people and ~3000 column of attributes that correspond to that person and is ...
sheroam's user avatar
  • 11
0 votes
0 answers
22 views

What algorithm can I use to get the product that would give me the most profit given its buying cost, selling cost, sell volume and buy volume?

I'm working on an app for evaluating the products sold on a market in a videogame. The products can be bought for cheap and sold for expensive. I want to evaluate the best product to buy and sell, but ...
EustakiosPower's user avatar
1 vote
0 answers
31 views

How to build this data pipeline?

I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should ...
Wissem Boujlida's user avatar
0 votes
0 answers
16 views

Ways to check whether your predictors are even correlated with their label?

I scraped a dataset of pre-match data in a video game and am trying to classify them by outcome, i.e. wins v. losses. The models I've tried so far have poor accuracy of around 50%, so I'm thinking ...
Lilian Shi's user avatar
0 votes
1 answer
18 views

Colab file errors

and thank you to whoever answers this. I'm pretty new to data science and machine learning, and I've been trying to immerse myself in it. I feel like there's so much to learn. One of the things that's ...
Ahmed's user avatar
  • 1
1 vote
1 answer
80 views

pos_label=1 is not a valid label. Should be one of [2,4]

I am trying to retrieve my precision score but I am getting an error as follows: pos_label=1 is not a valid label. It should be one of [2 ,4] And here is the code ...
Hanh's user avatar
  • 13
0 votes
0 answers
6 views

Connect and publish with Tableau and Azure Analysis Services

My organization is migrating to another platform and we need to replicate the reports we have made with Power BI in Tableau, but the problem is that our server data is in Azure Analysis Services, but ...
Ozdran Yankovic's user avatar
0 votes
0 answers
11 views

Excluding files in Azure Data Studio when they don't have a file extension

I need to SFTP files generated every 15 minutes up to an external SAAS product. The problem is the app generating these files has a hard lock on them. It monitors realtime and keeps appending. The ...
Transitioned's user avatar
0 votes
1 answer
23 views

What are these 'code' columns in this csv and what does Pandas pivot method do with it?

In a csv discussed in the book HoML (hands on machine learning), what do the values in capital letters represent? Are these some sort of code? Also, it seems the pandas.pivot() methods does something ...
Yorick Dantuma's user avatar
0 votes
0 answers
20 views

Inverse Laplacian of xarray

my question is related to calculating the inverse Laplacian of a xarray. Specifically, I want to solve the equation ∇^2 s = h+r+e for s in Python (where ∇^2 denotes the laplacian). Here, s is an ...
random human's user avatar
0 votes
1 answer
40 views

Is it possible to use the SODA tool as open source?

I was reading about the SODA tool and was wondering if it was free to use. As far as I understand, it has a SODA library and a SODA core, but they are all just clients for connecting to a cloud server ...
JustUser's user avatar
0 votes
1 answer
22 views

How should I plot 2D data as 3D plot?

I got data (1503, 38697) (cell(datapoint), genes(features)), each elements has integer values (gene expression). so something like ...
containletters's user avatar
0 votes
0 answers
79 views

Looking for datasets on automobile parts information for machine learning

I'm embarking on a machine learning project that requires a comprehensive dataset of automobile parts information. The goal is to train a model that can identify and categorize various auto parts, ...
Anand 's user avatar
0 votes
2 answers
483 views

Solve a non-linear system, in Python, with the GAUSS-NEWTON algorithm? (Jacobian matrix J, etc.)

I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
Zollikofen4's user avatar
0 votes
1 answer
19 views

Hassle-free platform for a small data science team?

our team of 4 data scientists has been exploring options to automate our ETL, data storage and model updates and are looking for a hassle-free platform that can help with this. We've had several ...
Rotten_Cabbagges's user avatar
0 votes
0 answers
14 views

Packet data needed fot the model

I want to use this data set https://huggingface.co/datasets/rdpahalavan/network-packet-flow-header-payload for network attack classification Here is packet structure from the README file. Assuming I ...
Roma's user avatar
  • 101
0 votes
0 answers
24 views

statistical analysis of two data sets have different number of sampling

I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
Fadi's user avatar
  • 1
1 vote
1 answer
80 views

Should I apply the same data transformations in production for my classification model's inference steps

I am now moving my best classification model to production and doing tests currently. Should I use the same scaler() I used in training during my inference in ...
MAA's user avatar
  • 11
2 votes
2 answers
108 views

How to deal with high data volumes? (Tools, techniques, concepts, etc.)

I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and ...
tms's user avatar
  • 21
0 votes
0 answers
5 views

Data Tracking For Process Improvement

I work in a manufacturing plant with 6 production lines. As part of the maintenance crew, it is my job to record all maintenance work performed on the equipment inside of our shared OneNote. This ...
Austin's user avatar
  • 1
0 votes
0 answers
24 views

What was used before data augmentation (e.g. SMOTE) to train ML models on imbalanced data? Please provide citations

I am seriously curious on how imbalance data was treated in machine learning and statistical learning before modern time data augmentation solutions such as SMOTE appeared. Please provide citations ...
Full Array's user avatar
0 votes
1 answer
78 views

Loading Data with column Headers

I have a file called adult.data that contains my test set in a CSV file format. Unfortunately there are no column headers in the file to describe the variables. But I have a file called adult.names ...
Sean's user avatar
  • 1
0 votes
0 answers
50 views

Outliers problem

everyone hope you all are good. I have a hard time with outliers First i found some outliers in train set then i capped them and did the same with test set. So when i checked my rsme root mean ...
Shoaib Ahmed's user avatar
0 votes
0 answers
12 views

Seeking Guidance on Creating Sample Data for AI Business Use Case

Subject: Seeking Guidance on Creating Sample Data for AI Business Use Case Dear StackExhange Family I am currently working on an AI project aimed at providing AI embaded solutions to HealthCare, ...
TARUN KUMAR's user avatar
1 vote
0 answers
30 views

Determining the start and stop of vehicle movement by GPS

There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
Abraham Lincoln's user avatar
0 votes
0 answers
29 views

Data Science Communities for complete beginners

Any recommendation as to where i can join communities and forums so i can get to learn Data Science please? i am a total beginner and i'm having a hard time learning it alone.
Jibril Yahaya Jibril's user avatar
0 votes
1 answer
31 views

Very basic python question [closed]

I am doing the free interactive learn python tutorial, and I am on the first conditions lesson. The task was to make it so that every statement in the following code resolved true. The code below was ...
auditorchick's user avatar
0 votes
0 answers
13 views

Pre-trained text classifier to check whether a document is business or personal

I need a pre-trained classifier that checks if a document is either a business or personal document. I don't have time to train one myself so if anyone knows of a classifier that is pre-trained and ...
user155097's user avatar
0 votes
0 answers
11 views

in data science if a feature has a dominant category we drop it before model building. why do we do so?

In machine learning and data science, we drop the feature in which one category heavily dominates the other category. Why do we do so?
Lakshya Kumar's user avatar
0 votes
1 answer
17 views

Dates countries joined/left international organisations

Does anyone know of a reliable data source containing the dates countries left or joined international organisations, for example,WTO, EU. etc? Ideally I'd like this going back to 1946. Even the year ...
James's user avatar
  • 121
0 votes
0 answers
9 views

Which Frameworks/Libs Best Support Integer-Based Features, Scaling, Training, etc?

Papers such as Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference have interested me in exploring integer-based data science. In particular, I'm thinking of ...
ezekiel68's user avatar
0 votes
0 answers
29 views

Using PowerBI calculated columns vs calculated measures to calculate a frequency flag column

Sorry for the long question. In PowerBI I have fields called 'user ID' and 'service'. Each row represents an activity, so may not be unique. I want to create a new variable called '5+ flag' that for ...
Molly's user avatar
  • 1
0 votes
0 answers
7 views

Terminology: What is it called when the filter criteria is added to the data table?

When we want to focus on a group within our dataset, it is common to filter it. Suppose we have a data table showing how many points 4 players earned in a game. Table 1: Game Scores for 4 Players Age ...
madprogramer's user avatar
0 votes
2 answers
41 views

Lack of imagination when creating projects

Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
Davi Komura de Castro's user avatar
0 votes
0 answers
15 views

Mapping two unrelated representation spaces with different distributions, roughly preserving similarities

I've got two embedding / representation spaces which are completely unrelated to begin with, yet I wish to find a (=any) mapping between them. Space A are feature histograms with a somewhat normal ...
nussbaum's user avatar
0 votes
0 answers
14 views

How should I save data in deeplearning with nosql (mongodb)?

I usually use file system to manage data for my deep learning model, but one of my boss told me to make nosql database to manage data. Datasets I use have m rows, and n columns of count matrix and ...
containletters's user avatar
0 votes
0 answers
12 views

Detect Recent Anomalies

I'm trying to see how to approach this problem: I have a dataset of fraud transactions. There are several categorical columns, like country, type, merchant, etc. All of the records are considered ...
F_M's user avatar
  • 131
0 votes
0 answers
12 views

(Apparently) Inconsistent Results Between Visual Patterns and Chi-Square Test in Categorical Data Analysis

I'm stuck while working on a categorical data analysis project and would appreciate any insight. Here are the first 4 rows of the dataset: successful_upload feat_1 feat_2 0 T>2 <\$10K 1 0.5&...
Sherwood's user avatar

1
2 3 4 5
18