Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

dataset where the data changes occur irregularly, alternating between every two and three days

I'm working with a dataset where the data changes occur irregularly, alternating between every two and three days. what should i do to generate a corresponding date series to match this irregular ...
REDA OUMLIL's user avatar
0 votes
1 answer
25 views

I have some topics from python out of which I want to know the one which actually needs to be studied for data science roles specifically

The topics are as follows: Collections Itertools Regex and parsing Error and exceptions Date and time
Kanika Dixit's user avatar
0 votes
0 answers
10 views

Help required in opening files of a dataset (.phys, .thermal, .pts, .ass extensions)

We have received a dataset that consists of audio, visual, thermal, and physiological modalities. Upon exploring the dataset, we encountered some challenges in opening the following file types: .phys ...
Anup Kumar Gupta's user avatar
0 votes
0 answers
9 views

How do we set daterange in Sailpoint IIQ

I'm trying to set the daterange of a report dynamically. But it seems as if the package I'm using doesn't implement the interface Serializable as it is supposed to. Do you have any idea how I can ...
srikanthbollu's user avatar
0 votes
0 answers
16 views

Weighted average for multiple confusion matrix

I have problem and i have no idea how to resolve it. I have 4 Confusion Matrixes and i need to calculate for example Matthews Correlation for each CM. In second step i want calculate BIG KPI result ...
Debosy's user avatar
  • 1
0 votes
0 answers
36 views

improve my f1_score for classification - pandas/sklearn

I would like advice on how to improve my f1_score for classification. I currently have something around 0.57. Dataset: lotWaferDie - lot, board and chip on which defects were measured string values ...
user162343's user avatar
1 vote
0 answers
13 views

Are there any public medical image databases?

I'm not sure if I'm asking this in the right place, please direct me to the right place for me to ask this if I am not. I'm working on a project to look at images of the inner ear from an otoscope and ...
Skymero's user avatar
  • 11
0 votes
0 answers
12 views

Building a dataset for classification

I'm thinking of building a powershell script classifier using different architectures of neural networks. I have approximately 6k powershell scripts (3k malicious, 3k benign). My questions are: How ...
freaks's user avatar
  • 1
0 votes
0 answers
10 views

Missing Value in a dataset

i'm curently cleaning a dataset and theres a column called " Institution", it is needed for encoding and training the classification model later so it needs to be cleaned. In that column, ...
Harry lou's user avatar
2 votes
0 answers
23 views

Why is it called 'Test' dataset and not 'Testing' Dataset?

Training data is named as such as it is used for 'training'. Then why not the same for 'Testing' dataset? I apologise if this is not very relevant, but its something I believe only the community can ...
Sameervdtc's user avatar
0 votes
0 answers
16 views

how to eal with a column name having comma space or and

I am taking the Coursera assignments on SQL and came across to a question where one of the tables has a column with this name: Elementary, Middle, or High School I do not know how to deal with this ...
Baghban's user avatar
  • 101
0 votes
0 answers
23 views

Using Reinforcement learning for minimisation

I would like to use reinforcement learning for the optimisation of a given function under some contraints. Take for example the following problems: ...
Ach Raf's user avatar
0 votes
0 answers
13 views

Dataset for access control policies?

I'm working on a dynamic access control policy generation model using machine learning and I need a real dataset to evaluate the performance. Is there any access control policy dataset consisting ...
Binod Karunanayake's user avatar
0 votes
0 answers
20 views

Continuous Function from Binned Data with Consistent Integrals

I have binned energy production data (in Wh) in 5-minute intervals. This means that, at the timestamp of t + 5 minutes, the value represents the amount of energy generated in the interval from t to t +...
Benjamin K's user avatar
0 votes
0 answers
13 views

Implementing Fuzzy Matching and NLP for Transaction Classification

I’m a trainee at a fintech startup, and I’m working on a project that involves classifying transactions using Natural Language Processing (NLP) and fuzzy matching techniques. The main goal is to ...
RAN's user avatar
  • 1
0 votes
1 answer
11 views

issue with data sample

...
lyel's user avatar
  • 1
0 votes
0 answers
20 views

What is the best library for time-series analysis?

Recently, I've been using statsmodels, but I would like to know if others work fine for you.
L1rola's user avatar
  • 13
0 votes
0 answers
18 views

Permanent account number

Permanent account number PAN is a unique 10 digit alphanumeric. Will there be requirement in future to be changed to a 11 digit alphanumeric taking into consideration the growing population of India? ...
Prashant Akerkar's user avatar
1 vote
1 answer
25 views

Treat Age as Categorical or Continuous

I am in the middle of processing data for feature reduction (haven't decided the method yet). The data consists of a row of people and ~3000 column of attributes that correspond to that person and is ...
sheroam's user avatar
  • 11
0 votes
0 answers
22 views

What algorithm can I use to get the product that would give me the most profit given its buying cost, selling cost, sell volume and buy volume?

I'm working on an app for evaluating the products sold on a market in a videogame. The products can be bought for cheap and sold for expensive. I want to evaluate the best product to buy and sell, but ...
EustakiosPower's user avatar
2 votes
0 answers
42 views

How to build this data pipeline?

I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should ...
Wissem Boujlida's user avatar
0 votes
0 answers
18 views

Ways to check whether your predictors are even correlated with their label?

I scraped a dataset of pre-match data in a video game and am trying to classify them by outcome, i.e. wins v. losses. The models I've tried so far have poor accuracy of around 50%, so I'm thinking ...
Lilian Shi's user avatar
0 votes
1 answer
19 views

Colab file errors

and thank you to whoever answers this. I'm pretty new to data science and machine learning, and I've been trying to immerse myself in it. I feel like there's so much to learn. One of the things that's ...
Ahmed's user avatar
  • 1
1 vote
1 answer
106 views

pos_label=1 is not a valid label. Should be one of [2,4]

I am trying to retrieve my precision score but I am getting an error as follows: pos_label=1 is not a valid label. It should be one of [2 ,4] And here is the code ...
Hanh's user avatar
  • 13
0 votes
0 answers
6 views

Connect and publish with Tableau and Azure Analysis Services

My organization is migrating to another platform and we need to replicate the reports we have made with Power BI in Tableau, but the problem is that our server data is in Azure Analysis Services, but ...
Ozdran Yankovic's user avatar
0 votes
0 answers
12 views

Excluding files in Azure Data Studio when they don't have a file extension

I need to SFTP files generated every 15 minutes up to an external SAAS product. The problem is the app generating these files has a hard lock on them. It monitors realtime and keeps appending. The ...
Transitioned's user avatar
0 votes
1 answer
25 views

What are these 'code' columns in this csv and what does Pandas pivot method do with it?

In a csv discussed in the book HoML (hands on machine learning), what do the values in capital letters represent? Are these some sort of code? Also, it seems the pandas.pivot() methods does something ...
Yorick Dantuma's user avatar
1 vote
0 answers
26 views

Inverse Laplacian of xarray

my question is related to calculating the inverse Laplacian of a xarray. Specifically, I want to solve the equation ∇^2 s = h+r+e for s in Python (where ∇^2 denotes the laplacian). Here, s is an ...
random human's user avatar
0 votes
1 answer
54 views

Is it possible to use the SODA tool as open source?

I was reading about the SODA tool and was wondering if it was free to use. As far as I understand, it has a SODA library and a SODA core, but they are all just clients for connecting to a cloud server ...
JustUser's user avatar
0 votes
1 answer
23 views

How should I plot 2D data as 3D plot?

I got data (1503, 38697) (cell(datapoint), genes(features)), each elements has integer values (gene expression). so something like ...
containletters's user avatar
0 votes
0 answers
123 views

Looking for datasets on automobile parts information for machine learning

I'm embarking on a machine learning project that requires a comprehensive dataset of automobile parts information. The goal is to train a model that can identify and categorize various auto parts, ...
Anand 's user avatar
0 votes
2 answers
518 views

Solve a non-linear system, in Python, with the GAUSS-NEWTON algorithm? (Jacobian matrix J, etc.)

I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
Zollikofen4's user avatar
0 votes
1 answer
20 views

Hassle-free platform for a small data science team?

our team of 4 data scientists has been exploring options to automate our ETL, data storage and model updates and are looking for a hassle-free platform that can help with this. We've had several ...
Rotten_Cabbagges's user avatar
0 votes
0 answers
15 views

Packet data needed fot the model

I want to use this data set https://huggingface.co/datasets/rdpahalavan/network-packet-flow-header-payload for network attack classification Here is packet structure from the README file. Assuming I ...
Roma's user avatar
  • 101
0 votes
0 answers
24 views

statistical analysis of two data sets have different number of sampling

I have two datasets from my statistical fault analysis: one with 400 iterations and another with 500 iterations. Both datasets have the same maximum and minimum values, and some data points are ...
Fadi's user avatar
  • 1
1 vote
1 answer
142 views

Should I apply the same data transformations in production for my classification model's inference steps

I am now moving my best classification model to production and doing tests currently. Should I use the same scaler() I used in training during my inference in ...
easymoneysniper's user avatar
2 votes
2 answers
139 views

How to deal with high data volumes? (Tools, techniques, concepts, etc.)

I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and ...
tms's user avatar
  • 21
0 votes
0 answers
5 views

Data Tracking For Process Improvement

I work in a manufacturing plant with 6 production lines. As part of the maintenance crew, it is my job to record all maintenance work performed on the equipment inside of our shared OneNote. This ...
Austin's user avatar
  • 1
0 votes
0 answers
24 views

What was used before data augmentation (e.g. SMOTE) to train ML models on imbalanced data? Please provide citations

I am seriously curious on how imbalance data was treated in machine learning and statistical learning before modern time data augmentation solutions such as SMOTE appeared. Please provide citations ...
Full Array's user avatar
0 votes
1 answer
86 views

Loading Data with column Headers

I have a file called adult.data that contains my test set in a CSV file format. Unfortunately there are no column headers in the file to describe the variables. But I have a file called adult.names ...
Sean's user avatar
  • 1
0 votes
0 answers
50 views

Outliers problem

everyone hope you all are good. I have a hard time with outliers First i found some outliers in train set then i capped them and did the same with test set. So when i checked my rsme root mean ...
Shoaib Ahmed's user avatar
0 votes
0 answers
12 views

Seeking Guidance on Creating Sample Data for AI Business Use Case

Subject: Seeking Guidance on Creating Sample Data for AI Business Use Case Dear StackExhange Family I am currently working on an AI project aimed at providing AI embaded solutions to HealthCare, ...
TARUN KUMAR's user avatar
1 vote
0 answers
30 views

Determining the start and stop of vehicle movement by GPS

There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
Abraham Lincoln's user avatar
0 votes
0 answers
29 views

Data Science Communities for complete beginners

Any recommendation as to where i can join communities and forums so i can get to learn Data Science please? i am a total beginner and i'm having a hard time learning it alone.
Jibril Yahaya Jibril's user avatar
0 votes
1 answer
36 views

Very basic python question [closed]

I am doing the free interactive learn python tutorial, and I am on the first conditions lesson. The task was to make it so that every statement in the following code resolved true. The code below was ...
auditorchick's user avatar
0 votes
0 answers
13 views

Pre-trained text classifier to check whether a document is business or personal

I need a pre-trained classifier that checks if a document is either a business or personal document. I don't have time to train one myself so if anyone knows of a classifier that is pre-trained and ...
user155097's user avatar
0 votes
0 answers
15 views

in data science if a feature has a dominant category we drop it before model building. why do we do so?

In machine learning and data science, we drop the feature in which one category heavily dominates the other category. Why do we do so?
Lakshya Kumar's user avatar
0 votes
1 answer
17 views

Dates countries joined/left international organisations

Does anyone know of a reliable data source containing the dates countries left or joined international organisations, for example,WTO, EU. etc? Ideally I'd like this going back to 1946. Even the year ...
James's user avatar
  • 121
0 votes
0 answers
9 views

Which Frameworks/Libs Best Support Integer-Based Features, Scaling, Training, etc?

Papers such as Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference have interested me in exploring integer-based data science. In particular, I'm thinking of ...
ezekiel68's user avatar
0 votes
0 answers
32 views

Using PowerBI calculated columns vs calculated measures to calculate a frequency flag column

Sorry for the long question. In PowerBI I have fields called 'user ID' and 'service'. Each row represents an activity, so may not be unique. I want to create a new variable called '5+ flag' that for ...
Molly's user avatar
  • 1

1
2 3 4 5
18