Questions tagged [data-analysis]
Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making.
281 questions
1
vote
0
answers
227
views
Plot aggregated variable across two categorical variables
I have a dataset of GitHub data with two categorical columns: Repository type and research domain. I want to visualize many variables (booleans and numerical) that showcase the differences across the ...
- 11
1
vote
0
answers
59
views
Is there a python package that can replicate this model?
I'm trying to convert a short R data analysis script to python and can't find the right modules/packages to do so.
This is the script in R:
...
- 11
0
votes
0
answers
70
views
What are some "best practices" for discovering true patterns in data without relying on "scores" as measures of accuracy?
What are some "best practices" for discovering true patterns in data without knowing about them and without relying on "scores" as measures of accuracy?
In university I was always ...
- 416
3
votes
2
answers
1k
views
Storing Large dataset for processing and analysis of data
I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I ...
CommunityBot
- 1
5
votes
2
answers
33k
views
Plot Two Categorical Variables
How to plot two categorical variables in Python or using any library? I want to plot the Playing Role of a Cricketer (Batsman, Bowler, etc.) VS Bought_By (Franchise Names, e.g., CSK, DC, etc.). The ...
- 21
4
votes
2
answers
205
views
find most dense neighborhood of points in high dimensional space
I'm working on a project where I have many high-dimensional points and I want to find the most dense neighborhood of them. Ideally, out of my ~500 points that are each a 4x300 matrix (300 ms time ...
CommunityBot
- 1
0
votes
0
answers
27
views
How can "profiling" be done in a dataset? What are the different techniques?
I am currently doing an analysis in which I need to "profile" each record. For example, let's say I have a dataset of accounts with customer information (name, id, address, money spent, ...
0
votes
1
answer
786
views
How do I not display rows that have an empty value when trying to output a dataframe with pandas [closed]
I have this table where there are missing values under the value2 column.
Value1
Value2
1000
1000
1000
500
1000
560
1000
560
What I would like to do is to display the above table but without ...
- 8,197
1
vote
1
answer
248
views
Having weird accuracy graph on deep learning binary classification model
I am doing deep learning binary classification on some data and got very weird results with the accuracy metric. In the first few epochs, it doesn't change at all but then it goes on this weird linear ...
- 566
1
vote
1
answer
312
views
How to return the number of values that has a specific count
I would like to find how many occurrences of a specific value count a column contains. For example, based on the data frame below, I want to find how many values in the ID column are repeated twice
<...
- 263
0
votes
1
answer
28
views
Determine the effect on margins of a price increase
I hope you can help guide me in the right direction! Any advice is appreciated!
Situation
I'm currently analyzing the effect of a price increase from a retailer on a few 100 products. I'm interested ...
- 421
1
vote
0
answers
23
views
How to re-generate dataset df at time t2 while having df at time t1 and cross-sectional dataset df' at time t1 and t2?
I have a travel survey dataset df' collected in 2017 and also 2019. Note that individuals (households here) are not necessarily identical in 2017 and 2019 but their features are.
Dataset df' in 2017:
...
- 11
1
vote
1
answer
19
views
How do I estimate the benefit of one business method over another when all my data is confounded?
I'm trying to estimate the benefit of using one method over another, but the existing data is confounded and I'm struggling to figure it out.
So my company submits debit orders to customers to collect ...
- 25.9k
0
votes
1
answer
98
views
Calculating relative risk in R
I am working with the nhanes data from 2017-2018. I want to determine the relative risk of being overweight (Adipös) when being "Poor". And this grouped by Gender and Ethnicity.
So far I ...
- 8,197
0
votes
1
answer
715
views
Clustering a set of matrices with kmeans/other approaches
I am posting here as overflow community told me it was better to ask it here. I was trying to cluster a set of matrices in order to obtain $n$ clusters based on matrix similarity.
In practice, my ...
- 2,074
4
votes
1
answer
6k
views
Monthly trend with fb prophet-Interpreting the graph
I have monthly data with month/year in one column and price on another. I would like to get a yearly trend with fb prophet library in python (how to use monthly data with the library is explained at ...
CommunityBot
- 1
26
votes
7
answers
7k
views
Is Python a viable language to do statistical analysis in?
I originally came from R, but Python seems to be the more common language these days. Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using ...
- 7,776
0
votes
2
answers
131
views
Which analytics metrics are useful?
I am working on a home project and the question is :
A session is defined as a period of activity in the app. If there is a period longer than 5 minutes when the user does not do anything actively ...
- 11
4
votes
1
answer
304
views
How to interpret standard deviation calculated by excel
I have some percentage values in my excel spreadsheet like:
When i use the STDEV function to calculate the standard deviation with these values, i get ...
- 3,069
1
vote
2
answers
19k
views
ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')
X = train_encoded_df.iloc[:, 1: ]
y = train_encoded_df["Loan_Status"]
print("Precision:",metrics.precision_score(y_test, y_pred))
My training data contains the ...
3
votes
2
answers
78
views
is it a good idea to take the derivative or integral of some features and add them as new features in machine learning?
I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column ...
CommunityBot
- 1
1
vote
1
answer
339
views
SKLearn PCA explained_variance_ration cumsum gives array of 1
I have a problem with PCA. I read that PCA needs clean numeric values. I started my analysis with a dataset called trainDf with shape ...
- 7,776
0
votes
1
answer
1k
views
Study names not showing in forest plot (R)
Context: I followed an existing R script to create a forest plot in R using the forest function from the metafor package.
Problem: The plot is showing, however, the study names are not showing on it ...
- 8,197
0
votes
2
answers
91
views
Evaluating if metric of one group is higher than the metric of another group when group sizes differ significantly
I am working with a dataset that contains data of applicant income, gender, and loan status (whether or not the person was approved for a loan). I've created the following plots from the data. The ...
- 22.3k
1
vote
1
answer
30
views
How to assess a trend during time?
I wanted to know which analysis method would you suggest for assessing a trend during years. Let us say we have data on antibiotic resistance in a particular bacteria from 2010 until 2021. How can we ...
- 1,657
1
vote
0
answers
47
views
Tableau - Filter Buttons not filtering correctly
I created filter buttons to use as a filter for 6 category's (measures) in Tableau Public (latest version) so when you click on them the values for each display on my "Military Enrollment By ...
- 11
3
votes
1
answer
126
views
Detecting specific points in (noisy) dataset
In my recent work I've came across a problem where I need to find certain points in quickly oscillating data. Let's work with syntethic data instead of real measurements and ignore the noise for the ...
CommunityBot
- 1
1
vote
0
answers
151
views
How to integrate Zoho analytics with Jupyter notebook? [closed]
I am trying to connect Zoho Analytics and Python for importing data from Zoho Analytics.
I have already installed !pip install zoho-analytics-connector. What should ...
- 11
1
vote
1
answer
206
views
Knit with R markdown [closed]
I have tried running some code on RStudio Desktop, within the Chunk the codes runs smoothly, but when knitting to view as html, I get the message that something is wrong with a line of code. What can ...
1
vote
2
answers
147
views
For a student who is a beginner in quantitative research and statistics, which is the better statistical tool to start: R or IBM SPSS? Why? [closed]
Currently, I am writing my research design. However, I am still indecisive on what statistical tool should I use for the data analysis. I tried looking up on the internet and there are disparate ...
- 7,776
1
vote
0
answers
94
views
How to represent/aggregate data belonging to same group, and feed to model for training?
I have a problem in which there are couple of records that belong to the same group, like multiple observations. For example this data set
has three groups, For each group, I have three observations....
- 123
0
votes
1
answer
4k
views
How to find median/average values between data frames with slightly different columns?
I am trying to combat run-to-run variance of the data I collect by combining the data from different runs and finding the mean/average. The problem is that in each run there is a chance that some of ...
- 8,197
2
votes
1
answer
121
views
Log scaling whole dataset
I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value.
I am wondering how I can get rid of this -infinity?
I have been advised to add a small value ...
CommunityBot
- 1
0
votes
2
answers
941
views
If I have two variables with strong correlation, should I delete one and leave the other in my data
I have a large dataset, where I should make a binary prediction. The fact is that, after analyzing the data, I found that some variables are positively correlated to each other. So, I was wondering ...
- 868
1
vote
0
answers
9
views
Identifying the best group that are most productive
can you please give some idea to solve this problem, Below is my data, I want to identify the best inventory based on the CPM(target value) .. For example: I want to be able to say - Business section, ...
- 11
1
vote
1
answer
20
views
Numbers of participants in crowdsourcing tasks
Within the recent years, leveraging crowdsourcing for performing, mainly intelligent-intensive, tasks has become prevalent. An important question in this regard, that I could not find any specific ...
- 22.3k
0
votes
2
answers
276
views
Building predictive model with low correlated data [closed]
I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or ...
- 1,657
0
votes
1
answer
48
views
Creative Analysis - What are the possible approaches?
We're analyzing some creatives from past advertising campaigns and are looking for the best way to approach. Currently we've broken down each creative into 20+ objective characteristics, for example:
...
CommunityBot
- 1
0
votes
1
answer
101
views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company.
Beforehand we want to test this new price list in several countries.
A/B test cannot be conducted here ...
- 22.3k
1
vote
0
answers
31
views
Finding correlations between plant features?
I have a list of flowers, and each has a color and a fragrance intensity. Each data row has a color and intensity for one breed of flower, like this:
...
0
votes
1
answer
42
views
Document Clustering for given specific clusters in python
How can we classify text in to given specific number of clusters in python? I'm aware that the number of clusters can be specified using some mechanisms like k-means but I need to classify the given ...
- 884
1
vote
3
answers
1k
views
How to handle a large number of categories in one column effectively in machine learning?
In my dataset one column has a large no of categories. For model training or building we need to convert these categories to numeric. There are several methods we can use to convert labels to numeric, ...
CommunityBot
- 1
1
vote
1
answer
912
views
If two features are highly correlated to each other and giving same impact on target variable in this case, which feature we need to select?
in my dataset, I am having plenty of features and two features are highly correlated to each other and giving same impact on target variable, in this case which feature we need to select in order to ...
- 672
3
votes
2
answers
441
views
why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding
We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data.
And I real a blog which give the difference between Label Encoding and One-Hot Encoding.
So ...
0
votes
3
answers
515
views
Calculating optimal 'phase shift' when comparing two linear datasets to maximise correlation?
Let's say I have two columns of data, when graphed they look like this:
This does not have a particularly wonderful correlation.
On the other hand if I shift one graph 10 datapoints relative to the ...
- 231
1
vote
0
answers
168
views
Detect Bias in YOLO format object detection dataset
I work on YOLO data annotation. I want to know how to detect bias in the data before training in this kind of datasets. What are the metrics and analysis methods needed to use to detect bias in ...
- 161
1
vote
0
answers
124
views
How to plot visualization with Interactive Feature Selection in Bokeh, Python
The task is to automate the Visualization. The CSV file contains large nos of features (column names e:g. 32 nos it may increase in future). The task is to plot Interactive Visualization. All the ...
1
vote
0
answers
10
views
Does anyone know of literature regarding a Neural Net boosted GBM?
For obvious reasons, most GBMs created in the private sector are tree boosted.
Occasionally, one might want a linear boosted GBM so that the residual models collapse into a simple linear combination. ...
- 11
1
vote
0
answers
34
views
I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?
In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of ...
2
votes
0
answers
23
views
Can you apply technical analysis indicators to general data?
I collect data, for example, "how many people mentioned 'Bitcoin' on an investing community", I take the data for daily, so it's like "5/1,231;5/2,156;5/3,223" and so on. So ...
- 121