Data Science
Sponsored by
Sponsored logo

Questions tagged [python]

Ask Question

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

5,967 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Adding a group specific penalty to binary cross-entropy

I want to implement a custom Keras loss function that consists of plain binary cross-entropy plus a penalty that increases the loss for false negatives from one class (each observation can belong to ...
0
votes
2answers
113 views

Why is XGBClassifier in Python outputting different feature importance values with the same data across different repetitions?

I am fitting an XGBClassifier to a small dataset (32 subjects) and find that if I loop through the code 10 times the feature importances (gain) assigned to the features in the model varies slightly. I ...
0
votes
0answers
5 views

Is there a way to output feature importance based on the outputted class?

I'm running a random forest classifier in Python (two classes). I am using the feature_importances_ method of the ...
4
votes
1answer
328 views

Multiple activation functions with TensorFlow estimator DNNClassifier

I just want to know if is it possible to use tf.estimator.DNNClassifier with multiple different activation functions. I mean, could I use a DNNClassifier estimator which use different activation ...
0
votes
1answer
57 views

Whether to replace NaN values in result column

I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "...
0
votes
1answer
15 views

How to fit Word2Vec on test data?

I am working on a Sentiment Analysis problem. I am using Gensim's Word2Vec to vectorize my data in the following way: ...
2
votes
2answers
220 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
3
votes
1answer
2k views

One hot encoding as input to recurrent neural networks

I'm trying to predict next label in a pattern based on previous labels using recurrent neural network. In total I have 100 labels Example of input pattern: ...
1
vote
1answer
3k views

Training Inception V3 based model using Keras with Tensorflow Backend

I am currently training a few custom models that require about 12Gb GPU memory at the most. My setup has about 96Gb of GPU memory and python/Jupyter still manages to hog up all the gpu memory to the ...
0
votes
1answer
36 views

Transitioning from a python script for data transformation to BigQuery

So I have a dataset spread over multiple and ever-growing excel files all of which looks like: email order_ID order_date user1@email.com 1234 23-Mar-2021 user2@email.com 1235 23-Mar-2021 user2@...
1
vote
1answer
80 views

Naives Bayes Text Classifier Confidence Score

I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any ...
0
votes
3answers
62 views

p-value of chi squared test is exactly 0.0

I need to do a chi square test of two of my dataset's categorical variables. This two variables have basically the same meaning but comes from two different sources, so my idea is to use a chi square ...
1
vote
1answer
240 views

CASE WHEN SQL Type Statement in pandas

I'm trying to create a new column called Class which is a calculated column. Basically when the Math class is NA then the class will be the CS class, otherwise it will be vice versa. How can I do ...
2
votes
1answer
241 views

How to tune / choose the preference parameter of AffinityPropagation?

I have large dictionary of "pairwise similarity matrixes" that would look like the following: similarity['group1']: ...
0
votes
0answers
4 views

measuring variance of differences between two sets of corresponding means with significance

I have a dataframe id, w, x, y1, y2 (two categorical variables, two dependent variables) id is the index which is not particularly informative w, x are categorical variables - w in {0,1}, x in {0,1,2,...
0
votes
2answers
503 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
1answer
35 views

need an explanation of the For Loop in the DBSCAN algorithm Demo

In the following code of the DBSCAN algorithm, as a beginner I need an explanation for what happens to the data in the bottom for loop and why ? Generate sample data ...
4
votes
2answers
2k views

Multidimensional scaling producing different results for different seeds

I took the data from here and wanted to play around with multidimensional scaling with this data. The data looks like this: In particular, I want to plot the cities in a 2D space, and see how much it ...
0
votes
0answers
5 views

Clustering with multiindex DataFrame

I have a huge amount of multiindexed data that, very simplified, looks like this: Code: ...
0
votes
2answers
16 views

How to set vocabulary size, padding length and embedding dimension in LSTM network?

Usually in a LSTM network, we have certain parameters that need to be set before the model can begin training. I am specifically talking about vocabulary size, padding length and embedding dimension. ...
0
votes
0answers
7 views

How does ConvLSTM layer calculates the output

I'm building ConvLSTM for forecasting weather, but I'm not able to understand the math calculation behind the layer. Like in simple Conv layer we simply do the dot product of input with the kernel. ...
0
votes
0answers
10 views

PCA in Tensorflow

I am currently trying to understand the code of PCA in TensorFlow Extended. I tried to understand how the Covariance Matrix is calculated once we have a PCollection (Beam). I wanted to execute the ...
0
votes
0answers
12 views

Help: something weird up with my model

I have the following model... ...
0
votes
1answer
6 views

CREATE TABLE USING Oracle DATA_SOURCE

I am trying to create a table using ORACLE as a data source using spark query but getting an error. %sql CREATE TABLE TEST USING org.apache.spark.sql.jdbc OPTIONS ( url "jdbc:oracle:thin:@...
1
vote
1answer
48 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
2
votes
1answer
360 views

Model prediction on meshgrid in python

Suppose I have data with two independent variable $X_1$, $X_2$ and one dependent variable say $y$, as follows: $X_1$: $x_{1,1}$, $x_{1,2}$ , $x_{1,3}$ $X_2$: $x_{2,1}$, $x_{2,2}$, $x_{2,3}$ $y$: $...
0
votes
1answer
364 views

partial fitting, how to ensure one hot captures all features consistently

Doing some data science on ~4 million samples, with lots of columns being categorical. One column has ~1000 categories and my boss insists on including it in the analysis. My output is also ...
2
votes
1answer
52 views

In a Time Series Problem, is it possible to forecast quantities by learning the patterns of other items? What are my options?

Suppose I own a store that sells a variety of apples and I have the following stats each month. Report Date Type of Apple (TA) Quantity Available(QA) Quantity Sold in the Past 30 days(QS30) ...
40
votes
3answers
25k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
0
votes
0answers
12 views

Training an LSTM

For training an LSTM, does multi dimensional data need to be flattened first? (This is in reference to pytorch, but the pytorch forums haven't answered either of my questions) For example: Suppose i ...
0
votes
1answer
43 views

Ordering a material science dataset (properties names, properties scalars, formulas)

I'm dealing with a materials science dataset and I'm in the following situation, I have data organized like this: ...
2
votes
3answers
2k views

Custom conditional Keras metric

I am trying to create the following metric for my neural network using keras $$ s = \left\{ \begin{array}{ll} \sum_{i=1}^{n} e^{\frac{-d_i}{10}}-1 & \quad d < 0 \\ ...
0
votes
0answers
9 views

Creating table in Databricks using the table from Oracle

I am trying to create a table in Databricks using a object in Oracle Db, but getting an error ...
0
votes
1answer
2k views

Dimension Mismatch Error during dot product in Python

I have two matrices user_vecs and item_vecs I am trying to take the dot product of the two to build a recommendation engine: The ...
4
votes
3answers
396 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
1
vote
1answer
18 views

Converting binary valued data to boolean or one hot?

I am dealing with a dataset that contains multiple columns (features) that contain binary variables, e.g., a gender feature that contains 'male' and 'female'. I want to apply some supervised learning ...
1
vote
2answers
184 views

Predicting Customer Activity Absence

Could you please assist me with to following question? I have a customer activity dataframe that looks like this: It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and ...
0
votes
0answers
23 views

Where to start with ChatBots?

I want to start my journey into ChatBots and how I can create them. I have read some articles regarding the type of chatbots. Basically there are 2 types of chatbots, one is a rule based and the other ...
1
vote
2answers
622 views

Which algorithm can be used to reduce dimension of multiple time series?

In my dataset, a data point is essentially a Time series of 6 feature over a year per month so in all, it results in 6*12=72 features. I need to find class outliers so I perform dimensionality ...
6
votes
1answer
2k views

Why Scikit and statsmodel provide different Coefficient of determination?

First of all, I know there is a similar question, however, I didn't find it so much helpful. My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can ...
1
vote
1answer
22 views

Best way to optimize dataframe row by row sum of squared errors calculation?

I am seeing a significant slowdown with the following small snippet of code which computes the sum of squared errors between two dataframes - e.g. it takes approximately 2.5 seconds to run when ...
0
votes
0answers
21 views

What is the best algorithm for feature selection in categorical variables?

Imagine this question in two parts: Part 1 : My Y variable can be only 1 and 0 Part 2 : My Y variable can be only -1, 0, 1 In both cases my X is continous with float variables. What are the ...
0
votes
0answers
8 views

Method for predicting future state, based on time spent in previous states

So what I'm looking for is the best approach to predict a future state. Say we have three states: A, B, C. I want to predict if in the next time-interval (f.e. a day or a week) the state will become C....
3
votes
1answer
461 views

Complete a Hungarian stem to a real word

I'm quite new to the NLTK package of Python and to NLP too (I usually work in R but for NLP purposes and scraping maybe Python is more able). I scrap articles from Hungarian newsportals and want to ...
0
votes
2answers
1k views

How to deal with errors of defining data types in pandas' read_csv ()?

I have a table with 118,000 rows and 80 columns. I would like to select 8 columns from the table. I am reading the file using the pandas function pd.read_csv ...
1
vote
1answer
887 views

Model Parallelism not working in Inception v3 with Keras and TensorFlow

I have been stuck with a problem like this for a while now. I have an AWS setup with 500 GB of RAM and about 7 GPUs. Now the issue is that each time I try to run my Keras with TensorFlow as back-end ...
0
votes
1answer
55 views

Trying to convert a .txt file into .npy file to split and use for train/validation/test

I have a large data set in .txt form, and was trying to split it into three for testing, validation, and training by transforming the data into a ...
0
votes
0answers
19 views

Algorithms or statistical methods for detecting trends based on multiple time series

First of all, I have little experience with data science, statistics, and working with time series, so please excuse my perhaps foolish and amateurish question. I have a data set with many different ...

1
2 3 4 5
…
120

Your privacy

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.