Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,700
questions
3
votes
2
answers
12k
views
How to get a count the number of observations for each year with a Pandas datetime column?
I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year.
This is the column: ...
1
vote
0
answers
371
views
What would be the equivalent of R's mboost in Python?
I am looking for the Python equivalent of R's mboost package ( mboost ).
Would that be xgboost?
3
votes
2
answers
5k
views
A single column has many values per row, separated by a comma. How to create an individual column for each of these?
As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
1
vote
0
answers
104
views
Time array ( 10 min ) for start time to next time in time series import from csv file using python
Here I have time series import from csv file. Start time 7.00 a.m next time is 8.00 a.m next time is 9.00a.m like that it will continue till to midnight. Then again next day it will start with time 7....
3
votes
1
answer
528
views
Creating similarity metric with Doc2Vec and additional features
I have a dataset which contains many features. Each record is company that has many features.
For example...
Company A:
Keywords - data, big data, tableau, dashboards, etc.
Industry - Information ...
1
vote
1
answer
520
views
Target data values are not evenly distributed
Data nature:
I have features with 10 numeric type, and other 10 categorical, with a lot of values, at the end, using one-hot encoding I got a matrix of 600 columns. My problem is with accuracy ...
2
votes
1
answer
186
views
Convert a csv file so that all attributes will become columns with corresponding element [closed]
so I have this dataset in a csv file. I would like to convert it into a matrix form.
The data currently looks like this:
...
5
votes
1
answer
514
views
Find matching text from a text column
This is my first time to use Data Analytics tool to figure out a solution to a problem.
I have a table with following columns ...
4
votes
1
answer
4k
views
Credit scoring using scorecardpy with XGBoost
I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried ...
1
vote
1
answer
348
views
Batch data before feed into CNN network
I am working on a project to classify CT scan images using the CNN model, the image size is huge and I want to feed it into the network using the idea of batches, tried doing that with this pieces of ...
2
votes
0
answers
104
views
Reduce number of vectors in dataset to achieve the "same average dimensions result"? [closed]
Edit for re-opening the question, I'll try to answer questions made by @user2974951:
I have a large user preference statistics for trichotomic data sets. You can visualize each data trio as a 3D ...
0
votes
1
answer
111
views
Predict compatibility of 2 people as boolean classification problem
How can I predict the compatibility of 2 people as a boolean classification problem?
I want to know if below is an appropriate approach to modelling compatibility, or if I should be using "market ...
1
vote
1
answer
614
views
Comparing XGBR with CatBoost performance
I saw on a CatBoost site that it supposed to outperform any other boosted training model and decided to try it myself on a Kaggle's https://www.kaggle.com/c/house-prices-advanced-regression-techniques....
2
votes
1
answer
6k
views
LSTM train on varying length sequences
So I have around 600 classes and 17k sentences for text classification.The feature vectors are trained using word2vec.
Vector_size = 100 ;
LSTM hidden units= 100
I am using class weight for ...
2
votes
1
answer
85
views
Coefficient of determination is close to 1 but the value of RMSE is large. What does it mean?
I am working with the DecisionTreeRegressor and trying to understand how well the data fits the model. I calculated both RMSE ...
0
votes
2
answers
132
views
Face Recognition Using HOG+SVM
This is my approach
...
14
votes
3
answers
21k
views
Increasing SpaCy max NLP limit
I'm getting this error:
...
1
vote
1
answer
226
views
Can I retrain my keras model on small dataset with some empty folders
I am working on a vehicle classification problem.
I trained my model with 605 labels(folders) and 300k photos.
After testing my trained model, I checked my wrong answers' training and validation ...
2
votes
2
answers
75
views
Is it acceptable for a forecasting model to predict moving averaged version of the data?
I'm working on a project in which I'm developing a precipitation forecasting model.
When I try to predict the original data, the model (LSTM neural network) is not able to predict the peaks.
This ...
2
votes
1
answer
456
views
Mismatching target dimension in Keras LSTM
I have built this network for text classification:
...
11
votes
2
answers
4k
views
Is max_depth in scikit the equivalent of pruning in decision trees?
I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I ...
1
vote
1
answer
6k
views
Adding a new custom column in a python data frame [closed]
I need to add a new column in a python data frame that has the dates of January in a column. Each date is repeated 24 times in the column without any intervention. The total number of entries will ...
2
votes
1
answer
202
views
Getting rid of maxpooling layer causes running cuda out memory error pytorch
Video card: gtx1070ti 8Gb, batchsize 64.
I had such UNET with resnet152 as encoder wich worket pretty fine:
...
1
vote
0
answers
901
views
Find partial derivative of softmax w.r.t logits in python
I have trouble implementing back propogation for multi class classification
My neural network has 2 layers
Forward propagation
X -> L1 -> L2
weights W are initialized as random
...
2
votes
1
answer
19k
views
TypeError: '<' not supported between instances of 'int' and 'str'
I have the following code
...
3
votes
0
answers
146
views
Feature Importance Python
My dataset has around 1000 features and 30k rows.
All the feautres have value either 1 or 0. My target variable is Size which 3 classes : Small, Medium and Large. I have around 5k "small" data ...
3
votes
0
answers
462
views
Genarate one hour time interval array using pandas in python (import from csv) to predict next value
I am trying to generate one hour one hour time interval to predict next value according to my data set imported from csv file. Here according to the time it will give outputs include in x column. This ...
1
vote
0
answers
127
views
How to calculate and print unseen time series values for LSTM after train, valid and test data
My dataset is time series data. It contains data of 10 packages. trainX = (1410,3,15), trainY= (1410,1), testX=(110,3,15), testY=(110,1). Input is three feature vectors of last three months and i am ...
4
votes
1
answer
307
views
Can someone please explain what this sample function is upto?
So there is a function in Dino_Name_Generator at Deeplearning.ai notebook
...
1
vote
0
answers
98
views
How to load label data presented in raster format into Keras/Tensorflow
I would like to use a CNN network to segment 2 objects (binary: "0: object not present 1: object present") into shapes, but I have an issue with data. The training data is 150 images and in &...
1
vote
2
answers
3k
views
LSTM future steps prediction with shifted y_train relatively to X_train
I'm trying to predict simple one feature time series data with shifted train data. The source looks like this:
...
1
vote
1
answer
2k
views
Prediction with unseen values in categorical variables
I have created an Artificial Neural Network with 4 features. I am at the point where I want to test the model with a live sample of a malicious file path/exe using:
...
1
vote
6
answers
226
views
How to statistically prove that a column in a dataframe is not needed
I have a pandas dataframe consisting of dimensions and features of a different fabric materials. I have several rows per product material type causing the dataset to seem very huge. From basic logic ...
1
vote
0
answers
79
views
Training a convoltion neural network for localization
I am new into convolutional neural networks and I am trying to build a cnn for localization of object on tensorflow. Currently I am trying to replicate a cnn similar to that of Alexnet and then want ...
16
votes
6
answers
8k
views
Are there any good out-of-the-box language models for python?
I'm prototyping an application and I need a language model to compute perplexity on some generated sentences.
Is there any trained language model in python I can readily use? Something simple like
<...
2
votes
1
answer
520
views
Downsample GPS track
I am working with GPS track files (list of X and Y coordinates). I have tracks with a high sampling rate and want to downsample the track for easier handling.
The obvious way would be to create a new ...
1
vote
1
answer
274
views
Boundaries of Reinforcement Learning
I finally developed a Game Bot that learns how to play the videogame Snake with Deep Q-Learning. I tried with different neural networks and hyper-parameters, and I found a working set-up, for a ...
1
vote
1
answer
2k
views
Voice recognition with fourier transformation with audio input in python
First of all is using the fourier transformation even a good method for recognizing different speakers? I'm not sure if it could recognize a voice if the things that are said are different. I know ...
1
vote
2
answers
912
views
Label Encoder encoding the same class as two integers
First I have defined classes of a label encoder with the keys of a dictionary. Then I have used that label encoder to encode some strings. But for the same string, its giving different integers. Why ...
1
vote
0
answers
207
views
Is there a way to use Orange on a remote server or in the cloud? [closed]
My question there is a way to run the Orange, not on a local computer, but on a remote server, even in the cloud? My notebook has only 8GB memory and limited number of CPU core, which is insufficient ...
2
votes
0
answers
68
views
Continuously train CoreML model after deploy on Server?
I am working on face recognition with CoreML.I make a CoreML model with python Turi Create and put it on Server. User download it and compile on device but there is one problem if we want to do any ...
2
votes
1
answer
1k
views
boosting an xgboost classifier with another xgboost classifier using different sets of features
What I would like to do, is train a first model $f_{1}(\underline{x})$, where $\underline{x}$ is a set of features, fix what model 1 has learned, and then train a second model $f_{2}(\underline{y})$ ...
-2
votes
1
answer
6k
views
How do I add a column to a Pandas dataframe based on other rows and columns in the dataframe? [closed]
I've tried a lot of different methods, but I can't seem to find the right way to do this. I want to create a new column based on the time and ...
1
vote
1
answer
337
views
More weightage to a categorical feature for an Autoencoder model
I am using autoencoder for anomaly detection. I don't have any labels already and so its unsupervised. If I have categorical variables, I usually one hot encode before giving it to the model. I would ...
7
votes
3
answers
2k
views
0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial
My goal is to classify products pictures into categories such as dress, sandals, etc.
I am using the MNIST fashion dataset, following this official tutorial word-per-word: https://www.tensorflow.org/...
1
vote
1
answer
2k
views
What is the significance of underflow during parameter update using stochastic gradient descent?
Background
I am using scikit-learn's MLPRegressor to learn a model with the following arguments:
...
2
votes
0
answers
643
views
Output size is too small for SpatialAveragePooling in Unet
Tried this to use Resnext as encoder in Unet from here, but keep getting RuntimeError: Given input size: (4320x4x4). Calculated output size: (4320x-6x-6). Output size is too small at /opt/conda/conda-...
9
votes
1
answer
5k
views
When does decision tree perform better than the neural network?
I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
6
votes
0
answers
217
views
Julia programming language [closed]
I've heard a lot of buzz for Julia programming language and how it's the future of AI. Should I buy into the hype and start learning Julia? Or do you think its better to stick to a better established ...
1
vote
1
answer
503
views
Value extraction from a python dataframe [ problem statement specific ]
This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that ...