Shamit Verma
  • Member for 4 years, 2 months
  • Last seen more than a week ago
How much of data wrangling is a data scientist's job?
26 votes

Feels like most of the work is not related to data science at all. Is this accurate? This is the reality of any data science project. Google actually measured it and published a paper "Hidden ...

View answer
How to handle columns with categorical data and many unique values
Accepted answer
7 votes

For categorical columns, you have two options : Entity Embeddings One Hot Vector For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change ...

View answer
Date Extraction in Python
6 votes

Stanford CoreNLP has a very good implementation of NER for date/time. https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process) Though this is written in ...

View answer
What is a good way to store processed CSV data to train model in Python?
5 votes

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second. Most of the time is going to be spent by dataframe runtime in parsing data and creation of ...

View answer
Classifier performance evaluation
Accepted answer
4 votes

Useful metrics in such scenario are: F1 Score (and precision / recall) ROC Curves (Metric is : Area Under the ROC Curve (AUC)) Few articles on how to choose metrics for a specific project are: ...

View answer
Calculating Feature Importance of Time Series Data
Accepted answer
4 votes

For time series data, Sensitivity analysis can help with overall Importance of a feature. For example, is "Day of the week" a good feature for stock price forecasting. LIME is one approach that can ...

View answer
Products classification by name
4 votes

This should be doable with pre-trained word vectors + document/sentence vectors. Tutorial : https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e All Product labels with "...

View answer
Loss is decreasing but val_loss not!
4 votes

This indicates that model is not generalizing (it is over-fitting). Few options are : Get more training data Reduce complexity of model (Number of LSTM layers, complexity of dense layers) Andrew NG ...

View answer
How to create clusters based on sentence similarity?
4 votes

For this problem, I would start with a simple bag of words model and use that as a baseline. This is an example : https://medium.com/@MSalnikov/text-clustering-with-k-means-and-tf-idf-f099bcf95183 , ...

View answer
Multivariate Time Series Binary Classification
Accepted answer
3 votes

You can add all features as input to RNN/LSTM (Day #, F1, F2, ... F5) and binary class as output. This article has an example of such network.

View answer
Can I create a good Speech Recognition Engine while having millions of recorded conversations?
Accepted answer
3 votes

Yes, having lots of recorded conversations is great for building a speech recognition system. You will still have to create training samples (Each sample will be parts of Wave file --> text), but you ...

View answer
How can I detect blocks of text from scanned document images
3 votes

There are two options : Scan the images with a higher DPI. This should accentuate vertical separation between paragraphs. Train a Deep learning model for Text Detection in scene. Examples : https://...

View answer
Understanding LSTM structure
3 votes

Adding model.summary() Produces this output _________________________________________________________________ Layer (type) Output Shape Param # ====================...

View answer
ML algorithm for Music Features
3 votes

If the problem you are trying to solve is "I need to guess if the new songs are fit to user's taste with the song feature data I got." AND if you have some indication what user has liked / listened-...

View answer
Sentence similarity using Doc2vec
Accepted answer
2 votes

Doc2Vec (and words vectors) need significant amount of data to learn useful vector representation. 50k sentences is not sufficient for this. To overcome this, you can feed word vectors as initial ...

View answer
Multichannel numpy array to PIL image
2 votes

Try specifying mode so that PIL is aware of data format. img = Image.fromarray(source_array, mode="CMYK") If that does not work, what is the shape of source array ?

View answer
Reducing search iteration over millions of data
2 votes

You are on the right track. In an interview situation, this should be a good answer. Another answer would be to pre-calculate list of 20 stores in ETL (or CRUD services or DB triggers) and store list ...

View answer
How to encode a time series as an image to feed it into CNN?
2 votes

1D CNN : You do not have to convert it into an image for CNN. CNN can work directly on time-series (1D Convolution Network). More Details : What is a 1D Convolutional Layer in Deep Learning? ...

View answer
What kind of algorithm should I choose for this music classification system?
2 votes

There are two high level approaches (Approach 2 was a better fit for a music-classification problem that I worked on) : Signal processing + CNN : Output of signal processing is saved as image. ...

View answer
How long would it take to become proficient in machine learning for someone with a non-statistical mathematical background?
2 votes

I think you already know enough applied mathematics to begin with. You can pick-up rest of it as required. One option is : Start with an online course that provides high level overview of machine ...

View answer
Gradient Descent Convergence
2 votes

Gradient Descent does not always converge to Global minima. It only converges if function is convex and learning rate is appropriate. For most real life problems, function will have local minimums ...

View answer
How to decide the processing power required based on the dataset?
Accepted answer
2 votes

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another. For example, linear regression will be much faster as compared to 4 layer neural network (and will ...

View answer
Correlation between Time Series Indicators ( Stock Prices )
2 votes

For time series, correlation is a different. A variable might b related past N values of other variables. This article explains theory behind finding relationships in time series (Skip to section "...

View answer
Can feedback loops occur in machine learning that cause the model to become less precise?
2 votes

Yes, this is a real problem that manifests once system is used by real users. Most prominent example is News Echo Chamber (accentuated by ML based recommendation systems) ML algo sees that you ...

View answer
Splitting a neural network in 2 microservices
2 votes

Yes, you can split the model into two parts after training. Not sure, what would be the advantage, but it is possible. High level steps : Define and train the model Save trained model Create two ...

View answer
How to reshape data for LSTM training in multivariate sequence prediction
Accepted answer
2 votes

This makes sense. It should work for input and first couple of layers. For output layers, you can have a softmax if you need to generate only next record in sequence. Following Keras code has an ...

View answer
operating on a dataset with 125,497,040 records
2 votes

For 127 million rows, it is better to perform data prep on DB. It will be a select + insert query and will not require whole data to be loaded in memory. SELECT YEAR(date) AS 'year', MONTH(date) AS '...

View answer
Help in NLP Problem
2 votes

Next steps should be : Build a "Bag of Words" classifier as a baseline. Example : https://medium.freecodecamp.org/text-classification-and-prediction-using-bag-of-words-8aeb1396cded Build a ...

View answer
Protein interaction prediction- how to input this data structure
Accepted answer
2 votes

https://en.wikipedia.org/wiki/Adjacency_matrix For such problems, you can tabulate these connections as adjacency matrix and create a network to predict weights for the matrix given some properties ...

View answer
Deep learning performance on classifying simple geometric figures
2 votes

A CNN network should easily achieve close to 100% accuracy on this task. Few aspects make this task easier : Clean High-contrast images generated from program Only 1 shape per image Given few ...

View answer