Shamit Verma
  • Member for 4 years, 2 months
  • Last seen more than a week ago
Curation of a dataset of audio files in different formats
0 votes

If you have enough storage, covert all these to some common format like 320kbps MP3 or WAV. This will save some issue with analysis pipeline (and move these issues to ETL pipeline). Issues are : ...

View answer
Multichannel numpy array to PIL image
2 votes

Try specifying mode so that PIL is aware of data format. img = Image.fromarray(source_array, mode="CMYK") If that does not work, what is the shape of source array ?

View answer
Why is data science not yet widely applied to Law?
1 votes

NLP is very widely used in certain aspects of law. I worked on few use cases related to contract management. While I can't talk about specifics, general areas where NLP is applied are: Distance ...

View answer
Reducing search iteration over millions of data
2 votes

You are on the right track. In an interview situation, this should be a good answer. Another answer would be to pre-calculate list of 20 stores in ETL (or CRUD services or DB triggers) and store list ...

View answer
Text extraction from large pool of documents of different formats
1 votes

One option is to use Apache SOLR + Apache TIKA. Apache TIKA has support for most common file formats, it extracts test content from files. Extracted text can be stored in SOLR. SOLR supports various ...

View answer
Difference - Batch Training or training multiple times?
1 votes

There will not be any difference between : Training all batches + Epoch in one go Saving a checkpoint and resuming the training later Saving a checkpoint is a very good practice. It is not ...

View answer
How to encode a time series as an image to feed it into CNN?
2 votes

1D CNN : You do not have to convert it into an image for CNN. CNN can work directly on time-series (1D Convolution Network). More Details : What is a 1D Convolutional Layer in Deep Learning? ...

View answer
Classifier performance evaluation
Accepted answer
4 votes

Useful metrics in such scenario are: F1 Score (and precision / recall) ROC Curves (Metric is : Area Under the ROC Curve (AUC)) Few articles on how to choose metrics for a specific project are: ...

View answer
Classifying ultrasound videos with a small dataset
1 votes

One option in similar problems is to use transfer learning on raw images. For similar problems (few hundred images for training), solution that worked for me is transfer learning + image pre-...

View answer
How to handle columns with categorical data and many unique values
Accepted answer
7 votes

For categorical columns, you have two options : Entity Embeddings One Hot Vector For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change ...

View answer
How to create word2vec for phrases and then calculate cosine similarity
0 votes

Vector representation of phrases (called term-vectors) are used in projects like search results optimization and question answering. A textbook example is "Chinese river" ~ {"Yangtze_River","...

View answer
How to do webscrapping in R on this webpage?
1 votes

This should be possible with rvest in R. Two things make is possible URL pattern is predictable, https://www.ecb.europa.eu/press/pressconf/2012/html/index.en.html (replace 2012 with other year ...

View answer
Training an acoustic model for a speech-to-text engine
Accepted answer
1 votes

This paper has details on how to prepare Audio data (and merge it with Language Models) for speech-to-text : http://slazebni.cs.illinois.edu/spring17/lec26_audio.pdf Slide 16 has very high level ...

View answer
Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text?
1 votes

This can be solved with "text segmentation". NLP libraries have code for breaking given text into : Sentences Phrases Words With this, you can break text into sentences and insert . or ? for each ...

View answer
Learning curve of CNN model
1 votes

https://en.wikipedia.org/wiki/Overfitting This model is over-fitting. Better train accuracy (and validation accuracy that gets worse with successive iterations) indicates over-fit. For CNN Next ...

View answer
Why image is more blurred through PIL?
0 votes

Try specifying the resampling filter. Default filter (Nearest Neighbor) is fast but results are not as good. image=Image.open("../content/cell_images/Parasitized/"+i) size_image = image....

View answer
How much of data wrangling is a data scientist's job?
26 votes

Feels like most of the work is not related to data science at all. Is this accurate? This is the reality of any data science project. Google actually measured it and published a paper "Hidden ...

View answer
Sentence similarity using Doc2vec
Accepted answer
2 votes

Doc2Vec (and words vectors) need significant amount of data to learn useful vector representation. 50k sentences is not sufficient for this. To overcome this, you can feed word vectors as initial ...

View answer
What kind of algorithm should I choose for this music classification system?
2 votes

There are two high level approaches (Approach 2 was a better fit for a music-classification problem that I worked on) : Signal processing + CNN : Output of signal processing is saved as image. ...

View answer
is it necessary for Artificial NN to be fully connected or only fully connected NN is called ANN?
0 votes

No, there are other kinds of networks such as : RNN (Cells are connected vertically across the same layer) https://en.wikipedia.org/wiki/Recurrent_neural_network CNN (Moves a rectangle across 2 D ...

View answer
Sampling Big Data for Predictive Analytics in Python
1 votes

This is what I do in projects : Pre-process data in DB / Data Lake. aim is to : A. Form batches (might require a new table with shuffled indices) B. Create a copy with Normalization and other feature ...

View answer
How long would it take to become proficient in machine learning for someone with a non-statistical mathematical background?
2 votes

I think you already know enough applied mathematics to begin with. You can pick-up rest of it as required. One option is : Start with an online course that provides high level overview of machine ...

View answer
How to compute f1 in TensorFlow
0 votes

F1 score can be defined as a custom metric. Keras will evaluate this metric on each batch/epoch as applicable. import keras.backend as K def f1_metric(y_true, y_pred): true_positives = K.sum(K....

View answer
How to checkpoint by minibatch in Keras
Accepted answer
1 votes

You have to write a custom callback for this. Steps are : Subclass ModelCheckpoint (https://github.com/keras-team/keras/blob/master/keras/callbacks.py) or create new one if you do not need filename ...

View answer
Can I create a good Speech Recognition Engine while having millions of recorded conversations?
Accepted answer
3 votes

Yes, having lots of recorded conversations is great for building a speech recognition system. You will still have to create training samples (Each sample will be parts of Wave file --> text), but you ...

View answer
Integrate remote sensing & GIS data in a CNN
Accepted answer
1 votes

Keras functional API can be used to create models with multiple inputs (E.g. : Image, Categorical and Numerical). From Geotiff, you can extract : TIFF CSV with other features TIFF will be an input ...

View answer
What is the difference between "offline trained model" and "pretrained model"?
0 votes

Online Model : Model that continuously learns in production. If 10 new training samples are available, we do not need to retrain with all previous samples. Pre-trained model Model has already been ...

View answer
What are CRF (Conditional Random Field)
Accepted answer
1 votes

Stanford CoreNLP is a very good implementation of CRF (In Natural Language Processing domain). https://nlp.stanford.edu/software/CRF-NER.html . CRF specific implementation is : https://github.com/...

View answer
Crop all written letters from image to form a website
Accepted answer
1 votes

This can be done with OpenCV. Code needs to : Prep-process image to enhance contrast Use OpenCV to identify 'bounding box' for each character Save each character's image as a new file Code ...

View answer
Calculating Feature Importance of Time Series Data
Accepted answer
4 votes

For time series data, Sensitivity analysis can help with overall Importance of a feature. For example, is "Day of the week" a good feature for stock price forecasting. LIME is one approach that can ...

View answer