Hot answers tagged

11

If your categorical columns are currently character/object you can use something like this to do each one: char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index for c in char_cols: df[c] = pd.factorize(df[c])[0] If you need to be able to get back to the categories I'd create a dictionary to save the encoding; something like: char_cols = df....


9

Update: we have updated TRAINSET to include the ability to upload multiple series as well as apply multiple labels! See demo in GIF below. We had this same problem again and again at Geocene, so we came up with this open-source web app called TRAINSET. You can use TRAINSET to brush labels onto time series data. You import data in a defined CSV format, then ...


9

Yes, you should do this. Given the initialization schemes and normalized inputs, the expected values for the outputs are 0. This means that you will not be too far off from the start, which helps convergence. If your target is 1000, your mean squared error will be huge which means your gradients will also be huge which can lead to numerical instabiliy.


6

First, let's create a mcve to play with: import pandas as pd import numpy as np In [1]: categorical_array = np.random.choice(['Var1','Var2','Var3'], size=(5,3), p=[0.25,0.5,0.25]) df = pd.DataFrame(categorical_array, columns=map(lambda x:chr(97+x), range(categorical_array.shape[1]))) ...


5

If you are specifically looking to outline the whales, seems like FastAnnotationTool could work: https://github.com/christopher5106/FastAnnotationTool Other options here: https://en.wikipedia.org/wiki/List_of_Manual_Image_Annotation_Tools When had to annotate many images for a project, I built a fairly simple MATLAB gui that displayed images. I cycled ...


5

Label flipping is a training technique where one selectively manipulates the labels in order to make the model more robust against label noise and associated attacks - the specifics depend a lot on the nature of the noise. Label flipping bears no benefit only under the assumption that all labels are (and will always be) correct and that no adversaries exist. ...


4

With t-SNE none of the input parameters are weighted more than any other parameter so the differences you want to see like students forming islands by grade level will not happen because there is so much other data present to pull those students/data points in different directions. I highly encourage you to have a specific question in mind and tailor your ...


4

It sounds like you are looking for active learning. In active learning, the classifier learns which samples would be most useful to have labelled by a human. There are many techniques for active learning, and many ways to adapt an existing (standard) learning algorithm to the active learning setting. The particular approach you mentioned is called "...


4

Audacity is a free and open source audio editing software, available for all common desktop operating systems. It can be used to annotate audio, by using Label Tracks. You can have a label at a certain position, or covering a selection of time. Annotating Open or import your audio into Audacity Select the audio track you want to annotate. Click Track -> ...


4

TL;DR Yes, with overfitting all data becomes (non-linearly) separable (as long as the points don't precisely overlap). Explanation The problem with your argument is that you are using circles on a 2D plane, which is very difficult to learn. However, I think your argument can be made stronger with a decision-tree. (0.2, 3.1)? --> yes -> star ...


4

So when we generate labels via machine learning models like clustering above, is it a recommended approach? Only if you can really make highly distinct 2 clusters/groups. This will be highy unlikely, especially for complicated and high dimensional datasets. One of the reasons is that clustering algorithms are just weaker than the supervised algorithms. If ...


4

The way most people gain an initial understanding of label smoothing (and what most common explanations have to say on the subject) plays a great role in how one would approach this question. At first glance, label smoothing is exactly what the name suggests: we modify the labels or some portion of them in order to get a better, more general, more robust ...


3

This question is a very common question. How to handle samples belong to categories that are not included in the data set. There is no optimal way to handle this: Creating a separate class is possible but not recommended. First, you need images to start with, second this class will have a very large variation within the class, a property that is not ...


3

I am currently developing a set of tools to annotate and detect patterns in time series data: https://github.com/avenix/WDK check the AnnotationApp in 1-Annotation


3

Here's a great simple tool. It's fully in Python so you can play around with it to fit your needs more properly.


3

Having consulted my professor, the person that wrote the question from the exercise book featured in the OP, here is their perspective: Groups of data points can always be separated. The exception is when two points are at the same location. However, the thing to consider is whether or not your decision boundary can separate unseen data, generated by the ...


2

I suggest you try building multiple networks: one network for building type (outputs "house", "apartment", or "condominum"), another network for build year, another for garage (yes vs no), and so on. This keeps the number of classes small for each network, and allows each network to tune itself for the specific task it is focusing on. If you want to avoid ...


2

Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code. Yes. You can treat your automated labelling as a model in its own right (it essentially is an "expert" model, that takes in additional features). Collect ground truth data with known accurate labels, and use a metric such as accuracy, AUROC,...


2

In order to convert types of multiple columns at once I would use something like this : df2 = df.select_dtypes(include = ['type_of_insterest']) df2[df2.columns].apply(lambda x:x.astype('category')) Then I would join them back to original df.


2

There are several tools for this you could check the Stanford Simple Annotation Tool, also Brat has online demos. For a comprehensive list you could check this question on Quora


2

I think your best bet would be transfer learning. Start with a model that has already been trained with a wider dataset such as the ones presented here. From there you can train the model with your specific dataset. You can then use output nodes for the labels which you have available to you, and you can get the predictions for the other images from the pre-...


2

This type of problem is considered to be part of 'active learning'. There is a lot of research being done on this topic at the moment, but some first approaches are relatively easy, depending on the type of model that you are using. Since you mentioned that you are using deep learning bounding box detectors, I will showcase a few examples of how to approach ...


2

In general, you want to show your model all types of "resistors" it would see in the real world. Is it feasible for you to manually go through your dataset and label each of these images as resistor A or resistor B? Do you care if your model can make this distinction? If the answer to either of these is no, I would just leave them all as one label and ...


2

The chicken egg dilemma What came first: the labeled data or the machine learning model? If you have labeled data, then you can train a machine learning model. If you have a trained machine learning model, then you can label data. Precision vs. Recall Suppose you are in case 2 (you have a model). Then how is the precision and recall of your model? Unless ...


2

I also need such a tool to annotate data but did not found any suitable tool. Therefore, i wrote a small python app by myself, just abused matplotlib for this task. I used matplotlib.use('TkAgg') and SpanSelector with my own onselect(xmin, xmax) method called for this task. Check this code example: https://matplotlib.org/gallery/widgets/span_selector.html


2

Give a shot to DataTurks It gives you simply slick UI + web based collaborative framework to work with your colleagues/mates to build the dataset. Image from a sample Project which classifies images to celberity Some more sample projects: https://dataturks.com/projects/trending


2

There is no common practice in labeling the bounding boxes. It is always problem dependent. For example, if you want to count the chickens then you should also label the whole chicken as one instance of a chicken. If you simply what to detect if there is a chicken in the picture you should label the unoccluded part. You have to think about your problem. ...


2

Yes! This could be an excellent test case for your classification algorithm. With only 5% mislabeling a good algorithm will be easily able to identify "outliers" by having much worse predictions for these mislabeled records. If you were able to at least identify "correct" records to generate a training set that would be even better but with 5% mislabeling ...


2

There are several general approaches to this but almost all of them compare the values to some baseline and make a decision whether the individual value is close enough. You can compare the similarity of strings using different methods e.g. I often use the Levenshtein distance which basically measures how many characters you would have to change to convert ...


2

This seems intuitive. When we have a single-label response (Multi-Class setting), our response prior to encoding looks like: [1, 4, 2, 3, 5] After encoding,it becomes [[1,0,0,0,0], [0,0,0,1,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,0,1]] Indicating the presence of ith Class. But only one. Similarly, in a multi-label setting- Input should be - [[1,2,5], [4,1], [...


Only top voted, non community-wiki answers of a minimum length are eligible