31

Shallow Natural Language Processing technique can be used to extract concepts from sentence. ------------------------------------------- Shallow NLP technique steps: Convert the sentence to lowercase Remove stopwords (these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words) Extract n-gram i.e., a ...


26

For instance: rs<-c("copyright @ The Society of mo","I want you to meet me @ the coffeshop") s<-gsub("@.*","",rs) s [1] "copyright " "I want you to meet me " Or, if you want to keep the @ character: s<-gsub("(@).*","\\1",rs) s [1] "copyright @" "I want you to meet me @" EDIT: If what you want is to remove ...


22

R contains some standard functions for data manipulation, which can be used for data cleaning, in its base package (gsub, transform, etc.), as well as in various third-party packages, such as stringr, reshape/reshape2, and plyr/dplyr. Examples and best practices of usage for these packages and their functions are described in the following paper: http://vita....


22

You can specify the unit of a pandas to_datetime call. Stolen from here: # assuming `df` is your data frame and `date` is your column of timestamps df['date'] = pandas.to_datetime(df['date'], unit='s') Should work with integer datatypes, which makes sense if the unit is seconds since the epoch.


20

There are at least two general considerations to make: Domain-related If an attribute potentially has predictive power in your domain and more specifically for your task your models might benefit from a direct encoding. For example: if being trans is correlated with different psychological disorders then I'd include a direct feature for this. This way it ...


19

One of the references I mentioned in the OP led me to a potential solution that seems quite powerful, described in "Privacy-preserving record linkage using Bloom filters" (doi:10.1186/1472-6947-9-41): A new protocol for privacy-preserving record linkage with encrypted identifiers allowing for errors in identifiers has been developed. The protocol is based ...


18

That paper gives a nice answer, where i quoted from. Search for Should I standardize the target variables (column vectors)? in that page. Standardizing target variables is typically more a convenience for getting good initial weights than a necessity. However, if you have two or more target variables and your error function is scale-sensitive like the ...


17

From my point of view, this question is suitable for a two-step answer. The first part, let us call it soft preprocessing, could be taken as the usage of different data mining algorithms to preprocess data in such a way that makes it suitable for further analyses. Notice that this could be the analysis itself, in case the goal is simple enough to be tackled ...


16

Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving beyond this level of metadata. I would especially pick a format that can be used regardless of development environment and one that can keep some basic structure ...


15

1. Do not modify the original data Having the original data source intact is important. You may find that updates you make to the data are not valid. You may also find a more efficient way to make updates and you will want to regression test those updates. Always work with a copy of the data, and add columns/properties/metadata that includes any ...


15

Well if the dataset is not too large I would suggest using pandas to clean the data. So you would need to first do Python2 python2 -m pip install pandas Python3 python3 -m pip install pandas If you already have anaconda installed you can skip the above step. Next you could go through an IDE (like jupyter) or through the shell type the following ...


14

R and SQL are two completely different beasts. SQL is a language that you can use to query data that is stored in databases as you already experienced. The benefits of SQL versus R lays mostly in the fact of the database server (MS SQL, Oracle, PostgreSQL, MySQL, etc.). Most, if not all, modern database servers permit multiple users to query data from the ...


13

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart. A common step I think you have missed is imputation of missing values. I would put that before feature engineering. Overall I ...


12

Even if you are effectively modifying certain records by hand, as in the city name example you give, I would recommend doing it in code. The reason to strongly prefer code over hand-tweaking records is that the code makes a result reproducible. You want to make sure that you can always go from raw data to final result without any human intervention. Here's ...


12

The error here seems to be because you want train and test data (so two data sets), meaning that each class must be present in each of the data sets. This would mean that each class must have at least two samples. It is a design choice of whoever implemented train_test_split. I guess it might not technically be stratified otherwise. You can see where it is ...


12

As a rule of thumb, removing outliers without a good reason to remove outliers rarely does anyone any good. Without a deep and vested understanding of what the possible ranges exist within each feature, then removing outliers becomes tricky. Often times, I see students/new hires plot box-plots or check mean and standard deviation to determine an outlier ...


11

One reason that data cleaning is rarely fully automated is that there is so much judgment required to define what "clean" means given your particular problem, methods, and goals. It may be as simple as imputing values for any missing data, or it might be as complex as diagnosing data entry errors or data transformation errors from previous automated ...


11

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see: The current context is very favorable to data scientists looking for a job, so anybody with some basic knowledge of ML has a chance. You're already above this level so ...


10

As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for linking . See for more information on the Jaro and Jaro-Winkler distance in this journal. For a comparison of different matching techniques, read this paper


10

They are used for two different purposes. StandardScaler changes each feature column $f_{:,i}$ to $$f'_{:,i} = \frac{f_{:,i} - mean(f_{:,i})}{std(f_{:,i})}.$$ Normalizer changes each sample $x_n=(f_{n,1},...,f_{n,d})$ to $$x'_n = \frac{x_n}{size(x_n)},$$ where $size(x_n)$ for l1 norm is $\left \| x_n \right \|_1=|f_{n,1}|+...+|f_{n,d}|$, l2 norm is $\...


9

You can use Multi-label data stratification in skmultilearn library


9

It is quite an interesting question. I guess that you can call it "dealing with non-binary gender roles in a binary language" or something like this. In the past I did once something similar. I created 3 features: sex at birth [male,female] sex identification [male,female] Attracted sexually to [male,female]. All these features are binary and you can ...


8

Halfway through reading your question, I realized Levenshtein Distance could be a nice solution to your problem. Its good to see that you have a link to a paper on the topic, let me see if I can shed some light into what a Levenshtein solution would look like. Levenshtein distance is used across many industries for entity resolution, what makes it useful ...


8

I'd start with the melt() function in pandas. I wrote an article about it: https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python?lang=en


8

The ten times rule seems like a rule of thumb to me, but it is true that the performance of your machine learning algorithm may decrease if you do not feed it with enough training data. A practical and data-driven way of determining whether you have enough training data is by plotting a learning curve, like the one in the example below: The learning curve ...


8

You simply need to do: df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern) df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})


7

The brat annotation tool might be useful for you as per my comment. I have tried many of them and this is the best I have found. It has a nice user interface and can support a number of different types of annotations. The annotations are stored in a separate .annot file which contain each annotation as well as its location within the original document. A ...


7

In general, you don't want to use XML tags to tag documents in this way because tags may overlap. UIMA, GATE and similar NLP frameworks denote the tags separate from the text. Each tag, such as Person, ACME, John etc. is stored as the position that the tag begins and the position that it ends. So, for the tag ACME, it would be stored as starting a ...


7

If feasible I would link related records (e.g., Dave, David, etc.) and replace them with a sequence number (1,2,3, etc.) or a salted hash of the string that is used to represent all related records (e.g., David instead of Dave). I assume that third parties need not have any idea what the real name is, otherwise you might as well give it to them. edit: You ...


7

There's a much more pythonic solution in pandas... This takes less than a second on 10 Million rows on my laptop: for x in X11.E.unique(): X11[x]=(X11.E==x).astype(int) X11 Here are the details laid out: Simple small dataframe - import numpy as np import pandas as pd X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD')) X11['E'] = [25223, ...


Only top voted, non community-wiki answers of a minimum length are eligible