I have a complete Hadoop platform with HDFS, MR, Hive, PIG, Hbase, etc., Python, R, Java. All data sets have a large size.
The data set A, describing the jobs of people working in a company, is composed of the following fields:
- Id Person: a unique alphanumeric identifier per person.
- Start Date: a date format iso entry in the post
End Date: iso size release date of the position. If the date is not given, it is the current position
Job Title: a text field containing the title and the name of the company. The text is free, non-standardized, French and / or English and can contain typos. Ex: Director Big Data Analytics with Google, Commercial Manager at [missing text] , Manager at googole ...
My question is; how can I create a feature to easily process the name of company of the job (jobtitle)?
Thank you in advance