I have structure data with csv or parquet format, I would like to extract the data type of the column by analyzing the data. when I looked at the NER from Hugging phase transformers, it actually dealing with context, but here my data is tabular format. is there any way to extract data type of the each column by analyzing the data, actually volume of the data is so big (GB's), please suggest best solution for this.

  • 1
    $\begingroup$ Can you please clarify what you mean by extracting the data type of each column? Have you tried the using the dtypes property? $\endgroup$ Oct 28, 2022 at 10:14
  • $\begingroup$ Your question is not clear: normally the type of a column is just "string" or "integer" for instance. But you also talk about NER which assigns advanced semantic categories like "person name" or "location" (fyi the only way to do this without context requires a dictionary). Please give an example of input data and what you would expect as type. $\endgroup$
    – Erwan
    Oct 28, 2022 at 10:37
  • $\begingroup$ Hi Erwan & DarknessPlusPlus, Thanks a lot for your response. currently data type of each column is string, by looking at the data, need to assign proper data type based on data, like int, big int, long, double and date etc... i understand when we go with NER, it will give category based on context, even if we get category , we will try map the category and get data type based on category. please let me know apart from NER, is there any better solution to deal with this problem $\endgroup$
    – Anantha
    Oct 28, 2022 at 11:34
  • $\begingroup$ if you are interested in the technical data type only (like int, double, date...), this is not NER at all. Normally detecting types in python is simpler, see for example this and afaik libraries for reading csv files can usually detect the correct type. You should probably give more detail about your data and your specific problem. $\endgroup$
    – Erwan
    Oct 29, 2022 at 15:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.