1
$\begingroup$

I want to create a data model out of a .csv file using Python. I mean to create dependencies, for example the primary key and stuff such that I can check if the new .csv complies with the given data model. I would appreciate some suggestions regarding how to do that, the libraries, frameworks etc

$\endgroup$
1
$\begingroup$

Adding on top of @El Burro's answer, most of the training/testing proof of concepts of model making happens on manipulating DataFrame which provides easy functionalities like chaining operations, broadcasting, filling missing values etc and pandas is one such library. It has datatype inferring too and it uses python stack numpy which is fast manipulating arrays.

Other than that, as you asked if you want to check format of incoming data that is being passed to the model. You can use pandas here like (this is just a demo that you can achieve all kinds of stuff, there might be a better way to pull this off)

>> import pandas as pd
>> df1 = pd.DataFrame({'a': [1, 2, 3], 'b': ['a', 'b', 'c']})
>> df1
   a  b
0  1  a
1  2  b
2  3  c
>> df2 = pd.DataFrame({'a': [1, 2, 3], 'b': [1.0, 2.0, 3.0]})
>> df2
   a  b
0  1  1.0
1  2  2.0
2  3  3.0
>> df1.dtypes
a     int64
b    object
dtype: object
>> df2.dtypes
a      int64
b    float64
dtype: object
>> df1.dtypes.to_dict() == df2.dtypes.to_dict()
False
$\endgroup$
  • $\begingroup$ Thanks for the tip @KiriteeGak. I am using pandas myself. $\endgroup$ – zimmer Jan 30 '18 at 13:12
2
$\begingroup$

In my opinion this is probably achieved fastest with a rule set. You can easily load and manipulate data via pandas. Then you just have to determine what are rules. But without any details its difficult to go more into details. Regular expressions might be a powerful tool to use but that is speculation given the sparse info you provide.

$\endgroup$
  • $\begingroup$ Thanks @El Burro. I am using pandas already and thought of making a rule set as well. Just wondering if there's another way to do create a data model. For instance if I want to remove all the columns which do not belong to the predefined data model, I might just have to stick with the rules. $\endgroup$ – zimmer Jan 30 '18 at 13:11
1
$\begingroup$

A loop might give you more control. The csv package lets you read a CSV file line by line.

import csv

# Open the file in read mode. If encoding is an issue, specify the correct encoding here.
with open('my_csv_file.csv', mode='r', encoding='UTF8') as file:

    # Read CSV file. Specify the correct delimiter and how to handle quotation marks.
    lines = csv.reader(file, delimiter=',', quoting=csv.QUOTE_NONE)

    # Exctract header.
    header = next(data)
    ... # Check if the header complies to your data model.

    # Loop through body.
    for line in lines:
        ... # Check if the line complies to your data model.

For the actual checking, have a look at the assert command. For example:

x = 1.0
try:
    assert(isinstance, x, int)    # Check if x is an integer.
except AssertionError:
    x = int(x)    # Handle the error.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.