0
$\begingroup$

How do I correct my data or format it so that it is presentable, and fix my graphs?

  • Dataset is 345551 rows × 7 columns.
  • I am using numpy, pandas, seaborn and matplot lib.
  • It seems that my pricing data is being displayed in scientific notation.

When I fit a linear regression model I get the following coefficients

property_type= -3.096186e+05
new_build= -1.909146e+04

When I use a train/test split and check my predictions they don't make sense.

index=246862
actual=440000
predicted=4.252606e+05

graph data is labelled in scientific notation distribution is skewed left, and logged in scientific notation matplotlib scatter graph does not render correctly, why?

$\endgroup$
1
  • $\begingroup$ if you provide code to make dummy data, and to make plots then answerers can show you results in the language you are asking the question: visually. $\endgroup$ – EngrStudent Aug 26 '20 at 15:36
0
$\begingroup$

Try using numpy's log or log1p. The math module also offers logarithmic functionality.

You can use sklearn to transform your variables - see MinMaxScaler, StandardScaler, etc.

Other transformations like Box-Cox can be found in scipy.stats

$\endgroup$
1
  • $\begingroup$ Thanks, this answers part of my question, I'll investigate and see if it solves the overall issue. $\endgroup$ – davidmaceachern Jul 1 '19 at 15:46
0
$\begingroup$

To clearly visualise your data without worrying about scientific notation, you can transform your data by using either of methods MinMaxScalar , StandardScalar or Normalisation technique. Apart from that there are other data transformation methods but you can use either of the above methods.

Now you can use these transformed data to train your model. Of course parameters value will be different after fitting model with transformed data than fitting model with original data. To get the actual output you need to transform the individual input by using the same method and then convert the output into original scale by using the reverse of the same method.

For example, you transform your data by dividing it with max of your data and trained your model. Now you have an individual input to predict the output. So you have to devide the input with same max data. After getting the output, you need to perform the reverse transformation means multiply the output with max data.

Note: Each column will have their own max, so use respective column's max to transform. Similarly target column will also be having it's max value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.