I'm trying to create a model which predicts Real estate prices with xgboost in machine learning, my question is : Can i combine two datasets to do it ? First dataset : 13 features Second dataset : 100 features Thé différence between the two datasets is that the first dataset is Real estate transaction from 2018 to 2021 with features like area , région And the second is also transaction but from 2011 to 2016 but with more features like balcony, rénovation, and Much more features which are not present in thé first dataset The idea is that i need the first dataset because it's New and actual with New prices inflation And i need the second because i need to include balcony and more features like 5 features only in my prédiction. Can i do that ? And what is the best approach to replace the missing features in first dataset which exist only in the second dataset
1 Answer
# importing pandas library
import pandas as pd
# creating dataframe1
df1 = pd.DataFrame({
"Price" : [1, 2, 3, 4, 2, 3, 8],
"Area" : [20, 30, 40, 50, 30, 54, 45],
"Region" : ["Mumbai", "Delhi", "Hyderabad", "Bangalore", "Hyderabad",
"Bangalore", "Vizag"],
"Year" : [2018, 2018, 2019, 2020, 2020, 2021, 2021],
"NearToSea" : [1, 0, 0, 0, 0, 0, 1]
})
df1.head()
# output
Price Area Region Year NearToSea
0 1 20 Mumbai 2018 1
1 2 30 Delhi 2018 0
2 3 40 Hyderabad 2019 0
3 4 50 Bangalore 2020 0
4 2 30 Hyderabad 2020 0
# creating dataframe2
df2 = pd.DataFrame({
"Price" : [1, 2, 3, 4],
"Area" : [20, 30, 40, 50],
"Region" : ["Mumbai", "Delhi", "Hyderabad", "Bangalore"],
"Balcony" : [2, 1, 0, 3],
"Bedrooms" : [2, 3, 4, 3],
"Kitchen" : [1, 1, 2, 1],
"Year" : [2015, 2016, 2014, 2015]
})
df2.head()
# output
Price Area Region Balcony Bedrooms Kitchens Year
0 1 20 Mumbai 2 2 1 2015
1 2 30 Delhi 1 3 1 2016
2 3 40 Hyderabad 0 4 2 2014
3 4 50 Bangalore 3 3 1 2015
# make list of columns in each dataframe
df1Columns = list(df1.columns)
df2Columns = list(df2.columns)
# Initilize empty list of columns to add to df1 and df2
ExtraColumnsToAddToDf1 = list()
ExtraColumnsToAddToDf2 = list()
# loop through each column in df2
for column in df2Columns:
# if column not in df1
if column not in df1Columns:
# add column to ExtraColumnsToAddToDf1
ExtraColumnsToAddToDf1.append(column)
# loop through each column in df1
for column in df1Columns:
# if column not in df2
if column not in df2Columns:
# add column to ExtraColumnsToAddToDf2
ExtraColumnsToAddToDf2.append(column)
# list of columns that are not present in df1, but exists in df2
ExtraColumnsToAddToDf1
# output
['Balcony', 'Bedrooms', 'Kitchen']
# list of columns that are not present in df2, but exists in df1
ExtraColumnsToAddToDf2
# output
['NearToSea']
# add columns that are not present in dataframe1, 2
for column in ExtraColumnsToAddToDf1:
df1[column] = ""
for column in ExtraColumnsToAddToDf2:
df2[column] = ""
# create a dataframe from those two dataframes
DataFrame = pd.concat([df1, df2], axis = 0)
# shuffling dataframe to involve samples from both dataframe to show at top 5
DataFrame = DataFrame.sample(frac=1)
DataFrame.head()
# output
Price Area Region Year NearToSea Balcony Bedrooms Kitchen
0 3 54 Bangalore 2021 0
1 8 45 Vizag 2021 1
2 1 20 Mumbai 2015 2 2 1
3 2 30 Delhi 2016 1 3 1
4 3 40 Hyderabad 2014 0 4 2
# now we have dataframe that has data from both the files
# we can deal with nan values using df.fillna(), df.interpolate()
refer to pandas fillna documentation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html
since it is a time series data I suggest you to read about pandas interpolation
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html
And also please edit your question into meaningful format, so that all stack exchangers will be able to find what it matters..
-
$\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$– Community BotJun 11, 2022 at 1:08