Merging two datasets with different features for machine learning prediction

Question

I'm trying to create a model which predicts Real estate prices with xgboost in machine learning, my question is : Can i combine two datasets to do it ? First dataset : 13 features Second dataset : 100 features Thé différence between the two datasets is that the first dataset is Real estate transaction from 2018 to 2021 with features like area , région And the second is also transaction but from 2011 to 2016 but with more features like balcony, rénovation, and Much more features which are not present in thé first dataset The idea is that i need the first dataset because it's New and actual with New prices inflation And i need the second because i need to include balcony and more features like 5 features only in my prédiction. Can i do that ? And what is the best approach to replace the missing features in first dataset which exist only in the second dataset

Answer 1

# importing pandas library
import pandas as pd

# creating dataframe1
df1 = pd.DataFrame({
"Price" : [1, 2, 3, 4, 2, 3, 8],
"Area" : [20, 30, 40, 50, 30, 54, 45],
"Region" : ["Mumbai", "Delhi", "Hyderabad", "Bangalore", "Hyderabad", 
"Bangalore", "Vizag"],
"Year" : [2018, 2018, 2019, 2020, 2020, 2021, 2021],
"NearToSea" : [1, 0, 0, 0, 0, 0, 1]
})
df1.head()

# output
       Price    Area    Region     Year   NearToSea
   0    1       20      Mumbai     2018   1
   1    2       30      Delhi      2018   0
   2    3       40      Hyderabad  2019   0
   3    4       50      Bangalore  2020   0
   4    2       30      Hyderabad  2020   0

# creating dataframe2
df2 = pd.DataFrame({
"Price" : [1, 2, 3, 4],
"Area" : [20, 30, 40, 50],
"Region" : ["Mumbai", "Delhi", "Hyderabad", "Bangalore"],
"Balcony" : [2, 1, 0, 3],
"Bedrooms" : [2, 3, 4, 3],
"Kitchen" : [1, 1, 2, 1],
"Year" : [2015, 2016, 2014, 2015]    
})
df2.head()

# output
   Price    Area    Region     Balcony  Bedrooms   Kitchens   Year
0   1       20      Mumbai     2        2           1         2015
1   2       30      Delhi      1        3           1         2016
2   3       40      Hyderabad  0        4           2         2014
3   4       50      Bangalore  3        3           1         2015

# make list of columns in each dataframe
df1Columns = list(df1.columns)
df2Columns = list(df2.columns)

# Initilize empty list of columns to add to df1 and df2
ExtraColumnsToAddToDf1 = list()
ExtraColumnsToAddToDf2 = list()

# loop through each column in df2
for column in df2Columns:
    # if column not in df1
    if column not in df1Columns:
        # add column to ExtraColumnsToAddToDf1
        ExtraColumnsToAddToDf1.append(column)

# loop through each column in df1
for column in df1Columns:
    # if column not in df2
    if column not in df2Columns:
        # add column to ExtraColumnsToAddToDf2
        ExtraColumnsToAddToDf2.append(column)


# list of columns that are not present in df1, but exists in df2
ExtraColumnsToAddToDf1

# output
['Balcony', 'Bedrooms', 'Kitchen']

# list of columns that are not present in df2, but exists in df1
ExtraColumnsToAddToDf2

# output
['NearToSea']

# add columns that are not present in dataframe1, 2
for column in ExtraColumnsToAddToDf1:
    df1[column] = ""
for column in ExtraColumnsToAddToDf2:
    df2[column] = ""

# create a dataframe from those two dataframes
DataFrame = pd.concat([df1, df2], axis = 0)
# shuffling dataframe to involve samples from both dataframe to show at top 5
DataFrame = DataFrame.sample(frac=1)
DataFrame.head()

# output
   Price    Area    Region      Year    NearToSea   Balcony  Bedrooms   Kitchen
0   3       54      Bangalore   2021    0           
1   8       45      Vizag       2021    1           
2   1       20      Mumbai      2015                2        2          1
3   2       30      Delhi       2016                1        3          1
4   3       40      Hyderabad   2014                0        4          2

# now we have dataframe that has data from both the files
# we can deal with nan values using df.fillna(), df.interpolate()

refer to pandas fillna documentation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html
since it is a time series data I suggest you to read about pandas interpolation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html
And also please edit your question into meaningful format, so that all stack exchangers will be able to find what it matters..