I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. I suggest you first look at this very similar post, where I discuss some techniques to convert categorical variables to numerical values.
Since you provide little information about your categorical variables, for example how many levels each categorical variable have or how you do label encoding (just out-of-the-box method?) it is hard to give better guidelines. Generally speaking though, label encoding is mostly used for encoding target variables (dependent variable) if it is categorical, while you mention you use label encoding for other 6 independent variables (inputs). Here your target is rather numerical values (regression), so I would go beyond label encoding to convert your independent categorical variables.
Image the following very simplified scenario (just for educational purposes) to understand why label encoding is not a good idea where there is more than 1 independent categorical variables:
Color Year
0 Red 2010
1 Blue 2011
2 Green 2012
Now label encoding this would converting this to (naively speaking):
Color Year
0 1 1
1 2 2
2 3 3
There ways like constructing your own dictionary for each categorical variable and values associated to its levels while label encoding to prevent this obvious mistake. Apart from that, since you hardly go beyond RMSE value of 0.11, one could think of your input values (maybe not encoded properly!) and choice of your model.
Maybe try using this code for handling your independent categorical variables. Please note here, as a general rule of thumb, avoid using one-hot encoding although it is widely used in XGBoost (explained in the post). Maybe try "Target Encoding"!
Now speaking of the choice of model:
- Have you tried regularization (L1, L2)?
- Have confident you are about the independency of your independent variables? In simple regression you are assuming this holds, but may it is not which is easily the case that your independent variables are correlated. Maybe you should analyze this first using confusion matrix (see here for a brief tutorial)
- Have you tried gradient boosting decision trees (GBDT) for regression?
Note1: If you want to go with GBDTs, do not leave that other 1 independent variable alone. You said you have 6 independent variables out of 7. I assume the other one is continuous numerical values? If that so, you have to handle that properly in GBDTs like binarizing (put in bins).
Note2: You may try either XGBoost, or Catboost, and a nice tutorial using Catboost. Each has its pros and cons. For the former hyperparameter tuning could be challenging and there is no ways to automatically account for categorical variables but have larger community. While the latter is recently released (mid last year), it is much easier in terms of hyperparameter tuning (often defaults works great), and there are ways to automatically account for categorical features without explicitly encode them, but less support and community is smaller.
Note3: The good thing about GBDT is that you do not need to worry about the correlation of features per se. Algorithm automatically ignore the coupling of strongly correlated features.
Note4: You may easily overfit, although there are built-in regularization. Catboost for example have a interactive learning curve for training and validation to control overfitting (see the last tutorial).
I strongly believe taking care of your inputs together with a proper choice of algorithm would boost your RMSE error.
