0
$\begingroup$

I'm having trouble with a Machine Learning project, and knowing if I am performing methods properly or not. We are given data from a real satellite, and we have to predict when it will pass through the magnetopause of earth. I have attempted logistic regression, LDA, QDA, Ridge Regression, SVC, SVM with polynomial kernel, sigmoid kernel, and radial kernel, decision trees with boosting, bagging, and random forests, cubic and quadratic splines, and logistic regression with lasso. QDA and Logistic regression gave us a lot of true positives and true negatives, while other models overfitted.

I am wondering if I could get help to see if it is a problem in how I am sampling my data, or performing a method itself.

I'm just going to go over what I tried for boosting.

Here is a summary of the data itself:

   X               Time               DES.N       
  Min.   :     1   Min.   :536594401   Min.   :  0.00  
1st Qu.: 98865   1st Qu.:537362798   1st Qu.:  6.17  
Median :197730   Median :538147162   Median : 15.85  
Mean   :197730   Mean   :538203547   Mean   : 20.31  
3rd Qu.:296594   3rd Qu.:538920402   3rd Qu.: 29.74  
Max.   :395458   Max.   :539955155   Max.   :252.57  

 DES.Vx             DES.Vy              DES.Vz          
Min.   :-2283.52   Min.   :-4295.546   Min.   :-1703.7500  
1st Qu.:  -97.94   1st Qu.:  -74.483   1st Qu.:  -22.1474  
Median :  -56.62   Median :  -33.585   Median :   -0.4084  
Mean   :  -74.63   Mean   :  -33.728   Mean   :   -0.2937  
3rd Qu.:  -19.18   3rd Qu.:    6.444   3rd Qu.:   22.2976  
Max.   : 1060.05   Max.   : 1158.961   Max.   : 1808.3413  

 DES.T_para         DES.T_perp           FGM.Bx        
Min.   :    0.00   Min.   :    0.00   Min.   :-64.7443  
1st Qu.:   38.60   1st Qu.:   39.93   1st Qu.: -4.3714  
Median :   55.19   Median :   55.41   Median :  0.1823  
Mean   :  181.53   Mean   :  200.90   Mean   :  0.6486  
3rd Qu.:   86.27   3rd Qu.:   85.33   3rd Qu.:  5.2224  
Max.   :11490.63   Max.   :12612.85   Max.   : 59.6272  

 FGM.By              FGM.Bz            FGM.Bt        
Min.   :-116.4996   Min.   :-91.529   Min.   :  0.2131  
1st Qu.: -13.0983   1st Qu.: -6.353   1st Qu.: 17.3176  
Median :  -0.1073   Median :  7.070   Median : 28.1939  
Mean   :  -0.9013   Mean   : 14.724   Mean   : 32.5449  
3rd Qu.:   9.9636   3rd Qu.: 39.692   3rd Qu.: 46.0648  
Max.   :  98.9936   Max.   : 97.863   Max.   :122.8706  

 DIS.N              DIS.Vx            DIS.Vy       
Min.   :  0.1976   Min.   :-676.04   Min.   :-417.87  
1st Qu.:  6.9935   1st Qu.:-102.97   1st Qu.: -72.77  
Median : 16.6294   Median : -58.83   Median : -30.66  
Mean   : 22.2347   Mean   : -78.43   Mean   : -33.01  
3rd Qu.: 31.7479   3rd Qu.: -16.20   3rd Qu.:   4.15  
Max.   :294.8410   Max.   : 416.74   Max.   : 401.58  

 DIS.Vz           DIS.T_para        DIS.T_perp     
Min.   :-468.342   Min.   :  16.18   Min.   :  16.04  
1st Qu.: -14.404   1st Qu.: 230.55   1st Qu.: 281.84  
Median :   4.477   Median : 385.32   Median : 500.22  
Mean   :   2.964   Mean   : 806.99   Mean   :1045.55  
3rd Qu.:  21.866   3rd Qu.: 762.22   3rd Qu.: 960.69  
Max.   : 426.431   Max.   :7497.61   Max.   :9805.26  

 Priority        Selected                            Comments     
Min.   :  0.0   Min.   :0.00000   None                     :355974  
1st Qu.:  0.0   1st Qu.:0.00000   Partial BS crossing      :  1707  
Median :  0.0   Median :0.00000   Partial MP               :   945  
Mean   : 12.7   Mean   :0.09984   Partial MP crossing      :   860  
3rd Qu.:  0.0   3rd Qu.:0.00000   Full MP crossing with jet:   727  
Max.   :200.0   Max.   :1.00000   MP                       :   628  
                                  (Other)                  : 34617

It is a large data set, about 400,000 rows.

I sample my data like so:

 set.seed(1)

 #Take only 4,000 rows out of the 400,000
 samples <- sample(1:nrow(MyData), 0.01 * nrow(MyData))
 SampleData <- na.omit(MyData[samples, c(3:18, 20)])

 #Split into 70% training and 30% test set
 samples <- sample(nrow(SampleData), 0.7 * nrow(SampleData))
 SampleData$Selected <- as.factor(SampleData$Selected)

 train = SampleData[samples, ]
 test = SampleData[-samples, ]

Then I do Boosting:

 MyData.boost <- gbm(Selected ~ DES.Vz + DIS.Vx + DIS.Vz, 
 data=train, shrinkage=0.01, distribution = 'bernoulli', 
 cv.folds=5, n.trees=500, verbose=F)

 pred.boost <- predict(MyData.boost, newdata=test, n.trees=500, 
 type="response")
 predict_class <- pred.boost > 0.5
 table(predict_class, test$Selected)
 mean(predict_class == test$Selected)

 predict_class    0    1
         FALSE 1065  112
          TRUE    5    5

 0.9014322

I do not feel these results are correct, what could I be doing wrong? Notice there is 1065 true negatives, it basically predicated zero most the time. Then 112 false positives is also not good.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.