0
$\begingroup$

When calling CoxPHFitter() on my full dataset I'm getting the following error:

Users/anaconda3/lib/python3.7/site-packages/lifelines/fitters/coxph_fitter.py:557: ConvergenceWarning: Newton-Rhapson failed to converge sufficiently in 50 steps.
  warnings.warn("Newton-Rhapson failed to converge sufficiently in %d steps." % max_steps, ConvergenceWarning)

But I couldn't find how to increase number of steps. I've also tried to play with the values of the params: step_size, enalizer and alpha - with no success.

This is the function that I'm running and the params:

def cox_proportional_hazard_model(data, survival_duration, survival_status, strata=None):

    cph = CoxPHFitter(alpha=0.05, tie_method='Efron', penalizer=0.1, strata=None)

    cph.fit(df=data, 
            duration_col=survival_duration, event_col=survival_status, 
            strata=strata, show_progress=True, step_size=0.1)

    cph.print_summary()

    return cph

and here is the output and the deltas:

Iteration 1: norm_delta = 22.95175, step_size = 0.1000, ll = -383.78983, newton_decrement = 224.62787, seconds_since_start = 0.0
Iteration 2: norm_delta = 8.59969, step_size = 0.0250, ll = -344.73687, newton_decrement = 100.90631, seconds_since_start = 0.1
Iteration 3: norm_delta = 8.00526, step_size = 0.0225, ll = -339.71541, newton_decrement = 95.05309, seconds_since_start = 0.1
Iteration 4: norm_delta = 7.61510, step_size = 0.0243, ll = -335.44970, newton_decrement = 90.63796, seconds_since_start = 0.1
Iteration 5: norm_delta = 7.29316, step_size = 0.0262, ll = -331.05741, newton_decrement = 86.53240, seconds_since_start = 0.2
Iteration 6: norm_delta = 7.02757, step_size = 0.0283, ll = -326.52929, newton_decrement = 82.69935, seconds_since_start = 0.2
Iteration 7: norm_delta = 6.80949, step_size = 0.0306, ll = -321.85618, newton_decrement = 79.10759, seconds_since_start = 0.2
Iteration 8: norm_delta = 6.63229, step_size = 0.0331, ll = -317.02892, newton_decrement = 75.73047, seconds_since_start = 0.3
Iteration 9: norm_delta = 6.49106, step_size = 0.0357, ll = -312.03837, newton_decrement = 72.54478, seconds_since_start = 0.3
Iteration 10: norm_delta = 6.38213, step_size = 0.0386, ll = -306.87533, newton_decrement = 69.52988, seconds_since_start = 0.3
Iteration 11: norm_delta = 6.30276, step_size = 0.0416, ll = -301.53059, newton_decrement = 66.66703, seconds_since_start = 0.3
Iteration 12: norm_delta = 6.25096, step_size = 0.0450, ll = -295.99496, newton_decrement = 63.93876, seconds_since_start = 0.4
Iteration 13: norm_delta = 6.22523, step_size = 0.0486, ll = -290.25932, newton_decrement = 61.32840, seconds_since_start = 0.4
Iteration 14: norm_delta = 6.22451, step_size = 0.0525, ll = -284.31480, newton_decrement = 58.81965, seconds_since_start = 0.4
Iteration 15: norm_delta = 6.24798, step_size = 0.0567, ll = -278.15291, newton_decrement = 56.39613, seconds_since_start = 0.5
Iteration 16: norm_delta = 6.29497, step_size = 0.0500, ll = -271.76578, newton_decrement = 54.04106, seconds_since_start = 0.5
Iteration 17: norm_delta = 6.35101, step_size = 0.0441, ll = -266.33229, newton_decrement = 52.14109, seconds_since_start = 0.5
Iteration 18: norm_delta = 6.40935, step_size = 0.0389, ll = -261.68076, newton_decrement = 50.57648, seconds_since_start = 0.6
Iteration 19: norm_delta = 6.46639, step_size = 0.0343, ll = -257.67951, newton_decrement = 49.26847, seconds_since_start = 0.6
Iteration 20: norm_delta = 6.52023, step_size = 0.0302, ll = -254.22460, newton_decrement = 48.16262, seconds_since_start = 0.6
Iteration 21: norm_delta = 6.57000, step_size = 0.0267, ll = -251.23237, newton_decrement = 47.21966, seconds_since_start = 0.7
Iteration 22: norm_delta = 6.61537, step_size = 0.0235, ll = -248.63436, newton_decrement = 46.41034, seconds_since_start = 0.7
Iteration 23: norm_delta = 6.65633, step_size = 0.0207, ll = -246.37393, newton_decrement = 45.71217, seconds_since_start = 0.7
Iteration 24: norm_delta = 6.69308, step_size = 0.0183, ll = -244.40374, newton_decrement = 45.10750, seconds_since_start = 0.8
Iteration 25: norm_delta = 6.72589, step_size = 0.0161, ll = -242.68394, newton_decrement = 44.58214, seconds_since_start = 0.8
Iteration 26: norm_delta = 6.75508, step_size = 0.0142, ll = -241.18077, newton_decrement = 44.12455, seconds_since_start = 0.8
Iteration 27: norm_delta = 6.78100, step_size = 0.0126, ll = -239.86547, newton_decrement = 43.72517, seconds_since_start = 0.9
Iteration 28: norm_delta = 6.80396, step_size = 0.0111, ll = -238.71344, newton_decrement = 43.37602, seconds_since_start = 0.9
Iteration 29: norm_delta = 6.82427, step_size = 0.0098, ll = -237.70355, newton_decrement = 43.07037, seconds_since_start = 0.9
Iteration 30: norm_delta = 6.84223, step_size = 0.0086, ll = -236.81763, newton_decrement = 42.80250, seconds_since_start = 1.0
Iteration 31: norm_delta = 6.85809, step_size = 0.0076, ll = -236.03994, newton_decrement = 42.56752, seconds_since_start = 1.0
Iteration 32: norm_delta = 6.87209, step_size = 0.0067, ll = -235.35688, newton_decrement = 42.36124, seconds_since_start = 1.0
Iteration 33: norm_delta = 6.88445, step_size = 0.0059, ll = -234.75663, newton_decrement = 42.18003, seconds_since_start = 1.1
Iteration 34: norm_delta = 6.89535, step_size = 0.0052, ll = -234.22893, newton_decrement = 42.02075, seconds_since_start = 1.1
Iteration 35: norm_delta = 6.90497, step_size = 0.0046, ll = -233.76482, newton_decrement = 41.88069, seconds_since_start = 1.1
Iteration 36: norm_delta = 6.91345, step_size = 0.0041, ll = -233.35650, newton_decrement = 41.75748, seconds_since_start = 1.2
Iteration 37: norm_delta = 6.92093, step_size = 0.0036, ll = -232.99717, newton_decrement = 41.64906, seconds_since_start = 1.2
Iteration 38: norm_delta = 6.92753, step_size = 0.0032, ll = -232.68085, newton_decrement = 41.55361, seconds_since_start = 1.2
Iteration 39: norm_delta = 6.93335, step_size = 0.0028, ll = -232.40235, newton_decrement = 41.46957, seconds_since_start = 1.3
Iteration 40: norm_delta = 6.93847, step_size = 0.0025, ll = -232.15707, newton_decrement = 41.39556, seconds_since_start = 1.3
Iteration 41: norm_delta = 6.94300, step_size = 0.0022, ll = -231.94104, newton_decrement = 41.33037, seconds_since_start = 1.3
Iteration 42: norm_delta = 6.94698, step_size = 0.0019, ll = -231.75071, newton_decrement = 41.27294, seconds_since_start = 1.4
Iteration 43: norm_delta = 6.95050, step_size = 0.0017, ll = -231.58303, newton_decrement = 41.22233, seconds_since_start = 1.4
Iteration 44: norm_delta = 6.95360, step_size = 0.0015, ll = -231.43526, newton_decrement = 41.17774, seconds_since_start = 1.4
Iteration 45: norm_delta = 6.95634, step_size = 0.0013, ll = -231.30504, newton_decrement = 41.13844, seconds_since_start = 1.5
Iteration 46: norm_delta = 6.95875, step_size = 0.0012, ll = -231.19026, newton_decrement = 41.10379, seconds_since_start = 1.5
Iteration 47: norm_delta = 6.96087, step_size = 0.0010, ll = -231.08909, newton_decrement = 41.07326, seconds_since_start = 1.5
Iteration 48: norm_delta = 6.96275, step_size = 0.0009, ll = -230.99991, newton_decrement = 41.04634, seconds_since_start = 1.6
Iteration 49: norm_delta = 6.96440, step_size = 0.0008, ll = -230.92129, newton_decrement = 41.02261, seconds_since_start = 1.6
Iteration 50: norm_delta = 6.96586, step_size = 0.0007, ll = -230.85198, newton_decrement = 41.00169, seconds_since_start = 1.7
Convergence failed. See any warning messages.
Concordance index of the model 0.9980554205153136 

<lifelines.CoxPHFitter: fitted with 115 observations, 19 censored>
      duration col = 'Survival from onset'
         event col = 'survival status'
         penalizer = 0.1
number of subjects = 115
  number of events = 96
    log-likelihood = -230.85
  time fit was run = 2019-07-29 18:06:24 UTC

---
                          coef exp(coef)  se(coef)     z    p  -log2(p)  lower 0.95  upper 0.95
hsa-miR-1-3p              0.00      1.00      0.00  0.10 0.92      0.12       -0.00        0.00
hsa-miR-101-3p            0.00      1.00      0.00  0.35 0.73      0.45       -0.00        0.01
hsa-miR-103a-3p           0.00      1.00      0.00  0.78 0.44      1.19       -0.00        0.00
hsa-miR-103b             -0.00      1.00      0.00 -0.35 0.73      0.46       -0.01        0.01
hsa-miR-106b-3p          -0.00      1.00      0.01 -0.10 0.92      0.12       -0.02        0.01
hsa-miR-107               0.00      1.00      0.01  0.11 0.91      0.13       -0.01        0.01
hsa-miR-10a-5p           -0.00      1.00      0.02 -0.20 0.84      0.25       -0.04        0.03
hsa-miR-10b-5p           -0.01      0.99      0.02 -0.34 0.73      0.45       -0.04        0.02
hsa-miR-122-5p            0.00      1.00      0.00  0.05 0.96      0.06       -0.00        0.00
hsa-miR-125a-5p          -0.00      1.00      0.00 -0.22 0.83      0.27       -0.01        0.01
hsa-miR-125b-2-3p        -0.00      1.00      0.01 -0.17 0.86      0.21       -0.01        0.01
hsa-miR-125b-5p          -0.00      1.00      0.01 -0.29 0.77      0.38       -0.01        0.01
hsa-miR-126-3p            0.00      1.00      0.00  0.09 0.93      0.11       -0.00        0.00
hsa-miR-126-5p            0.00      1.00      0.00  0.29 0.77      0.38       -0.00        0.00
hsa-miR-1268b             0.01      1.01      0.02  0.41 0.68      0.56       -0.04        0.06
hsa-miR-127-3p            0.01      1.01      0.03  0.46 0.64      0.64       -0.04        0.07
hsa-miR-128-3p           -0.00      1.00      0.01 -0.12 0.91      0.14       -0.03        0.03
hsa-miR-1287-5p          -0.00      1.00      0.02 -0.02 0.98      0.03       -0.04        0.04
hsa-miR-1301-3p           0.00      1.00      0.03  0.03 0.98      0.03       -0.05        0.05
hsa-miR-1306-5p          -0.01      0.99      0.02 -0.33 0.74      0.44       -0.05        0.04
hsa-miR-1307-3p          -0.00      1.00      0.00 -0.21 0.83      0.27       -0.01        0.01
hsa-miR-1307-5p           0.00      1.00      0.03  0.04 0.96      0.05       -0.06        0.06
hsa-miR-130a-3p           0.01      1.01      0.03  0.22 0.82      0.28       -0.05        0.06
hsa-miR-133a-3p           0.00      1.00      0.00  0.08 0.94      0.09       -0.01        0.01
hsa-miR-133b             -0.00      1.00      0.02 -0.17 0.87      0.21       -0.03        0.03
hsa-miR-134-5p           -0.00      1.00      0.01 -0.26 0.80      0.33       -0.01        0.01
hsa-miR-139-3p            0.00      1.00      0.02  0.06 0.95      0.08       -0.03        0.03
hsa-miR-140-3p            0.00      1.00      0.00  0.20 0.84      0.25       -0.01        0.01
hsa-miR-140-5p            0.02      1.02      0.05  0.33 0.74      0.43       -0.09        0.12
hsa-miR-142-3p           -0.00      1.00      0.00 -0.03 0.97      0.04       -0.00        0.00
hsa-miR-142-5p            0.00      1.00      0.00  0.01 0.99      0.01       -0.00        0.00
hsa-miR-143-3p           -0.00      1.00      0.00 -0.09 0.93      0.10       -0.00        0.00
hsa-miR-144-3p           -0.00      1.00      0.01 -0.21 0.83      0.26       -0.03        0.02
hsa-miR-144-5p           -0.00      1.00      0.03 -0.09 0.93      0.11       -0.07        0.06
hsa-miR-145-5p            0.01      1.01      0.04  0.26 0.80      0.33       -0.07        0.09
hsa-miR-146a-5p           0.00      1.00      0.00  0.01 0.99      0.01       -0.00        0.00
hsa-miR-146b-5p           0.00      1.00      0.01  0.04 0.97      0.05       -0.02        0.02
hsa-miR-148a-3p           0.00      1.00      0.00  0.54 0.59      0.76       -0.00        0.00
hsa-miR-148b-3p          -0.00      1.00      0.00 -0.50 0.62      0.69       -0.01        0.01
hsa-miR-150-5p           -0.00      1.00      0.00 -0.25 0.81      0.31       -0.00        0.00
hsa-miR-151a-3p          -0.00      1.00      0.00 -0.09 0.93      0.10       -0.00        0.00
hsa-miR-151b/151a-5p      0.00      1.00      0.02  0.22 0.82      0.28       -0.03        0.04
hsa-miR-152-3p           -0.00      1.00      0.02 -0.20 0.84      0.25       -0.04        0.03
hsa-miR-155-5p            0.00      1.00      0.01  0.58 0.56      0.83       -0.01        0.01
hsa-miR-15a-5p            0.00      1.00      0.02  0.15 0.88      0.19       -0.03        0.03
hsa-miR-15b-5p           -0.00      1.00      0.01 -0.11 0.91      0.13       -0.01        0.01
hsa-miR-16-2-3p          -0.00      1.00      0.01 -0.26 0.80      0.33       -0.02        0.01
hsa-miR-16-5p             0.00      1.00      0.00  0.15 0.88      0.18       -0.00        0.00
hsa-miR-17-5p            -0.01      0.99      0.02 -0.32 0.75      0.41       -0.04        0.03
hsa-miR-181a-2-3p        -0.01      0.99      0.07 -0.08 0.94      0.09       -0.14        0.13
hsa-miR-181a-5p           0.00      1.00      0.00  0.13 0.90      0.16       -0.01        0.01
hsa-miR-181b-5p           0.02      1.02      0.03  0.67 0.51      0.98       -0.04        0.07
hsa-miR-182-5p           -0.00      1.00      0.01 -0.02 0.98      0.03       -0.02        0.02
hsa-miR-183-5p            0.00      1.00      0.01  0.22 0.83      0.27       -0.02        0.02
hsa-miR-185-3p            0.00      1.00      0.05  0.09 0.93      0.10       -0.09        0.10
hsa-miR-185-5p            0.00      1.00      0.00  0.14 0.89      0.17       -0.01        0.01
hsa-miR-186-5p           -0.00      1.00      0.01 -0.03 0.97      0.04       -0.02        0.02
hsa-miR-18a-5p            0.02      1.02      0.06  0.32 0.75      0.42       -0.09        0.13
hsa-miR-1908-5p          -0.00      1.00      0.03 -0.14 0.89      0.16       -0.06        0.05
hsa-miR-190a-5p           0.00      1.00      0.04  0.13 0.90      0.15       -0.07        0.08
hsa-miR-191-5p           -0.00      1.00      0.00 -0.47 0.64      0.65       -0.00        0.00
hsa-miR-192-5p            0.00      1.00      0.01  0.10 0.92      0.12       -0.02        0.02
hsa-miR-193a-5p          -0.00      1.00      0.03 -0.11 0.91      0.14       -0.06        0.05
hsa-miR-194-5p           -0.00      1.00      0.04 -0.09 0.93      0.11       -0.08        0.07
hsa-miR-195-5p           -0.00      1.00      0.01 -0.28 0.78      0.36       -0.02        0.02
hsa-miR-196b-5p          -0.01      0.99      0.02 -0.63 0.53      0.92       -0.06        0.03
hsa-miR-197-3p           -0.00      1.00      0.02 -0.14 0.89      0.18       -0.04        0.03
hsa-miR-199a-3p          -0.00      1.00      0.00 -0.54 0.59      0.76       -0.00        0.00
hsa-miR-199a-5p           0.02      1.02      0.04  0.42 0.68      0.56       -0.06        0.09
hsa-miR-199b-3p          -0.00      1.00      0.00 -0.41 0.68      0.55       -0.01        0.01
hsa-miR-19a-3p            0.00      1.00      0.04  0.03 0.97      0.04       -0.08        0.09
hsa-miR-19b-3p            0.01      1.01      0.02  0.35 0.73      0.46       -0.03        0.04
hsa-miR-200a-3p           0.00      1.00      0.04  0.04 0.97      0.05       -0.07        0.07
hsa-miR-200b-3p           0.01      1.01      0.02  0.48 0.63      0.66       -0.03        0.04
hsa-miR-200c-3p           0.00      1.00      0.04  0.02 0.98      0.02       -0.08        0.08
hsa-miR-203a-3p          -0.00      1.00      0.02 -0.24 0.81      0.30       -0.03        0.03
hsa-miR-205-5p           -0.01      0.99      0.03 -0.18 0.86      0.22       -0.06        0.05
hsa-miR-206               0.00      1.00      0.00  0.13 0.90      0.16       -0.00        0.00
hsa-miR-20a-5p           -0.00      1.00      0.01 -0.14 0.89      0.17       -0.02        0.02
hsa-miR-20b-5p           -0.02      0.98      0.04 -0.45 0.65      0.61       -0.09        0.05
hsa-miR-21-5p             0.00      1.00      0.00  0.02 0.98      0.03       -0.00        0.00
hsa-miR-2110             -0.02      0.98      0.05 -0.39 0.70      0.52       -0.12        0.08
hsa-miR-22-3p             0.00      1.00      0.01  0.49 0.62      0.69       -0.01        0.02
hsa-miR-221-3p           -0.00      1.00      0.00 -0.07 0.95      0.08       -0.00        0.00
hsa-miR-222-3p           -0.00      1.00      0.01 -0.13 0.90      0.16       -0.02        0.02
hsa-miR-223-3p            0.00      1.00      0.00  0.46 0.64      0.64       -0.00        0.00
hsa-miR-223-5p            0.00      1.00      0.01  0.12 0.91      0.14       -0.03        0.03
hsa-miR-23a-3p            0.00      1.00      0.00  0.03 0.98      0.03       -0.01        0.01
hsa-miR-23b-3p            0.01      1.01      0.01  0.40 0.69      0.53       -0.02        0.03
hsa-miR-24-3p            -0.00      1.00      0.00 -0.03 0.97      0.04       -0.01        0.01
hsa-miR-25-3p            -0.00      1.00      0.00 -0.01 1.00      0.01       -0.00        0.00
hsa-miR-26a-5p            0.00      1.00      0.00  0.05 0.96      0.06       -0.00        0.00
hsa-miR-26b-5p           -0.00      1.00      0.00 -0.17 0.87      0.20       -0.00        0.00
hsa-miR-27a-3p           -0.00      1.00      0.01 -0.27 0.78      0.35       -0.03        0.02
hsa-miR-27b-3p           -0.00      1.00      0.01 -0.21 0.83      0.26       -0.01        0.01
hsa-miR-28-3p             0.00      1.00      0.01  0.05 0.96      0.06       -0.02        0.02
hsa-miR-29a-3p           -0.00      1.00      0.00 -0.58 0.56      0.82       -0.01        0.00
hsa-miR-29b-3p           -0.00      1.00      0.03 -0.15 0.88      0.19       -0.06        0.05
hsa-miR-29c-3p           -0.00      1.00      0.01 -0.18 0.86      0.22       -0.01        0.01
hsa-miR-29c-5p           -0.00      1.00      0.05 -0.02 0.98      0.02       -0.09        0.09
hsa-miR-30a-5p           -0.00      1.00      0.01 -0.10 0.92      0.12       -0.01        0.01
hsa-miR-30b-5p           -0.01      0.99      0.04 -0.12 0.90      0.15       -0.09        0.08
hsa-miR-30c-5p            0.00      1.00      0.02  0.15 0.88      0.19       -0.03        0.04
hsa-miR-30d-5p            0.00      1.00      0.00  0.19 0.85      0.24       -0.00        0.00
hsa-miR-30e-3p           -0.00      1.00      0.01 -0.14 0.89      0.18       -0.02        0.02
hsa-miR-30e-5p           -0.00      1.00      0.00 -0.55 0.59      0.77       -0.01        0.00
hsa-miR-3135b             0.00      1.00      0.01  0.14 0.89      0.17       -0.02        0.03
hsa-miR-3168              0.00      1.00      0.00  0.30 0.76      0.39       -0.00        0.00
hsa-miR-320a             -0.00      1.00      0.00 -0.04 0.97      0.04       -0.00        0.00
hsa-miR-320b             -0.00      1.00      0.01 -0.23 0.82      0.29       -0.03        0.02
hsa-miR-320c             -0.00      1.00      0.02 -0.14 0.89      0.17       -0.05        0.04
hsa-miR-323b-3p           0.04      1.04      0.05  0.72 0.47      1.09       -0.06        0.13
hsa-miR-324-5p           -0.01      0.99      0.04 -0.16 0.88      0.19       -0.07        0.06
hsa-miR-326              -0.00      1.00      0.03 -0.12 0.90      0.15       -0.06        0.05
hsa-miR-328-3p           -0.00      1.00      0.01 -0.06 0.95      0.07       -0.01        0.01
hsa-miR-335-5p           -0.01      0.99      0.02 -0.50 0.62      0.69       -0.04        0.03
hsa-miR-339-3p           -0.00      1.00      0.04 -0.05 0.96      0.05       -0.07        0.07
hsa-miR-339-5p            0.00      1.00      0.01  0.42 0.67      0.57       -0.01        0.02
hsa-miR-340-5p            0.01      1.01      0.02  0.34 0.73      0.45       -0.04        0.05
hsa-miR-342-3p           -0.00      1.00      0.00 -0.05 0.96      0.06       -0.00        0.00
hsa-miR-345-5p            0.00      1.00      0.05  0.02 0.98      0.03       -0.09        0.10
hsa-miR-34a-5p            0.00      1.00      0.03  0.16 0.87      0.20       -0.05        0.06
hsa-miR-361-3p            0.01      1.01      0.03  0.44 0.66      0.61       -0.04        0.06
hsa-miR-361-5p            0.00      1.00      0.01  0.38 0.70      0.50       -0.02        0.03
hsa-miR-3613-3p           0.01      1.01      0.02  0.32 0.75      0.41       -0.03        0.04
hsa-miR-3615              0.00      1.00      0.01  0.24 0.81      0.30       -0.02        0.03
hsa-miR-363-3p            0.01      1.01      0.02  0.44 0.66      0.60       -0.03        0.05
hsa-miR-3687              0.02      1.02      0.04  0.62 0.54      0.90       -0.05        0.10
hsa-miR-370-3p           -0.00      1.00      0.02 -0.11 0.91      0.14       -0.03        0.03
hsa-miR-374a-5p          -0.01      0.99      0.04 -0.37 0.71      0.49       -0.09        0.06
hsa-miR-374b-5p          -0.02      0.98      0.08 -0.29 0.77      0.38       -0.17        0.13
hsa-miR-375               0.00      1.00      0.01  0.07 0.94      0.09       -0.02        0.02
hsa-miR-378a-3p           0.00      1.00      0.01  0.11 0.92      0.13       -0.02        0.03
hsa-miR-378c              0.02      1.02      0.04  0.40 0.69      0.53       -0.06        0.10
hsa-miR-379-5p            0.00      1.00      0.02  0.21 0.83      0.26       -0.03        0.04
hsa-miR-381-3p            0.00      1.00      0.04  0.12 0.90      0.15       -0.07        0.08
hsa-miR-382-5p            0.00      1.00      0.01  0.19 0.85      0.24       -0.01        0.01
hsa-miR-3940-3p          -0.01      0.99      0.02 -0.26 0.79      0.33       -0.04        0.03
hsa-miR-3974             -0.00      1.00      0.02 -0.21 0.83      0.26       -0.03        0.03
hsa-miR-409-3p           -0.00      1.00      0.01 -0.48 0.63      0.67       -0.01        0.01
hsa-miR-423-3p           -0.00      1.00      0.01 -0.72 0.47      1.08       -0.01        0.01
hsa-miR-423-5p            0.00      1.00      0.00  0.15 0.88      0.19       -0.00        0.00
hsa-miR-425-3p            0.00      1.00      0.02  0.08 0.94      0.09       -0.04        0.05
hsa-miR-425-5p           -0.00      1.00      0.00 -0.21 0.84      0.26       -0.00        0.00
hsa-miR-4254             -0.00      1.00      0.01 -0.30 0.77      0.38       -0.03        0.02
hsa-miR-4286              0.03      1.03      0.06  0.56 0.57      0.80       -0.08        0.14
hsa-miR-431-5p           -0.00      1.00      0.02 -0.15 0.88      0.18       -0.04        0.03
hsa-miR-432-5p            0.00      1.00      0.00  0.28 0.78      0.35       -0.00        0.01
hsa-miR-4433a-3p         -0.00      1.00      0.03 -0.01 0.99      0.01       -0.06        0.06
hsa-miR-4433b-5p          0.00      1.00      0.01  0.20 0.84      0.25       -0.01        0.01
hsa-miR-4446-3p           0.01      1.01      0.04  0.24 0.81      0.30       -0.07        0.09
hsa-miR-4451             -0.00      1.00      0.02 -0.00 1.00      0.00       -0.04        0.04
hsa-miR-4454             -0.01      0.99      0.03 -0.28 0.78      0.36       -0.08        0.06
hsa-miR-451a              0.00      1.00      0.00  0.23 0.82      0.29       -0.00        0.00
hsa-miR-454-3p           -0.00      1.00      0.02 -0.12 0.91      0.14       -0.05        0.04
hsa-miR-4655-5p          -0.00      1.00      0.01 -0.05 0.96      0.06       -0.02        0.02
hsa-miR-4732-5p           0.00      1.00      0.02  0.02 0.98      0.03       -0.03        0.03
hsa-miR-483-5p            0.01      1.01      0.01  0.50 0.62      0.70       -0.02        0.03
hsa-miR-484               0.00      1.00      0.00  0.11 0.91      0.13       -0.01        0.01
hsa-miR-485-3p           -0.00      1.00      0.02 -0.11 0.91      0.14       -0.03        0.03
hsa-miR-485-5p            0.01      1.01      0.05  0.16 0.87      0.19       -0.10        0.12
hsa-miR-486-3p            0.00      1.00      0.00  0.13 0.89      0.16       -0.00        0.00
hsa-miR-486-5p            0.00      1.00      0.00  0.09 0.93      0.11       -0.00        0.00
hsa-miR-487b-3p          -0.00      1.00      0.05 -0.04 0.97      0.05       -0.10        0.09
hsa-miR-501-3p            0.02      1.02      0.04  0.42 0.68      0.57       -0.07        0.10
hsa-miR-532-5p           -0.01      0.99      0.04 -0.34 0.73      0.45       -0.08        0.06
hsa-miR-548ad-3p          0.00      1.00      0.02  0.17 0.87      0.21       -0.03        0.04
hsa-miR-548ap-5p/548j-5p -0.00      1.00      0.04 -0.10 0.92      0.12       -0.09        0.08
hsa-miR-574-3p            0.01      1.01      0.02  0.25 0.80      0.32       -0.03        0.04
hsa-miR-584-5p            0.00      1.00      0.00  0.53 0.60      0.74       -0.00        0.01
hsa-miR-625-3p           -0.00      1.00      0.00 -0.03 0.97      0.04       -0.01        0.01
hsa-miR-625-5p            0.01      1.01      0.02  0.29 0.77      0.37       -0.03        0.04
hsa-miR-628-3p           -0.00      1.00      0.03 -0.03 0.97      0.04       -0.07        0.07
hsa-miR-629-5p           -0.00      1.00      0.02 -0.21 0.83      0.26       -0.03        0.03
hsa-miR-652-3p            0.00      1.00      0.04  0.11 0.92      0.13       -0.08        0.09
hsa-miR-654-3p           -0.01      0.99      0.02 -0.26 0.80      0.33       -0.05        0.04
hsa-miR-660-5p           -0.01      0.99      0.02 -0.76 0.45      1.16       -0.05        0.02
hsa-miR-664a-5p          -0.01      0.99      0.03 -0.26 0.79      0.34       -0.06        0.05
hsa-miR-671-3p           -0.01      0.99      0.06 -0.17 0.87      0.21       -0.13        0.11
hsa-miR-671-5p            0.02      1.02      0.05  0.29 0.77      0.38       -0.09        0.12
hsa-miR-6728-5p          -0.00      1.00      0.01 -0.13 0.89      0.16       -0.03        0.02
hsa-miR-6749-5p          -0.01      0.99      0.01 -0.50 0.62      0.70       -0.04        0.02
hsa-miR-6787-5p          -0.00      1.00      0.03 -0.19 0.85      0.24       -0.06        0.05
hsa-miR-6852-5p          -0.01      0.99      0.06 -0.15 0.88      0.19       -0.12        0.10
hsa-miR-6890-5p           0.00      1.00      0.02  0.10 0.92      0.12       -0.04        0.05
hsa-miR-7-5p              0.01      1.01      0.02  0.38 0.70      0.51       -0.03        0.05
hsa-miR-744-5p           -0.00      1.00      0.00 -0.07 0.95      0.08       -0.01        0.01
hsa-miR-760               0.03      1.03      0.05  0.49 0.62      0.69       -0.08        0.13
hsa-miR-769-5p            0.00      1.00      0.06  0.02 0.99      0.02       -0.11        0.11
hsa-miR-92a-3p            0.00      1.00      0.00  0.17 0.87      0.20       -0.00        0.00
hsa-miR-92b-3p           -0.00      1.00      0.01 -0.08 0.94      0.09       -0.02        0.02
hsa-miR-93-5p            -0.00      1.00      0.00 -0.30 0.76      0.39       -0.00        0.00
hsa-miR-941              -0.03      0.97      0.04 -0.65 0.51      0.96       -0.12        0.06
hsa-miR-98-5p             0.01      1.01      0.01  0.51 0.61      0.71       -0.02        0.03
hsa-miR-99a-5p            0.00      1.00      0.02  0.23 0.82      0.28       -0.04        0.05
hsa-miR-99b-5p            0.00      1.00      0.01  0.13 0.90      0.15       -0.01        0.01
---
Concordance = 1.00
Log-likelihood ratio test = 305.88 on 196 df, -log2(p)=20.20
/Users/nancy/anaconda3/lib/python3.7/site-packages/lifelines/fitters/coxph_fitter.py:557: ConvergenceWarning: Newton-Rhapson failed to converge sufficiently in 50 steps.
  warnings.warn("Newton-Rhapson failed to converge sufficiently in %d steps." % max_steps, ConvergenceWarning)
<lifelines.CoxPHFitter: fitted with 115 observations, 19 censored>
Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

👋 hi there, lifelines author here. Let me try to help.

1) Do you see any Python warnings when the fit starts running?

2) I noticed that you have 115 observations, but over 190 variables. It's very likely that system is overdetermined: there isn't a unique solution, and your model will completely overfit to the data (more evidence of this: the concordance ~= 1.0). Since your coefficients look to be very small, you may need a very high penalizer to "fix" this, but the real solution is to get much more data.

3) alpha doesn't change anything in the fit, it's only for confidence intervals. So no point in playing with that.

4) Geometrically what is happening is that the point is near the minimum, but it's very flat there, so the point is kinda making large jumps. This is why the delta is still so large. Again, this is a consequence of a lot of variables, but not many data points.

Improve this answer
$\endgroup$
1
  • $\begingroup$ First, thank you very much for putting this wonderful and well documented package. We all love it! I'm happy to announce that I'vs fixed the all the assumption violations by using your tutorials - I've changed some variables to be non linear and added variables to be as "strata". Indeed, I know that I have a problem of dimension - but that's the data I have so I'll remove variables to solve the n<p problem. Again, thank you! $\endgroup$
    – Serendipity
    Commented Jul 31, 2019 at 16:06
0
$\begingroup$

The package comes with an excellent tutorial on the assumptions of the CPH algorithm and even provide a function for Testing-the-proportional-hazard-assumptions.

I've first had a convergence problem as stated above, but once resolving that I got a message that one of the variables is violating the assumptions:

    The ``p_value_threshold`` is set at 0.05. Even under the null hypothesis of no violations, some
covariates will be below the threshold by chance. This is compounded when there are many covariates.
Similarly, when there are lots of observations, even minor deviances from the proportional hazard
assumption will be flagged.

With that in mind, it's best to use a combination of statistical tests and visual tests to determine
the most serious violations. Produce visual plots using ``check_assumptions(..., show_plots=True)``
and looking for non-constant lines. See link [A] below for a full example.

1. Variable 'hsa-miR-181a-5p' failed the non-proportional test: p-value is 0.0331.

   Advice 1: the functional form of the variable 'hsa-miR-181a-5p' might be incorrect. That is,there may be non-linear terms missing. The proportional hazard test used is very sensitive to
incorrect functional forms. See documentation in link [D] below on how to specify a functional form.

   Advice 2: try binning the variable 'hsa-miR-181a-5p' using pd.cut, and then specify it in
`strata=['hsa-miR-181a-5p', ...]` in the call in `.fit`. See documentation in link [B] below.

   Advice 3: try adding an interaction term with your time variable. See documentation in link [C]
below.

---
[A]  https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html
[B]  https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html#Bin-variable-and-stratify-on-it
[C]  https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html#Introduce-time-varying-covariates
[D]  https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html#Modify-the-functional-form
[E]  https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html#Stratification

Here are the steps that I took resolve the problem:

  1. Although the full error message states that I have a time-varying variables, I 100% don't have any time-varying variables. Particularly, that variable (which the error message claims it is violating the assumption) is a fixed number for every t.

    After some deep reading, this could happen!

    One alternative is to correct that variable/s directly; I've engineered the specific variable that is violating the hazard assumption to be non-linear (quadratic). In addition, I've scaled all variables to the same range.

  2. I've moved one of variables (age) to be "strata" - w/o that the problem is not fully resolved. Note that I binned "age" to decades, so it's now a categorical variable with cardinality of $9$ (which is relatively low) . Also, we know that age has an impact on the "event" (time of death), since age is negatively correlated with survival time.

  3. I indeed had serious overfitting, since $n<p$ in my data. I've removed correlated variables and also added a regularization term ($L2$ penalty) - did the work.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.