I am training a pyspark logistic regression model using pyspark mllib. I am noticing that the weights are not being consistent in between runs. I have set the random seed in the training script and also am making a deterministic split between training and testing data using TrainValidationSplit.

With low number of records i.e., less than 1million input records, the model weights match exactly between runs. But as I increase the volume of the data (to >20M records), I am noticing that the weights are fluctuating. Some of them are zero during one iteration while in other iterations, they have values in the 10^(-3) range. Can anyone help me understand why this could be happening?

Note: I am doing a constrained optimization - I am setting lower bounds weights and intercept to be zero and non-negative.

  • $\begingroup$ Not sure this is a problem. What is the log_probability of your data for models with these slightly different coefficients? Could it be that your data is equally probable under these different coefficients? If it is, you can't really blame the optimizer - it did its job, it simply happens that there is more than one set of parameters that works $\endgroup$
    – Cryo
    Jul 19, 2023 at 20:44
  • $\begingroup$ Thanks @cryo for the response! From what I have observed, this seems to be arising from subtle numerical differences in the objective during the iterations. Although the initial objective values are same, as the training happens through multiple iterations, model convergence happens at different iterations. Is this a artifact of loss landscape and numerical optimization limits? $\endgroup$ Jul 20, 2023 at 15:39
  • $\begingroup$ I don't think that answers my question. What I am getting at is - is this a problem? Your input data is used to fit the coefficients. What if there are several combinations of coefficients that are equally probable? For example I could try to find solution to x+y=4, and my solver could say x=1, y=3, or it could say x=0, y=4, so the inconsistency is very large, but both combinations fit the problem. So my question is whether the different coefficients you get equally fit the problem you are posing. If they do, then yes, some numerical artifacts could be the immediate cause of difference $\endgroup$
    – Cryo
    Jul 22, 2023 at 14:43
  • $\begingroup$ But the underlying cause is that problem is under-specified. i.e. either you accept that several possible solutions can arise, or you add more data, or you change the question you are posing $\endgroup$
    – Cryo
    Jul 22, 2023 at 14:45
  • $\begingroup$ Yes, I understand this issue better now. It might be that the data is equally likely for both set of coefficients. How do I check if the data is equally likely with these two set of coefficients? I was thinking if the predictions would be same, but I don’t think that will be true $\endgroup$ Jul 23, 2023 at 16:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.