I use deepAR RNN on AWS via python. I run the estimator.fit(inputs=data_channels) and want to get insight on the algorithm performance and the fit that the model obtained, i.e. epoch to epoch convergence, quantile loss, RMSE, etc..

Instead I get a log dump like this:

 [05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, wQuantileLoss[0.5]): 0.0768988
[05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, wQuantileLoss[0.9]): 0.037788
[05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, RMSE): 4433.98308522
#metrics {"Metrics": {"totaltime": {"count": 1, "max": 125569.84615325928, "sum": 125569.84615325928, "min": 125569.84615325928}, "setuptime": {"count": 1, "max": 14.33110237121582, "sum": 14.33110237121582, "min": 14.33110237121582}}, "EndTime": 1525945442.991945, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1525945442.891469}

which is completely unreadable and unprocessable.

Is there a way to get those metrics back to python?

PS. The folder in output_path of estimator contains some files, but they do not seem to have the results I am looking for

I managed to handle this way, yet horribly inefficient:

  1. First I capture the console output with this trick:

    f = io.StringIO() with redirect_stdout(f): estimator.fit(inputs=data_channels) dr_log = f.getvalue()

  2. Then I manually process the long multiline string (blaaah:/)

    def dr_metrics(log, r): rr = list() final_res = dict() for line in log: if "Epoch" in line: l = line.split("]") print(l)

        e = l[-3].split("[")[-1]
        b = l[-2].split("[")[-1]
        val = l[-1][:-2].split("=")[1]
        rr.append([e, b, val, "Speed" in line])
        print([e, b, val])
    if "Final loss" in line:
        final_res['final_loss'] = line.split(":")[-1].split(" ")[1]
    if "#quality_metric" in line:
        if "mean_wQuantileLoss" in line:
            final_res['mean_wQuantileLoss'] = line.split("=")[-1]
        if "RMSE" in line:
            final_res['RMSE'] = line.split("=")[-1]

    res_df = pd.DataFrame(rr, columns=["Epoch", "Batch", "AvgLoss", "EpochEnd"]) res_df.head(10) print(final_res)

  • $\begingroup$ I found this way around, but still needs some processing f = io.StringIO() with redirect_stdout(f): estimator.fit(inputs=data_channels) print(f.getvalue()) $\endgroup$ – Intelligent-Infrastructure May 24 '18 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.