Hello everybody,
My team and I are facing a sudden and unexpected issue with training Forecasting models utilizing AutoML on Azure Databricks.
As of two weeks ago, we privatized our Azure Databricks environment behind a private network, accessible only by using a VPN.
Since then, Serverless training of Forecasting ML models fails for every type of forecasting ML framework, including ARIMA, Prophet and DeepAR. To be more precise:
- ARIMA training experiments complete. However, the deployment of trained ARIMA models in Endpoints fails. The respective Service logs are available here.
- Prophet training experiments fail to complete, being cancelled at the Tuning stage. The only output in the run indicating that something is wrong is "databricks.automl.shared.databricks_utils: No host name to create absolute URL".
- DeepAR training experiments also fail to complete, being cancelled at the Tuning stage. The only output in the run indicating that something is wrong is "databricks.automl.shared.databricks_utils: No host name to create absolute URL".
On the other hand, using a compute cluster to train ARIMA and Prophet models, the training completes successfully. The compute runtime is 17.3 LTS for Machine Learning. However, when trying to use these models for inference, we get the following issues:
- For the ARIMA models, the deployment fails, indicating that the failure reason is "mlflowserving.scoring_server.diagnostic_classifier._PhaseTaggedError: [mlflow_parse] ModuleNotFoundError: No module named 'databricks.automl_runtime'"
- For the Prophet models, the deployment completes. However, when trying to use any Prophet model for inference, the procedure fails, indicating that the failure reason is "Inference failed: Encountered an unexpected error while evaluating the model. Verify that the input is compatible with the model for inference. Error ''Prophet' object has no attribute 'stan_backend'' (Support code: INFERENCE_FAILED-9430702a5a9f)"
The serverless training of Forecasting ML models worked perfectly end-to-end a few weeks back and so did the deployment and the inference.
Could we get some help on how to resolve this issues? Is this a temporary bug? Do we need to take any actions?
I have also opened a respective community thread.
Thank you in advance.
Hello everybody,
My team and I are facing a sudden and unexpected issue with training Forecasting models utilizing AutoML on Azure Databricks.
As of two weeks ago, we privatized our Azure Databricks environment behind a private network, accessible only by using a VPN.
Since then, Serverless training of Forecasting ML models fails for every type of forecasting ML framework, including ARIMA, Prophet and DeepAR. To be more precise:
On the other hand, using a compute cluster to train ARIMA and Prophet models, the training completes successfully. The compute runtime is 17.3 LTS for Machine Learning. However, when trying to use these models for inference, we get the following issues:
The serverless training of Forecasting ML models worked perfectly end-to-end a few weeks back and so did the deployment and the inference.
Could we get some help on how to resolve this issues? Is this a temporary bug? Do we need to take any actions?
I have also opened a respective community thread.
Thank you in advance.