Written by

Businnes Intelligence Developer at Shift (www.shift.com.br)
Question José Pereira · Aug 4, 2021

Cross Validation in IntegratedML

Hi!

I'd like to know if the VALIDATE MODEL statement in IntegratedML performs a cross validation (CV) procedure internally. If it doesn't, is there a ready to use method for CV?

Thanks!

Comments

Jon Willeke · Aug 10, 2021

No, the VALIDATE MODEL statement does not do cross validation. It calculates validation metrics for the given trained model and dataset. As described in the "Model Selection Process" section of the documentation, however, the TRAIN MODEL statement does this to some extent for classification models when using the AutoML provider:

These scoring metrics are then computed for each model using Monte Carlo cross validation, with three training/testing splits of 70%/30%, to determine the best model.

I also believe that the DataRobot provider incorporates cross validation into its training. I'm not sure about H2O.

0
José Pereira  Aug 11, 2021 to Jon Willeke

Thanks Jon, I'll check it out.

0