autostats

plot variable contributions

multiclass target

Species is a 3-level factor so it will be automatically modelled with a multiclass neural network and a light gbm with multiclass objective function.

First set define the formula to use for modeling.

iris %>% 
  tidy_formula(target = Species) -> species_formula

species_formula
#> Species ~ Sepal.Length + Sepal.Width + Petal.Length + Petal.Width
#> <environment: 0x7ff874cb9ed0>

iris %>% 
  auto_variable_contributions(species_formula)
#> [17:43:12] WARNING: amalgamation/../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

iris %>% 
  auto_model_accuracy(species_formula)
#> ! Fold1: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold2: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold3: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold4: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.

4 - fold cross-validated accuracy for classification model of Species on dataset iris
regularized linear model	accuracy	0.954	0.03109
xgboost	0.947	0.01499
regularized linear model	roc_auc	0.991	0.00532
xgboost	0.972	0.01629

binary target

Linear models uses weighted logistic regression for modeling the coefficients

iris %>% 
  filter(Species != "setosa") %>% 
  auto_variable_contributions(species_formula)
#> [17:43:21] WARNING: amalgamation/../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

For the variable contributions the linear model uses penalized logistic regression provided by glmnet.

iris %>% 
  filter(Species != "setosa") -> iris_binary

iris_binary %>% 
  auto_model_accuracy(species_formula)
#> ! Fold1: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold2: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold3: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold4: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.

4 - fold cross-validated accuracy for classification model of Species on dataset iris_binary
regularized linear model	accuracy	0.960	0.01633
xgboost	0.940	0.02000
regularized linear model	roc_auc	0.995	0.00309
xgboost	0.983	0.01291

continuous target

Models are automatically adapted for a continuous target.

Define the new formula

iris %>% 
  tidy_formula(target = Petal.Length) -> petal_formula

petal_formula
#> Petal.Length ~ Sepal.Length + Sepal.Width + Petal.Width + Species
#> <environment: 0x7ff865a37540>

iris %>% 
  auto_variable_contributions(petal_formula)

iris %>% 
  auto_model_accuracy(petal_formula)
#> ! Fold1: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold2: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold3: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.
#> ! Fold4: preprocessor 1/1, model 1/1: `early_stop` was reduced to 14.

4 - fold cross-validated accuracy for regression model of Petal.Length on dataset iris
regularized linear model	rmse	0.269	0.01165
xgboost	0.273	0.01366
regularized linear model	rsq	0.976	0.00162
xgboost	0.975	0.00417

auto anova

auto anova automatically regresses each continuous variable supplied against each categorical variable supplied. Lm is called separately for each continuous/ categorical variable pair, but the results are reported in one dataframe. Whether the outcome differs amongst categorical levels is determined by the p.value. The interpretation is affected by the choice of baseline for comparison. Traditionally the first level of the factor is used, however option to use the mean of the continuous variable as the baseline intercept is a helpful comparison.

iris %>% 
  auto_anova(Species, matches("Petal"), baseline = "first_level")
#> # A tibble: 6 × 12
#>   target   predictor level    estimate target_mean     n std.error level_p.value
#>   <chr>    <chr>     <chr>       <dbl>       <dbl> <int>     <dbl>         <dbl>
#> 1 Petal.L… Species   (Interc…    1.46        1.46     50    0.0609      9.30e-53
#> 2 Petal.L… Species   versico…    2.80        4.26     50    0.0861      5.25e-69
#> 3 Petal.L… Species   virgini…    4.09        5.55     50    0.0861      4.11e-91
#> 4 Petal.W… Species   (Interc…    0.246       0.246    50    0.0289      1.96e-14
#> 5 Petal.W… Species   versico…    1.08        1.33     50    0.0409      1.25e-57
#> 6 Petal.W… Species   virgini…    1.78        2.03     50    0.0409      7.95e-86
#> # … with 4 more variables: level_significance <chr>, predictor_p.value <dbl>,
#> #   predictor_significance <chr>, conclusion <chr>

4 - fold cross-validated accuracy for classification model of Species on dataset iris
model	metric	mean_score	std_err
regularized linear model	accuracy	0.954	0.03109
xgboost	accuracy	0.947	0.01499
regularized linear model	roc_auc	0.991	0.00532
xgboost	roc_auc	0.972	0.01629