evimp {earth} | R Documentation |
Estimate variable importances in an earth
object
evimp(obj, trim=TRUE)
obj |
An earth object.
|
trim |
If TRUE (default), delete rows in the returned matrix for variables that don't appear in any subsets. |
A matrix showing the relative importances of the variables in the model.
There is a a row for each variable.
The row name is the variable name, but with -unused
appended
if the variable does not appear in the final model.
The columns of the matrix are:
col
: column index of the variable in the x
argument to earth
.
used
: 1 if the variable is used in the final model, else 0.
Equivalently, 0 if the row name has a -unused
suffix.
nsubsets
: variable importance using the "number of subsets" criterion.
Is the number of subsets that include the variable (see below).
gcv
: variable importance using the GCV criterion (see below).
rss
: ditto but for the RSS criterion.
The rows are sorted on the nsubsets
criterion.
This means that values in the nsubsets
column decrease as you go down the column.
The values in the gcv
and rss
columns decrease except where the
gcv
or rss
ranking differs from the nsubsets
ranking.
Additionally, there are unnamed columns after the gcv
column and the rss
column.
These have a 0 where the ranking using the gcv
or rss
criteria differs from
that using the nsubsets
criterion.
Equivalently, there is a 0 for values that do not decrease as you go
down the gcv
or rss
column.
Estimating variable importance
Establishing predictor importance is in general a tricky and even controversial problem.
There is no completely reliable way to estimate the importance of the variables
in a standard MARS model,
unless you make further lengthy tests after the model is built.
The evimp
function just makes an educated (and in practice useful)
guess as described below.
Three criteria for estimating variable importance
The evimp
functions uses three criteria for estimating variable importance.
1. The nsubsets
criterion counts the number of model subsets that include the variable.
Variables that are included in more subsets are considered more important.
By "subsets" we mean the subsets of terms generated by the pruning pass.
There is one subset for each model size,
and each subset is the best set of terms for that model size.
(These subsets are specified by $prune.terms
in earth's return value.)
Only subsets that are smaller than or equal in size to the final model are used
for estimating variable importance.
2. The rss
criterion first calculates the decrease in the RSS
for each subset relative to the previous subset.
(For multiple response models, RSS's are calculated over all responses.)
Then for each variable it sums these decreases over all subsets that include the variable.
Finally is scales these decreases so the maximum decrease is 100.
Variables which cause larger net decreases in the RSS are considered more important.
3. The gcv
criterion is the same, but using the GCV instead of the RSS.
Adding a variable can sometimes increase the GCV.
When this happens, the variable could even have a negative total importance,
and thus appear less important than unused variables.
Note that using RSq's and GRSq's instead of RSS's and GCV's would give identical estimates of variable importance.
Example
a <- earth(O3 ~ ., data=ozone1, degree=2) evimp(a, trim=FALSE)Yields the following matrix:
col used nsubsets gcv rss temp 4 1 10 100.00 1 100.00 1 humidity 3 1 8 12.68 1 14.78 1 ibt 7 1 8 12.68 1 14.78 1 doy 9 1 7 11.26 1 12.93 1 dpg 6 1 5 6.75 1 7.84 1 ibh 5 1 4 9.58 0 10.46 0 vis 8 1 4 4.38 1 5.30 1 wind 2 1 1 0.74 1 0.98 1 vh-unused 1 0 0 0.00 1 0.00 1The rows are sorted on
nsubsets
.
We see that temp
is considered the most important variable,
followed by humidity
, and so on.
We see that vh
is unused in the final model,
and thus is given an unused
suffix and a 0 in the used
column.
The col
column gives the the column indices of the variables
in the x
argument to earth
after factors have been expanded.
The nsubsets
column is the number of subsets that included the corresponding variable.
For example, temp
appears in 10 subsets and humidity
in 8.
The gcv
and rss
columns are scaled so
the largest net decrease is 100.
The unnamed columns after the gcv
and rss
columns have a 0 if the corresponding criterion increases instead of decreasing
(i.e. the ranking disagrees with the nsubsets
ranking).
We see that ibh
is considered less important than dpg
using the nsubsets
criterion, but not with the gcv
and rss
criteria.
Other techniques
Running plotmo
with ylim=NULL
(the default)
gives an idea of which predictors make the largest changes to the predicted value
(but only with all other predictors at their median values).
You can also use drop1
(assuming you are using the formula interface to earth).
Calling drop1(my.earth.model)
will delete each predictor in turn from your model,
rebuild the model from scratch each time, and calculate the GCV each time.
You will get warnings that the earth library function extractAIC.earth
is
returning GCVs instead of AICs — but that is what you want so you can
ignore the warnings.
(You can turn off just these warnings by passing warn=FALSE
to drop1
).
The column labeled AIC
in the printed response
from drop1
will actually be a column of GCVs not AICs.
The Df
column is not much use in this context.
Remember that this technique only tells you how important
a variable is with the other variables already in the model.
It does not tell the effect of a variable in isolation.
Note that drop1
drops predictors from the model
while earth's pruning pass drops terms.
You will get lots of output from drop1
if you built your original earth
model with trace>0
.
You can set trace=0
by updating your model before calling drop1
.
Do it like this:
my.model <- update.earth(my.model, trace=0)
.
Remarks
This function is useful in practice but the following issues can make it misleading.
MARS models have a high variance — if the data changes a little, the set of basis terms created by the forward pass can change a lot. So estimates of predictor importance can be unreliable because they can vary with even slightly different training data.
Colinear (or related) variables can mask each other's importance, just as in linear models. This means that if two predictors are closely related, the forward pass will somewhat arbitrarily choose one over the other. The chosen predictor will incorrectly appear more important.
For interaction terms, each variable gets credit for the entire term — thus interaction terms are counted more than once and get a total higher weighting than additive terms (questionably). Each variable gets equal credit in interaction terms even though one variable in that term may be far more important than the other.
One can question if it is valid to estimate variable importance using model subsets that are not part of the final model. It is even possible for a variable to be rated as important yet not appear in the final model.
An example of conflicting importances
(however, the results are fine with the default pmethod
):
evimp(earth(mpg~., data=mtcars, pmethod="none"))
Acknowledgment
Thanks to Max Kuhn for the original evimp
code and for helpful discussions.
data(ozone1) a <- earth(O3 ~ ., data=ozone1, degree=2) evimp(a, trim=FALSE)