Citing cobalt

When using cobalt, please cite your use of it along with the conditioning package used. The full APA reference for cobalt is the following:

Greifer, N. (2016). cobalt: Covariate Balance Tables and Plots. R package version 1.1.0.

For example, if you use Matching for propensity score estimation and matching and cobalt for balance assessment and/or reporting, a possible citation might go as follows:

“Matching was performed using the Matching package (Sekhon, 2011), and covariate balance was assessed using cobalt (Greifer, 2016), both in R 3.3.0 (R Core Team, 2016).”

Visual clarity

cobalt presents one table in its balance output, and it contains all the information required to assess balance. twang and CBPS present two tables, MatchIt presents three tables, and Matching presents as many tables as there are covariates. Although each of these tables contains valuable information, the bal.tab() function in cobalt allows for a quick and easy search for the information desired, which is often a single column containing a balance statistic (such as the standardized mean difference) for the adjusted sample.

Useful summaries

Although a thorough balance assessment requires examining the balance of each covariate individually, cobalt’s bal.tab() function can also produce quick balance summaries that can aid in model selection when there are many covariates or higher order terms to examine. These summaries include the proportion of covariates that have met a user-specified threshold for balance and the covariate with the highest degree of imbalance, two values that have been shown to be effective in diagnosing imbalance and potential bias (Stuart, Lee, & Leacy., 2013).

One tool to rule them all

Because there is no a priori way to know which conditioning method will work best for a given sample, users should try several methods, and these methods are spread across various packages; for example, full matching is available only in MatchIt, generalized boosted modeling only in twang, covariate balancing propensity score weighting only in CBPS, and genetic matching only in MatchIt and Matching. If a user wants to compare these methods on their ability to generate balance in the sample, they cannot do so on the same metrics and with the same output. Each package computes balance statistics differently, and the relevant balance measures are in different places in each package, or not available at all. By using cobalt to assess balance across packages, users can be sure they are using a single, equivalent balance metric across methods, and the relevant balance statistics will be in the same place and computed the same way regardless of the conditioning package used.

Flexibility

cobalt gives users choice for what statistics are presented and how they are calculated, but intelligently uses defaults that are in line with the goals of unified balance assessment and with the data available. Rather than displaying all values calculated, bal.tab() only displays what the user wants; at a bare minimum, the standardized mean difference for each covariate is displayed, which is traditionally considered sufficient for model selection and justification in preprocessing analysis. Even if the user doesn’t want other values displayed, they are all still calculated, and thus available for use in programming.

Pretty plots

The main conditioning packages produce plots that can be useful in assessing balance, summarizing balance, and understanding the intricacies of the conditioning method for which simple text would be insufficient. Many of these plots are unique to each package, and cobalt has not attempted to replace or replicate them. For other plots, though, cobalt uses ggplot2 to present clean, clear, customizable, and high-quality displays for balance assessment and presentation. The two included plotting functions are bal.plot(), which generates plots of the distributions of covariates in the groups so that more complete distributional balance can be assessed beyond numerical summaries, and love.plot(), which generates a plot summarizing covariate balance before and after conditioning, popularized by Dr. Thomas E. Love. Because these plots use ggplot2 as their base, users familiar with ggplot2 can customize various elements of the plots for use in publications or presentations.

f.build

f.build() is a small tool that can be helpful in quickly specifying formula inputs to functions. An example is provided below:

data(lalonde, package = "cobalt")
covs <- subset(lalonde, select = -c(treat, re78))
f.build(treat, covs)

## treat ~ age + educ + black + hispan + married + nodegree + re74 + 
##     re75
## <environment: 0x0000000006890be0>

The function creates a formula object from two inputs: the first argument is the unquoted name of the variable to be left hand side (response) variable in the formula; the second argument is a data frame, the variable names of which are to be the right hand side (predictor) variables. The utility of f.build() is that the user does not have to manually type out the name of every covariate when entering a formula into a function. It can be used simply in place of a formula, as in the following examples, which make use of the variables defined above:

# Generating propensity scores using logistic regression
p.score <- glm(f.build(treat, covs), data = lalonde, family = "binomial")$fitted.values

# Using matchit() from the MatchIt package
library("MatchIt")
m.out <- matchit(f.build(treat, covs), data = lalonde, method = "nearest")

f.build() can also be used in the Matching and formula interfaces in bal.tab().

Note: f.build() uses the first argument exactly as it is given, with no evaluation or checking; therefore, the named variable does not have to exist in any data set (but, of course, it will be useful if it does). The names of the second argument are used to create the right hand side; therefore, the data frame does not actually have to contain the observations of the variables, just columns with names corresponding to the desired variables. In this way, users can add variables to be included in f.build() by expanding the data frame with a new variable without having to fill in the variable with its true values.

bal.tab

bal.tab() is the primary function of cobalt. It produces balance tables for the objects given as inputs. The balance tables can be customized with a variety of inputs, which affect both calculation and presentation of values. It performs similar functions to summary() in MatchIt; bal.table(), summary(), and dx.wts() in twang; MatchBalance() and summary() in Matching; and balance() in CBPS. It can be seen as a replacement or a supplement to these functions.

For more help using bal.tab(), see ?bal.tab in R, which contains information on how certain values are calculated and links to the help files for the bal.tab() methods that integrate with the above packages.

For simplicity, the description of the use of bal.tab() will be most complete in its use with MatchIt. The demonstration will display bal.tab()’s many options, most of which are not unique to its use with MatchIt. The other demonstrations will be minimal, highlighting how to use bal.tab() effectively with the other packages, but not detailing all its possible options with those packages, to avoid redundancy.

data(lalonde, package = "cobalt")
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

# Nearest neighbor 1:1 matching with replacement
library("MatchIt") #if not yet loaded
m.out <- matchit(f.build(treat, covs0), data = lalonde, method = "nearest", replace = TRUE)
bal.tab(m.out)

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "nearest", 
##     replace = TRUE)
## 
## Balance Measures:
##               Type Diff.Adj
## .distance Distance  -0.0002
## age        Contin.   0.1964
## educ       Contin.  -0.0726
## black       Binary   0.0108
## hispan      Binary  -0.0054
## re74       Contin.   0.0667
## re75       Contin.   0.0936
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched        83     185
## Unmatched     346       0
## Discarded       0       0

bal.tab(m.out, binary = "std", continuous = "std") #Same as MatchIt's summary(..., standardize = TRUE)

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "nearest", 
##     replace = TRUE)
## 
## Balance Measures:
##               Type Diff.Adj
## .distance Distance  -0.0002
## age        Contin.   0.1964
## educ       Contin.  -0.0726
## black       Binary   0.0297
## hispan      Binary  -0.0228
## re74       Contin.   0.0667
## re75       Contin.   0.0936
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched        83     185
## Unmatched     346       0
## Discarded       0       0

# Balance on all covariates in data set, including interactions and squares
bal.tab(m.out, addl = lalonde[,c("nodegree", "married")], int = TRUE)

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "nearest", 
##     replace = TRUE)
## 
## Balance Measures:
##                      Type Diff.Adj
## .distance        Distance  -0.0002
## age               Contin.   0.1964
## educ              Contin.  -0.0726
## black              Binary   0.0108
## hispan             Binary  -0.0054
## re74              Contin.   0.0667
## re75              Contin.   0.0936
## nodegree           Binary   0.0486
## married            Binary  -0.0541
## age_2             Contin.   0.1024
## educ_2            Contin.  -0.1181
## re74_2            Contin.   0.0741
## re75_2            Contin.   0.0745
## age_educ          Contin.   0.1190
## age_black         Contin.   0.1197
## age_hispan        Contin.   0.0088
## age_re74          Contin.   0.0303
## age_re75          Contin.   0.0824
## age_nodegree      Contin.   0.1670
## age_married       Contin.  -0.2090
## educ_black        Contin.   0.0039
## educ_hispan       Contin.  -0.0278
## educ_re74         Contin.   0.0260
## educ_re75         Contin.   0.0602
## educ_nodegree     Contin.   0.1262
## educ_married      Contin.  -0.1502
## black_hispan       Binary   0.0000
## black_re74        Contin.   0.0626
## black_re75        Contin.   0.0862
## black_nodegree     Binary   0.0324
## black_married      Binary  -0.0270
## hispan_re74       Contin.   0.0525
## hispan_re75       Contin.   0.0838
## hispan_nodegree    Binary   0.0054
## hispan_married     Binary   0.0000
## re74_re75         Contin.   0.0690
## re74_nodegree     Contin.   0.1138
## re74_married      Contin.   0.0484
## re75_nodegree     Contin.   0.1774
## re75_married      Contin.   0.0233
## nodegree_married   Binary   0.0162
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched        83     185
## Unmatched     346       0
## Discarded       0       0

# Balance tables with variance ratios and statistics for the unadjusted sample
bal.tab(m.out, disp.v.ratio = TRUE, un = TRUE)

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "nearest", 
##     replace = TRUE)
## 
## Balance Measures:
##               Type Diff.Un V.Ratio.Un Diff.Adj V.Ratio.Adj
## .distance Distance  0.3786             -0.0002            
## age        Contin. -0.3094     2.2728   0.1964      1.5297
## educ       Contin.  0.0550     2.0166  -0.0726      1.4078
## black       Binary  0.6404              0.0108            
## hispan      Binary -0.0827             -0.0054            
## re74       Contin. -0.7211       1.93   0.0667      1.4254
## re75       Contin. -0.2903     1.0457   0.0936      1.4696
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched        83     185
## Unmatched     346       0
## Discarded       0       0

# Balance tables with thresholds for mean differences and variance ratios
bal.tab(m.out, m.threshold = .1, v.threshold = 2)

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "nearest", 
##     replace = TRUE)
## 
## Balance Measures:
##               Type Diff.Adj        M.Threshold V.Ratio.Adj  V.Threshold
## .distance Distance  -0.0002                                            
## age        Contin.   0.1964 Not Balanced, >0.1      1.5297 Balanced, <2
## educ       Contin.  -0.0726     Balanced, <0.1      1.4078 Balanced, <2
## black       Binary   0.0108     Balanced, <0.1                         
## hispan      Binary  -0.0054     Balanced, <0.1                         
## re74       Contin.   0.0667     Balanced, <0.1      1.4254 Balanced, <2
## re75       Contin.   0.0936     Balanced, <0.1      1.4696 Balanced, <2
## 
## 
## Balance tally for mean differences:
##                    count
## Balanced, <0.1         5
## Not Balanced, >0.1     1
## 
## 
## Variable with the greatest mean difference:
##     Diff.Adj        M.Threshold
## age   0.1964 Not Balanced, >0.1
## 
## 
## Balance tally for variance ratios:
##                  count
## Balanced, <2         4
## Not Balanced, >2     0
## 
## 
## Variable with the greatest variance ratio:
##     V.Ratio.Adj  V.Threshold
## age      1.5297 Balanced, <2
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched        83     185
## Unmatched     346       0
## Discarded       0       0

# Subclassification with 6 subclasses (the default in MatchIt)
m.out.sub <- matchit(f.build(treat, covs0), data = lalonde, method = "subclass")

bal.tab(m.out.sub, disp.subclass = TRUE)

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "subclass")
## 
## Balance by subclass:
##  - - - Subclass 1 - - - 
##               Type Diff.Adj
## .distance Distance   0.0458
## age        Contin.  -0.5026
## educ       Contin.   0.1733
## black       Binary   0.0587
## hispan      Binary   0.1775
## re74       Contin.  -0.5534
## re75       Contin.  -0.1218
## 
##  - - - Subclass 2 - - - 
##               Type Diff.Adj
## .distance Distance   0.0044
## age        Contin.  -0.7229
## educ       Contin.   0.1530
## black       Binary   0.0000
## hispan      Binary   0.0000
## re74       Contin.   0.0655
## re75       Contin.   0.1939
## 
##  - - - Subclass 3 - - - 
##               Type Diff.Adj
## .distance Distance   0.0009
## age        Contin.   0.5897
## educ       Contin.  -0.0917
## black       Binary   0.0000
## hispan      Binary   0.0000
## re74       Contin.  -0.1047
## re75       Contin.  -0.2348
## 
##  - - - Subclass 4 - - - 
##               Type Diff.Adj
## .distance Distance  -0.0006
## age        Contin.   0.1898
## educ       Contin.  -0.2842
## black       Binary   0.0000
## hispan      Binary   0.0000
## re74       Contin.  -0.0656
## re75       Contin.   0.1585
## 
##  - - - Subclass 5 - - - 
##               Type Diff.Adj
## .distance Distance  -0.0027
## age        Contin.   0.6183
## educ       Contin.  -0.2132
## black       Binary   0.0000
## hispan      Binary   0.0000
## re74       Contin.  -0.0474
## re75       Contin.   0.0302
## 
##  - - - Subclass 6 - - - 
##               Type Diff.Adj
## .distance Distance  -0.0046
## age        Contin.   0.0902
## educ       Contin.  -0.0791
## black       Binary   0.0000
## hispan      Binary   0.0000
## re74       Contin.  -0.0553
## re75       Contin.  -0.3557
## 
## Balance measures across subclasses:
##               Type Diff.Adj
## .distance Distance   0.0078
## age        Contin.   0.2071
## educ       Contin.   0.0738
## black       Binary   0.0098
## hispan      Binary   0.0297
## re74       Contin.   0.0963
## re75       Contin.   0.0849
## 
## 
## Sample sizes by subclass:
##         Subclass 1 Subclass 2 Subclass 3 Subclass 4 Subclass 5 Subclass 6
## Control        344         26         17         21          7         14
## Treated         31         31         30         31         31         31
## Total          375         57         47         52         38         45

library("twang")
data(lalonde, package = "cobalt") ##If not yet loaded
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

ps.out <- ps(f.build(treat, covs0), data = lalonde, stop.method = c("es.mean",
             "es.max"), estimand = "ATT", n.trees = 1000, verbose = FALSE)
bal.tab(ps.out, full.stop.method = "es.mean.att")

## 
## Call:
## ps(formula = f.build(treat, covs0), data = lalonde, n.trees = 1000, 
##     verbose = FALSE, estimand = "ATT", stop.method = c("es.mean", 
##         "es.max"))
## 
## Balance Measures:
##               Type Diff.Adj
## .distance Distance   0.1263
## age        Contin.   0.0135
## educ       Contin.  -0.0861
## black       Binary   0.0446
## hispan      Binary   0.0113
## re74       Contin.  -0.0042
## re75       Contin.   0.0468
## 
## 
## Effective sample sizes:
##            Control Treated
## Unadjusted  429.00     185
## Adjusted     53.45     185

library("Matching")
data(lalonde, package = "cobalt") #If not yet loaded
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

fit <- glm(f.build(treat, covs0), data = lalonde, family = "binomial")
p.score <- fit$fitted.values
match.out <- Match(Tr = lalonde$treat, X = p.score, estimand = "ATT")

bal.tab(match.out, formula = f.build(treat, covs0), data = lalonde)

## 
## Balance Measures:
##           Type Diff.Adj
## age    Contin.   0.2144
## educ   Contin.  -0.1025
## black   Binary   0.0108
## hispan  Binary  -0.0070
## re74   Contin.   0.0500
## re75   Contin.   0.0886
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched       159     185
## Unmatched     270       0

bal.tab(match.out, treat = lalonde$treat, covs = covs0)

library("CBPS")
data(lalonde, package = "cobalt") #If not yet loaded
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

cbps.out <- CBPS(f.build(treat, covs0), data = lalonde, standardize = FALSE)

## [1] "Finding ATT with T=1 as the treatment.  Set ATT=2 to find ATT with T=0 as the treatment"

bal.tab(cbps.out)

## 
## Call:
## CBPS(formula = f.build(treat, covs0), data = lalonde, standardize = FALSE)
## 
## Balance Measures:
##               Type Diff.Adj
## .distance Distance  -0.0034
## age        Contin.  -0.0081
## educ       Contin.   0.0055
## black       Binary  -0.0002
## hispan      Binary   0.0000
## re74       Contin.  -0.0014
## re75       Contin.  -0.0018
## 
## 
## Effective sample sizes:
##            Control Treated
## Unadjusted   429.0     185
## Adjusted     110.8     185

# Generating ATT weights as specified in Austin (2011)

data(lalonde, package = "cobalt") #If not yet loaded
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

fit <- glm(f.build(treat, covs0), data = lalonde, family = "binomial")
lalonde$p.score <- fit$fitted.values
lalonde$att.weights <- with(lalonde, treat + (1-treat)*p.score/(1-p.score))

bal.tab(covs0, treat = lalonde$treat, weights = lalonde$att.weights,
        method = "weighting")

## 
## Balance Measures:
##           Type Diff.Adj
## age    Contin.   0.1112
## educ   Contin.  -0.0641
## black   Binary  -0.0044
## hispan  Binary   0.0016
## re74   Contin.  -0.0039
## re75   Contin.  -0.0428
## 
## 
## Effective sample sizes:
##            Control Treated
## Unadjusted   429.0     185
## Adjusted     108.2     185

bal.tab(f.build(treat, covs0), data = lalonde, weights = "att.weights",
        distance = "p.score", method = "weighting")

## 
## Balance Measures:
##               Type Diff.Adj
## .distance Distance  -0.0084
## age        Contin.   0.1112
## educ       Contin.  -0.0641
## black       Binary  -0.0044
## hispan      Binary   0.0016
## re74       Contin.  -0.0039
## re75       Contin.  -0.0428
## 
## 
## Effective sample sizes:
##            Control Treated
## Unadjusted   429.0     185
## Adjusted     108.2     185

bal.plot

The gold standard for assessing covariate balance is multidimensional distributional similarity between the treated and control groups. Because this is hard to visualize and assess with the large numbers of covariates typical of causal effect analysis, univariate balance is typically assessed as a proxy. Most conditioning packages, as well as cobalt, will provide numerical summaries of balance, typically by comparing moments between the treated and control groups. But even univariate balance is more complicated than simple numerical summaries can address; examining distributional balance is a more thorough method to assess balance between groups. Although there are statistics such as the Komologrov-Smirnov distance that summarize distributional balance beyond the first few moments (Ali et al., 2016; Austin & Stuart, 2015), complimenting statistics with a visual examination of the distributional densities can be an effective way of assessing distributional similarity between the groups (Ho et al., 2007).

bal.plot() allows users to do so by displaying density plots (for continuous variables) or bar graphs (for binary variables) for the treated and control groups so that users can compare the distributional shapes between them. Below is an example of the use of bal.plot() after using MatchIt to perform matching:

data(lalonde, package = "cobalt")
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

# Nearest neighbor 1:1 matching with replacement
library("MatchIt") #if not yet loaded
m.out <- matchit(f.build(treat, covs0), data = lalonde, method = "nearest", replace = TRUE)

bal.plot(m.out, "age")
bal.plot(m.out, "black")

The first argument (or set of arguments) is the sufficient set of arguments for a simple call to bal.tab(), defining the data object (e.g., the output of a conditioning function), the treatment indicators, and the weights or subclasses. See above for examples. The second argument is the name of the covariate for which distributional balance is to be assessed. If subclassification is used (i.e., if subclasses are present in the input data object or arguments), an additional argument which.sub must be specified, with a number corresponding to the subclass number for which balance is to be assessed on specified covariate.

The user can also specify whether distributional balance is to be shown before or after adjusting by using the argument to un, as with bal.tab(). If un = TRUE, balance will be displayed for the unadjusted sample only. The default is to display balance for the adjusted sample.

The output of bal.plot() is a density plot for the two groups on the given covariate. For categorical or binary variables, a bar graph is displayed instead. The x-axis is the value of the variable, and the y-axis is the proportion density or proportion of the variable in the given group. When multi-category categorical variables are given, bars will be created for each level, unlike in bal.tab(), which splits the variable into several binary variables. The degree to which the density plots for the two groups overlap is a good measure of group balance on the given covariate; significant differences in shape can be indicative of poor balance, even when the mean differences and variance ratios are well within thresholds. Strong distributional similarity is especially important for variables strongly related to the outcome of interest.

The output plot is made using ggplot2, which means that users familiar with ggplot2 can adjust the plot with ggplot2 commands. For example, to change the colors and theme to black and white, and to change the axis and title names, the following code can be used:

library("ggplot2")
bp <- bal.plot(m.out, "age")
bp + theme_bw() + scale_fill_manual( values = c("black","white")) + 
    labs(title = "Distributional Balance for Age", x = "Age")

Distributional balance can also be assessed on the distance measure, and this can form an alternative to other common support checks, like MatchIt’s plot(..., type = "hist") or twang’s plot(..., plots = "boxplot"). To examine the distributions of the distance measure, the input to var.name must be ".distance" (noting the “.” at the beginning). If the data input object doesn’t already contain the distance measure (e.g., if not using one of the conditioning packages), the distance measure must be manually added as an input to bal.plot() through distance, in addition to being called through var.name. Below is an example using the formula input to bal.plot() with to display the distributions of propensity scores before and after adjustment:

data(lalonde, package = "cobalt") #If not yet loaded
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

fit <- glm(f.build(treat, covs0), data = lalonde, family = "binomial")
lalonde$p.score <- fit$fitted.values
lalonde$att.weights <- with(lalonde, treat + (1-treat)*p.score/(1-p.score))

bal.plot(f.build(treat, covs0), data = lalonde, var.name = ".distance",
         weights = "att.weights", distance = "p.score", 
         method = "weighting", un = TRUE)
bal.plot(f.build(treat, covs0), data = lalonde, var.name = ".distance",
         weights = "att.weights", distance = "p.score", 
         method = "weighting", un = FALSE)

It is generally not a useful assessment of balance to examine the overlap of the distance measure distributions after adjustment, as most conditioning methods will yield good distributional overlap on the distance measure whether or not balance is achieved on the covariates (Stuart et al., 2013). However, it may be useful to see the new range of the distance measure if calipers or common support pruning are used.

love.plot

The Love plot is a summary plot of covariate balance before and after conditioning popularized by Dr. Thomas E. Love. In a visually appealing and clear way, balance can be presented to demonstrate to readers that balance has been met within a threshold, and that balance has improved after conditioning (which is not always the case; cf. King & Nielson, 2016). Although Love plots are used with some frequency and there are tools to generate them, none of the packages used for conditioning include a function for quickly and easily generating them. love.plot() does just this, providing the user with several options to customize their plot for presentation. Below is an example of its use:

data(lalonde, package = "cobalt")
covs0 <- subset(lalonde, select = -c(treat, re78, nodegree, married))

# Nearest neighbor 1:1 matching with replacement
library("MatchIt") #if not yet loaded
m.out <- matchit(f.build(treat, covs0), data = lalonde, method = "nearest", replace = TRUE)

love.plot(bal.tab(m.out), threshold = .1)

love.plot() takes as its first argument the output of a call to bal.tab(); this can be accomplished simply by inserting the call into the first argument or by saving the result of a call to bal.tab() to an object and inserting the object as the argument. There are several other arguments, all of which control display, and are described below.

The output is a plot with the distance measure on the X-axis and the covariates output in bal.tab() on the Y-axis. Each point represents the balance statistic for that covariate, colored based on whether it is calculated before or after adjustment. The dotted lines represent the threshold set in the threshold argument; if most or all of the points after adjustment are within the threshold, that is good evidence that balance has been achieved. Like the output of bal.plot(), the output of love.plot() is a ggplot2 object, which means ggplot2 users can modify the plot to their liking for presentation or publication. This might involve changing the colors, shapes, or labels used in the plot.

The default is to present mean differences as they are calculated in the call to bal.tab(); by specifying stat = "variance.ratios" (abbreviations allowed), the user can request variance ratios instead. Because some variables don’t have variance ratios calculated (e.g., binary variables), there will be empty rows if variance ratios are requested, but these rows can be deleted by setting no.missing = TRUE.

The threshold works similarly to m.threshold and v.threshold in bal.tab(); specifying it is optional, but doing so will provide an additional point of reference on which to evaluate the displayed balance measures. If these thresholds are specified in bal.tab(), love.plot() will use them in the resultant plot with the corresponding balance statistic. The threshold argument to love.plot() takes precedence.

If mean difference are requested, love.plot() will use the mean differences as they are calculated by bal.tab() and presented in the mean differences columns of the balance table. See the section on using bal.tab() to see what the default calculations are for these values. If abs = TRUE in love.plot(), the plot will display absolute mean differences, which can aid in display clarity since the magnitude is generally the more important aspect of the statistic.

The order of the covariates displayed can be adjusted using the argument to var.order. If left empty or NULL, the covariates will be listed in alphabetical order. If "adjusted", covariates will be ordered by the requested balance statistic of the adjusted sample. If "unadjusted", covariates will be ordered by the requested balance statistic of the unadjusted sample, which tends to be more visually appealing. Abbreviations are allowed here.

The plot uses the original variable names as they are given in the data set, which may not be the names desired for display in publication. By using the argument to var.names, users can specify their own variable names to be used instead. This can also be done using ggplot2 commands after the plot has been generated. To specify new variable names with var.names, the user must enter an object containing the new variable names and, optionally, the old variable names to replace. For options of how to do so, see the help file for love.plot() with ?love.plot. Below is an example, creating a publication-ready plot:

v <- data.frame(old = c("age", "educ", "black", "hispan", 
                        "re74", "re75"),
                new = c("Age", "Years of Education", "Black", 
                        "Hispanic", "Earnings 1974", 
                        "Earnings 1975"))
                
love.plot(bal.tab(m.out), stat = "mean.diffs", threshold = .1, 
          var.order = "unadjusted", var.names = v, abs = TRUE)

This plot shows that balance was improved on all variables after adjustment, bringing all but one below the threshold of .1 for absolute mean differences.

Using the prognostic score for balance assessment

The prognostic score is the model-predicted outcome for an individual, excluding the treatment variable in the model (Hansen, 2008). Stuart et al. (2013) found that prognostic scores can be an extremely effective tool for assessing balance, greatly outperforming mean differences on covariates and significance tests. This is true even if the prognostic score model is slightly misspecified. Although the use of prognostic scores appears to violate the spirit of preprocessing in that users observe the outcome variable prior to treatment effect estimation, typically the prognostic score model is estimated in just the control group, so that the outcome of the treated group (which may contain treatment effect information) is excluded from analysis.

Assessing balance on the prognostic score is simple in cobalt, and highly recommended when available. The steps are:

1. Estimate the outcome model in the control group
2. Generate model-predicted outcome values for both the treated and control groups
3. Assess balance on prognostic scores by comparing standardized mean differences

To use prognostic scores in cobalt, simply add the prognostic score as a variable in the argument to addl. Below is an example of how to do so after a call to matchit():

ctrl.data <- lalonde[lalonde$treat == 0,]
ctrl.fit <- glm(re78 ~ age + educ + black + hispan + 
                married + nodegree + re74 + re75,
                data = ctrl.data)
lalonde$prog.score <- predict(ctrl.fit, lalonde)

bal.tab(m.out, addl = data.frame(prog.score = lalonde$prog.score))

## 
## Call:
## matchit(formula = f.build(treat, covs0), data = lalonde, method = "nearest", 
##     replace = TRUE)
## 
## Balance Measures:
##                Type Diff.Adj
## .distance  Distance  -0.0002
## age         Contin.   0.1964
## educ        Contin.  -0.0726
## black        Binary   0.0108
## hispan       Binary  -0.0054
## re74        Contin.   0.0667
## re75        Contin.   0.0936
## prog.score  Contin.   0.0573
## 
## 
## Sample sizes:
##           Control Treated
## All           429     185
## Matched        83     185
## Unmatched     346       0
## Discarded       0       0

Although the prognostic score is sensitive to the outcome estimation model used, a defensible prognostic score model can yield valid prognostic scores, which can then be used in balance assessment. In the above example, balance on the estimated prognostic score was good, so we can have some confidence that the effect estimate will be relatively unbiased, even though the “age” variable remains imbalanced. The logic is that that age is not a highly prognostic variable, which could be demonstrated by examining the standardized regression output of prognostic score model, so even though imbalance remains, such imbalance is unlikely to affect the effect estimate. The variables “re74” and “re75”, though, which are highly prognostic of the outcome, are quite balanced, thereby supporting an unbiased treatment effect estimate.

Variance in Standardized Mean Differences

When computing a standardized mean difference, the raw mean difference is divided by a standard deviation, yielding a d-type effect size statistic. In bal.tab(), the user can control whether the standard deviation is that of the treated group or control group or a pooled estimate, calculated as the square root of the average of the group variances. In most applications, the standard deviation corresponding to the default for the method is the most appropriate.

A key detail is that the standard deviation, no matter how it is calculated, is always calculated using the unadjusted sample¹. This is line with how MatchIt computes standardized mean differences, and is recommended by Stuart (2008; 2010). One reason to favor the use of the standard deviation of the unadjusted sample is that it prevents the paradoxical situation that occurs when adjustment decreases both the mean difference and the spread of the sample, yielding a larger standardized mean difference than that prior to adjustment, even though the adjusted groups are now more similar. By using the same standard deviation before and after adjusting, the change in balance is isolated to the change in mean difference, rather than being conflated with an accompanying change in spread. It is important to note that both twang and Matching calculate standardized mean differences using the standard deviation of the sample in question, not the unadjusted sample, though CBPS uses the standard deviation of the unadjusted sample.

Weighted Variance

When using weighting or matching, summary values after adjustment are calculated by using weights generated from the matching or weighting process. For example, group means are computed using the standard formula for a weighted mean, incorporating the weighting or matching weights into the calculation. To estimate a weighted sample variance for use in variance ratios², there are two formulas that have been proposed:

1. \[\frac{\sum_{i=1}^{n} w_{i}(x_{i} - \bar{x}_{w})^2}{(\sum_{i=1}^{n} w_{i}) - 1}\]

2. \[\frac{\sum_{i=1}^{n} w_{i}}{(\sum_{i=1}^{n} w_{i})^2 - \sum_{i=1}^{n} w^2_{i}} \sum_{i=1}^{n} w_{i}(x_{i} - \bar{x}_{w})^2\]

The weights used in the first formula are often called “frequency weights”, while the weights in the second formula are often called normalized or “reliability weights”. MatchIt, twang, and Matching all use the first formula when calculating any weighted variance (CBPS does not compute a weighted variance). However, Austin (2008) and Austin & Stuart (2015) recommend the second formula when considering matching weights for k:1 matching or weights for propensity score weighting. In cobalt the first formula is used to remain in line with the other packages. For some applications (i.e., when all weights are either 0 or 1), the two formulas yield the same variance estimate.

Test Statistics and P-values

Some of the early literature on propensity score matching included measures for balance assessment that relied on hypothesis tests for the independence of treatment assignment and covariates after adjustment (e.g., Rosenbaum & Rubin, 1983; Hansen, 2004). In a review of propensity score applications in the social sciences, Thoemmes & Kim (2011) found that over 66% of studies used significance tests to assess balance. Likewise, Austin (2008) found that over 70% of studies using propensity scores in the medical literature used significance tests to assess balance, a finding replicated by Ali et al. (2015). These hypothesis tests can come in many forms: t-tests for the difference in means between groups, chi-square tests for the difference in proportion between groups, Kolmogorov-Smirnov tests for the difference in cumulative density between groups, or F-tests for the difference in means between groups across subclasses.

The use of hypothesis tests appears natural here: if balance is achieved, we would not expect extreme values for the test statistics, and we can quantify the probability of observing imbalance as extreme as the one observed if no imbalance is present as a way to assess whether there is balance, as we can do with standard hypothesis testing. But this view is not shared by the methodological community: virtually all contemporary propensity score methodologists recommend against using hypothesis tests for balance assessment (e.g., Austin, 2009; 2011; Ho et al., 2007; Imai et al., 2008; Stuart, 2010; Thoemmes & Kim, 2011; Ali et al., 2015). There are logical reasons for this preference against hypothesis tests, noted clearly in Ali et al. (2015): they are influenced by sample size, which fluctuates during adjustment, and the theory behind them is inappropriate because balance is a quality solely of the sample in question, not as it relates to a population. The relevant information in a hypothesis test for group differences is the standardized magnitude of the group difference, and so such a measure is preferred.

Because hypothesis tests can be misleading and their use is discouraged by leading methodologists, they have been completely excluded in cobalt in favor of summary statistics. This stands in contrast to twang and Matching, both of which report hypothesis test p-values in their balance output.

Q-Q Plots and Summaries

Q-Q plots have been recommended as tools to assess distributional balance of covariates between groups (Ho et al., 2007), and are implemented in MatchIt and Matching (twang implements them but for a different purpose). Statistics summarizing the degree of imbalance in Q-Q plots are also reported in both MatchIt and Matching. MatchIt’s summary() command by default reports “eQQ Med”, “eQQ Mean”, and “eQQ Max” for each covariate before and after adjustment. These are the median, mean, and maximum distance between the treated and control group empirical Q-Q plots on the scale of the covariate. When standardize = TRUE, the values reported are “eCDF Med”, “eCDF Mean”, and “eCDF Max”, which are the same values as above but computed using the empircal cumulative densities. The Matching package also provides all of these values. Values close to 0 indicate good balance between groups for the given covariate.

A weakness of empirical Q-Q plots is that they don’t reveal much about the differences in the shapes of the distributions between the groups, which is a key aspect of distributional balance. A density plot essentially contains the same information, but clearer and more intuitive. Although the assessment of balance using an empirical Q-Q plot is straightforward (i.e., deviations from the 45-degree line indicate imbalances), density plots present the information in a way more in line with the actual goals of conditioning, in particular, that the distributions of treated and control units are similar (Ho et al., 2007).

Empirical Q-Q plot summary statistics may be useful in quantifying imbalance, but currently there are no recommendations for their use, and no precedents in the applied literature either. Similar methods, the Kolmogorov-Smirnov distance and Levy distance, were reviewed in Ali et al. (2015), and were found to have significant limitations not shared by the standardized mean difference. Belitser et al. (2011) found empirically that both measures were poor predictors of bias, especially in small samples. Although there may be promise for the use of these statistics, without guidance researchers may not find them useful or worth the space in a quick balance summary.

Density Plots

As mentioned above, cobalt displays density plots and bar charts rather than empirical Q-Q plots for the assessment of distributional similarity. These charts are not standard in any of the conditioning packages, but can be an intuitive and helpful tool for deciding whether adjustment has yielded similar distributions between the groups for given covariates. Though there are no obvious heuristics for deciding how much dissimilarity is too much dissimilarity, density plots do avoid the sometimes confusing logic of empirical Q-Q plots in favor of simplicity and interpretability. Austin (2009) considered density plots as a compliment to empirical Q-Q plots to more finely examine balance after adjusting.

Variance Ratios

Although mean differences (including t-tests and chi-square tests) are the most reported balance statistic (Thoemmes & Kim, 2011), variance ratios have been recommended in the literature as a means to further examine balance between groups (Austin, 2009, Imai et al., 2008; Ho et al., 2007). When group variances are similar, the variance ratio will be close to 1. Common thresholds for the variance ratio for balanced groups are 1/2 and 2 (Rubin, 2001; Stuart, 2010), though ratios closer to 1 are preferred.

The Matching package is the only one of the conditioning packages that presents variance ratios for covariates. Although bal.tab() in cobalt does not display variance ratios by default, they can be easily requested and have thresholds set.

Distinguishing Continuous and Binary Variables

Continuous and binary variables are treated differently by default in bal.tab() and bal.plot(). For continuous variables, the standard manipulations apply: standardized mean differences, variance ratios, and density plots. For binary variables, raw differences in proportion and bar charts are preferable, and variance ratios are useless.

The value of standardized mean differences for continuous variables is that they are on the same scale so that they can be compared across variables, and they allow for a simple interpretation even when the details of the variable’s original scale are unclear to the analyst. None of these advantages are passed to binary variables because binary variables are already on the same scale (i.e., a proportion), and the scale is easily interpretable. In addition, the details of standardizing the proportion difference of a binary variable involve dividing the proportion difference by a variance, but the variance of a binary variable is a function of its proportion (Austin, 2009). Standardizing the proportion difference of a binary variable can yield the following counterintuitive result: if \(X_{T} = .2\) and \(X_{C} = .3\), the standardized difference in proportion would be different from that if \(X_{T} = .5\) and \(X_{C} = .6\), even though the expectation is that the balance statistic should be the same for both scenarios because both would yield the same degree of bias in the effect estimate.

MatchIt allows users to view either standardized mean differences for all covariates or raw differences for all covariates, and twang and Matching display standardized differences for all variables but calculate test statistics depending on whether the covariate is continuous or binary (CBPS does not calculate mean differences, but presents both standardized and unstandardized means for all covariates). cobalt allows the user to select how the differences are to be calculated separately for continuous and binary variables, and uses the intuitive default that mean differences for continuous variables should be standardized while proportion differences for binary variables should be raw.

Because the variance of a binary variable is a function only of its proportion, the variance ratio of a binary variable in two groups is a function only of their proportions, thereby containing no more information than the simple difference in proportion. Therefore, for binary variables, cobalt does not compute variance ratios, as they can be misleading. Matching, the only package that computes variance ratios for balance assessment, does so both for continuous and binary variables.

Love Plots

Love plots and other similar plots that graphically display covariate balance before and after matching appear with some frequency in the applied and methodological literature (e.g., Ahmed et al., 2007; Austin, 2009; Austin & Stuart, 2015). Several of the conditioning packages implement a similar plot: MatchIt’s plot(summary(..., standardize = TRUE)) and twang’s plot(..., plots = "es") present line plots showing the change in balance for each covariate. Below are examples of twang’s balance plot (above) and cobalt’s Love plot (below).

A Love plot displays the same information, but allows for the clear display of both the balance statistic value and the variable name, unlike the line plot above which displays only the balance statistic value. The Love plot in cobalt allows for a great deal of flexibility in appearance, and can present variance ratios as well as mean differences.