Baseline: practical control variates for agent evaluation in zero-sum domains
Josh Davidson, Christopher Archibald, Michael Bowling · 2013
Agent evaluation in stochastic domains can be difficult. The commonplace approach of Monte Carlo evaluation can in-volve a prohibitive number of simulations when the variance of the outcome is high. In such domains, variance reduc-tion techniques are necessary, but these techniques require careful encoding of domain knowledge. This paper intro-duces baseline as a simple approach to creating low vari-ance estimators for zero-sum multi-agent domains with high outcome variance. The baseline method leverages the self play of any available agent to produce a control variate for variance reduction, subverting any extra complexity inher-ent with traditional approaches. The baseline method is also applicable in situations where existing techniques either require extensive implementation overhead or simply can-not be applied. Experimental variance reduction results are shown for both cases using the baseline method. Baseline is shown to surpass state-of-the-art techniques in three-player computer poker and is competitive in two-player computer poker games. Baseline also shows variance reduction in hu-man poker and in a mock Ad Auction tournament from the Trading Agent Competition, domains where variance reduc-tion methods are not typically employed.