SCHNEIDER: System for Counterfactual Human Network Evaluation of Individual Differences, Errors, and Residuals

Background:

Models of human performance have traditionally relied on data averaged across a large number of participants to understand the impacts of variables on outcomes. Traditional approaches to accounting for individual differences have tried to accomplish this by trying to develop a more nuanced understanding of the impacts of individual variables on outcomes. Modern advances in computing power and machine learning allow for the exploration of more data-intensive approaches. The DARPA TAILOR program, of which SCHNEIDER is a part, starts from the assumption that traditional human models are valid for a wide range of people, and that we can capture more individual nuance by determining what information might be contained in the residuals of those traditional models. Once models can capture individual differences better in this fashion, they can be used to do more nuanced counterfactual reasoning about what conditions can be modified to maximize human performance.

SIFT Approach:

The SIFT approach is two-pronged. The first involves extraction of a Bayesian network structure from a large dataset. This extraction is mostly automated, but can be guided by subject-matter experts providing limits on possible network topologies, such as edges that must exist (whitelist) or cannot possibly exist (blacklist) in the network. This Bayesian network is then translated into a network in the SUNNY system, which will actually do the reasoning and counterfactual reasoning.

Benefits:

  • Increased understanding of how to construct models that are best able to customize recommendations to individual users so as to maximize performance.
  • Increased experience with complex data processing and machine learning that can be applied to future SIFT projects