Intelligence analysis is one of the major functions performed by an Army staff in battlefield management. In particular, intelligence analysts develop intelligence requirements based on the commander's information requirements, develop a collection plan, and then monitor messages from the battlefield with respect to the commander's information requirements.
 
 

The goal of the CoRAVEN project is to develop an intelligent collaborative multimedia system to support intelligence analysts. Key ingredients of our design approach include (1) significant knowledge engineering and iterative prototyping activities with domain experts, (2) graphical user interfaces to provide flexible support for the multiple tasks in which analysts are engaged, (3) the use of Bayesian belief networks as a way to structure inferences that relate observable data to the commander's information requirements, (4) sonification of data streams and alarms to support enhanced situation awareness,(5) collaboration technologies, and (6) psychological studies of reasoning and judgment under uncertainty
 
 

Cognitive systems engineering research studies human activity in context, identifies problems in human-machine interaction, and designs and evaluates technology solutions to those problems. The CoRAVEN project is an example of such research. The goal of this project is to study the process of Army intelligence collection management and analysis, define issues, and apply advanced technology towards solutions. In so doing, we plan to make substantive contributions to the theories and design methodologies of cognitive engineering as well as provide a proof-of-concept prototype that is a promising tool for the Army.
 
 

Currently, we have analyzed Army doctrine and other archival materials and conducted informal interviews and collaborative verbal protocol sessions with domain experts. Our analysis so far has highlighted (1) the problem of data overload and information filtering, (2) the importance of maps in reasoning and planning, and (3) significant individual differences in reasoning under uncertainty.
 
 

To address these problems, the CoRAVEN project is engaged in developing a proof-of-concept tool in which analysts are able to view spatial data (maps), temporal data (the synchronization matrix which represents the schedule of collection assets) and graph-based models for fusing evidence (Bayesian networks). The coordination of multiple views by a single user, and collaboration among multiple users, are also key issues. Furthermore, viewing rich dynamic data is not merely a visual process, but auditory as well; we are exploring methods for data sonification to better support situation assessment. Finally, we continue to engage domain experts in knowledge elicitation and design feedback sessions with our evolving prototypes as well as conduct more rigorous psychological experiments to validate portions of our approach and gain insight into the individual variability of our target population.
 
 

The work of Intelligence Collection Management (CM) and Analysis is a good example of a cognitively complex task for which cognitive tools such as CoRAVEN are needed. Doctrinally, CM consists of several sub-functions including Requirements Management, Mission Management, Asset Management, Analysis, and Dissemination. The earlier planning sub-functions turn the intelligence needs of the commander's operational plan into formalized Intelligence Requirements (IRs) and Priority Intelligence Requirements (PIRs), a Collection Plan organized as a set of Named Areas of Interest (NAIs), and a Synchronization Matrix for allocating limited collection resources to NAIs such that the PIRs and IRs can be satisfied in a timely manner. CM begins while operational preparations are still being made and is repeated as necessary during the course of operations. The later stages of the Intelligence Cycle consist of analyzing, communicating and presenting the intelligence gathered by collection assets. It is the responsibility of the Intelligence Officer, and the central organizing principle of this officer's staff, to present the commander with a full, timely, and organized account of the intelligence collected and its significance for the commander’s operational decision making.
 
 

CoRAVEN is intended to be a flexible resource for intelligence analysis by providing easy navigation among three interrelated models and views: information abstractions about how observable evidence maps to PIRs and IRs, spatial abstractions such as NAIs that are used to organize planning and analysis, and temporal view of the synchronization and operations matrices. Providing flexible ways for analysts to map among these three interrelated models/views is a critical feature of CoRAVEN.
 
 

Currently, CoRAVEN has been designed to support the analysis of collected information and the communication of its significance among the analysis staff and to decision makers, and in particular addresses the following challenges: 1) identifying all of the information relevant to a given decision, 2) efficiently and reliably assessing the significance of all of the relevant information, and 3) effectively communicating the significance and relevance of information to a given decision. CoRAVEN seeks to address these issues by using Bayesian networks (BNs) to structure the relationship of evidence to PIRs and IRs and providing a collaborative audio-visual environment for the visualization and sonification of BNs, their evidential sources, and their relationship to the geographic and temporal structure of the situation.
 
 

The name CoRAVEN comes from "Collaborative RAVEN". RAVEN is a research project on using Bayesian networks as a reasoning tool for intelligence analysts (Mengshoel and Wilkins, 1997, 1998a, 1998b]). Bayesian networks (BNs) are an important knowledge representation that are used for reasoning and learning under uncertainty [(Pearl, 1988]) [(Lauritzen & Spiegelhalter, 1988]). Probability theory and graph theory form their basis: random variables are nodes and conditional dependencies are edges in a directed acyclic graph. Edges typically point from cause to effect.
 
 

Consider a simple Bayesian network that consists of five nodes A, B, C, D, and E. In addition to the graph, there are conditional probability tables associated with each node V and its parents Pa(V), expressing the conditional probability Pr(V | Pa(V)). If the node D has two parents B and C, assuming discrete binary nodes with values {0,1}, Pr(D=0 | B=1, C=0)) is one of the entries in D’s conditional probability table. Static and temporal BN variants can be used to model static and dynamic environments such as battlefields [(Mengshoel & Wilkins, 1997]).
 
 

Inference in Bayesian networks is one focus of our research. The inference task of belief updating amounts to the following: Given evidence at node E and query node Q, infer posterior probability Pr(Q | E=ei). Any nodes in the network can be evidence or query nodes. For the example BN, this leads to different types of inference: diagnostic as in Pr(A | E=ei); causal as in Pr(E | A=aj), and mixed as in Pr(D | E=ei, A=aj). A variety of approaches to Bayesian network belief updating and belief revision have been investigated [(Pearl, 1988; Lauritzen & Spiegelhalter, 1988]). These inference algorithms vary in many respects: they are exact, approximate, or heuristic; work on singly or multiply connected graphs; and are used for different inference tasks. Computational hardness has been shown both for belief updating and belief revision. Research into non-exact algorithms for solving these tasks approximately or heuristically is therefore important.
 
 

A commercial BN tool, HUGIN, uses an exact algorithm known as cluster propagation [(Lauritzen & Spiegelhalter, 1988]). For sparse BNs this algorithm works well; however, for large and highly connected BNs, it can become too slow for practical use. For this reason, a heuristic approach to belief revision in BNs is also investigated [(Mengshoel & Wilkins, 1998a]). More specifically, we consider a BN as encoding a genetic algorithm (GA) fitness. This is a restriction on the fitness function, but probability theory in general and BNs in particular have proven sufficiently rich to make this an interesting restriction. Part of our research has focused on GA selection and BN abstraction, and we have shown promising results for GA-based belief revision [(Mengshoel & Wilkins, 1998b]) as well as integrating BN abstraction and refinement into GA-based belief revision [(Mengshoel & Wilkins, 1998a]).
 
 

CoRAVEN currently relies on the standard HUGIN implementation of BNs. Our approach is that each PIR and IR has an associated Bayesian network, with the top node being the PIR or IR itself, and the leaf nodes representing observable evidence. Thus, in our demonstration, analysts using CoRAVEN must navigate among a number of BNs (currently eight), where each BN can be fairly large (the largest networks in our demonstration are about 650 nodes). Hence, one critical issue is how analysts will be able to monitor dynamic updates to all these networks as messages are received from intelligence assets, thus triggering state changes in leaf nodes with inferences propagating throughout the networks. Part of our answer to this is data sonification, which is the subject of the next section.
 
 

Sonification is the transformation of numerical data into sound for purposes of observing that data. The essential research task of sonification is to identify and construct an intuitive perceptual space for the auditory display of data. This task includes the assimilation of technological, creative and scientific advances in sound synthesis and signal processing and in human perception and cognition (see Brady et al., 1996; Choi, 1997; http:// www.ncsa.uiuc.edu / ~audio).
 
 

The NCSA Sound Server (VSS) (Bargar et al., 1994) facilitates the application of sonification in scientific research by providing a distributed system and language for ubiquitous sound production in computational environments. The VSS supports both sound computation and sound authoring; the latter is the process of establishing automated relationships between objects or events in a silent computing application, and algorithms for sound production which operate in parallel to the silent application.
 
 

In CoRAVEN, sound authoring is applied to the Bayesian network display in two different ways: (1) as a way of monitoring the dynamic evolution of weights on the nodes and (2) as a means of users setting alarms related to certain node (Bargar and Choi, 1998). The complexity of the BN is difficult to visualize, particularly the relative contribution of internal nodes to the final outcome. We apply sound authoring in layers of musical patterns that represent the probabilities at internal nodes. The use of musical patterns facilitates the ability to maintain coherence when information from many nodes is presented at the same time. Temporal patterns provide a high-dimensional space for differentiating elements in a complex state. Gradient alarms may be configured to report the onset of special conditions at a node. A gradient alarm informs a listener continuously as a system approaches or recedes from a designated alarm state, by the degree of onset of a notable change in the auditory texture.
 
 

Sonification supports the background monitoring of parallel processes in eyes-busy, hands-busy scenarios. The system architecture supporting data-driven sound with distributed sound synthesis engines allows efficient transmission of sonification among remote collaborators, and can assist the rapid exchange of alternative versions of a PIR representation during the decision-making process.

Intelligence analysis is a multi-person process, and thus, CoRAVEN also needs to address issues in collaboration and cooperative problem solving. Our analysis has revealed a large repertoire of collaboration support features necessary. For example, analysts may want to share their current Bayesian networks with colleagues for comments, or may want to collaborate synchronously on setting alarms, or simply have shared displays of the networks as they are updated.
 
 

Thus, in CoRAVEN, we are designing two complementary demonstrations of collaboration technology. One demonstration is Java-based and will support shared displays and simple generic collaboration mechanisms such as real-time chatting. The other demonstration in Visual C++ exploits the Watch-and-Notify feature of the POET™ commercial object-oriented database management system to support synchronous collaboration with flexibility at run time.
 
 

Evaluations of decision support systems by users are generally recognized as crucial to successful implementation (Mahmood & Sniezek, 1989). However, evaluations of user input at the earliest design stages may be even more important. The first concern in evaluation CoRAVEN was the validation of the actual Bns used in CoRAVEN.
 
 

Two studies, reported in full in Sniezek & Chernyshenko, 1998, were conducted to obtain data on the subjective probabilities from six domain experts. The first study was designed to evaluate the probabilities in terms of calibration, i.e., the correspondence between experts’ probabilities and objective criteria. To make it possible to have objective criteria, a series of 80 choice items were written from military intelligence field manuals obtained from the Army Training and Doctrine Command. Each item had two alternative choices: a correct one taken from the manual and an incorrect distractor. Items were presented to the experts individually, via computer. The expert selected one of the alternatives and gave a subjective probability that this choice was correct. The data from all six experts revealed overconfidence bias. The mean probabilities (.79) exceeded proportions correct (.64) by a mean of .15. Although bias was minimal with the easiest items—those with a proportion correct over .75—it was especially severe with the more difficult items. This is of concern because the problems encountered by military intelligence officers will more often be best characterized as difficult. Another limiting factor of the probabilities was in terms of resolution, i.e., the ability to discriminate among difficulty levels. Mean probabilities of .75 were observed for items that were moderately difficult (mean proportion correct of .61) as well as for those that were extremely difficult (mean proportion correct of .35, which is .15 below chance). The conclusion from this study is that the probability assessments of army experts in military intelligence show the same biases as experts in a variety of other domains (cf. Sniezek, Paese, & Switzer, 1990). Additional research is needed to develop and test procedures for calibrating the probabilities that will be used in the BN. Strategies to be employed would include those based on the work of Chernyshenko & Sniezek, 1998, Paese & Sniezek, 1991; and Sniezek & Buckley, 1991
 
 
 
 

A second study with the same experts was designed to measure reliability and consensus for probabilities assigned to events from a sample BN. Consistency within each expert was satisfactory, with an average reliability coefficient of .69. However, expert agreement was poor. The average st. dev. across 26 probability judgments was over .16. Thus, it will be necessary to elicit probabilities from a larger sample of experts and apply correction procedures to the probabilities input to the BN. Again it must be emphasized that the problems noted here are the norm among domain experts and not unique to military intelligence analysts. To ensure a good foundation for the BN, future research should use techniques based on procedures for aggregating expert probability judgments (cf., Rantilla & Budescu, 1999; Sniezek & Buckley, 1995; Sniezek & Henry, 1989).
 
 

In the context of developing and evaluating CoRAVEN, there exist a number of other potential problems that have been identified in the psychological research literature. For example, obtaining contributions of unique as well as common knowledge from the multiple users of a collaborative interface such as CoRAVEN (Savadori, Van Swol, & Sniezek, 1988).
 
 

* The views and conclusions contained in this document are those of the authors and should not be interpreted as presenting the official policies, either expressed or implied, of the Army Research Laboratory or the US Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon..