A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis

Prepared by the US Government March 2009

This primer highlights structured analytic techniques—some widely used in the private

sector and academia, some unique to the intelligence profession. It is not a

comprehensive overview of how intelligence officers conduct analysis. Rather, the

primer highlights how structured analytic techniques can help one challenge judgments,

identify mental mindsets, stimulate creativity, and manage uncertainty. In short,

incorporating regular use of techniques such as these can enable one to structure

thinking for wrestling with difficult questions.

Introduction ………………………………………………………………………………………………….. 1

How To Use These Techniques ………………………………………………………………………. 5

Diagnostic Techniques …………………………………………………………………………………… 7

Key Assumptions Check …………………………………………………………………………. 7

Quality of Information Check ………………………………………………………………….. 10

Indicators or Signposts of Change ………………………………………………………….. 12

Analysis of Competing Hypotheses (ACH) ………………………………………………. 14

Contrarian Techniques ………………………………………………………………………………….. 17

Devil’s Advocacy …………………………………………………………………………………… 17

Team A/Team B ……………………………………………………………………………………. 19

High-Impact/Low-Probability Analysis ……………………………………………………… 22

“What If?” Analysis ………………………………………………………………………………… 24

Imaginative Thinking Techniques …………………………………………………………………… 27

Brainstorming ……………………………………………………………………………………….. 27

Outside-In Thinking ……………………………………………………………………………….. 30

Red Team Analysis ………………………………………………………………………………… 31

Alternative Futures Analysis ……………………………………………………………………. 34

Strategies for Using Structured Analytic Techniques ………………………………………… 38

Selective Bibliography ………………………………………………………………………………….. 40

T A B L E O F C O N T E N T S

1

Intelligence analysts should be self-conscious about their reasoning processes. They should think about how they make judgments and reach conclusions, not just about the judgments and conclusions themselves.

—Richards Heuer, The Psychology of Intelligence Analysis

THE “MIND-SET” CHALLENGE

Using the analytic techniques contained in this primer will assist analysts in dealing with the perennial problems of intelligence: the complexity of international developments, incomplete and ambiguous information, and the inherent limitations of the human mind. Understanding the intentions and capabilities of adversaries and other foreign actors is challenging, especially when either or both are concealed. Moreover, transnational threats today pose even greater complexity, in that they involve multiple actors—including nonstate entities—that can adapt and transform themselves faster than those who seek to monitor and contain them. Finally, globalization has increased the diversity of outcomes when complex, interactive systems such as financial flows, regional economies or the international system as a whole are in flux.2

The first hurdle for analysts is identifying the relevant and diagnostic information from the increasing volume of ambiguous and contradictory data that is acquired through open source and clandestine means. Analysts must also pierce the shroud of secrecy—and sometimes deception—that state and nonstate actors use to mislead. A systematic approach that considers a range of alternative explanations and outcomes offers one way to ensure that analysts do not dismiss potentially relevant hypotheses and supporting information resulting in missed opportunities to warn.

Cognitive and perceptual biases in human perception and judgment are another important reason for analysts to consider alternatives. As Richards Heuer and others have argued, all individuals assimilate and evaluate information through the medium of “mental models” (sometimes also called “frames” or “mind-sets”). These are experience- based constructs of assumptions and expectations both about the world in general and more specific domains. These constructs strongly influence what information analysts will accept—that is, data that are in accordance with analysts’ unconscious mental models are more likely to be perceived and remembered than information that is at odds with them.

Mental models are critical to allowing individuals to process what otherwise would be an incomprehensible volume of information. Yet, they can cause analysts to overlook, reject, or forget important incoming or missing information that is not in accord with their assumptions and expectations. Seasoned analysts may be more susceptible to these mind-set problems as a result of their expertise and past success in using time-tested mental models. The key risks of mind- sets are that: analysts perceive what they expect to perceive; once formed, they are resistant to change; new information is assimilated, sometimes erroneously, into existing mental models; and conflicting information is often dismissed or ignored.

I N T R O D U C T I O N

1 Richards J. Heuer, Jr., The Psychology of Intelligence Analysis (Washington: Center for the Study of Intelligence, 1999).

2 These observations were drawn from a lengthier treatment of cognitive bias found in the Sherman Kent Center’s Occasional Paper, Making Sense of Transnational Threats, Vol. 3, No. 1, October 2004.

1

“

”

2

Common Perceptual and Cognitive Biases

Perceptual Biases

Expectations. We tend to perceive what we expect to perceive. More (unambiguous) information is needed to recognize an unexpected phenomenon.

Resistance. Perceptions resist change even in the face of new evidence.

Ambiguities. Initial exposure to ambiguous or blurred stimuli interferes with accurate perception, even after more and better information becomes available.

Biases in Evaluating Evidence

Consistency. Conclusions drawn from a small body of consistent data engender more confidence than ones drawn from a larger body of less consistent data.

Missing Information. It is difficult to judge well the potential impact of missing evidence, even if the information gap is known.

Discredited Evidence. Even though evidence supporting a perception may be proved wrong, the perception may not quickly change.

Biases in Estimating Probabilities

Availability. Probability estimates are influenced by how easily one can imagine an event or recall similar instances.

Anchoring. Probability estimates are adjusted only incrementally in response to new information or further analysis.

Overconfidence. In translating feelings of certainty into a probability estimate, people are often overconfident, especially if they have considerable expertise.

Biases in Perceiving Causality

Rationality. Events are seen as part of an orderly, causal pattern. Randomness, accident and error tend to be rejected as explanations for observed events. For example, the extent to which other people or countries pursue a coherent, rational, goal-maximizing policy is overestimated.

Attribution. Behavior of others is attributed to some fixed nature of the person or country, while our own behavior is attributed to the situation in which we find ourselves.

Intelligence analysts must actively review the accuracy of their mind-sets by applying structured analytic techniques that will make those mental models more explicit and expose their key assumptions. The techniques found in this primer are designed to assist in this regard by:

• Instilling more structure into the intelligence analysis.

• Making analytic arguments more transparent by articulating them and challenging key assumptions.

• Stimulating more creative, “out-of-the- box” thinking and examining alternative outcomes, even those with low probability, to see if available data might support these outcomes.

• Identifying indicators of change (or signposts) that can reduce the chances of surprise.

Incorporating findings derived from these techniques into our intelligence products also serves the policymaker by:

• Highlighting potential changes that would alter key assessments or predictions.

• Identifying key assumptions, uncertainties, intelligence gaps and disagreements that might illuminate risks and costs associated with policy choices.

• Exploring alternative outcomes for which policy actions might be necessary.

3

Strategic Assumptions That Were Not Challenged

1941 World War II

Japan would avoid all-out war because it recognized US military superiority.

Given that US superiority would only increase, Japan might view a first strike as the only way to knock America out of the war.

1950s Korean War

China would not cross the Yalu River in support of the North Korean government.

Red China could make good on its threats to counter “US aggression” against the North.

1962 Cuban Missile Crisis

The Soviet Union would not introduce offensive nuclear weapons into Cuba.

The Kremlin could miscalculate and believe it could create a fait accompli that a young US President would not be prepared to reverse.

1973 Yom Kippur War

Arabs knew they could not win because they had failed to cooperate in the past and still lacked sufficient air defenses to counter Israeli airpower.

A surprise Arab attack, even if repelled, could wound Israel psychologically and prompt international calls for cease-fires and diplomatic negotiations.

1989 German Unification

East Germany could not unify with the West Germany against the wishes of the Soviet Union.

The Soviet Union—under Gorbachev—might not be prepared to intervene militarily in Eastern Europe as it had in the past.

1998 Indian Nuclear Test

Conducting a nuclear test risked international condemnation and US sanctions and would threaten a newly elected coalition government.

A successful and surprise nuclear test could boost Indian nationalist pride and solidify public support for a shaky coalition government.

2003 Iraq’s WMD Programs

Saddam failed to cooperate with UN inspectors because he was continuing to develop weapons of mass destruction.

If Iraqi authorities had destroyed their WMD stocks and abandoned their programs, they might refuse to fully acknowledge this to the UN to maintain Iraq’s regional status, deterrence, and internal regime stability.

4

5

The analytic techniques in this primer are designed to help individual analysts, as well as teams, explore and challenge their analytical arguments and mind- sets. Some techniques are fairly simple to understand and employ—such as Brainstorming and Devil’s Advocacy. Others are more complex and demand a greater degree of analytical sophistication, resource commitment, and time. All the techniques are included because they have helped other analysts avoid rigid ways of thinking or assisted them in exploring new outcomes or implications of an intelligence problem.

The techniques are grouped by their purpose: diagnostic techniques are primarily aimed at making analytic arguments, assumptions, or intelligence gaps more transparent; contrarian techniques explicitly challenge current thinking; and imaginative thinking

H O W T O U S E T H E S E T E C H N I Q U E S

techniques aim at developing new insights, different perspectives and/or develop alternative outcomes. In fact, many of the techniques will do some combination of these functions. However, analysts will want to select the tool that best accomplishes the specific task they set out for themselves. Although application of these techniques alone is no guarantee of analytic precision or accuracy of judgments, it does improve the sophistication and credibility of intelligence assessments as well as their usefulness to policymakers. As Richards Heuer notes in his own work on cognitive bias, “analysis can be improved.”3

3 Heuer, Psychology of Intelligence Analysis, p. 184.

7

KEY ASSUMPTIONS CHECK

List and review the key working assumptions on which fundamental judgments rest.

WHEN TO USE A Key Assumptions Check is most useful at the beginning of an analytic project. An individual analyst or a team can spend an hour or two articulating and reviewing the key assumptions. Rechecking assumptions also can be valuable at any time prior to finalizing judgments, to insure that the assessment does not rest on flawed premises. Identifying hidden assumptions can be one of the most difficult challenges an analyst faces, as they are ideas held—often unconsciously—to be true and, therefore, are seldom examined and almost never challenged.

A key assumption is any hypothesis that analysts have accepted to be true and which forms the basis of the assessment. For example, military analysis may focus exclusively on analyzing key technical and military variables (sometimes called factors) of a military force and “assume” that these forces will be operated in a particular environment (desert, open plains, arctic conditions, etc.). Postulating other conditions or assumptions, however, could dramatically impact the assessment. Historically, US analysis of Soviet-Warsaw Pact operations against NATO had to “assume” a level of non- Soviet Warsaw Pact reliability (e.g., would these forces actually fight?). In this case, there was high uncertainty and depending on what level of reliability one assumed, the analyst could arrive at very different

conclusions about a potential Soviet offensive operation. Or when economists assess the prospects for foreign economic reforms, they may consciously, or not, assume a degree of political stability in those countries or the region that may or may not exist in the future. Likewise, political analysts reviewing a developing country’s domestic stability might unconsciously assume stable oil prices, when this key determinant of economic performance and underlying social peace might fluctuate. All of these examples highlight the fact that analysts often rely on stated and unstated assumptions to conduct their analysis. The goal is not to undermine or abandon key assumptions; rather, it is to make them explicit and identify what information or developments would demand rethinking them.

VALUE ADDED Explicitly identifying working assumptions during an analytic project helps:

• Explain the logic of the analytic argument and expose faulty logic.

• Understand the key factors that shape an issue.

• Stimulate thinking about an issue.

• Uncover hidden relationships and links between key factors.

• Identify developments that would cause you to abandon an assumption.

• Prepare analysts for changed circumstances that could surprise them.

D I A G N O S T I C T E C H N I Q U E S

8

Key Assumptions Check: The 2002 DC Sniper Case

The outbreak of sniper shootings in the Washington, DC area in the fall of 2002 provides a good example of how this technique could have been applied. After the initial flurry of shootings, the operating assumption that quickly emerged was that the shootings were the work of a single, white male who had some military training and was driving a white van. If law enforcement officials had conducted a Key Assumptions Check, they could have broken this statement into its key components and assessed the validity of each statement as follows:

Key Assumption Assessment

The sniper is a male.

The sniper is acting alone.

The sniper is white.

The sniper has military training/experience.

The sniper is driving a white van.

Highly likely (but not certain) given past precedent with serial killers. We are taking little risk by not looking for a female.

Highly likely (but not certain) given past precedents.

Likely, but not as certain, given past precedents. We would be taking some risk if we rule out nonwhites as suspects.

Possible, but not sufficient reason to exclude from consideration potential suspects who have not had any military training.

Possible because you have a credible eyewitness account but worthy of continuing scrutiny given the number of white vans in the area (more than 70,000 registered in the Maryland suburbs of Metropolitan Washington, DC) and that different kinds of vehicles are being described.

A Key Assumptions Check could have allowed law enforcement officials to:

• Avoid jumping to conclusions (the sniper is white, has military training, and is driving a white van) that did not hold up under closer scrutiny. By explicitly examining each assumption, officials could have avoided prematurely narrowing down the potential pool of suspects to a group that did not include the actual perpetrator. Similarly, they might have been more cautious about accepting that the sniper was driving a white van.

• Be receptive to new leads and citizen tips, such as eyewitness reports that the sniper fled the scene driving a specific model Chevrolet.

• More seriously consider evidence that subsequently became available, which contradicted a key assumption. If officials had stated explicitly that they were assuming the sniper was acting alone, they might have been sensitive to new information that contradicted that key assumption. Often this type of information gets “lost in the noise” if the analyst has not already thought about what key assumptions he or she is making.

9

THE METHOD Checking for key assumptions requires analysts to consider how their analysis depends on the validity of certain premises, which they do not routinely question or believe to be in doubt. A four- step process will help analysts:

1. Review what the current analytic line on this issue appears to be; write it down for all to see.

2. Articulate all the premises, both stated and unstated in finished intelligence, which are accepted as true for this analytic line to be valid.

3. Challenge each assumption, asking why it “must” be true and whether it remains valid under all conditions.

4. Refine the list of key assumptions to contain only those that “must be true” to sustain your analytic line; consider under what conditions or in the face of what information these assumptions might not hold.

QUESTIONS TO ASK DURING THIS PROCESS INCLUDE: • How much confidence exists that this

assumption is correct?

• What explains the degree of confidence in the assumption?

• What circumstances or information might undermine this assumption?

• Is a key assumption more likely a key uncertainty or key factor?

• Could the assumption have been true in the past but less so now?

• If the assumption proves to be wrong, would it significantly alter the analytic line? How?

• Has this process identified new factors that need further analysis?

10

QUALITY OF INFORMATION CHECK

Evaluates completeness and soundness of available information sources.

WHEN TO USE Weighing the validity of sources is a key feature of any critical thinking. Moreover, establishing how much confidence one puts in analytic judgments should ultimately rest on how accurate and reliable the information base is. Hence, checking the quality of information used in intelligence analysis is an ongoing, continuous process. Having multiple sources on an issue is not a substitute for having good information that has been thoroughly examined. Analysts should perform periodic checks of the information base for their analytic judgments. Otherwise, important analytic judgments can become anchored to weak information, and any “caveats” attached to those judgments in the past can be forgotten or ignored over time.

If a major analytic assessment is planned, analysts should individually or collectively review the quality of their information and refresh their understanding of the strengths and weaknesses of past reporting on which an analytic line rests. Without understanding the context and conditions under which critical information has been provided, it will be difficult for analysts to assess the information’s validity and establish a confidence level in an intelligence assessment.

VALUE ADDED A thorough review of information sources provides analysts—as well as policymakers—with an accurate assessment of “what we know” and “what we do not know.” It is also an opportunity to confirm that sources have been cited accurately. In the case of HUMINT, this will require extensive review of the sources’ background information and access as well as his or her motivation for providing the information. Similarly, reviewing technical sourcing can sometimes reveal inadvertent errors in processing, translation, or interpretation that otherwise might have gone unnoticed.

In addition, a quality of information check can be valuable to both collectors and policymakers:

• It can help to detect possible deception and denial strategies by an adversary.

• It can identify key intelligence gaps and new requirements for collectors.

• It can assist policymakers in understanding how much confidence analysts are placing on analytic judgments.

11

THE METHOD An analyst or a team might begin a quality of information check by developing a database in which information is stored according to source type and date, with additional notations indicating strengths or weaknesses in those sources.4 Ideally, analysts would have a retrieval and search capability on the database, so that periodic reviews are less labor intensive and result in a more complete review of all sources used in past analysis. For the information review to be fully effective, analysts will need as much background information on sources as is feasible. Knowing the circumstances in which reporting was obtained is often critical to understanding its validity. With the data in hand, analysts can then:

• Review systematically all sources for accuracy.

• Identify information sources that appear most critical or compelling.

• Check for sufficient and strong corroboration of critical reporting.

• Reexamine previously dismissed information in light of new facts or circumstances that cast it in a different light.

• Ensure that any recalled reporting is identified and properly flagged for other analysts; analysis based on recalled reporting should also be reviewed to determine if the reporting was essential to the judgments made.

• Consider whether ambiguous information has been interpreted and caveated properly.

• Indicate a level of confidence that analysts can place in sources, which are likely to figure in future analytic assessments.

Quality of Information Problem on Iraq

“. . . Analysts community wide are unable to make fully informed judgments on the information they received, relying instead on nonspecific source lines to reach their assessments. Moreover, relevant operational data is nearly always withheld from analysts, putting them at a further analytical disadvantage . . .”

—Senate Select Committee on Intelligence, Report on the US Intelligence Community’s Prewar Intelligence Assessments on Iraq

“Analytic errors included over-reliance on a single, ambiguous source, [and, in addition to collection shortfalls] failure of analysts to understand fully the limitations of technical collection . . .”

“The Intelligence Community relied too heavily on ambiguous imagery indicators . . .” —Commission on the Intelligence Capabilities of the United States Regarding

Weapons of Mass Destruction.

4 Analysis of Competing Hypotheses (ACH), discussed later in the primer, is a useful technique for exploring the possibility that deception could explain the absence of evidence.

12

INDICATORS OR SIGNPOSTS OF CHANGE

Periodically review a list of observable events or trends to track events, monitor targets, spot emerging trends, and warn of unanticipated change.

WHEN TO USE An analyst or team can create an indicators or signposts list of observable events that one would expect to see if a postulated situation is developing; e.g., economic reform, military modernization, political instability, or democratization. Constructing the list might require only a few hours or as much as several days to identify the critical variables associated with the targeted issue. The technique can be used whenever an analyst needs to track an event over time to monitor and evaluate changes. However, it can also be a very powerful aid in supporting other structured methods explained later in this primer. In those instances, analysts would be watching for mounting evidence to support a particular hypothesis, low- probability event, or scenario.

When there are sharply divided views on an issue, an indicators or signposts list can also “depersonalize” the argument by shifting analytic attention to a more objective set of criteria. Using an indicators list can clarify substantive disagreements, once all sides agree on the set of objective criteria used to measure the topic under study.

VALUE ADDED By providing an objective baseline for tracking events or targets, indicators instill rigor into the analytic process and enhance the credibility of analytic judgments. An indicators list included

in a finished product also allows the policymaker to track developments and builds a more concrete case for the analytic judgments. By laying out a list of critical variables, analysts also will be generating hypotheses regarding why they expect to see the presence of such factors. In so doing, analysts make the analytic line much more transparent and available for scrutiny by others.

THE METHOD Whether used alone, or in combination with other structured analysis, the process is the same:

• Identify a set of competing hypotheses or scenarios.

• Create separate lists of potential activities, statements, or events expected for each hypothesis or scenario.

• Regularly review and update the indicators lists to

Related Tags