Instilling rigor and imagination in analysis Analysts Roadmap Katherine Hibbs Pherson Mary C. Boardman 23 May 2012 9/11 Attack on World Trade Center o Failure of imagination Iraq WMD o Poor analytic tradecraft Missing null hypothesis Inadequate vetting of sources The Call for Analytic Rigor The Community has developed a series of guidelines to help analysts produce solid products and avoid the shortfalls of the past. Copyright 2012 Pherson Associates, LLC. All Rights Reserved.2 Defines core elements for conducting evaluations of analytic production Mandates evaluation programs using the standards Allows for additional standards Foresees using them to assess performance trends and establish new training and development initiatives Copyright 2012 Pherson Associates, LLC. All Rights Reserved.3 Intelligence Community Directive 203: Standards for Intelligence Analysis 1.Objectivity 2.Independent of Political Considerations 3.Timeliness 4.Based on All Available Sources of Intelligence 5.Exhibits Proper Standards of Analytic Tradecraft, Specifically: Properly describes quality and reliability of underlying sources Properly caveats and expresses uncertainties or confidence in analytic judgments Properly distinguishes between underlying intelligence and analysts assumptions and judgments Incorporates alternative analysis where appropriate Demonstrates relevance to U.S. national security Uses logical argumentation Exhibits consistency of analysis over time, or highlights changes and explains rationale Makes accurate judgments and assessments Copyright 2012 Pherson Associates, LLC. All Rights Reserved.4 Consistent and structured sourcing Source reference statements (endnotes) that protect sensitive sources methods Source descriptors Source summary statements Copyright 2012 Pherson Associates, LLC. All Rights Reserved.5 Know your customers and what they need Write for tailored reuse Products must be discoverable by those who need them Tradecraft essential, not expendable Timely electronic dissemination is always the goal Train to think of customers inclusively Copyright 2012 Pherson Associates, LLC. All Rights Reserved.6 7 Similar, but Slightly Different 8 MessageAnalyticConvincingStructuredWell-Written Main Point Clear Judgments, Not Just Facts Evidence for Judgments Sections Advance Story Precise Language Beyond the Obvious Provide ContextNo Contradictions One Point per Section Concise So What? Anticipated Questions Explain Reliability Consistent Tics No Typos Note Changed Assessment Alternate Views Free of BiasNo Redundancy No Awkward Constructions Opportunities for the US Forward Looking Identify Gaps Graphics Aid Text Provide Warning Confidence Levels Key Essential Desired As Appropriate 9 Copyright 2012 Pherson Associates, LLC. All Rights Reserved. 10 Getting to the So What? or News that Can Be Used What is new or different about an issue or event? Is it an indicator of significant change? Do you need to warn of an imminent or potential threat? Does the issue or event relate to your primary consumers interests or schedule? What can you add that will support your primary consumers as they respond to crises, make difficult decisions, attend meetings, or think about how to meet current or future challenges? Does the event or issue present opportunities or pitfalls for your organizations programs, actions, or policies? Is the issue or event important your organization or U.S. national security? Copyright 2012 Pherson Associates, LLC. All Rights Reserved.11 Copyright 2012 Pherson Associates, LLC. All Rights Reserved.12 Analysts Are Not Only Awash in Data, but in Guidance and Requirements Copyright 2012 Pherson Associates, LLC. All Rights Reserved.13 Analytic Best Practices Canada, Office of the Privy Council 1.Reflect on the problem, determining possible approaches 2.Be resourceful and systematic in collecting information, documenting sources, and noting caveats 3.Critically evaluate the quality of all information 4.Develop multiple hypotheses and explanations, show tolerance for uncertainties 5.Challenge mindsets, assumptions, and biases 6.Build collaborative networks 7.Use Structured Analytic Techniques 8.Write clear, concise, well-documented, and client- focused reports Copyright 2012 Pherson Associates, LLC. All Rights Reserved.14 The Analysts Role in the Operating Environment Analyst Informs Decisionmaker Impacts Issue Interprets Source: Pherson, Katherine Hibbs and Randolph Pherson, Critical Thinking for Strategic Intelligence, Washington, DC: CQ Press, Jerry Ratcliffe has produced a similar model that better explains the specific components of the law enforcement model for Intelligence-Led Policing. Copyright 2012 Pherson Associates, LLC. All Rights Reserved.15 Analysts Roadmap Key consolidated elements o Get your act together o Critical thinking o Analytic standards o Structured analytic techniques Based on questions asked by thousands of analysts in training classes. Based on what you do, but how you do it. Beta tested in government, private industry, and overseas. Copyright 2012 Pherson Associates, LLC. All Rights Reserved.16 AIMSS: The Key to Conceptualizing Intelligence Products Audience Intelligence Question Message Storyline Standards Copyright 2012 Pherson Associates, LLC. All Rights Reserved.17 18 Copyright 2012 Pherson Associates, LLC. All Rights Reserved. 19 Copyright 2012 Pherson Associates, LLC. All Rights Reserved. Four Steps for Communicating with Analysts 1.Preparation Follow the Analysts Roadmap 2.Analytic Guidance Use the IC Standards 3.Evaluation against the Standards Adapt an Evaluation Form 4.Mentoring and Feedback Dont skimp on tradecraft support for analysts 20 Copyright 2012 Pherson Associates, LLC. All Rights Reserved. Can DHS make a unique contribution? Does the analysis highlight new information and further our understanding? Can we deliver the analysis in time to make a difference? Should an explicit warning message be delivered? To whom should we reach out at each stage of the process? With whom do we need to coordinate? Would the analysis benefit from structured analytic techniques? Are key drivers and trend lines highlighted? Did we identify contrary evidence, and information gaps? Are analytic judgments and assumptions distinguished from the underlying intelligence? Did we consider past customer feedback, explore opportunities, and assess both benefits and risks? Is the writing clear, precise, and well- organized (is the bottom line up front)? Is the credibility of the source(s) clearly articulated? Is denial or deception a possibility? Are levels of confidence in the judgments and uncertainties clearly expressed? Is the analysis free of bias, advocacy, and value laden terms? Have we double- checked the facts? Are the graphics consistent with the analysis? Are key terms well- defined? Is a glossary needed? Have grammatical errors, typographical errors, and misspellings been corrected? Are classification markings correct and handling caveats prominent? Do endnotes follow DHS guidelines? Will the product go to all our key customers? Who are the primary customer(s)? Will we publish at different classification levels? What are the key intelligence questions? What is the broader context for this topic or issue? Have we identified and challenged my key assumptions? Are we changing our overall analytic line? If so, did we explain why? Can graphics be used to advance and define the message? Is the topic clearly relevant to the DHS mission? Is the topic clearly relevant to the DHS mission? Is the main point prominent and clearly stated? Did we identify alternative hypotheses and provide context? Is there sufficient reasoning and compelling evidence to support the judgments? Does each section, paragraph, and sentence advance the storyline? Do my titles and headings effectively capture the message? Are they in synch with the text? Analytic Managers Roadmap Best Practices for Applying Analytic Tradecraft Standards Task 1: Stop and Reflect Task 1: Stop and Reflect Task 2: Focus the Message Task 2: Focus the Message Task 3: Develop the Storyline Task 3: Develop the Storyline Task 4: Prepare the Draft Task 4: Prepare the Draft Task 5: Perfect the Presentation Task 5: Perfect the Presentation Critical Requirements Copyright 2012 Pherson Associates, LLC. All Rights Reserved.21