Data for Decision-Making with Project IDEA

Data for Decision Making – ISR Analysis and IDEA

Starting in 2015, Redhorse NatSec partnered with IBM to conduct data-driven studies and analysis to improve the effectiveness of ISR systems deployed around the world. Our study teams always began by assembling a comprehensive, coherent, factual dataset of all information relating to the problem. It accessed databases, tasking orders, and intelligence collections in the Pentagon and with Combatant Commands (CCMD). They visited intelligence and operations centers overseas to observe operations and workflows, interview deployed Warfighters, and find data held locally in mission and logistics databases. The teams created custom algorithms and data pipelines for each study – to ingest all of this data from various isolated silos, transform it into a common format, and make it available for analysis as they worked to solve DoD challenges.

As an example of an effective study, in 2015, when a pilot shortage led to gaps in unmanned aerial system (UAS) coverage of war zones, DoD asked our ISR analysis team to find a way to ensure warfighters continued to receive critical intelligence with fewer platforms flying. Conducting this study was difficult, because different UAS platforms, and even different systems on a single UAS, each fed information into dozens of different databases with different data structures and formats that were not interoperable. To solve this data challenge, our analysis team created a technical tool to extract and stitch together structured and unstructured data from isolated sources in different formats to create a comprehensive, enterprise-wide look at UAV performance.

We accomplished this because we understood what intelligence the warfighters were relying on, were comfortable with the various ISR data systems, and had the technical expertise to handle data however we found it in each system. We designed customized VBA and Python code to pull data from DIA’s ROME, US Air Force’s UNICORN, Air Combat Command’s Skynet, and US Central Command’s (CENTCOM) MISREP and ISR Analysis Tools. We then developed recommendations that optimized tasking by what sensors and capabilities each platform had, rather than by who owned it, and we considered factors like weather patterns and disruptions. The deputy secretary of defense (DEPSECDEF) ordered our plan into action, resulting in a 30% recovery of lost flight hours, largely offsetting the pilot shortage.

As NatSec was helping to execute the DEPSECDEF order for global, dynamic ISR allocation, we were setting up a separate demonstration to help DoD leaders with global ISR situational awareness. We used the data tools created for several studies to present a consolidated picture of how and where DoD’s ISR data was held. DoD leaders were startled by how disjointed and incomplete their understanding of the ISR picture was – and how flawed their deployment and employment decisions had become.

For decades, intelligence officers had planned and conducted ISR operations and processing, exploitation, and dissemination (PED) of intelligence from ISR data in a slow and inefficient manual process. Practitioners had little insight into how their work related to that done by others, and decision makers who depended on ISR data and analysis had varying levels of trust in the intelligence they were receiving. After our late-2015 briefing, the Office of the Under Secretary of Defense for Intelligence and Security (OUSD(I&S)), the senior intelligence office for all DoD services and agencies, gave the NatSec team clearance to fix this longstanding problem with our ISR Data Enrichment and Aggregation (IDEA) pilot for the U.S. Indo-Pacific Command (INDOPACOM). 

In 2016, we employed tools from Python, PostgreSQL, machine learning (ML), natural language processing (NLP), and MS Office to:

• Ingest data from dozens of siloed sources into IDEA’s data pipeline, working with data in its original form for processing on DoD’s classified systems,
• Transform data to improve its quality and usability, structuring unstructured data, applying an IDEA-preferred schema, correcting typos, inferring missing data, enriching data with information from outside sources, and linking new data to enable more robust, contextual insight to users,
• Match data for the same “event” (for instance collection on a specific intelligence target by several platforms at the same time) and merge data to create a master record of that event that represents the superset of data elements, and
• Present the “best” (most accurate and useful) data in an intuitive way to anyone who has the proper access and could make use of the data. 

In 2017, after our successful demonstration of the IDEA pilot for all ISR flights in the INDOPACOM area of responsibility, OUSD(I&S) contracted with NatSec to build IDEA into a global data service. NatSec ISR experts and data engineers have since employed advanced technologies in machine learning, data visualization, geospatial analysis, web services, web development, database development, and network architecting to build the IDEA data service within DoD’s classified cloud architecture, conducting live data science work on DoD’s real data, where it already resides.

To ensure IDEA did not recreate the data silos we had purposefully broken down, we developed leveled application programming interfaces (API) technologies and protocols that allow senior leaders and novice users to interface with user-friendly web layouts, intelligence analysts and those comfortable with MS Excel to conduct bulk data exports, and expert users, developers, and machine-to-machine programs to connect directly to build or enrich their own applications with IDEA’s standardized, clean, and AI-ready data. Coded with Python, MongoDB, Redis, Elastic (ELK), NodeJS and React, IDEA is stored in a Mongo non-relational database and resides on Titan, allowing us to push updates without downtime, which is unique for the Top Secret JWICS domain.

(As of April 2025), IDEA holds the ground truth record of every ISR flight record, collect record, intercept, and “event” that occurs globally. Its more-than 10,000 active users include OUSD(I&S), Joint Staff intelligence and operations elements (J2, J3), DoD’s Office of Cost Assessment and Program Evaluation (CAPE), Unified Combatant Commands, DoD intelligence agencies including NSA, NGA, DIA, and NRO, and operational ISR users around the globe. Over seven years, with the work of just 24 full-stack data engineers, IDEA has become the capability DoD uses to manage its global ISR enterprise, is expanding to present data on other networks and at lower classification levels and will soon supplant ROME as DoD’s ISR program of record.

In 2025, IDEA began integrating AI and generative AI (GenAI) / large language Model (LLM) technology to perform analytics to help ISR collection managers understand recent operations to inform current and future operational decisions. Our LLM capability writes post-mission analytic reports, aggregating data such as what frequencies were collected (within the past day, week, or month), any new intelligence requirements that were introduced, the impacts of weather on operations and collection, any adversarial intercepts of ISR flights (for instance adversarial fighters intercepting UAS), and measures of performance such as flights flown and flights canceled. Anyone with access can generate these GenAI reports with custom prompts for data fields, including selecting a specific operational area. As of April 2025, this technology has advanced enough to allow no code / low code prompting of new requirements.

IDEA is an exemplar for DoD data science – Lean, Agile, user-centered, with continuous delivery and continuous authority to operate (ATO). It is government owned, built on open-source tech, modular and open, reusable, and flexible. We started with an idea for how DoD could better understand how its ISR platforms operated. We built IDEA into the authoritative master-data solution for ISR – a single source of truth and facts that is always current and always available. When asked by Congress to show that it is a data-driven organization, OUSD(I&S) consistently points to IDEA as its proof.