5 Open Source Intelligence Gathering Techniques

Open source intelligence, or OSINT, is the collection, processing and analysis of open source data (OSD) for investigative purposes. OSD itself is publicly available or licensable information that can be accessed from various sources. From threat intelligence to compliance, due diligence and background reporting, OSINT can assist in a wide range of investigative scenarios. However, gathering OSINT isn’t always straightforward.

For example, the sheer enormity of the internet presents a challenge for intelligence gathering — manually sifting through all of the available data is an impossible task. There is also increasing pressure on organisations to utilise open source data. In 2018, the European Banking Authority (EBA) stated that financial institutions should be carrying out “open source searches” as part of due diligence processes.

Fortunately, OSINT solutions that solve many of these challenges are available to investigators. Armed with these tools and a variety of techniques, investigators can cut through the noise and extract the full range of insights OSINT has to offer.

Today, we’re going to examine the most effective methods investigators can deploy to gather OSINT and drive successful investigation outcomes.

Suggested reading: You can read more about the benefits and challenges of using OSINT in our free eBook — The OSINT Handbook

#1 Stay aligned with the Intelligence Cycle

The intelligence cycle should be considered a fundamental part of intelligence gathering, and OSINT investigations are no exception. The cycle provides a simple framework for analysts to conduct an investigation and gather intelligence. The typical intelligence cycle involves fives stages:

1. Direction and planning: Identify the fundamental questions that need answering. Define who or what is under investigation and why, and the OSINT sources you intend to use.
2. Collection: The process of extracting open source data in its raw form from the sources you have previously identified.
3. Processing: After data has been gathered, it needs to be translated into a comprehensible format, which might mean decryption or sorting based on topic, relevance and reliability.
4. Analysis: Extract insights to better understand the meaning behind the data gathered. 
5. Reporting and dissemination: Collate and visualise findings to make decisions more understandable to stakeholders who may not have the time to go through a detailed report.

OSINT investigations usually begin with either a question or a problem. Conducting initial searches based on that starting point might be plain and clear — like searching for public filings for a background report on a prospective client. However, for more complex investigations, ethical considerations need to be made to ensure compliance and operational integrity. 

#2 Utilise disparate data sources

The internet is not a singular, monolithic database accessible only with commercial search engines. In fact, 96% of the internet is not accessible using standard search engines, and is either unindexed or otherwise obscured from crawlers. 

Effective OSINT gathering must strike below the surface web, drawing insights from disparate data sources. This should include: 

• Corporate records 
• Blogs
• Forums 
• Grey literature 
• Publicly accessible databases. 

That’s not to suggest that the surface is useless — far from it — but it represents just a tiny portion of what OSINT can offer. Diving beneath the surface allows investigators to obtain insights crucial to their investigations.

For example, the deep web contains all manner of corporate, governmental and NGO papers, as well as academic literature and company filing information. Much of this is OSD, and can therefore be used in OSINT gathering processes, but can only be obtained with database access or specialised search tools. 

OSINT researchers can also extend their searches into databases which are specifically designed for due diligence, risk management and background checking. These can include:

• Global corporate records aggregators such as Bureau Van Dijk or OpenCorporates
• Adverse media aggregators such as Dow Jones Factiva
• Compliance databases such as Refinitiv World-Check 
• Sanctions databases such as OpenSanctions

OSINT is often an iterative process, and findings from one source might trigger new investigation pathways. Recording disparate information in a singular, centralised environment is essential, as this will streamline the research process when the investigation begins to evolve.

#3 Expand searches to capture all available evidence

The surface web provides an excellent starting point for virtually any intelligence gathering or research process. However, effective OSINT gathering needs to go further and extend searches into all public data sources.

This includes the dark web, a rich source of OSD that provides insight, commentary and evidence on everything from smuggling and financial crime to terrorist networking and wildlife trafficking. 

During the OSINT gathering process, investigators should expand their searches into the dark web in order to: 

• Better understand criminal networks, their communications, ideas, trends and practices. 
• Locate information on dark web mirrors of surface websites, such as Facebook, which are used for political dissent and censorship-proof communications.
• Correlate, compare and validate information found on the dark web or deep web, e.g., matching usernames and profile pictures.

However, it’s important to remember that extending OSINT-gathering into the dark web requires specialist tools. It is of the utmost importance that investigators remain anonymous during the information-gathering process. Failure to do so risks alerting the subject to the existence of the investigation.

Furthermore, the dark web is home to an array of illegal and potentially distributing media. In light of this, it’s important to take precautions to protect investigators from this content to ensure ethical data collection throughout investigations. 

#4 Deploy intelligent automation

The volume of OSD available to investigators across disparate sources is vast and ever-growing, meaning that there is now far too much data to undertake OSINT investigations manually. Investigators need to start deploying new and innovative tools to effectively scale up their investigations and drive optimised outcomes.

That’s one of the reasons why many are looking to artificial intelligence (AI) and machine learning (ML) for a solution. However, automating the process of OSINT gathering should not completely replace human decision-making. AI has benefits in many situations, but can’t always make the nuanced, ethical decisions a human investigator would.

An alternative solution is the use of intelligent automation (IA) in the OSINT gathering process. IA tools can streamline the intelligence cycle by automating menial, low-level tasks, saving time and resources while leaving high-value, operation-critical decisions to human experts.

Intelligent automation can provide investigators with a number of benefits during the process of gathering intelligence, including:

• The automatic collation of information from all available data points and sources into one seamless workflow, saving research time without sacrificing choice and direction. 
• The revealing of named entities, relationship links and other critical features for deriving insight, saving analysts time without replacing essential human judgment. 
• The creation of maps, networks, charts and tables that organise information effectively, helping analysts make connections and obtain insights in a timely manner.

#5 Choose the right software

As the amount of OSD available to investigators has increased, we have seen the emergence of more sophisticated software to augment the OSINT gathering process.

The various OSINT tools now available come with a wide range of functionality. However, in order to ensure an effective and efficient OSINT-gathering process, investigators need tools that provide key benefits, including: 

• Targeted searches across multiple data sets for terms/keywords that allow for incisive yet inclusive OSINT gathering across all available sources.
• Secure searches, which keep investigators anonymous during OSINT gathering, especially important when utilising the dark web.
• Rapidly linking names, addresses, locations and other named entities, even when working with a wide array of text-rich sources. 
• Combining all gathered OSINT into one easily accessible platform, regardless of its source and format.
• Intelligent automation of repetitive and time-consuming tasks, including drawing up networks, maps and graphs.

Moreover, using a singular OSINT ecosystem for the entire end-to-end investigation ensures operational accountability, integrity and security. Researchers need the ability to store all discovered data in a singular, secure location that is accessible anywhere at any time.

Leverage powerful tools for effective OSINT gathering

Modern intelligence analysts require cutting-edge tools to harness the full capabilities of OSINT. The data is out there in the form of publicly available information — it’s just a matter of finding it and converting it into usable, actionable insights.

In response to the rising demand for quality OSINT software, at Blackdot, we developed Videris, a platform that arms investigators with the tools they need to conduct robust, detailed and accurate investigations.

Videris simplifies and enhances the OSINT gathering processes with a range of essential functionality, including:

• Videris Search: Automatically sort and categorise results in order to find key information in a sea of noise.
• Charts and Corporate Network Mapping: Rapidly convert data into powerful visuals for review and analysis.
• The exposure of links across sources: Using entity recognition and cross-matching, Videris can help ensure you never miss a link between sources.
• Videris Notes: Investigators can securely record their findings with sourcing as they go, also aiding in the efficiency and preparation of final reports.
• A single ecosystem: Videris ensures security and anonymity and provides certainty that the investigation is untraceable.

Videris can be deployed in the cloud or on a hosted or standalone network, depending on your needs. Book a demo today, and explore how Videris could optimise the outcomes of your investigations.

---

¹EBA Final Guidelines 2018

²CS Online – What is the Dark Web?