Open source intelligence, or OSINT, is the collection, processing and analysis of open source data (OSD) for investigative purposes. OSD itself is publicly available or licensable information that can be accessed from various sources. From threat intelligence to compliance, due diligence and background reporting, OSINT can assist in a wide range of investigative scenarios. However, gathering OSINT isn’t always straightforward.
For example, the sheer enormity of the internet presents a challenge for intelligence gathering — manually sifting through all of the available data is an impossible task. There is also increasing pressure on organisations to utilise open source data. In 2018, the European Banking Authority (EBA) stated that financial institutions should be carrying out “open source searches” as part of due diligence processes.
Fortunately, OSINT solutions that solve many of these challenges are available to investigators. Armed with these tools and a variety of techniques, investigators can cut through the noise and extract the full range of insights OSINT has to offer.
Today, we’re going to examine the most effective methods investigators can deploy to gather OSINT and drive successful investigation outcomes.
Suggested reading: You can read more about the benefits and challenges of using OSINT in our free eBook — The OSINT Handbook
The intelligence cycle should be considered a fundamental part of intelligence gathering, and OSINT investigations are no exception. The cycle provides a simple framework for analysts to conduct an investigation and gather intelligence. The typical intelligence cycle involves fives stages:
OSINT investigations usually begin with either a question or a problem. Conducting initial searches based on that starting point might be plain and clear — like searching for public filings for a background report on a prospective client. However, for more complex investigations, ethical considerations need to be made to ensure compliance and operational integrity.
The internet is not a singular, monolithic database accessible only with commercial search engines. In fact, 96% of the internet is not accessible using standard search engines, and is either unindexed or otherwise obscured from crawlers.
Effective OSINT gathering must strike below the surface web, drawing insights from disparate data sources. This should include:
That’s not to suggest that the surface is useless — far from it — but it represents just a tiny portion of what OSINT can offer. Diving beneath the surface allows investigators to obtain insights crucial to their investigations.
For example, the deep web contains all manner of corporate, governmental and NGO papers, as well as academic literature and company filing information. Much of this is OSD, and can therefore be used in OSINT gathering processes, but can only be obtained with database access or specialised search tools.
OSINT researchers can also extend their searches into databases which are specifically designed for due diligence, risk management and background checking. These can include:
OSINT is often an iterative process, and findings from one source might trigger new investigation pathways. Recording disparate information in a singular, centralised environment is essential, as this will streamline the research process when the investigation begins to evolve.
The surface web provides an excellent starting point for virtually any intelligence gathering or research process. However, effective OSINT gathering needs to go further and extend searches into all public data sources.
This includes the dark web, a rich source of OSD that provides insight, commentary and evidence on everything from smuggling and financial crime to terrorist networking and wildlife trafficking.
During the OSINT gathering process, investigators should expand their searches into the dark web in order to:
However, it’s important to remember that extending OSINT-gathering into the dark web requires specialist tools. It is of the utmost importance that investigators remain anonymous during the information-gathering process. Failure to do so risks alerting the subject to the existence of the investigation.
Furthermore, the dark web is home to an array of illegal and potentially distributing media. In light of this, it’s important to take precautions to protect investigators from this content to ensure ethical data collection throughout investigations.
The volume of OSD available to investigators across disparate sources is vast and ever-growing, meaning that there is now far too much data to undertake OSINT investigations manually. Investigators need to start deploying new and innovative tools to effectively scale up their investigations and drive optimised outcomes.
That’s one of the reasons why many are looking to artificial intelligence (AI) and machine learning (ML) for a solution. However, automating the process of OSINT gathering should not completely replace human decision-making. AI has benefits in many situations, but can’t always make the nuanced, ethical decisions a human investigator would.
An alternative solution is the use of intelligent automation (IA) in the OSINT gathering process. IA tools can streamline the intelligence cycle by automating menial, low-level tasks, saving time and resources while leaving high-value, operation-critical decisions to human experts.
Intelligent automation can provide investigators with a number of benefits during the process of gathering intelligence, including:
As the amount of OSD available to investigators has increased, we have seen the emergence of more sophisticated software to augment the OSINT gathering process.
The various OSINT tools now available come with a wide range of functionality. However, in order to ensure an effective and efficient OSINT-gathering process, investigators need tools that provide key benefits, including:
Moreover, using a singular OSINT ecosystem for the entire end-to-end investigation ensures operational accountability, integrity and security. Researchers need the ability to store all discovered data in a singular, secure location that is accessible anywhere at any time.
Modern intelligence analysts require cutting-edge tools to harness the full capabilities of OSINT. The data is out there in the form of publicly available information — it’s just a matter of finding it and converting it into usable, actionable insights.
In response to the rising demand for quality OSINT software, at Blackdot, we developed Videris, a platform that arms investigators with the tools they need to conduct robust, detailed and accurate investigations.
Videris simplifies and enhances the OSINT gathering processes with a range of essential functionality, including:
Videris can be deployed in the cloud or on a hosted or standalone network, depending on your needs. Book a demo today, and explore how Videris could optimise the outcomes of your investigations.