What is OSINT? Open Source Intelligence and the Future of Data Analysis
By Blackdot Solutions
Get the latest news and insights sent straight to your inbox
The proliferation of publicly available data presents a significant opportunity to improve investigatory outcomes across a wide range of contexts. By gathering, cross-referencing and analysing this information, connections and insights can be obtained that would have previously been impossible. The potential for greater insight is facilitated by the sheer volume of open data sources. However, this scale is simultaneously the largest stumbling block to putting open source data to use: It’s simply too easy to get lost in a sea of noise.
OSINT (Open Source Intelligence) is the targeted collection, processing and analysis of public (open source) data to gain insights, drive decision-making, and mitigate risks. It uses technology and investigatory techniques to overcome challenges associated with the vast size of public data and home in on relevant information.
The origins of OSINT are apparent in its name. Like other military intelligence disciplines — for example, HUMINT (Human Intelligence) and SIGINT (Signals Intelligence) — OSINT is defined by the type of data being collected, and focuses on the capture of relevant information and its appropriate distribution to guide actions/investigations.
Historically, OSINT has been applied in countering terrorism, proliferation, espionage and serious/organised crime settings. However, we are focused on the future of OSINT. Tools, techniques and technology developed for military and government application are now moving into the private sector. Paired with an increased focus on intelligence-led data analysis (aligned with Big Data trends more generally), OSINT is disrupting investigatory best practices and creating far more robust outcomes in scenarios ranging across industries. These include, but are not limited to —
- Anti-money laundering and anti-financial crime
- Third-party due diligence
- Fraud and corruption investigations
- Brand protection and illicit trade
- Insider threat identification
The OSINT market is expected to reach nearly $12bn (£8.5bn) by 2026, registering a CAGR of 17.4%. We expect to see OSINT become an increasingly important part of compliance (particularly in financial services) as adoption rates increase across industries.
At Blackdot, we’ve pioneered open source intelligence software within a government context, and are helping bring OSINT into the mainstream with our product Videris. This article is about helping you understand OSINT, its implications for the future, and how you can use it to improve outcomes today.
What is open source data?
Open source data (OSD) refers to all publicly available or publicly licensable information. OSD comes from an incredibly broad range of external internet and public data sources. These include, but are not limited to —
- Government and global data sources (e.g. data.gov/census bureau/etc.)
- Corporate data sources in countries where this data is publicly available (e.g. UK Companies House)
- Crime statisitcs (e.g. uniform crime reporting program/bureau of justice statistics/etc.)
- Health and scientific data sources (e.g. World Health Organization/NHS digital/etc.)
- Social media profiles (e.g. Facebook, Linkedin, Twitter, Instagram etc.)
The OSINT framework is an interesting free resource that really demonstrates the sheer expanse of open source data sources. When taken together, open source data provides the potential for unparalleled analysis of individuals and groups, helping contextualise investigations and make connections that would have otherwise been missed. However, this is only possible if the right data points can be identified at the right time.
Filtering through these vast pools of data and identifying the most relevant information is perhaps the fundamental challenge of working with open source data. Organisations and investigators need to focus on tailoring open source data to their unique needs and applications.
Pro tip: Some open source intelligence tools combine these wide-reaching data sources with internal and even paid data, allowing for even more sophisticated cross-checking and analysis when used in conjunction with effectively calibrated intelligent automation.
Open source intelligence vs open source data
Open source intelligence and open source data are often used interchangeably, but they are fundamentally different concepts.
- Open source data (OSD): Generic data from a primary source that’s freely available to the public.
- Open source intelligence (OSINT): The end result once open source data has been collected, processed, analysed and then used to drive decision-making.
Open source data is wide-ranging and easily available, open source intelligence is not. And it’s open source intelligence that provides insights, and ultimately action, from those data sources.
The bottom line: OSINT prioritises the targeted and controlled application of data. In light of increasing volumes of all types of data, a targeted approach is the only way to focus on what’s important and identify what’s not. This not only improves the effectiveness of data capture, it allows it to be deployed in ethical and targeted ways that avoid mass surveillance in favour of specific intelligence gathering.
The future will be defined by intelligence (not data)
As data becomes increasingly more available, the importance of intelligence-led strategies will continue to grow. This is for two reasons —
- Intelligence enables investigators to draw insights through sensitive and informed data handling, without overexposure to mass data collection.
- Intelligence is the antidote to information overload, and enables organisations to use information to drive practical outcomes.
Solving these dual challenges has been a fundamental struggle within Big Data analytics for years. Adding open source data to private and proprietary sources only compounds the struggle. OSINT’s success relies on effectively sorting information, and significant strides have been made within the field of open source investigations towards resolving the challenge of data collection and use.
For investigators, the right OSINT tools will play a huge part in ensuring easy access to open source data, and providing targeted data collation that sits alongside in-house focuses. This can cut through the noise and provide insights that drive informed decision-making, which is the goal of using data in the first place.
How can OSINT be used in practice?
OSINT is already paving the way for the future of investigative practices in a wide range of industries. It enables organisations to map, track, and oversee risk factors, threat actors, and even consumer behaviours. Use cases provide an example of how benefit can be derived from incorporating OSINT more deeply into operations.
1. Financial services: Using OSINT to fight financial crime
OSINT has powerful implications for anti-money laundering (AML) and anti-financial crime (AFC) investigations. By cross-checking open data sources such as social media with corporate and financial records, connections can be made that might have otherwise slipped through the cracks, and financial crime investigations and due diligence can be taken to new levels of effectiveness.
There are two critical sides to the adoption of OSINT in financial services —
- Doing what matters: Financial institutions have a responsibility to prevent financial crime. By taking an intelligence-led approach to AFC, it’s possible to win public trust and demonstrate relevance in an era of increased digitalisation and disintermediation of financial services.
- Getting ahead of future compliance trends: Although OSINT requires the use of proprietary tools and techniques, OSD is available to everyone. As OSINT becomes standard within the industry, regulators are likely to look poorly on financial institutions that fail to use the information already at their disposal. Compliance requires demonstrating best practices, and OSINT will increasingly become part of that equation — driving future AML compliance trends.
With the high stakes of AFC investigations in financial institutions, access to a diverse range of data within an easy-to-access platform is especially crucial for ensuring comprehensive intelligence that prevents vital risks from slipping under the net.
2. Corporate: Using OSINT to protect your brand
As the financial and reputational cost of breaches in the corporate world grows, companies need to investigate every avenue of risk. The best OSINT tools are especially poised to bring a range of investigative benefits to the corporate world, including —
- Due diligence: Fast and efficient due diligence investigations on senior hires/ suppliers/contractors to mitigate against integrity risks such as corruption and bribery etc
- Brand protection: Identifying counterfeit networks and unauthorised sellers.
- Corporate security: Mapping and collecting information on threat actors to inform threat mitigation measures
- Insider threat identification: Tracing the social networks of threat actors to uncover connections between employees and malicious actors.
3. Government and law enforcement: Using OSINT to track criminals
OSINT has been a primary tool for government and law enforcement investigations for many years, but as open source data grows, OSINT practices will become even more integral to investigatory competence and success. OSINT can especially help government agencies in a range of key focus areas, including —
- Counter-terrorism: Understanding terrorist networks in real-time to identify targets and mitigate risks.
- Counter-proliferation: Identifying where industry might be deliberately or unwittingly supplying goods and services to parties with links to regimes involved in the development and distribution of WMD. .
- Serious and organised crime: Investigating and mapping networks of criminal actors.
OSINT can also provide invaluable benefits for risk consultancies working with clients in each of the above sectors.
What is needed to make OSINT effective?
While OSINT stands to simplify data collation, poor or unprofessional OSINT techniques can create as many problems as they solve. Fundamentally, organisations risk drowning in data and failing to make meaningful connections within the data they’ve captured. What’s more, data collection and analysis always create risks related to compliant data storage, use and the possibility of exposing an investigation.
Intelligent automation (IA) is the critical capability needed to manage the large datasets inherent to OSINT, and Big Data more generally. Rather than handing decision-making over to AI algorithms, IA augments human decision-making by focusing operators on core pieces of information and cross-referenced connections. This is important to keep OSINT investigations targeted, ethical and effective.
At Blackdot, we built Videris to transform the way that professional investigators can use OSINT across a wide range of use cases. No matter how you achieve these outcomes, there are a number of core capabilities that you should look for in any OSINT platform. Videris provides functionality that enables users to —
- Collect: Videris search enables the collection of single-click information from a wide range of sources such as search engines, corporate records, etc.
- Analyse: Intelligent automation makes it easy to spot hidden connections through robotic process automation and natural language processing, which cross-references data sources, ensuring context that drives human decisions.
- Visualise: Data and connections are displayed on charts, graphs, or maps for easier visualisations which are then exported straight into your report.
- Combine: One-source, single-point-of-access records save time and simplify workloads, and data integrations make it easier to spot hidden connections.
- Secure: The ability to anonymise investigations ensures your identity remains secure, and that no data leaves your servers without permission.
- Source: Full data sourcing as you work ensures transparency and decisions that can be easily explained.
Videris provides an unrivalled ability to capture information and map connections, keeping in-house teams in control of any decisions made. With speed, accuracy, security, and transparency, this one-source tool offers the benefits of OSINT at its best, while directly working to overcome the setbacks that have led organisations to deprioritise this intelligence until now.
The application of data requires intelligence
Companies need to demonstrate an understanding of modern risk landscapes, and data alone doesn’t do that. Ensuring that data can be converted into intelligence is critical. Identifying the right data, and manipulating it to meet organisational requirements is essential for bringing this benefit within easy reach. OSINT makes that possible.
OSINT is the solution to the tsunami of publicly available data. Videris was designed to tackle this challenge by providing:
- Ability to collect, analyse and visualise complex data in a single platform
- Intelligent automation that informs human decisions
- Anonymity and data security
- A single point of access
Together, these benefits make it possible to leverage open source data with unparalleled speed. The ability to integrate and search across data sources within a single platform strikes a balance between searching across large amounts of data while filtering out the noise and rapidly identifying connections in seconds. This not only ensures inclusive data collation, but also the informed, intelligence-driven decisions that modern business simply can’t do without.
Get in touch to book a demo and learn how to take your capabilities to the next level.