5 Advanced Open Source Investigation Techniques

The proliferation of open source data – found across social media posts, self-published blogs, and other open source repositories – is radically transforming the world of investigations. However, generating meaningful insights from these volumes of publicly available data is the largest challenge facing effective investigations today. 

What is OSINT? Open source intelligence is the application of intelligence gathering techniques and technology to investigations that make use of open source data (OSD). When done well, this is the key to managing OSD and using it to drive tangible investigatory outcomes. Advanced technologies and investigative techniques help to differentiate intelligence from data, and draw connections across investigative landscapes. 

At Blackdot, our goal is to transform the accessibility and value of OSD for both government and private investigators. Our tool, Videris, delivers simple and powerful capabilities that improve outcomes without engaging in indiscriminate data capture. Here, we’re going to look at the advanced techniques that have driven our technology development and can be used to improve outcomes in any context. This article will help you to understand what those techniques are, and how the right open source intelligence tools help you to utilise them.

Technique # 1: Integrate internal and external databases

Internal data is information created inside a business. External data can come from numerous other sources, including news media, search engines, social media and corporate records data. To ensure that all available data is being utilised (and that no potential intelligence is being missed), investigations tools that integrate external and internal databases make it possible to recognise hidden connections, and highlight suspicious behaviours across disparate data sets.

As ever more people choose to share their lives through social media, Social media intelligence (SOCMINT) is particularly important. SOCMINT involves intelligence across two data categories, which are —

1. Original content: Facebook updates/uploaded images/video uploads.
2. Metadata associated with original content: Multimedia/geo-location/date/time.

When integrated with other types of publicly available information, or OSD, the insights derived from these information sources enable organisations to resolve complex problems, ranging from financial crime, to due diligence, to fraud and more. For example, the use of external intelligence such as SOCMINT can allow organisations to identify networks of actors wanting to harm, steal or defraud the organisation. 

Fundamentally, it’s vital to centralise your investigation processes. Multiple data sources and intelligence platforms create inefficiency and increase the risks of data loss or exposures that reduce effectiveness. Investigators need the ability to gather and analyse data across a comprehensive landscape in order to advance decision-making and actions.

Technique # 2: Harness intelligent automation to improve decision-making

It’s often not realistic to undertake open source investigations manually — there is simply too much data. However, fully-automated and AI-based decision-making cannot match the expertise and nuance that experienced investigators bring to the table. Moreover, there are legitimate ethical concerns when it comes to automating decisions as impactful as those that occur within the course of most OSINT investigations.

The approach we’ve embraced at Blackdot is Intelligent Automation (IA), which automates repetitive tasks in order to improve and accelerate human decision-making. This is done through the use of — 

• Intelligent automation of manual processes
• Automated red-flagging
• Cross-matching technology
• Visualisation capabilities
• Prioritised human oversight and reporting

Intelligent automation (IA) makes it possible to automate time-consuming processes while still leaving experienced investigators in charge; augmenting, rather than replacing, human decision-making. 

Technique # 3: Investigation security is paramount

OSINT, like most forms of intelligence, is typically used to identify and manage risks. Compromises to investigatory security prevent these objectives from being realised and risk creating more threat entry-points than they address. The poor handling of OSINT can also threaten your compliance with legal regulatory requirements. For example, hoarding the open source data you’ve collected through OSINT when not all of it will be relevant to your investigation contravenes GDPR regulations. What’s more, the data you have collected that is relevant needs to also be safely and securely stored so as to not breach GDPR. 

Another security threat to consider is revealing investigators’ identities or tipping off the subject of an investigation, which will be counter-productive to the development of valuable insights. For example, if a bank is investigating a customer’s source of funds, it is vital that the investigator doesn’t reveal that this investigation is taking place. If this does happen, they are at risk of compromising the investigation by causing the subject to further conceal their assets – or of reputational damage and loss of business.

Secure OSINT investigation ecosystems and effective threat identification should be part of all investigations. However, investigative integrity is still at significant risk if these processes aren’t deployed through an OSINT platform that prioritises security. This can be realised through the implementation of centralised data repositories, IP address security, direct exports, and flexible deployments either on-premises or in the cloud among other priorities.

Technique # 4: Align your investigations to the Intelligence Cycle

The Intelligence Cycle is a process designed to help investigators to focus on transforming data into actionable insights. The four stages of the Intelligence Cycle play a key part in open source investigation best practices by answering the fundamental questions that follow —

1. Direction: What is the problem and why is intelligence necessary?
2. Collection: What information is needed and where should it be sourced from?
3. Analysis: What does that information/intelligence mean?
4. Reporting and dissemination: How can that information be presented to the people who can put it into action?

When dealing with vast amounts of publicly available data, deriving insights can be challenging. You can quickly become lost in the volumes of data available, and irrelevant data collection is inevitable if context isn’t considered throughout the investigation. By accounting for the objectives, utilisation, and outcomes of an investigation before actually handling the data, investigators can assure the relevance of their outcomes and the final intelligence product. 

When paired with an OSINT tool that facilitates filtering and focus, the direction and control of the Intelligence Cycle makes it possible to ensure intelligent outcomes. 

Technique # 5: Always investigate ethically

Mass surveillance and indiscriminate data hoarding can place your organisation at risk of non-compliance and reputational damage. OSD might be publicly available information, but it still covers personal data that is subject to data privacy regulations in varying degrees. You need to consider legal, financial and PR components, along with their associated risks. 

Ethical OSD usage depends on an organisation’s ability, and desire, to target their data handling towards specified objectives that limit the flow of information to only valuable insights. What’s more, effective solutions will offer a simple means of saving sourcing information and collated data — this allows for easy referencing and cross-referencing of investigations. By simplifying and streamlining access to open source data, OSINT solutions provide an opportunity to gain insights that might otherwise only be available through more expansive and intrusive forms of intelligence collection. Instead, we can step toward the ethical collation and dissemination of publicly available data for a more targeted and sophisticated approach to investigations. As excessive automation with no human involvement can lead to wrong — and unethical — decisions, an IA approach that leaves humans in charge of OSINT decision-making minimises the risk of poor and potentially unethical decisions. The following factors also have an impact on the extent to which investigations can be considered ethical: 

• Targeted searches: collect only the information that is relevant to your investigation
• Legitimate use cases: collecting data to prevent crime, protect national security or meet regulatory requirements is considered more ethical than collection for marketing or sales purposes
• Only public data: using data that is freely available online is easily explainable; using privileged data is more likely to be viewed negatively

Deploying the power of a 3rd party platform

The growth of open source data has the potential to improve the effectiveness of investigations in both the public and private sphere, but it has also left companies at risk of drowning in data they can’t make sense of. Advanced investigation techniques, implemented with the help of the right 3rd party platform, are the best way through. At Blackdot, we developed Videris to address this problem by providing the intelligent automation necessary to professionalise OSINT practices, while always guaranteeing human-led decisions that keep investigations firmly between the lines of an ethical and targeted approach.

With extensive experience in government applications, as well as in the private sector, we’ve developed Videris to bring advanced open source investigation techniques to all users, regardless of their level of familiarity with OSINT. Videris does this through a range of capabilities, including —

• Targeted searches: The ability to highlight search terms or search keywords across multiple public data sets at the same time with Videris Search ensures that investigations, and the data used within them, are always targeted towards outcomes.
• Automatic analysis: Intelligent automation, robotic process automation and natural language processing make it far easier to sieve through expansive data sets, while bringing any valuable findings back to the analysts who can turn them into action.
• Visualised risk: Visualisations, including maps and charts, make it easier to draw connections across illicit networks and understand all risks, including those that may inform further investigations, or drive an investigation in a direction that a manual approach would never have recognised
• Combined findings: The one-source nature of Videris ensures that you can combine all your findings in one easily accessible, secure platform to prevent exposure or data breaches, as well as simplifying the overall investigation process.
• Secure searches: The ability to anonymise investigations provides security that ensures the integrity and effectiveness of your investigations.
• Data and source capture: By capturing full data sourcing as you work, Videris also ensures that you’re able to export sources directly into the reports that drive decisions based on your findings. This allows organisations to keep an immutable record of their findings, regardless of whether the evidence is taken down after the fact. 

These capabilities let investigators work at an advanced level to achieve better outcomes. Ready to utilise these advanced techniques that can take your investigations to the next level? Book a demo to find out exactly how we can help you.