The State of AI in Cybersecurity: Understanding AI Technologies
24
Jul 2024
Part 4: This blog explores the findings from Darktrace’s State of AI Cybersecurity Report on security professionals' understanding of the different types of AI used in security programs. Get the latest insights into the evolving challenges, growing demand for skilled professionals, and the need for integrated security solutions by downloading the full report.
About the State of AI Cybersecurity Report
Darktrace surveyed 1,800 CISOs, security leaders, administrators, and practitioners from industries around the globe. Our research was conducted to understand how the adoption of new AI-powered offensive and defensive cybersecurity technologies are being managed by organizations.
How familiar are security professionals with supervised machine learning
Just 31% of security professionals report that they are “very familiar” with supervised machine learning.
Many participants admitted unfamiliarity with various AI types. Less than one-third felt "very familiar" with the technologies surveyed: only 31% with supervised machine learning and 28% with natural language processing (NLP).
Most participants were "somewhat" familiar, ranging from 46% for supervised machine learning to 36% for generative adversarial networks (GANs). Executives and those in larger organizations reported the highest familiarity.
Combining "very" and "somewhat" familiar responses, 77% had familiarity with supervised machine learning, 74% generative AI, and 73% NLP. With generative AI getting so much media attention, and NLP being the broader area of AI that encompasses generative AI, these results may indicate that stakeholders are understanding the topic on the basis of buzz, not hands-on work with the technologies.
If defenders hope to get ahead of attackers, they will need to go beyond supervised learning algorithms trained on known attack patterns and generative AI. Instead, they’ll need to adopt a comprehensive toolkit comprised of multiple, varied AI approaches—including unsupervised algorithms that continuously learn from an organization’s specific data rather than relying on big data generalizations.
Different types of AI
Different types of AI have different strengths and use cases in cyber security. It’s important to choose the right technique for what you’re trying to achieve.
Supervised machine learning: Applied more often than any other type of AI in cyber security. Trained on human attack patterns and historical threat intelligence.
Large language models (LLMs): Applies deep learning models trained on extremely large data sets to understand, summarize, and generate new content. Used in generative AI tools.
Natural language processing (NLP): Applies computational techniques to process and understand human language.
Unsupervised machine learning: Continuously learns from raw, unstructured data to identify deviations that represent true anomalies.
What impact will generative AI have on the cybersecurity field?
More than half of security professionals (57%) believe that generative AI will have a bigger impact on their field over the next few years than other types of AI.
Figure 1: Chart from Darktrace's State of AI in Cybersecurity Report
Security stakeholders are highly aware of generative AI and LLMs, viewing them as pivotal to the field's future. Generative AI excels at abstracting information, automating tasks, and facilitating human-computer interaction. However, LLMs can "hallucinate" due to training data errors and are vulnerable to prompt injection attacks. Despite improvements in securing LLMs, the best cyber defenses use a mix of AI types for enhanced accuracy and capability.
AI education is crucial as industry expectations for generative AI grow. Leaders and practitioners need to understand where and how to use AI while managing risks. As they learn more, there will be a shift from generative AI to broader AI applications.
Do security professionals fully understand the different types of AI in security products?
Only 26% of security professionals report a full understanding of the different types of AI in use within security products.
Confusion is prevalent in today’s marketplace. Our survey found that only 26% of respondents fully understand the AI types in their security stack, while 31% are unsure or confused by vendor claims. Nearly 65% believe generative AI is mainly used in cybersecurity, though it’s only useful for identifying phishing emails. This highlights a gap between user expectations and vendor delivery, with too much focus on generative AI.
Key findings include:
Executives and managers report higher understanding than practitioners.
Larger organizations have better understanding due to greater specialization.
As AI evolves, vendors are rapidly introducing new solutions faster than practitioners can learn to use them. There's a strong need for greater vendor transparency and more education for users to maximize the technology's value.
To help ease confusion around AI technologies in cybersecurity, Darktrace has released the CISO’s Guide to Cyber AI. A comprehensive white paper that categorizes the different applications of AI in cybersecurity. Download the White Paper here.
Do security professionals believe generative AI alone is enough to stop zero-day threats?
No! 86% of survey participants believe generative AI alone is NOT enough to stop zero-day threats
This consensus spans all geographies, organization sizes, and roles, though executives are slightly less likely to agree. Asia-Pacific participants agree more, while U.S. participants agree less.
Despite expecting generative AI to have the most impact, respondents recognize its limited security use cases and its need to work alongside other AI types. This highlights the necessity for vendor transparency and varied AI approaches for effective security across threat prevention, detection, and response.
Stakeholders must understand how AI solutions work to ensure they offer advanced, rather than outdated, threat detection methods. The survey shows awareness that old methods are insufficient.
Oops! Something went wrong while submitting the form.
Newsletter
Stay ahead of threats with the Darktrace blog newsletter
Get the latest insights from the cybersecurity landscape, including threat trends, incident analysis, and the latest Darktrace product developments – delivered directly to your inbox, monthly.
Thanks, your request has been received
A member of our team will be in touch with you shortly.
Oops! Something went wrong while submitting the form.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
A Snake in the Net: Defending Against AiTM Phishing Threats and Mamba 2FA
What are Adversary-in-the-Middle (AiTM) phishing kits?
Phishing-as-a-Service (PhaaS) platforms have significantly lowered the barriers to entry for cybercriminals, enabling a new wave of sophisticated phishing attacks. Among the most concerning developments in this landscape is the emergence of Adversary-in-the-Middle (AiTM) phishing kits, which enhance traditional phishing tactics by allowing attackers to intercept and manipulate communications in real-time. The PhaaS marketplace offers a wide variety of innovative capabilities, with basic services starting around USD 120 and more advanced services costing around USD 250 monthly [1].
These AiTM kits are designed to create convincing decoy pages that mimic legitimate login interfaces, often pre-filling user information to increase credibility. By acting as a man-in-the-middle, attackers can harvest sensitive data such as usernames, passwords, and even multi-factor authentication (MFA) tokens without raising immediate suspicion. This capability not only makes AiTM attacks more effective but also poses a significant challenge for cybersecurity defenses [2].
Mamba 2FA is one such example of a PhaaS strain with AiTM capabilities that has emerged as a significant threat to users of Microsoft 365 and other enterprise systems. Discovered in May 2024, Mamba 2FA employs advanced AiTM tactics to bypass MFA, making it particularly dangerous for organizations relying on these security measures.
What is Mamba 2FA?
Phishing Mechanism
Mamba 2FA employs highly convincing phishing pages that closely mimic legitimate Microsoft services like OneDrive and SharePoint. These phishing URLs are crafted with a specific structure, incorporating Base64-encoded parameters. This technique allows attackers to tailor the phishing experience to the targeted organization, making the deception more effective. If an invalid parameter is detected, users are redirected to a benign error page, which helps evade automated detection systems [5].
Figure 1: Phishing page mimicking the Microsoft OneDrive service.
Real-Time Communication
A standout feature of Mamba 2FA is its use of the Socket.IO JavaScript library. This library facilitates real-time communication between the phishing page and the attackers' backend servers. As users input sensitive information, such as usernames, passwords, and MFA tokens on the phishing site, this data is immediately relayed to the attackers, enabling swift unauthorized access [5].
Multi-Factor Authentication Bypass
Mamba 2FA specifically targets MFA methods that are not resistant to phishing, such as one-time passwords (OTPs) and push notifications. When a user enters their MFA token, it is captured in real-time by the attackers, who can then use it to access the victim's account immediately. This capability significantly undermines traditional security measures that rely on MFA for account protection.
Infrastructure and Distribution
The platform's infrastructure consists of two main components: link domains and relay servers. Link domains handle initial phishing attempts, while relay servers are responsible for stealing credentials and completing login processes on behalf of the attacker. The relay servers are designed to mask their IP addresses by using proxy services, making it more difficult for security systems to block them [3].
Evasion Techniques
To evade detection by security tools, Mamba 2FA employs several strategies:
Sandbox Detection: The platform can detect if it is being analyzed in a sandbox environment and will redirect users to harmless pages like Google’s 404 error page.
Dynamic URL Generation: The URLs used in phishing attempts are frequently rotated and often short-lived to avoid being blacklisted by security solutions.
HTML Attachments: Phishing emails often include HTML attachments that appear benign but contain hidden JavaScript that redirects users to the phishing page [5].
Darktrace’s Coverage of Mamba 2FA
Starting in July 2024, the Darktrace Threat Research team detected a sudden rise in Microsoft 365 customer accounts logging in from unusual external sources. These accounts were accessed from an anomalous endpoint, 2607:5500:3000:fea[::]2, and exhibited unusual behaviors upon logging into Software-as-a-Service (SaaS) accounts. This activity strongly correlates with a phishing campaign using Mamba 2FA, first documented in late June 2024 and tracked as Mamba 2FA by Sekoia [2][3].
Darktrace / IDENTITY was able to identify the initial stages of the Mamba 2FA campaign by correlating subtle anomalies, such as unusual SaaS login locations. Using AI based on peer group analysis, it detected unusual behavior associated with these attacks. By leveraging Autonomous Response actions, Darktrace was able to neutralize these threats in every instance of the campaign detected.
On July 23, a SaaS user was observed logging in from a rare ASN and IP address, 2607:5500:3000:fea::2, originating from the US and successfully passed through MFA authentication.
Figure 2: Model Alert Event Log showing Darktrace’s detection of a SaaS user mailbox logging in from an unusual source it correlates with Mamba 2FA relay server.
Almost an hour later, the SaaS user was observed logging in from another suspicious IP address, 45.133.172[.]86, linked to ASN AS174 COGENT-174. This IP, originating from the UK, successfully passed through MFA validation.
Following this unusual access, the SaaS user was notably observed reading emails and files that could contain sensitive payment and contract information. This behavior suggests that the attacker may have been leveraging contextual information about the target to craft further malicious phishing emails or fraudulent invoices. Subsequently, the user was detected creating a new mailbox rule titled 'fdsdf'. This rule was configured to redirect emails from a specific domain to the 'Deleted Items' folder and automatically mark them as read.
Implications of Unusual Email Rules
Such unusual email rule configurations are a common tactic employed by attackers. They often use these rules to automatically forward emails containing sensitive keywords—such as "invoice”, "payment", or "confidential"—to an external address. Additionally, these rules help conceal malicious activities, keeping them hidden from the target and allowing the attacker to operate undetected.
Figure 3: The model alert “SaaS / Compliance / Anomalous New Email Rule,” pertaining to the unusual email rule created by the SaaS user named ‘fdsdf’.
Blocking the action
A few minutes later, the SaaS user from the unusual IP address 45.133.172[.]86 was observed attempting to send an email with the subject “RE: Payments.” Subsequently, Darktrace detected the user engaging in activities that could potentially establish persistence in the compromised account, such as registering a new authenticator app. Recognizing this sequence of anomalous behaviors, Darktrace implemented an Autonomous Response inhibitor, disabling the SaaS user for two hours. This action effectively contained potential malicious activities, such as the distribution of phishing emails and fraudulent invoices, and gave the customer’s security team the necessary time to conduct a thorough investigation and implement appropriate security measures.
Figure 4: Device Event Log displaying Darktrace’s Autonomous Response taking action by blocking the SaaS account.
Figure 5: Darktrace / IDENTITY highlighting the 16 model alerts that triggered during the observed compromise.
In another example from mid-July, similar activities related to the campaign were observed on another customer network. A SaaS user was initially detected logging in from the unusual external endpoint 2607:5500:3000:fea[::]2.
Figure 6: The SaaS / Compromise / SaaS Anomaly Following Anomalous Login model alert was triggered by an unusual login from a suspicious IP address linked to Mamba 2FA.
A few minutes later, in the same manner as demonstrated in the previous case, the actor was observed logging in from another rare endpoint, 102.68.111[.]240. However, this time it was from a source IP located in Lagos, Nigeria, which no other user on the network had been observed connecting from. Once logged in, the SaaS user updated the settings to "User registered Authenticator App with Notification and Code," a possible attempt to maintain persistence in the SaaS account.
Figure 7: Darktrace / IDENTITY highlighted the regular locations for the SaaS user. The rarity scores associated with the Mamba 2FA IP location and another IP located in Nigeria were classified as having very low regularity scores for this user.
Based on unusual patterns of user behavior, a Cyber AI Analyst Incident was also generated, detailing all potential account hijacking activities. Darktrace also applied an Autonomous Response action, disabling the user for over five hours. This swift action was crucial in preventing further unauthorized access, potential data breaches and further implications.
Figure 8: Cyber AI Analyst Incident detailing the unusual activities related to the SaaS account hijacking.
Since the customer had subscribed to Darktrace Security Operations Centre (SOC) services, Darktrace analysts conducted an additional human investigation confirming the account compromise.
How Darktrace Combats Phishing Threats
The initial entry point for Mamba 2FA account compromises primarily involves phishing campaigns using HTML attachments and deceptive links. These phishing attempts are designed to mimic legitimate Microsoft services, such as OneDrive and SharePoint, making them appear authentic to unsuspecting users. Darktrace / EMAIL leverages multiple capabilities to analyze email content for known indicators of phishing. This includes looking for suspicious URLs, unusual attachments (like HTML files with embedded JavaScript), and signs of social engineering tactics commonly used in phishing campaigns like Mamba 2FA. With these capabilities, Darktrace successfully detected Mamba 2FA phishing emails in networks where this tool is integrated into the security layers, consequently preventing further implications and account hijacks of their users.
Mamba 2FA URL Structure and Domain Names
The URL structure used in Mamba 2FA phishing attempts is specifically designed to facilitate the capture of user credentials and MFA tokens while evading detection. These phishing URLs typically follow a pattern that incorporates Base64-encoded parameters, which play a crucial role in the operation of the phishing kit.
The URLs associated with Mamba 2FA phishing pages generally follow this structure [6]:
https://{domain}/{m,n,o}/?{Base64 string}
Below are some potential Mamba 2FA phishing emails, with the Base64 strings already decoded, that were classified as certain threats by Darktrace / EMAIL. This classification was based on identifying multiple suspicious characteristics, such as HTML attachments containing JavaScript code, emails from senders with no previous association with the recipients, analysis of redirect links, among others. These emails were autonomously blocked from being delivered to users' inboxes.
Figure 9: Darktrace / EMAIL highlighted a possible phishing email from Mamba 2FA, which was classified as a 100% anomaly.
Figure 10: Darktrace / EMAIL highlighted a URL that resembles the characteristics associated with Mamba 2FA.
Conclusion
The rise of PhaaS platforms and the advent of AiTM phishing kits represent a concerning evolution in cyber threats, pushing the boundaries of traditional phishing tactics and exposing significant vulnerabilities in current cybersecurity defenses. The ability of these attacks to effortlessly bypass traditional security measures like MFA underscores the need for more sophisticated, adaptive strategies to combat these evolving threats.
By identifying and responding to anomalous activities within Microsoft 365 accounts, Darktrace not only highlights the importance of comprehensive monitoring but also sets a new standard for proactive threat detection. Furthermore, the autonomous threat response capabilities and the exceptional proficiency of Darktrace / EMAIL in intercepting and neutralizing sophisticated phishing attacks illustrate a robust defense mechanism that can effectively safeguard users and maintain the integrity of digital ecosystems.
Credit to Patrick Anjos (Senior Cyber Analyst) and Nahisha Nobregas (Senior Cyber Analyst)
Appendices
Darktrace Model Detections
SaaS / Access / M365 High Risk Level Login
SaaS / Access / Unusual External Source for SaaS Credential Use
SaaS / Compromise / Login From Rare Endpoint While User Is Active
SaaS / Compliance / M365 Security Information Modified
SaaS / Compromise / Unusual Login and New Email Rule
The Gartner Magic Quadrant for Email Security is designed to help organizations evaluate which email security solutions might be the best fit for their needs by providing a visual representation of the market vendors and the strengths and cautions of different vendors. We encourage our customers to read the full report to get the complete picture.
Darktrace / EMAIL has a unique AI approach to identifying threats, including NLP and behavioral analysis, instead of traditional security measures like signatures and sandboxing – providing protection against advanced attacks like Business Email Compromise (BEC) and spear phishing. We believe our AI-first approach delivers high-quality solutions that our customers trust, allowing them to stay ahead of sophisticated threats that other tools miss.
We’re proud of Darktrace’s rapid growth, geographic scale, and ability to execute effectively in the email security market, which reflect our commitment to delivering high-quality, reliable solutions that meet the evolving needs of our customers.
What do we believe makes Darktrace the fastest growing email security solution on the market?
An AI-first approach to innovation: Catching the threats others miss
As one of the founders of the ICES category, Darktrace has a long history of innovation, backed by over 200 patents. While other email security solutions are only just starting to apply machine learning (ML) techniques to outdated methods like signature analysis, reputation lists, and sandboxing, Darktrace has redefined the approach to email threat detection with its pioneering AI-driven anomaly detection engine.
Traditional ESPs often miss advanced threats because they rely on rules and signatures that focus on payloads and blindly trust known sources. This approach requires constant updates and frequently fails to detect threats like Business Email Compromise and Spear Phishing. In contrast, Darktrace / EMAIL uses advanced anomaly detection to identify the most sophisticated threats by focusing on unusual patterns and behaviors. This innovative approach has consistently delivered superior detection, stopping on average 58% of the threats that other solutions in the security stack miss.1
But our AI-first approach doesn’t stop at the inbox. At Darktrace, we transcend the limitations of traditional email security by leveraging a platform that unifies insights across multiple domains, providing robust protection against multi-domain threats. Our award-winning solutions defend the most popular attack vectors, including email, messaging, network, and identity protection. By combining signals from all domains, we establish unique behavioral profiles for each device and user, significantly enhancing detection precision.
This pioneering approach has led to introducing industry-first advancements like QR code analysis and automated incident investigations, alongside game-changing functionality including:
Microsoft Teams security with advanced messaging analysis: The ability to identify critical early phishing and insider threats across both email and Microsoft Teams messaging.
AI analyst narratives for improved end user reporting: that reduces phishing investigations by 60% by exposing unique narratives that provide the context of each received email and give feedback to each employee as they interact with their mail.2
Mailbox Security Assistant: to perform advanced behavioral browser analysis and stop malicious links within webpages, detecting and remediating 70% more malicious phishing links than traditional tools.3
AI based, autonomous data loss prevention: to immediately secure your organization from misdirected emails, insider threats, and data loss—both classified and unclassified- without any administrative overhead.
Customer trust that fuels exponential growth
With almost 5,000 customers in under 5 years, we've doubled the growth rate of other vendors in the email security market. Our rapid market penetration, fueled by customer satisfaction and pioneering technology, showcases our revolutionary approach and sets new industry standards.
Darktrace’s exceptional customer retention is fueled by an unparalleled customer experience, extensive regional support, dedicated account teams, and cutting-edge scalable technology. We pride ourselves on having a global network with local expertise, consisting of 110 worldwide offices which provide local language and technical support to offer multilingual, in-house assistance to our customer base.
Check it out – Darktrace / EMAIL has the highest percentage of 5-star ratings with a 4.8 rating on Gartner® Peer Insights™.4
Supporting every stage of your email security journey
Darktrace / EMAIL supports your security maturity journey, from first time security buyers to mature security stacks looking to augment their existing ESPs – by handling advanced threats without extensive tuning. And unlike other solutions that create a siloed and parallel solution, it works harmoniously with native email providers to create a modern email security stack. That’s why Darktrace performs well with first-time email security buyers and has strong renewal rates.
Integrating with Microsoft and Google via API, we replace traditional Secure Email Gateways (SEGs) with a modern, comprehensive email security stack. By combining approaches, our solution merges attack-centric analysis, which learns attack patterns and threat intelligence, with a business-centric approach that understands user behavior and inbox activity to deliver a unified stack that defends the entire threat spectrum – leading Darktrace to be recognized as Microsoft Partner of the year UK 2024.
Our user-friendly, self-learning AI solution requires minimal tuning and deployment, making it perfect for customers looking for a highly usable but lightly configurable solution that will accompany them throughout their lifetime as they mature their email security stack in line with the evolving threat landscape.
Learn more
Get complimentary access to the full Gartner® Magic Quadrant™ for Email Security Platforms here.
1 From September 1 – December 31 2023, 58% of the phishing emails analyzed by Darktrace / EMAIL had already passed through native spam filtering and email security controls. (Darktrace End of Year Threat Report 2023)
2 When customers deployed the Darktrace / EMAIL Outlook Add-in there was a 60% decrease in incorrectly reported phishing emails. Darktrace Internal Research, 2024
3 Once a user reports phishing that contains a link, an automated second level triage engages our link analysis infrastructure expanding the signals analyzed. Darktrace Internal Research, 2024
.jpg)
