MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH? A SPECIAL ANALYSIS

YOU DESERVE THE BEST SECURI TY

About This Report This repor t shares an analysis of Microsoft 365 Email Secur ity with its Defender Secur ity product. To prepare this repor t, the Check Point Email Research Team (previously Avanan) analyzed three mill ion real-wor ld emails to evaluate the efifcacy of Microsoft Defender for Microsoft 365.

In general, Microsoft 365 is a very secure service. That is a result of a massive and continuous investment from Microsoft. What this repor t does note is the challenge that Microsoft faces. As the default security for most organizations, many hackers design their attacks to bypass Microsoft’s secur ity. An excellent example of how hackers focus on Microsoft 365 comes in a series of blogs from Microsoft that details the attempts of a state-sponsored group to compromise their services.

This is most l ikely why we see a signiifcant percentage of attacks bypassing Microsoft secur ity. In this context, it’s impor tant to note that this does not mean that Microsoft's secur ity has worsened. The hackers improved and learned more methods to obfuscate and bypass the default security.

For this reason, Check Point’s secur ity exper ts recommend that our customers add an extra layer of secur ity on top of the default security provided by their cloud email service.

This repor t will offer data on Defender’s effectiveness and explore how Defender fares in organizations of different sizes. We’l l closely examine how the “Dumpster Diving” phenomenon stil l hur ts organizations and show how Defender fares against var ious types of phishing. And ifnally, we’l l summarize how these numbers affect your SOC and your business.

This repor t obviously could not analyze all Microsoft 365 customers. It is derived from analyzing customers using Microsoft Defender who also have Check Point’s Email Secur ity Solution. The results should be taken as a general guide to Microsoft’s efifcacy. We estimate that the data set is large enough to constitute a statistically representative sample of Microsoft 365 customers with some var iance in the results for different organizations.

2MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

3MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

The Results Check Point’s research team found that 18.8% of phishing emails bypassed Microsoft Exchange Online Protection (EOP) and Defender to make it to a user ’s inbox.

This is a 74% increase compared to our previous repor t from February 2020, where we used a similar analysis method and found that only 10.8% of phishing emails were missed.

For the 2022 repor t, Check Point analyzed nearly 3,000,000 emails scanned by Microsoft Defender dur ing one week. The four organizations in our sample ranged from 500 to 20,000 users, were from major industr ies, and were located in all par ts of the United States.

We picked these organizations because they run both Microsoft Defender and Harmony Email and Collaboration Secur ity (HEC). Because HEC runs inline behind Defender and connects via API, we can look at the following to help with classiifcations:

• Email headers added by Defender after ifles have been scanned but before they are released into quarantine, junk folders, or user inboxes

• Defender quarantine folder and event logs

We then analyzed the messages that Check Point’s HEC classiifed as phishing, but EOP and Microsoft Defender did not. Without Check Point, these emails would have been delivered to the inbox. That number of emails classiifed as phishing by Check Point divided by the total email attacks gives us the percentage of missed phishing emails.

This increase may be due to a few factors. We conducted our previous analysis in February 2020, before COVID-19. At that time, we found that 10.8% of phishing emails bypassed Microsoft and made it into the inbox. Since then, the number of attacks has increased by almost 100%, focusing on sophisticated phishing campaigns that bypass built- in secur ity.

4MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

Defender’s Missed Phishing Rate is Higher in Larger Organizations Our analysis found that when deployed in larger organizations, Microsoft Defender missed more phishing emails. The missed phishing rate for two large organizations in our research was 50-70%.

This is a change from our 2020 analysis. In 2020, we found no correlation between company size and missed phishing rate. We speculate that hackers are now focusing more of their effor ts on more prominent organizations and, in par ticular, the advanced attacks that they don’t want to ‘waste’ on targets less lucrative for them.

Remote work is another factor that may be at play. We conducted our 2020 analysis just before the COVID-19 pandemic. Now, many more employees are working from home, the volume of attacks has increased, and Business Email Compromise (BEC) and ransomware demands have become headline news wor ldwide.

For reasons we will discuss later in the repor t, the type of attacks used by bad actors is changing. Our analysis found that targeted ifnancial attacks are speciifcally crafted to bypass Defender. Financial-based phishing attacks refer to many attack types, including fake invoice scams, fraudulent Bitcoin transfers, phony business proposals, fake wire transfer requests, and more. Defender misses 42% of these types of attacks. Financial attacks are more common against large organizations. The Ver izon Data Breach Investigation Repor t conifrms that ifnancially motivated attacks are the most common attack vector.

The Dumpster Diving Problem Another par t of our analysis included emails Defender forwarded to a user ’s Junk Folder. In May 2020, Avanan coined the term Dumpster Diving. This refers to the practice where marketing emails, subscr iptions, and targeted phishing attacks are commingled in the Junk folder, making them immediately accessible to the end-user. Many organizations send all Defender detections to the Junk folder. They deem this preferable to sending Defender detections to quarantine because it reduces the r isk of blocking legitimate emails.

We found that, on average, Defender sends 7% of phishing messages to the Junk folder. End-users become accustomed to dumpster diving in the Junk folder for legitimate messages. Users may act on a phishing email by mistake with many emails to root through in the junk folder with no distinction between treasure and trash.

5MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

In some attacks, an additional email of high reputation points the recipient to the search their Junk folder for the message. We’ve seen the text in attacks such as: “Did you get my message? Please check your Junk folder. If you don’t ifnd it, let me know, and I’l l resend it. The subject should be ‘X.’” This attack serves two purposes. First, it sends the recipients Dumpster Diving through dangerous phishing emails. Second, when the email is found, the end-user tends to give it more validity as if the second email conifrms the ifrst is legitimate.

Forwarding phishing to the junk folder is very r isky and one of the critical weaknesses in the Microsoft Defender and EOP.

Defender Versus Different Types of Threats The following pie char t breaks down the attacks by categories:

As a quick overview, here are brief deifnitions of these types of phishing:

• Finance, 42%: This category refers to attacks having to make money, ranging from bitcoin scams to invoice fraud, fake wire transfers, and more

• Brand impersonation, 22%: This category refers to phishing emails that look like they are coming from a popular and legitimate brand

Categories of Phish that Defender Misses

6MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

• Credential harvest, 21%: This is a broad category that refers to types of phishing emails with the overall goal of stealing usernames and passwords

• Social engineering, 12%: This is another broad category that refers to the idea of hackers tr icking a user into doing something they don’t want to do

• Business Email Compromise, 2%: Business Email Compromise (BEC) is a popular attack form that sees a hacker impersonating an executive, asking an underling for urgent action

• Taxes, 1%: This is a subset of phishing that takes advantage of tax fears to extract money

As discussed previously, targeted ifnancial attacks are often successful against Defender. This broad category includes anything involving money – fake invoices, bitcoin transfers, etc. These scams tend to hit the enterpr ise space in larger quantities.

As discussed previously, targeted ifnancial attacks are often successful against Defender. This broad category includes anything involving money – fake invoices, bitcoin transfers, etc. These scams tend to hit the enterpr ise space in larger quantities.

The email below shows an example of a ifnancial scam missed by Defender. If a user clicks the purpor ted business proposal l ink, it takes them to a fake Microsoft 365 por tal .

There are a few ‘tells’ in the email. In the body, notice how a zero is used in place of the letter ‘o’ in the subject.

7MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

After clicking, users are directed to these pages. Pay attention to the strange URL and the off-center text.

After clicking “View Proposal,” the user is taken to this spoof of a Microsoft login page. Notice the random URL.

Brand impersonation is another method hackers choose to bypass Defender. These emails claim to come from a well-known brand but are a hacker tr ying to get information.

In the attack shown below, hackers spoof Best Buy. However, the reply-to address offers some clues as to its legitimacy.

Finally, credential harvesting attacks saw success against Defender. The idea is to steal something from the end user. These attacks range from attempts to harvest Microsoft 365 login credentials to something more sophisticated, l ike the attack seen below.

In this attack, the end-user sees an email from what appears to be PayPal. It shows a purchase of $509.49. At the bottom, it encourages recipients to call a phone number if they do not recognize the transaction. When they reach the number, it gets routed to a ‘suppor t rep’ who asks for banking information.

9MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

Malicious Files Malware propagating through ifles remains a signiifcant attack vector. These are often considered separately from phishing, but too often, phishing techniques lead to malicious content and vice-versa. The goal is to infect the endpoint for remote control or ransomware.

For example, Microsoft Defender missed the real-wor ld email below and delivered it to an end-user’s inbox. It describes a business proposal. While this email doesn't look like a traditional phishing attack, the ifle attached to the document is a macro-infested Excel document.

Check Point Research’s analysis of the infected ifle shows it’s malicious and a critical secur ity r isk.

10MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

The Impact on the Security Operations Center Combating the r ise in missed phishing attacks falls on the SOC team. To evaluate the impact on the SOC, we compared our customers in HEC’s prevent mode, which actively blocks malicious emails, with two customers using HEC’s Monitor Only mode. In Monitor Only, SOC staff will be aler ted to issues, but the problems will need to be assessed and responded to manually. Check Point found that manually managing email problems takes up to 23% of the staff ’s total time. That work involves remediating email threats, responding to end-user requests, and more. In some environments, that number is even higher.

Another method we used to evaluate the load on the SOC team was to look at the time spent by SOC staff before and after deploying Check Point’s email security. In one large company, before deploying Check Point, in an average week, there were 910 repor ted phishing emails. The IT team could not catch up; they could only remediate an average of 59 per week (or less than 7% of total repor ts). This organization estimated they would need 16 full- time employees to deal with the user-repor ted phishing problem. When moving to full HEC’s prevent mode, this number was reduced to two user-repor ted phishing emails per week. We analyzed another organization prior to deploying Check Point and found that they spend 879 hours or 36 full days annually researching phishing reports. A third organization spent 2,500 hours a year reviewing suspicious email repor ts from end-user. That’s the equivalent of 104 days.

That time drain leads to other prior ities being overlooked and massive burnout among the IT and engineering staff. With users repor ting both actual and imagined phishing emails, the SOC spends far too much time sifting through the smoke, unable to locate the ifre.

What Defender Does Well In addition to the sections outlined below, we’ve observed that Defender does an excellent job with malware – Check Point has found that, with unknown malware, Microsoft catches 90%. Fur ther, Microsoft Defender includes URL rewriting, a key feature to prevent time-of-click attacks. In our analysis, we’ve observed that in the environments sampled; Defender limited the amount of phishing in these categories:

11MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

DMARC Spooifng DMARC, or Domain-based Message Authentication, Repor ting & Conformance, is an email standard used to authenticate an email. It’s a way for operators to identify legitimate emails. The idea is to prevent hackers from spooifng an organization. However, in our analysis, we found that in 2.5% of cases, when Defender ifnds a case of a spoofed email and DMARC failed, it stil l sends it to the Junk Folder.

Business Email Compromise Business Email Compromise (BEC) is one of the fastest growing and most successful attack vectors. Though simple, it works because it uses social engineering and doesn’t include any malware or malicious links. The idea is to spoof an executive asking an underling for an urgent favor. This can be the purchase of gift cards or other ifnancial transactions. Since 2016, BEC-related losses have totaled over $43 bil l ion.In our analysis, Microsoft did an excellent job limiting these attacks from reaching the inbox, allowing just 2.0% to pass.

Defender vs. Secure Email Gateways In one study analyzing 300 million emails, we found that Microsoft is in the middle of the pack compared to other Secure Email Gateways. Per every 100,000 emails, Microsoft’s catch rate of phishing emails is better than some Secure Email Gateways and worse than others.

12MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

Where Microsoft does shine, as mentioned above, is malware. After analyzing 360 samples of both known and unknown malicious PDF, DOC, XLS, and other executable ifles, we were able to determine the catch rate for all malicious ifle types:

To break this down fur ther, Microsoft's efifcacy is even higher for just unknown malicious Ofifce Docs and PDFs.

When considering email security options, it’s essential to consider the entirety of a solution’s offer ing.

13MICROSOFT DEFENDER FOR OFFICE 365: IS IT GOOD ENOUGH?

Worldwide Headquarters 5 Ha’Solelim Street, Tel Aviv 67897, Israel | Tel: 972-3-753-4555 | Fax: 972-3-624-1100 | Email: info@checkpoint.com

U.S. Headquarters 959 Skyway Road, Suite 300, San Carlos, CA 94070 | Tel: 800-429-4391; 650-628-2000 | Fax: 650-654-4233

www.checkpoint.com

© 2022 Check Point Software Technologies Ltd. All rights reserved.

Conclusion Microsoft 365 is the most used and targeted email service wor ldwide. After thoroughly analyzing near ly three mill ion emails, the Check Point’s Research team found that Microsoft Defender misses 18.8% of phishing emails. This represents a 74% increase in missed attacks compared to the previous repor t from February 2020. This is not necessar ily a decline in Microsoft’s effectiveness, but rather an increase in hacker sophistication and speciifcally an increase in targeted attacks designed speciifcally to bypass Microsoft Defender. Hackers, in other words, have stepped up their game faster than Defender can respond.

Our security exper ts recommend a “defense-in-depth” strategy for all security, including email. Layer ing an additional level of protection on top of the default, whether Microsoft or Google, is recommended as a best practice for email security. It is essential as hackers continue to target and ifnd ways to bypass the layers of protection included in the service by the service provider.

About Check Point Harmony Email & Collaboration Check Point’s Harmony Email and Collaboration Security (HEC) platform is the only security solution that can prevent malicious content from reaching a user’s inbox. HEC operates “ Inline,” meaning that the platform scans all emails and attachments before they get a user’s account. The platform scans for potentially malicious content using AI and Machine Learning algorithms, aler ts SOC staff when an issue is found, and quarantines the harmful content.

Check Point’s unique architecture allows it to learn from the speciifc emails that Microsoft misses. These are often highly targeted attacks explicitly designed to bypass Microsoft’s protections. HEC’s email security architecture sits behind Microsoft. When Microsoft blocks an attack, the attack is blocked, entirely stopped. When it doesn’t, however, HEC sits between it and the inbox, giving a ifnal analysis. Our AI has the unique ability to be trained on these speciifc, sophisticated, and evasive attacks.