How To Test Antivirus and EDR Software: A Complete Guide

At Blumira, we regularly onboard new technologies to help our customers integrate their existing security solutions with our cloud SIEM. This means we often need to test new endpoint tools like antivirus and EDR to look for actionable alerts generated by the source tooling. 

Here are some ways to test antivirus and EDR software.

Why You Should Test Antivirus Software and EDR

As security professionals tasked with touching so many technologies, we get a chance to test and see what endpoint tools detect or prevent. In our experience, some of them are sadly far behind the curve when it comes to providing actionable alerts, detection depth, or simply prevention effectiveness.

Testing also helps confirm if the endpoint tool has some EDR features that look at attacker behaviors like process activity, network connections and registry content rather than just raw file inspection. 

We also recommend that you deploy the Sysinternals logging tool, Sysmon, in addition to EDR and antivirus software. Blumira offers many threat actor behaviors mapped to detections based on Sysmon logs, which provide a kind of flight recorder view of all the system’s activity.

Learn How To Deploy Sysmon Here >

An Overview of Antivirus and EDR Testing

To test antivirus and EDR tools, a good starting point is to see if the tooling can at least compete with a default Windows 10 install using Windows Defender with Real-Time Protection, as this is installed and free on all Windows systems.

The main goal of the testing is to push our endpoint software to see if it can detect any of these activities and allow us to write meaningful detections that can prevent a breach or ransomware incident.

Environment

For complete testing, the environment requires three hosts, a threat actor system, an initial user endpoint, and a domain controller server. If you only want to emulate phase one tests, the requirements are just a threat actor system and the initial user endpoint.

Initial Configuration Requirements

For this test regime, we will be using the Metasploit Framework, a tried and true adversary and red team platform still in use both by penetration testers and real-world attackers. We will be doing some initial configuration that will be a little bit more than default configuration to make sure that there is at least a little bit of a bar for the tool to be tested to have to pass. But we will refrain from more advanced techniques as our goal is to be detected, if possible.

For initial setup, you’ll need to get a host running Kali. This can be hosted in your local lab’s LAN or created in a cloud environment like AWS.

Kali install docs:

https://www.kali.org/docs/installation/

AWS Market Kali Image:

https://aws.amazon.com/marketplace/pp/B01M26MMTT

After getting Kali installed or started and connecting either via SSH or the GUI (depending on your installation environment), start the Metasploit console and run some collection tasks to change the SSL certificate used by the framework.

See step-by-step instructions here.

For the targets, set up a server that you’ll want to make a domain controller and an endpoint to join to the test domain.

Once you’ve configured the hosts, it’s time to generate the payloads for testing. This can better help judge how an endpoint tool handles unknown malware. Protection against old threats, while helpful, is less actionable. We want to generate alerts that require security intervention.

We will be giving a handicap and generating several payloads at various iterations of encoding complexity using MSFvenom and the shikata ga nai encoder. This will make the payloads unique to our test. The shikata ga nai encoder used to be a very effective evasion method, but the encoding method should be readily detected by most antivirus programs today.

See the payload generation steps here.

Phase One

This first phase of testing helps to establish a baseline of how a particular endpoint tool handles common attack tools and methodologies. 

As such, it’s important to provide a lot of handicaps to the test threat actor — things like local admin privileges on the initial host and domain admin later on for lateral movement. You want to apply an “assumed breach” methodology rather than test every minute requirement of an attack chain. Should a tool prove proficient enough, you can return with a finer tooth comb to look at bypasses and more in-depth detection opportunities. 

Initial Access

The first test is meant to simulate a simple phishing link that directs a user to a poorly-configured attacker stager website. We are looking to see if the tool either inspects traffic to the host or hooks into the browser to block malicious files.

Step-by-step instructions here.

For our “assume breach” methodology, in this example we’d turn off Microsoft Defender’s Real-Time Protection to allow the rest of the testing to continue.

Execution

For the next test, we’ll need to execute the downloaded files, At this point you can try both with all detections re-enabled or the blocking controls disabled if it prevents further testing. (But mark that as a good outcome.)

Before you start the execution, start the meterpreter listener on our Kali host. Those instructions can be found here.

After executing the files, record what, if any, alerting or logs that are generated. 

Discovery

After execution, most threat actors will run a variety of commands to discover information about the local system and connected domains so next we’ll emulate that and look for any alerting or logs that the endpoint technology generates.

The step-by-step instructions can be found here.

A Windows administrator may notice that all the commands run here are built in Windows utilities that get used by normal users and system administrators. While that’s true, they are favored by threat actors for that exact reason, but a modern requirement is to be able to trigger on suspicious invocation of these utilities that fall out of baseline usage.

Local Privilege Escalation

The following phase in our intrusion chain is elevation of privilege that allows the threat actor to collect sensitive data further down the attack path. Here we’re looking to see if the endpoint technology can detect the common Named Pipe impersonation and process injection.

The step-by-step instructions can be found here.

Credential Access

For the final part of the first phase we will dump credentials using the meterpreter shell. This may be difficult for most endpoint tools to determine, as this will all be occurring in memory, but is a common step in an intrusion.

You can find the step-by-step guide here.

This completes the first round of testing localized to a single endpoint, for the next phase we’ll begin lateral movement and look for taking actions against a domain controller.

Phase Two

Once you’ve completed tests for a single system, you’ll want to take it a bit further and look at lateral movement and some later-stage actions that a threat actor would take. So for phase two of testing, model out lateral movement to a domain controller (DC) and threat actor activity on a DC. 

Like in Phase One, you should provide a few handicaps using an assumed breach methodology and not require some tedious actions that would be necessary for a full, real-life intrusion; this will include things like providing a user with privileges high enough to pivot to our server, and password or hash material to allow authentication.

Lateral Movement

Once a threat actor has a foothold in an environment and has gathered enough initial information of the target environment, they’ll want to move laterally. In a full, live environment it may take many steps of lateral movement for the threat actor to reach their goals. For our test, we’re going to speed up and use domain administrator credentials to pivot to our domain controller server and see if our endpoint tool can pick up on remote service execution, which is used by many post-exploitation frameworks. 

For step-by-step instructions see here.

Process Injection

In any intrusion there is never a straight path following the traditional “kill-chain” model; you often see a circular pattern as threat actors repeat certain steps as they move through to their goals. Here, we will repeat a process injection on the server and see if the tool we are testing can detect it.

See the step-by-step instructions here.

Domain Credential Access

Once on a domain controller, most threat actors want to get access to all of the domain credentials, allowing them many ways to return to a domain later, or to sell access off to others. These are some of the crown jewels of any enterprise network and we want to see if the endpoint tool being tested will observe unusual access on the domain controller to the files that store the domain credentials.

See the step-by-step instructions here.

Defense Evasion

Now we are getting toward the final steps of our simulated intrusion. Some threat actors will attempt to clear evidence of their actions before leaving an environment, so we will simulate that with clearing the Windows logs on our domain controller here.

The step-by-step instructions for the test are here.

Impact

For the last test, we’ll execute another built-in Windows utility that ransomware groups often execute to lock systems in return for a payment. If you haven’t detected or stopped a threat actor by this point it is already too late, but in a large environment drastic measures may limit some fallout. Here, we want to see if the endpoint technology can detect unusual access and use of this administrative utility. 

Step-by-step instructions can be found here.

Conclusion

With these tests, you should now have a handle on the basic capabilities of the endpoint technology being tested. You can also craft actionable detections for a security team to take action on, prioritizing the high fidelity alerts from basic adware noise.

Blumira Passes The Test

Blumira’s built-in detections can alert you about each of these suspicious attacker behaviors that lead to a data breach or ransomware attack.

You can deploy Blumira Free SIEM in a matter of minutes. Try it today and execute some of these tests to see the immediate security value that Blumira can offer. 

In addition to our free edition, we also offer a free Domain Security Assessment to help you quickly assess your security using only your domain name. Sign up for the assessment to get a detailed report of your risks and what actions to take, before attackers do. 

Learn More About EDR Testing

Lastly, you can hear from Joff Thyer, Penetration Tester, Developer and Researcher at Black Hills Information Security, along with Blumira’s Brian Laskowski, Incident Detection Engineer, as they go through ways to test your EDR. Watch the on-demand webinar here.