Remediating Networks Affected by the SolarWinds and Active Directory/M365 Compromise

Updated April 15, 2021: The U.S. Government attributes this activity to the Russian Foreign Intelligence Service (SVR). Additional information may be found in a statement from the White House.

Note: although the guidance on this webpage is tailored to federal departments and agencies, the Cybersecurity and Infrastructure Security Agency (CISA) encourages critical infrastructure and private sector organizations to review and apply it, as appropriate. For more information on CISA’s response to this activity, refer to cisa.gov/supply-chain-compromise.

Since December 2020, the Cybersecurity and Infrastructure Security Agency (CISA) has been responding to a significant cybersecurity incident affecting networks of multiple U.S. government agencies, critical infrastructure entities, and private sector organizations. Although the malicious activity varied among affected entities, an advanced persistent threat (APT) actor targeted and gained long-term access to select organizations’ enterprise networks and moved laterally to Microsoft cloud systems—i.e., Azure Active Directory (AD) and Microsoft 365 (M365) environments. The actor used privileged access to collect and exfiltrate sensitive data and created backdoors to enable their return.

CISA is providing the guidance below to support federal departments and agencies in evicting this threat activity from compromised on-premises and cloud environments. This guidance addresses tactics, techniques, and procedures (TTPs) leveraged by the threat actor. Given that the threat actor may be deeply burrowed in networks, eviction will be challenging and complex; this guidance provides short- and intermediate-term actions that agencies can take to mitigate this activity and prevent the actor’s re-use of similar TTPs. By taking steps to evict this adversary from compromised on-premises and cloud environments, agencies will position themselves for long-term actions to build more secure, resilient networks.

Threat Actor Activity

Threat Actor Activity

In December 2020, CISA was made aware of a supply chain compromise of certain versions of the SolarWinds Orion platform. An APT actor added malicious code to multiple versions of SolarWinds Orion and, in some instances, leveraged it for initial access to enterprise networks of U.S. government agencies, critical infrastructure entities, and private sector organizations.

Through incident response, CISA determined that, in other instances, the threat actor obtained initial access by password guessing, password spraying, and exploiting inappropriately secured administrative credentials via remote services.

Once inside the network, the threat actor bypassed multi-factor authentication (MFA) and moved laterally to Microsoft cloud systems by compromising federated identity solutions. The threat actor:

  • Stole the Active Directory Federation Service (ADFS) token-signing certificate to forge tokens. By using this technique, referred to as “Golden SAML” (Security Assertion Markup Language), the threat actor was able to move laterally to M365 environments by authenticating into the federated resource provider, bypassing MFA and password requirements.
  • Modified or added trusted domains in Azure AD. By using this technique, the threat actor was able to move laterally to Azure AD environments by adding new federated identity providers (iDPs). (See FireEye White Paper: Remediation and Hardening Strategies for Microsoft 365 to Defend Against UNC2452.)

After gaining access to cloud environments, the actor established persistence mechanisms for Application Programming Interface (API)-based access and collected and exfiltrated data.

The threat actor has demonstrated sophisticated defense evasion skills. The actor:

  • Hid their command and control (C2) communications with extensive obfuscation,
  • Hid their activity among legitimate user traffic, and
  • Established difficult-to-detect persistence mechanisms (e.g., in API).

Note: for more information on this activity, including TTPs, refer to CISA Activity Alerts:

Introduction: Mitigating and Remediating Malicious Activity

Introduction: Mitigating and Remediating Malicious Activity

Note: this remediation guidance applies to organizations that use SolarWinds Orion. CISA is aware of other initial access vectors—agencies should not assume they are not compromised by this actor solely because they have never used affected versions of SolarWinds Orion. Those agencies should investigate to confirm they have not observed related threat actor TTPs. If related activity is detected, agencies may find this guidance helpful. Those agencies are encouraged to review Joint Advisory AA20-245A: Technical Approaches to Uncovering and Remediating Malicious Activity and to contact CISA for further assistance.

Because the actor moved laterally to multiple systems and established difficult-to-detect persistence mechanisms, their activity will be difficult to detect and eradicate. CISA has provided the following short- and intermediate-term steps agencies can take to detect, mitigate, and remediate this activity. By following the ordered steps, agencies will harden their networks  and prevent re-use of the TTPs. CISA will update this webpage with further guidance on specific actions as new information becomes available.

Conduct a Risk/Impact Assessment

Conduct a Risk/Impact Assessment

Networks with SolarWinds Orion products will generally fall into one of three categories. Agencies will need to conduct a risk/impact assessment to determine which category they fall into.

  • Category 1 includes agency networks that do not have the identified malicious binary code on their network and can forensically confirm that the binary was never present on their systems. This includes networks that do not, and never did, use the affected versions of SolarWinds Orion products.
  • Category 2 includes agency networks where the presence of the malicious binary has been identified—with or without beaconing to avsvmcloud[.]com.
  • Category 3 includes agency networks that used affected versions of SolarWinds Orion and have evidence of follow-on threat actor activity, such as binary beaconing to avsvmcloud[.]com and secondary C2 activity to a separate domain or IP address (typically but not exclusively returned in avsvmcloud[.]com Canonical Name record [CNAME] responses).

Resources

Remediating Malicious Activity: Category 1 and 2 Organizations

Remediating Malicious Activity: Category 1 and 2 Organizations

Although unaffected by this incident, Category 1 organizations should work to maintain strong network posture and resilience. Refer to https://www.cisa.gov/cybersecurity and https://us-cert.cisa.gov/resources/federal for assistance. CISA recommends Category 1 organizations:

Category 2 organizations should continue enhanced monitoring for any possible follow-on adversary activity. Refer to resources below for more information.

Resources:

According to ED 21-01 and associated supplemental guidance, all federal agencies that ran affected versions of SolarWinds Orion must “conduct system memory, host storage, network, and cloud forensic analysis,” “hunt for indicators of compromise (IOCs) or other evidence of threat actor activity, such as secondary actions on objectives (AOO),” and “[i]dentify and remove all threat actor-controlled accounts and identified persistence mechanisms.”

Remediating Malicious Activity: Category 3 Organizations

Remediating Malicious Activity: Category 3 Organizations

Note: Category 3 organizations should use out-of-band communications for all mitigation and remediation documentation and conversations, i.e., do not use any compromised systems to communicate remediation plans or actions.

For Category 3 organizations, completing all the steps provided in this guidance is necessary to fully accomplish eviction. (Note: CISA will release detailed information on eviction steps to agencies via the Homeland Security Information Network [HSIN]). These are resource-intensive and highly complex and will require the enterprise network to be disconnected from the internet for 3–5 days; however, failure to perform a comprehensive and thorough remediation will expose enterprise networks and cloud environments to substantial risk of long-term undetected APT activity, including email monitoring and data collection and exfiltration. CISA recommends organization leadership read the CISA Insights, Remediating Networks Affected by the SolarWinds and Active Directory/M365 Compromise: Risk Decisions for Leaders, for more information.

Remediation plans for dealing with malicious compromises are necessarily unique to every organization, and success requires careful consideration. There are three phases for evicting the actor:

  • Phase 1: Pre-Eviction. Actions to detect and identify APT activity and prepare the network for eviction.
  • Phase 2: Eviction. Actions to remove the APT actor from on-premises and cloud environments. This phase includes rebuilding devices and systems.
  • Phase 3: Post-Eviction. Actions to ensure eviction was successful and the network has good cyber posture.

Phase 1: Pre-Eviction

Define the True Scope

  • Action 1: Identify trust boundaries and determine the enterprise assets to which this guidance applies (i.e., determine what assets are within the trust boundary).

Resources:

Investigate Suspicious Account Activity

  • Action 1: Investigate suspicious account activity associated with SolarWinds Orion servers, especially service accounts used by SolarWinds Orion.
  • Action 2: Enumerate and investigate any credentials stored or used on the SolarWinds server, including network administration and device credentials. If a Category 3 organization cannot fully confirm all activity of these credentials is benign, they should proceed as if the highest administrative level of credentials on their affected Category 3 SolarWinds server have been compromised. In many cases, the adversary may have had months with this access.

Resources:

Investigate Potential SAML Abuse

  • Action 1: Investigate attacks on identity sources, such as SAML forgery. If the adversary has compromised administrator credentials in an environment—or if organizations identify SAML abuse in the environment—simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network. In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears emphasizing that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action. For many organizations, remediation from this level of compromise may require third-party assistance.

Resources:

Scope the Intrusion

  • Action 1: Look for the artifacts from known TTPs associated with this activity. Refer to SolarWinds and Active Directory/M365 Compromise: Detecting APT Activity from Known Tactics, Techniques, and Procedures and corresponding detection artifacts. Prioritize these by the biggest value for investment (e.g., prioritize by techniques or technologies that cover multiple tactics or that provide visibility into shared data sources).
  • Action 2: Audit all network device configurations stored or managed on the SolarWinds monitoring server for signs of unauthorized or malicious configuration changes.
  • Action 3: Audit all network device configurations for signs of unauthorized or malicious configuration changes. Also be sure to audit the current network device running configuration and any local configurations that could be loaded at boot time.
  • Action 4: Assess the current endpoint telemetry collection level and configure endpoint detection and response (EDR) or detection solution for aggressive collection; prioritize this by value of asset and account.

Resources:

Harden the Enterprise Attack Surface

  • Action 1: Review and validate perimeter firewall rulesets. Remove all rules the organization does not have a documented need for.
  • Action 2: Implement host-based firewalls to make lateral movement more difficult for the actor.
  • Action 3: Close and/or monitor high-risk ports (e.g., Remote Desktop Protocol [RDP], Server Message Block [SMB], File Transfer Protocol [FTP], Trivial File Transfer Protocol [TFTP], Secure Shell, and WebDAV).
  • Action 4: Employ allowlisting, especially for systems providing remote access to the enterprise.
  • Action 5: Enforce enterprise Domain Name System (DNS) resolution for all systems.

Identify Federation Model for On-Premises Resources to Cloud Trust Relationship and Identify Adversary Activity in M365/Azure Environment

  • Action 1: Identify the Source Anchor and commensurate configuration for Azure AD Connect in the current Azure Tenant, if applicable.
  • Action 2: Identify permission and credential changes to applications and service principals. Identify overly permissive applications, unusual credentials in applications, or modifications to federation trust settings. See CISA Activity Alert AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments for more information.
  • Action 3: Review M365 tenant configuration and perform a risk assessment for administrative accounts and applications.

Resources:

Phase 2: Eviction

To evict the actor from the network, agencies should take steps to regain sole control over their AD, remove malware implants from network and cloud systems, and rebuild or re-image network and cloud systems. CISA will release detailed information on eviction steps to agencies via HSIN.

Note: this phase will require agencies to disconnect their enterprise network from the internet for several days. Agencies should plan accordingly.

Resources:

Phase 3: Post-Eviction

Report to CISA

Post-eviction, all Category 3 agencies should report actions taken to CISA, any actions left undone, and their assessments of the residual risk. CISA will release a checklist to agencies via HSIN.

Maintain Vigilance

From a security operations center (SOC) perspective, it is important to continue monitoring for malicious activity related to this campaign. Permanent eviction is complicated; the adversary may have been inside networks for a long period, so they likely understand various weaknesses, which they can attempt to leverage to regain network access in various forms. SOCs should continue monitoring for:

  • Known TTPs associated with this activity, and
  • Signs of persistence, such as C2 connections to new domains and attempts to run unusual code.

CISA also recommends that affected agencies implement behavior analysis and reduce traffic to the internet.

Resources: CISA, Federal Government, and International Partner Publications

Resources: CISA, Federal Government, and International Partner Publications

Note: The following publications focus on the SolarWinds Orion Compromise and Related Activity

Table 1: CISA, Federal Government, SLTT, and International Partners Publications

 

Publication Date

 

Title

 

SolarWinds Orion Compromise and Related Activity

4/15/2021 CISA Malware Analysis Report: MAR-MAR-10327841-1.v1 – SUNSHUTTLE
4/15/2021 CISA Current Activity: CISA and CNMF Analysis of SolarWinds-related Malware
4/15/2021 NSA-CISA-FBI Joint Cybersecurity Advisory: Russian SVR Targets U.S. and Allied Networks
4/15/2021 CISA Current Activity: NSA-CISA-FBI Joint Advisory on Russian SVR Targeting U.S. and Allied Networks
4/8/2021 CISA Current Activity: Using Aviary to Analyze Post-Compromise Threat Activity in M365 Environments
3/18/2021 CISA Alert AA21-077A: Detecting Post-Compromise Threat Activity Using the CHIRP IOC Detection Tool
3/18/2021 CISA Current Activity: Using CHIRP to Detect Post-Compromise Threat Activity in On-Premises Environments
3/9/2021 CISA Insights: SolarWinds and Active Directory/M365 Compromise: Risk Decisions for Leaders 
3/9/2021 CISA Current Activity: Guidance on Remediating Networks Affected by the SolarWinds and Active Directory/M365 Compromise
2/8/2021 CISA Malware Analysis Report: MAR-10318845-1.v1 - SUNBURST
2/8/2021 CISA Malware Analysis Report: MAR-10320115-1.v1 - TEARDROP
2/8/2021 CISA Activity Alert AA20-352A: APT Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations
Note: initial publication of Alert was 12/17/2020; latest update was 4/15/2021.
1/8/2021 CISA Alert AA21-008A: Detecting Post- Compromise Threat Activity in Microsoft Cloud Environments
1/8/2021 CISA Current Activity: CISA Releases New Alert on Post-Compromise Threat Activity in Microsoft Cloud Environments and Tools to Help Detect This Activity
1/6/2021 CISA Emergency Directive 21-01: Mitigate SolarWinds Orion Code Compromise and Supplemental Guidance
Note: initial publication of ED 21-01 was 12/13/2021; latest update to supplemental guidance was 1/6/2021.
1/6/2021 CISA Current Activity: CISA Updates Emergency Directive 21-01 Supplemental Guidance and Activity Alert on SolarWinds Orion Compromise
1/5/2021 CISA/FBI/NSA/ODNI Joint Statement
12/30/2020 Canadian Centre for Cyber Security Alert: Recommendations for SolarWinds Supply-Chain Compromise - Update 1
12/29/2020 Australian Cyber Security Centre Alert: Potential SolarWinds Orion compromise
12/26/2020 CERT/CC: Vulnerability Note VU#843464: SolarWinds Orion API authentication bypass allows remote command execution
12/24/2020 CISA Current Activity: CISA Releases Free Detection Tool for Azure/M365 Environment
12/24/2020 Canadian Centre for Cyber Security Alert: Recommendations for SolarWinds Supply-Chain Compromise
12/23/2020 CISA: Supply Chain Compromise webpage
12/23/2020 CISA Current Activity: CISA Releases CISA Insights and Creates Webpage on Ongoing APT Cyber Activity
12/23/2020 CISA Insight: What Every Leader Needs to Know About the Ongoing APT Cyber Activity
12/22/2020 MS-ISAC: The SolarWinds Cyber-Attack: What SLTTs Need to Know Note: latest update was 12/22/2020.
12/21/2020 UK NCSC statement on the SolarWinds compromise
12/19/2020 CISA Current Activity: CISA Updates Alert and Releases Supplemental Guidance on Emergency Directive for SolarWinds Orion Compromise
12/17/2020 CISA Current Activity: NSA Releases Cybersecurity Advisory on Detecting Abuse of Authentication Mechanisms
12/17/2020 NSA Cybersecurity Advisory: Detecting Abuse of Authentication Mechanisms
12/17/2020 Canadian Centre for Cyber Security Alert: Advanced Persistent Threat Compromises (CISA)
12/16/2020 CISA/FBI/ODNI Joint Statement

12/15/2020

UK National Cyber Security Centre: Dealing with the SolarWinds Orion compromise
12/14/2020 Australian Cyber Security Centre Alert: Potential SolarWinds Orion compromise
12/13/2020 CISA Current Activity: Active Exploitation of SolarWinds Software

 

General Cybersecurity Information

5/19/2019 NCSC: Security Architecture Anti-Patterns

 

Table 2: Industry Publications

 

Publication Date

 

Title

 

SolarWinds Orion Compromise and Related Activity

3/4/2021 MITRE's Center for Threat-Informed Defense Public Resources (GitHub): Solorigate Note: latest update was 3/4/2021.
1/12/2021 Cisco Event Response: SolarWinds Orion Platform Software Attack Note: latest update was 1/12/2021.
12/31/2020 Microsoft: Internal Solorigate Investigation Update
12/21/2020 Microsoft: Solorigate Research Center
12/21/2020 Microsoft: Understanding “Solorigate”’s Identity  IOCs - for Identity Vendors and their customers
12/18/2020 MITRE (Medium): Identifying UNC2452-Related Techniques for ATT&CK
12/17/2020 Microsoft: Latest Threat Intelligence (15 December 2020) - FireEye and SolarWinds Events
12/15/2020 CrowdStrike: The Imperative to Secure Identities: Key Takeaways from Recent High- Profile Breaches
12/14/2020 Volexity: Dark Halo Leverages SolarWinds Compromise to Breach Organizations
12/14/2020 Symantec: Sunburst: Supply Chain Attack Targets SolarWinds Users
12/14/2020 Cisco Talos: FireEye Breach Detection Guidance
12/14/2020 Cisco Talos Threat Advisory: SolarWinds supply chain attack
12/14/2020 Cisco Talos: SolarWinds Orion Platform Supply Chain Attack
12/13/2020 FireEye: Global Intrusion Campaign Leverages Software Supply Chain Compromise
12/13/2020 FireEye: Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims with SUNBURST Backdoor
12/13/2020 Microsoft: Important steps for customers to protect themselves from recent nation- state cyberattacks
12/13/2020 Microsoft: Customer Guidance on Recent Nation-State Cyber Attacks
12/8/2020 FireEye: Unauthorized Access of FireEye Red Team Tools

 

Malware Analysis

1/20/2021 Microsoft: Deep dive into the Solorigate second- stage activation: From SUNBURST to TEARDROP and Raindrop
1/18/2021 Symantec: Raindrop: New Malware Discovered in SolarWinds Investigation
1/11/2021 CrowdStrike: SUNSPOT: An Implant in the Build Process
12/24/2020 FireEye: SUNBURST Additional Technical Details
12/22/2020 CheckPoint Research: SUNBURST, TEARDROP and the NetSec New Normal
12/18/2020 Microsoft: Analyzing Solorigate, the compromised DLL file that started a sophisticated cyberattack, and how Microsoft Defender helps protect customers
12/17/2020 McAfee: Additional Analysis into the SUNBURST Backdoor
12/17/2020 Palo Alto Networks: SUPERNOVA: A Novel .NET Webshell

 

Incident Response, Remediation, and Hardening

1/19/2021 FireEye: Remediation and Hardening Strategies for Microsoft 365 to Defend Against UNC2452
12/28/2020 Microsoft: Using Microsoft 365 Defender to protect against Solorigate
12/22/2020 Microsoft: Protecting Microsoft 365 from on-premises attacks
12/22/2020 Microsoft: Azure Active Directory Workbook to Assess Solorigate Risk
12/21/2020 Microsoft: Advice for incident responders on recovery from systemic identity compromises
12/21/2020 FireEye (GitHub): FireEye Mandiant SunBurst Countermeasures
12/16/2020 Microsoft: SolarWinds Post-Compromise Hunting with Azure Sentinel
10/28/2020 Trimarc: Securing Microsoft Azure AD Connect
8/9/2018 Microsoft: AD Forest Recovery - Resetting the krbtgt Password
2/18/2016 CrowdStrike: Investigating PowerShell: Command and Script Logging
4/8/2015 FireEye: Windows Management Instrumentation (WMI) Offense, Defense, and Forensics

 

Technical and Investigation Information from SolarWinds

2/24/2021 FAQ: Security Advisory Note: latest update was 2/24/2021.
1/19/2021 CISA/CERT Upgrading Your Environment Note: latest update was 1/19/2021.
1/11/2021 New Findings from Our Investigation of SUNBURST
12/17/2020 SolarWinds Security Advisory
N/A Secure Configuration for the Orion Platform

 

Detection Tools

N/A CISA: CHIRP
N/A CISA: Sparrow
N/A

CrowdStrike:

N/A FireEye Mandiant: Azure AD Investigator
N/A Microsoft:

 

Note: The information you have accessed or received is being provided “as is” for informational purposes only. DHS and CISA do not endorse any commercial product or service, including any subjects of analysis. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation, or favoring by DHS or CISA.