Remediating Networks Affected by the SolarWinds and Active Directory/M365 Compromise

Threat Actor Activity

Threat Actor Activity

In December 2020, CISA was made aware of a supply chain compromise of certain versions of the SolarWinds Orion platform. An APT actor added malicious code to multiple versions of SolarWinds Orion and, in some instances, leveraged it for initial access to enterprise networks of U.S. government agencies, critical infrastructure entities, and private sector organizations.

Through incident response, CISA determined that, in other instances, the threat actor obtained initial access by password guessing, password spraying, and exploiting inappropriately secured administrative credentials via remote services.

Once inside the network, the threat actor bypassed multi-factor authentication (MFA) and moved laterally to Microsoft cloud systems by compromising federated identity solutions. The threat actor:

• Stole the Active Directory Federation Service (ADFS) token-signing certificate to forge tokens. By using this technique, referred to as “Golden SAML” (Security Assertion Markup Language), the threat actor was able to move laterally to M365 environments by authenticating into the federated resource provider, bypassing MFA and password requirements.
• Modified or added trusted domains in Azure AD. By using this technique, the threat actor was able to move laterally to Azure AD environments by adding new federated identity providers (iDPs). (See FireEye White Paper: Remediation and Hardening Strategies for Microsoft 365 to Defend Against UNC2452.)

After gaining access to cloud environments, the actor established persistence mechanisms for Application Programming Interface (API)-based access and collected and exfiltrated data.

The threat actor has demonstrated sophisticated defense evasion skills. The actor:

• Hid their command and control (C2) communications with extensive obfuscation,
• Hid their activity among legitimate user traffic, and
• Established difficult-to-detect persistence mechanisms (e.g., in API).

Note: for more information on this activity, including TTPs, refer to CISA Activity Alerts:

• AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations, and
• AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments.

Introduction: Mitigating and Remediating Malicious Activity

Introduction: Mitigating and Remediating Malicious Activity

Note: this remediation guidance applies to organizations that use SolarWinds Orion. CISA is aware of other initial access vectors—agencies should not assume they are not compromised by this actor solely because they have never used affected versions of SolarWinds Orion. Those agencies should investigate to confirm they have not observed related threat actor TTPs. If related activity is detected, agencies may find this guidance helpful. Those agencies are encouraged to review Joint Advisory AA20-245A: Technical Approaches to Uncovering and Remediating Malicious Activity and to contact CISA for further assistance.

Because the actor moved laterally to multiple systems and established difficult-to-detect persistence mechanisms, their activity will be difficult to detect and eradicate. CISA has provided the following short- and intermediate-term steps agencies can take to detect, mitigate, and remediate this activity. By following the ordered steps, agencies will harden their networks  and prevent re-use of the TTPs. CISA will update this webpage with further guidance on specific actions as new information becomes available.

Conduct a Risk/Impact Assessment

Conduct a Risk/Impact Assessment

Networks with SolarWinds Orion products will generally fall into one of three categories. Agencies will need to conduct a risk/impact assessment to determine which category they fall into.

• Category 1 includes agency networks that do not have the identified malicious binary code on their network and can forensically confirm that the binary was never present on their systems. This includes networks that do not, and never did, use the affected versions of SolarWinds Orion products.
• Category 2 includes agency networks where the presence of the malicious binary has been identified—with or without beaconing to avsvmcloud[.]com.
• Category 3 includes agency networks that used affected versions of SolarWinds Orion and have evidence of follow-on threat actor activity, such as binary beaconing to avsvmcloud[.]com and secondary C2 activity to a separate domain or IP address (typically but not exclusively returned in avsvmcloud[.]com Canonical Name record [CNAME] responses).

Resources

• CISA Activity Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations
• CISA Emergency Directive (ED) 21-01: Mitigate SolarWinds Orion Code Compromise

Remediating Malicious Activity: Category 1 and 2 Organizations

Remediating Malicious Activity: Category 1 and 2 Organizations

Although unaffected by this incident, Category 1 organizations should work to maintain strong network posture and resilience. Refer to https://www.cisa.gov/cybersecurity and https://us-cert.cisa.gov/resources/federal for assistance. CISA recommends Category 1 organizations:

• Maintain up-to-date antivirus signatures and engines. See Protecting Against Malicious Code.
• Ensure systems have the latest security updates. See Understanding Patches and Software Updates.
• Enforce a strong password policy. See Choosing and Protecting Passwords.
• Exercise caution when opening email attachments, even if the attachment is expected and the sender appears to be known. See Using Caution with Email Attachments.
• Sign up to receive CISA’s alerts on security topics and threats.
• Sign up for CISA’s free vulnerability scanning and testing services to help organizations secure internet-facing systems from weak configuration and known vulnerabilities.
• Email vulnerability_info@cisa.dhs.gov to sign up. See https://www.cisa.gov/cyber-resource-hub for more information about vulnerability scanning and other CISA cybersecurity assessment services.

Category 2 organizations should continue enhanced monitoring for any possible follow-on adversary activity. Refer to resources below for more information.

Resources:

• CISA Activity Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations
• CISA Activity Alert AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments
• CISA Emergency Directive (ED) 21-01: Mitigate SolarWinds Orion Code Compromise

According to ED 21-01 and associated supplemental guidance, all federal agencies that ran affected versions of SolarWinds Orion must “conduct system memory, host storage, network, and cloud forensic analysis,” “hunt for indicators of compromise (IOCs) or other evidence of threat actor activity, such as secondary actions on objectives (AOO),” and “[i]dentify and remove all threat actor-controlled accounts and identified persistence mechanisms.”

Remediating Malicious Activity: Category 3 Organizations

Remediating Malicious Activity: Category 3 Organizations

Note: Category 3 organizations should use out-of-band communications for all mitigation and remediation documentation and conversations, i.e., do not use any compromised systems to communicate remediation plans or actions.

For Category 3 organizations, completing all the steps provided in this guidance is necessary to fully accomplish eviction. (Note: CISA will release detailed information on eviction steps to agencies via the Homeland Security Information Network [HSIN]). These are resource-intensive and highly complex and will require the enterprise network to be disconnected from the internet for 3–5 days; however, failure to perform a comprehensive and thorough remediation will expose enterprise networks and cloud environments to substantial risk of long-term undetected APT activity, including email monitoring and data collection and exfiltration. CISA recommends organization leadership read the CISA Insights, Remediating Networks Affected by the SolarWinds and Active Directory/M365 Compromise: Risk Decisions for Leaders, for more information.

Remediation plans for dealing with malicious compromises are necessarily unique to every organization, and success requires careful consideration. There are three phases for evicting the actor:

• Phase 1: Pre-Eviction. Actions to detect and identify APT activity and prepare the network for eviction.
• Phase 2: Eviction. Actions to remove the APT actor from on-premises and cloud environments. This phase includes rebuilding devices and systems.
• Phase 3: Post-Eviction. Actions to ensure eviction was successful and the network has good cyber posture.

Phase 1: Pre-Eviction

Define the True Scope

• Action 1: Identify trust boundaries and determine the enterprise assets to which this guidance applies (i.e., determine what assets are within the trust boundary).

Resources:

• CISA Emergency Directive (ED) 21-01: Mitigate SolarWinds Orion Code Compromise

Investigate Suspicious Account Activity

• Action 1: Investigate suspicious account activity associated with SolarWinds Orion servers, especially service accounts used by SolarWinds Orion.
• Action 2: Enumerate and investigate any credentials stored or used on the SolarWinds server, including network administration and device credentials. If a Category 3 organization cannot fully confirm all activity of these credentials is benign, they should proceed as if the highest administrative level of credentials on their affected Category 3 SolarWinds server have been compromised. In many cases, the adversary may have had months with this access.

Resources:

• FireEye Threat Research Blog: Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor
• CISA Activity Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations

Investigate Potential SAML Abuse

• Action 1: Investigate attacks on identity sources, such as SAML forgery. If the adversary has compromised administrator credentials in an environment—or if organizations identify SAML abuse in the environment—simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network. In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears emphasizing that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action. For many organizations, remediation from this level of compromise may require third-party assistance.

Resources:

• CISA Activity Alert AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments
• FireEye White Paper: Remediation and Hardening Strategies for Microsoft 365 to Defend Against UNC2452
• National Security Agency (NSA) Cybersecurity Advisory: Detecting Abuse of Authentication Mechanisms
• Microsoft Azure Active Directory Identity Blog: Understanding "Solorigate"'s Identity IOCs

Scope the Intrusion

• Action 1: Look for the artifacts from known TTPs associated with this activity. Refer to SolarWinds and Active Directory/M365 Compromise: Detecting APT Activity from Known Tactics, Techniques, and Procedures and corresponding detection artifacts. Prioritize these by the biggest value for investment (e.g., prioritize by techniques or technologies that cover multiple tactics or that provide visibility into shared data sources).
• Action 2: Audit all network device configurations stored or managed on the SolarWinds monitoring server for signs of unauthorized or malicious configuration changes.
• Action 3: Audit all network device configurations for signs of unauthorized or malicious configuration changes. Also be sure to audit the current network device running configuration and any local configurations that could be loaded at boot time.
• Action 4: Assess the current endpoint telemetry collection level and configure endpoint detection and response (EDR) or detection solution for aggressive collection; prioritize this by value of asset and account.

Resources:

• CISA Activity Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations
• CISA Activity Alert AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments

Harden the Enterprise Attack Surface

• Action 1: Review and validate perimeter firewall rulesets. Remove all rules the organization does not have a documented need for.
• Action 2: Implement host-based firewalls to make lateral movement more difficult for the actor.
• Action 3: Close and/or monitor high-risk ports (e.g., Remote Desktop Protocol [RDP], Server Message Block [SMB], File Transfer Protocol [FTP], Trivial File Transfer Protocol [TFTP], Secure Shell, and WebDAV).
• Action 4: Employ allowlisting, especially for systems providing remote access to the enterprise.
• Action 5: Enforce enterprise Domain Name System (DNS) resolution for all systems.

Identify Federation Model for On-Premises Resources to Cloud Trust Relationship and Identify Adversary Activity in M365/Azure Environment

• Action 1: Identify the Source Anchor and commensurate configuration for Azure AD Connect in the current Azure Tenant, if applicable.
• Action 2: Identify permission and credential changes to applications and service principals. Identify overly permissive applications, unusual credentials in applications, or modifications to federation trust settings. See CISA Activity Alert AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments for more information.
• Action 3: Review M365 tenant configuration and perform a risk assessment for administrative accounts and applications.

Resources:

• CISA Activity Alert AA21-008A: Detecting Post-Compromise Threat Activity in Microsoft Cloud Environments
• CISA Analysis Report AR21-013A: Strengthening Security Configurations to Defend Against Attackers Targeting Cloud Services
• FireEye White Paper: Remediation and Hardening Strategies for Microsoft 365 to Defend Against UNC2452

Phase 2: Eviction

To evict the actor from the network, agencies should take steps to regain sole control over their AD, remove malware implants from network and cloud systems, and rebuild or re-image network and cloud systems. CISA will release detailed information on eviction steps to agencies via HSIN.

Note: this phase will require agencies to disconnect their enterprise network from the internet for several days. Agencies should plan accordingly.

Resources:

• CISA Activity Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations
• FireEye White Paper: Remediation and Hardening Strategies for Microsoft 365 to Defend Against UNC2452

Phase 3: Post-Eviction

Report to CISA

Post-eviction, all Category 3 agencies should report actions taken to CISA, any actions left undone, and their assessments of the residual risk. CISA will release a checklist to agencies via HSIN.

Maintain Vigilance

From a security operations center (SOC) perspective, it is important to continue monitoring for malicious activity related to this campaign. Permanent eviction is complicated; the adversary may have been inside networks for a long period, so they likely understand various weaknesses, which they can attempt to leverage to regain network access in various forms. SOCs should continue monitoring for:

• Known TTPs associated with this activity, and
• Signs of persistence, such as C2 connections to new domains and attempts to run unusual code.

CISA also recommends that affected agencies implement behavior analysis and reduce traffic to the internet.

Resources: CISA, Federal Government, and International Partner Publications

Resources: CISA, Federal Government, and International Partner Publications

Note: The following publications focus on the SolarWinds Orion Compromise and Related Activity

Table 1: CISA, Federal Government, SLTT, and International Partners Publications

Table 2: Industry Publications

Note: The information you have accessed or received is being provided “as is” for informational purposes only. DHS and CISA do not endorse any commercial product or service, including any subjects of analysis. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation, or favoring by DHS or CISA.