PRCCDC 2019 Retrospective

I’ve documented my PRCCDC experience.

The Competition

The National Collegiate Cyber Defense Competition (CCDC) is a competition in which multiple college teams act as blue teams and are dropped into an unfamiliar network in the midst of a cyberattack. This year’s PRCCDC regional event was my first time competing.

TLDR: Team gets along well. I was impressed by everyone’s teamwork and morale. Some basic problems ended up being significant time sinks for us. We could’ve written more scripts and located configuration templates ahead of time to speed up system hardening.

Skip to the bottom of the post if you just want to see my list of lessons learned. You’ll miss all the fun red team stories though.

Competition Recap

Day One

Azcatraz: a prison for wizards?

During day one, I and some of my teammates got our very first taste of CCDC. For the first hour, we were plagued with non-red team related network issues. Our first two business injects were delivered via word of mouth from the white team (our team’s judges and answerers of general questions). Not until later did I realize that the red team had achieved some level of access to our network’s only domain controller by 29 minutes into the game. At least, that’s the timestamp I saw along with the first evidence of their execution of some code that caused an unexpected crash and memory dump. By 2 hours or so into the competition, it became clear that we had been soundly owned by the red team and we decided to cut our losses and restore the domain controller to its initial state.

Passwords

I think our password strategy was pretty solid, though we discovered a frustrating flaw in it. One of my teammates generated several hundred strong passwords, numbered them, and provided space to write descriptions. We were each assigned a block of passwords. When someone asked for an account password, we told them the corresponding number. This was a sensible system and was not painful to use.

As it turns out, MySQL, like many other technologies, has special characters that must be escaped if they are to be used in strings. You might be able to see where I’m going with this. Our teammate working with the MySQL database experienced significant issues using one of the pre-generated passwords as a database user password. We had a few different issues over the course of the competition stemming from passwords/password management; however, it was really just a couple of edge cases causing problems.

Not your mom’s CCDC!

Thinking back, we may have focused too much on service uptime and completing business injects and too little time on hardening during the first few hours of competition. We communicated well as a team but we didn’t go into day one expecting the red team to come in hot. We were a bit taken aback when we realized they were going for the throat from the beginning!

Cat pics

We moved through day one trying to restore services and harden servers opportunistically. We got our domain owned again and this time the red team defaced our company web site by dumping usernames and password hashes of every account on our domain on it along with an assortment of cat pictures. We scrambled to pick up the pieces and try to stop any further damage from occurring.

A plan

By the end of the day, we were exhausted but morale remained high. That night we discussed some strategy for day two and decided that we should attempt to lock down the network perimeter as much as we could as early as we could (even at the cost of some service uptime) and harden servers methodically before allowing more traffic in/out of the network.

Plans have a funny way of not working out.

Day Two

Business is hard

Both days I noticed myself getting sidetracked from the hardening tasks I wanted to accomplish due to business injects, random issue troubleshooting, etc. Early in day two, our team captain was repeatedly rushed into meetings with executives and yelled at by the CISO. We had a few unhappy customers on the phone complaining about the website being down or other services not working…we also had some random person trying to sell us firewalls!

A different type of open door policy

Speaking of firewalls…I ended up spending some time on day two working with my teammate who had been focused primarily on the Palo Alto firewall. While we worked on hardening the newly-restored firewall management server, we went over the firewall policies. Very quickly, I noticed a policy named “Alohomora,” which means “open the door” in Harry Potter wizarding speak. I didn’t know that but I did call attention to this rule because it was allowing any/any traffic in to a specific host on the network and I wanted to know more. Firewall person had a “d’oh!” moment and then we both had a laugh as we removed that rule.

Everything was going great (besides the chaos of the competition, business injects, and hilariously frustrating help desk phone service). During hardening of the firewall management server, we went through the motions of creating an additional backup user in addition to the changing the password of the primary admin user and removing administrative users that weren’t documented. We cleaned up the firewall policies and prepared to lock down the perimeter and attempt to kick the attackers out of our network. We kicked off one last restart of the management server to allow updates to complete. At this point, I was shifting focus to the domain controller as my teammate was going to take over firewall stuff.

Presto chango

Then it happened. After the restart, none of the credentials worked! I am honestly still not sure what happened there as our red team person told us after the fact that they hadn’t really been messing with the firewall…?

Negotiating with dementors

We submitted incident reports left and right as the red team became more and more brazen. Servers were crashing, services were going down, we were progressively losing access to our jump boxes, our antagonists were leaving us messages on screen, and at one point I had a full-on battle for control of my mouse. It was pretty silly. At one point, our adversaries sent a very young red team member into the room who had been told to plug something in to a computer in the room. They didn’t get far.

One thing we wrongly assumed is that our workspaces, the jump boxes we RDP-ed into to get into the network, weren’t in scope of the red team’s assault. During our initial debrief, we discovered that they totally were in scope and were totally owned by the red team.

We got to a certain point where we decided that we’d do our best and just have fun with it–there was no point in being negative or upset. At one point we recited a poem to our red team antagonist over the phone in exchange for a password we needed! The competition ended, rather appropriately, with a wizard prison riot.

Red meets blue

At the end of day two, we met with the white team and red team for a quick debrief. A few interesting notes from that discussion:

• Jump boxes were definitely owned.
• We could have annoyed our attackers and forced them to establish more persistent footholds by restarting servers periodically.
• During the last hour or two of the game, the red team got louder and bolder because the rules on their side were shifted to allow pretty much anything toward the end.

Lessons Learned

Strategy

• Assume that the red team will go for the kill at the first opportunity.
• Proceed as if things will be the worst and strategize accordingly.
• Develop a back up plan and practice the skills you’ll need to do it! (for example, if the perimeter firewall is hosed, can your team rapidly configure host-based firewalls to help fill that gap?)
• Decide which methods the team will use to automate patching and hardening tasks and practice them well before the competition.
• Figure out low cost (time and energy) ways to annoy the red team (like a script that restarts servers randomly sorta like Chaos Monkey).

Password Management

• Ensure that you have a method for managing your team’s passwords that isn’t based in software.
• Be aware of common reserved characters and avoid using them in pre-generated passwords.

System Administration

• Lock down the network perimeter then patch and harden endpoints.
• Review existing policies, roles, and user accounts and determine if they are overly permissive or if they simply should not exist. Lots of systems were configured insecurely or had backdoor admin users.
• Disable privileged accounts if your team isn’t in control of them.
• Regarding firewall configurations: my teammate discovered after the competition that it’s easy to download pre-built templates so that firewall configuration becomes a simple low-hanging fruit win instead of a long, drawn out headache. This could likely be applied to other services too.
• Disable unnecessary features. Disabling PowerShell in as many places as possible (except where needed obviously) would’ve been a good thing as I did find evidence of the red team using PowerShell against us.

Monitoring

• Sysinternals tools like TCPview and Process Explorer are really good for pinpointing which processes are generating network traffic (and the destinations of course), making registry changes, etc.
• Even if centralized logging isn’t possible, log events provide really good data about what’s happening on a system. At the very least, you can use what you find to get some points for filing well-documented incident reports.

All in all, it was super fun and I am so glad I took the time to compete! I really wish I could participate next year but I’m graduating so I won’t be eligible.

If you are in college and have an opportunity to participate, I highly recommend it. I learned a lot and most importantly it was fun!