Unpacking NSIS-based Crypter – step by step

Nowadays we can encounter many malware samples packed by a crypter using installer scripts. We can distinguish them by a NSIS tag on Virus Total:

Often, (but not always) they come with a standard NSIS icon:

In this tutorial, I will show how to approach static decryption of such packages.

UPDATE: See also and example of unpacking a similar crypter in a dynamic way, using memory dumping: https://www.youtube.com/watch?v=4PM2TPyAqzA

Analyzed samples

1. d0f6cfb56f9b23eedce0f2ac30233fd1
2. 17fcd7a7162298225b06d85d1d5a90ea

Steps

1. Decompress the package – you can use 7zip under Windows or a standard archive manager under Linux
2. Find a DLL and the exported function, that will be used for unpacking. See the referenced strings – you will get names of the files that are opened.
3. Find those files. One of them will be your searched payload (encrypted by a simple XOR-based algorithm)
4. Try to find the key and the encryption algorithm (XOR based). In some cases, it is not a pure XOR, but usually you can figure out the modifications by looking at the output. In some (rare) cases, you will need to analyze the function in the DLL, that is used for unpacking, to find out the correct algorithm.

Details

1. Decompress

After decompressing the executable you will see two directories – one of them contains standard NSIS elements (just skip it, nothing to see here):

The other directory contains the stuff of our interest. In the best case [1], no rubbish is added and we can easily see the  files containing valuable data:

But often it looks like this [2] (junk files were added for the purpose of obfuscation), and we need to figure out which of the files contain the payload:

2. Find the DLL in this directory and see what it exports. Usually it is just one function, responsible for unpacking the payload. Pay attention on the function’s name and RVA of it’s code:

Example 1:

The unpacking function is named: Orchil and its code starts at RVA 0x1000

Example 2:

The unpacking function is named: Simile and its code starts at RVA 0x106E.

Enter in the function (using your favorite debugger/disassembler) and see which of the files it uses (it is especially helpful if there are junk files stored in the same directory). Example – using OllyDbg:

Example 1:

Example 2:

3. Find the referenced files in the same directory where the DLL is:

Look at their sizes. The smaller one contains encrypted functions that are loaded and used during payload injection. The bigger one contains encrypted payload. Copy the payload file aside. (In the case 2 the payload file is Postfix.3 – in the case 1 – ShopDemise.RkT.

4. Try to crack the payload

This type of crypter may use a simple XOR as well as some more complex XOR-based algorithms. Start from checking the simplest cases. If the typical algorithms does not work, you will need to analyze the unpacking function from the DLL. I already did analysis of some popular variant (read more here) and prepared a helper script:  nsisdec.py

Sometimes it is also beneficial to see the visualization of the encrypted files, so that we can guess what type of the encryption to expect. Left – Postfix.3 right – ShopDemise.RkT. In both cases we can see some patterns, like padding, that can be inside the PE file. Both visualizations suggest weak encryption – but in case of Postfix.3 it seems to be simpler.

Example [2] – file: Postfix.3
For the simplest case – XOR the payload file with any valid PE file. This trick uses the fact that all the PE files have similar headers plus the property of XOR function (applying it twice reverse the effect) – so, if we see repeating patterns at the beginning, it may be the key.

For simple XOR you can use the script dexor.py – as well as nsisdec.py in a default, 0 mode.

./nsisdec.py --file Postfix.3 --keyfile IP.dll > out.bin

Output:

00000000  6e 6f 74 6f 6f 74 6f 6f  74 6f 6f 74 6f 6f 74 6f  |notootootootooto|
00000010  6f 74 6f 6f 74 6f 6f 74  6f 6f 74 6f 6f 74 6f 6f  |otootootootootoo|
00000020  74 6f 6f 74 6f 6f 74 6f  6f 74 6f 6f 74 6f 6f 74  |tootootootootoot|
00000030  6f 6f 74 6f 6f 74 6f 6f  74 6f 6f 74 57 6f 74 6f  |ootootootootWoto|
00000040  6f 74 6f 6f 74 6f 6f 74  6f 6f 74 6f 6f 74 6f 6f  |otootootootootoo|
00000050  74 6f 6f 74 6f 6f 74 6f  6f 74 6f 6f 74 6f 6f 74  |tootootootootoot|

Seeing this, we may suspect that the key is “oot” (first character of the file is modified).
Let’s try:

./nsisdec.py --file Postfix.3 --key "oot" > out.bin

Output:

00000000  4c 5a 90 00 03 00 00 00  04 00 00 00 ff ff 00 00  |LZ..............|
00000010  b8 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 e8 00 00 00  |................|
00000040  0e 1f ba 0e 00 b4 09 cd  21 b8 01 4c cd 21 54 68  |........!..L.!Th|
00000050  69 73 20 70 72 6f 67 72  61 6d 20 63 61 6e 6e 6f  |is program canno|
00000060  74 20 62 65 20 72 75 6e  20 69 6e 20 44 4f 53 20  |t be run in DOS |
00000070  6d 6f 64 65 2e 0d 0d 0a  24 00 00 00 00 00 00 00  |mode....$.......|

At the beginning we can see that the first character is invalid, but the rest of the file looks like a perfectly normal PE file. Let’s change the “LZ” to the valid “MZ” header…
So, indeed we got a valid PE file (it’s Cerber ransomware, you can see it here).

Example [1] , file: ShopDemise.RkT

In this case after the XOR operation we cannot see the familiar patterns. It means, this sample must be handled differently. We will try another common algorithm used by this type of crypter. But let’s try to find the key first. Usually it is hardcoded in the binary and not obfuscated. Seeing all the strings and checking in the code where they are used, we can suspect that the key for this file is: “CaratSerinIntermittency”

Ok, let’s apply it along with another algorithm commonly used by this crypter (mode 2):

./nsisdec.py --file ShopDemise.RkT --key "CaratSerinIntermittency" --mode 2 > out2.bin

Output:

00000000  58 5a 90 00 03 00 00 00  04 00 00 00 ff ff 00 00  |XZ..............|
00000010  b8 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 e8 00 00 00  |................|
00000040  0e 1f ba 0e 00 b4 09 cd  21 b8 01 4c cd 21 54 68  |........!..L.!Th|
00000050  69 73 20 70 72 6f 67 72  61 6d 20 63 61 6e 6e 6f  |is program canno|
00000060  74 20 62 65 20 72 75 6e  20 69 6e 20 44 4f 53 20  |t be run in DOS |
00000070  6d 6f 64 65 2e 0d 0d 0a  24 00 00 00 00 00 00 00  |mode....$.......|

Again, we see a PE file with first character invalid. Let’s change the “XZ” to the valid “MZ” header… It works! Again we got a valid executable (in this case it is also Cerber, you can see it here).

Ending note

Ok, we unpacked the payloads  – and what was in the remaining files? You can uncover their content using their name as the key and the dedicated algorithm, that you can find in the same script (mode 1). Usual key length is 10. Example:

./nsisdec.py --file NardooDeposal.W --key NardooDeposal.W --maxkey 10 --mode 1 > out.bin

They contain functions that are going to be loaded, some junk for obfuscation and the code that is used for injecting payload to the remote process. Example (beginning of the decoded file):

00000000  43 72 65 61 74 65 50 72  6f 63 65 73 73 41 0a 4e  |CreateProcessA.N|
00000010  74 55 6e 6d 61 70 56 69  65 77 4f 66 53 65 63 74  |tUnmapViewOfSect|
00000020  69 6f 6e 0a 56 69 72 74  75 61 6c 41 6c 6c 6f 63  |ion.VirtualAlloc|
00000030  45 78 0a 56 69 72 74 75  61 6c 41 6c 6c 6f 63 0a  |Ex.VirtualAlloc.|
00000040  57 72 69 74 65 50 72 6f  63 65 73 73 4d 65 6d 6f  |WriteProcessMemo|
00000050  72 79 0a 47 65 74 54 68  72 65 61 64 43 6f 6e 74  |ry.GetThreadCont|
00000060  65 78 74 0a 53 65 74 54  68 72 65 61 64 43 6f 6e  |ext.SetThreadCon|
00000070  74 65 78 74 0a 52 65 73  75 6d 65 54 68 72 65 61  |text.ResumeThrea|
00000080  64 0a 47 65 74 46 69 6c  65 53 69 7a 65 0a 52 65  |d.GetFileSize.Re|
00000090  61 64 50 72 6f 63 65 73  73 4d 65 6d 6f 72 79 0a  |adProcessMemory.|
000000a0  6e 74 64 6c 6c 2e 64 6c  6c 0a 4c 6f 63 61 6c 41  |ntdll.dll.LocalA|
000000b0  6c 6c 6f 63 0a 53 6c 65  65 70 0a 47 65 74 4d 6f  |lloc.Sleep.GetMo|
000000c0  64 75 6c 65 46 69 6c 65  4e 61 6d 65 41 0a 47 65  |duleFileNameA.Ge|
000000d0  74 43 75 72 73 6f 72 50  6f 73 0a 4e 74 52 65 73  |tCursorPos.NtRes|
000000e0  75 6d 65 54 68 72 65 61  64 0a 75 73 65 72 33 32  |umeThread.user32|
000000f0  0a 6c 73 74 72 63 61 74  41 0a 45 78 69 74 50 72  |.lstrcatA.ExitPr|
00000100  6f 63 65 73 73 0a 47 65  74 43 6f 6d 6d 61 6e 64  |ocess.GetCommand|
00000110  4c 69 6e 65 41 73 77 65  61 74 69 6e 65 73 73 0a  |LineAsweatiness.|
00000120  6b 61 6e 73 0a 67 6f 6f  64 66 65 6c 6c 6f 77 73  |kans.goodfellows|
00000130  68 69 70 0a 69 73 61 74  69 6e 0a 66 75 6e 67 69  |hip.isatin.fungi|
00000140  62 6c 65 73 0a 63 61 72  74 6f 6e 70 69 65 72 72  |bles.cartonpierr|

That’s all!

Read also Part 2

Appendix

Helper script: https://github.com/hasherezade/malware_analysis/blob/master/nsisdec.py

Read also about manual unpacking of this crypter in my post written for Malwarebytes

http://nsis.sourceforge.net – NSIS homepage