Flare-On 9 – Task 9

For those of you who don’t know, Flare-On is an annual “reverse engineering marathon” organized by Mandiant (formerly by FireEye). It runs for 6 weeks, and contains usually 10-12 tasks of increasing difficulty. This year I completed as 103 (solves board here). In this short series you will find my solutions of the tasks I enjoyed the most.

Time for some crypto challenge:

You can find the package here: 09_encryptor.7z , password: flare

After unpacking the archive we see:

It is a 64-bit PE – “a ransomware”, plus a file encrypted by it, that needs to be recovered. So, we have an emulation of the ransomware decryption scenario.

I used to crack ransomware in the past, and I still find this kind of cryptoanalysis tasks very enjoyable. As usually in such cases, two algorithms are used:

  1. symmetric, to encrypt a file (with a random key)
  2. asymmetric, to protect the generated random key

A flaw can be in one of the following:

  • how the random key is generated (was the strong random generator used?)
  • how the symmetric encryption is implemented (any implementation flaws making it weaker?)
  • how the asymmetric encryption is implemented
  • finally: are the algorithms applied correctly?

The task is written in C, and the code is pretty small, and focused on the main goal, so the analysis is easy.

The file is encrypted with ChaCha:

This version of ChaCha uses 32 byte key, and 12 byte nonce. The implementation of ChaCha seems correct. Also, for the generation of the key and nonce, a strong random generator is used (SystemFunction036). So at this point my guess is that the bug must be somewhere around the asymmetric algorithm.

After the file is encrypted, the buffer containing the key and nonce is encrypted with a private key from a newly generated keypair.

So, the 4 hex strings that we see at the end of the file suppose to contain the following elements:

{RSA master public key - the hardcoded master public key}
{RSA generated public key - the public key from the generated keypair}
{RSA generated private key, protected by the RSA master public key}
{ChaCha key and nonce, protected by the RSA generated private key}

If everything is correct there, we need the RSA master private key, in order to decrypt the RSA generated private key, in order to decrypt the ChaCha key and nonce… Let’s take a closer look if it really is this way.

A good cheatsheet describing all the RSA building blocks is available here.

Snippet describing the parts related to RSA implementation:

int __fastcall encrypt_file_content_and_save_keys(FILE *out_file, FILE *in_file)
__int64 v4; // rcx
_DWORD *v5; // rdi
__int128 *_key; // rdi
__int64 i; // rcx
_QWORD key_out_buf[17]; // [rsp+20h] [rbp+0h] BYREF
__int128 key[2]; // [rsp+A8h] [rbp+88h] BYREF
__int128 nonce[9]; // [rsp+C8h] [rbp+A8h] BYREF
v4 = 34i64;
v5 = key_out_buf;
while ( v4 )
*v5++ = 0;
_key = key;
for ( i = 34i64; i; –i )
*(_DWORD *)_key = 0;
_key = (__int128 *)((char *)_key + 4);
SystemFunction036(key, 32u);
SystemFunction036((char *)nonce + 4, 12u);
chacha_encrypt(out_file, in_file, key, nonce);
protect_by_assymetric_crypt(key_out_buf, key, RSA_d, RSA_n);
print_in_hex_to_file(out_file, RSA_master_public_key);
putc(10, out_file);
print_in_hex_to_file(out_file, RSA_n);
putc(10, out_file);
print_in_hex_to_file(out_file, RSA_protected_gen_priv_key);
putc(10, out_file);
print_in_hex_to_file(out_file, key_out_buf); // protected ChaCha key
return putc(10, out_file);
__int64 init_stuff()
__int64 rsa_p[17]; // [rsp+30h] [rbp-348h] BYREF
__int64 rsa_q[17]; // [rsp+B8h] [rbp-2C0h] BYREF
__int64 buf1_sub1[17]; // [rsp+140h] [rbp-238h] BYREF
__int64 buf2_sub1[17]; // [rsp+1C8h] [rbp-1B0h] BYREF
__int64 rsa_euler[17]; // [rsp+250h] [rbp-128h] BYREF
char RSA_generated_private[160]; // [rsp+2D8h] [rbp-A0h] BYREF
while ( !(unsigned int)is_prime((unsigned __int64 *)rsa_p) );
while ( !(unsigned int)is_prime((unsigned __int64 *)rsa_q) );
bignum_mul(RSA_n, (unsigned __int64 *)rsa_p, (unsigned __int64 *)rsa_q);
calc_sub1((unsigned __int64 *)buf1_sub1, (unsigned __int64 *)rsa_p);
calc_sub1((unsigned __int64 *)buf2_sub1, (unsigned __int64 *)rsa_q);
bignum_mul(rsa_euler, (unsigned __int64 *)buf1_sub1, (unsigned __int64 *)buf2_sub1);
calculate_d(RSA_d, RSA_d, rsa_euler);
return protect_by_assymetric_crypt(
view raw notes.cpp hosted with ❤ by GitHub

I made a small loader for the original app, and hooked the functions with detours (loader.cpp), in order to quickly log all their input and output parameters. At some point, I noticed something very suspicious: instead of the generated private key being provided to encrypt the generated ChaCha key, what was passed was the standard public exponent! So, in reality is is RSA signing.

To recover the “encrypted” content, all we have to do is to use the exponent 10001 as a private key.

For solving the final equation, I used the following online tool: https://www.boxentriq.com/code-breaking/rsa

By looking at the output we can see that it is in the correct format of key and nonce. However, we still need to reverse the bytes before using.

Now in order to decode the file content, we can just rename the file to “.EncryptMe” and we can set a breakpoint after the key and nonce are generated, to replace them in memory.

And we get the original content decrypted:


The flag is:


About hasherezade

Programmer and researcher, interested in InfoSec.
This entry was posted in CrackMe and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s