Write up Reverse Engineering IDSECCONF CTF 2022

Maulvi Alfansuri
10 min readNov 8, 2022


Idsecconf CTF 2022 is CTF competition held by IDSECCONF Conference and is part of #IDSECCONF2022 as a security conference. I participated with a teams name kaela_simp and finished in 4th position when the competition ended. I managed to solve 4 reverse engineering challenges and 1 web challenge. All of the challenges can be downloaded on my github here. Here is a write-up of reverse engineering challenges that I solved in this CTF.

CTF scoreboard

Table of contents

· InjinirPemula (100 Points / 56 solved)
· NontonSerial (200 Points / 5 solved)
· AnggapAjaPiem (250 Points / 5 solved)
· Pagarin (200 Points / 4 solved)
· Conclusion

InjinirPemula (100 Points / 56 solved)

We are provided with ELF Binary with a simple 64-bit binary.

Binary spec

Execute the binary would show output like this

Run the binary would ask correct argument

Hmm, argument checking, let's view the code on the decompiler.

Check the string and we would see strings “Gut Lakk” and it references to main function. Main function had argv pass on the check function.

Main function

View the decompiled function of the check function would show output like this

Anti decompiler instruction exist to broke decompilation

There is some code that breaks the decompiler. Let's see function disassembly.

disassembly code

Disassembly shows there are analysis errors on the assembly and we see that function is not created on the checker code. Checker code also contains hex that supposed to be flag character. Lets created the function.

Create function manuallly

Now we see better-decompiled code. Extract the char and we got the flag

Flag compared per character

NontonSerial (200 Points / 5 solved)

On this challenge, we introduced with ELF binary with a strange machine type TI msp430.

FIle property

A few popular decompilers sometimes had a few different behaviors when dealing with strange ELF like this. But in this case, I used Ghidra because Ghidra had many architectures and machine types to handle this scenario. Load the binary on the Ghidra and select language TI MSP430.

Importing binary to ghidra and manually selecting a language

After completing the analysis, we try to inspect the function that handles the string or checker validation and found this function.

Program function that contains wrong flag string statement

There are wrong statement strings stated on the function. On line 35, there are xor operations on the static address 0xfebe. That address is supposed to be encrypted data and should be decrypted after we got the right check validation. Extract the variable and do xor the value with 0xcc

This data xor with 0xcc

Below is implementation code.

Script to xor the data
Run the script would show the flag

Run the code and the flag should acquire.

AnggapAjaPiem (250 Points / 5 solved)

On this challenge, we are provided with windows binary this time.

File property

Run the .exe files would show a window form that would check your input.

Program windows show input form

Let's open the binary on the decompiler to further analysis.

On the binary, this is the dialog function that controls the window.

Function main

There are function checker sub_4012D0. Function checker gets param from GetDlgItemTextA that had return value user input string length. Let's check the function checker.

Function checker that has been renamed

On the function checker, there our input is processed by some kind of calculation. After analyzing functions one by one, there are a few functions that do the arithmetic calculations, we renamed the function with name adder, multiplication, division, modulus, and subtract.

On the function checker also exist function 1 and function 2 that combined arithmetic functions. To get the flag, we should return the value 0 on this checker function.

We know that this function is processing our character flags one by one to calculate, and after replicating function 2 on the self c code, we know that the function2 return value would always have returned 0 value if the first parameter had 0 value. On this assumption, the flag is supposed to be shown up if the value of v12 always had 0 value on all character flags processed. If this scenario succeeds, we would get the flag if we brute force the flag character one by one.

However, We still did not know how long our flag length was and the value of v6. To get the value of v6, I try to debug the process and found out the value is 0x42 and never changed on every step.

Check value v6 from the debugger

Below is my code implementation of the brute force. In a simple way, I try to brute-force string length and brute-force flag per character, if return value of function2 is 0 and the flag character is printable, show the character to the output.


Compile the code and run, we should get the flag on flag length 54.

Run the compiled code, and the flag would show on the output

Pagarin (200 Points / 4 solved)

On this challenge, we provided with Linux binary. If we run the binary would close immediately.

File property and run the program that immediately closed

Opening the binary on the decompiler would show interesting output.

Below is the code of the main function.

Main function

Main function would check our env PAGARNYA_GAN, and hash the string value, and compare to memory on &unk_30E0.

Memory that compared in the function

Extract this memory, and search on google, we should find this md5 hash of string “true” would return the same hash.

Look up the memory would show reverse md5 look up

Let's set our env and run the binary again.

Run the binary with env would ask the input

Now binary actually asks the input, if we check again the main function we should know why this binary had some strange behavior.

There are strange function BUG on the main function

On the code on line 19, if we already pass the first env comparison, we should execute a function named BUG with decompiler.

If we look at the disassembly graph, we should find our instruction code is ud2.

UD2 Instruction code exist as function named BUG

If we look up the assembly documentation, the ud2 instruction is used to generate an invalid opcode for testing purposes. On this binary ud2 is used as anti-decompiler protection.

If we debug this execution flow, after the program executes instruction ud2, the program would jump to function sub_1A28. Below is the code of this function

Handler function that called after ud2 executed

If we analyze this function further, we know this function is declared by the init function below.

Init param that set handler function exception

The Init function would set an exception and set function sub_1A28 as an exception handler. So if we execute ud2 as an invalid opcode, function sub_1A28 would be executed.

Let's view the function sub_1A28 again. On line 10, we view variable v4 assigned with some pointer from parameter 3. If we check this value on the debugger, we should see this value is the address of ud2 code before we entered the exception. So we can assume this value is used as some kind of saved instruction pointer.

Instruction code before ud2 executed on 0x555555555fb6
rax contains 0x555555555fb6 and the address added with 2

If we check the handler function again, we know that this v4 is assigned with this pointer + 2, and this value is used as a comparison. There are 3 block codes that are executed.

block code executed based on the ud2 address+2

The red code would be executed if our value is 3, the green code is executed if our value is 2 and the blue code would be executed if we got the value 1. After our block code is executed, we found out that our a3 + 168 is added with value 3. And after we debug this on the debugger, we found out that the application back to the instruction after we called ud2 and add with value + 3.

We know that there are few ud2 codes, so we created a simple script to get all ud2 codes, and get value passed to the handler function.

UD2 identifier script
Left side is ud2 address and right side is ud2 address+2 value

After running the script, we got a few addresses that contained ud2 instruction and the passed value. Let's view 0x5555555558d0 which contains value 2.

Function that had value 2 on the ud2+2 code

If we view this function, we know that this function contains ud2 code, there is a BUG function on the decompiled view.

On the previous analysis, we know that our code would return to ud2 code + 2 address after we entered the exception. Let's patch the code to view what code we executed after we returned to this function. We patch this three bytecode

Patch this code to view after exception code

After patching this bytecode, we now should view all the code. Don't forget to patch other function bytecode too, below is the main function code that also contains ud2. We know that main function code after ud2 is executed, would assign some global array parameter that we would used later.

Main function had initial assignment of array paramsatu, paramdua and compariso1

After we done with main function, we should move to function sub_17ce.

Function sub_17ce

Let's rename this function and the handler function to make the code easier to read.

Function sub17ce renamed. Red code executed before exception, green code inserted in the loop executed in the exception and blue code executed after exception
Green code is inserted into the loop. This code exists on the handler function

On the function we renamed functionaneh, we found out that our code is split into two sections. Red block code would be executed first. After that, the exception is raised with ud2 instruction, and the code would go to green block on the exception handler. After green block was executed, the function returned and executed blue code.

After we know the execution flow, now we should find out what the code doing.

On the red code, the program would check our input to make sure our input only contains characters 1, 2, or 3. The application also make sure our input had length 310 character. After that, every two characters of our input is processed with green block code that also process array paramsatu and paramdua.

After all character pairs are processed, array paramsatu is compared with array name compariso1 that contains 4e,4e,0 that already initialized on the main function. If the comparison is true, we should get our input hashed with sha256 and the value xor with global variable aJ

Xor hash of our input with global array aJ
Value for array aJ

Now how do we get the correct input? Bruteforce the character did not seems feasible (3 ** 310) and the key to xor encrypted value is the hash of our input. To analyze this thing easier, I try to duplicate code to C to analyze every parameter.

Simulation of the function to trace each parameter value after pair input

After trying the simulation, I know that every pair of our inputs would process paramsatu. In the example, if we try to input 12, paramsatu would change from 9c,0,0 to 49,53,0

Pair input after input 12

If we input 21, we should view paramsatu changed from 49,53,0 back to 9c,0,0

Pair input after input 21

Some value would come back to the value that we already visited. This problem can be mapped as a simple graph below.

Problem mapped as a graph

After I know that this problem can be represented with a graph problem, I had the idea to solve this problem using BFS (breadth first search) to find the shortest path to make sure paramsatu had value 4e,4e,0. If we found the parameter input had a value that already exists, stops the traverse for the node. Our success input would be represented as traverse path from the beginning node that had value 9c,0,0 to final found node that had value 4e,4e,0. Below is the implementation of the code.

BFS code to bruteforce path

Run the script, and we should find the traverse output had a length 310, a length that exactly the program wants.

Run the BFSode, which would return a string that had a length of 310

Let's input this code

input the code would return the flag

And we got the flag that appeared on the output


I had so much fun doing reverse engineering challenges from this CTF, especially with challenge “Pagarin”, which contains anti-decompiler, anti-debugging techniques that trap me for an hour, and had some algorithm stuff like BFS, kindly rare on the local CTF used this approach as solving method. Congratulations to the winner and thanks to the problem setter who already provided us with good CTF.