The Joy of Low-Level Programming: Writing a Windows Game in x64 Assembly - Part 2
Sunday, 29 December 2024
In part one, I talked about programming for Windows using pure x64 Assembly. In this part, we are going deeper into coding.
Get hands-on with coding
To start, let's create the skeleton of the program and make sure that you have a working development environment. The program starts and peacefully terminates. This is all it does.
bits 64 ; We are going to target 64-bit Assembly
default rel ; Our addresses are offsets relative to rip no absolute values
global main ; Telling enverybody that the entry-point is "main"
; We are going to use ExitProcess to gracefully terminate the
; program but it resides in an external library (kernel32.lib)
extern ExitProcess
section .data
; All initialized variables
section .bss
; All uninitialized variables
section .text ; Our code starts from here
main:
; Setting up the stack frame (32 bytes for shadow space, 8 bytes for padding)
sub rsp, 40
mov rcx, 0 ; The first and only parameter of ExitProcess
; It will be the programm exit code
call ExitProcess
Save the code above to starter.asm, open the x64 Native Tools Command Prompt created by your Visual Studio installer, and use the following command to assemble the source code:
nasm starter.asm -f win64 -o starter.obj
Next, use the following command to generate the final executable:
link starter.obj /entry:main /subsystem:console /defaultlib:kernel32.lib /out:starter.exe /incremental:no
Without the x64 Native Tools Command Prompt, your system won't be able to find link.exe. In that case, you need to manually navigate to its folder to invoke it.
After linking, a new file called starter.exe will be generated in the current directory. This is a valid Windows executable. To test it, type the following command:
.\starter.exe && echo %errorlevel%
If you see 0, everything works as expected. The program is designed to return 0 as its exit code. If you prefer PowerShell, you can use .\starter.exe; $LASTEXITCODE instead.
Global varaibles and strings
Our program requires some global variables, most of which are string messages displayed throughout the game.
section .data
upper_bound equ 100_000
max_tries equ 20
newline db 0xd, 0xa
welcome_msg1 db 'Hello and welcome to "Guess Me". I have picked a number for you to guess between 1 and '
welcome_msg1_len equ $-welcome_msg1
welcome_msg2 db '.', 0xd, 0xa, 'You have '
welcome_msg2_len equ $-welcome_msg2
welcome_msg3 db ' guesses. Good luck :)'
welcome_msg3_len equ $-welcome_msg3
ask_guess_msg1 db 'Enter your guess (#'
ask_guess_msg1_len equ $-ask_guess_msg1
ask_guess_msg2 db '): '
ask_guess_msg2_len equ $-ask_guess_msg2
too_low_msg db ' too LOW.'
too_low_msg_len equ $-too_low_msg
too_high_msg db ' too HIGH.'
too_high_msg_len equ $-too_high_msg
nan_guess_msg db 'Enter a number please :('
nan_guess_msg_len equ $-nan_guess_msg
winner_msg db 'Congratulations! You made it :) :) :)'
winner_msg_len equ $-winner_msg
loser_msg db 'Ooooh Nooo it was not your day :( My number was '
loser_msg_len equ $-loser_msg
play_again_msg db 'Do you want to play again (Y)?'
play_again_msg_len equ $-play_again_msg
goodbye_msg db 'Goodby :)'
goodbye_msg_len equ $-goodbye_msg
no_rdrand_msg db 'Sorry but your CPU does not support the RDRAND instruction :('
no_rdrand_msg_len equ $-no_rdrand_msg
buffer_len equ 6
error_code equ 4_000_000
section .bss
buffer resb buffer_len ; Stores player's input
picked_number resq 1 ; the chosen number
current_try resw 1 ; your current try (1 to 20)
For each string message, we define its length as a constant, calculated at compile time using the $ token, which expands to the current location in the source code.
The first procedure is responsible for printing a string on the screen. For this task, we need to make use of two Win32 functions: - GetStdHandle that returns the handle to the standard output (the current console window) - WriteFile that writes a sequence of characters into a file specified by its handle
In Windows x64 calling convention, the first four integer parameters are passed to a function using rcx, rdx, r8, and r9 registers and other paramters are push on the stack, and the return value is must to be stored in rax register.
To obtain the standard output handle, we need:
mov rcx, -11 ; STD_OUTPUT_HANDLE
call GetStdHandle
and rax will hold the handle that we can use to pass to WriteFile:
mov rcx, rax ; file handle
lea rdx, [our string] ; the string
mov r8, our string len ; length
lea r9, [bytes] ; address of the variable to store
; the actual number of bytes we have
; read
call WriteFile
And here is the final print procedure. It takes three arguments: - rcx point to the string to print - rdx length of the string - r8 determines if a newline must be added at the end
The procedure does not return any value so rax is not set to any specific value (it might be used internally by the procedure though, but the caller should not expect any meaningful value in rax).
print:
; rcx: points to the string
; rdx: the length of the string
; r8 : 1=> add a newline
; Defining the stack frame
sub rsp, 72 ; 4 qword local variables (32)
; 1 qword stack parameter
; (the last parameter of WriteFile) (8)
; shadow space (32)
; Storing local variable on the stack
mov [rsp+64], r8 ; has newline?
mov [rsp+56], rdx ; string length
mov [rsp+48], rcx ; string
mov qword[rsp+40], 0 ; STD OUT handle
mov qword[rsp+32], 0 ; Stack parameter
; Getting the handle
mov rcx, -11 ; STD_OUTPUT_HANDLE
call GetStdHandle
mov [rsp+40], rax
; Printing the string
mov rcx, rax ; file handle
mov rdx, [rsp+48] ; string
mov r8, [rsp+56] ; length
mov r9, 0 ; we don't need to know how many
; bytes are written
call WriteFile
cmp qword[rsp+64], 1 ; is newline needed?
jne print_exit
; Printing a newline if specified
mov rcx, [rsp+40] ; file handle
lea rdx, [newline] ; string
mov r8, 2 ; length
mov r9, 0 ; we don't need to know how many
; bytes are written
call WriteFile
print_exit:
add rsp, 72 ; Cleaning up the stack frame
ret
Random Generator
It is obvious that the program must generate random numbers. To achieve this, we will use the rdrand instruction available on x86-64 CPUs. This procedure does not accept any parameters, but returns a value in rax — either a random number or an error.
rand:
; Using RDRAND to return a random number. error_code is returned in the case of errors.
; Checking whether RDRAND is supported
mov eax, 1
mov ecx, 0
cpuid
shr ecx, 30
and ecx, 1
jz rand_no_rdrand
retry_rand:
xor rax, rax
rdrand eax
cmp eax, upper_bound
ja retry_rand
ret
rand_no_rdrand:
mov rax, error_code
ret
Integer to String - String to Integer
To print integers, they must first be converted to strings. Similarly, strings containing numeric values need to be converted back to integers. Below are two procedures that perform these operations:
int_to_str:
; rcx: holds the integer
; Returns number of digits
sub rsp, 8 ; 1 qword local variable (8)
mov [rsp], rcx ; clear_buffer manipulates rcx and we need it after it
mov rcx, 0
call clear_buffer
mov rax, [rsp]
mov rcx, 10 ; we'll keep dividing by 10 to extract all digits
xor rbx, rbx ; stores the number of digits temporarily
int_to_str_loop:
xor rdx, rdx ; rdx is not needed for this division operation
div rcx ; eax = Quotient, edx = Remainder
inc rbx ; one more digit was extracted
; filling the buffer
add rdx, 030h ; finding the ascii character
lea r8, [buffer]
add r8, buffer_len
sub r8, rbx
mov [r8], dl
cmp rax, 0
je int_to_str_exit
jmp int_to_str_loop
int_to_str_exit:
mov rcx, rbx
call align_buffer ; we've built the string from end to start so we need to shift it to the begining of the buffer
mov rax, rbx
add rsp, 8
ret
str_to_int:
; Converts the string inside the buffer to an integer
; Returns the integer or error_code if the string is not convertible
cmp byte[buffer], 0 ; do we have anything in buffer?
je str_to_int_error
lea rcx, [buffer]
xor rax, rax ; the result
mov r9, 10 ; we'll keep mutiplying by 10 to aggregate all digits
xor r8, r8 ; will hold each step character temporarily
mov rbx, 1 ; loop counter
str_to_int_loop:
mov r8b, [rcx+rbx-1] ; current digit
cmp r8b, 0 ; is the character string terminator?
je str_to_int_exit
cmp r8b, 030h ; character '0'
jb str_to_int_error
cmp r8b, 039h ; character '9'
ja str_to_int_error
sub r8b, 030h ; char to int conversion
mul r9
add rax, r8
inc rbx
cmp rbx, buffer_len ; have we reached the end of the buffer?
ja str_to_int_exit
jmp str_to_int_loop
str_to_int_error:
mov rax, error_code
str_to_int_exit:
ret
Reading the guess
Another important procedure reads an integer from the standard input (keyboard).
This time, we use GetStdHandle to get the standard input handle:
mov rcx, -10 ; STD_INPUT_HANDLE
call GetStdHandle
Then we will use ReadFile to read from the standard input.
Although the play can type as many characters as they want, we are only interested in the first 6 characters (because the computer's chosen number has at most 6 digits). Due to input buffering, this procedure becomes slightly complicated, as we need to read the characters we are interested in and flush the rest of the input buffer.
read_int:
; Tries to read an integer from the standard input and returns the integer value.
; error_code is returned if no integer convertible value is read.
; NOTE: The user input always ends with "\r\n" and it might contain more characters than buffer_len.
; As a result, we either need to remove the "\r\n" if it occurs inside "buffer", or flush the
; excess characters that did not fit in "buffer" but still reside in the standard input buffer as
; they will be read automatically by the next ReadFile invocation. ReadFile reads maximum of
; buffer_len bytes each time so the number of actual bytes read that gets reported by ReadFile,
; never surpasses buffer_len. The following pseudocode shows the whole logic where #OfBytesRead
; is the actual bytes read reported by ReadFile:
;
; if #OfBytesRead < buffer_len
; find the \r starting from the end
; clear_buffer from \r to the end
; call str_to_int
; else
; if buffer ends with \r\n
; clear_buffer from the second to the end byte
; call str_to_int
; else
; if the last byte == \r
; clear_buffer from the last byte
; call str_to_int
; flush stdin
;
; And it can be summarized to:
;
; flushStdin=false
; if #OfBytesRead == buffer_len // user has inputted either exactly buffer_len bytes or more
; if buffer last byte != \n // input length is greater than buffer_len, so StdIn buffer must be flushed
; flushStdin = true
; find the \r starting from the end
; clear_buffer from \r to the end
; if flushStdin == true
; flush stdin
sub rsp, 56 ; 2 qword local variables (16) + 1 qword stack parameter (the last parameter of ReadFile) (8) + shadow space (32)
mov qword[rsp+48], 0 ; local variable (StdIn handle)
mov qword[rsp+40], 0 ; local variable (byte count actually read)
mov qword[rsp+32], 0 ; Stack parameter
mov rcx, -10 ; STD_INPUT_HANDLE
call GetStdHandle
mov qword[rsp+48], rax
; r15b keeps track of two flags in its two least significant bits:
; 7 ... 1 0
; ---------------------------------------------------------------------------
; | | flush stdin buffer during | current iteration |
; | ... | the next iteration? | mode? |
; | | 0: no | 0: read-data mode |
; | | 1: yes | 1: flush-stdin mode |
; ---------------------------------------------------------------------------
mov r15b, 0
read_int_loop:
mov rcx, qword[rsp+48]
lea rdx, [buffer]
mov r8, buffer_len ; number of bytes to read
lea r9, [rsp+40] ; number of bytes actually read
call ReadFile
and r15b, 1111_1101b ; resetting the flush flag
cmp qword[rsp+40], buffer_len
jne read_int_process_input ; the input length is less than buffer_len
cmp byte[buffer+buffer_len-1], 0xa ; the input length might be equal to or more than buffer_len
je read_int_process_input ; the last byte is \n so, the input length is excatly buffer_len
or r15b, 10b ; setting the flush flag as the input legnth is greater than buffer_len
read_int_process_input:
test r15b, 01b ; are we in flush-stdin mode?
jz read_int_process_buffer ; ...no, processing user's guess
jmp read_int_loop_epilog
read_int_process_buffer:
; Searching for the first "\r" starting from the end to then Remove the possible trailing "\r\n" from the input.
mov rcx, buffer_len
read_int_check_cr_loop:
lea rax, [buffer]
lea rax, [rax+rcx-1]
cmp byte[rax], 0xd
je read_int_clear
loop read_int_check_cr_loop
jmp read_int_convert ; no remove is needed as we have at least buffer_len charaters
read_int_clear:
dec rcx
call clear_buffer
read_int_convert:
call str_to_int
mov r14, rax ; the final result
read_int_loop_epilog:
test r15b, 10b ; does stdin need to be flushed? ...
jz read_int_exit ; ... no
or r15b, 01b ; entering flush-stdin mode
jmp read_int_loop
read_int_exit:
add rsp, 56
mov rax, r14
ret
The next procedure prompts the player for their guess and waits for a valid input:
ask_for_guess:
; Returns an integer.
sub rsp, 8 ; stack alignment
ask_for_guess_loop:
; Printing the prompt
lea rcx, [ask_guess_msg1]
mov rdx, ask_guess_msg1_len
mov r8, 0
call print
mov rcx, [current_try]
mov rdx, 0
call print_int
lea rcx, [ask_guess_msg2]
mov rdx, ask_guess_msg2_len
mov r8, 0
call print
; Waiting for the user input
call read_int
cmp rax, error_code
jne ask_for_guess_exit ; everything is fine
; Invalid (Not A Number) input
lea rcx, [nan_guess_msg]
mov rdx, nan_guess_msg_len
mov r8, 1
call print
call print_empty_line
jmp ask_for_guess_loop
ask_for_guess_exit:
add rsp, 8
ret
Play Again?
At the end of the game, we want to ask the player if they want to play again.
ask_for_play_again:
; Asks the user if they want to play again and returns 1 if yes otherwise returns 0.
sub rsp, 40 ; one qword local variable (8) + shadow space (32)
mov qword[rsp+32], 0 ; local variable (byte count actually read)
lea rcx, [play_again_msg]
mov rdx, play_again_msg_len
mov r8, 0
call print
mov rcx, -10 ; STD_INPUT_HANDLE
call GetStdHandle
mov rcx, rax
lea rdx, [buffer]
mov r8, 1 ; number of bytes to read
lea r9, [rsp+32] ; number of bytes actually read
call ReadFile
cmp byte[buffer], 0x59 ; 'Y'
je ask_for_play_again_yes
cmp byte[buffer], 0x79 ; 'y'
je ask_for_play_again_yes
mov rax, 0 ; it's a no anwser
jmp ask_for_play_again_exit
ask_for_play_again_yes:
mov rax, 1
ask_for_play_again_exit:
add rsp, 40
ret
Program Introduction
The following procedure prints the introduction which is a collection of strings and the program logo.
print_intro:
sub rsp, 8 ; stack alignment
lea rcx, [logo]
mov rdx, logo_len
mov r8, 0
call print
lea rcx, [welcome_msg1]
mov rdx, welcome_msg1_len
mov r8, 0
call print
mov rcx, upper_bound
mov rdx, 0
call print_int
lea rcx, [welcome_msg2]
mov rdx, welcome_msg2_len
mov r8, 0
call print
mov rcx, max_tries
mov rdx, 0
call print_int
lea rcx, [welcome_msg3]
mov rdx, welcome_msg3_len
mov r8, 1
call print
add rsp, 8
ret
Logo
The program logo is an ASCII art saved in ascii.txt, generated using https://patorjk.com/software/taag. I have also written a Python script (ascii_2_asm_str.py) that generates a byte sequence from the ASCII art, making it usable in Assembly code.
All together
You can find the final source code on GitHub (guessme.asm).
How to Build
I have provided a batch file (build.bat) that invokes NASM and the Microsoft Linker to generate the final executable. It also includes sections for using gcc or clang linkers if you have them set up on your Windows machine and you prefer to use them instead. To enable a specific linker, uncomment its section and comment out the others.
Remember that the Microsoft Linker requires the Visual Studio Developer Command Prompt to function properly.
If you are not interested in building the program yourself and prefer a ready-to-go executable, you can download it from GitHub (guessme.exe). The executable was built using Visual Studio 2022 Build Tools on Windows 11.
Resources
- Understanding Windows x64 Assembly (This is the first place to go in you're serious)
- Cracking Assembly (An in-depth tutorial serie)
- Overview of x64 ABI conventions (The official Microsoft documentation)
- Windows x64 Calling Convention: Stack Frame (Visually describes the x64 stack layout)
- X64 Deep Dive (Yet another in-depth tutorial)
- assembly-fun (A repository full of samples)