OpenSSL “HollowByte” Vulnerability Lets Attackers Crash Servers With Just 11 Bytes

Overview

Security researchers from Okta's Red Team have disclosed a denial-of-service (DoS) vulnerability in OpenSSL, dubbed HollowByte, that allows an unauthenticated remote attacker to consume significant server memory using a malicious payload of only 11 bytes. The flaw affects vulnerable OpenSSL implementations during the TLS handshake process and may lead to memory exhaustion, service degradation, or server crashes. 

According to public reporting, OpenSSL has already fixed the issue and backported patches to supported releases. However, the issue was reportedly addressed without a CVE identifier or dedicated security advisory.

Threat Details

Technical Details

HollowByte exploits the way vulnerable OpenSSL versions process TLS handshake messages. During a TLS handshake, a client sends a message containing a 4-byte header, where three bytes specify the size of the upcoming message body. Vulnerable OpenSSL versions allocate memory based solely on the size declared in this header before validating that the corresponding payload actually exists. 
An attacker can:

1. Establish a TLS connection.

2. Send an 11-byte packet containing a malicious size declaration.

3. Force OpenSSL to allocate memory based on the claimed size.

4. Never send the expected payload.

The affected worker thread then waits indefinitely while allocated memory remains reserved. Repeating this process across many connections can dramatically increase memory consumption. 

Why the Attack Is Dangerous

Researchers observed that on systems using glibc, memory released by OpenSSL may not immediately return to the operating system. By sending multiple requests with varying declared sizes, attackers can trigger memory fragmentation that prevents efficient memory reuse. This causes the application's Resident Set Size (RSS) to grow over time, even after malicious connections terminate. 
Researchers reported:

• A 1 GB server accumulated approximately 547 MB of fragmented memory before being terminated by the operating system.

• A 16 GB server accumulated approximately 25% of total system memory while remaining below typical connection limits.

These findings were observed during testing reported by Okta. 

Potentially Affected Technologies

Because OpenSSL is widely embedded across enterprise infrastructure, exposure may extend to:

• NGINX

• Apache HTTP Server

• Node.js

• Python

• Ruby

• PHP

• MySQL

• PostgreSQL

• Linux-based TLS-enabled services

These technologies were explicitly referenced as common OpenSSL consumers. 

Impact

Successful exploitation may allow attackers to:

• Exhaust available server memory.

• Cause service instability.

• Trigger process crashes.

• Force application restarts.

• Disrupt TLS-enabled services.

No authentication or prior access is required to trigger the condition. 

Recommendations

• Upgrade OpenSSL to a fixed release immediately.

• Verify whether internet-facing services rely on vulnerable OpenSSL versions.

• Monitor TLS services for abnormal memory growth.

• Validate that web servers, application servers, databases, and VPN services are using patched OpenSSL libraries.

OpenSSL fixed the issue through changes that move away from the vulnerable pre-allocation behavior and use incremental buffer growth. 

References

• BleepingComputer – HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload 

• Cyber Security News – OpenSSL "HollowByte" Vulnerability Lets Hackers Crash Servers With Just 11 By…

• The Hacker News – OpenSSL HollowByte Flaw Could Freeze Server Memory with 11-Byte TLS Requests