Directory Brute-forcing at Scale

Introduction

Directory brute-forcing, a fundamental skill in the hacker's toolbelt, has evolved significantly over the years. Tools such as dirbuster, gobuster, feroxbuster, dirb, and ffuf have been instrumental in uncovering hidden content on websites. These tools and numerous wordlists designed to discover files and directories intelligently have become staples in the toolkits of penetration testers and bug bounty hunters. Now more sophisticated than ever, technology plays a vital role in cybersecurity practices.

Understanding Directory Brute-forcing

Directory brute-forcing involves systematically checking a list of potential directories and file names against a web server to discover hidden or non-publicly linked content. The goal is to uncover resources that may contain sensitive information or vulnerabilities, which are not meant to be publicly accessible but are inadequately protected.

The Problem with Current Techniques

Despite advancements, significant challenges persist in the realm of directory brute-forcing, especially at scale:

1. Scanning Multiple URLs: Tools like ffuf, considered among the best, do not natively support brute-forcing multiple URLs, a limitation the developer acknowledges.
2. Diverse Web Technologies: The vast array of web technologies today necessitates specialized wordlists to target specific tech stacks effectively. Unfortunately, no tools currently offer dynamic execution of commands with different wordlists tailored to the technology stack.
3. Web Application Firewalls (WAFs): WAFs can quickly detect and block requests from brute-forcing tools, hindering their effectiveness.
4. Varied Server Responses: Different tech stacks, load balancers, and WAFs lead to varied server responses. Whereas a 200 status code once reliably indicated the existence of a file, modern web applications exhibit a range of behaviors, complicating the task of determining valid or interesting endpoints.
5. Reviewing Output: Analyzing the results from directory brute-forcing tools is laborious, often involving manual review of text or JSON files and clicking through URLs. While some techniques streamline this process, a more efficient solution is needed.

Elevating ffuf with ffufw

ffufw, standing for "ffuf with that special sauce," is a robust wrapper around the widely-used ffuf tool, designed to make directory brute-forcing easier and smarter. This tool is a testament to the ongoing evolution in cybersecurity tactics, offering features that elevate directory brute-forcing to a new level of efficiency and intelligence.

puzzlepeaches / ffufw

ffufw is a wrapper around ffuf to make directory bruteforcing easier and more intelligent.

icon-github: https://github.com/puzzlepeaches/ffufw

Key Features of ffufw

1. Multithreaded Execution: It enables multithreaded execution of ffuf commands across multiple URLs, significantly enhancing the speed and scale of operations.
2. Dynamic Detection and Customization: The tool intelligently detects technologies using gowappalyzer and dynamically generates ffuf commands based on these detections. This includes the use of custom wordlists and extensions tailored to the detected technologies.
3. Enhanced Output Analysis: ffufw parses ffuf's output, filtering out irrelevant data and pinpointing potentially interesting endpoints, thus making the analysis more focused and less time-consuming.
4. Integration with gowitness: Discovered URLs can be passed to gowitness for visual review, providing a more intuitive understanding of the findings.
5. WAF Exclusion: The tool offers the ability to exclude URLs protected by Web Application Firewalls (WAFs) from the scans, thereby increasing efficiency and reducing the risk of being blocked.

Getting Started with ffufw

Installation

ffufw requires Go for installation. The following command can be used for a swift installation:

go install github.com/puzzlepeaches/ffufw@latest

Additionally, it leverages other tools like ffuf and ffufPostProcessing, which must be installed for the tool to run.

go install github.com/Damian89/ffufPostprocessing@latest
go install github.com/ffuf/ffuf/v2@latest

Usage

ffufw is designed to enhance the efficiency and effectiveness of directory brute-forcing through its advanced features and user-friendly interface. Let's delve into more detailed usage examples and potential use cases to better understand how this tool can be leveraged in various cybersecurity scenarios.

Basic Usage Example

For a simple start, you can run ffufw with a list of target URLs and specify an output directory for the results. This is useful for quick scans over a predefined list of URLs:

ffufw -i /path/to/targets.txt -o /path/to/output

Nothing quite special is happening here. The general execution workflow of the tool is as follows:

• Built-in wordlists are downloaded if not already existing and used for directory brute-forcing
• Directories are created in your output location for each individual URL
• Technologies in use for the URL are extracted from the targets
• ffuf commands are crafted for execution
• ffuf commands are sent into a queue and executed sequentially based on the concurrency rate specified

The resulting output of the tool can be found in folders located within your specified output location. Each URL processed has it’s own separate directory that stores results. In this directory, you will find result files storing requests and responses for review.

In addition to this, you will find JSON files containing the output from ffufPostProcessing. This output file is extremely similar to ffuf’s JSON file output with one addition — the reason ffufPostProccessing determined the endpoint found may be of interest. The logic ffufPostProccessing uses to do this is outside of the scope of this blog article, but you can find more detail in the tools wiki and code linked below.

https://github.com/Damian89/ffufPostprocessing/blob/master/Readme.md

https://github.com/Damian89/ffufPostprocessing/blob/master/pkg/results/clean.go

Advanced Usage with Custom Configurations

You can customize the scan by using a specific ffuf config file and enable verbose output for more detailed information. This approach is ideal for users who require deeper insights into the scan process and results:

ffufw -o /path/to/output -i /path/to/targets.txt -c /path/to/.ffufrc -v

Usage of the .ffufrc file is extremely powerful in the context of ffufw. With it, all configuration and command line options are possible in the context of ffufw as a wrapper.

In your ffuf configuration file, for example, you can specify headers and socks5 proxies for evasion purposes. An example of a ffuf config file we like to use is below.

[http]
 headers = [
 "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",
 "X-Forwarded-For: 127.0.0.1",
 "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
 "Sec-Ch-Ua: ",
 "Sec-Ch-Ua-Mobile: ?0",
 "Sec-Fetch-Site: none",
 "Sec-Fetch-Mode: navigate",
 "Sec-Fetch-User: ?1",
 "Sec-Fetch-Dest: document",
 "Accept-Encoding: gzip, deflate",
 "Accept-Language: en-US,en;q=0.9"
 ]

 timeout = 5
 proxyurl = "socks5://127.0.0.1:9999"

Integrating with gowitness for Visual Review

To enhance the review process, you can integrate ffufw with gowitness. This allows you to visually inspect the discovered URLs, making it easier to spot anomalies or interesting findings:

ffufw -o /path/to/output -i /path/to/targets.txt -g <http://127.0.0.1:9999>

This feature set utilizes the not often used gowitness API.

The API allows you to submit arbitrary URLs and gowitness will go out and take screenshots. While this functionality can be greatly useful for a small number of URLs, the results can quickly get out of control. As we stated earlier in the article, directory bruteforcing is not a perfect science and ffufpostprocessing can’t always infer what you want to see. With that, you may end up in situations where ffufpostprocessing miscalculates and gowitness becomes filled with screenshots of identical endpoints making review quite difficult.

Integrating with Burpsuite

To enhance the review process, you can integrate ffufw with Burpsuite or some other proxy utility. This allows you to easily inspect the discovered URLs, and requests/response making it easier to spot anomalies or interesting endpoints:

ffufw -o /path/to/output -i /path/to/targets.txt -r <http://127.0.0.1:8080>

This is where the magic happens. Many know that ffuf has similar functionality built right in using the replay proxy feature, but ffufw breaks this due to it’s use of ffufpostprocessing. With the -r flag, you can easily replicate the same behavior, however.

Customizing Tool Path and Concurrency

For environments with custom installations of ffuf and ffufPostProcessing, you can specify the paths to these binaries. Additionally, adjusting the concurrency level can optimize the scanning speed based on your system’s capabilities:

ffufw --ffuf /usr/local/bin/ffuf --ffufPostprocessing /usr/local/bin/ffufPostprocessing -o /path/to/output -i /path/to/targets.txt -t 5

No setups are made equal. Especially when it comes to go utilities. Therefore, we simply added functionality to directly specify the paths of required tools. Nothing special, but very useful in certain edge cases.

In addition, here, we up our threading for the tool. Threading if ffufw simply places more ffuf commands into the queue for execution. This is nice if you have a large number of URLs to work with and want to get things done quickly. Be careful though, you can lock your box up pretty quickly if you aren’t careful.

Excluding WAF-Protected URLs

In scenarios where you want to avoid triggering WAFs, ffufw allows you to exclude URLs protected by WAFs from the scans. This is particularly useful when scanning sensitive targets where avoiding detection is crucial:

ffufw -o /path/to/output -i /path/to/urls.txt -e

This was a feature added later in the tools inception. WAFs are, let’s admit, very annoying and often quite good at their jobs. Therefore, dependent on your use case, you may want to ignore WAFs altogether. Using a great library from projectdiscovery, cdncheck we can quickly determine if a WAF is in play and ignore scanning of it all together. This is nice, but you will miss stuff, so be careful.

Wrapping Up

This has been a problem we’ve been attempting to tackle for, well, let’s face it, years now. Instead of trying to reinvent the wheel or using cough cough, rust tools that segfault every ten minutes (you know who you are), ffufw was built to simply wrap the best of the bunch into a single utility.

Here at Sprocket, testing at scale is what we do. Directory brute forcing yields so much fruit that you can’t simply ignore it or use tools as they are. While we do think ffufw tackles the problem for us, bug bounty hunters and other penetration testing firms alike, we would love to receive suggestions and more importantly MR’s to help improve the tooling for years to come. In the future, we hope to iterate onto the great logic already in place in ffufpostproccessing to improve identification of standout endpoints even further.