One Proxy to Rule Them All

Nicholas Anastasi and Joseph Morris co-authored this blog. Joseph Morris wrote the Terraform code, and Nicholas Anastasi did the mitmproxy scripting.

Welcome! If you’re looking for a powerful tool to help you bypass Web Application Firewalls (WAFs) during external penetration tests and bug bounty programs, you’re in the right place. Our Gigaproxy tool is designed to rotate IPs using mitmproxy, AWS API Gateway, and Lambda. The tool's repository and setup instructions are below and a full explanation of why and how we created it, follows.

gigaproxy

One proxy to rule them all - rotating IPs using mitmproxy, AWS API Gateway, and Lambda.

icon-github: https://github.com/Sprocket-Security/gigaproxy

A Background

Many of us are familiar with various tools that rotate IP addresses using the AWS API Gateway. While these tools are excellent for targeting a single URL, they fall short for larger scopes. For broader applications, we previously used a solution our founder wrote back in the day called proxycannon-ng, a private botnet leveraging multiple cloud environments. However, it has limitations in scale and cost.

proxycannon-ng

A private botnet using multiple cloud environments for pentesters and red teamers.

icon-github: https://github.com/proxycannon/proxycannon-ng

All in all, there hadn’t really been a great solution for scaled IP rotation that was cheap, worked against multiple targets, and was easy to connect to and maintain.

Introducing Gigaproxy

Gigaproxy solves these issues by enabling IP rotation across multiple targets. By leveraging AWS Lambda and API Gateway, it provides a scalable, cost-effective solution for pentesters and red teamers.

How did we get there, though? Well, let's start with an interchange that happened recently.

To explain Gigaproxy, we ought to mention Fireprox (fireprox-ng), a tool that rotates IP addresses by creating and staging an AWS API Gateway endpoint. The technique used by fireprox as a tool essentially creates and stages an AWS API gateway endpoint that uses the HTTP integration method. When using this integration as part of the API gateway, you can specify the following.

This example creates an API endpoint that, when requested, passes the request on to https://sprocketsecurity.com/, which essentially serves as a transparent reverse proxy. While this is great for a single endpoint URL, with this current integration type, there is no way to target multiple endpoints via API gateway.

Upon creating this API gateway endpoint, you are provided with a unique URL generated by AWS that, when requested with arbitrary HTTP verbs, will relay the request to the URL you specified. When the request comes out on the other end, that target endpoint will see the request as coming from AWS-owned infrastructure and IP space. This IP space is so large and the API gateway functionality is built so that every HTTP request coming out the other side to your target will originate from a unique IP.

As we said, one downside to this technique is that you can only target a single endpoint at a time. Therefore, if you are running tests at scale or spraying Nuclei against a target organization, you won’t be able to rotate your IP addresses without creating API gateway endpoints for every URL you want to target. This is very time-consuming and not viable for anything outside of single application engagements.

However, what if we put code into a Lambda function that processed an incoming request to the API gateway and subsequently relayed the request onto an arbitrary endpoint? Well, you can, and it might go a little something like this.

To test this, an API gateway endpoint was crafted that triggers a Lambda function containing the code above. To work correctly, the header "x-forward-me-to" must be included along with the HTTP request to the API gateway endpoint, telling the Lambda function where to send the request. When relayed by the lambda function, this request will include all of the original characteristics of the request we are attempting to relay. I’m sure some of you already see the caveats here, but we will get to that momentarily.

While this technique works, we must tackle some additional problems before we can Nuclei the entire Earth. One is Lambda cold starts, and the other is that none of the tooling out there supports the addition of an "x-forward-me-to" header when sending requests without significant modification.

Cooling off

First, let’s talk about Lambda cold start.

Lambda Execution Environments

Lambda execution details and lifecycle

icon-github: https://docs.aws.amazon.com/lambda/latest/operatorguide/execution-environments.html

When a Lambda function is run, it is loaded into an execution environment. If the Lambda function is called in quick succession, the same execution environment is used. This is fine in a normal use case, but we want to rotate our IP addresses. If you execute in the same environment repeatedly, your IP address will never change, making this technique unviable for us.

We began to ask, well, how does AWS determine that the same Lambda function is being called? We found after some testing that AWS must be hashing the configuration of the entire Lambda function definition and using that to determine if the same function is being executed in succession. Therefore, a simple change to the description of our Lambda function would lead AWS to believe that the code we are running must be loaded into a new execution environment to execute, resulting in a fresh public IP address.

How do you automate this? Simple: You can utilize an AWS EventBridge on a one-minute cron; we update the function description to the current time stamp every time it's triggered.

Amazon EventBridge

Build event-driven applications at scale across AWS, existing systems, or SaaS applications

icon-github: https://aws.amazon.com/eventbridge/

We found, regardless, that as the number of API gateway requests and Lambda function executions scaled to meet the needs of our tooling, multiple concurrent functions were loaded into new execution environments, resulting in even more frequent IP rotations. A cascading effect occurs wherein, at the start, you will only see rotations every minute, but as you send more requests to the gateway, dozens of public IPs begin to be used for outbound requests.

You said to forward me where?

Now for "x-forward-me-to". Nuclei, ffuf, katana, etc., have no native ability to include headers with dynamic values for outgoing requests. Modifying existing tools to support this would be a nightmare. That’s where mitmproxy comes in. We wrote a short script to leverage mitmproxy to modify our outgoing requests in flight, creating a double hop from our local host to our API gateway for actual relaying. The script looks a little something like this.

Super simple. Once you have installed mitmproxy and its root certificate, you should be able to get rolling with the script. More details on the actual setup are below.

Using Gigaproxy

Once your infrastructure is live and you have your API endpoint and API key, you can start forwarding to the gigaproxy using the command below.

mitmdump -s gigaproxy.py --set auth_token=$YOUR_TOKEN --set proxy_endpoint=$YOUR_ENDPOINT

Is it working?

With the mitmproxy script running in the background, you can take any tool that supports an HTTP proxy and utilize our infrastructure. To test that the script is working correctly, you can execute the following in your command line after the mitmproxy script is running:

while true; do for i in https://ifconfig.me https://api.ipify.org https://ipv4.rawrify.com/ip; 
 do curl -s $i -x http://127.0.0.1:8888 && echo ; 
done; done

This will continuously issue cURL requests to public IP grabber endpoints. By default, you should see your public IP changing every minute. The cool thing about gigaproxy, however, is as you send a higher rate of requests through the tool, AWS will begin to scale up the number of execution environments required to run your Lambda functions, meaning that you will begin to see your public IP rotating on nearly every request.

Sharing is caring

We envision that teams will likely set up this infrastructure and then subsequently share the API key they provisioned with the rest of their team. We have not noticed any level of rate limiting exhibited by AWS here, but there are some considerations you will need to make regarding cost relative to your team's size.

The biggest issue you may encounter is Mitmproxy's inability to handle the amount of traffic being passed through it. Therefore, request rate limiting features in Nuclei, Katana, etc. may have to be implemented to prevent request failure. Alternatively, and probably the best route here, is not to share the same Mitmproxy instance among teammates. Instead, use separate infrastructure for engagements. You should be doing this already.

Viewing logs

While throughout this article and the projects README, we call out the usage of mitmdump to provide the request hop proxying, you also can use mitmproxy or mitmweb on the command line with the same script to get an idea of what your requests look like going in and out. This can assist with troubleshooting tooling going through this proxy and with modifications to the mitmproxy script, could log all pentesting traffic to help with attribution.

You can even add the -w flag to store the contents of the requests and responses passed through mitmproxy to the API gateway. You have been warned. However, the output is ugly. You can also mod the script we have included to log the output of mitmproxy quite easily if needed.

If you’d like to enable logging on the Lambda function that forwards your requests, you can specify a "enable_function_logging" variable in your Terraform build. This will create a Cloudwatch log group called the following.

/aws/lambda/${var.project_identifier}-forwarder-function 

The Lambda function will forward logs to this group. To prevent log bloat, logs are retained for 14 days by default.

If Cloudwatch logging is enabled, you can also pass a X-Debug header with your requests to log the JSON input object that gets passed to the Lambda function via API Gateway (basically your request plus its metadata). Be careful with this, as it will log your API key, too — delete the relevant log stream(s) in Cloudwatch after inspection.

Okay, how much is it going to cost me?

Many of you are likely asking how much this will cost to run. Well, it’s pretty darn cheap. According to our calculations 🤓 even if you were to send 10,000,000 requests through gigaproxy monthly, you would still pay less than $200. With all of this being serverless, you don't get charged if you don’t use it.

Theoretical maximum monthly and annual costs assuming 100% utilization (10 million requests/month) every month with default API key quota limit.

If you want to do some calculations of your own, check out our pre-made pricing calculator.

Is there anything it can’t do?

At this time, the Lambda code can not handle binary data in responses. It can only take the response data if it is in the form of binary data and return it as a base64 blob. Take note of this, as favicon grabber tools and similar techniques will not work with this rotation technique.

If anyone can help resolve this, it would be greatly appreciated.

Wrapping Up

Before we close out, this excerpt from the fireprox GitHub repositories README sums up an obligatory message we would like to pass on to our readers.

icon-alert-triangle:

Use of this tool on systems other than those that you own are likely to violate the AWS Acceptable Use Policy and could potentially lead to termination or suspension of your AWS account. Further, even use of this tool on systems that you do own, or have explicit permission to perform penetration testing on, is subject to the AWS policy on penetration testing.

Be careful and have fun, everyone. As always, pull requests to the repository are very welcome and encouraged. We will likely be making some significant updates to this tool soon and hopefully provide the ability to execute this across multiple cloud environments, better manage authentication, and give some much more verbose logging capabilities to help your team provide context on testing activities to customers.

At Sprocket Security, we test thens of thousands of assets at scale, continuously making evasion techniques like this necessary. If you are interested in learning more about how we test and our platform, contact our team using the link below.