Sam - September 23, 2016

How to test proxies using bash and cURL with multi-threaded async requests

Tutorials

Introduction

If you’ve recently found some public proxies or even paid proxies you may want to test to see if they’re working correctly.

The quickest and easiest way to do this is with using the cURL cli binary on OSX (or any linux distro).

With this we can easily test HTTP, HTTPS, SOCKS4, SOCKS4a & SOCKS5 proxy protocols using just a few commands.

In this tutorial I will assume you have access to a linux terminal or OSX terminal.

Step 1 – Set your proxies

Copy all of your proxies into a file, in this case we will paste them all into a file called “usa.txt”

Find the perfect Proxy Product.

Proxyrack offers a multiple options to suit most use cases, if you are unsure our 3 Day Trial allows you to test them all.

Residential proxies

Never get blocked, choose your location

View all option available

Datacenter proxies

Super fast and reliable

View all option available

3 Day Trial

Test all products to find the best fit

View all option available

Example:

209.205.211.251:2000
209.205.211.251:2001
209.205.211.251:2002
209.205.211.251:2003
209.205.211.251:2004
209.205.211.251:2005
209.205.211.251:2006
209.205.211.251:2007
209.205.211.251:2008
209.205.211.251:2009
209.205.211.251:2010
209.205.211.251:2011
209.205.211.251:2012
209.205.211.251:2013
209.205.211.251:2014
209.205.211.251:2015
209.205.211.251:2016
209.205.211.251:2017
209.205.211.251:2018
209.205.211.251:2019

Step 2 – Write our bash / cURL script

Using your favourite text editor (vim/nano/etc) create your shell script file in this case we will call it test_usa.sh

Here are the seperate scripts for each protocol

HTTP protocol:

for proxy in `cat usa.txt`
do 
   (curl -s --max-time 10 -x $proxy -U YOURUSER:YOURPASS ip-api.com/json;echo) & 
done | grep country | wc -l

HTTPS Protocols

for proxy in `cat usa.txt`
do 
   (curl -s --max-time 10 -x https://$proxy -U YOURUSER:YOURPASS ip-api.com/json;echo) & 
done | grep country | wc -l

SOCKS4 Protocol

for proxy in `cat usa.txt`
do 
   (curl -s --max-time 10 --socks4 $proxy ip-api.com/json;echo) & 
done | grep country | wc -l

SOCKS4a Protocol

for proxy in `cat usa.txt`
do 
   (curl -s --max-time 10 --socks4a $proxy -U YOURUSER:YOURPASS ip-api.com/json;echo) & 
done | grep country | wc -l

SOCKS5 Protocol

for proxy in `cat usa.txt`
do 
   (curl -s --max-time 10 --socks5 $proxy -U YOURUSER:YOURPASS ip-api.com/json;echo) & 
done | grep country | wc -l

There is a max execution time of 10 seconds to prevent the proxy from never timing out.
Once it is completed it will output a number into your console and this is the number of successful requests it was able to complete

Example output:

sbook:~$ bash test_usa.sh
      19

Finished!

That was pretty easy, right?

It wouldn’t be hard to create a much more advanced script that takes parameters and all that jazz but this is a simple, easy solution that most people can understand more importantly simply copy + paste to get going.

How it works (if you’re new to bash shell scripting)

Line 1:

This is a for loop and every new line it reads it stores it in the variable $proxy

What is cat? This command reads a file and outputs it into your terminal

for proxy in `cat usa.txt`

Line 2:

This is the syntax we need to create the for loop (bash is really quite like reading english)

do

Line 3:

This executes the cURL command line with our $proxy (the line we just read)

-s Parameter is setting silent mode otherwise we get a bunch of ambigious text on our screen from the multi-threaded approach

-U YOURUSER:YOURPASS is the authentication process

--max-time 10 This sets the max time for our request to load the page ip-api.com/json

The & symbol sends our command into the background and moves onto the next part of the loop, thus creating our async / multi-threaded approach

   (curl -s --max-time 10 -x $proxy -U YOURUSER:YOURPASS ip-api.com/json;echo) &

Line 4:

Now we finish the loop with done just like we started with do

grep country then echos/prints only if we find the text string “country” from the page we loaded (only successful page loads would be counted)

wc -l This command counts how many of these instances we find of the above

This is probably not the most in depth explanation you’ll find on these single commands but hopefully it gives you a better understanding. If you genuinely want a further explanation of these commands or want to ask me a question just write below in the comments and I’ll see how I can assist 🙂