Web scraping is a process to gather bulk data from internet or web pages. The data can be consumed using an API. But there are sites where API is not provided to get the data. During this condition, we can use Web

where we can directly connect to the webpage and collect the required data. Now, python is one of the best programming languages, to use for development and web applications. It further helps in the analysis of the data with numeric and scientific calculations. So for this tutorial, we would prefer using python language for scrapping.

For web scrapping, there are multiple tools used but the best one is the Scrapy framework. It is specially designed to crawl websites using structured data. It can easily scaled form scrapping one page to multiple pages. It gives an option to extract this data into various formats like CSV, JSON, XML, JSON lines, etc.

In this tutorial, we will extract the data from Reddit website, for title, date, upvotes and image links.

We need an initial system setup before we start with our scraping project. We can install Scrapy framework either by using Python or Anaconda framework. We will see both the ways of installation; you can pick any of these ways to install Scrapy.

For Scrapy to run we need Python version 2.7 or 3.4 and above. Using below command we can have scrapy using python.

If you are using python, it is good to know the dependencies used in the Scrapy framework. You may need to install dependencies for

for any issues during installations. Also, note that these packages may depend on the non-python framework which you would require, for Scrapy run successfully.

If you are familiar with Anaconda, you can use it to install Scrapy. You just need to make sure that you are running

version required for your Anaconda framework to run. Check out their website to know the required python version mentioned for the Anaconda framework. With Anaconda installed you need not worry of other dependencies required for Scrapy, Anaconda will take care of it while installing.

To install Scrapy, open Anaconda Prompt, and type below command.

I would recommend using Anaconda, as it’s faster and would save from getting into installations of the dependencies.

Make a directory for your project. Point your prompt to your project directory.

To create a project using Scrapy framework type below command.

Once your project is created, you will get the below folder structure.

We will take a brief look at these files, but for this project, we will create a spider in the spider folder and set a few settings in the setting.py file.

├── scrapy.cfg          # configuration file

└── scraping_reddit     # This is project's Python module, you need to import your code from this

├── __init__.py         # Needed to manage the spider in the project

├── items.py            # define modules of scraped items

├── middlewares.py      # Define modules of spider middleware

├── pipelines.py        # Item pipeline file of the project

├── settings.py         # add settings here

└── spiders             # directory to locate spiders

├── __init__.py

Next, we will create a spider for our webpage to be crawled. Spider is a python code or a template used to crawl the given page using XPath and CSS expressions. We will take a look at these selectors and XPath while writing a spider code. To create spider, navigate to the inner folder of your project. i.e to …/scrapying_reddit/scrapying_reddit, and type below command.

scrapy genspider redditSpider <your-reddit-link>.com

This command will create a template of a spider, named as “redditSpider” with the link to set as “allowed_domains”. Our Reddit link will be “

”. Below is the command we will be using for the given url.

scrapy genspider redditSpider https://www.reddit.com/r/cats/

We will now see the web page where our redittSpider is pointed to and would code it to extract the marked items on this site. We are extracting, time content, image and number of votes.

We will inspect the path for each element, which needs to be extracted. Let’s inspect the image element.

While scraping the web pages, most tasks performed are extracting the data from HTML source. There are options to use various libraries such as, BeautifulSoap which is a python libery, Ixml. Scrapy has its own mechanism to work on, these are called Selectors. They select certain parts of the HTML document specified either by XPath or CSS expressions. names = response.xpath(‘//span[@class=”a-profile-name”]/text()’).extract()

This is the basic structure of the Spider. The variable ‘allowed_domain’ is taken from the command above. You can set the ‘allowed_domain’ to the domain name of the site. Now we will create XPath for the items and crawl the spider recursively.

# -*- coding: utf-8 -*-

import scrapy

class RedditspiderSpider(scrapy.Spider):

    name = 'redditSpider'
    allowed_domains = ['https://www.reddit.com/r/cats/']
    start_urls = ['http://https://www.reddit.com/r/cats//']

    def parse(self, response):
        pass



Below are the XPath we will use to extract Title, Images, UpVotes, Date and Time.

titles = response.xpath('//*[@class="_eYtD2XCVieq6emjKBH3m"]/text()').extract()
imgs = response.xpath('//img[@alt="Post image"]/@src').extract()
upVotesList = response.xpath('//*[@class="_1rZYMD_4xY3gRcSS3p8ODO"]/text()').extract()
datetimes = response.xpath('//*[@data-click-id="timestamp"]/text()').extract()

We will use attribute selector ‘//*’, to search titles, votes and date time. Once we extract the data, we will then yield it using the zip. Zip() is to map a similar index of different containers, so they can be used using a single entity.

for (title, img, upVotes, datetime) in zip(titles, imgs, upVotesList, datetimes):
    yield {'Title': title.encode('utf-8'), 'Image': img, 'Up Votes': upVotes, 'Date Time': datetime}

You know that Reddit only sends a few posts when you make a request to its subreddit. To scrape more data, you need to set up Scrapy to scrape recursively. the first step is to find out the XPath of the Next button. Then use response.follow function with a call back to parse function. This is how I did it in my code.

next_page = response.xpath('//link[@rel="next"]/@href').extract_first()
if next_page is not None:
    yield response.follow(next_page, self.parse)

Anyhow, you don’t need to scrape pages endlessly. 

you can control how many pages you follow using DEPTH_LIMIT property.

Below is the complete code for scrapping the Reddit page.

# -*- coding: utf-8 -*-
import scrapy


class RedditspiderSpider(scrapy.Spider):
    name = 'redditSpider'
    allowed_domains = ['www.reddit.com']
    start_urls = ['https://www.reddit.com/r/cats/']

    custom_settings = {
        'DEPTH_LIMIT': 10
    }

    def parse(self, response):
        titles = response.xpath('//*[@class="_eYtD2XCVieq6emjKBH3m"]/text()').extract()
        imgs = response.xpath('//img[@alt="Post image"]/@src').extract()
        upVotesList = response.xpath('//*[@class="_1rZYMD_4xY3gRcSS3p8ODO"]/text()').extract()
        datetimes = response.xpath('//*[@data-click-id="timestamp"]/text()').extract()
        for (title, img, upVotes, datetime) in zip(titles, imgs, upVotesList, datetimes):
            yield {'Title': title.encode('utf-8'), 'Image': img, 'Up Votes': upVotes, 'Date Time': datetime}

        next_page = response.xpath('//link[@rel="next"]/@href').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)



Scrapy gives us the flexibility to store the scraped data in the formats link JSON, CSV, XML, JSON lines given below. We will set the output format in csv file. So add these lines in settings.py file.

#FEED FORMAT
FEED_FORMAT = "csv"
FEED_URI = "reddit.csv"

To run our spider we will use runspider command.

Below is the output of the above code. You get the file in the path given in feed uri. “..\scraping_reddit\scraping_reddit\output”.

Here the output file shows title, image, upvotes and date time extracted using XPath.

Once the command is run, Scrapy will search for the spider and will run it through its crawler engine. The crawl is started by making a request to URL’s specified in the spider and the default method “parse” is called back. It will parse the response object and then will loop through the elements using a CSS Selector or using the mentioned XPath. These requests are scheduled and processed asynchronously. Thus Scrapy framework is built to not wait for a request to be finished, it will send next request to URL. If one request is failed, it won’t hamper the functioning of other requests.

With the concurrent request which helps you to crawl fast, it will also help to delay the download between each request by limiting concurrent request per domain or per IP. It even has an auto-throttling extension that figures out these automatically.

Scraping is the best way to get the data required in bulk amount. You can use the Scrapy framework to crawl to the Reddit using XPath. Scrapy can be easily installed using the Anaconda framework. It gives us the flexibility to set the format for output files in its setting file. You can even set the concurrent request per domain or per IP or set auto-throttling extension. You can further use this extracted data as input to analyzing tools of Python like Panda to convert these scrapped data into meaningful charts.

Scraping of Reddit using Scrapy: Python

21st Century enterprises now recognize the power of Big Data to boost profits, develop new sales pipelines, and develop and maintain a competitive edge in today’s highly contested online business environments. In any business, the mantra “time is money” has always been a core fundamental, and this vital rule of thumb hasn’t changed when it applies to the gathering, collating, and analyzing of Big Data for use by the enterprise’s decision-makers.

applications can drive traffic and increase conversions in a way which can level the playing field for startups who no longer need to lay out huge initial capital investments for expensive IT hardware onsite. Established SMBs need to stay current on pricing to maintain their competitive edge and foster robust growth, and it’s not as if there is any choice in the matter.

 and Big Data analytics in today’s digital environment has forced the issue. Any enterprise which chooses not to avail themselves of this critical business information can rest assured that their rivals certainly will, and relying on human resources alone to keep up to speed in the hyper-dynamic Big Data world of internet business is like entering a golf cart in the Grand Prix. You can’t win.

That said, many who recognize the value of Big Data and venture into the

of web scraping will find that the doors to the data they need are not left wide open. Many websites and search engines take sophisticated defensive measures to safeguard the data they contain and identifying a single IP as the source of a torrent of web crawling bots and queries is the first line of defense. The highest levels of anonymity and security are essential for any advanced data scraping operation. In this post, the experts at ProxyRack take a detailed look at just how a backconnect proxy can optimize advanced SaaS and Big Data applications with unsurpassed levels of anonymity while enhancing the power of advanced Big Data software with the sheer power of numbers.

Businesses today have at their disposal armies of automated information gathering bots thanks to advanced SaaS, all primed and ready to crawl the web and root out the most pertinent business data such as competitors’ pricing, customer preferences, and market trends. Today’s Big Data applications search and acquire precisely targeted consumer contact information that can make the “cold” sales lead an anachronism of the past. If your niche business manufactures pink 3-legged stools, web scraping can find the exact consumers who are looking for and purchasing pink 3-legged stools. Big Data applications can show you what your competitors are charging for their version of the products, online reviews from consumers, seasonal market trends, and the leading geo-locations for developing the most lucrative sales pipelines.

Of course, the websites being scraped are aware of the value of their information as well and take defensive measures to block it, and identifying the IP which is the source of queries plays a key role in that defense. Game on. It’s a case of go with a proxy or go home. But as we’ll find, for advanced Big Data applications, all

Web scraping applications have become irreplaceable tools for accomplishing the ambitious business tasks we discussed above, but there are limitations which are too frequently discovered the hard way soon after the data collecting bot army has already been deployed to the information battlefield. A single residential proxy does a fine job of masking the user’s IP address and preserving anonymity, but defensive measures at the target websites and search engines can still block access simply based on the sheer number or type of queries coming from the IP of that particular proxy over time.

Getting your IP address blocked is one of the most common obstacles faced in data scraping operations, and the banning may not occur until hours into the process when the data stream is lost not to mention the loss of precious business hours invested. Applications can’t function with incomplete data, and there are various red flags which can trigger an IP ban including:

A residential proxy is comprised of a single unit which acts as the middleman, forwarding and retrieving data with different referrers and headers. They are very beneficial for preserving anonymity and security for casual browsing but they are not the optimal tool for advanced web scraping applications. Data is sent from point A to point B, and every outgoing connection comes out as point B. This is where the rotating residential proxy plays a role.

Adding more residential proxies, say B, C, and D which rotate as the IP source, still leaves a definable footprint which is easily recognized as website accesses continue to go through a discernable B, C, D rotation. This can cause a search engine to automatically require a “

” signature for all actions or trigger a ban on the IP, bringing the web scraping process to a grinding halt.

The rotating pool of a small number of available residential IPs is only a partial solution. As we’ll see, the backconnect proxy takes the rotating IP concept a giant step farther. If we think of the Big Data stream as analogous to water flow, the residential proxy is the average garden hose (prone to kinks) while the backconnect proxy is the powerful and far more reliable firehose, built for full stream high-pressure performance.

Backconnect Proxy Advantages For Big Data Applications

The backconnect proxy is the rotating residential IP concept on steroids. Backconnect proxies are much more than the residential proxy unit accessing and forwarding data by rotating the IP with a limited and easily detectable number of addresses. Backconnect proxies are configured specifically to serve the demanding requirements of Big Data applications, comprised of a multitude of different machines and configurations linked together in a private network. From 50 to 500,000 proxies can all be linked together to form the single gateway known as the backconnect proxy.

Obviously, that multitude of IPs is the key advantage here. Rather than limiting the query sources to B, C, and D, websites and search engines see queries originating from a multitude of individual and separately geo-located points provided by the backconnect proxy network. Every connection is made from a different IP address, accessing the website from a unique connection point to keep data flowing without triggering red flag actions from the website. The only limitation to the power of the backconnect proxy is the number of accessible IP addresses available from the proxy provider.

At ProxyRack we can provide access to upwards of 1,250,000 IP addresses in 40 different countries including the US, UK, Australia, Russia, and Europe. For now. We are still continually growing our IP structure to allow for dynamic scaling of HTTP and SOCKS requests. While standard residential proxies do provide a good level of anonymity for the average browser, the backconnect proxy provides advantages for advanced users including:

Generally, the power of the multitude of rotating IPs networked together in a backconnect proxy allows a greater number of requests per minute. This eliminates the delays between requests which can occur with a standard proxy server. From the perspective that you can collect more data in a shorter time with a single command, the speed of the backconnect proxy is a significant advantage for Big Data applications.

That said, there is the other element of speed to be considered, that of broadband connections. All of the advantages of the multitude of residential IPs available in the backconnect proxy outweigh the inherent fluctuations in broadband speed which can occur with the varying quality of the proxies in the residential IP pool. Some connections may be lightning fast while others tend to be slower due to variations in normal broadband connections in different locations around the world. To minimize these speed fluctuations it’s important to choose an industry-leading provider such as ProxyRack which has an enormous pool of high-quality residential proxies available to optimize performance in the backconnect proxy rotation.

Any business conducting advanced online operations can benefit from the power provided by the backconnect proxy. These include:

These advanced applications are rarely “one and done” operations. Even the most advanced Big Data software is useless when your IP has been banned. Effective

programs require continuous deployment and monitoring to ensure SERP ranking is maintained. Web scraping and data mining bots must have reliable access in order to keep you abreast of changing business environments and Whois requests to develop pertinent contact information are all ongoing processes requiring repetitive access to the relevant sources on the web. The backconnect proxy can take anonymity, security, and scalability to unsurpassed levels to ensure that your data is available in a predictably consistent stream whenever you need it.

Most traditional proxy services allow you to purchase just a limited number of proxies, perhaps up to just 25, and they won’t change until the end of the month. That could be a severe handicap for Big Data operations conducted on an ongoing daily basis. At Proxy Rack we’ve set ourselves apart from traditional proxy services by accommodating the Big Data demands of today’s business applications by offering access to 100,000 unique IP addresses daily, and more than 1,250,000 unique monthly IP addresses. That means you spend your time collecting and analyzing your Big Data to boost profits and grow your business, rather than working around the delays caused by incomplete or inaccessible data from a blocked or banned IP.

Our multifunction rotating ports assign a new random unique IP on every connection or thread your software uses. You can also use the 10-minute rotating port to complete a sequence of requests from the same IP address. We don’t use any public proxies whatsoever, all IP addresses are exclusively private to ensure that your true IP address is never leaked. All of our plans include access to all of our available unique IP addresses, and we support HTTP, HTTPS, and SOCKS protocols that work with all existing software. You can choose from 50 simultaneous connections with our Standard Plan, 100 connections with the Elite Plan, and the ultimate 200 connections available with our Guru Plan. All

three of our rotating residential IP plans

include unmetered bandwidth and are backed by a 3-day money back guarantee.

If your business is ready to embrace the benefits of Big Data you’ll need the advantages that one of the largest private proxy services available to the public can provide, and getting started is just a click away. 

The Backconnect Proxy: Empowering Big Data For Big Profits

Web Scraping in the 21st Century Business Environment

Call it web scraping, data scraping, data extraction, screen scraping, web harvesting, or more generally DaaS (Data as a Service). By any name, Big Data has become a fundamental business tool in the 21st Century global business environment and collecting and analyzing that data is crucial for any business which finds itself in a highly competitive market. In this post, the techno-wizards at Proxy Rack explore the important role of web scraping in specific business applications across a broad range of industries.

We’inkedinl define exactly what web scraping is and how businesses like yours can use it to boost profits, foster robust growth, and streamline Big Data processes. We’ll explain why web scraping is an essential business tool, and how it is accessible for enterprises of all sizes in a variety of markets and industries.

Web Scraping Defined: The Secret Weapon of Fortune 500 Companies

 is the general term for the various automated methods applied for collecting information from the internet. To be effective, this is accomplished by software which simulates human web browsing, or web surfing, to collect information from websites. In the old school business environment, the value of knowing your competitor and keeping up to speed with supply and demand variables and other changing market conditions for any enterprise are well-established business fundamentals.

The value of business data is nothing new, but the demands on any business relying on internet marketing and data go far beyond the limited human capacity to collect that information from thousands of relevant websites, and that’s where the automated web scraping technique comes in. Web

“bots” comprise an automated workforce which is able to go about the assigned data collection tasks on a relentless 24/7/365 routine when required.

Businesses extract information from a website for a number of reasons, two of the most common being to grow the business by establishing a sales pipeline and to discover where competitors are setting their prices. 

 describes how web scraping is used as a growth hack by setting up a common robot.txt file which tells a web crawler what to look for on a web page. In Medal’s example, a

reseller assigns a bot to browse for the terms “Jordan” and “Air Jordan” at popular competing retail sites such as eBay and StockX. In this way, the reseller is able to access the aggregate prices charged by the competition and use that information as a competitive edge.

Of course, the reseller’s competitors are using the same essential web

techniques themselves, which leads us to the web scraping measures/countermeasures scenario reminiscent of the old “Spy vs. Spy” comics.

As data security consultant and user acquisition expert Eran Halevy notes in his own 

, web scraping has quickly developed into an unavoidable online arms race for the internet marketing sector. His article describes the web scraping slug out between retail giants Amazon and Walmart.

made industry news in 2017 by successfully blocking Walmart’s digital army of bots from web scraping Amazon’s listings “several million times a day”. This online cold war spawned a whole new sector of third-party service providers who specialize in identifying and blocking web scraping by competitors. That’s how valuable the data is.

/Walmart skirmish also highlights an important reality for enterprises at all levels concerning web scraping. If you’re not doing it yet, you can safely bet that your competitors are, which may explain your enterprise’s otherwise mysteriously diminishing returns. The CEO of NY wholesaler 

 explained his reasons for scraping his competitors’ sites every 20 minutes saying, “If we’re not decently priced, we’ll see it almost immediately in sales declines.”

Retail price competition is just one rather obvious aspect of the business value of Big Data. Now let’s look at some other rather surprising and creative ways businesses can profit from the Big Data collected by web scraping.

Development: Web Scraping and the Beauty Industry

The Beauty Industry is a $445 billion dollar industry 

 by HuffPost’s style and beauty reporter Julia Brucculieri, with the average American woman spending up to $300,000 just on face products in her lifetime. To date, most “big beauty” brands aren’t selling products tailored to fit the needs of individual consumers, but companies such as 

are changing that “one-size-fits-all” mentality with advanced web scraping combined with Artificial Intelligence (AI) technology.

and their tailored product development strategy

was 2 years in the making. The data was compiled by web scraping more than 8 million consumer reviews about 100,000 skin care products. Bots also scanned for data on 20,000 beauty ingredients while glomming through 4,000 scientific articles about skin and ingredient details. Specific keywords such as “acne” or “wrinkles” are connected to product reviews and ratings using machine learning. In this way, products can be tailored to use the ingredients proven as most successful for various skin conditions.

Consumers contribute to the success of their personal skin care products by taking a short dermatology survey to determine age, skin type, skin goals, ethnicity, and geographic location. Calculations are made using the web scraped data to develop a unique skin profile and a customized skin care regimen tailored to each customer’s specific needs. The massive web scraped database also lets 

avoid ingredients which are not a good fit with certain skin types and could actually cause harm.

Web Scraping to Predict Crowdfunding Success at Kickstarter

Tristan Dresbach of the NYC Data Science Academy

 came up with a creative use for web scraping when he asked the question “What characteristics maximize the probability of a successful 

 Campaign?” The popular crowdfunding platform 

 has drawn nearly 4 billion in US dollars pledged for business start-up campaigns.

The crowdfunding platform provides an exciting alternative to traditional start-up funding sources such as small business loans, finding an angel, or risking your own hard-earned cash. The percentage for a successful, fully funded campaign as of October 2018 at 

 is a daunting 36.4%, with a 63.6% failure rate 

(Donations are returned to the donors in any failed 

 campaign, those which do not achieve full funding.)

Dresbach decided to use web scraping to analyze the winning 36% to identify the key characteristics of successful campaigns. He created a script to extract 20+ variables including city, state, number of updates, reward levels, campaign duration, category, and creator to name just a few. Dresbach was able to determine important parameters for success at 

Dresbach has just “scraped the surface” of this project and plans to expand to 200 sub-categories to more precisely predict the best ways to create a start-up project, set the minimum funding goal, set reward levels, and choose a deadline for successful funding campaigns at 

Web Scraping For Recruiters and Employment Seekers

Big Data and analytics are enhancing recruiting and talent management in the human resources sector of industries across the board. Companies can 

 using web scraping to locate and attract the best-qualified candidates for the positions they have available. They no longer need to rely on the intuition and limited resources of individual human recruiters when it’s time to build the dream teams that will attract investors and inspire customer or

Web scraping allows recruiters to expand the search for precisely qualified talent beyond the usual resume sources at

or Indeed. Though these massive employment sites contribute a significant amount of hiring data to the recruiting process, web scraping can expand the search to social media and industry websites to aggregate data which enhances hiring with decisions based on facts and eliminates much of the risk and guesswork that is inherent in traditional hiring. In the IT field, for example, web scraping can be used to grade programmer candidates based on their coding abilities and the track record of actual programming contributions they have made online.

Of course, from the job seeker’s side, web scraping can also be useful, as self-described “aspiring data scientist” Michael Salmon explains in his article 

“Web Scraping Job Postings from Indeed”.

 Salmon describes his method as working smarter, not harder when parsing massive amounts of job listings at Indeed, which by the way also uses web scraping to compile it’s huge aggregated job lists.

Generating Quality Sales Leads With Web Scraping

” reveals what is probably the most well-known and widely applied application of web scraping. What enterprise could resist the potential to generate 10,000 leads in

? Andrew Fogg, Chief Data Officer and co-founder of 

. explains how to use web scraping to generate sales leads “in masse” in his

Web scraping provides a much higher quality of leads than the old technique of buying databases full of phone numbers and email addresses. The quantity is there, but without important “inside information” about the names in the data it’s impossible to sift out the hot prospects from the cold. Web scraping can be used to change all that.

As Fogg explains, quantity and quality are both enhanced when web-based data is the source tapped using a simple 3-step procedure:

The ideal user defined in step 1 is the key to quality leads. The web scraping tools can filter through the massive amounts of bulk information on the internet, extracting only the specific and relevant data using a set of your company’s pre-defined parameters.

Search Engine Optimization (SEO) and Web Scraping

Ranking on the almighty SERP, or Search Engine Result Pages, is fundamental to success in today’s competitive business environment, and SEO plays a key role in the online marketing world. Online reviews carry more authority with consumers today than a word-of-mouth recommendation from someone they know personally. SERP ranking is significantly affected by the website’s authority as assessed by the almighty Google search engine algorithms which take into

the number of backlinks to a site, the relevance of keywords users are searching, and the queries, or “long-tailed keywords” which are answered by informative content at the site.

software suites in the digital marketing field is ScrapeBox. Web scraping functions allow users to:

As we mentioned above, web scraping is a highly effective technique to gain a competitive edge over rival enterprises. That means that you’ll want to keep your web scraping programs confidential, and that means you need a reliable proxy service to mask your machine’s IP address. Our proxies work with any kind of software which supports HTTP or SOCKS. We’ve tested our proxies and they operate well in support of:

ProxyRack serves more than 50,000,000 page requests and powers some of the largest data mining companies on the web, in data mining operations spanning 3 continents. When you’re ready to give your enterprise the Big Data competitive edge that web scraping provides, don’t hesitate to contact us for the

services and technology which support Big Data extraction performance.

Web Scraping, Big Data, and How Successful Businesses Use Them

ProxyRack WHOIS Tool is a multi-threaded, socks5 proxy supported NodeJS application that can be used to scrape WHOIS record data or check domain availability.

ProxyRack WHOIS Tool requires node >=8.93

Install the dependencies and devDependencies and start the web UI server.

$ git clone https://github.com/proxyrack/whois-proxy-scraping-tool.git
$ cd whois-proxy-scraping-tool
$ npm install
$ node index.js --mode server --port 8080 --web-port 8000


$ node index.js --mode file --input domains.txt --output result.txt --output-available available.txt --output-registered registered.txt


{
  "proxies": [
    "yourusername:yourpassword@megaproxy.rotating.proxyrack.net:222 socks5"
  ]
}


You can modify your MongoDB settings here, proxies file path, results directories and set the number of concurrent threads you want to run

It is OK to completely ignore the MongoDB line here, just specify localhost even if you haven’t set up and installed it.

The software will still run and save results to disk.

'use strict';
const config = {
  db: {
    url: 'mongodb://localhost:27017',
    name: 'whois'
  },
  proxies: {
    path: 'proxies.json'
  },
  numThreads: 50,                        // number of threads per search
  outputDir: './results',
  outputUrl: '/results/'
};

