ProxyBroker - HTTP(S), SOCKS4/5.CONNECT80/23

[Elusive]

ProxyBroker - HTTP(S), SOCKS4/5.CONNECT80/23


ProxyBroker is an open source tool that asynchronously finds public proxies from multiple sources and concurrently checks them.
Features
Finds more than 7000 working proxies from ~50 sources.
Support protocols: HTTP(S), SOCKS4/5. Also CONNECT method to ports 80 and 23 (SMTP).
Proxies may be filtered by type, anonymity level, response time, country and status in DNSBL.
Work as a proxy server that distributes incoming requests to external proxies. With automatic proxy rotation.
All proxies are checked to support Cookies and Referer (and POST requests if required).
Automatically removes duplicate proxies.
Is asynchronous.
Usage
CLI Examples
Find
Find and show 10 HTTP(S) proxies from United States with the high level of anonymity:
Code:

$ proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10

Grab
Find and save to a file 10 US proxies (without a check):
Code:

$ proxybroker grab --countries US --limit 10 --outfile ./proxies.txt

Serve
Run a local proxy server that distributes incoming requests to a pool of found HTTP(S) proxies with the high level of anonymity:
Code:

$ proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High

Code:

Requirements

    Python 3.5 or higher

    aiohttp

https://pypi.python.org/pypi/aiohttp

    aiodns

https://pypi.python.org/pypi/aiodns

    maxminddb

https://pypi.python.org/pypi/maxminddb

Installation

To install last stable release from pypi:

$ pip install proxybroker

The latest development version can be installed directly from GitHub:

$ pip install -U git+https://github.com/constverum/ProxyBroker.git

Code:

ProxyBroker$ proxybroker

usage: proxybroker [--max-conn MAX_CONN] [--max-tries MAX_TRIES]

                   [--timeout SECONDS] [--judge JUDGES] [--provider PROVIDERS]

                   [--verify-ssl]

                   [--log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]]

                   [--version] [--help]

                   {find,grab,serve,update-geo} ...

Proxy [Finder | Checker | Server]

Commands:

  These are common commands used in various situations

  {find,grab,serve,update-geo}

    find                Find and check proxies

    grab                Find proxies without a check

    serve               Run a local proxy server

    update-geo          Download and use a detailed GeoIP database

Options:

  --max-conn MAX_CONN   The maximum number of concurrent checks of proxies

  --max-tries MAX_TRIES

                        The maximum number of attempts to check a proxy

  --timeout SECONDS, -t SECONDS

                        Timeout of a request in seconds. The default value is

                        8 seconds

  --judge JUDGES        Urls of pages that show HTTP headers and IP address

  --provider PROVIDERS  Urls of pages where to find proxies

  --verify-ssl, -ssl    Flag indicating whether to check the SSL certificates

  --log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]

                        Logging level

  --version, -v         Show program's version number and exit

  --help, -h            Show this help message and exit