Bot 'n Scraper Webserver Banning

Discussion in 'Domains and Hosting' started by graybeard, May 22, 2019.

  1. graybeard

    graybeard Well-Known Member

    How I ban bad bots, spy tools and web scrapers --


    [*]you need root access in the server (VPS or above)
    [*]you will need to edit the root files on the server in SSH (or a local terminal)

    [*]you need rather obscure knowledge of networks, who to block, who NOT to block
    [*]you need to scan your server logs CONTINUOUSLY finding new threats -- bad bots and scraping ips.
    [*]you need to know how to find AS assignments and block entire network CIDR ranges
    [*]you need to learn to use $ ipcalc
    [*]you will need to spend too much time [​IMG] on defeating the bottomless pit of SEO scammers and thieves
    Good luck with your spy tools and scrapers ...

    $ curl -I 'https://www.[target-server].com/'
    curl: (52) Empty reply from server
    ^^^ 444= a server fart
    _______________________________

    In a Nginx domain configuration;

    in the main
    server{



    Code:
    #########
    ###access control
    #allows

    #first in order allow the ips you want here if
    #they may be blocked in your .conf files
    #example below
    allow 34.35.67.102;

    ###deny conf

    # BLOCK SCAMMERS IP ADDRESSES
    include /etc/nginx/conf.d/blockips.conf;
    include /etc/nginx/conf.d/block_asns-hisec.conf;

    #BAD BOTS

    #map $http_user_agent $bad_bot
    #map include in /etc/nginx/nginx.conf
    #include /etc/nginx/conf.d/block_bots.conf;

    if ($bad_bot) { return 444; }

    if ($server_protocol = HTTP/1.0) { return 444; }

    if ($http_user_agent = "") { return 444; }

    ##############

    }
    named referrers can be easily blocked also -- just need to know the names ...
    This will never be 100% unfortunately. But you will see if the traffic you buy is junk or not [​IMG]