Blocking malicious crawler in nginx

Some crawlers and user agents are not helpful and could be potentially harmful. Some are just wasting your server resources.

For a simple blocking, you can add this code in your server block:

if ($http_user_agent ~* (libwww) ) { return 403; }

This tells nginx to response 403 to any connections initiated by user-agent which contains libwww. Depends on your requirement, usually you choose between a 403 or 444 response.

Most of the time, you are not dealing with just 1 bad user-agent. You want to block as many of them as possible. A community maintained list is available here. Just download and include the 2 files in your nginx directory and add these 2 lines in your nginx.conf http block:

include blacklist.conf;
include blockips.conf;

Finally, reboot your nginx server.