Knowledge

Log bot requests in nginx

#nginx

If you want to know when and how often bots visit your website, you can easily track this using the following configuration in nginx.

Published by Mark van Eijk on November 18, 2024
Updated on November 18, 2024 · 1 minute read

  1. Detect bots using nginx
  2. Readble log format
  3. Log requests when it's a bot
  4. Log files per bot

Detect bots using nginx

First we need to detect if the current visitor's user agent indicates it is a bot. We can do this with the map directive in nginx:

map $http_user_agent $bot {
    default "";
    "~*googlebot" "google";
    "~*bingbot" "bing";
    "~*slurp" "yahoo";
    "~*duckduckbot" "duckduckgo";
    "~*baiduspider" "baidu";
    "~*yandexbot" "yandex";
    "~*sogou" "sogou";
    "~*exabot" "exabot";
    "~*applebot" "apple";
    "~*twitterbot" "twitter";
}

This checks the user agent string for matches for known bot names and then it maps it to a specific name that is set to variable $bot.

Readble log format

To make the entries readable, you can optionally choose to define a specific log_format for the bot requests:

log_format bots "$time_local: $request_method $scheme://$host$request_uri [$status] $bytes_sent @ $request_time ($http_referer)";

Log requests when it's a bot

Now we can log these bot requests by creating a specific bots.log file using the access_log directive that logs requests only if $bot is filled and set the log_format to the newly created bots format.

access_log /var/www/logs/bots.log bots if=$bot;

Log files per bot

If you prefer separating the logs per bot, so you can more easily see how many times specifically the Googlebot has come by your website, you can define a variable per bot:

if ($bot = "google") {
    set $google "1";
}
if ($bot = "bing") {
    set $bing "1";
}
if ($bot = "yahoo") {
    set $yahoo "1";
}
if ($bot = "duckduckgo") {
    set $duckduckgo "1";
}
if ($bot = "baidu") {
    set $baidu "1";
}
if ($bot = "yandex") {
    set $yandex "1";
}
if ($bot = "sogou") {
    set $sogou "1";
}
if ($bot = "exabot") {
    set $exabot "1";
}
if ($bot = "apple") {
    set $apple "1";
}
if ($bot = "twitter") {
    set $twitter "1";
}

And therefore setup log files per bot:

access_log /var/www/logs/bots/google.log bots if=$google;
access_log /var/www/logs/bots/bing.log bots if=$bing;
access_log /var/www/logs/bots/yahoo.log bots if=$yahoo;
access_log /var/www/logs/bots/duckduckgo.log bots if=$duckduckgo;
access_log /var/www/logs/bots/baidu.log bots if=$baidu;
access_log /var/www/logs/bots/yandex.log bots if=$yandex;
access_log /var/www/logs/bots/sogou.log bots if=$sogou;
access_log /var/www/logs/bots/exabot.log bots if=$exabot;
access_log /var/www/logs/bots/apple.log bots if=$apple;
access_log /var/www/logs/bots/twitter.log bots if=$twitter;

Subscribe to our newsletter

Do you want to receive regular updates with fresh and exclusive content to learn more about web development, hosting, security and performance? Subscribe now!

Related articles

Server Side Includes (SSI) in nginx

If you want to know when and how often bots visit your website, you can easily track this using the following configuration in nginx.

Read more →

Configure Content Security Policy with nonce using nginx

If you want to know when and how often bots visit your website, you can easily track this using the following configuration in nginx.

Read more →