Knowledge

Detect Googlebot visits using nginx

#nginx

How to detect when and how often Googlebot visits your website using a few configuration lines in nginx.

Published by Mark van Eijk on November 18, 2024 · 1 minute read

  1. Detecting Googlebot
  2. Logging the requests

Detecting Googlebot

It's very easy to detect the Googlebut using the user agent in nginx, here we use the map directive to set the variable $googlebot to yes or keep it empty depending on the given user agent:

map $http_user_agent $googlebot {
    default "";
    "~*googlebot" "yes";
}

Logging the requests

When $googlebot is filled, we want to log the request in a log file. This can be done using the access_log directive:

access_log /var/www/logs/googlebot.log bots if=$googlebot;

That's it!

You can read here how you can log multiple bots how to log multiple bots for your virtual host in nginx.

Subscribe to our newsletter

Do you want to receive regular updates with fresh and exclusive content to learn more about web development, hosting, security and performance? Subscribe now!

Related articles

Server Side Includes (SSI) in nginx

How to detect when and how often Googlebot visits your website using a few configuration lines in nginx.

Read more →

Configure Content Security Policy with nonce using nginx

How to detect when and how often Googlebot visits your website using a few configuration lines in nginx.

Read more →