block

How to Block or Control YandexBot Crawling

Control YandexBot crawling

Use robots.txt first, then a server block only for abusive traffic

Yandex officially supports robots.txt and offers crawl-rate controls in Yandex Webmaster. Add a Yandex rule to a valid root-level robots.txt file and test it. A web-server block is stronger, but it also blocks legitimate access and can be bypassed by software that copies the Yandex user-agent name.

Use the robots.txt RuleServer blocking
FIRSTUse robots.txtOfficial crawl instruction
VERIFYCheck server logsConfirm the real problem
STRONG BLOCKUse only if necessaryUser agents can be spoofed

How do you know YandexBot is causing the load?

Review access logs

Compare request rate, paths, status codes and user-agent with the period of high resource use.

Find expensive URLs

Search, calendar, filter and parameter pages can create much more work than ordinary articles.

Separate bots from attacks

A user-agent string is not proof of identity. Malicious software can claim to be YandexBot.

Fix the bottleneck

Caching, database queries and unlimited URL combinations may need work even if one crawler exposes them.

How do you block Yandex in robots.txt?

Add this group to the plain-text robots.txt file at the website root:

User-agent: Yandex
Disallow: /

Confirm the public file returns HTTP 200 and plain text. Preserve existing sitemap and crawler rules. Yandex says a valid file can restrict crawling and reduce server load.

  1. 1
    Back up the current file

    Preserve all existing directives and sitemap locations before editing.

  2. 2
    Add the Yandex group

    Use the exact user-agent and disallow lines shown above.

  3. 3
    Test the response

    Open the file without signing in and validate it in Yandex Webmaster where available.

  4. 4
    Monitor requests

    Allow time for the crawler to refetch the file, then compare logs and load.

When should you block at the server?

Use a server block when traffic remains harmful, is impersonating a crawler or must stop immediately. The correct method depends on active Apache and Nginx configuration. A copied legacy Order Deny,Allow snippet may not suit a modern server.

How do you reverse the change?

robots.txt rule

Remove only the Yandex group and confirm the rest of the file still returns HTTP 200.

Server block

Restore the backup, test the site immediately and clear relevant caches.

Search visibility

Allowing crawl again does not guarantee immediate recrawling or indexing.

Long-term control

Block infinite URL spaces and fix expensive pages instead of blocking every crawler that finds them.

Read Yandex’s official robots.txt guidance.

YandexBot questions

Does Yandex ignore robots.txt?

Yandex states that its indexing robot supports robots.txt. A missing, inaccessible or invalid file may be treated as allowing crawl.

Will Disallow remove pages from search?

Not necessarily. Blocking crawl and removing an indexed URL are separate tasks.

Can I block by IP address?

Ranges can change and need verification. Do not maintain an unverified copied list.

Why is the old Apache rule gone?

It used legacy syntax and an imprecise pattern. Modern configurations need a rule suited to the actual server.

Still seeing heavy crawling?

Send UKC the time window and hostname

Include sample log lines with customer information removed.

Open a Support TicketView UKC Hosting