Control unwanted crawler traffic safely
Ask Baiduspider to stop in robots.txt first, then return 403 only if it keeps consuming resources
Use the least disruptive control that solves the problem. A robots.txt rule is easy to reverse and preserves normal visitor access. A server or Cloudflare block is stronger but must be tested carefully because user-agent matching can be spoofed.
Should you block Baiduspider?
Block or limit it when
Logs show sustained requests that consume bandwidth, PHP workers, database time or crawl capacity and the site does not need visibility in Baidu search.
Leave it available when
The business serves Chinese-speaking customers, depends on Baidu discovery or has no measurable resource problem from the crawler.
Check the evidence first
Confirm the user-agent, requested paths, response codes, request rate and time window in access logs. Do not act on an isolated visit.
Fix expensive pages too
A crawler may expose slow searches, broken links, endless calendar URLs or uncached dynamic pages that should be corrected regardless of the bot.
How do you ask Baiduspider not to crawl the site?
Add this group to the site’s root robots.txt file:
User-agent: Baiduspider
Disallow: /- 1Open the correct website root
In Plesk File Manager, open the document root for the domain, normally the directory containing the live site’s index file.
- 2Back up the existing robots.txt
Download or copy it before editing. Existing search-engine, sitemap and plugin rules must be preserved.
- 3Add the Baiduspider group
Place the two lines in a separate user-agent group. Do not add the disallow rule beneath a wildcard group unless every crawler should be blocked.
- 4Open the public file
Visit https://yourdomain.example/robots.txt and confirm it returns plain text with the new rule.
WordPress can generate a virtual robots.txt when no physical file exists. If an SEO or security plugin manages it, use that plugin’s supported editor or create a carefully reviewed root file. Avoid having several tools compete to generate different versions.
What does the robots.txt rule do?
It requests no crawling
A compliant crawler that identifies as Baiduspider should stop requesting paths covered by the slash rule after it refreshes the file.
It does not return an error
The website remains available if that user-agent requests it. robots.txt is instruction, not enforcement.
It can take time
The crawler may cache robots.txt and already queued URLs may still appear in logs for a while.
It does not guarantee removal
Blocking crawling is different from removing known URLs from a search index. Search engines may retain a URL reference without revisiting the content.
How do you return 403 to Baiduspider on Apache?
If requests continue and resource use justifies enforcement, back up the live .htaccess file and place this block before the normal application rewrite rules:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
RewriteRule ^ - [F,L]
</IfModule>The F flag returns HTTP 403 Forbidden and L stops later rewrite processing for that request. The old 2010 advice used BrowserMatchNoCase with Deny from env, an Apache 2.2-era access-control pattern that should not be copied into a current Apache 2.4 configuration.
Can Cloudflare block the crawler before it reaches hosting?
Yes. When the domain is proxied through Cloudflare, a custom security rule can match requests whose user-agent contains Baiduspider and block or challenge them at the edge. This can reduce origin traffic more effectively than .htaccess.
Use a narrow expression
Match the intended user-agent text case-insensitively and apply it only to the relevant hostname or site.
Review before enabling
Use rule preview, security events or log data where available. Check that normal browsers and monitoring services do not match.
Choose block or managed challenge
A block is predictable for an unwanted crawler. A challenge is less absolute but can still consume requests and may not suit an automated bot.
Keep the rule documented
Record why it exists, when it was added and how to disable it if the business later wants Baidu search visibility.
How do you test the result?
| Test | Command or action | Expected result |
|---|---|---|
| robots.txt | Open /robots.txt in a normal browser | HTTP 200 and the Baiduspider group appears once |
| Normal visitor | Open several public pages normally | The same successful pages and assets as before |
| Spoofed crawler header | curl -A "Baiduspider" -I https://example.com/ | HTTP 403 only when the hard block is enabled |
| Logs | Compare request counts and status codes after the change | Requests fall after robots.txt or receive 403 at the chosen layer |
| Other search engines | Review Google and Bing crawling or test their user-agents | They remain unaffected by the specific rule |
Which approaches should you avoid?
Copying old Apache 2.2 directives
Deny from may rely on compatibility modules and can behave differently in modern hosting. Use a current, tested method.
Maintaining random IP lists
Crawler networks change. Third-party lists become stale and can block unrelated visitors or miss new addresses.
Blocking every bot
A wildcard ban can remove useful search, monitoring and service traffic. Match only the crawler or paths causing the problem.
Editing without rollback
A syntax error in .htaccess can make the whole site return HTTP 500. Keep a copy and know how to restore it through Plesk.
Baiduspider blocking questions
Will robots.txt stop Baiduspider immediately?
Not necessarily. The crawler may use a cached copy and have requests already queued. Monitor logs over time before deciding the request was ignored.
Will blocking Baiduspider affect Google?
Not when the rule specifically targets Baiduspider and is implemented correctly. Test normal pages and review the complete robots.txt so a wildcard rule has not been changed accidentally.
Can I block only an expensive path?
Yes. In robots.txt replace the slash with the path prefix, or narrow the server or edge rule. This can preserve general discovery while protecting searches, calendars or other costly sections.
Why do requests continue after a 403 rule?
A server block stops application access but does not force the remote client to stop sending requests. An edge block can protect the origin, while rate limiting or provider assistance may be needed for persistent abusive traffic.
Can I block Baiduspider by IP?
It is possible, but static lists age quickly and require careful verification. Prefer crawler instructions or an edge security rule unless you have current, authoritative network data.
Should I ask support before editing .htaccess?
Yes if you are not comfortable restoring the file after a syntax problem. Include the domain, sample log lines and the observed resource effect, but remove passwords and unrelated personal data.
Need help checking the logs?
Start with evidence and keep a rollback copy
Send support the affected domain, time window, example requests and resource symptom. Do not send account passwords.