{"id":437,"date":"2010-11-15T23:10:12","date_gmt":"2010-11-15T22:10:12","guid":{"rendered":"https:\/\/www.uk-cheapest.co.uk\/support\/?p=437"},"modified":"2011-11-15T23:10:34","modified_gmt":"2011-11-15T22:10:34","slug":"i-dont-want-google-to-crawl-part-or-all-of-my-site","status":"publish","type":"post","link":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/","title":{"rendered":"I don’t want Google to crawl part or all of my site"},"content":{"rendered":"

There is a standard method involving a “robots.txt” file for excluding robot crawlers. This will prevent Googlebot or other crawlers from visiting your site. Googlebot has a user-agent of “Googlebot”. In addition, Googlebot understands some extensions to the robots.txt standard: Disallow patterns may include * to match any sequence of characters, and patterns may end in $ to indicate that the $ must match the end of a name. For example, to prevent Googlebot from crawling files that end in gif, you may use the following robots.txt entry:<\/p>\n

User-agent: Googlebot
\nDisallow: \/*.gif$<\/p>\n

There is another standard for telling robots not to index a particular web page or follow links on it, which may be more helpful, since it can be used on a page-by-page basis. This method involves placing a “META” element into a page of HTML.<\/p>\n

Remember, changing your server’s robots.txt file or changing the “META” elements on its pages will not cause an immediate change in what results Google returns. It is likely that it will take a while for any changes you make to propagate to Google’s next index of the web.<\/p>\n

Excerpt taken from\u00a0Google Webmaster Info<\/strong><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

There is a standard method involving a “robots.txt” file for excluding robot crawlers. This will prevent Googlebot or other crawlers from visiting your site. Googlebot has a user-agent of “Googlebot”. In addition, Googlebot understands some extensions to the robots.txt standard: Disallow patterns may include * to match any sequence of characters, and patterns may end … Read more…<\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"generate_page_header":"","footnotes":""},"categories":[19],"tags":[9,177,39],"class_list":["post-437","post","type-post","status-publish","format-standard","hentry","category-optimisation-search-engines","tag-google","tag-search-engines-2","tag-seo"],"yoast_head":"\nI don't want Google to crawl part or all of my site - UK-Cheapest.co.uk<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"I don't want Google to crawl part or all of my site\" \/>\n<meta property=\"og:description\" content=\"There is a standard method involving a “robots.txt” file for excluding robot crawlers. This will prevent Googlebot or other crawlers from visiting your site. Googlebot has a user-agent of “Googlebot”. In addition, Googlebot understands some extensions to the robots.txt standard: Disallow patterns may include * to match any sequence of characters, and patterns may end ... Read more...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/\" \/>\n<meta property=\"og:site_name\" content=\"UK-Cheapest.co.uk\" \/>\n<meta property=\"article:published_time\" content=\"2010-11-15T22:10:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2011-11-15T22:10:34+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.uk-cheapest.co.uk\/support\/wp-content\/uploads\/2023\/12\/IMG_3064.jpeg\" \/>\n\t<meta property=\"og:image:width\" content=\"768\" \/>\n\t<meta property=\"og:image:height\" content=\"439\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"UKC\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ukchelpdesk\" \/>\n<meta name=\"twitter:site\" content=\"@ukchelpdesk\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"UKC\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/\",\"url\":\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/\",\"name\":\"I don't want Google to crawl part or all of my site - UK-Cheapest.co.uk\",\"isPartOf\":{\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#website\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"UK-Cheapest.co.uk\",\"item\":\"https:\/\/www.uk-cheapest.co.uk\/support\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"I don’t want Google to crawl part or all of my site\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#website\",\"url\":\"https:\/\/www.uk-cheapest.co.uk\/support\/\",\"name\":\"UK-Cheapest.co.uk\",\"description\":\"Cheap Domain Names, Web Hosting, Site Builder & Web Security\",\"publisher\":{\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.uk-cheapest.co.uk\/support\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#organization\",\"name\":\"UKC\",\"url\":\"https:\/\/www.uk-cheapest.co.uk\/support\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.uk-cheapest.co.uk\/support\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-14-at-17.40.07.png\",\"contentUrl\":\"https:\/\/www.uk-cheapest.co.uk\/support\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-14-at-17.40.07.png\",\"width\":217,\"height\":55,\"caption\":\"UKC\"},\"image\":{\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/ukchelpdesk\",\"https:\/\/www.instagram.com\/ukc_hosting\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/person\/135cf981b8a3a120603fcbc027eed746\",\"name\":\"UKC\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f39b95f178ec21ebe3134a71baa1146d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f39b95f178ec21ebe3134a71baa1146d?s=96&d=mm&r=g\",\"caption\":\"UKC\"},\"sameAs\":[\"https:\/\/www.uk-cheapest.co.uk\"],\"url\":\"https:\/\/www.uk-cheapest.co.uk\/support\/author\/colin\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"I don't want Google to crawl part or all of my site - UK-Cheapest.co.uk","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/","og_locale":"en_GB","og_type":"article","og_title":"I don't want Google to crawl part or all of my site","og_description":"There is a standard method involving a “robots.txt” file for excluding robot crawlers. This will prevent Googlebot or other crawlers from visiting your site. Googlebot has a user-agent of “Googlebot”. In addition, Googlebot understands some extensions to the robots.txt standard: Disallow patterns may include * to match any sequence of characters, and patterns may end ... Read more...","og_url":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/","og_site_name":"UK-Cheapest.co.uk","article_published_time":"2010-11-15T22:10:12+00:00","article_modified_time":"2011-11-15T22:10:34+00:00","og_image":[{"width":768,"height":439,"url":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-content\/uploads\/2023\/12\/IMG_3064.jpeg","type":"image\/jpeg"}],"author":"UKC","twitter_card":"summary_large_image","twitter_creator":"@ukchelpdesk","twitter_site":"@ukchelpdesk","twitter_misc":{"Written by":"UKC","Estimated reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/","url":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/","name":"I don't want Google to crawl part or all of my site - UK-Cheapest.co.uk","isPartOf":{"@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#website"},"breadcrumb":{"@id":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/i-dont-want-google-to-crawl-part-or-all-of-my-site\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"UK-Cheapest.co.uk","item":"https:\/\/www.uk-cheapest.co.uk\/support\/"},{"@type":"ListItem","position":2,"name":"I don’t want Google to crawl part or all of my site"}]},{"@type":"WebSite","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#website","url":"https:\/\/www.uk-cheapest.co.uk\/support\/","name":"UK-Cheapest.co.uk","description":"Cheap Domain Names, Web Hosting, Site Builder & Web Security","publisher":{"@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.uk-cheapest.co.uk\/support\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#organization","name":"UKC","url":"https:\/\/www.uk-cheapest.co.uk\/support\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/logo\/image\/","url":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-14-at-17.40.07.png","contentUrl":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-14-at-17.40.07.png","width":217,"height":55,"caption":"UKC"},"image":{"@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/ukchelpdesk","https:\/\/www.instagram.com\/ukc_hosting\/"]},{"@type":"Person","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/person\/135cf981b8a3a120603fcbc027eed746","name":"UKC","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.uk-cheapest.co.uk\/support\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f39b95f178ec21ebe3134a71baa1146d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f39b95f178ec21ebe3134a71baa1146d?s=96&d=mm&r=g","caption":"UKC"},"sameAs":["https:\/\/www.uk-cheapest.co.uk"],"url":"https:\/\/www.uk-cheapest.co.uk\/support\/author\/colin\/"}]}},"_links":{"self":[{"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/posts\/437"}],"collection":[{"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/comments?post=437"}],"version-history":[{"count":1,"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/posts\/437\/revisions"}],"predecessor-version":[{"id":438,"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/posts\/437\/revisions\/438"}],"wp:attachment":[{"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/media?parent=437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/categories?post=437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.uk-cheapest.co.uk\/support\/wp-json\/wp\/v2\/tags?post=437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}