{"id":1562,"date":"2020-09-04T10:29:52","date_gmt":"2020-09-04T10:29:52","guid":{"rendered":"http:\/\/ubiq.co\/tech-blog\/?p=1562"},"modified":"2025-10-07T04:43:35","modified_gmt":"2025-10-07T04:43:35","slug":"prevent-seo-bots-crawling-site","status":"publish","type":"post","link":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/","title":{"rendered":"How To Prevent Bots From Crawling Your Site"},"content":{"rendered":"<p>Every website administrator needs to protect their site from malicious bots and users. You may even need to prevent bots from crawling your website, if you don&#8217;t want its content to be indexed online without your permission, or even stolen. Here&#8217;s how to prevent bots from crawling your site. You can also use these steps to stop all\u00a0 spam bots and malicious bots from crawling your website.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#How_To_Prevent_Bots_From_Crawling_Your_Site\" >How To Prevent Bots From Crawling Your Site<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#1_Using_robotstxt\" >1. Using robots.txt<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#Disallow_all_search_engines_to_crawl_website\" >Disallow all search engines to crawl website<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#Allow_all_search_engines_to_crawl_website\" >Allow all search engines to crawl website<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#Disallow_One_Specific_Search_Engine_from_crawling_website\" >Disallow One Specific Search Engine from crawling website<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#Disallow_All_Search_Engines_from_Crawling_specific_folders\" >Disallow All Search Engines from Crawling specific folders<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#Disallow_All_Search_Engines_from_Crawling_specific_files\" >Disallow All Search Engines from Crawling specific files<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#2_Using_CAPTCHA\" >2. Using CAPTCHA<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#3_By_Blocking_IP_Address\" >3. By Blocking IP Address<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#4_Using_HTTP_Authentication\" >4. Using HTTP Authentication<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#5_Using_htaccess_file\" >5. Using .htaccess file<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"How_To_Prevent_Bots_From_Crawling_Your_Site\"><\/span>How To Prevent Bots From Crawling Your Site<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Here are the different ways to prevent bots from crawling your site.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"1_Using_robotstxt\"><\/span>1. Using robots.txt<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Robots.txt is a text file that contains crawling instructions for incoming bots. Search bots, spam bots and other bots look for this file before they crawl your website. They proceed depending on the instructions present in this file. Robots.txt must be served at <em>www.yourdomain.com\/robots.txt<\/em> URL. So if your website is www.helloworld.com, then robots.txt should be served at <em>www.helloworld.com\/robots.txt<\/em><\/p>\n<p>You can use robots.txt to tell search bots to not crawl your entire website, or specific folders and pages in it.<\/p>\n<p>There are quite a few rules available to instruct crawl bots. The most common ones are:<\/p>\n<ul>\n<li><strong>User-agent<\/strong>: Search bots user User-agent attribute to identify yourself. You can allow\/disallow crawl bots by mentioning their user agent names.<\/li>\n<li><strong>Disallow<\/strong>: Specifies the files or folders that are not allowed to be crawled.<\/li>\n<li><strong>Crawl-delay<\/strong>: Specifies the number of seconds a bot should wait before crawling each page<\/li>\n<li><strong>Wildcard (*)<\/strong>: Used to mean all bots<\/li>\n<\/ul>\n<p>Bonus Read : <a href=\"http:\/\/ubiq.co\/tech-blog\/nginx-ssl-configuration-step-step-details\/\">NGINX SSL Configuration (Step by Step)<\/a><\/p>\n<p>We will look at a few examples to disallow robots from crawling your site. Here are the user-agent names of common bots for your reference &#8211; Googlebot, Yahoo!, Slurp bingbot, AhrefsBot, Baiduspider, Ezooms, MJ12bot, YandexBot<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Disallow_all_search_engines_to_crawl_website\"><\/span>Disallow all search engines to crawl website<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Here&#8217;s what you need to add to your robots.txt file if you want to disallow all bots from crawling your website<\/p>\n<pre>User-agent: *\nDisallow: \/\n<\/pre>\n<p>In the above configuration, we use a wildcard * for user-agent rule to disallow all in robots.txt. We use home url (\/) in Disallow rule to specify entire website.<\/p>\n<p>In this case, we disallow all bots from crawling our entire website.<\/p>\n<p>Bonus Read : <a href=\"http:\/\/ubiq.co\/tech-blog\/linux-list-processes-name-user-pid\/\">Linux List All Processes by Name, User, PID<\/a><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Allow_all_search_engines_to_crawl_website\"><\/span>Allow all search engines to crawl website<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Here&#8217;s what you need to add to your robots.txt file if you want to allow all bots from crawling your website<\/p>\n<pre>User-agent: *\nDisallow:\n<\/pre>\n<p>In the above configuration, we use a wildcard * for user-agent to specify all crawl bots. We leave the Disallow rule as blank.<\/p>\n<p>In this case, we allow all bots from crawling our entire website.<\/p>\n<p>Bonus Read : <a href=\"http:\/\/ubiq.co\/tech-blog\/prevent-image-hotlinking-nginx\/\">How to Prevent Image Hotlinking in NGINX<\/a><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Disallow_One_Specific_Search_Engine_from_crawling_website\"><\/span>Disallow One Specific Search Engine from crawling website<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If you want to disallow only one specific crawl bot from crawling your website, mention its user name in User-name rule<\/p>\n<pre>User-agent: BaiduSpider\nDisallow: \/\n<\/pre>\n<h3><span class=\"ez-toc-section\" id=\"Disallow_All_Search_Engines_from_Crawling_specific_folders\"><\/span>Disallow All Search Engines from Crawling specific folders<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If you want to disallow all search engines from crawling specific folders (e.g \/product, \/uploads), mention them separately in Disallow rule<\/p>\n<pre>User-agent: *\nDisallow: \/uploads\nDisallow: \/product\n<\/pre>\n<h3><span class=\"ez-toc-section\" id=\"Disallow_All_Search_Engines_from_Crawling_specific_files\"><\/span>Disallow All Search Engines from Crawling specific files<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If you want to disallow all search engines from crawling specific files (e.g \/signup.html, \/payment.php), mention them separately in Disallow rule<\/p>\n<pre>User-agent: *\nDisallow: \/signup.html\nDisallow: \/payment.php\n<\/pre>\n<p>You can always use a combination of the above configurations in your robots.txt file.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"2_Using_CAPTCHA\"><\/span>2. Using CAPTCHA<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>CAPTCHA stands for Completely Automated Public Turing tests to Tell Computers and Humans Apart. It uses programmatic approach to determine whether a website visitor is a human or a bot. They do this by requiring the visitor to complete a task such as solving a puzzle, or typing characters, that can be done only by humans for sure. They are very effective in blocking bots on websites.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"3_By_Blocking_IP_Address\"><\/span>3. By Blocking IP Address<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In most cases, these bots operate using one or more specific IP addresses. If they are repeatedly hurting your site, then simply open your server log using an log management tool, identify the suspicious IP addresses, and block them using a security plugin or firewall. If your website is using a CMS such as WordPress, Magento, Joomla, etc. then you can easily use one of the many security plugins that are available our of the box.<\/p>\n<p>You can also use firewall services such as iptables, ufw, and fail2ban to block malicious IP addresses. If you are using a cloud service such as AWS, then you can also block these IP addresses directly using the cloud provider&#8217;s service management console.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"4_Using_HTTP_Authentication\"><\/span>4. Using HTTP Authentication<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>HTTP Authentication is a basic authentication service provided by almost every web server such as Apache and NGINX. Website administrators can enable it to require users to enter a username and password to access one or more web pages on their site. Unless the user enters proper login credentials, they will not be able to access the requested web page. This will automatically block all bots.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"5_Using_htaccess_file\"><\/span>5. Using .htaccess file<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If your website or blog runs on Apache server, then you can use .htaccess file to block web traffic from bots. It is usually used for redirecting traffic and rewriting URLs. It allows you to easily identify and block users based on different parameters in their request URLs. Most bots can be identified using their USER-AGENT attribute in request header. For example, if you have already enabled mod_rewrite (.htaccess) on your site, then add the following lines to this file to block Google Bot from crawling your site.<\/p>\n<pre>Rewrite Engine On\nRewriteCond %{HTTP_USER_AGENT}Googlebot{NC}\nRewriteRule .* - [R=403, L]<\/pre>\n<p>Most bots target ports 80 and 443. You may also want to <a href=\"_wp_link_placeholder\" data-wplink-edit=\"true\">change port number of Apache server<\/a> so that it becomes difficult for bots to guess.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Hopefully, now you can easily disallow bots from crawling your website. In this article, we have learnt several powerful ways to easily identify and block bot traffic from your website. You may need to use one or more of them to effectively stop this menace.<\/p>\n<p><a href=\"http:\/\/ubiq.co\/\">Ubiq<\/a>\u00a0makes it easy to visualize data in minutes, and monitor in real-time dashboards.\u00a0<a href=\"http:\/\/ubiq.co\/accounts\/register\">Try it<\/a> today!<\/p>\n<p>Also read:<br \/>\n<a href=\"https:\/\/ubiq.co\/tech-blog\/nginx-ssl-configuration-step-step-details\/\">NGINX SSL Configuration Step by Step<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here&#8217;s how to prevent bots from crawling your site.<\/p>\n","protected":false},"author":1,"featured_media":1567,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[10],"tags":[135],"class_list":["post-1562","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-website-administration","tag-crawl-bots"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How To Prevent Bots From Crawling Your Site - Ubiq BI<\/title>\n<meta name=\"description\" content=\"Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here&#039;s how to prevent bots from crawling your site.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How To Prevent Bots From Crawling Your Site - Ubiq BI\" \/>\n<meta property=\"og:description\" content=\"Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here&#039;s how to prevent bots from crawling your site.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\" \/>\n<meta property=\"og:site_name\" content=\"Ubiq BI\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ubiqbi\" \/>\n<meta property=\"article:published_time\" content=\"2020-09-04T10:29:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-07T04:43:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png\" \/>\n\t<meta property=\"og:image:width\" content=\"760\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Sreeram Sreenivasan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@UbiqBI\" \/>\n<meta name=\"twitter:site\" content=\"@UbiqBI\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sreeram Sreenivasan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\"},\"author\":{\"name\":\"Sreeram Sreenivasan\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc\"},\"headline\":\"How To Prevent Bots From Crawling Your Site\",\"datePublished\":\"2020-09-04T10:29:52+00:00\",\"dateModified\":\"2025-10-07T04:43:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\"},\"wordCount\":1012,\"image\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1\",\"keywords\":[\"crawl bots\"],\"articleSection\":[\"Website Administration\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\",\"url\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\",\"name\":\"How To Prevent Bots From Crawling Your Site - Ubiq BI\",\"isPartOf\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1\",\"datePublished\":\"2020-09-04T10:29:52+00:00\",\"dateModified\":\"2025-10-07T04:43:35+00:00\",\"author\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc\"},\"description\":\"Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here's how to prevent bots from crawling your site.\",\"breadcrumb\":{\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1\",\"width\":760,\"height\":400,\"caption\":\"prevent seo bots\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ubiq.co\/tech-blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How To Prevent Bots From Crawling Your Site\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/#website\",\"url\":\"https:\/\/ubiq.co\/tech-blog\/\",\"name\":\"Ubiq BI\",\"description\":\"Build dashboards &amp; reports in minutes\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ubiq.co\/tech-blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc\",\"name\":\"Sreeram Sreenivasan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g\",\"caption\":\"Sreeram Sreenivasan\"},\"description\":\"Sreeram Sreenivasan is the Founder of Ubiq. He has helped many Fortune 500 companies in the areas of BI &amp; software development.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/sreeram-sreenivasan\/\"],\"url\":\"https:\/\/ubiq.co\/tech-blog\/author\/wordpress\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How To Prevent Bots From Crawling Your Site - Ubiq BI","description":"Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here's how to prevent bots from crawling your site.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/","og_locale":"en_US","og_type":"article","og_title":"How To Prevent Bots From Crawling Your Site - Ubiq BI","og_description":"Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here's how to prevent bots from crawling your site.","og_url":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/","og_site_name":"Ubiq BI","article_publisher":"https:\/\/www.facebook.com\/ubiqbi","article_published_time":"2020-09-04T10:29:52+00:00","article_modified_time":"2025-10-07T04:43:35+00:00","og_image":[{"width":760,"height":400,"url":"https:\/\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png","type":"image\/png"}],"author":"Sreeram Sreenivasan","twitter_card":"summary_large_image","twitter_creator":"@UbiqBI","twitter_site":"@UbiqBI","twitter_misc":{"Written by":"Sreeram Sreenivasan","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#article","isPartOf":{"@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/"},"author":{"name":"Sreeram Sreenivasan","@id":"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc"},"headline":"How To Prevent Bots From Crawling Your Site","datePublished":"2020-09-04T10:29:52+00:00","dateModified":"2025-10-07T04:43:35+00:00","mainEntityOfPage":{"@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/"},"wordCount":1012,"image":{"@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1","keywords":["crawl bots"],"articleSection":["Website Administration"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/","url":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/","name":"How To Prevent Bots From Crawling Your Site - Ubiq BI","isPartOf":{"@id":"https:\/\/ubiq.co\/tech-blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage"},"image":{"@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1","datePublished":"2020-09-04T10:29:52+00:00","dateModified":"2025-10-07T04:43:35+00:00","author":{"@id":"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc"},"description":"Sometimes you may need to prevent seo bots and spam bots from crawling your site. Here's how to prevent bots from crawling your site.","breadcrumb":{"@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#primaryimage","url":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1","contentUrl":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1","width":760,"height":400,"caption":"prevent seo bots"},{"@type":"BreadcrumbList","@id":"https:\/\/ubiq.co\/tech-blog\/prevent-seo-bots-crawling-site\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ubiq.co\/tech-blog\/"},{"@type":"ListItem","position":2,"name":"How To Prevent Bots From Crawling Your Site"}]},{"@type":"WebSite","@id":"https:\/\/ubiq.co\/tech-blog\/#website","url":"https:\/\/ubiq.co\/tech-blog\/","name":"Ubiq BI","description":"Build dashboards &amp; reports in minutes","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ubiq.co\/tech-blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/ubiq.co\/tech-blog\/#\/schema\/person\/db98d49a766a3a111d8510935ab90abc","name":"Sreeram Sreenivasan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b3127ed2d4bb8efb3fa0bbb52cf2efd4d0156c97fc05a503537c883e8279947?s=96&d=mm&r=g","caption":"Sreeram Sreenivasan"},"description":"Sreeram Sreenivasan is the Founder of Ubiq. He has helped many Fortune 500 companies in the areas of BI &amp; software development.","sameAs":["https:\/\/www.linkedin.com\/in\/sreeram-sreenivasan\/"],"url":"https:\/\/ubiq.co\/tech-blog\/author\/wordpress\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ubiq.co\/tech-blog\/wp-content\/uploads\/2020\/09\/prevent-search-bot.png?fit=760%2C400&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/pbGGTT-pc","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts\/1562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/comments?post=1562"}],"version-history":[{"count":19,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts\/1562\/revisions"}],"predecessor-version":[{"id":9723,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/posts\/1562\/revisions\/9723"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/media\/1567"}],"wp:attachment":[{"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/media?parent=1562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/categories?post=1562"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ubiq.co\/tech-blog\/wp-json\/wp\/v2\/tags?post=1562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}