[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"cms-page:\u002Fblog\u002Fthings-consider-before-adding-disallow-robots-txt-file":3},{"date":4,"date_gmt":4,"content":5,"sticky":7,"format":8,"categories":9,"collections":12,"coauthors":13,"class_list":15,"acf":25,"head_links":445,"breadcrumbs":446,"reading_minutes":174,"category":454,"head":457},"2021-11-05T12:00:00",{"rendered":6},"\n\u003Cp>The robots.txt file is a core tool in any \u003Ca rel=\"noreferrer noopener\" href=\"\u002Fseo\u002Ftechnical-seo\u002F\">technical SEO specialist’s\u003C\u002Fa> arsenal. Auditing it is paramount in any standard technical SEO review due to the hard-line nature that typically comes with the file.\u003C\u002Fp>\n\n\n\n\u003Cp>Unlike other SEO components, directives within the file are \u003Cem>usually\u003C\u002Fem> more respected (for search engines, anyway) as opposed to just being hints or general suggestions. Configuring them incorrectly, therefore, can prove harmful to your site’s organic visibility.\u003C\u002Fp>\n\n\n\n\u003Ch2 class=\"wp-block-heading\" id=\"what-is-a-robots-txt-file\">What is a robots.txt file?\u003C\u002Fh2>\n\n\n\n\u003Cp>Found at root domain level, i.e. \u002Frobots.txt, the robots.txt file is an exclusion protocol targeted at web crawlers. It’s designed to communicate which parts of a site they are allowed and, more commonly, not allowed to crawl. This is done by setting crawl directives.\u003C\u002Fp>\n\n\n\n\u003Cp>Crawl directives are signified by the \u003Cstrong>Disallow:\u003C\u002Fstrong> or \u003Cstrong>Allow:\u003C\u002Fstrong> syntax whereby you state which URLs on your site are permitted for crawling. Crawl directives then need to be coupled with a user agent target, i.e. \u003Cstrong>User-agent:\u003C\u002Fstrong>, in order to form the “rules” for web crawlers to abide by.\u003C\u002Fp>\n\n\n\n\u003Cp>Here’s an example of a robots.txt rule that’s typically found on a WordPress site;\u003C\u002Fp>\n\n\n\n\u003Cp>User-agent: *\u003Cbr>Disallow: \u002Fwp\u002Fwp-admin\u002F\u003Cbr>Allow: \u002Fwp\u002Fwp-admin\u002Fadmin-ajax.php\u003C\u002Fp>\n\n\n\n\u003Cp>Here, all user-agents are targeted, as signified by the asterisk symbol. Two directives then follow; the first stating to not crawl \u003Cem>\u002Fwp\u002Fwp-admin\u002F\u003C\u002Fem> with the second stating to override the first disallow directive, but only for the following URL; \u003Cem>\u002Fwp\u002Fwp-admin\u002Fadmin-ajax.php\u003C\u002Fem>.\u003C\u002Fp>\n\n\n\n\u003Cp>At its core, its purpose and setup is relatively straightforward. However, I still witness some misconceptions regarding the robots.txt file where SEOs still overlook it’s impact on the remainder of a site’s organic presence. I’d like to unpack these here.\u003C\u002Fp>\n\n\n\n\u003Cp>Here are to some things to remember before opting to disallow URLs and directories via robots.txt.\u003C\u002Fp>\n\n\n\n\u003Ch2 class=\"wp-block-heading\" id=\"what-is-a-robots-txt-files-purpose\">What is a robots.txt file’s purpose?\u003C\u002Fh2>\n\n\n\n\u003Cp>Robots.txt is concerned with crawling – no more, no less.\u003C\u002Fp>\n\n\n\n\u003Cp>However, depending on where you look, it’s still used interchangeably with the notion of indexing. Here’s a screenshot from Ryte.com’s Website Success tool where they knowingly muddle the differentiation.\u003C\u002Fp>\n\n\n\n\u003Cfigure class=\"wp-block-image\">\u003Cimg loading=\"lazy\" decoding=\"async\" width=\"1149\" height=\"266\" src=\"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002FTlmVbAba-ryte-crawl-configurations.png?auto=compress%2Cformat&q=90\" alt=\"ryte crawl configurations\" class=\"wp-image-4210\"\u002F>\u003C\u002Ffigure>\n\n\n\n\u003Cp>When setting a crawl, Ryte asks the user whether they want their crawler to obey the robots.txt or not, in order to find “which pages are set to noindex”.\u003C\u002Fp>\n\n\n\n\u003Cp>This is incorrect.\u003C\u002Fp>\n\n\n\n\u003Cp>A page disallowed by robots.txt can still find its way into a search engine index. The notion of “noindex” (alluding to the use of the meta robots noindex tag) becomes applicable after a crawl and determines whether or not to include an item of content into a search engine’s index.\u003C\u002Fp>\n\n\n\n\u003Cp>Google explains this clearly in their guidance below;\u003C\u002Fp>\n\n\n\n\u003Cfigure class=\"wp-block-image\">\u003Cimg loading=\"lazy\" decoding=\"async\" width=\"775\" height=\"131\" src=\"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002FMt4L1mh5-google-robots-txt.png?auto=compress%2Cformat&q=90\" alt=\"\" class=\"wp-image-4211\"\u002F>\u003C\u002Ffigure>\n\n\n\n\u003Cp>This is where understanding the definition of crawling and indexing is essential, especially when applying the two to robots.txt files.\u003C\u002Fp>\n\n\n\n\u003Cp>Crawling is the process search engines take to navigate around the web to find information – this is where robots.txt becomes applicable.\u003C\u002Fp>\n\n\n\n\u003Cp>Indexing, on the other hand, is the process after this, where new or updated information is stored, processed and interpreted – this is where the notion of a meta robots noindex tag would apply.\u003C\u002Fp>\n\n\n\n\u003Cp>Granted, my point here is simply regarding nomenclature. You may well be aware of the differentiation between crawling and indexing and the tools you can use to control these two constructs. However, to those less versed in the nuances of technical SEO, it’s a fundamental requirement to ensure you understand how search works.\u003C\u002Fp>\n\n\n\n\u003Cp>What’s more, if these definitions are not understood, you can run into problems when it comes to applying robots.txt into your SEO strategies.\u003C\u002Fp>\n\n\n\n\u003Ch2 class=\"wp-block-heading\" id=\"how-it-can-impact-your-index-presence\">How it can impact your index presence\u003C\u002Fh2>\n\n\n\n\u003Cp>While the robots.txt file is there to manage crawl traffic, a consideration still needs to be made on the impact it can have on your site’s index presence.\u003C\u002Fp>\n\n\n\n\u003Cp>We know that a disallowed page can still find its way into the index – this has been confirmed and documented by Google several times. When this happens, snippets like this find themselves in SERPs;\u003C\u002Fp>\n\n\n\n\u003Cfigure class=\"wp-block-image\">\u003Cimg loading=\"lazy\" decoding=\"async\" width=\"325\" height=\"108\" src=\"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002F7blxDv73-Indexed-though-blocked-via-robots.png?auto=compress%2Cformat&q=90\" alt=\"\" class=\"wp-image-4212\"\u002F>\u003C\u002Ffigure>\n\n\n\n\u003Cp>But what causes this? Likely three things, either;\u003C\u002Fp>\n\n\n\n\u003Col class=\"wp-block-list\">\n\u003Cli>The page is being linked-to via an external source, or;\u003C\u002Fli>\n\n\n\n\u003Cli>The page was already crawled and indexed, with the disallow directive being applied after;\u003C\u002Fli>\n\n\n\n\u003Cli>The directive that the page applies to has been ignored (this is increasingly uncommon, though it’s still \u003Ca href=\"https:\u002F\u002Fwww.contentkingapp.com\u002Facademy\u002Frobotstxt\u002F\" target=\"_blank\" rel=\"noopener\">accepted that it can happen\u003C\u002Fa>).\u003C\u002Fli>\n\u003C\u002Fol>\n\n\n\n\u003Cp>So when Google attempts to index the page again, the content is largely unavailable (aside from a few components). Finding exactly where this occurring on your site is now made simple via the new Search Console by navigating to \u003Cstrong>Coverage\u003C\u002Fstrong> > \u003Cstrong>Valid with warnings\u003C\u002Fstrong> > \u003Cstrong>Indexed, though blocked via robots.txt\u003C\u002Fstrong>.\u003C\u002Fp>\n\n\n\n\u003Cfigure class=\"wp-block-image\">\u003Cimg loading=\"lazy\" decoding=\"async\" width=\"913\" height=\"431\" src=\"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002FxsdNFlNI-Indexed-though-blocked-via-robots-report.png?auto=compress%2Cformat&q=90\" alt=\"\" class=\"wp-image-4213\"\u002F>\u003C\u002Ffigure>\n\n\n\n\u003Cp>Here, you’re able to pinpoint exactly which URLs are being blocked by your robots.txt file but still discoverable via search.\u003C\u002Fp>\n\n\n\n\u003Cp>Now, it’s short-sighted to instantly presume these are critical errors. Google themselves class them as warnings, acknowledging how they may be intentionally set like that. If it is a necessary evil then so be it. However, if it was unintentional, then some further investigation is worthwhile by looking at each blocked but indexed URL individually.\u003C\u002Fp>\n\n\n\n\u003Cp>What you need to decipher is why these URLs were disallowed in the first place. If it was to reduce duplication, then a far more optimal solution is to consider canonicalisation. If the intention was to remove the page from search entirely then a meta robots noindex tag is more appropriate. Finally, and arguably most importantly, if a URL has been blocked by mistake then this needs to be resolved altogether by removing the disallow.\u003C\u002Fp>\n\n\n\n\u003Cp>Making blanket recommendations in technical SEO is difficult since the discipline is so dependent on context. However, what remains true is keeping an eye on these URLs. After all, having several blocked URLs indexed can dilute your index presence and make for a sub-optimal search experience. This is especially the case if disallowed URLs are still receiving organic traffic, which can be identified via DeepCrawl and their “\u003Ca href=\"https:\u002F\u002Fwww.deepcrawl.com\u002Fknowledge\u002Freports\u002Fdisallowed_pages_with_traffic\u002F\" target=\"_blank\" rel=\"noopener\">Disallowed Pages with Traffic\u003C\u002Fa>” report.\u003C\u002Fp>\n\n\n\n\u003Cfigure class=\"wp-block-image\">\u003Cimg loading=\"lazy\" decoding=\"async\" width=\"639\" height=\"137\" src=\"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002F9gjvemog-disallowed-urls-with-traffic-deepcrawl.png?auto=compress%2Cformat&q=90\" alt=\"\" class=\"wp-image-4214\"\u002F>\u003C\u002Ffigure>\n\n\n\n\u003Ch2 class=\"wp-block-heading\" id=\"how-it-impacts-link-equity\">How it impacts link equity\u003C\u002Fh2>\n\n\n\n\u003Cp>Another point to consider before disallowing via robots.txt is how it prevents the spread of link equity. This is important to bear in mind if you’re looking to introduce robots.txt directives as a means to control crawl efficiency.\u003C\u002Fp>\n\n\n\n\u003Cp>Again, it’s not a one size fits all solution as managing crawl budget typically comes with some trade-offs depending on the avenue you go with. Luke Carthy conveyed this perfectly during his talk at SearchLeeds 2018, “\u003Ca rel=\"noreferrer noopener\" href=\"https:\u002F\u002Fwww.slideshare.net\u002FLukeCarthy\u002Fhow-to-optimise-the-st-out-of-your-internal-search\">How to Optimise the S*** out of Your Internal Search\u003C\u002Fa>”. It’s the below slide in particular that outlines your options well;\u003C\u002Fp>\n\n\n\n\u003Cfigure class=\"wp-block-image\">\u003Cimg loading=\"lazy\" decoding=\"async\" width=\"1230\" height=\"590\" src=\"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002F7Ly2z5Fa-luke-carthy-slide.png?auto=compress%2Cformat&q=90\" alt=\"\" class=\"wp-image-4215\"\u002F>\u003C\u002Ffigure>\n\n\n\n\u003Cp>As we can see, while disallow directives via robots.txt “fix” duplicate content issues and improve crawl efficiency, the setup comes with negatives implications your regarding internal and inbound links.\u003C\u002Fp>\n\n\n\n\u003Cp>It’s the inbound links we need to be particularly concerned with, especially if they’re of quality. Ensuring these link to URLs that are able to redistribute link equity is the main objective here to ensure your site benefits from the authority. DeepCrawl can visualise this for you with their “\u003Ca rel=\"noreferrer noopener\" href=\"https:\u002F\u002Fwww.deepcrawl.com\u002Fknowledge\u002Freports\u002Fbacklink_pages_disallowed_uncrawled\u002F\">Disallowed URLs with Backlinks\u003C\u002Fa>” report.\u003C\u002Fp>\n\n\n\n\u003Cp>Working directly with DeepCrawl’s Majestic integration, this report allows you to investigate all links with disallow directives applied to them. From there, it’s a case determining a resolution such as removing the disallow, redirecting the URL to a more appropriate destination or even asking the site owner to reposition the link to a landing page that’s more integral to your SEO strategy.\u003C\u002Fp>\n\n\n\n\u003Ch2 class=\"wp-block-heading\" id=\"rounding-up-robots-txt\">Rounding up robots.txt\u003C\u002Fh2>\n\n\n\n\u003Cp>Robots.txt is a powerful tool that needs to be handled carefully. Typically, you’re safe to use the file to avoid user agents crawling unimportant content but even then, the definition of “unimportant” becomes inherently problematic!\u003C\u002Fp>\n\n\n\n\u003Cp>Above all, it’s important to remember the implications it can have on your index presence and link equity. Always consider all the options available to you, such as canonicalisation, meta robots tagging or even authentication if the content you’re looking to deter search engines from is particularly sensitive or private.\u003C\u002Fp>\n\n\n\n\u003Cp>If you have any further thoughts about this post or when you think it’s best to use robots.txt, feel free to let me know on \u003Ca href=\"https:\u002F\u002Ftwitter.com\u002FPetejov\" target=\"_blank\" rel=\"noopener\">twitter\u003C\u002Fa>.\u003C\u002Fp>\n",false,"standard",[10,11],1,42,[],[14],143,[16,17,18,19,20,21,22,23,24],"post-4209","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo","category-technical-seo",{"page_content_modules":26,"hero":170},[27,124],{"acf_fc_layout":28,"reduced":29,"top_overlap":-1,"bottom_underlap":-1,"disable_observer":29,"title":30,"content":6,"authors":31},"blog_post_content",true,"Things to consider before adding a disallow to your robots.txt file",[32],{"id":33,"path":34,"display_name":35,"job_title":36,"description":37,"avatar_url":38,"acf":39},59,"\u002Fabout\u002Four-team\u002Fpetar-jovetic\u002F","Petar Jovetic","Executive Director - Organic Media","Executive Director - Organic Media at Impression, specialising in organic performance, data, content marketing, technical SEO and user experience.","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress&fit=crop&fm=jpg&h=150&q=80&w=150&bg=CFE6F9",{"full_bio":40,"job_title":36,"specialist_service_links":41,"photo":68},"\u003Cp>Executive Director – Organic Media. Joined Impression in July 2015.\u003C\u002Fp>\n\u003Cp>I am the Executive Director of Organic Media at Impression, overseeing our SEO, PR, Social and Influencer departments. As an experienced organic marketer, I have worked with hundreds of online businesses, helping them to improve their organic performance through strategies rooted in data, content marketing, technical SEO and user experience. These have included a mix of growing brands and larger, more established businesses nationally and internationally. I work closely with my team of Organic Directors to continually improve Impression’s organic product offering, ensuring this continues to foster long-lasting and prosperous relationships with our clients. I also work within Impression’s leadership and management teams, helping the agency scale while maintaining an industry-leading position in the market.\u003C\u002Fp>\n\u003Cp>\u003Cb>Something that I’m proud of:\u003Cbr \u002F>\n\u003C\u002Fb>Our organic teams, because honestly, they’re continually kicking ass! Right now, they’re the best they ever have been, and I’m so proud of what we’ve all collectively achieved. The exciting part is we’re not even close to being done. There are still many exciting developments on the horizon to further strengthen our services.\u003C\u002Fp>\n\u003Cp>\u003Cb>Outside of work:\u003Cbr \u002F>\n\u003C\u002Fb>Raising my two young boys. I also love listening to music, playing my guitar and writing and learning new songs. I also enjoy hogging the TV to play my PS5, much to my fiancée’s dismay.\u003C\u002Fp>\n",[42,59],{"page":43,"link_text":48},{"ID":44,"post_author":45,"post_date":46,"post_date_gmt":46,"post_content":47,"post_title":48,"post_excerpt":47,"post_status":49,"comment_status":50,"ping_status":50,"post_password":47,"post_name":51,"to_ping":47,"pinged":47,"post_modified":52,"post_modified_gmt":52,"post_content_filtered":47,"post_parent":53,"guid":54,"menu_order":53,"post_type":55,"post_mime_type":47,"comment_count":56,"filter":57,"url_path":58},35,"98","2020-03-12 15:02:03","","SEO","publish","closed","seo","2026-09-10 09:29:45",0,"\u002F?page_id=35","page","0","raw","\u002Fseo\u002F",{"page":60,"link_text":63},{"ID":61,"post_author":45,"post_date":62,"post_date_gmt":62,"post_content":47,"post_title":63,"post_excerpt":47,"post_status":49,"comment_status":50,"ping_status":50,"post_password":47,"post_name":64,"to_ping":47,"pinged":47,"post_modified":65,"post_modified_gmt":65,"post_content_filtered":47,"post_parent":53,"guid":66,"menu_order":53,"post_type":55,"post_mime_type":47,"comment_count":56,"filter":57,"url_path":67},37897,"2026-04-13 11:44:52","PR & Social","pr-social","2026-06-30 09:19:16","\u002F?page_id=37897","\u002Fpr-social\u002F",{"ID":69,"id":69,"title":70,"filename":71,"filesize":72,"url":73,"link":74,"alt":47,"author":75,"description":47,"caption":47,"name":76,"status":77,"uploaded_to":53,"date":78,"modified":78,"menu_order":53,"mime_type":79,"type":80,"subtype":81,"icon":82,"width":83,"height":84,"sizes":85},24289,"Pete-Jovetic-NEW","Pete-Jovetic-NEW.png",274057,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&q=90","\u002Fpete-jovetic-new\u002F","12","pete-jovetic-new","inherit","2023-08-02 10:44:57","image\u002Fpng","image","png","\u002Fwp\u002Fwp-includes\u002Fimages\u002Fmedia\u002Fdefault.png",696,600,{"thumbnail":86,"thumbnail-width":87,"thumbnail-height":87,"medium":88,"medium-width":89,"medium-height":90,"medium_large":91,"medium_large-width":92,"medium_large-height":93,"large":94,"large-width":95,"large-height":96,"1536x1536":97,"1536x1536-width":98,"1536x1536-height":99,"2048x2048":100,"2048x2048-width":101,"2048x2048-height":102,"post-thumbnail":103,"post-thumbnail-width":104,"post-thumbnail-height":105,"12_col":106,"12_col-width":107,"12_col-height":108,"default_landscape":109,"default_landscape-width":110,"default_landscape-height":84,"default_portrait":111,"default_portrait-width":84,"default_portrait-height":110,"6_col_and_margin_short":112,"6_col_and_margin_short-width":104,"6_col_and_margin_short-height":108,"11_col_and_margin":113,"11_col_and_margin-width":114,"11_col_and_margin-height":108,"9_col_rectangle":115,"9_col_rectangle-width":116,"9_col_rectangle-height":117,"11_col":118,"11_col-width":119,"11_col-height":108,"staff_image_696_600":120,"staff_image_696_600-width":83,"staff_image_696_600-height":84,"rectangular_thumbnail":121,"rectangular_thumbnail-width":122,"rectangular_thumbnail-height":123},"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=640&q=90&w=640","640","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=scale&h=172&q=90&w=200",200,172,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=scale&h=662&q=90&w=768",768,662,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=1000&q=90&w=1892","1892","1000","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=scale&h=1324&q=90&w=1536",1536,1324,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=scale&h=1766&q=90&w=2048",2048,1766,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=1180&q=90&w=1860",1860,1180,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=870&q=90&w=2880",2880,870,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=600&q=90&w=800",800,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=800&q=90&w=600","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=870&q=90&w=1860","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=870&q=90&w=3080",3080,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=1400&q=90&w=2150",2150,1400,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=870&q=90&w=2640",2640,"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=600&q=90&w=696","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress%2Cformat&fit=crop&h=300&q=90&w=750",750,300,{"acf_fc_layout":125,"reduced":29,"top_overlap":126,"bottom_underlap":126,"disable_observer":-1,"heading":127,"cta_text":128,"cta_link":129,"items":130},"related_module","small","See more posts","See all articles","\u002Fblog\u002F",[131,144,153],{"type_key":17,"title":132,"excerpt":133,"to":134,"postedAt":135,"readingTime":136,"image":137,"authors":138,"category":142},"August 2026 Google algorithm and search industry updates","\u003Cp>This month, we bring you some exciting developments and updates from the world of search, with a clear focus on changes made to Google’s AI Overviews, the personalisation of search, and the way AIOs are affecting zero-click behaviour. The start of a large shift, and a slightly controversial one, is happening in the world of […]\u003C\u002Fp>\n","\u002Fblog\u002Faugust-2026-google-algorithm-and-search-industry-updates\u002F","2026-09-09T10:48:44",5,"https:\u002F\u002Fimages.impression.co.uk\u002F2026\u002F09\u002Ft-i-m-e-l-o-r-d-eCC0dhkg_pY-unsplash.jpg?auto=compress%2Cformat&fit=crop&h=640&q=90&w=640",[139],{"name":140,"image":141},"Scott Lynam","https:\u002F\u002Fimages.impression.co.uk\u002F2025\u002F05\u002FScott-Lynam.png?auto=compress&fit=crop&fm=jpg&h=150&q=80&w=150&bg=CFE6F9",{"to":143,"name":48},"\u002Fblog\u002Fcategory\u002Fseo\u002F",{"type_key":17,"title":145,"excerpt":146,"to":147,"postedAt":148,"readingTime":136,"image":149,"authors":150,"category":152},"July 2026 Google Algorithm and Search Industry Updates","\u003Cp>This month, we bring you some exciting developments and updates from the world of search, with a clear focus on tailoring AI search results in Google and an upgrade to WordPress e-commerce websites. The start of a large shift, and a slightly controversial one, is happening in e-commerce: Google has started initial tests for AI-generated […]\u003C\u002Fp>\n","\u002Fblog\u002Fjuly-2026-google-algorithm-and-search-industry-updates\u002F","2026-08-04T14:15:43","https:\u002F\u002Fimages.impression.co.uk\u002F2026\u002F08\u002Frubaitul-azad-58HvxSq7rrU-unsplash.jpg?auto=compress%2Cformat&fit=crop&h=640&q=90&w=640",[151],{"name":140,"image":141},{"to":143,"name":48},{"type_key":17,"title":154,"excerpt":155,"to":156,"postedAt":157,"readingTime":158,"image":159,"authors":160,"category":167},"How information gain solves commodity SEO","\u003Cp>When building informational assets as an SEO professional, how can you distinguish your brand when so many articles often mirror one another? The answer lies in information gain. By moving away from replication and toward the creation of unique value, you provide both Google and users with a compelling reason to prefer your brand over […]\u003C\u002Fp>\n","\u002Fblog\u002Fhow-information-gain-solves-commodity-seo\u002F","2026-07-20T11:37:53",12,"https:\u002F\u002Fimages.impression.co.uk\u002F2026\u002F07\u002Fgetty-images-lwAc6CL9tlM-unsplash.jpg?auto=compress%2Cformat&fit=crop&h=640&q=90&w=640",[161,164],{"name":162,"image":163},"Rhianne Moriarty","https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F05\u002FOxCBEwDu-Rhianne-Moriaty.png?auto=compress&fit=crop&fm=jpg&h=150&q=80&w=150&bg=CFE6F9",{"name":165,"image":166},"Milly Green","https:\u002F\u002Fimages.impression.co.uk\u002F2025\u002F02\u002FMilly-Green.png?auto=compress&fit=crop&fm=jpg&h=150&q=80&w=150&bg=CFE6F9",{"to":168,"name":169},"\u002Fblog\u002Fcategory\u002Ftechnical-seo\u002F","Technical SEO",{"type":171,"style":172,"heading":30,"date":4,"updatedAt":173,"reading_minutes":174,"image":175,"imageAlt":30,"categories":176,"collections":444},"blog-post","light","2025-05-07T16:01:05",7,"https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F07\u002FQBAX7xiT-050720211625513367_robots.png?auto=compress%2Cformat&fit=crop&h=1180&q=90&w=1860",[177,359],{"id":10,"count":178,"description":179,"link":143,"name":48,"slug":51,"taxonomy":180,"parent":53,"meta":181,"acf":182,"head_links":183,"resource_count":184,"yoast_title":185,"yoast_meta":186,"yoast_json_ld":231,"_links":328,"path":143},431,"\u003Cp class=\"p1\">Looking for the latest news in the world of SEO? Then you’ve come to the right place. We regularly publish blogs on the announcements, updates, and strategies that are shaking up the industry.\u003C\u002Fp>\r\n\r\n\u003Cp class=\"p1\">Find out more about our \u003Ca href=\"\u002Fseo\u002F\">SEO services\u003C\u002Fa>, or browse our blog posts below for our latest tips, advice and guides. If you’re interested, why not check out our \u003Ca href=\"\u002Four-work\u002Fservice\u002Fseo\u002F\">SEO case studies\u003C\u002Fa> too?\u003C\u002Fp>","category",[],[],[],27,"SEO Blog | SEO Tips, Advice & Guides | Impression",[187,189,192,195,198,201,204,206,208,210,213,216,219,222,225,228],{"property":188,"content":185},"title",{"property":190,"content":191},"description","Keeping up to date with the latest SEO trends, strategies, and announcements can be difficult. Browse our tips, advice and opinions in our latest SEO blogs!",{"name":193,"content":194},"robots","index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1",{"property":196,"content":197},"canonical","https:\u002F\u002Fadmin.impressiondigital.com\u002Fblog\u002Fcategory\u002Fseo\u002F",{"name":199,"content":200},"og:locale","en-GB",{"property":202,"content":203},"og:type","article",{"property":205,"content":185},"og:title",{"property":207,"content":191},"og:description",{"property":209,"content":197},"og:url",{"property":211,"content":212},"og:site_name","Impression",{"property":214,"content":215},"og:image","https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F06\u002FABFPXguH-Opengraph-BLUE.jpg?auto=compress%2Cformat&q=90",{"property":217,"content":218},"og:image:width",1200,{"property":220,"content":221},"og:image:height",630,{"property":223,"content":224},"og:image:type","image\u002Fjpeg",{"property":226,"content":227},"twitter:card","summary_large_image",{"property":229,"content":230},"twitter:site","@impressiontalk",{"@context":232,"@graph":233},"https:\u002F\u002Fschema.org",[234,242,251,261],{"@type":235,"@id":236,"url":236,"name":185,"isPartOf":237,"description":191,"breadcrumb":239,"inLanguage":241},"CollectionPage","https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fcategory\u002Fseo\u002F",{"@id":238},"https:\u002F\u002Fwww.impressiondigital.com\u002F#website",{"@id":240},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fcategory\u002Fseo\u002F#breadcrumb","en",{"@type":243,"@id":240,"itemListElement":244},"BreadcrumbList",[245,249],{"@type":246,"position":10,"name":247,"item":248},"ListItem","Home","https:\u002F\u002Fwww.impressiondigital.com\u002F",{"@type":246,"position":250,"name":48},2,{"@type":252,"@id":238,"url":248,"name":212,"description":47,"publisher":253,"inLanguage":200,"potentialAction":255},"WebSite",{"@id":254},"https:\u002F\u002Fwww.impressiondigital.com\u002F#organization",{"@type":256,"target":257,"query-input":260},"SearchAction",{"@type":258,"urlTemplate":259},"EntryPoint","https:\u002F\u002Fwww.impressiondigital.com\u002Fsearch?q={search_term_string}","required name=search_term_string",{"@type":262,"@id":254,"name":212,"url":248,"logo":263,"image":269,"sameAs":270,"@context":274,"email":275,"foundingDate":276,"legalName":277,"vatID":278,"description":279,"telephone":280,"address":281,"location":288,"areaServed":321},"Organization",{"@type":264,"inLanguage":200,"@id":265,"url":266,"contentUrl":266,"width":267,"height":268,"caption":212},"ImageObject","https:\u002F\u002Fwww.impressiondigital.com\u002F#\u002Fschema\u002Flogo\u002Fimage\u002F","https:\u002F\u002Fcdn.impression.co.uk\u002F2021\u002F06\u002FA5LuTDrq-Impression_Wordmark.png",920,176,"https:\u002F\u002Fcdn.impression.co.uk\u002F2021\u002F04\u002FFAVxDeOS-Nottingham.jpg",[271,272,273],"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fimpression-digital-limited","https:\u002F\u002Fwww.facebook.com\u002Fimpression\u002F","https:\u002F\u002Fwww.instagram.com\u002Fimpressiontalk\u002F","https:\u002F\u002Fwww.schema.org\u002F","hello@impressiondigital.com","2012-11-07","Impression Digital Limited","GB186363774","Impression is an award-winning digital marketing agency based in the UK and USA. We offer SEO agency, PPC agency, Digital PR agency and Analytics services to companies of all sizes across the UK, North America, Europe and beyond. Get in touch with us if you think we can work together.","01158 242 212",{"@type":282,"streetAddress":283,"addressLocality":284,"addressRegion":285,"postalCode":286,"addressCountry":287},"PostalAddress","Fothergill House, 16 King Street","Nottingham","Nottinghamshire","NG1 2AS","United Kingdom",[289,298,310],{"@type":290,"name":291,"geo":292,"address":296,"photo":269,"telephone":297},"Place","Impression Digital - Nottingham",{"@type":293,"latitude":294,"longitude":295},"GeoCoordinates","52.95501401025914","-1.1494887460285232",{"@type":282,"streetAddress":283,"addressLocality":284,"addressRegion":285,"postalCode":286,"addressCountry":287},"+44 01158 242 212",{"@type":290,"name":299,"geo":300,"address":303,"photo":308,"telephone":309},"Impression Digital - London",{"@type":293,"latitude":301,"longitude":302},"51.50237208377361","-0.09229841534284115",{"@type":282,"streetAddress":304,"addressLocality":305,"addressRegion":306,"postalCode":307,"addressCountry":287},"201 Borough High St","Southwark","London","SE1 1JA","https:\u002F\u002Fcdn.impression.co.uk\u002F2021\u002F04\u002FwD0pPVWz-London.jpg","+44 020 7112 8148",{"@type":290,"name":311,"geo":312,"address":315},"Impression Digital - New York",{"@type":293,"latitude":313,"longitude":314},"40.726722746949","-74.00527778220652",{"@type":282,"streetAddress":316,"addressLocality":317,"addressRegion":318,"postalCode":319,"addressCountry":320},"160 Varick St","New York","NY","10013","USA",[322,325,327],{"@type":323,"name":324},"Country","Worldwide",{"@type":323,"name":326},"GBR",{"@type":323,"name":320},{"self":329,"collection":338,"about":341,"wp:post_type":344,"curies":355},[330],{"href":331,"targetHints":332},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fcategories\u002F1",{"allow":333},[334,335,336,337],"GET","POST","PUT","PATCH",[339],{"href":340},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fcategories",[342],{"href":343},"\u002Fwp-json\u002Fwp\u002Fv2\u002Ftaxonomies\u002Fcategory",[345,347,349,351,353],{"href":346},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fposts?categories=1",{"href":348},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fwhite_papers?categories=1",{"href":350},"\u002Fwp-json\u002Fwp\u002Fv2\u002Ftools?categories=1",{"href":352},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fwebinars?categories=1",{"href":354},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fevents?categories=1",[356],{"name":357,"href":358,"templated":29},"wp","https:\u002F\u002Fapi.w.org\u002F{rel}",{"id":11,"count":360,"description":361,"link":168,"name":169,"slug":362,"taxonomy":180,"parent":53,"meta":363,"acf":364,"head_links":365,"resource_count":136,"yoast_title":366,"yoast_meta":367,"yoast_json_ld":387,"_links":420,"path":168},34,"\u003Cp class=\"p1\">Here at Impression, ongoing research and training is a huge part of how we keep our team up-to-date with the latest happenings in the world of technical SEO. We regularly share our thoughts on the latest digital news right here on our blog, so you can keep up to date too!\u003C\u002Fp>\r\n\r\n\u003Cp class=\"p1\">Find out more about our \u003Ca href=\"\u002Fseo\u002Ftechnical-seo\u002F\">Technical SEO services\u003C\u002Fa>, or browse our latest tips, advice and opinions below.\u003C\u002Fp>","technical-seo",[],[],[],"Technical SEO Blog | Improve your Crawling & Indexing with Impression",[368,369,371,372,374,376,377,378,379,380,381,382,383,384,385,386],{"property":188,"content":366},{"property":190,"content":370},"Keeping up to date with the latest tech SEO trends, strategies, and industry announcements can be difficult. Browse our tips and advice in our latest blogs!",{"name":193,"content":194},{"property":196,"content":373},"https:\u002F\u002Fadmin.impressiondigital.com\u002Fblog\u002Fcategory\u002Ftechnical-seo\u002F",{"name":199,"content":375},"en_GB",{"property":202,"content":203},{"property":205,"content":366},{"property":207,"content":370},{"property":209,"content":373},{"property":211,"content":212},{"property":214,"content":215},{"property":217,"content":218},{"property":220,"content":221},{"property":223,"content":224},{"property":226,"content":227},{"property":229,"content":230},{"@context":232,"@graph":388},[389,394,398,402],{"@type":235,"@id":390,"url":390,"name":366,"isPartOf":391,"description":370,"breadcrumb":392,"inLanguage":241},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fcategory\u002Ftechnical-seo\u002F",{"@id":238},{"@id":393},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fcategory\u002Ftechnical-seo\u002F#breadcrumb",{"@type":243,"@id":393,"itemListElement":395},[396,397],{"@type":246,"position":10,"name":247,"item":248},{"@type":246,"position":250,"name":169},{"@type":252,"@id":238,"url":248,"name":212,"description":47,"publisher":399,"inLanguage":200,"potentialAction":400},{"@id":254},{"@type":256,"target":401,"query-input":260},{"@type":258,"urlTemplate":259},{"@type":262,"@id":254,"name":212,"url":248,"logo":403,"image":269,"sameAs":404,"@context":274,"email":275,"foundingDate":276,"legalName":277,"vatID":278,"description":279,"telephone":280,"address":405,"location":406,"areaServed":416},{"@type":264,"inLanguage":200,"@id":265,"url":266,"contentUrl":266,"width":267,"height":268,"caption":212},[271,272,273],{"@type":282,"streetAddress":283,"addressLocality":284,"addressRegion":285,"postalCode":286,"addressCountry":287},[407,410,413],{"@type":290,"name":291,"geo":408,"address":409,"photo":269,"telephone":297},{"@type":293,"latitude":294,"longitude":295},{"@type":282,"streetAddress":283,"addressLocality":284,"addressRegion":285,"postalCode":286,"addressCountry":287},{"@type":290,"name":299,"geo":411,"address":412,"photo":308,"telephone":309},{"@type":293,"latitude":301,"longitude":302},{"@type":282,"streetAddress":304,"addressLocality":305,"addressRegion":306,"postalCode":307,"addressCountry":287},{"@type":290,"name":311,"geo":414,"address":415},{"@type":293,"latitude":313,"longitude":314},{"@type":282,"streetAddress":316,"addressLocality":317,"addressRegion":318,"postalCode":319,"addressCountry":320},[417,418,419],{"@type":323,"name":324},{"@type":323,"name":326},{"@type":323,"name":320},{"self":421,"collection":427,"about":429,"wp:post_type":431,"curies":442},[422],{"href":423,"targetHints":424},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fcategories\u002F42",{"allow":425},[334,335,336,337,426],"DELETE",[428],{"href":340},[430],{"href":343},[432,434,436,438,440],{"href":433},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fposts?categories=42",{"href":435},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fwhite_papers?categories=42",{"href":437},"\u002Fwp-json\u002Fwp\u002Fv2\u002Ftools?categories=42",{"href":439},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fwebinars?categories=42",{"href":441},"\u002Fwp-json\u002Fwp\u002Fv2\u002Fevents?categories=42",[443],{"name":357,"href":358,"templated":29},[],[],[447,449,451,452],{"path":448,"name":247},"\u002F",{"path":129,"name":450},"Blog",{"path":143,"name":48},{"path":453,"name":30},null,[455,456],{"id":10,"name":48,"path":143},{"id":11,"name":169,"path":168},{"title":458,"script":459,"meta":532,"link":558},"Robots.txt for SEO: Things To Consider Before Disallowing - Impression",[460],{"type":461,"key":462,"innerHTML":463},"application\u002Fld+json","yoast-json-ld",{"@context":232,"@graph":464},[465,481,489,492,501,505,523],{"@type":466,"@id":467,"isPartOf":468,"author":470,"headline":30,"datePublished":472,"dateModified":473,"mainEntityOfPage":474,"wordCount":117,"publisher":475,"image":477,"thumbnailUrl":479,"articleSection":480,"inLanguage":200},"Article","https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fthings-consider-before-adding-disallow-robots-txt-file\u002F#article",{"@id":469},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fthings-consider-before-adding-disallow-robots-txt-file\u002F",{"name":35,"@id":471},"https:\u002F\u002Fadmin.impressiondigital.com\u002F#\u002Fschema\u002Fperson\u002Fbee0a5f3728d9ea70a88abf24a323c8e","2021-11-05T12:00:00+00:00","2025-05-07T16:01:05+00:00",{"@id":469},{"@id":476},"https:\u002F\u002Fadmin.impressiondigital.com\u002F#organization",{"@id":478},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fthings-consider-before-adding-disallow-robots-txt-file\u002F#primaryimage","https:\u002F\u002Fimages.impression.co.uk\u002F2021\u002F07\u002FQBAX7xiT-050720211625513367_robots.png?auto=compress%2Cformat&q=90",[48,169],{"@type":482,"@id":469,"url":469,"name":458,"isPartOf":483,"primaryImageOfPage":484,"image":485,"thumbnailUrl":479,"datePublished":472,"dateModified":473,"description":486,"breadcrumb":487,"inLanguage":241},"WebPage",{"@id":238},{"@id":478},{"@id":478},"It's important to understand the intracies of the robots.txt file before using it in your optimisations. Here are some things to consider before disallowing.",{"@id":488},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002Fthings-consider-before-adding-disallow-robots-txt-file\u002F#breadcrumb",{"@type":264,"inLanguage":200,"@id":478,"url":479,"contentUrl":479,"width":490,"height":491},2004,921,{"@type":243,"@id":488,"itemListElement":493},[494,495,497,499],{"@type":246,"position":10,"name":247,"item":248},{"@type":246,"position":250,"name":450,"item":496},"https:\u002F\u002Fwww.impressiondigital.com\u002Fblog\u002F",{"@type":246,"position":498,"name":48,"item":236},3,{"@type":246,"position":500,"name":30},4,{"@type":252,"@id":238,"url":248,"name":212,"description":47,"publisher":502,"inLanguage":200,"potentialAction":503},{"@id":254},{"@type":256,"target":504,"query-input":260},{"@type":258,"urlTemplate":259},{"@type":262,"@id":254,"name":212,"url":248,"logo":506,"image":269,"sameAs":507,"@context":274,"email":275,"foundingDate":276,"legalName":277,"vatID":278,"description":279,"telephone":280,"address":508,"location":509,"areaServed":519},{"@type":264,"inLanguage":200,"@id":265,"url":266,"contentUrl":266,"width":267,"height":268,"caption":212},[271,272,273],{"@type":282,"streetAddress":283,"addressLocality":284,"addressRegion":285,"postalCode":286,"addressCountry":287},[510,513,516],{"@type":290,"name":291,"geo":511,"address":512,"photo":269,"telephone":297},{"@type":293,"latitude":294,"longitude":295},{"@type":282,"streetAddress":283,"addressLocality":284,"addressRegion":285,"postalCode":286,"addressCountry":287},{"@type":290,"name":299,"geo":514,"address":515,"photo":308,"telephone":309},{"@type":293,"latitude":301,"longitude":302},{"@type":282,"streetAddress":304,"addressLocality":305,"addressRegion":306,"postalCode":307,"addressCountry":287},{"@type":290,"name":311,"geo":517,"address":518},{"@type":293,"latitude":313,"longitude":314},{"@type":282,"streetAddress":316,"addressLocality":317,"addressRegion":318,"postalCode":319,"addressCountry":320},[520,521,522],{"@type":323,"name":324},{"@type":323,"name":326},{"@type":323,"name":320},{"@type":524,"@id":525,"name":35,"image":526,"description":37,"sameAs":529,"url":531},"Person","https:\u002F\u002Fwww.impressiondigital.com\u002F#\u002Fschema\u002Fperson\u002Fbee0a5f3728d9ea70a88abf24a323c8e",{"@type":264,"inLanguage":200,"@id":527,"url":528,"contentUrl":528,"caption":35},"https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress&fit=fit&fm=jpg&h=96&q=90&w=962a3fd5db6f3dc00862f90bacced770aa","https:\u002F\u002Fimages.impression.co.uk\u002F2023\u002F08\u002FPete-Jovetic-NEW.png?auto=compress&fit=fit&fm=jpg&h=96&q=90&w=96",[530],"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fpetarjovetic\u002F","https:\u002F\u002Fwww.impressiondigital.com\u002Fabout\u002Four-team\u002Fpetar-jovetic\u002F",[533,534,535,536,537,538,539,540,541,544,546,548,549,550,551,552,554,555,557],{"name":190,"content":486},{"name":193,"content":194},{"property":199,"content":375},{"property":202,"content":203},{"property":205,"content":458},{"property":207,"content":486},{"property":209,"content":469},{"property":211,"content":212},{"property":542,"content":543},"article:publisher","https:\u002F\u002Fwww.facebook.com\u002Fimpression",{"property":545,"content":472},"article:published_time",{"property":547,"content":473},"article:modified_time",{"property":214,"content":479},{"property":217,"content":490},{"property":220,"content":491},{"property":223,"content":79},{"name":553,"content":35},"author",{"name":226,"content":227},{"name":556,"content":230},"twitter:creator",{"name":229,"content":230},[]]