{
  "$type": "site.standard.document",
  "canonicalUrl": "https://www.simoncox.com/post/2025-04-07-recovery-from-a-bing-de-indexing/",
  "description": "My personal site vanished from Bings index and it took quite a while to get it back in - here is what I found and did.",
  "path": "/post/2025-04-07-recovery-from-a-bing-de-indexing/",
  "publishedAt": "2025-04-07T00:00:00.000Z",
  "site": "at://did:plc:tki7vwlanxbwrz2er67eaeqa/site.standard.publication/3mp4h4md7zv2y",
  "tags": [
    "Web",
    "SEO"
  ],
  "textContent": "This has been quite a journey and as professional Technical SEO, quite the challenge.\n\nI have been using Bing Webmaster Tools for many years, mostly as a confirmation tool when comparing issues with Google Search Console. As Bing tends to only provide 5% or less of organic search results it hasn't been that important to keep track of whats going on. \n\nWhy Bing is becoming more important\nHowever with the introduction of Large Language Models (LLMs) and the Bing index being used to populate various AI tools, it has become more important that your site is being indexed by Bing. Bing is also used by some other search engines to complement their own databases DuckDuck go being one of these.\n\nSomething's up  \nIn January 2025 I checked how my personal website was doing and there wasn't a great deal of impressions or clicks - which is fairly normal. I checked the home page in the URL inspection tool and realised there was a bit of an issue!\n\nThe dreaded note: \"The inspected url is known to bing but has some issues which are preventing indexation. We recommend you follow Bing Webmaster Guidelines to increase your chances of indexation.\" This really is as much information Bing will give you about the error on your website that is preventing indexing. You are on your own finding out where the issue is.\n\nIt should normally have a couple of green ticks stating that the page is Indexed successfully, URL can appear on Bing and No SEO issue found - that's where we need to be.\n\nI checked the Live Page tab:\n\t\t\t\t\t\n\nOdd - the URL can be indexed by Bing.\n\nThe Crawl and Indexing report\nI checked the Crawl and indexing report which is in Search Performance last position in the Filter by drop down menu.\n\t\t\noh dear me. At this point breathing excersises come in handy.\n\n\t\t\nSure there were some crawl errors but that didn't correlate with the drop in August 2024 from 120 pages index to zero in a few weeks. \n\nSite check\nSo I did a site check in Bing Search: and sure enough there were no results in the Search Engine Results Page (SERP). None of my sites pages were in the index at all - disaster! \n\t\t\n\nSite scans\nBing Webmaster tools allows you to run and keep, up top 5 separate Site Scans that are useful for troubleshooting.\n\t\n\nFive http 500 errors was concerning.\n\t\n\nThe information on what actual errors were occurring and which URLs just sin't provided. None of the crawls I ran, using Sitebulb or Screaming Frog, were giving me any errors at all so I could only put this down to Bing crawler errors at the time of running. Five URLs with server errors would not course the whole site to be de-indexed though, would they?\n\nManually adding pages to the index\nI began trying to resolve this by manually adding the pages using the URL Inspection and Requesting indexing. This should normally work, and is good practice when you publish a new page. \n\t\t\n\nI did this for 37 pages and then left it for a week or so and then checked to see if they had been indexed. No they had not.\n\t\t\n\nIn the IndexNow section in indexing Insights I was able to find some details.\nIt appeared the issue was Content Quality.\n\nAnd yes I did read the Bing Webmaster  and no they were no help as they were as ambiguous as you would expect, as I knew they would be. They have to cover a huge amount of potential situations without giving too much away - because search engines change aspects of how they work a lot. \n\nContent quality\nContent quality is always a worry though. There isn't any AI written content on the site - you can tell that by my deplorable spelling mistakes.\n\t\n\nA while back I published a short, Cobbles of Mousehole, a few days before the start of the decline in the indexed pages. It contained several paragraphs lifted from online news articles and I may have been auto penalised for plagiarism, but if you read the piece it is a review of those articles. So I considered that I might have fallen foul of the Guidelines and replaced the live text with screenshots. I believed that my content was squeaky clean at that point.\n\nNot the first to have been de-indexed\nSo I had a look around the web and found that a lot of peoples websites had been de-indexed by Bing - this wasn't a niche issue that I had. Fortunately there was plenty of advice to hand on how to get your site re-indexed. The one that I found to be most helpful at this stage was \nDeindexed in Bing?.\n\n To summarise the tips for a quick for for Bing De-indexed sites:\n1. Confirm whether you have a Bing indexation problem\n2. Have a working XML sitemap file\n3. Check your robots.txt for weird issues\n4. Make sure your site 404s correctly\n5. Setup a Bing Webmaster Tools account\n6. Log a ticket with Bing support \n7. Check your Uptime + other technical SEO elements\n\nMy Results from that checklist:\n1. Yes I do have an issue - no pages are showing in Bing search.\n2. XML file works fine. Tested with SEO Testing XML Sitemap Validator\n3. Robots.txt validates in Bing webmaster tools.\n4. Seems to be ok - I got the correct responses in testing\n5. Already done.\n6. Not going to do that if I can help it.\n7. Uptime should be ok - Netlify is a CDN… Speed test all ok.\n\nWhen checking the 404 page I checked the headers and noticed there were three missing headers but on comparing with some other sites that I knew were not de-indexed there was no difference.\n\nSitemaps\nI deleted an old sitemap entry and resubmitted the main sitemap and also added the rss feed in the Submit Sitemap section.\n\nBlocking AI crawlers\nAt this time I was blocking all known AI crawlers using the ai.robots.txt list was this the problem. I knew Bing were testing Co-pilot with a view to adding it to search, and now has been launched, and did wonder if by blocking AI crawlers If might be running fowl of some terms and conditions that I had not found.\n\nI took those out of my robots.txt file, deployed and left it a week. No change.\n\nAt that point I thought nearly everything else has been ruled out. So it time to raise a Ticket with Bing.\n\nBing Help ticket\nSo I did raise a ticket with Webmasters Support. and explained what I had done and waited for their response.\n\nBing support replied\n\"After further review, it appears that your site https://www.simoncox.com/ did not meet the standards set by Bing the last time it was crawled. Testing: https://www.simoncox.com/post/2024-05-28-log-store-build/ Bing URL Inspection states: The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you follow Bing Webmaster Guidelines to increase your chances of indexation.\"\n\nI wasn't sure that their response was that helpful. \n\nMore changes and a switch in hosting\nI made some more subtle changes to the site and ran the Bing webmaster site scan several times to eliminate all issues. I was able to identify a 500 error while doing this so ended up switching the site from Netlify to Cloudflare hosting. This didn't take too long and I now have the site deploying to both so with a switch of the DNS, managed through hover.com, I can flick between them if needed. \n\nAfter the hosting migration the Bing Site scan gave me no errors. \n\nRedirect issues with Cloudflare\nHaving moved to Cloudflare I checked a few things and some redirects were not working. What is good is that the _redirect files works on both Netlify and Cloudflare, even the splats. Cloudflare state that you can have 2000 static and 100 dynamic redirects on the free plan - but anything after about 100 static didn’t work for me. So I had to work them into splats to get below the threshold. \n\nWith all that done I waited a week and again raised a support request.\nThis time I got a really quick reply:  \n\n“After further review, it appears that your site https://www.simoncox.com/ did not meet the standards set by Bing the last time it was crawled.” ...\n\nIndexNow\nOne feature that Bing has is IndexNow which allows a webmaster to get pages indexed almost instantaneously and shared with other Search Engines using the IndexNow system. Delightfully Cloudflare can do this for you automatically in their Crawler Hints - just switch it on. \n\nHaving moved the site to Cloudflare I can make use of their IndexNow feature!\nHad to shift theDNS to Cloudflare as well so that I could turn on Crawler Hints to get the IndexNow feature. Then had to turn on Custom domain in Cloudflare pages and reactivate the site as I think it got turned off when I switched the DNS.\n\nIn no time at all I had 839 URLs submitted to the index. \n\nWhat happened next was a surprise! There was a massive rise in errors - of course there would be as the pages were getting indexed but this got me thinking.\n\nHTML validation\nI checked the site on the W3C Nu HTML checker tool and I had the Fathom Analytics script inserted after the closing body tag and before the closing html tag. Years and years ago this was ok. Google thinks its ok - well it wasn't de-indexing the site so I presume it forgave me and got on with the job in hand.\n\nApparently that was a change in HTML 4.01 (1999) and I had forgotten. I looked back through my deployment sand I had switched the Fathom script from the head to the base of the html just before the site was de-indexed. Coincidence? Had I found the cause?\n\nI moved the Fathom code inside the body tag, ensured the html validated and then resubmitted all the page urls using the URL submission tool.\n\nFinally success and pages started to get indexed \nI kept a check on progress and suddenly the number of pages crawled started going up daily.\n\t\n\nThe site started appearing in the SERPs again!\nHere we go, good content for people to read and enjoy back in the SERPs where they belong.\n\t\t\n\ninteresting differences between the layouts depending on your search terms!\n\t\n\nConclusion\nIt is a bit of an odd one this. For years we have been bowing our heads to Google and its ways. Google have constantly stated that they really do not care if your web page is valid or not. (Though it really does matter if you break the head with a body tag.) This is probably for the best as there are many web pages out there that don't validate but still provide the user with a good experience. Bing however is a little more fussy and I applaud it for that. I like pages to be written with valid html as it helps other developers sort your code out when they need to and for people learning html to understand  how it works.\n\t\n\nI let my standards slip though and suffered for it (well 5% less people visiting wasn't really suffering but my fragile ego was complaining).\n\nAnd I was done. Or at least I thought I was...\n\nThe aftermath of the Bing de-indexing\nWhen I started writing this article a search on Bing with the site operator for my site was providing a full list of the pages. As of today there is nothing showing again - yet the webmaster tools is showing impressions and clicks as well as the pages being in the index. Google has some of my content ranking really well but since the initial euphoria of the pages appearing in the Bing SERP again I think the new site honeymoon period may be over as nothing is cutting the mustard for Bing. \n\nAgain I think I know what may have done it this time and have made a change and booked a test in for next week. I will update this article if I have fixed it again.\n\nI am Simon Cox. I really am.\nIn looking at the SERPs today I though that the new Copilot might shed some light on whats indexed on the site. I asked it to tell me about simoncox.com and it clearly mentioned Technical SEO but links to a different Simon Cox - who happens to be a Microsoft Regional Director... Am I being blacklisted now? A conspiracy tin foil hat has been ordered.  Back to the keyboard.\n\nUpdate 30th July 2025\nThe site is indexed according to Bing Webmaster tools but not showing in the SERPs.\nIt did come back in the SERPs for a couple of weeks in March, after I added IndexNow, and then vanished again in April. The Bing Webmaster team have escalated my support request and I have been waiting since.\n\n\t\n\n\t\n\nUpdate 19th November 2025\nAm I finally back in Bing's good books? \n\nI have spent an extraordinary amount of time trying fixes to the site, rebuilding some posts and even renaming a whole section in case the URL segment I was using caused issues with a reviewer, AI or human. \n\nWhat moved the needle - what was causing Bing to de-index my site?\nNo idea. All I was told is that the site had been reviewed and would be back in the SERP in a week or two as it got recrawled. Actually was a lot faster than that because, I assume, I am using Index Now in Cloudflare.\n\nYou can see that in February to March and then from April to November there is barely any impressions. In the first week of November the impressions start up again as the site is released back into the index. \n\n.",
  "title": "Recovery from a Bing de-indexing"
}