{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/javascript/bulk-request-google-search-index/",
  "description": "Learn how to programmatically request Google search indexing for multiple URLs using the Indexing API and NodeJS for faster reindexing.",
  "path": "/javascript/bulk-request-google-search-index/",
  "publishedAt": "2023-05-26T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "JavaScript",
    "API",
    "Web"
  ],
  "textContent": "Recently, I purchased a domain for this blog and migrated the content from\n[rednafi.github.io] to [rednafi.com]. This turned out to be a much bigger hassle than I\noriginally thought it'd be, mostly because, despite setting redirection for almost all the\nURLs from the previous domain to the new one and submitting the new [sitemap.xml] to the\nSearch Console, Google kept indexing the older domain. To make things worse, the search\nengine selected the previous domain as canonical, and no amount of manual requests were\nchanging the status in the last 30 days. Strangely, I didn't encounter this issue with Bing,\nas it reindexed the new site within a week after I submitted the sitemap file via their\nwebmaster panel.\n\nWhile researching this, one potential solution suggested that along with submitting the\nsitemap via [Google Search Console], I'd have to make individual indexing requests for each\nURL to encourage faster indexing. The problem is, I've got quite a bit of content on this\nsite, and it'll take forever for me to click through all the links and request indexing that\nway. Naturally, I looked for a way to do this programmatically. Luckily, I found out that\nthere's an [indexing API] that allows you to make bulk indexing requests programmatically.\nThis has one big advantage - Google [responds to API requests faster] than indexing requests\nwith sitemap submission.\n\nAll you've to do is:\n\n- List out the URLs that need to be indexed.\n- Fulfill the [prerequisites] and download the private key JSON file required to make\n  requests to the API. From the docs:\n\n    > _Every call to the Indexing API must be authenticated with an OAuth token that you get\n    > in exchange for your private key. Each token is good for a span of time. Google\n    > provides API client libraries to get OAuth tokens for a number of languages._\n\n    The private key file will look like this:\n\n    \n\n- Use an API client to make the requests.\n\nIn my case, this site's [sitemap.xml] lists out all the URLs as follows:\n\nHere's a NodeJS script that collects the URLs from sitemap.xml and makes requests to the\nindexing API:\n\nBefore executing the script, npm install googleapis and xml2js. Now running the script\nwill give you an output similar to this:\n\nHere, the getUrls function is defined to fetch the sitemap content from a specified URL,\nparse the XML content and extract the URLs. It uses the fetch function to retrieve the file,\nthen uses xml2js to parse the XML and extract the URLs from the result.\n\nThe script then initializes an authentication client using the imported private key and\nspecifies the required API scope. The authorize function is called to authenticate the\nclient and obtain access tokens. Inside the authorization callback, the script prepares the\nnecessary options for making API requests to the Google indexing API. It then calls the\ngetUrls function to fetch the URLs from the sitemap.xml file. For each URL, it updates\nthe options with the URL and makes a POST request to the Indexing API to request indexing.\nThe response from the API is then logged into the console.\n\nOne thing to keep in mind is that by default, the daily request quota per project is 200.\nBut you can [request more quota] if you need it.\n\n\n\n\n[rednafi.github.io]:\n    https://rednafi.github.io\n\n[rednafi.com]:\n    /\n\n[sitemap.xml]:\n    /sitemap.xml\n\n[google search console]:\n    https://search.google.com/search-console/about\n\n[indexing API]:\n    https://developers.google.com/search/apis/indexing-api/v3/quickstart\n\n[responds to API requests faster]:\n    https://developers.google.com/search/apis/indexing-api/v3/quickstart#sitemaps\n\n[prerequisites]:\n    https://developers.google.com/search/apis/indexing-api/v3/quickstart\n\n[request more quota]:\n    https://developers.google.com/search/apis/indexing-api/v3/quota-pricing",
  "title": "Bulk request Google search indexing with API"
}