{
"$type": "site.standard.document",
"content": {
"$type": "site.standard.content.markdown",
"text": "\n\n[DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/) is an object storage service that is S3 compatible (you can use all the AWS S3\nlibraries, which is nifty) with a CDN automatically built in. It even lets you bring your own domain and will handle\nthe SSL certificates for you through [Let's Encrypt](https://letsencrypt.org/). Based upon my experience with Gatsby and S3, this seemed like\na great fit and since I'm already a DigitalOcean customer and figured I might as well give it a shot instead of signing\nup for another service.\n\nLong story short, I was able to get it working, but I ended up ditching it to go with [Netlify](https://www.netlify.com/). Still, if you\nare here, you might have a do or die situation where you must host your site on Spaces and maybe this will be useful.\nAlso, it's a fun story for me to tell!\n\n## How to deploy Gatsby to DigitalOcean Spaces\n\nYou will build and deploy your site in the standard Gatsby way:\n\n1. Build your Gatsby site using `gatsby build --prefix-paths`\n - Set the `PREFIX_PATHS` environment variable to what your CDN URL is.\n - Mine is `https://cdn.eligundry.com/site` because I host the site in a folder in the bucket.\n2. Deploy your site using [gatsby-plugin-s3][gatsby-plugin-s3].\n - This plugin will correctly set the cache headers on your files for optimal performance.\n - You will need to provide the option `customAwsEndpointHostname` set to the endpoint provided to you in your\n DigitalOcean Console. Mine was `nyc3.digitaloceanspaces.com`.\n\nAfter you do this, your site's files will be in your Spaces bucket, but it definitely will not be working. Read on for\nthe issues and how to fix them!\n\n## No Cloudfront or S3 Static Website Hosting Equivalent\n\nDigitalOcean Spaces is like if [S3][s3] and [Cloudfront][cloudfront] were merged together in a time machine that went back\nto when both were launched. It is object storage (S3) with a built in CDN (Cloudfront). As previously mentioned, Spaces\nis S3 compatible, which means you can use the AWS CLI/SDKs for S3 for bucket operations. And the CDN is pretty user\nfriendly in that it provides SSL automatically through Let's Encrypt and you can bring your own domain.\n\nBut, the similarities stop there. Spaces' object storage is missing a bunch of S3 features, like [object\nexpiration][s3-object-expiration], [a rich ecosystem of programmatic add-ons via Lambda][s3-lambda], and, most\nimportantly for our use case, [static website hosting][s3-static-website-hosting]. Gatsby likes to serve it's routes\nwithout an extension, which means that all paths end up being folders with an `index.html` in them for the page. S3's\nstatic website hosting will rewrite that request so that it will serve up `index.html` when a path is requested. It also\nhas other cool features that Gatsby can leverage, like error pages and [programmatic redirects][gatsby-redirect].\n\nSince Spaces has none of these features, I wanted to at least get the URL rewriting working. I already was running an\nnginx instance on a DigitalOcean VPS, so I wrote this quick 'n dirty config to reverse proxy the HTML files from the\nCDN.\n\n```nginx\nserver {\n listen 443 ssl http2;\n\n # I have an API that I serve on the same domain and it must come\n # first because of the rules that come after\n location ~ ^/api {\n proxy_pass http://api:8080;\n }\n\n # All these rules could probably be collapsed into one meta location.\n # I didn't have enough patience to really hammer that out, sue me.\n\n # All non html assets that are not on the CDN\n location / {\n resolver 1.1.1.1 [::1]:5353 valid=30s;\n set $cdn_url \"https://cdn.eligundry.com/site/\";\n proxy_pass $cdn_url;\n proxy_set_header Host \"cdn.eligundry.com\";\n # Without this, SSL handshakes fail?\n # https://stackoverflow.com/a/52199403\n proxy_ssl_server_name on;\n }\n\n # /index.html\n location = / {\n resolver 1.1.1.1 [::1]:5353 valid=30s;\n set $cdn_url \"https://cdn.eligundry.com/site/\";\n proxy_pass $cdn_url/index.html;\n proxy_set_header Host \"cdn.eligundry.com\";\n proxy_ssl_server_name on;\n }\n\n # All other HTML pages\n location ~ /(?<req_path>.+) {\n resolver 1.1.1.1 [::1]:5353 valid=30s;\n set $cdn_url \"https://cdn.eligundry.com/site/\";\n proxy_pass $cdn_url/$req_path/index.html;\n proxy_set_header Host \"cdn.eligundry.com\";\n proxy_ssl_server_name on;\n }\n}\n```\n\nBecause I built the site with `--prefix-paths`, all the HTML files being served point straight to the CDN making it\nunnecessary to reverse proxy those resources. In practice, the only requests hitting the reverse proxy would the\nrequests for HTML pages on first load. This is slightly slower and doesn't provide the benefits to the end user a normal\nCDN would provide via proximity, but at least it's working, so onward to the next issue!\n\n## No Automatic GZIP\n\nIf you want to get that sweet sweet 💯 from [Lighthouse][lighthouse], you need your content gzip'd. Unfortunately,\n[DigitalOcean Spaces' CDN does not support gzip'ing on the fly][no-gzip-issue] like Cloudfront does, so you have to precompress your\nassets as a part of the build. I used [`gatsby-plugin-zopfli`][gatsby-plugin-zopfli], which hooks into the Gatsby build\nprocess to compress the assets, with the following config:\n\n```javascript\n{\n resolve: 'gatsby-plugin-zopfli',\n options: {\n extensions: ['css', 'js', 'html', 'json']\n }\n}\n```\n\nThis will compress all your assets while leaving the original files intact. In order for routing to work, you need to\noverwrite the existing files with these compressed files. I used the following shell snippet in my deploy script.\n\n```bash\n$ for f in \"public/*.gz\"; do mv -v -- \"$f\" \"${f%.gz}\"; done\n```\n\nA quick note on the above snippet: If you plan on having this in a CI pipeline and want to leverage\n[Gatsby incremental builds][gatsby-incremental-builds], this will break that, as it will be unable to detect if files\nhave changed or not. Such are the sacrifices we make for speed.\n\nFinally, when you upload your files to DigitalOcean Spaces with [`gatsby-plugin-s3`][gatsby-plugin-s3], you will need to\nmanually set the `Content-Encoding: gzip` header via the `params` option. It took me a little bit to understand how to\ndo this so maybe this will help you:\n\n```javascript\n{\n resolve: 'gatsby-plugin-s3',\n options: {\n mergeCachingParams: true,\n params: {\n '**/**': {\n ACL: 'public-read',\n },\n '**/**.js': {\n ContentEncoding: 'gzip',\n },\n '**/**.css': {\n ContentEncoding: 'gzip',\n },\n '**/**.html': {\n ContentEncoding: 'gzip',\n },\n '**/**.json': {\n ContentEncoding: 'gzip',\n },\n },\n },\n}\n```\n\nWith these settings, I was able to deploy my site! I opened my site, popped open up the network inspector to ensure\neverything is gzip'd. I opened up a JS file and looked at the headers. Strange, there wasn't a `Content-Encoding: gzip`\nheader? Did this even work? I opened up the CDN console, purged the cache, redeployed 3 times (for luck), but still no\nheader. What was I doing wrong!?\n\nI was doing nothing wrong. In the network panel, when I hovered over the request size, I saw that the transfer size\nwas smaller than the resource size, just like it would if it was gzip'd.\n\n\n\nI did a little bit of Googling to figure out what's happening here and found this:\n\n> Due to a known issue, file metadata headers like Content-Encoding are not passed through the CDN. Metadata headers are\n> correctly set when fetching content directly from the origin.\n>\n> [DigitalOcean Docs](https://docs.digitalocean.com/products/spaces/how-to/set-file-metadata/)\n\nFrom what I can tell, the browser is treating this like gzip because of legacy quirks and is doing a content scan of the\nfile when it discovers it isn't actually a plain text `application/javascript` file. There is probably a trivial\nperformance hit here as it's harder than explicitly saying that it's a gzip'd file, but not enough to matter because\nI have this site running!\n\n## HTTP/2 Is Not Supported\n\nNow that my site was live and somewhat functional, I started digging into the network tab to make sure everything is\norder. I was greeted with this waterfall:\n\n\n\n**Those highlighted gray bars are bad.** They are connections that are stalling because of too many other connections.\nThis is happening because [DigitalOcean Spaces does not support HTTP/2][do-spaces-no-http2]. [People have been voting on\nadding this feature since 2018][pokemon-go-to-the-polls] and yet there is still no progress on this.\n\nYou can see how this is bad by how the grey bars become green as the bars starting with orange finish. For most web\nbrowsers, [only six HTTP/1.1 connections can be made to the same host at once][6-http1-connections]. Before HTTP/2,\ndevelopers used to use CDNs that could provide multiple domains to side step this issue, but I really don't want to go\nthat route as it isn't 2011 anymore.\n\nOne of the big benefits to HTTP/2 is that this limit is removed and common connection resources, like SSL handshakes,\ncan be shared between connections. **The fact that DigitalOcean Spaces does not support this in 2021 is going to prevent\nyou from getting the max score you could in Lighthouse.** With HTTP/3 around the corner, I am not optimistic in Spaces\nsupporting that anytime soon.\n\n# Results\n\n\n\nI was able to get a 100 from Lighthouse! But at what cost?\n\n\n\nThe site was still a little glitchy, I suspect because of the nginx routing I setup poorly and maybe some script loading\nissues related to scripts sometimes loading out of order because of HTTP/1.1 restrictions.\n\n**I decided that this was too much work.** As much as I love maintaining my own infrastructure and writing Github Actions,\nthis is a solved problem by [S3][s3], [Gatsby Cloud][gatsby-cloud], [Vercel][vercel], and [Netlify][netlify]. I've used Gatsby\nCloud and became frustrated with it (too \"convention over configuration\" for my tastes, though the build speeds are\nfast). I'm using Vercel and S3 at work and they are pretty good, but just to round out the list, I decided to host my\nsite through Netlify.\n\n## Netlify\n\nNetlify was so easy to setup! I had my site running in about 10 minutes and I got a slightly better Lighthouse score\nwithout any of the bugginess that I was seeing on DigitalOcean Spaces. And look at how much flatter and full of happy\ncolors this connection waterfall is (the time scale is roughly the same as the waterfall for DigitalOcean Spaces).\n\n\n\nAlso, how can you not enjoy their developer content!?\n\n<Tweet id=\"1362440545604415493\" />\n\n# Conclusion\n\nI still love DigitalOcean (I'm never leaving my cute Droplet for a soulless EC2 instance in Ohio), but I cannot\nrecommend DigitalOcean Spaces for static site hosting. For what it's worth, it doesn't appear that DigitalOcean\nrecommends it either. They have thousands of guides on how to do things, but none on hosting static sites here. Also,\nthere is this [support answer saying \"good idea, maybe our product team will look at it?\"][do-support-answer].\n\nBut, you should not use Spaces for anything related to static sites. It might be okay of simple object storage for an\napp (which seems like their target use case). Otherwise, avoid the like the plauge and use a service that knows Gatsby\ninside and out!\n\n[do-spaces]: https://www.digitalocean.com/products/spaces/\n[letsencrypt]: https://letsencrypt.org/\n[lighthouse]: https://developers.google.com/web/tools/lighthouse\n[gatsby-plugin-zopfli]: https://www.gatsbyjs.com/plugins/gatsby-plugin-zopfli/\n[gatsby-plugin-s3]: https://www.gatsbyjs.com/plugins/gatsby-plugin-s3/\n[gatsby-incremental-builds]: https://www.gatsbyjs.com/docs/reference/release-notes/v3.0/#incremental-builds-in-oss\n[do-support-answer]: https://www.digitalocean.com/community/questions/how-to-configure-a-static-website-from-a-do-space?answer=63554\n[s3]: https://aws.amazon.com/s3/\n[s3-object-expiration]: https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/\n[s3-lambda]: https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html\n[s3-static-website-hosting]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html\n[gatsby-redirect]: https://www.gatsbyjs.com/docs/reference/config-files/actions/#createRedirect\n[cloudfront]: https://aws.amazon.com/cloudfront/\n[gatsby-cloud]: https://www.gatsbyjs.com/products/cloud/\n[vercel]: https://vercel.com/\n[netlify]: https://www.netlify.com/\n[do-spaces-no-http2]: https://www.digitalocean.com/community/questions/spaces-cdn-http-2-support\n[pokemon-go-to-the-polls]: https://www.digitalocean.com/community/questions/spaces-cdn-http-2-support\n[6-http1-connections]: https://docs.pushtechnology.com/cloud/latest/manual/html/designguide/solution/support/connection_limitations.html\n[no-gzip-issue]: https://www.digitalocean.com/community/questions/how-do-i-enable-gzip-compression-on-spaces",
"version": "1.0"
},
"description": "You shouldn't use DigitalOcean Spaces to host a Gatsby site, but I had to do it to them for science.",
"path": "/blog/hosting-gatsby-on-digitalocean-spaces/",
"publishedAt": "2021-06-02T05:00:00.000Z",
"site": "https://eligundry.com",
"tags": [
"code"
],
"textContent": "DigitalOcean Spaces is an object storage service that is S3 compatible (you can use all the AWS S3\nlibraries, which is nifty) with a CDN automatically built in. It even lets you bring your own domain and will handle\nthe SSL certificates for you through Let's Encrypt. Based upon my experience with Gatsby and S3, this seemed like\na great fit and since I'm already a DigitalOcean customer and figured I might as well give it a shot instead of signing\nup for another service.\n\nLong story short, I was able to get it working, but I ended up ditching it to go with Netlify. Still, if you\nare here, you might have a do or die situation where you must host your site on Spaces and maybe this will be useful.\nAlso, it's a fun story for me to tell!\n\nHow to deploy Gatsby to DigitalOcean Spaces\n\nYou will build and deploy your site in the standard Gatsby way:\nBuild your Gatsby site using \nSet the environment variable to what your CDN URL is.\nMine is because I host the site in a folder in the bucket.\nDeploy your site using [gatsby-plugin-s3][gatsby-plugin-s3].\nThis plugin will correctly set the cache headers on your files for optimal performance.\nYou will need to provide the option set to the endpoint provided to you in your\n DigitalOcean Console. Mine was .\n\nAfter you do this, your site's files will be in your Spaces bucket, but it definitely will not be working. Read on for\nthe issues and how to fix them!\n\nNo Cloudfront or S3 Static Website Hosting Equivalent\n\nDigitalOcean Spaces is like if [S3][s3] and [Cloudfront][cloudfront] were merged together in a time machine that went back\nto when both were launched. It is object storage (S3) with a built in CDN (Cloudfront). As previously mentioned, Spaces\nis S3 compatible, which means you can use the AWS CLI/SDKs for S3 for bucket operations. And the CDN is pretty user\nfriendly in that it provides SSL automatically through Let's Encrypt and you can bring your own domain.\n\nBut, the similarities stop there. Spaces' object storage is missing a bunch of S3 features, like [object\nexpiration][s3-object-expiration], [a rich ecosystem of programmatic add-ons via Lambda][s3-lambda], and, most\nimportantly for our use case, [static website hosting][s3-static-website-hosting]. Gatsby likes to serve it's routes\nwithout an extension, which means that all paths end up being folders with an in them for the page. S3's\nstatic website hosting will rewrite that request so that it will serve up when a path is requested. It also\nhas other cool features that Gatsby can leverage, like error pages and [programmatic redirects][gatsby-redirect].\n\nSince Spaces has none of these features, I wanted to at least get the URL rewriting working. I already was running an\nnginx instance on a DigitalOcean VPS, so I wrote this quick 'n dirty config to reverse proxy the HTML files from the\nCDN.\n\nBecause I built the site with , all the HTML files being served point straight to the CDN making it\nunnecessary to reverse proxy those resources. In practice, the only requests hitting the reverse proxy would the\nrequests for HTML pages on first load. This is slightly slower and doesn't provide the benefits to the end user a normal\nCDN would provide via proximity, but at least it's working, so onward to the next issue!\n\nNo Automatic GZIP\n\nIf you want to get that sweet sweet 💯 from [Lighthouse][lighthouse], you need your content gzip'd. Unfortunately,\n[DigitalOcean Spaces' CDN does not support gzip'ing on the fly][no-gzip-issue] like Cloudfront does, so you have to precompress your\nassets as a part of the build. I used [][gatsby-plugin-zopfli], which hooks into the Gatsby build\nprocess to compress the assets, with the following config:\n\nThis will compress all your assets while leaving the original files intact. In order for routing to work, you need to\noverwrite the existing files with these compressed files. I used the following shell snippet in my deploy script.\n\nA quick note on the above snippet: If you plan on having this in a CI pipeline and want to leverage\n[Gatsby incremental builds][gatsby-incremental-builds], this will break that, as it will be unable to detect if files\nhave changed or not. Such are the sacrifices we make for speed.\n\nFinally, when you upload your files to DigitalOcean Spaces with [][gatsby-plugin-s3], you will need to\nmanually set the header via the option. It took me a little bit to understand how to\ndo this so maybe this will help you:\n\nWith these settings, I was able to deploy my site! I opened my site, popped open up the network inspector to ensure\neverything is gzip'd. I opened up a JS file and looked at the headers. Strange, there wasn't a \nheader? Did this even work? I opened up the CDN console, purged the cache, redeployed 3 times (for luck), but still no\nheader. What was I doing wrong!?\n\nI was doing nothing wrong. In the network panel, when I hovered over the request size, I saw that the transfer size\nwas smaller than the resource size, just like it would if it was gzip'd.\n\nI did a little bit of Googling to figure out what's happening here and found this:\n\nDue to a known issue, file metadata headers like Content-Encoding are not passed through the CDN. Metadata headers are\ncorrectly set when fetching content directly from the origin.\nDigitalOcean Docs\n\nFrom what I can tell, the browser is treating this like gzip because of legacy quirks and is doing a content scan of the\nfile when it discovers it isn't actually a plain text file. There is probably a trivial\nperformance hit here as it's harder than explicitly saying that it's a gzip'd file, but not enough to matter because\nI have this site running!\n\nHTTP/2 Is Not Supported\n\nNow that my site was live and somewhat functional, I started digging into the network tab to make sure everything is\norder. I was greeted with this waterfall:\n\nThose highlighted gray bars are bad. They are connections that are stalling because of too many other connections.\nThis is happening because [DigitalOcean Spaces does not support HTTP/2][do-spaces-no-http2]. [People have been voting on\nadding this feature since 2018][pokemon-go-to-the-polls] and yet there is still no progress on this.\n\nYou can see how this is bad by how the grey bars become green as the bars starting with orange finish. For most web\nbrowsers, [only six HTTP/1.1 connections can be made to the same host at once][6-http1-connections]. Before HTTP/2,\ndevelopers used to use CDNs that could provide multiple domains to side step this issue, but I really don't want to go\nthat route as it isn't 2011 anymore.\n\nOne of the big benefits to HTTP/2 is that this limit is removed and common connection resources, like SSL handshakes,\ncan be shared between connections. The fact that DigitalOcean Spaces does not support this in 2021 is going to prevent\nyou from getting the max score you could in Lighthouse. With HTTP/3 around the corner, I am not optimistic in Spaces\nsupporting that anytime soon.\n\nResults\n\nI was able to get a 100 from Lighthouse! But at what cost?\n\nThe site was still a little glitchy, I suspect because of the nginx routing I setup poorly and maybe some script loading\nissues related to scripts sometimes loading out of order because of HTTP/1.1 restrictions.\n\nI decided that this was too much work. As much as I love maintaining my own infrastructure and writing Github Actions,\nthis is a solved problem by [S3][s3], [Gatsby Cloud][gatsby-cloud], [Vercel][vercel], and [Netlify][netlify]. I've used Gatsby\nCloud and became frustrated with it (too \"convention over configuration\" for my tastes, though the build speeds are\nfast). I'm using Vercel and S3 at work and they are pretty good, but just to round out the list, I decided to host my\nsite through Netlify.\n\nNetlify\n\nNetlify was so easy to setup! I had my site running in about 10 minutes and I got a slightly better Lighthouse score\nwithout any of the bugginess that I was seeing on DigitalOcean Spaces. And look at how much flatter and full of happy\ncolors this connection waterfall is (the time scale is roughly the same as the waterfall for DigitalOcean Spaces).\n\nAlso, how can you not enjoy their developer content!?\n\nConclusion\n\nI still love DigitalOcean (I'm never leaving my cute Droplet for a soulless EC2 instance in Ohio), but I cannot\nrecommend DigitalOcean Spaces for static site hosting. For what it's worth, it doesn't appear that DigitalOcean\nrecommends it either. They have thousands of guides on how to do things, but none on hosting static sites here. Also,\nthere is this [support answer saying \"good idea, maybe our product team will look at it?\"][do-support-answer].\n\nBut, you should not use Spaces for anything related to static sites. It might be okay of simple object storage for an\napp (which seems like their target use case). Otherwise, avoid the like the plauge and use a service that knows Gatsby\ninside and out!\n\n[do-spaces]: https://www.digitalocean.com/products/spaces/\n[letsencrypt]: https://letsencrypt.org/\n[lighthouse]: https://developers.google.com/web/tools/lighthouse\n[gatsby-plugin-zopfli]: https://www.gatsbyjs.com/plugins/gatsby-plugin-zopfli/\n[gatsby-plugin-s3]: https://www.gatsbyjs.com/plugins/gatsby-plugin-s3/\n[gatsby-incremental-builds]: https://www.gatsbyjs.com/docs/reference/release-notes/v3.0/#incremental-builds-in-oss\n[do-support-answer]: https://www.digitalocean.com/community/questions/how-to-configure-a-static-website-from-a-do-space?answer=63554\n[s3]: https://aws.amazon.com/s3/\n[s3-object-expiration]: https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/\n[s3-lambda]: https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html\n[s3-static-website-hosting]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html\n[gatsby-redirect]: https://www.gatsbyjs.com/docs/reference/config-files/actions/#createRedirect\n[cloudfront]: https://aws.amazon.com/cloudfront/\n[gatsby-cloud]: https://www.gatsbyjs.com/products/cloud/\n[vercel]: https://vercel.com/\n[netlify]: https://www.netlify.com/\n[do-spaces-no-http2]: https://www.digitalocean.com/community/questions/spaces-cdn-http-2-support\n[pokemon-go-to-the-polls]: https://www.digitalocean.com/community/questions/spaces-cdn-http-2-support\n[6-http1-connections]: https://docs.pushtechnology.com/cloud/latest/manual/html/designguide/solution/support/connection_limitations.html\n[no-gzip-issue]: https://www.digitalocean.com/community/questions/how-do-i-enable-gzip-compression-on-spaces",
"title": "Hosting Gatsby on DigitalOcean Spaces",
"updatedAt": "2025-02-12T02:34:47.000Z"
}