{
"path": "/3m5fknwepe22y",
"site": "at://did:plc:rnpkyqnmsw4ipey6eotbdnnf/site.standard.publication/3lxn3sgnjpc2a",
"tags": [
"atproto",
"guide"
],
"$type": "site.standard.document",
"title": "A blob in the bucket",
"content": {
"$type": "pub.leaflet.content",
"pages": [
{
"id": "019a75c6-8736-7995-afa2-1cad32195941",
"$type": "pub.leaflet.pages.linearDocument",
"blocks": [
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.image",
"image": {
"$type": "blob",
"ref": {
"$link": "bafkreif5hpzzvpc3nlcmyjj46u62nlzfdh6axr7osml5sb3kjuzk367a4i"
},
"mimeType": "image/png",
"size": 69216
},
"aspectRatio": {
"width": 400,
"height": 225
}
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.blockquote",
"facets": [
{
"index": {
"byteEnd": 45,
"byteStart": 34
},
"features": [
{
"uri": "https://baileytownsend.dev/articles/s3-blob-store",
"$type": "pub.leaflet.richtext.facet#link"
}
]
}
],
"plaintext": "Originally posted on 8/12/2025 on my old blog"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 198,
"byteStart": 163
},
"features": [
{
"uri": "https://upcloud.com/products/object-storage/#managed-object-storage",
"$type": "pub.leaflet.richtext.facet#link"
}
]
},
{
"index": {
"byteEnd": 291,
"byteStart": 212
},
"features": [
{
"uri": "https://upcloud.com/pricing-usd/",
"$type": "pub.leaflet.richtext.facet#link"
}
]
}
],
"plaintext": "I'm going to keep this one short and sweet. S3 cloud object storage is going to be less expensive in most cases than upping a VPS to a bigger drive. For instance, UpCloud offers $5 a month for 250GB, compared to an UpCloud server with 8GB of ram, 4 CPU Cores, and 160Gb of storage for $48/mo. There's a good chance you will not need that much server to run a PDS, but if you have a lot of users on it you'll need the disk space for user blobs."
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.header",
"level": 1,
"facets": [],
"plaintext": "Table of Contents"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.unorderedList",
"children": [
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [],
"plaintext": "PDS Setup"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [],
"plaintext": "Using rclone to migrate from a blobstore on disk to S3"
},
"children": []
}
]
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.header",
"level": 1,
"facets": [],
"plaintext": "PDS Setup"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 85,
"byteStart": 68
},
"features": [
{
"uri": "https://upcloud.com/docs/guides/get-started-managed-object-storage/",
"$type": "pub.leaflet.richtext.facet#link"
}
]
},
{
"index": {
"byteEnd": 111,
"byteStart": 90
},
"features": [
{
"uri": "https://bsky.app/profile/did:plc:7tx2qool7ehjpmo5xm2l574l",
"$type": "pub.leaflet.richtext.facet#link"
}
]
},
{
"index": {
"byteEnd": 157,
"byteStart": 129
},
"features": [
{
"uri": "https://turtleisland.blog/bluesky-pds-digitalocean-spaces-cdn-configuration/",
"$type": "pub.leaflet.richtext.facet#link"
}
]
}
],
"plaintext": "I'm not going to go over the setup of a cloud S3 object store since UpCloud does here and @yehuda.turtleis.land does a great job here for DigitalOcean Spaces."
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.unorderedList",
"children": [
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 17,
"byteStart": 10
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 59,
"byteStart": 47
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Open your pds.env file on your PDS, usually at /pds/pds.env."
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 42,
"byteStart": 15
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Remove the env PDS_BLOBSTORE_DISK_LOCATION (Can't have both set)"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 41,
"byteStart": 34
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Set some new env variables in the pds.env as below"
},
"children": []
}
]
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [],
"plaintext": "For UpCloud"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.code",
"language": "dotenv",
"plaintext": "PDS_BLOBSTORE_S3_BUCKET={What you named the bucket}\nPDS_BLOBSTORE_S3_REGION={Bottom left of Overview top section}\nPDS_BLOBSTORE_S3_ENDPOINT={S3 endpoint under Public access}\nPDS_BLOBSTORE_S3_ACCESS_KEY_ID={Under users tab}\nPDS_BLOBSTORE_S3_SECRET_ACCESS_KEY={Under users tab}\nPDS_BLOBSTORE_S3_FORCE_PATH_STYLE=false\n"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 105,
"byteStart": 53
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 204,
"byteStart": 146
},
"features": [
{
"uri": "https://turtleisland.blog/bluesky-pds-digitalocean-spaces-cdn-configuration/",
"$type": "pub.leaflet.richtext.facet#link"
}
]
}
],
"plaintext": "Digital ocean gives an Origin endpoint url like this https://{bucketname}.{region}.digitaloceanspaces.com that holds most of the info. Can always look here for a much more indepth setup for Digital Ocean."
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.code",
"language": "dotenv",
"plaintext": "PDS_BLOBSTORE_S3_BUCKET=bucketname\nPDS_BLOBSTORE_S3_REGION=region\nPDS_BLOBSTORE_S3_ENDPOINT=https://{region}.digitaloceanspaces.com\nPDS_BLOBSTORE_S3_ACCESS_KEY_ID={Settings -> Access Keys}\nPDS_BLOBSTORE_S3_SECRET_ACCESS_KEY={Settings -> Access Keys}\nPDS_BLOBSTORE_S3_FORCE_PATH_STYLE=false\n"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.unorderedList",
"children": [
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 23,
"byteStart": 4
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 46,
"byteStart": 26
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 58,
"byteStart": 54
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Run docker compose down & docker compose up -d in the /pds directory and if everything is right your PDS should be now storing and reading blobs from the object store! But this doesn't move the already existing blobs over, for that check out the next section."
},
"children": []
}
]
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.header",
"level": 1,
"facets": [],
"plaintext": "Using rclone to migrate from a blobstore on disk to S3"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 201,
"byteStart": 188
},
"features": [
{
"uri": "https://bsky.app/profile/bnewbold.net",
"$type": "pub.leaflet.richtext.facet#link"
}
]
}
],
"plaintext": "When I first started looking into doing this, I was wrongly assuming that I would need to write a script or a simple Rust binary to help users migrate to an S3 object storage. Thankfully, @bnewbold.net pointed out I was trying to reinvent the wheel."
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.bskyPost",
"postRef": {
"cid": "bafyreigomw5bo6q4t36ni5ioc34faidj34qf7znurexhlgv3b5nq67asyy",
"uri": "at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3lw5rrc43es2s"
}
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.blockquote",
"facets": [],
"plaintext": "I recommend before doing this making sure you have a backup of your server. Data lose is not expected, but you never know"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.unorderedList",
"children": [
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 73,
"byteStart": 54
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 108,
"byteStart": 104
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "You'll want to turn of the PDS while you do this with docker compose down in the PDS directory (usually /pds)"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 14,
"byteStart": 8
},
"features": [
{
"uri": "https://rclone.org/install/",
"$type": "pub.leaflet.richtext.facet#link"
}
]
},
{
"index": {
"byteEnd": 79,
"byteStart": 23
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Install rclone. I used sudo -v ; curl https://rclone.org/install.sh | sudo bash"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 37,
"byteStart": 9
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Create a ~/.config/rclone/rclone.conf"
},
"children": []
}
]
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.code",
"language": "plaintext",
"plaintext": "[pds-blobs]\ntype = s3\nprovider = {UpCloud or DigitalOcean}\nenv_auth = false\naccess_key_id = {your_access_key_id}\nsecret_access_key = {your_secret_key\nendpoint = {S3 endpoint minus the https so like atl1.digitaloceanspaces.com}\nacl = private\n"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.blockquote",
"facets": [],
"plaintext": "A couple of notes before you run the command to move the blobs over"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.unorderedList",
"children": [
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [],
"plaintext": "Once you're good with everything and prepared to take the time to move the blobs, you can run this command."
},
"children": []
}
]
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.code",
"language": "plaintext",
"plaintext": "rclone sync /pds/blocks pds-blobs:{bucket-name}/blocks -P\n"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [],
"plaintext": "To break that command down a bit"
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.unorderedList",
"children": [
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 6,
"byteStart": 0
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "rclone - the tool"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 4,
"byteStart": 0
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "sync - the command"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 11,
"byteStart": 0
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "/pds/blocks - the local location of your blobs on the PDS currently"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 10,
"byteStart": 0
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 61,
"byteStart": 33
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "pds-blobs: - The name you set in ~/.config/rclone/rclone.conf"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 7,
"byteStart": 0
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 86,
"byteStart": 79
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "/blocks - folder location on the object storage. The PDS expects there to be a /blocks folder when reading"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 2,
"byteStart": 0
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "-P - Shows the progress of it"
},
"children": []
},
{
"$type": "pub.leaflet.blocks.unorderedList#listItem",
"content": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 75,
"byteStart": 55
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
},
{
"index": {
"byteEnd": 87,
"byteStart": 83
},
"features": [
{
"$type": "pub.leaflet.richtext.facet#code"
}
]
}
],
"plaintext": "Once the upload is done can start your pds backup with docker compose up -d in the /pds directory"
},
"children": []
}
]
}
},
{
"$type": "pub.leaflet.pages.linearDocument#block",
"block": {
"$type": "pub.leaflet.blocks.text",
"facets": [
{
"index": {
"byteEnd": 182,
"byteStart": 162
},
"features": [
{
"uri": "https://signup.upcloud.com/?promo=P48GC6",
"$type": "pub.leaflet.richtext.facet#link"
}
]
},
{
"index": {
"byteEnd": 286,
"byteStart": 258
},
"features": [
{
"uri": "https://m.do.co/c/3316422ea20d",
"$type": "pub.leaflet.richtext.facet#link"
}
]
}
],
"plaintext": "And that's it! Your PDS should now be using the S3 object storage for blobs, saving hard drive space and some money. If you're wanting to try out UpCloud you can use my referral link to sign up for $25 of free credits or if you want to go with Digital Ocean you can use my referral link to get $200 in credit over 60 days."
}
}
]
}
]
},
"bskyPostRef": {
"cid": "bafyreiesgclrg2grikgmyts3varqkrgwo3pd5yfbtlqrqywodcmlblfbau",
"uri": "at://did:plc:rnpkyqnmsw4ipey6eotbdnnf/app.bsky.feed.post/3m5fko57ajc2y",
"commit": {
"cid": "bafyreigklcdcu7hzcsgk7eyorphe3yywr3juqpyhhrnurqsvpimkkoncue",
"rev": "3m5fko5dp5y26"
},
"validationStatus": "valid"
},
"description": "Learn how to setup your self hosted PDS to use an S3 object store for blob storage.",
"publishedAt": "2025-11-12T02:02:23.560Z"
}