{
"$type": "site.standard.document",
"content": {
"$type": "site.subgraph.content.markdown",
"body": "\nThe technical details of this setup are in [Obsidian as a CMS](1670659200001-obsidian-as-a-cms.md) and [Revisiting Obsidian as a CMS](1699332127006-revisiting-obsidian-as-a-cms.md). The thought process that got me here is its own thing.\n\nI wanted a backend for my personal site that was as close to free as possible. Airtable burned through the free tier fast, Google Sheets felt like dragging a desk through a doorway. What I actually needed was a flat-file way to publish markdown with metadata, plus the ability to write on the go (Git clients on iOS make that painful).\n\nObsidian was already my knowledge base. Folder-based templates, private Git backup. With the right YAML frontmatter, it covered all the meta and feature-flag fields I cared about. The only gap was getting the markdown out of the repo and into a front-end framework.\n\n## Inner workings\n\nThe GitHub GraphQL API can fetch a single file or a whole directory in one query. YAML gets parsed with [gray-matter](https://github.com/jonschlinkert/gray-matter), then the markdown renders. In Next.js I used [next-mdx-remote](https://github.com/hashicorp/next-mdx-remote). I'm on Astro now to ship less JS, with [astro-remote](https://github.com/natemoo-re/astro-remote), which uses [marked](https://marked.js.org/) and sanitizes output with [ultrahtml](https://github.com/natemoo-re/ultrahtml). Component overrides work the same as MDX.\n\n### Fetching posts\n\n```typescript\nasync function fetchFromGitHubGraphQL(query: string, variables: any) {\n const response = await fetch(\"https://api.github.com/graphql\", {\n method: \"POST\",\n headers: {\n \"Content-Type\": \"application/json\",\n Authorization: `Bearer ${github}`,\n },\n body: JSON.stringify({ query, variables }),\n });\n\n if (!response.ok) {\n console.error(\"HTTP Error:\", response.status);\n return response;\n }\n\n return response.json();\n}\n```\n\n`fetchFromGitHubGraphQL` used to be load-bearing across several callers. Now it sits behind one function, `getObsidianEntries`.\n\n```typescript\nexport async function getObsidianEntries(path: string, slug?: string) {\n const expression = slug ? `HEAD:content/${path}/${slug}.md` : `HEAD:content/${path}`;\n\n const {\n data: {\n repository: { object },\n },\n } = await fetchFromGitHubGraphQL(\n `\n query fetchEntries($owner: String!, $name: String!, $expression: String!) {\n repository(owner: $owner, name: $name) {\n object(expression: $expression) {\n ... on Tree {\n entries {\n name\n object {\n ... on Blob {\n text\n }\n }\n }\n }\n ... on Blob {\n text\n }\n }\n }\n }\n `,\n {\n owner: `GITHUB_USERNAME`,\n name: `REPO_NAME`,\n expression,\n }\n );\n\n if (slug) {\n if (!object || !object.text) {\n console.error(\"No data returned from the GraphQL query for the single entry.\");\n return null;\n }\n return parseMarkdownContent(object.text, path);\n }\n\n if (!object || !object.entries) {\n console.error(\"No data returned from the GraphQL query for multiple entries.\");\n return [];\n }\n\n const parsedEntries = await Promise.all(\n object.entries.map((entry: { object: { text: any } }) => {\n const content = entry.object.text;\n return parseMarkdownContent(content, path);\n })\n );\n\n parseAndMergeTags(parsedEntries);\n\n return parsedEntries;\n}\n```\n\n### File structure\n\n```\n.\n├── README.md\n├── content\n│ ├── art\n│ │ └── txt.md\n│ ├── notes\n│ │ └── txt.md\n│ ├── posts\n│ │ └── txt.md\n│ └── recipes\n│ └── txt.md\n└── templates\n └── base_template.md\n```\n\nThe folder layout doubles as routing on the front end. `getObsidianEntries` takes a path, with an optional slug. Slug matches the filename, so `path + slug.md` returns one entry, and `path` alone returns the whole directory. That one trick made it easy to spin up new content types.\n\n```typescript\n---\nimport { Markdown } from \"astro-remote\";\n\nconst { path, slug } = Astro.params;\nconst entry = await getObsidianEntries(path, slug);\nconst { body, frontmatter } = entry;\n---\n\n<article>\n <Markdown\n components={{ img: Image, p: Paragraph }}\n sanitize={{\n dropElements: [\"head\", \"style\"],\n allowCustomElements: true,\n }}\n >\n {body}\n </Markdown>\n</article>\n```\n\n```typescript\n---\nimport { getObsidianEntries } from \"@lib/github\";\n\nconst { path } = Astro.params;\nentries = entries.sort((a, b) => new Date(b.frontmatter.created).getTime() - new Date(a.frontmatter.created).getTime());\n---\n\n<>\n {\n entries.map((entry) => (\n <li>\n <p>\n <a href={`/${path}/${entry.frontmatter.slug}`}>{entry.frontmatter.title}</a>\n </p>\n </li>\n ))\n }\n</>\n```\n\n### Frontmatter\n\nA `base_template` populates each new file. It prompts for a title, formats a URL-safe slug, and stamps creation and modified dates.\n\n```yaml\n---\n<%*\nlet title = await tp.system.prompt(\"Please enter a value\");\nlet slug = tp.file.creation_date(\"x\") + \" \" + title;\nlet formatted_slug = slug.trim().replace(/\\W+/g, '-').toLowerCase();\nawait tp.file.rename(`${formatted_slug}`);\n%>\ntitle: <%* tR += title; %>\nslug: <%* tR += formatted_slug; %>\npublished: false\ncreated: <% tp.file.creation_date(\"YYYY-MM-DD HH:mm\") %>\nupdated: <% tp.file.last_modified_date(\"YYYY-MM-DD HH:mm\") %>\ntags:\n -\n---\n```\n\nTags are the rough edge. Right now I aggregate them into a flat file on Cloudflare R2, which is hacky and unreliable. A hashing scheme to keep tags in sync with published content is on the list.\n\n### Images\n\nThe old setup hashed each image with md5, dumped it to `assets`, and a GitHub Action pushed it to R2 on push. The [S3 Image Uploader](https://github.com/jvsteiner/s3-image-uploader) plugin replaced that: it hashes the filename and uploads from the Obsidian editor directly. My PR adding concurrent uploads landed in [`0.2.10`](https://github.com/jvsteiner/s3-image-uploader/releases/tag/0.2.10).\n\nMarkdown wraps `<img>` in `<p>`, which I don't want. Astro-Remote makes it easy to unwrap by checking the rendered slot:\n\n```\n---\nlet slots = await Astro.slots.render(\"default\");\nlet slotsString = slots.toString();\n---\n\n{\n slotsString.includes(\"img src\") ? (\n <slot />\n ) : (\n <p>\n <slot />\n </p>\n )\n}\n```\n\nThe whole thing is cheap, fast to write in, and mine.\n"
},
"description": "Obsidian as CMS: Using YAML frontmatter and GitHub's GraphQL API to create a free, flexible publishing system for markdown content that.",
"publishedAt": "2024-03-11T17:21:44.945Z",
"site": "at://did:plc:p5xem22ammiafn5kxonaksfa/site.standard.publication/3mlp3ywhyv2kx",
"tags": [
"obsidian",
"cms",
"markdown",
"github",
"astro",
"nextjs",
"content-management",
"web-development"
],
"textContent": "The technical details of this setup are in Obsidian as a CMS and Revisiting Obsidian as a CMS. The thought process that got me here is its own thing.\n\nI wanted a backend for my personal site that was as close to free as possible. Airtable burned through the free tier fast, Google Sheets felt like dragging a desk through a doorway. What I actually needed was a flat-file way to publish markdown with metadata, plus the ability to write on the go (Git clients on iOS make that painful).\n\nObsidian was already my knowledge base. Folder-based templates, private Git backup. With the right YAML frontmatter, it covered all the meta and feature-flag fields I cared about. The only gap was getting the markdown out of the repo and into a front-end framework.\n\nInner workings\n\nThe GitHub GraphQL API can fetch a single file or a whole directory in one query. YAML gets parsed with gray-matter, then the markdown renders. In Next.js I used next-mdx-remote. I'm on Astro now to ship less JS, with astro-remote, which uses marked and sanitizes output with ultrahtml. Component overrides work the same as MDX.\n\nFetching posts\n\nasync function fetchFromGitHubGraphQL(query: string, variables: any) { const response = await fetch(\"https://api.github.com/graphql\", { method: \"POST\", headers: { \"Content-Type\": \"application/json\", Authorization: `Bearer ${github}`, }, body: JSON.stringify({ query, variables }), });\n\nif (!response.ok) { console.error(\"HTTP Error:\", response.status); return response; }\n\nreturn response.json(); }\n\nfetchFromGitHubGraphQL used to be load-bearing across several callers. Now it sits behind one function, getObsidianEntries.\n\nexport async function getObsidianEntries(path: string, slug?: string) { const expression = slug ? `HEAD:content/${path}/${slug}.md` : `HEAD:content/${path}`;\n\nconst { data: { repository: { object }, }, } = await fetchFromGitHubGraphQL( ` query fetchEntries($owner: String!, $name: String!, $expression: String!) { repository(owner: $owner, name: $name) { object(expression: $expression) { ... on Tree { entries { name object { ... on Blob { text } } } } ... on Blob { text } } } } `, { owner: `GITHUB_USERNAME`, name: `REPO_NAME`, expression, } );\n\nif (slug) { if (!object || !object.text) { console.error(\"No data returned from the GraphQL query for the single entry.\"); return null; } return parseMarkdownContent(object.text, path); }\n\nif (!object || !object.entries) { console.error(\"No data returned from the GraphQL query for multiple entries.\"); return []; }\n\nconst parsedEntries = await Promise.all( object.entries.map((entry: { object: { text: any } }) => { const content = entry.object.text; return parseMarkdownContent(content, path); }) );\n\nparseAndMergeTags(parsedEntries);\n\nreturn parsedEntries; }\n\nFile structure\n\n. ├── README.md ├── content │ ├── art │ │ └── txt.md │ ├── notes │ │ └── txt.md │ ├── posts │ │ └── txt.md │ └── recipes │ └── txt.md └── templates └── base_template.md\n\nThe folder layout doubles as routing on the front end. getObsidianEntries takes a path, with an optional slug. Slug matches the filename, so path + slug.md returns one entry, and path alone returns the whole directory. That one trick made it easy to spin up new content types.\n\n--- import { Markdown } from \"astro-remote\";\n\nconst { path, slug } = Astro.params; const entry = await getObsidianEntries(path, slug); const { body, frontmatter } = entry; ---\n\n<article> <Markdown components={{ img: Image, p: Paragraph }} sanitize={{ dropElements: [\"head\", \"style\"], allowCustomElements: true, }} > {body} </Markdown> </article>\n\n--- import { getObsidianEntries } from \"@lib/github\";\n\nconst { path } = Astro.params; entries = entries.sort((a, b) => new Date(b.frontmatter.created).getTime() - new Date(a.frontmatter.created).getTime()); ---\n\n<> { entries.map((entry) => ( <li> <p> <a href={`/${path}/${entry.frontmatter.slug}`}>{entry.frontmatter.title}</a> </p> </li> )) } </>\n\nFrontmatter\n\nA base_template populates each new file. It prompts for a title, formats a URL-safe slug, and stamps creation and modified dates.\n\n--- <%* let title = await tp.system.prompt(\"Please enter a value\"); let slug = tp.file.creation_date(\"x\") + \" \" + title; let formatted_slug = slug.trim().replace(/\\W+/g, '-').toLowerCase(); await tp.file.rename(`${formatted_slug}`); %> title: <%* tR += title; %> slug: <%* tR += formatted_slug; %> published: false created: <% tp.file.creation_date(\"YYYY-MM-DD HH:mm\") %> updated: <% tp.file.last_modified_date(\"YYYY-MM-DD HH:mm\") %> tags: - ---\n\nTags are the rough edge. Right now I aggregate them into a flat file on Cloudflare R2, which is hacky and unreliable. A hashing scheme to keep tags in sync with published content is on the list.\n\nImages\n\nThe old setup hashed each image with md5, dumped it to assets, and a GitHub Action pushed it to R2 on push. The S3 Image Uploader plugin replaced that: it hashes the filename and uploads from the Obsidian editor directly. My PR adding concurrent uploads landed in 0.2.10.\n\nMarkdown wraps <img> in <p>, which I don't want. Astro-Remote makes it easy to unwrap by checking the rendered slot:\n\n--- let slots = await Astro.slots.render(\"default\"); let slotsString = slots.toString(); ---\n\n{ slotsString.includes(\"img src\") ? ( <slot /> ) : ( <p> <slot /> </p> ) }\n\nThe whole thing is cheap, fast to write in, and mine.",
"title": "Revisiting Obsidian as a CMS, again"
}