Paper publishing workflow

A homage to mise-en-place

{{}}

Install a new project from paper-publishing-process with

• Automatic compilation of main.pdf, review-.pdf and diff-.pdf
• Git Tag-based versioning and changelog generation
• CI/CD pipeline with compilation and new releases

{{< video src="compile-all.mp4" loop="true" autoplay="true" muted="true" >}}

{{}}

A frustration from when I first started writing papers to be submitted to journals and go through a review cycle was keeping track of which version was submitted at which point in time. Which version should I respond to, which version should the updated version be diffed against? This was a very manual process where the work of tracking what needed to be submitted in the latest revision sometimes felt as large as the writing itself.

Fortunately, there are solutions.

Setting Up Tools and Repo

Everything should be able to live independently of the system you first develop on, so we create a Git repo where all the files can live. In this guide we will make extensive use of GitHub's workflow, so we create the repo there. I called mine paper-publishing-process, so let's clone it down:

Next we need to be able to install all software, and for that we use mise. First we must install mise; visit the website, or run the following command:

Then we create the file mise.toml in the project root:

We need to give mise permission to use it, mise trust, before installing everything with mise install.

In the first commit I added a simple license file (MIT) and a .gitignore suited for TeX development.

→ See commit ce3150c.

While we're at it, let's add some files that help keep the repo up to a good standard in terms of formatting:

• tex-fmt.toml: Formatting of .tex and .bib
• .taplo.toml: Formatting of .toml
• .yamlfmt.yml: Formatting of .yml
• hk.pkl: Formatting and linting

This change also updates mise.toml and creates the file tex/main.tex.

→ See commit b161e41.

We can verify that everything so far is as it should be:

Automatic Compilation of TeX

Local Compilation with mise

We are now ready to work on the TeX files! We want to write in tex/main.tex, and every time we save, the PDF should update automatically. We turn again to mise for this functionality.

→ See commit e7127b4.

{{< details "Motivation for mise tasks" >}}

In the commit above we add a mise task called "localize-bib-paths". The motivation behind it is that the "compile" task (introduced a little later) will run it so that when developing locally all references come from the global master reference. This is useful in cases where you want to add a new reference to the TeX document that already exists in the master reference. If you use a modern editor it will provide completion suggestions, which would not be available in the generated reference files for new references.

{{< /details >}}

We can now run

It fails! This is due to the first, slightly special choice we make in this workflow.

Bibfish

In order to keep the .bib files in the repo as minimal as possible, they are all generated from a local "master reference file". This file is what the LOCAL_BIB_PATH value we added in mise.toml points to. This master reference is intended to be a file that lives only locally, not in the Git repo, and should be able to supply references not just to this project, but to all your TeX projects! Bibfish will then read it and generate a reference file based only on the references found in a given TeX file.

Bibfish supports most common citation formats (e.g. \cite, \citet, etc.), but can easily be extended with custom formats. In mise.toml we have added citeA, and also specify cite, citet and citep (bibfish -c 'cite,citet,citep,citeA' -f main.tex {{ env.LOCAL_BIB_PATH }}.bib main.bib).

Let's fix the error from before by

1. Adding mise.local.toml to .gitignore
2. Specifying a local bib file in mise.local.toml

{{}}

Note that the local file's location must either be "absolute" (/home/user/the-file), or "relative" from the tex folder, and that it is specified without .bib.

{{< /callout >}}

I also added the local master reference here to illustrate with a minimal example, but normally I would place it outside the repo. Also add

→ See commit a149cdf.

When we now try to compile, no error!

BBL Files and References in Git

Let's add some references.

We also create a general compile task in mise:

When we now run mise watch compile a new file will be generated: tex/main.bbl. This is normally ignored, but we will need it in the future, so we remove it from .gitignore.

It contains a new .bbl file, and the .bib file has been updated with exactly one reference — the one we added in tex/main.tex!

→ See commit 1a523ea.

Managing TeX Packages with tlmgr

We use tinytex as the TeX distribution in this project. As the name suggests it is much smaller than the more common TeX-live/TeX-live-full. This also means we will more often find that packages are unavailable, but in return we get more control. Let's use a handful of packages not found in TinyTeX.

→ See commit fe28c24 for updated tex/main.tex and mise.toml.

If you first update tex/main.tex with the contents from the commit above and have mise watch compile running it should fail, complaining that underscore is not available. Then add the postinstall hook in mise.toml and run mise install.

This will ensure that various packages are installed via tlmgr. The advantage of this setup is that many journals are very particular about which packages they accept. If you have written a long manuscript that uses many packages and compiles fine locally because you used TeX-live-full, it can quickly become difficult to figure out what needs to go if compilation on the journal's server crashes.

Local Development Workflow

We have now completed the workflow for local TeX files! Run mise watch compile, write down all your good ideas, and every time tex/main.tex is saved the compilation will run. Notice how tex/main.bib updates in the terminal window at the top right when I add and remove references.

{{< video src="compile.mp4" loop="true" autoplay="true" muted="true" >}}

Next steps: automatic compilation on GitHub, support for versioning via Git, and routines for responding to feedback from journals.

CI/CD Pipeline on GitHub

Git Commit Hooks

Since we have tools for formatting the code as well as a working hk setup, we can easily set up Git commit hooks:

{{}}

Since mise run|watch compile converts all \bibliography{...} to use the local master reference, while the pre-commit hook via hk converts them to use the generated bib files, a conflict will arise between them.

Each time you create a commit it is therefore wise to stop mise watch compile, so that it does not overwrite before you have finished the commit.

{{< /callout >}}

Automatic Building and Release

To set up CI/CD we create the file mise.ci.toml, as well as three files in .github/workflows and .mise/tasks:

• .github/workflows/build.yml

  Compiles all PDFs for every push to the main branch on GitHub.

• .github/workflows/fix-bib-filepath.yml

  An extra check ensuring the correct bib file is used to compile the PDFs.

• .mise/tasks/package

  Puts all files a journal needs into an archive, for easy uploading when submitting.

→ See commit 93c8487 for the contents of the files.

That commit actually failed in CI on GitHub. I had forgotten that all files in .mise/tasks must be executable (executable bit), meaning I needed to

→ See commit 06262fd.

Below we see that CI/CD ran without errors and gave us a paper-assets artifact.

{{