{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/misc/process-substitution-in-bash/",
  "description": "Use process substitution <() to treat command output as files in Bash. Compare directories, process data, and avoid temporary files with this technique.",
  "path": "/misc/process-substitution-in-bash/",
  "publishedAt": "2023-04-30T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Shell",
    "Unix",
    "TIL"
  ],
  "textContent": "I needed to compare two large directories with thousands of similarly named PDF files and\nfind the differing filenames between them. In the first pass, this is what I did:\n\nListed out the content of the first directory and saved it in a file:\n\nDid the same for the second directory:\n\nCompared the difference between the two outputs:\n\nThis returned the name of the differing files likes this:\n\nIt does the job, but I asked BingChat if there's a better way to accomplish the task without\ncreating intermediate files, and it didn't let me down. Turns out that in Bash, process\nsubstitution allows you to do just that. Instead of running three commands, you can achieve\nthe same result with a simple one-liner:\n\nProcess substitution\n\nIn Bash, process substitution is a feature that allows you to treat the output of a command\nor commands as if it were a file. It enables you to use the output of a command as an input\nto another command or perform other operations that expect file input or output.\n\n- One important thing to point out is that process substitution is specific to Bash, Zsh,\n  and certain versions of Ksh. Other shells and Bash in POSIX mode don't understand it.\n  Bash, Zsh, and Ksh (88,93) support process substitution, but pdksh derivatives like mksh\n  don't currently have this capability.\\*\n\nThe syntax for process substitution is as follows:\n\n- <(command): This form allows you to use the output of a command as a file-like input.\n- >(command): This form allows you to use the output of a command as a file-like output.\n\nWhen using process substitution, Bash creates a named pipe (FIFO) or a special file\ndescriptor /dev/fd/<n> behind the scenes. The command within the parentheses is executed,\nand its output is redirected to the named pipe or file descriptor. Then, the path to the\nnamed pipe or file descriptor is substituted into the original command line.\n\nThis is different from the plain-old stdin or stdout redirection. Here's how:\n\n- Input\n    - _Plain redirection_: When using plain stdin redirection (<), you can redirect input\n      from a file, for example, < input.txt. The command reads the content of the file as\n      standard input (stdin).\n    - _Process substitution_: With process substitution, you can use the output of a command\n      as input. For example, command < <(echo \"input\"). Here, the output of the echo\n      command is treated as a file-like object and used as the input to command.\n\n- Output\n    - _Plain redirection_: Using plain stdout redirection (> or >>), you can redirect\n      the output (stdout) of a command to a file, for example, command > output.txt. The\n      command's output is written to the specified file.\n    - _Process substitution_: With process substitution, you can use the output of a command\n      as output. For example, command >(process_output). Here, the output of command is\n      treated as a file-like output, and it is passed as input to the process_output\n      command or operation.\n\nBy using process substitution, the output of a command can be seamlessly integrated into\nother commands as if it were a file, even if the command doesn't explicitly support stdin or\nstdout redirection. This allows for greater compatibility and enables the use of the output\nin situations where direct piping or redirection may not be possible.\n\nA few practical examples\n\nInspecting the descriptors involved in process substitution\n\nYou can inspect the descriptor used by a process substitution like this:\n\nThis returns:\n\nHere, the expression >(true) creates a temporary file-like object, and the true command\nserves as a placeholder for its input. Similarly, <(false) creates another temporary\nfile-like object with the false command serving as a placeholder for its output. When the\necho command is executed, it displays the filenames associated with these temporary\nfile-like objects, which are /dev/fd/13 and /dev/fd/11 in this specific scenario. These\nfilenames represent the underlying descriptors of the respective process substitutions,\nindicating the file descriptors associated with the temporary objects created during the\nprocess substitution.\n\nCalculating the total number of lines in a file\n\nThis command calculates the total number of lines in the input.txt file. Here, the\n<(cat input.txt) commad creates an input-type file descriptor containing the output of the\ncat command and wc -l reads that content from there. The extra < redirects the\nfile-like object as an input stream again. This is a roundabout way of doing the following:\n\nProcessing the content of a file line by line\n\nThis command reads each line from the file input.txt and echoes it. It uses a while loop\nwith the read command to iterate over the lines, assigning each line to the variable line\nand the echo $line command displays the line. Process substitution <() is used to treat\nthe output of cat input.txt as a temporary file, providing the input to the loop.\n\nComparing directory sizes\n\nThe command compares the disk usage of two directories, dir1 and dir2, using the diff\ncommand. The process substitution <() is employed to capture the output of the du -sh\ncommand, which calculates the disk usage of each directory and provides a summary in a\nhuman-readable format. The output of each du -sh command, representing the disk usage of\ndir1 and dir2, is treated as temporary files and passed as arguments to the diff\ncommand. This enables the comparison of the disk usage between the two directories,\nhighlighting any discrepancies in file sizes or subdirectories.\n\nPicking or rejecting lines common between two sorted files\n\nThis returns:\n\nThis performs a comparison between the sorted outputs of two separate commands using comm.\nThe comm command expects two files but we're using process substitution to make two\nfile-like objects from stdout.\n\nWithin the first process substitution <(), echo is used to generate a string containing\ntwo lines: hello world and hello mars. This string is then piped to the sort command,\nwhich sorts the lines alphabetically.\n\nSimilarly, the second part of the command <() uses process substitution as well. It\nfollows the same pattern as the first process substitution, but this time the string\ncontains hello world and hello venus.\n\nThe file-like objects containing the sorted output from the two process substitutions are\nthen passed as arguments to the comm command. Then comm compares the input files line by\nline and generates three columns of output: lines unique to the first input, lines unique to\nthe second input, and lines common to both inputs.\n\nFurther reading\n\n- [Process substitution in Bash]\n\n\n\n\n[process substitution in bash]:\n    https://tldp.org/LDP/abs/html/process-sub.html",
  "title": "Process substitution in Bash"
}