Conduit to yield first lines from file without leaking space or file handles?

tomjaguarpaw:

---

EDIT: Related to Conduit missing closing file handles - Haskell-Cafe - Haskell.org

Looks like the best thing to do in conduit may be to use a sort of CPS style:

github.com/snoyberg/conduit

Conduit to yield first lines from file without leaking space or file handles?

opened 07:34AM - 05 May 26 UTC

      tomjaguarpaw
    

How can I write a conduit that takes in file names and yields the first n line…s from each file without leaking space or file handles? I have the following, but it leaks file handles and crashes on my machine if there are more than about 1000 files. Since take does not force all of its upstream conduit it doesn't cause the finalizer of sourceFile to run immediately (it's delayed until the ResourceT block ends). It's very hard to see how to solve this problem. (Let's say the files may have arbitrarily many lines, so anything that requires sinking the whole file is not a valid solution.) ---- Also discussed at: https://discourse.haskell.org/t/conduit-to-yield-first-lines-from-file-without-leaking-space-or-file-handles/14030 ---- cabal-script-test27.hs: /tmp/manyfiles/1981: openBinaryFile: resource exhausted (Too many open files) .hs import Conduit import qualified Data.Conduit.Combinators as CC import System.Directory (doesFileExist) import Data.Text (Text) main :: IO () main = runConduitRes $ CC.sourceDirectoryDeep False "/tmp/manyfiles" .| firstNLines 1 .| CC.mapM_ (liftIO . print) firstNLines :: Int -> ConduitT FilePath Text (ResourceT IO) () firstNLines n = awaitForever $ \fp -> CC.sourceFile fp .| CC.decodeUtf8Lenient .| CC.linesUnbounded .| CC.take n