{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreihyzog753dug7dzvryhsc7jxpo6ru6vebxi53jgy4m7h66lnilnnq",
    "uri": "at://did:plc:pi6woz4d47bkuws673w2il2r/app.bsky.feed.post/3mhcpnaxq2wf2"
  },
  "path": "/t/botan-bindings-devlog/6855?page=10#post_196",
  "publishedAt": "2026-03-18T02:51:56.000Z",
  "site": "https://discourse.haskell.org",
  "tags": [
    "@RNG",
    "@Hash"
  ],
  "textContent": "# A Space Oddity Update: Big Changes Coming\n\nThis update has been a long time coming. I don’t mean since the last update - I mean that it has things in it that I’ve wanted to get out for over a year. I have been diligently clearing the way for this for a very long time, and it has produced 2 other libraries as a side effect - it is time to show you what all that effort has been for.\n\n# (Ten) Ground Control (Nine) to Major Tom (eight, seven)\n\n`memalloc` has turned out to be in fine shape. I finally was able to add the last piece I needed to make it minimally viable. The question here was how to deal with generalizing the `withPtr` function to support arbitrary memory types, and to allow for not just drilling down step-by-step, but for drilling down to a specific memory type - eg, if you have `Ptr Word8` inside a `ForeignPtr` inside a `ByteString`, you want to be able to call `withPtr` directly, instead of calling `withMem` twice - once for the `ByteString` to get the `ForeignPtr`, once with the `ForeignPtr` to get the `Ptr`.\n\n\n    class (Memory mem, Memorable memo) => WithMemory mem memo where\n        withMemory :: memo -> (mem (MemRep memo) -> IO a) -> IO a\n        default withMemory :: (Mem memo ~ mem) => memo -> (mem (MemRep memo) -> IO a) -> IO a\n        withMemory = withMem\n\n    instance {-# OVERLAPPABLE #-} (Memory mem, Memorable memo, Mem memo ~ mem) => WithMemory mem memo where\n\n    #if defined(USE_BYTESTRING)\n    instance WithMemory Ptr ByteString where\n        withMemory ptr action = withMem ptr $ \\ fPtr -> withMem fPtr action\n    instance WithMemory ForeignPtr ByteString where\n    #endif\n\n    type WithPtr memo = WithMemory Ptr memo\n\n    withPtr :: WithPtr memo => memo -> (Ptr (MemRep memo) -> IO a) -> IO a\n    withPtr = withMemory\n\n    type WithForeignPtr memo = WithMemory ForeignPtr memo\n\n    withForeignPtr :: WithForeignPtr memo => memo -> (ForeignPtr (MemRep memo) -> IO a) -> IO a\n    withForeignPtr = withMemory\n\n\nIn the end, I turned to my first ever use of `OverlappableInstances`, which I hope is both safe and justified. Normally I would resort to some type-level and family shenanigans, so I might revisit a few things in `memalloc` and the `math` library in some places where I did, to see if this is a more appropriate way of handling things than with a recursive type family\n\nI am quite pleased with there result here, however.\n\n# (Six) Commencing (five) countdown, engines (four) on (three)\n\nIn the most recent meeting on Monday (nothing of note has occurred recently due to travel and other events), Joris Jose and I discussed the needs and effects of integrating the new memory allocation code into botan-low, and came to the conclusion that it doesn’t really fit the old plan of `botan-bindings -> botan-low -> botan`, and that, surprisingly, I quite like the state that `botan-low` is in - _that is to say, its not that there arent improvements to be made, but that the IO-Bytestring interface is quite effective for what it is, and I see no sense in mucking that up for no reason_.\n\nSometimes you go back and look at old code and are embarrassed, other times you are proud - the underbelly still needs the refactoring that `memalloc` is designed for, but I can’t really justify changing what is a stable API for no reason. And since the old plan was only an idea that we are not beholden to, the plans have been changed.\n\nHow have they changed?\n\nThe new memory management library provides allocators, and by providing the allocator as an argument, we can ie specify that we want to use a `SecureByteStringAllocator` that guarantees cleanup, instead of just using the default `ByteStringAllocator` which cleans up lazily. This changes the API interface:\n\n\n    -- This\n    rngGet :: RNG -> Int -> IO ByteString\n\n    -- Changes to this\n    type ByteAllocator alr bytes =\n        ( Allocator alr\n        , Allocation alr ~ bytes\n        , Layout alr ~ Int\n        , Bytes bytes\n        )\n    -- ...\n    rngGet :: (ByteAllocator alr bytes) => alr -> RNG -> Int -> IO bytes\n\n\nThis is nice, it gives us quite a bit of freedom in interacting with the `botan-bindings` C interface. Given that this will change almost every function, we decided it best to make `botan-low` into a more expressive interface with a major version jump, and the existing IO-Bytestring -focused interface will be re-exposed in `botan-io-bytestring`. This also gives reason to bring on in some of the higher-level ergonomics that are laying fallow in the unpublished `botan`, since we are not beholden to keeping `botan-low` as a 1:1 reflection of the `botan-bindings` interface anymore.\n\nThe Bytes class needs just a wee bit more work before I can use it here in botan, so today, we are going to finish ripping the guts out of the `Make / Remake / mkFoo` classes first, and with great relish, replacing them with ByteString allocators from `memalloc`.\n\n# (Two) Check ignition (One) and may god’s love (lift off…) be with you\n\n## RNG\n\nLast time, we defined a `BotanObject` class, and used it to manage turning a `Ptr` into a `ForeignPtr` into a `BotanObject` and back. Now, it is time to use it. In fact, lets rewrite **the entire RNG module…** I know that sounds drastic, but trust me on this\n\nWe have a few more imports this time around, courtesy of the `memalloc` library:\n\n\n    import Botan.Bindings.RNG\n    import Botan.Bindings.ConstPtr (ConstPtr (..))\n\n    import Botan.Low.Internal.Object\n    import Botan.Low.Internal.Error\n\n    import Memory.Memory\n    import Memory.Pointer\n    import Memory.Castable\n    import Memory.Allocator\n    import Memory.Allocator.ByteString\n\n    import Data.ByteString (ByteString)\n    import qualified Data.ByteString as ByteString\n    import Foreign.Ptr (Ptr)\n    import Foreign.ForeignPtr (ForeignPtr)\n\n\nStep 1 of implementing a module with the new `Memory` types is to define your memory instances. We sort of did this last time, so we’ll revisit with minor improvements due to changes since then.\n\n\n    -- This should probably go in botan-bindings\n    instance Memorable BotanRNG where\n        type Mem BotanRNG = Ptr\n        type MemRep BotanRNG = BotanRNGStruct\n        withMem (MkBotanRNG ptr) action = action ptr\n\n    newtype RNG = MkRNG { foreignPtr :: ForeignPtr BotanRNGStruct }\n\n    instance Memorable RNG where\n        type Mem RNG = ForeignPtr\n        type MemRep RNG = BotanRNGStruct\n        withMem (MkRNG fptr) action = action fptr\n\n    instance BotanObject RNG where\n\n        type BotanStruct RNG = BotanRNGStruct\n        type BotanPtr    RNG = BotanRNG\n\n        toBotanPtr      = MkBotanRNG\n        toBotan         = MkRNG\n        botanFinalizer  = botan_rng_destroy\n\n\nAnd then its engines on - didn’t you hear the countdown?\n\nFirst, we define a `withRNG` helper, which is `withBotanPtr` specialized - it makes things easier to read.\n\n\n    withRNG :: RNG -> (BotanRNG -> IO a) -> IO a\n    withRNG = withBotanPtr\n\n\nThen, we define our RNG initializer - it is _very_ straightforward: We get the RNGType’s name’s pointer as a ConstPtr CString and pass it in and `createBotan` handles the actual initialization logic, so all we have to do is turn arguments into pointers.\n\n\n    rngInit :: RNGType -> IO RNG\n    rngInit t = ByteString.useAsCString (rngTypeName t) $ \\ tPtr -> do\n        createBotan @RNG $ \\ outPtr -> botan_rng_init outPtr (ConstPtr $ cast tPtr)\n\n\nI’d use `withPtr` here instead of `ByteString.useAsCSString` except that the trailing null-terminator actually matters, and so the copy is required. I will probably either put a convenience function in the `memalloc` library to make this more convenient, or have the `...TypeName` functions add the `\\0` themselves. It is a minor nit.\n\nPreviously `botan-low` passed around context and algorithm types as ByteStrings, and that wasn’t very fun to manage. The higher-level `botan` has had ADTs for the algorithm types just sitting there, so I’ve been bringing them down to `botan-low` because we’re going to want to reuse them in any library built on top.\n\n\n    data RNGType\n        = System\n        | User\n        | UserThreadsafe\n        | RDRand\n\n    rngTypeName :: RNGType -> ByteString\n    rngTypeName System         = BOTAN_RNG_TYPE_SYSTEM\n    rngTypeName User           = BOTAN_RNG_TYPE_USER\n    rngTypeName UserThreadsafe = BOTAN_RNG_TYPE_USER_THREADSAFE\n    rngTypeName RDRand         = BOTAN_RNG_TYPE_RDRAND\n\n\nWe still point to the constants declared in `botan-bindings`, but the ADT is much friendlier.\n\n`rngGet` is our first real test - both as the first testable action of any `BotanObject` using the new initializers, but also as a test of our allocators as a means of getting data out of the C / Botan world:\n\n\n    rngGet :: RNG -> Int -> IO ByteString\n    rngGet rng len = withRNG rng $ \\ botanRNG -> do\n        allocInit ByteStringAllocator len $ \\ fPtr -> withPtr fPtr $ \\ bytesPtr -> do\n            throwBotanIfNegative_ $ botan_rng_get botanRNG bytesPtr (fromIntegral len)\n\n    systemRNGGet :: Int -> IO ByteString\n    systemRNGGet len = allocInit ByteStringAllocator len $ \\ fPtr -> withPtr fPtr $ \\ bytesPtr -> do\n        throwBotanIfNegative_ $ botan_system_rng_get bytesPtr (fromIntegral len)\n\n\nAh! The only beef I have with my design here is that when the `ByteStringAllocator` is initialized, the `Mem ByteString` is actually `ForeignPtr Word8` and so we need an extra `withPtr` step to drill down to `Ptr Word8` - this isn’t necessarily a bad thing, but I will probably put a convenience function in the ByteStringAllocator module to do this automatically. Otherwise, this is dirt simple!\n\nThese functions didn’t really change (withRNG silently handled the transition):\n\n\n    rngReseed :: RNG -> Int -> IO ()\n    rngReseed rng bits = withRNG rng $ \\ botanRNG -> do\n        throwBotanIfNegative_ $ botan_rng_reseed botanRNG (fromIntegral bits)\n\n    rngReseedFromRNG :: RNG -> RNG -> Int -> IO ()\n    rngReseedFromRNG rng source bits = withRNG rng $ \\ botanRNG -> do\n        withRNG source $ \\ sourcePtr -> do\n            throwBotanIfNegative_ $ botan_rng_reseed_from_rng botanRNG sourcePtr (fromIntegral bits)\n\n\nFinally, we make use of `withPtr` to supply entropy to an RNG.\n\n\n    rngAddEntropy :: RNG -> ByteString -> IO ()\n    rngAddEntropy rng bytes = withRNG rng $ \\ botanRNG -> do\n        withPtr bytes $ \\ bytesPtr -> do\n            throwBotanIfNegative_ $ botan_rng_add_entropy botanRNG (ConstPtr bytesPtr) (fromIntegral $ ByteString.length bytes)\n\n\nThat is it. RNG module refactored. Got energy for one more?\n\n## Hash\n\nStep 1: Define our object and memory instances, same as RNG.\n\n\n    instance Memorable BotanHash where\n        type Mem BotanHash = Ptr\n        type MemRep BotanHash = BotanHashStruct\n        withMem (MkBotanHash ptr) action = action ptr\n\n    newtype Hash = MkHash { foreignPtr :: ForeignPtr BotanHashStruct }\n\n    instance Memorable Hash where\n        type Mem Hash = ForeignPtr\n        type MemRep Hash = BotanHashStruct\n        withMem (MkHash fptr) action = action fptr\n\n    instance BotanObject Hash where\n\n        type BotanStruct Hash = BotanHashStruct\n        type BotanPtr    Hash = BotanHash\n\n        toBotanPtr = MkBotanHash\n        toBotan    = MkHash\n        botanFinalizer  = botan_hash_destroy\n\n    withHash :: Hash -> (BotanHash -> IO a) -> IO a\n    withHash = withBotanPtr @Hash\n\n\nInitializing a hash is pretty much the same as RNG:\n\n\n    hashInit :: HashType -> IO Hash\n    hashInit htype = ByteString.useAsCString (hashTypeName htype) $ \\ htypePtr -> do\n        createBotan @Hash $ \\ outPtr -> botan_hash_init outPtr (ConstPtr htypePtr) 0\n\n\nThe `Hash` type is a bit more complicated though  In a _good_ way…\n\n\n    data HashType\n        = BLAKE2b BLAKE2bSize\n        | GOST_34_11\n        | Keccak1600 Keccak1600Size\n        | MD4\n        | MD5\n        | RIPEMD160\n        | SHA1\n        | SHA224\n        | SHA256\n        | SHA384\n        | SHA512\n        | SHA512_256\n        | SHA3 SHA3Size\n        | SHAKE128 SHAKE128Size\n        | SHAKE256 SHAKE256Size\n        | SM3\n        | Skein512 Skein512Size Skein512Salt\n        | Streebog256\n        | Streebog512\n        | Whirlpool\n        -- Combination strategies\n        | Parallel HashType HashType\n        | Comb4P HashType HashType\n        -- Checksums\n        | Adler32\n        | CRC24\n        | CRC32\n        deriving stock (Eq, Ord, Show)\n\n    hashTypeName :: HashType -> ByteString\n    hashTypeName = implementation omitted\n\n\nWe actually ran into our first major hiccup in the Hash module - in botan, many functions require that you supply a buffer function with a size pointer with the value set to `0` in order to query for the required buffer length. This is a bit awkward, but I have created this small helper function:\n\n\n    queryBufferLength :: (Ptr CSize -> IO CInt) -> IO Int\n    queryBufferLength fn = mask_ $ alloca $ \\ szPtr -> do\n        Ptr.poke szPtr 0\n        code <- fn szPtr\n        if code == BOTAN_FFI_ERROR_INSUFFICIENT_BUFFER_SPACE\n            then fromIntegral <$> Ptr.peek szPtr\n            else throwBotanError code\n\n\nThis works for a single buffer, but there are functions that take two buffers, with two size pointers - I’m not entirely sure how this will hold up, but we’ll get to that in due time. For now, this works quite well!\n\nHere we use it to query the length of the buffer required to retrieve the hash’s name from a context object:\n\n\n    hashName :: Hash -> IO ByteString\n    hashName h = mask_ $ withHash h $ \\ hPtr -> do\n        n <- queryBufferLength $ \\ szPtr -> botan_hash_name hPtr nullPtr szPtr\n        name <- alloca $ \\ szPtr -> do\n            Ptr.poke szPtr (fromIntegral n)\n            allocInit ByteStringAllocator n $ \\ fPtr -> withPtr fPtr $ \\ namePtr -> do\n                throwBotanIfNegative_ $ botan_hash_name hPtr (cast namePtr) szPtr\n        -- TODO Check n vs the actual length\n        return $!! ByteString.takeWhile (/= 0) name\n\n\nNow, I have mixed feelings about this because `queryBufferLength` allocates a temporary size and returns its value only to immediately shove it into a different temporary size pointer, I think we could re-use the same size pointer but I want to wait until I deal with multi-buffer functions first - so for now, this inefficiency is acceptable, and is dwarfed by the need to copy the name buffer anyway.\n\nThis function barely changed - it just uses `createBotan` now\n\n\n    hashCopyState :: Hash -> IO Hash\n    hashCopyState source = withHash source $ \\ sourcePtr -> do\n        createBotan @Hash $ \\ outPtr -> botan_hash_copy_state outPtr sourcePtr\n\n\nNo more `mkAction`, we just use `withHash`\n\n\n    hashClear :: Hash -> IO ()\n    hashClear h = withHash h $ \\ botanHash -> do\n        throwBotanIfNegative_  $ botan_hash_clear botanHash\n\n\nThese are a little bit longer since now they call `alloca` themselves instead of a convenience function, but they are still very clean:\n\n\n    hashBlockSize :: Hash -> IO Int\n    hashBlockSize h = mask_ $ alloca $ \\ szPtr -> do\n        withHash h $ \\ botanHash -> do\n            throwBotanIfNegative_ $ botan_hash_block_size botanHash szPtr\n            fromIntegral <$> Ptr.peek szPtr\n\n    hashOutputLength :: Hash -> IO Int\n    hashOutputLength h = mask_ $ alloca $ \\ szPtr -> do\n        withHash h $ \\ botanHash -> do\n            throwBotanIfNegative_ $ botan_hash_output_length botanHash szPtr\n            fromIntegral <$> Ptr.peek szPtr\n\n\nFinally, we come to the actually hashing functions. Because of `withPtr` and `allocInit`, they are pretty trivial - almost exactly the same as the `rngGet` in `RNG\n\n\n    hashUpdate :: Hash -> ByteString -> IO ()\n    hashUpdate h bs = withHash h $ \\ hPtr -> do\n        withPtr bs $ \\ bsPtr -> do\n            throwBotanIfNegative_ $ botan_hash_update hPtr (ConstPtr bsPtr) (fromIntegral $ ByteString.length bs)\n\n    type HashDigest = ByteString\n\n    hashFinal :: Hash -> IO HashDigest\n    hashFinal h = withHash h $ \\ hPtr -> do\n        sz <- hashOutputLength h\n        allocInit ByteStringAllocator sz $ \\ fPtr -> withPtr fPtr $ \\ digestPtr -> do\n            throwBotanIfNegative_ $ botan_hash_final hPtr digestPtr\n\n\nAnd that’s it! That’s 2 modules down, with more following soon.\n\n# Conclusion\n\nAs a first real test of the memalloc library, I am proud of the result. Refactoring these 2 modules shows that it is more than sufficient to replace the original `Make / Remake / mkBindings`, since the new modules don’t use _any_ of that anymore.  Exposing the allocator and generalizing from bytestring to bytes will only take a little more work once we are ready for that, too.\n\nThese newly refactored modules won’t be live until I get the `memalloc` library up on hackage, since it is nows a dependency, so I have got to finish getting a few last things in shape, but I’ll be continuing to update the `botan-low` modules in the meantime - there’s nothing quite like having a use case to drive development forward.",
  "title": "Botan bindings devlog"
}