Monday Update

No meeting notes today, I was the only one in attendance. Last week saw the start of the botan-low + memalloc refactor, mostly getting our feet planted beneath us by dissecting the existing RNG interface, and discussing what we wanted to preserve and what we wanted to replace. Major points of were the complexity of the Internal.ByteString, Make, and Remake modules, the resulting mkBindings as a method of constructing context objects, and a quick review of the RNG interface. Later, in the memory thread we presented an in-depth discussion of the new Memory interface that will be handling the allocation and object construction moving forwards.

Continuing where we left off:

A look at mkBindings and handling context objects - BotanRNG vs RNG

One of the things that I have been mulling over is the necessity of unwrapping and wrapping pointers. The mkBindings function is obviously a critical component, it performs a vital duty, so we cannot get rid of it - and yet it is ugly and terrible. If anything could be improved - here it is. So let us look closer to understand why it exists; then we can replace it.

Botan.Bindings.RNG defines BotanRNG in the following manner:

-- | Opaque RNG struct
data {-# CTYPE "botan/ffi.h" "struct botan_rng_struct" #-} BotanRNGStruct

-- | Botan RNG object
newtype {-# CTYPE "botan/ffi.h" "botan_rng_t" #-} BotanRNG
    = MkBotanRNG { runBotanRNG :: Ptr BotanRNGStruct }
        deriving newtype (Eq, Ord, Storable)

foreign import capi safe "botan/ffi.h botan_rng_init"
    botan_rng_init
        :: Ptr BotanRNG
        -> ConstPtr CChar
        -> IO CInt

foreign import capi safe "botan/ffi.h &botan_rng_destroy"
    botan_rng_destroy
        :: FinalizerPtr BotanRNGStruct

Then, Botan.Low.RNG defines RNG in the following manner:

newtype RNG = MkRNG { getRNGForeignPtr :: ForeignPtr BotanRNGStruct }

withRNG     :: RNG -> (BotanRNG -> IO a) -> IO a
rngDestroy  :: RNG -> IO ()
createRNG   :: (Ptr BotanRNG -> IO CInt) -> IO RNG
(withRNG, rngDestroy, createRNG)
    = mkBindings MkBotanRNG (.runBotanRNG) MkRNG (.getRNGForeignPtr) botan_rng_destroy

rngInit
    :: RNGType
    -> IO RNG
rngInit = mkCreateObjectCString createRNG botan_rng_init

If we proceed through this step by step (and ignore how awkwards mkBindings is), we note that it is not all that complicated, and that we have several reasons for doing what we did. Making improvements here is valuable, because most all Botan context objects follow this pattern, so whatever we do here probably will affect every other module in a similar manner (we do have a high degree of consistency in that regard).

• We have an opaque Botan struct type signified by an empty data type BotanRNGStruct - this makes sense because as an opaque type, we can never actually see an instance of it.
• We have a newtype wrapper around a Ptr to a BotanRNGStruct - we can actually have an instance of this, it is just a pointer to an opaque struct.
• The empty data BotanRNGStruct and newtype BotanRNG are necessary in order to use CApiFFI and CTYPE, due to type safety requirements.
• This requires boilerplate / wrapping / unwrapping to use the pointer, but is a zero-cost abstraction
• The Ptr is unmanaged, and we must allocate and provide one to be filled
• There is a FinalizerPtr in the form of botan_rng_destroy, but Ptr needs to be a ForeignPtr to use it
  • Maybe we should have both botan_rng_destroy and botan_rng_destroy_finalizer
• botan-low lifts the BotanRNG from a Ptr BotanRNGStruct to a ForeignPtr BotanRNGStruct so we can finalize the memory when no references remain
  • The RNG newtype isn’t strictly necessary, unlike BotanRNGStruct and BotanRNG; it provides value but could be reduced to a type alias.
  • The Botan API requires that we allocate a temporary pointer Ptr BotanRNG (that is, a Ptr (Ptr BotanRNGStruct)) for the ‘create’ function to populate
  • Then, mkBindings takes that unmanaged Ptr and stuffs it into a ForeignPtr for the GC to track
  • The temporary pointer uses alloca so it is good to know we are not leaking that pointer either
• mkBindings and eg mkCreateObjectCString are mostly just thin wrappers for converting to and interacting with the inner foreign pointer
• The mkCreateObjectFoo functions sort of combine this lifting while also handling initialization arguments, but may no longer provide as much / any convenience due to the new Memory classes
• We didnt have Memory.Pointer last time to handle things, so mkBindings filled the gap
• The new Memory.Memory and Memory.Pointer classes allow us to provide the same functionality more naturally - I can just access the inner pointer using withMem (which also replaces the withFoo that mkBindings generates)

So, it turns out that some of the complexity is unavoidable - especially the BotanRNGStruct vs BotanRNG vs RNG stuff. We really do need to unwrap the Ptr and pack it into a ForeignPtr - to allow it to be tracked by the GC, and because a ForeignPtr BotanRNG is a ForeignPtr (Ptr BotanRNGStruct) and not a ForeignPtr BotanRNGStruct. We could turn RNG into a type instead of a newtype but type safety would suffer and I’d rather take advantage of the new Memory classes.

But, it seems that we should be able to more or less get rid of mkBindings, if we don’t really need any of its functions anymore! The new memory classes can handle both allocation (create and destroy) and pointer access (withPtr), so all that complexity can just be tossed out.

This does mean that our context objects will have to conform to some classes, and the botan-low library will have slightly more responsiblities, but I think upkeep will be more tolerable, and as a bonus we’ll be polymorphic over the return type of various functions so we can use anything that conforms to the new memory classes instead of being stuck with ByteString. That, I think, makes all this worthwhile.