{
"$type": "site.standard.document",
"bskyPostRef": {
"cid": "bafyreiecegolpvtwemymshxszocrwl3azdfyd6yobvclzkoozbx26kxjsm",
"uri": "at://did:plc:46ti67tc37qcmwp2vaynk6fq/app.bsky.feed.post/3mmmjmr7wj6r2"
},
"coverImage": {
"$type": "blob",
"ref": {
"$link": "bafkreidmamcgl3qedoalfe4crdond4s4axnr6hq3fn42dev5dj6zamu4da"
},
"mimeType": "image/jpeg",
"size": 75814
},
"path": "/en/blog/2026-akvorado-rib-sharding",
"publishedAt": "2026-05-24T17:50:23.417Z",
"site": "https://vincent.bernat.ch",
"tags": [
"Akvorado",
"BGP Monitoring Protocol",
"1 million routes",
"1",
"2",
"Previous implementation",
"Storing routes in a map",
"Interning routes",
"Why does it not scale?",
"RIB sharding",
"First step: basic sharding",
"Second step: lock-free reads",
"bart",
"ART\nalgorithm",
"benchmarks",
"3",
"optimized Swiss\ntable",
"4",
"5",
"unique package",
"6",
"benchmark",
"AS 12322",
"7",
"Gerhard\nBogner",
"debug\nthis issue",
"8",
"9",
"Go’s\nunique package",
"confirmed",
"this blind change",
"concurrent benchmark",
"10",
"11",
"Akvorado 2.2",
"PR #2433",
"↩",
"PR #254",
"PR #255",
"PR #278",
"PR #2244",
"PR #2245",
"discussion #2287",
"bart 0.28.0",
"pmacct",
"ADD-PATH",
"hash/maphash",
"unsafe",
"Hash() function for the nlri structure",
"RFC 7854",
"RFC 8671",
"RFC 9069",
"KIP-932",
"data provided by Geoff Huston",
"eventually consistent"
],
"textContent": "To associate routing information—like AS paths or BGP communities—to flows, Akvorado can import routes through the BGP Monitoring Protocol (BMP). As the Internet routing table contains more than 1 million routes, Akvorado needs to **scale to tens of millions of routes**.1 This has been a long-standing challenge,2 but I expect this issue is now fixed by using **RIB sharding**, a method that splits the routing database into several parts to enable concurrent updates.\n\n * Previous implementation\n * Storing routes in a map\n * Interning routes\n * Why does it not scale?\n * RIB sharding\n * First step: basic sharding\n * Second step: lock-free reads\n\n\n\n# Previous implementation\n\nAkvorado connects 2 elements to build its RIB:\n\n 1. a **prefix tree** , and\n 2. a **list of routes** attached to each prefix.\n\n\n\nAkvorado BMP RIB implementation without sharding. One single read/write lock.\n\nIn the diagram above, the RIB stores five IPv4 prefixes and two IPv6 prefixes. One of them, `2001:db8:1::/48`, contains three routes:\n\n * from peer 3, next hop `2001:db8::3:1`, AS 65402, AS path `65402`, community `65402:31`,\n * from peer 4, next hop `2001:db8::4:1`, same ASN, AS path, and community,\n * from peer 5, next hop `2001:db8::5:1`, AS 65402, AS path `65401 65402`, community `65402:31`.\n\n\n\nThe `rib` structure is defined in Go as follows:\n\n\n type rib struct {\n tree *bart.Table[prefixIndex]\n routes map[routeKey]route\n nlris *intern.Pool[nlri]\n nextHops *intern.Pool[nextHop]\n rtas *intern.Pool[routeAttributes]\n nextPrefixID prefixIndex\n freePrefixIDs []prefixIndex\n }\n\n\nThe prefix tree uses the bart package, an adaptation of Donald Knuth’s ART\nalgorithm. The benchmarks demonstrate it outperforms other packages for lookups, insertions, and memory usage.3 Plus, the author is quite helpful.\n\n## Storing routes in a map\n\nThe list of routes for each prefix is not stored directly in the prefix tree: it would put too much pressure on the garbage collector by allocating per-prefix arrays.\n\nInstead, the RIB assigns a unique 32-bit prefix identifier for each prefix, either by picking the last available prefix identifier from the `freePrefixIDs` array if any, or using the `nextPrefixID` value before incrementing it. Then, the routes are stored in the `routes` map, leveraging the optimized Swiss\ntable in Go. To retrieve routes attached to a prefix, we look them up one by one in the `routes` map with a 64-bit key combining the 32-bit prefix index with a 32-bit route index matching the position of the route in the list. Akvorado scans routes from the first to the last to find the best one.4 It knows there is no more route if the route key returns no result.\n\n\n type prefixIndex uint32\n type routeIndex uint32\n type routeKey uint64\n\n\n## Interning routes\n\nA route contains a BGP peer identifier, a partial NLRI5, the next hop, and the attributes.\n\n\n type route struct {\n peer uint32\n nlri intern.Reference[nlri]\n nextHop intern.Reference[nextHop]\n attributes intern.Reference[routeAttributes]\n prefixLen uint8\n }\n\n type nlri struct {\n family bgp.Family\n path uint32\n rd RD\n }\n type nextHop netip.Addr\n type routeAttributes struct {\n asn uint32\n asPath []uint32\n communities []uint32\n largeCommunities []bgp.LargeCommunity\n }\n\n\nTo save memory and allocations, NLRI, next hops, and route attributes are “interned:” a 32-bit integer replaces the real value. The mechanism predates the unique package introduced in Go 1.23. We keep it because it has different trade-offs:\n\n * It uses **explicit reference counting** instead of relying on weak pointers.\n * It works with **non-comparable values** implementing `Hash()` and `Equal()` methods.6\n * It uses **explicit pool instances**. This will be useful for sharding.\n * It has **better performance**. See for example this benchmark.\n * It consumes **half the memory** thanks to unsigned 32-bit references instead of pointers.\n * But it is **not safe for concurrent use**.\n\n\n\n## Why does it not scale?\n\nNote\n\nAt AS 12322, we don’t use BMP yet.7 But Gerhard\nBogner had the patience, availability, and technical skills to help me debug\nthis issue.\n\nThe global read/write lock is a bottleneck in this implementation. But how? There are several users of the RIB, each with its own set of constraints:\n\n * The **Kafka workers** look up the RIB to enrich flows with routing information. They are bound by the number of Kafka partitions.8 Akvorado also adjusts their number to ensure efficient batching to ClickHouse. On our setup, the number of workers oscillates between 8 and 16. As we want to observe the latest data, we cannot afford for the Kafka workers to lag too much.\n\n * The **monitored routers** send route updates through the BMP protocol. When connecting, they can send millions of routes.9 After the initial synchronization, updates are sent continuously and may spike from time to time. The router detects a stuck BMP station when its TCP window is full and resets the session in this case. While Akvorado implements a large incoming buffer, it still needs to update the received routes with the write lock held fast enough to avoid being detected as stuck.\n\n * When a **remote BGP peer goes down**, Akvorado flushes the associated routes by walking the RIB with the write lock held. When a **monitored router goes down** , Akvorado waits a bit but eventually flushes all the associated routes.\n\n\n\n\nIn short: on a busy setup, lock contention is high for both readers and writers, and neither can lag too much behind.\n\n# RIB sharding\n\n## First step: basic sharding\n\nTo remove the global lock, the RIB is split into several “shards,” each one handling a subset of the prefixes:\n\nAkvorado BMP RIB implementation with sharding.\n\nThe prefix tree stays global and is protected by a single lock. Each shard gets its read/write lock, its route map, and its intern pools to store NLRIs, next hops, and route attributes, which would not have been possible with Go’s\nunique package. The prefix indexes are also sharded: the 8 most significant bits are the shard index and the 24 remaining bits are the local prefix index.\n\nGerhard confirmed that after this blind change, the BMP receiver chugged steadily. 🎉\n\nLater, I wrote a concurrent benchmark over half a million synthetic but plausible routes10 partitioned over 0 to 8 writers, churning routes as fast as possible, while 1 to 16 readers continuously look up a set of 10,000 routes. I don’t know if this benchmark is realistic, but it confirms the improvements for both read and write latencies:\n\nRead and write latency performance improvement after sharding.\n\nIt also shows that a high number of writers degrades read latency.\n\n## Second step: lock-free reads\n\nThe single read/write lock protecting the prefix tree is the next target. The bart package provides alternative mutation methods returning an updated tree using copy-on-write. Readers don’t need the global lock any more, leaving it only to synchronize writers. The prefix tree is boxed in an atomic pointer.\n\nAkvorado BMP RIB implementation with sharding and lock-free reads.\n\nWithout a lock, readers can now fetch a stale prefix index when walking their copy of the tree if a concurrent writer removes the last route attached to this prefix index and recycles it for another prefix. To avoid this issue, we combine the prefix index with a generation number and store them in the tree:\n\n\n type generation uint32\n type prefixRef struct {\n idx prefixIndex\n gen generation\n }\n type rib struct {\n mu sync.Mutex\n tree atomic.Pointer[bart.Table[prefixRef]]\n shards []*ribShard\n }\n\n\nEach shard stores the generation number for each local prefix index. The generation number increases by one if the associated prefix index is freed. When looking up the routes attached to a prefix index, the reader checks if the generation number matches. Otherwise, it assumes the index was recycled and the list of routes is empty.11 You can see this case in the diagram above for prefix index 5, stored with a generation index of 3, while the current value in the `[]generations` array is 4. The generation number could overflow, but it is not a problem as lookups are quick.\n\nRunning the concurrent benchmark against this new implementation shows the improvements for the read latency as soon as the cost of the copy-on-write prefix tree is amortized.\n\nRead and write latency performance improvement after lock-free reads. The middle column shows the cumulative improvements of both steps.\n\n* * *\n\nAmong the multiple attempts to optimize the BMP component, RIB sharding is one of the more satisfying. Akvorado 2.2 implements the first step. PR #2433, drafted while writing this blog post, implements the second step and will be released with Akvorado 2.4. 🪓\n\n* * *\n\n 1. Each router exporting flows doesn’t need to send its routes. When Akvorado does not find a route from a specific device, it falls back to a route sent by another device. It is up to the operator to decide if this is a good enough approximation. ↩\n\n 2. I made many attempts to scale the BMP component. See for example PR #254, PR #255, PR #278, PR #2244, and PR #2245. Despite these efforts, this component remained problematic for some users. See discussion #2287 as the latest example. ↩\n\n 3. It keeps improving: bart 0.28.0 features a new implementation that trades a bit of memory for greater lookup performance. I did not test it yet, as I have been preparing this blog post for a couple of months already. ↩\n\n 4. Akvorado prefers the route matching the exact next hop. Otherwise, it falls back to any other route. This is an approximation. An alternative would be to have one prefix tree for each BGP peer but it would require configuring all routers to export their routes. pmacct’s BMP daemon implements this approach. ↩\n\n 5. If we consider the BGP RIB as a database, the Network Layer Reachability Information (NLRI) is the primary key. Its content depends on the BGP family. With IPv4 or IPv6 unicast, this is the prefix. For VPNv4 and VPNv6 families, it includes the route distinguisher. If you enable the ADD-PATH extension, the NLRI also contains a path identifier.\n\nIn our implementation, we don’t store the prefix as we get it from the looked-up IP address using the separately-stored prefix length. ↩\n\n 6. The `Hash()` methods rely on the hash/maphash package and on the unsafe package to avoid memory copies. See for example the Hash() function for the nlri structure. ↩\n\n 7. Despite being an author or co-author of the first BMP-related RFCs since 2016 (RFC 7854, RFC 8671, RFC 9069), Cisco did not implement it in a usable way in IOS XR until version 24.2.1. We still need to upgrade a few routers to enable this feature. ↩\n\n 8. KIP-932 introduces, in Kafka 4.2, the concept of share groups to enable cooperative consumption on the same partition. This is not supported in Akvorado yet. ↩\n\n 9. You can configure BMP to send routes for each BGP peer before or after applying the incoming policies. In this case, you can get more than one million routes for each transit peer. You can also tell BMP to send the local RIB, which only contains the best path for each prefix. ↩\n\n 10. The prefixes are random, but the prefix size distribution and the AS path length distribution follow the data provided by Geoff Huston. ↩\n\n 11. Alternatively, we could retry the lookup, but it would be pointless: the RIB is an eventually consistent database, and an empty list was a correct answer at some point in the recent past. ↩\n\n\n\n *[BGP]: Border Gateway Protocol\n *[BMP]: BGP Monitoring Protocol\n *[RIB]: Routing Information Base\n *[ASN]: Autonomous System Number\n *[NLRI]: Network Layer Reachability Information\n *[ BGP]: Border Gateway Protocol",
"title": "Vincent Bernat: Scaling Akvorado BMP RIB with sharding",
"updatedAt": "2026-05-24T07:50:40.000Z"
}