[Pre-RFC] BTF relocations

Jules-Bertholet:

vad:

It's not strictly necessary, because matching container type, its name and matching field name are sufficient for the relocation to work.

In one of the posts you linked, I see mention of version suffixes in the names, which are ignored for relocations. Should that mechanism work in Rust as well? (If not, then an alternative mechanism, e.g. attribute, is presumably necessary.)

Good point, sorry for missing it. Supporting version suffixes sounds like something we should do and the way clang implemented it makes sense to me. I will try to implement it and report once done.

Unsure about alternatives. If we introduce a custom attribute instead, that would likely require more compiler and language-design work to carry the versioning information through the pipeline. The clang approach seems less intrusive and already has an existence proof.

Jules-Bertholet:

vad:

Jules-Bertholet:

Is there any connection to/similarity with Swift stable ABI?

Not really. Actually I don't even think Rust ABI should be used for BTF relocatable types. #[repr(btf)] should enforce the C ABI, I'm going to update the RFC and examples accordingly.

To clarify the point I was driving at: Swift's ABI supports making limited changes to struct layouts while preserving compatibility, so it's conceptually similar to BTF relocations. I was wondering if any new syntax or high-level language APIs we add to support BTF relocations could also be leveraged to support Swift interop, even if the backend implementation is very different.

Thanks for the clarification, I get your point now.

The syntax I initially proposed for BTF relocations was a new attribute (it's still in the pre-RFC text):

#[btf_preserve_access_index]
#[repr(C)]
pub struct foo {
    [...]
}

However, @ais523 proposed a new type representation repr(btf):

#[repr(btf)]
pub struct foo {
    [...]
}

If we end up sticking with the first idea, a separate attribute, it could be shared between BTF relocations and Swift ABI resilience, if we give it a more neutral name (e.g. resilent). I could imagine it being used as follows in BPF programs:

#[resilent]
#[repr(C)]
pub struct foo {
    [...]
}

And similarly for Swift resilient types, assuming there is repr(swift):

#[resilent]
#[repr(swift)]
pub struct foo {
    [...]
}

The question is whether such a syntax makes sense to developers who are used to Swift. From what I gather, Swift makes all public types resilient if the -enable-library-evolution flag is provided to swiftc. One can opt out for individual types using the @frozen attribute. The question is: what kind of assumptions would work best for repr(swift) types in Rust?

• Should we default to non-resilient types and require people to annotate them with #[resilent]? That would be compatible with the idea of sharing the attribute with BTF types.
• Should we default to resilient types and have an opt-out annotation #[freeze]? That would defeat the idea.

Given that ABI resilience is still opt-in in swiftc, I'm leaning towards the first option.

To sum it up, yes, I think there is a possibility to have the same annotations.

zackw:

If I remember correctly, something like this is also how Swift (or possibly Dart) handles dynamic linking of libraries that provide generics, so there might be some useful lessons there.

I couldn't find any mechanism like that in Dart.

But yes, Swift's ABI resilience and BTF relocations are similar on the surface, but they have some differences as well.

• Swift ABI resilience works at runtime. BTF relocations are applied at load time: the program loader patches the bytecode, and the BPF virtual machine is unaware of the concept of relocation, because it operates in terms of offsets.
• Swift's ABI resilience changes the way types are accessed. Access to BTF types is still based on offsets.

Given these differences, the implementations will likely be very different. One implementation similarity that comes to my mind would be extending PlaceRef::project_field and PlaceRef::project_index, but then the further steps would be completely different.

Jules-Bertholet:

vad:

But if the relocated access still stays within some valid memory region while pointing to the wrong field, then this can become a silent logic bug instead.

Logic bug, or UB?

Potentially UB. A wrong relocation can also remain a silent logic bug, but it may become UB if it causes the program to perform an invalid access, for example by reading a value with the wrong type, size, or alignment assumptions. I still need to experiment with some real examples and see how verifier reacts to such programs.

That said, in Rust we could prevent that by forbidding the value initialization.

comex:

Second, you may be able to come up with a subset of the feature that doesn't depend on the Sized hierarchy at all. This should be easier for BTF types than for scalable vector types. The prototype of scalable vector types is pretty hacky: the compiler treats these types as Sized (despite the lack of a compile-time constant size), with the intent to downgrade to runtime-sized once that becomes a thing. This is because scalable vectors need to be passed by value and treated as Copy, which isn't supported for !Sized types. For BTF types, though, passing by value is not that essential, so you could probably start by making them !Sized, with the intent to upgrade to runtime-sized once that becomes a thing. This would be backwards-compatible and wouldn't break any existing generic code, so maybe it would even be stabilizable? (but I'm only speculating on that.)

The innovation would be having a type be !Sized but still allowing struct field access on it. You still need to work out the proper operational semantics for that, but that seems reasonably orthogonal to the Sized hierarchy.

There is one problem with BTF types starting as !Sized - some kernel structures are nested by value. For example, struct task_struct has the following fields:

struct task_struct {
    [...]
    struct sched_entity		se;
    struct sched_rt_entity		rt;
    struct sched_dl_entity		dl;
    [...]
}

However, I think we could still start an experimental implementation with BTF types being !Sized, with a limitation that only primitive fields (including pointers to other BTF-relocatable structs) can be accessed. Kernel types accessible directly from a BPF program context can be accessed only as raw pointers.

That would still not solve the task_struct case in full. Accesses such as task->pid and task->tgid would fit within that subset, but accesses crossing an inline composite field boundary, such as task->se.vruntime, would remain unsupported.

To elaborate on that, BPF programs usually access kernel data through a context pointer, which in case of kprobe programs (one of the most common types) is pt_regs. Such programs usually limit themselves to inspecting a small number of fields, or sending them to the user-space, rather than traversing large portions of nested kernel state.

The vast majority of BPF programs interacting with kernel types, stripping away all the abstraction provided by Aya, look similar to the following example of a kprobe attached to the try_to_wake_up function:

use core::c_ulong;

// Representation of the registers stored on the stack during a system call.
#[repr(C)]
pub struct pt_regs {
    pub r15: c_ulong,
    pub r14: c_ulong,
    pub r13: c_ulong,
    pub r12: c_ulong,
    pub rbp: c_ulong,
    pub rbx: c_ulong,
    pub r11: c_ulong,
    pub r10: c_ulong,
    pub r9: c_ulong,
    pub r8: c_ulong,
    pub rax: c_ulong,
    pub rcx: c_ulong,
    pub rdx: c_ulong,
    pub rsi: c_ulong,
    pub rdi: c_ulong,
    pub orig_rax: c_ulong,
    pub rip: c_ulong,
    pub cs: c_ulong,
    pub eflags: c_ulong,
    pub rsp: c_ulong,
    pub ss: c_ulong,
}


#[repr(C)]
struct task_struct {
    [...]
    pid: i32,
    tgid: i32,
    [...]
}

// BPF expects programs to be functions in specific sections with integer
// return codes.
#[unsafe(no_mangle)]
#[unsafe(link_section = "kprobe/try_to_wake_up")]
pub fn my_kprobe(ctx: *mut core::ffi::c_void) -> u32 {
    match try_my_kprobe(ctx) {
        Ok(ret) => ret,
        Err(_) => 0,
    }
}

// Convenience helper that allows us to return `Result`.
fn try_my_kprobe(ctx: *mut core::ffi::c_void) -> Result<u32, i32> {
    let regs: *mut pt_regs = ctx.cast();

    // Retrieve the first argument of `try_to_wake_up` of type `task_struct`.
    let task: *const task_struct = unsafe { (*regs).rdi as *const _ };

    // Inspect the `task` - check the fields we are interested in, log them etc.
    let pid = unsafe { bpf_probe_read_kernel(&(*task).pid)? };
    let tgid = unsafe { bpf_probe_read_kernel(&(*task).tgid)? };
    info!(&ctx, "kprobe called: pid: {}, tgid: {}", pid, tgid);

    Ok(0)
}

With BTF relocations supported, the only difference would be a concise definition of task_struct, containing only the fields we intend to use, annotated with #[repr(btf)]:

#[repr(btf)]
struct task_struct {
    pid: i32,
    tgid: i32,
}

This is exactly the kind of access pattern I think an initial !Sized experiment can support. By contrast, an access such as (*task).se.vruntime would still be out of scope, because it requires traversing an inline nested struct by value.

So the point is not that !Sized fully solves CO-RE for task_struct, but rather that it may already cover a useful experimental subset for types one retrieves directly from the BPF context.

ais523:

I think the opsem here is to have what is in effect a global variable (static used like a const) that stores the offset of a particular field, for field projection to be implemented by pointer arithmetic using the offset from the global variable, and for other field accesses to be implemented in terms of field projection. One advantage of doing things this way is that it should desugar pretty easily into MIR or even surface Rust, so the opsem aspects would be confined to stating/implementing the lowering rather than needing to add new opsem rules.

I think my current PoC implementation is quite aligned with this.