CartoonEdgeFilter makes the entire background black

I found the problem. All you should need to do (in addition to the changes you already made) is set FilterPostProcessor’s internal framebuffer format:

fpp.setFrameBufferFormat(Format.RGBA8);

FilterPostProcessor sneakily swaps out the viewport’s target framebuffer with its own before anything gets rendered, and uses a packed RGB format by default.

Edit: interestingly enough, it seems that if the graphics driver doesn’t support packed floating-point textures, either RGB16F, RGBA16F, or RGB8 is used instead, regardless of what is passed to setFrameBufferFormat. So FilterPostProcessor could just randomly override whatever you pass in on some graphics cards.