Yes the state is cloned for each request. Otherwise you wouldn't be able to extract an owned state.

I don't know the specifics of your problem, but I'd probably approach this with two separate tasks and an actual external queue using e.g. redis streams. Have one task that answers the request and queues it into the stream, then another task that takes from the stream and forwards / does whatever you intend to do with the batch.

I've been working on async actors with axum and I've found using tokio channels (broadcast/mpsc) to be pretty nice. You could keep a list of receivers in some shared state and then have the GET endpoint stream the events from the running tasks through. I don't have a great idea to avoid locking while writing to the shared state though... I just figure it's fairly fast to write so it shouldn't cause too much contention.

let the type you send over the mpsc contain a oneshot and use this oneshot to notify sender that the message has been handled.

If your library function is blocking, try running it inside tokio::spwan_blocking function, and use mpsc channels to communicate. Since looks like you are blocking your async runtime. This just my guess, will have to check the full code tho.

Looks great, thank you for releasing it as a standalone crate! I'm happy to see an in-depth discussion of the trade-offs between various implementations. Does it use fences or inline assembly for synchronization? I'm asking because they usually cause false positives on Thread Sanitizer , which is my go-to tool for testing concurrency because you can run entire real-world programs with it with reasonable performance (unlike loom or miri).