This is pretty clever. As I understand it.

1. Because LLMs are slow most of them stream the response to the user.
2. The response is streamed as text, but generated in tokens.
3. This means that each “chunk” leaks the length of the text corresponding to the token.
4. You can then use heuristics to guess the text of the response based on the token lengths.

This is a good reminder any time you are sending content in small chunks over an encrypted channel, many encrypted channels don’t provide protection against size leaks by default.

It seems there are a few easy solutions to this:

1. Send the token IDs (as fixed-size integers) over the network rather than the text.
2. Pad the text representations of the tokens to a fixed length.
3. Batch the tokens more (and maybe add padding) to produce bigger chunks and obscure individual token size.

These still all leak the approximate length of the response, but that is probably acceptable.