LGTM
← Back to the feed

Post #737

Gergely Orosz @GergelyOrosz

Open on X

OK, this is IMPORTANT to understand for any and all software engineers: Inference engineering 101: reading from the KV cache is very cheap; speculative decoding makes sessions more efficient. Coding use cases make HEAVY use of both. THIS is why inference is SUPER profitable!!

Open on X ↗· 0 views · 0 clicks

Join the conversation with the author on X.