Some modifications to the agent image recognition service
Changed svs-viewer to a pi backend
Along the way, k3 kept recommending that I build my own solution, but in actual use, the image caching and discarding logic, cache hit rates, and auto-compression logic kept causing all sorts of problems. Using a mature framework is really more appropriate.
At the very beginning, I actually wanted to do it as: image screenshot → summarize and discard the base64 info. But it turned out the cache hit rate was basically 0, which didn't work either. Then I changed to a version that kept all base64, and that was even worse — after just a couple of exchanges, the context would hit around 400k, the model couldn't handle it, and the speed was very slow. Now, the overall approach is to fully switch to pi agent's image management and compression logic, keeping a dynamic cache of 4 images in the middle, compressing images whenever nearing the window limit, and discarding base64 that's too old. But I can already tell this latter logic is still going to have problems — let's see how it runs before making further changes.
Actually, if the model were really cheap or had fast response times, just stuffing everything into the context with auto-compression would be much simpler.
Would it be better to compress the images first, and then feed them into the context? I'll try tuning in that direction next.
Comments
Leave a comment (email only, never shown)
No comments yet.