Support auto-unload for unused JIT loaded models

[eckartal]

Problem

• Users need better memory management when running multiple models
• Currently, once a JIT-loaded model is loaded, it stays in memory unless manually unloaded, which can lead to high memory usage

Success Criteria

• Implement an option to automatically unload JIT-loaded models when they are not in use
• Allow users to set a configurable timeout for unloading models (e.g., X minutes of inactivity)
• Ensure smooth reloading of models when needed without performance drops

Context

https://www.reddit.com/r/LocalLLaMA/comments/1isazyj/comment/mdf099u/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button