Andy Black
Help people. Get paid. Help more people.
Staff member
FASTLANE INSIDER
EPIC CONTRIBUTOR
Read Fastlane!
Speedway Pass
Ha. It's been a while since I read something that made me feel this hopelessly clueless.Get LM Studio on your PC. Here you can download huggingface OpenSource models and run local LLMs on your machine with a clean UI. This gets you 90% of the way with most LLM capability. There is vision capability but it's limited in LM studio at the moment.
There are also dozens of wrappers for different diffusion models.
There are many user interfaces for linking and piping different models together. I think node based programming interfaces are ideal for this type of work - and I use them accordingly. Touchdesigner is a super powerful software I use for interfacing with these models - check it out if you've never heard of it.
Play with NDI and OSC protocols for piping data between devices. For instance you can create a UI for the phone or ipad to interface with the model running on your PC using OSC to send/receive data. The model's memory runs on your GPU and you can use NDI which provides access to your GPU memory across applications.
It's fun to experiment with - and the pace of opensource innovation really levels the playing field. It means that any day, anybody can pick up a brand new model and develop a unique engine.