TD Gemini Toolkit
Thank You UCLA
Development of the open source TouchDesigner Gemini Toolkit was supported by UCLA REMAP and the UCLA School of Theater, Film, and Television, as part of the 2026 AI + Live Arts Hackathon.
About this Project
The TouchDesigner Gemini toolkit is a streamlined bridge that plugs Google’s Gemini AI directly into the TouchDesigner environment. Here you'll find a collection of custom TouchDesigner Components that use the Google Gemini API. Focused on an library independent workflow, this set of components is a batteries included collection that requires no additional libraries or dependencies to get working. The development of these components has focused on using non-blocking procedures in TouchDesigner to ensure there are no external library dependencies and that your project doesn't unexpectedly freeze. Development has focused on allowing for the use of multiple API keys across both studio and enterprise endpoints.
Why Use Machine Learning Tools?
In the past few years we've seen a significant shift of the capabilities of Machine Learning models and applications. To this end we've also seen a growing trend in both educational and commercial sectors to explore the possible applications of AI tooling.
There are a wide variety of perspectives about what comes next in AI, the long term impacts (environment, social, sociopolitical, etc.), and this project isn't aimed at having a particular position on the "value" of Machine Learning's applications. It does however, create a opening for creators who are interested in experimenting and interrogating new tools to understand their benefits and limitations without needing to understand the "how" behind setting up the technical infrastructure.
Considerations and Limitations
Many Machine Learning models still produce inconsistent results. In our own testing we've discovered server timeouts, refusals to generate content (returned issues include trademark violation, content violation, failed generation by the remote serer, etc.) and all sorts of potential challenges. While there are several parameters exposed in the components to help track those issues there are still scenarios where you may find that content fails to generate. At the end of the day these components are an opinionated implementation of the Google Gemini API, and while they are intended to work for most use cases, you may find edges where it doesn't fit your use case.
A Prototyping Toolkit
One position to consider is that many of these tools are great for fast prototyping, but might not standup against real-world conditions (especially for the Studio AI endpoint). One workflow you might consider is using these components to test ideas first, before you do the harder work of figuring out how you want to use AI in your installation, experience, or performance.
Test, Test, Test
If you do find that you're using these components in your live projects, we highly recommend that you set-up a Vertex endpoint. Generally speaking, the pricing is more advantageous, and the permissions have a wider set of scope for controlling API use. We also highly encourage you to reserve time in your process for play testing and load testing. A deployed project will surprise you with how it breaks, and if you can it is well worth setting aside time to test your artwork, performance, installation, etc before it's out in the world. We also recommend performing some load testing - some of the endpoints return faster than others, some models are more expensive than others. Make sure you have some budget and time to test so you're not suddenly surprised by a high utilization bill.
From SudoMagic
We're excited to partner with REMAP from UCLA for this project. SudoMagic has worked on projects that have incorporated different AI mechanics we've frequently re-build some variation of the same set of prototyping components that work asynchronously with AI platforms. This project was an opportunity to look at what standardizing some of that practice might look like. It also helps illustrate where AI works well, and where it still requires a healthy degree of curation.
If you find something broken or want to start a discussion about how these components work, please don't hesitate to add an issue to the project so we can keep it moving forward. If you find that you need to make changes to this project you're also free to fork the repo and start customizing the components so they fit your workflow.
Acknowledgements
Summoned by SudoMagic
