Real-time AudioVisual Composition with RayTone at The Knoll – Stanford, CA

Overview

Are you interested in experimenting with sound and visuals to create an original multimedia performance? This five-day workshop offers an introduction to real-time audiovisual composition using RayTone – a node-based sequencing environment designed to promote a playful workflow for transforming creative ideas into artistic content.

The workshop is hosted by RayTone co-author Eito Murakami, an artist and engineer working in computer music and graphics. We will also welcome Ilan Salomon-Jacob as a guest lecturer to introduce ChucK – a strongly-timed music programming language. With ChucK embedded in RayTone, users can customize behaviors of audio units that perform digital signal processing at run time. Additionally, RayTone allows users to load OpenGL Shading Language (GLSL) scripts for graphics programming such that audio and visual elements can be controlled simultaneously.

The workshop is structured into three categories: an introduction to RayTone sequencing, lectures on ChucK and shader programming, and activities to design original RayTone patches. In addition to learning from instructors, attendees will exchange knowledge and collaborate with each other to develop creative ideas. Upon completion of the workshop, attendees will learn to combine digital signal processing with graphics programming to perform real-time audiovisual composition. There will be an opportunity to present an original piece using RayTone on the last day of the workshop. Prior musical and programming experience may be helpful but not required.

More information can be found at https://www.raytone.app.
If you have any questions, please feel free to contact Eito at eitom@ccrma.stanford.edu

Requirements

Please bring a laptop (Windows / MacOS) and a pair of headphones to the workshop. CCRMA has a few extra pairs of headphones for those who cannot bring their own.

Schedule

Day 1: Introduction to RayTone & Digital Signal Processing

• Self introduction

• RayTone example projects

• Digital Signal Processing (DSP) with ChucK

– discrete time sampling

– additive / subtractive synthesis

– filter design

Day 2: More DSP + Shader Programming

• More DSP with ChucK

– granular synthesis

– physical modeling

• Shader programming

– GLSL examples

Day 3: More Shaders + RayTone development

• More shader programming

• Integrating webcam, images, and videos with shaders

• (Optional: Anatomy of RayTone Engine)

Day 4: Final project preparation

• Musical composition with MIDI controllers

• Communication with other software via OSC

• Expressive audiovisual integration

Day 5: Presentations

• More preparation time

• Presentations by participants!

About the instructors

Eito Murakami is a Ph.D. student at Center for Computer Research in Music and Acoustics (CCRMA) at Stanford University. He graduated from University of California San Diego with bachelor’s degrees in Interdisciplinary Computing and the Arts Music (ICAM) and Political Science/International Relations. Eito is an electronic composer, performer, sound designer, and virtual reality developer. By combining his classical music training with proficiency in audio and graphics software, he creates digital interfaces and instruments that promote playful workflows for multimedia performances. At CCRMA, his research involves designing audio playback systems in virtual reality to process dynamic spatial reverb and multiplayer interactions.