CCRMA Workshop: Designing Physical Interactions for Music
Roger's page

This page contains some useful information for participants of the CCRMA workshop:

1) Assembly the Teensy 3.2
2) Assembling the Prop Shield and connecting it to the Teensy and breadboard
3) Assembling a neopixel LED strip and connecting it to the Prop Shield

Roger's presentations:
1) Monday: Musical ideas for various sensor and sound/light output
2) Tuesday: What's inside LinnStrument and how the sensing system works
3) Wednesday: Tips for your projects and presentations
4) Thursday: Making your project into a commercial product



Assembling the Teensy 3.2

You can buy a Teensy 3.2 with the pins pre-soldered from the maker PJRC here. Or if you buy it from Sparkfun or another source, you'll need to solder the pins yourself using the "breakaway headers" that are among the required parts for the workshop. Soldering these pins isn't difficult, and it's good experience if you've never soldered before.

Here's the Teensy without any pins installed, as well as one of the "breakaway headers"" 

Break off two strips of 14 pins each from the breakaway headers. Then insert them into the board as shown and solder them into place. Make sure it looks like the picture BEFORE you solder:

You'll notice that the Teensy has 5 holes at one end that we didn't put pins into. That's because those 5 pins aren't necessary for most things and you can't install them if you want to insert your Teeny into a breadboard like this:

That's it-- you've assembled a Teensy 3.2. Pat yourself on the back.

Assembling the Prop Shield and connecting it to the Teensy and breadboard

The Prop Shield is a board that is made by the same company that makes the Teensy, attaches onto the Teeny, and adds the following functions:
1) an accelerometer, 2) pins to drive cool animated LED strips, and 3) a little audio output amplifier.

Here's how to assemble a Prop Shield and connect it to your Teensy 3.2 and also your breadboard. First, here's a picture of how everything will look once you're done:

As you can see, the Teensy will sit on top of the Prop Shield, and the Prop Shield's pins will stick into the breakboard. To do this, the pins that we'll assemble on the Prop Shield will be "stackable headers" that have pins sticking down (for insertion into the breadboard) but also sockets on top for the Teensy's pins to stick into. Here's the "Teensy Header Kit" that contains the stackable headers:

Insert the two long sections of 13 pins each, and the shorter section of 7 pins, into the Prop Shield's holes as shown below. Make sure it looks exactly like this picture, then solder the stackable headers into place.


Now, cut the bottoms of the 5 pins shown in this picture in red, which will allow the Prop Shield to be inserted into your breakboard:

Now your Prop Shield is done, but you'll also need to solder a 5-pin strip of breakaway headers to the 5 holes at one end of your Teensy 3.2 as shown:

These pins will allow all the necessary connections between the Teensy and Prop Shield when you stack them together.

Now that you've assembled both the Prop Shield and added the final pins of the Teensy, you'll insert the Teensy into the stackable headers of the Prop Shield, and the pins of the Prop Shield into the breadboard, as shown:

Note that adding the 5 final pins to the Teensy won't permit it to be inserted into the breadboard any more, but it doesn't matter because its pins connect through the Prop Shield, which is inserted into the breadboard below, thereby giving the same connection to the breadboard as if the Teensy (without the added 5 pins) had been inserted into the breadboard.

You're done! Now that you've assembled the hardware, you may want to run the accelerometer code described on this page (under the Motion Sensors section) that displays a moving 3D model of the Prop Shield rotating on your screen exactly as you're rotating the real Prop Shield in your hand!


Assembling a neopixel LED strip and connecting it to the Prop Shield

The Prop Shield includes 4 pins to connect to a neopixel LED strip. The strips are lots of fun, allowing to creating animated light shows by individually addressing the color and brightness of each of the LEDs in the strip. And by combining it with the accelerometer, you could use 3D movements of Prop Shield to produce instant changes in light position, color and animation speed in the LED strip. Here's a list of items you'd need to get from Adafruit in order to construct a 5-meter-long LED strip that connects to the 4 pins at the end of the Prop Shield:

* Adafruit DotStar Digital LED Strip - Black 30 LED - Per Meter 5m (BLACK) [ID:2237] = $99.75
* 5V 10A switching power supply[ID:658] = $29.95
* 4-pin JST SM Plug + Receptacle Cable Set, PRODUCT ID: 578
* Female DC Power adapter - 2.1mm jack to screw terminal block[ID:368] = $8.00

Here's how to hook them up:

Here's a page showing how to load some striking animated LED code examples into your Teensy.



Musical ideas for various sensors and audio/light output


Force-sensing resistors

These are flat pads whose resistance varies in proportion to how hard you press on them. (Here's one on Adafruit.) For example, you might use finger pressure to control a lowpass filter frequency, or use 2 FSRs to control both filter frequency and resonance. You can also derive strike velocity by measuring the peak of the pressure envelope.

To read the pressure of an FSR, connect an analog input on Teensy to both one pin of the FSR and one end of a 10k resistor. Then connect the other pin of the FSR to the Teensy's 3.3v pin, and the other end of the 10k resistor to the Teeny's Ground pin. With no pressure on the FSR, the analog input will read 0 volts. With full pressure it will read about 3 volts.

Flex Sensor

These are long strips that change resistance between on how much you bend them. (Here's one on Adafruit.) For example, you could tape flex sensors to measure the bend of your elbows, knees and neck, in order to shape synthesized sounds based on arm and leg movements. These could also be used as a pick detector to sense the action and strength of picking as on a guitar string.


Accelerometer

These devices sense movements in three dimensions: pitch, roll and yaw. The easiest way to implement an accelerometer into your project is with the Prop Shield described above. For example, you could hold the Prop Shield in your hand and use Roll to vary oscillator frequency, Yah to control filter frequency, and Pitch to control filter resonance.
Links:
Product page for Prop Shield: https://www.pjrc.com/store/prop_shield.html 
Video demo: https://youtu.be/5mccPaAEya8?t=24 
Pitch modution: https://youtu.be/cTBMhoi8nhc?t=118


Force-sensing linear potentiometers

These are linear strips that sense both finger position and pressure when you touch the strip. For example, one of these could used to create a musical pitch strip that produces a sound when pressed, with pitch determined by position and loudness determined by pressure. Four of these strips could be made into a 4-string musical instrument. Here's a 4-inch example for $10. These sensors actually sense either pressure or position at a time, so your code needs to rapidly switch between sensing each in order to appear to sense both.


Force-sensing trackpads

These are trackpads that also sense finger pressure. For example, one of these could used to create a musical surface that produces a sound when pressed, with pitch determined by X-axis, filter frequency by Y-axis, and loudness determined by pressure. Here's an example for $18. These sensors actually sense either pressure, X-axis or Y-axis at a time, so your code needs to rapidly switch between sensing each axis in order to appear to sense all simultaneously. Links:
https://www.tangio.ca/single-touch-trackpad-force-sensors 
3D drum strike control: https://youtu.be/HHWrSs8o1V4


Force-sensing multi-touch trackpads

These sense the position and pressure of multiple touches, allow the sensing of chords. https://www.tangio.ca/multi-touch-neo-sensing  


Joysticks

Similar to a trackpad, you can use a 2D joystick to simultaneously control 2 synth parameters with one finger:
https://www.adafruit.com/product/2765 


Distance sensors
‍‍
These are sensors that measure the distance from the sensor to an object like your hand. For example, two of these could be used to make a Theremin, with one measuring left hand distance for volume and the other measure right hand distance for pitch. Here's such a sensor on Adafruit.


Light and color sensors

These are sensors that sensor the brightness of light, as well as the brightness of red, green and blue of that light source. For example, you could alter a synthesizer sound by altering the color of light shining on it. Here's such a sensor on Adafruit.


Air pressure sensors

These air pressure sensors can be used to create an electronic wind instrument, with your breath pressure controlling sound volume. Here's such a sensor on Adafruit.


Depth Camera

Sense 3D shapes like hands/fingers, objects:
https://www.sparkfun.com/products/17770 



Musical ideas for sound/light output

Neopixel strips

These are long strips of addressable LEDs, allowing you to individually control the color and brightness  of each LED within the strip. These can produce engaging light animations.
Here's one on AdaFruit: https://youtu.be/gJzwjhVEqBE
And here's an example of using a Force-Sensing Linear Potentiometer (see above) to move light along a Neopixel strip: https://youtu.be/gJzwjhVEqBE
See above on this page for info on asssembled Teensy and Prop Shield, and assembling the parts for a fully-working LED strip controlled by a Teent Prop Shield.


Teensy Audio Library

The Teensy 4.0 is fast enough to process audio in real time. When used with PJRC's Audio Shield, the Teensy becomes a self-contained stereo audio processing device. Here's the Audio Shield:
https://www.pjrc.com/store/teensy3_audio.html 
PJRC has also created a wonderful software tool called the Audio Library:
https://www.pjrc.com/teensy/td_libs_Audio.html 
The Audio Library is a visual programming interface like Max/MSP or PD. You can simply drag audio modules from the large library of synththesizer or audio effects, connect them with visual wires, then push a button to generate the code that you copy into your Teensy editor. Viola! Instant synthesizer or guitar effects box.


What's inside LinnStrument and how the sensing system works


What's a LinnStrument?

LinnStrument brings the expressive touch control of acoustic instruments to electronic performance. Unlike a piano, the semitones are equally spaced, which is necessary for expressive pitch gestures. So LinnStrument consists of 8 rows of equally-spaced semitones, like strings, tuned in musical fourths or however you like.

LinnStrument product page: https://www.rogerlinndesign.com/linnstrument 
Demo video: https://youtu.be/STz__28Scwc 
Owner video compilation: https://youtu.be/7TTp7GKltHQ 


So what?
 
1) “Why is expressive touch control important”: 
https://www.rogerlinndesign.com/support/test-page 

2) “Why LinnStrument's Fourths String Layout is superior to the piano layout for expressive touch control”:
https://www.rogerlinndesign.com/support/support-linnstrument-fourths-layout 


What’s inside?

Chassis, LED board, main board, touch sensor sheet, silicone playing surface, top panel.
Photo: Exploded view


How does the sensing system work?

Image: sensor sheet

Touch sensor consists of 2 plastic layers separated by a thin air gap. The layers touch only when the playing surface is pressed. The 2 layers are:

1) Bottom sheet: horizontal rows of horizontal position+pressure sensors, to sense X-axis + pressure. The upper side has a layer of fixed resistor material (for sensing X and Y), covered with a second layer of force-sensing material.

2) Top sheet: vertical columns of same position+pressure sensors, to sense Y-axis + pressure. The lower side has a layer of fixed resistor material (for sensing X and Y), covered with a second layer of force-sensing material.

Image: Exploded view of sensor layers: top sheet, spacer, bottom sheet.

The ends of the the rows and columns are connected to analog switches, which connect them to either +3.3v, ground, or ADC (with or without pull-up resistor to +3.3v) depending on whether X, Y or Z (pressure) is being measured. 

To measure X: Row left = ground, row right = +3.3v, column L+R = ADC. Touch position measure X-axis within column.
To measure Y: Column top = +3.3v, column bottom = ground, row L+R = ADC. Touch position measure Y-axis within row.
To measure Z: row L+R = ground, column top/bottom = ADC + pull-up resistor to +3.3v. Pressure on FSR pulls 3.3v closer to ground.

Image: simplified schematic of 3 row by 3 column sensor, with equivalent circuits
Document: patent

To allow multiple simultaneous touches, only one row/column intersection is scanned at a time. The intersections are scanned sequentially at high speed, so it appears to be simultaneous to the musician.

To allow lit pads, light from LED board shines though holes in main board and unprinted windows in touch sensor, then is diffused by silicone playing surface with added white pigment. 3mm diameter unprinted (non-sensing) windows in touch sensor are small enough to not affect sensing accuracy.

Image and video of original 3x3 proof of concept.
Image of 3x3 prototype connected to breadboard with Arduido board and custom analog switch hardware.
Image of complete prototype, including first full sensor, Arduino board, and breadboard with analog switches and LED drivers.

Operating software is 100% open source, based on Arduino Due board for easy source editing. I (Roger) wrote the code for minimal function, then Geert Bevin took over to write the large volume of remaining software, fine-tuning the code to make it into the precise, sensitive and professional musical instrument it is.


Making it manufacturable

Once the prototype worked, a 3D model of the manufactureable design was created in Rhino3D modeling and rendering application. Consideration was given for:

1) Simple manufacturing and test, to reduce cost.
2) Low volume manufacturing techniques wherever possible, like sheet metal and wood, to reduce financial risk.
3) Easy owner servicing by following a video and using only a screwdriver, in case of repair needs.
4) Resistance to transport vibration, heat and humidity
5) Compliance with electrical, emissions, and environmental regulations.

Portions of the 3D model were exported to various fabricators:

1) metal chassis bottom and top, fabricated from sheet metal,
2) wood sides, made from milling wood with CNC machines,
3) circuit boards, layed out by board engineer to meet requirement of mechanical drawings,
4) touch sensor, patented custom design fabricated by Tangio company, and
5) silicone playing surface, made by etching a steel mold and injection molding the parts.


Tips on making an effective project and presentation


My planned talk for Wednesday was "Creating a music product, part 1: How to create a proof of concept of your product idea at low cost". However, I've decided it would be more helpful to give some suggestions on how to make your projects most effective and to give an effective presentation, including time for Q&A about these topics. And I'll include my thoughts for "How to create a proof of concept of your product idea at low cost" in my Thursday presentation.

Here are some tips on making an effective project and presentation:

* If you haven't yet selected a project, here are some sites with ideas:
1) Sasha's site, with examples of past projects on the right side: http://sashaleitman.com/dpi4m-2022/
2) The Guthman Competition, an annual competition for instrument designers at Georgia Tech: https://guthman.gatech.edu. These projects are more elaborate than your can create in our short workshop, but might spark an idea.

* Simple ideas are best. The best instruments are simple to use and understand, but allow development of skills through practice.

* Get it functioning first, then make it pretty. In the short time we have, it's important to solve the technical problems first. Don't waste time at first on making it pretty, or you might be scrambling at the last minute trying to get it working.

* Make it reliable. Murphy's Law: your project will alway fail just before you do your presentation. Use rigid wires instead of flexible wires so that they won't accidentally get pulled out. Any cables to the project board will get pulled, so put a nail in the board and wrap the cables around it, so that cable pulls won't rip all the small wires out of your breadboard. Practice moving the entire project then turning it on, to see if it still works each time.

* Stop, breathe and think to solve problems. Often people will spend hours trying to make their code work, only to later discover that the problem was due to a resistor lead not being pressed fully into the hole in the breadboard. Test your connections with a beeping muli-meter.

* Practice your presentation. Clearly and concisely state 1) what it is, 2) why it's useful, and 3) tech details of how you created it, then 4) give a short and effective performance that shows the merits of the idea. Time it to be certain you won't run over. Practice your presentation on a friend first, then ask if he clearly understood it. Practice is important so that speaking in front of the class doesn't steal your brain function. :)

* Sasha and I are here to help. If you're stumped, we are likely to know how to solve the problem.

* Q & A? 


Making your project into a commercial product


After you make your project, you may be thinking "My project idea is very cool. I think other people would like it. I think I'll make it into a commercial product. If so, here are some tips:

Start small. Don't quit your day job. Take it a step at a time. Don't worry about investment, patents, business licenses, regulations, expensive industrial designs, etc. These things will simply slow you down. Start the product as a side hobby, working evenings and weekends, and focus on learning if other people value your idea as much as you do.

First, streamline your design a little so that it's easier to make in small quantities. For example, mount your Teensy board and controls in a stock box with holes drilled in it for knobs, etc. Make sure it survives drops and being throwing around in a shipping truck. Then try it out on friends to see if they find your UI useable, and guess at a price that is attractive enough for people to buy it. Then build 10, make a web site, send an announcement to some music product web sites, and see how many people buy it. If you sell out, build 25. If those sell out, build 50, etc. Use the money from each build to fund the next build.

Each time you build more, you'll have learned from your customers about what changes are necessary. Refine your design with those changes. With each new build, you'll get closer to what the correct design and price is. For this reason, it's best at first to not to commit to any design elements that are expensive to change.

If you keep making more and people are still buying more, then it's time to make it more manufacturable. Make a circuit board and design that is simple and low-cost to manufacture and test at lower cost.

Even at this stage, I still wouldn't worry about patents. First, it's unlikely that your idea is patentable. Also, patents take a lot of time and money and even if you get one, all it does is give your a better legal case when you must hire a lawyer to sue the infringer. Also, at this early stage I wouldn't worry about business licenses or regulations. No agency will care until you start making money. Plus, electrical/emissions/environmental regulation testing and certification is expensive, and every time your change your design you'll have to re-certify it.

Avoid taking investment. If you do, your company mission statement changes from "I want to make cool things" to "I need to make products that will pay my investor a multiple of his investment".

Try to use low-cost off-the-shelf parts wherever possible:

* Cheap metal boxes from Hammond Manufacturing.

* Try to use parts that are commonly available from digikey.com, mouser.com, McMaster.com. If you choose a rare part, you may find that your entire production is held up for a single part that is out of stock.

* For front panels, either use silk-screen printing or for a slicker look, use pre-printed adhesive labels with pre-cut holes for knobs and panel controls.

* Avoid custom processes with high up-front costs like injection molding. If you need to change your design, you'll have to pay those up-front costs again. Instead, use stock boxes or sheet metal, wood, or 3D printing.

Also, try to design the product for easy owner-servicing by following a video. If a customer problem arises, you can simply send him a replacement circuit board or other part that he can replace himself, which is much cheaper and more convenient than having to return the product to you for repair or send it to a repair shop.

On your site, make lots of videos about how to use it.

Instead of a printed manual, put your entire manual on your site, including videos, FAQs, etc., and keep them updated.

Build into your product an easy way to update the software. There will be bugs.

Most importantly, have fun!