K-Mix API, Part 2 – Controlling K-Mix with Leap Motion and the Web MIDI API

10 min read

If you've ever fantasized about mixing with your hands similar to how you control a Theremin or in the movie Minority Report, your dreams are now closer to reality! With the latest K-Mix update, which enables sending MIDI message to fully control K-Mix, combined with the the K-Mix API and the incredible Leap Motion controller, I'll demonstrate how you can control K-Mix with only your hands and fingers, without touching K-Mix at all! K-Mix used with the K-Mix API brings a new meaning to the term 'Programmable Mixer'.

The project below allows you to control K-Mix's faders 1-8 (plus the master fader), as well as stereo panning for each channel fader. We have a simple Ableton Live project setup with 8 looped tracks of the K-Mix 'theme song' by Emmett Corman with each track assigned to its own channel on K-Mix. You could also easily control K-Mix's other features such as onboard DSP and use instruments / mics directly connected to K-Mix's inputs.

Controls:

  • Left hand (fingers, excluding thumb): faders 1-4
  • Right hand (fingers, excluding thumb): faders 5-8
  • Fist: Master fader

Fader level is controlled in the vertical axis (moving your finger up/down), and stereo panning is controlled in the horizontal axis (moving your finger left/right).

Let's get started!

Sending MIDI to K-Mix

There are 3 different send modes with K-Mix:

k-mix-audio-control (default)

Sending to the 'k-mix-audio-control' port controls the K-Mix Mix Bank, which controls all of the audio features of K-Mix like changing fader levels, EQ, and spatialization in addition to sending preset change messages.

Inputs 1 – 8:
All of the per input channel [1-8] automatable control parameters are listed in section 4.3.3 of the Manual. The input channel is denoted by ':channel', so for controlling input channel 3's fader level, you would use 'fader:3'.

Main/Master Fader:
In order to control the main fader, send to 'main:fader', which controls the main fader level.

Misc (reverb/surround/auxes):
To control K-Mix's miscellaneous features (reverb/surround/auxes), covered in section 4.3.3.4 in the Manual, send to 'misc:reverb-level', which controls the K-Mix's main reverb level.

Presets:
To change a preset on K-Mix, just send a message to 'preset'. K-Mix has 12 onboard presets.

// inputs 1-8
kmix.send('fader:1', 127)
kmix.send('fader:1', 127, time) // with time
// control input 1 high-frequency EQ parameter
kmix.send('eq-high-frequency:1', 127)

// Main Fader
kmix.send('main:mute', 11) // auto channel 9

// Misc
kmix.send('misc:reverb-level', 100) // auto channel 10

// load preset #2
kmix.send('preset', 2)

k-mix-control-surface

If you wish to control the K-Mix's 3 MIDI Banks, covered in section 4.3.4 of the Manual, you send to the 'k-mix-control-surface' port, referred to in this API as 'control'. The main purpose of sending to the 'control' port is for K-Mix LED control.

As seen below, the desired control must be prepended by 'control:'. The optional 'time' and 'bank' [1-3] parameters follow, 'bank' defaults to 1. If you wish to send to a different bank, you must set the time argument to either 0 or millisecond value before the bank argument.

kmix.send('control:fader-3', 127 [, time, bank]) // control, value, time, bank
// if you wish to send to a different bank, you must set the time argument to either 0 or millisecond value
kmix.send('control:rotary-1', 64, time, 2) // set rotary-1 to 64 in bank 2
// in order to control buttons, a button must be in 'toggle' mode
kmix.send('control:button-vu', 127) // turn button-vu on; any value > 0
kmix.send('control:button-vu', 0) // turn button-vu off

raw MIDI messages

If you think in midi messages, you can send raw midi messages to K-Mix. You pass in the byte array with optional time and the port you want to send to, which defaults to 'k-mix-audio-control'. If you want to send to the 'k-mix-control-surface', add 'control' after time.

kmix.send([176, 1, 127] [, time])
kmix.send('control', [176, 1, 127] [, time]) // send to 'k-mix-control-surface'
// you can also use hex values [0xB0,1,0x7F]
/*
Note: all faders and rotaries are of type 176 + channel,
all buttons are of type 144 for on and 128 for off.
*/

Leap Motion

You can use a Leap Motion controller with various programming languages and platforms, but since we're working with the browser and the Web MIDI API, we want to use the JavaScript SDK, which is nicely documented here.

Getting data from Leap Motion was pretty easy, everything happens in the Leap loop as shown below:

Leap.loop({
  hand: function(hand){
    // the magic happens here...
    // Everything we want is in the returned 'hand' object.
  }
})

Leap Motion allows you to get a staggering amount of detail and data from the movement of your hands, fingers, and joints, but in this project, we're only interested in three things:

  1. which hand we're using
  2. what finger we're using
  3. is our hand is closed to a fist

We can get figure out what hand we’re using by simply using ‘hand.type‘ inside our Leap loop. This will return either ‘left’ or ‘right’. To determine which fader our fingers control, i’m figuring out how many fingers are extended, excluding thumbs, and if we’re using our right hand adding 4 to that number, see line 10 below. Leap Motion also makes it really easy to figure out if your hand is closed to a fist or not by seeing if ‘hand.grabStrength < 1’, where 1 signifies a closed fist.

Once we know which and and finger we’re using, we can get the position data (X, Y, and Z) for that finger. There are a few ways to get position data but we’re going to use ‘stabilizedTipPosition’ as that is, as its name implies, more stabile.

Once we have our finger position data we will need to scale it to MIDI range (0-127) using a simple range conversion function ‘convertRange’, see lines 39 – 49 below.

Lastly, we need to send our position data to K-Mix using the K-Mix API.

//kmix.send('fader:1', 127)
kmix.send('fader:' + controlNumberFader, percentageY * 127)
//kmix.send('pan-main:1', 64)
kmix.send('pan-main:' + controlNumberFader, convertRange(horizontalPosition, [-350,350], [0,127]))

Here is the complete code for our Leap loop:

// LeapMotion
let horizontalPosition, percentageX, verticalPosition, percentageY, zPosition, percentageZ, 
		extendedFingers = [], addFingers = 0, controlNumberFader = 1, controlNumberRotary = 1

Leap.loop({
 
  hand: function(hand){
    // console.log( hand );
    
    addFingers = (hand.type === 'left') ? 0 : 4;

    if(hand.grabStrength < 1){

	    fingers.forEach( f => {
	    	if(f === 'thumb') return;
	    	
	    	controlNumberFader = (extendedFingers.length + addFingers) ? (extendedFingers.length + addFingers) : 1;
	    	controlNumberRotary = (extendedFingers.length) ? (extendedFingers.length) : 1;

	    	if(hand[f].extended){
	    		// add fingers
	    		if(!extendedFingers.includes(f)) extendedFingers.push(f)

	  			horizontalPosition = hand[f].stabilizedTipPosition[0];
	  			percentageX = getPercent(horizontalPosition);

	    		verticalPosition = hand[f].stabilizedTipPosition[1];
	    		percentageY = getPercent(verticalPosition);

	    	} else {
	    		// remove fingers
	    		if(extendedFingers.indexOf(f) !== -1) extendedFingers.splice(extendedFingers.indexOf(f), 1)
	    	}
	    
	    })
    	// console.log('# extended', extendedFingers);
    	// console.log('controlNumberFader', controlNumberFader);

    	kmix.send('fader:' + controlNumberFader, percentageY * 127)
    			.send('pan-main:' + controlNumberFader, convertRange(horizontalPosition, [-350,350], [0,127]))    
  		
  		controlSVGFader('fader-' + controlNumberFader, percentageY * 127, svg)
  		controlSVGRotary('rotary-' + controlNumberRotary, convertRange(horizontalPosition, [-350,350], [0,127]), svg)
  	} else {
  		verticalPosition = hand.stabilizedPalmPosition[1];
  		percentageY = getPercent(verticalPosition);
  		kmix.send('main:fader', percentageY * 127)
  		
  		controlSVGFader('fader-master', percentageY * 127, svg)
  	}
  }
 
}).use('screenPosition');

function convertRange( value, r1, r2 ) { 
    return ( value - r1[ 0 ] ) * ( r2[ 1 ] - r2[ 0 ] ) / ( r1[ 1 ] - r1[ 0 ] ) + r2[ 0 ];
}

function getPercent(position){
	return Math.max(Math.min(((position - 50) / 350), 1), 0)
}

If you don’t have a K-Mix yet, you can watch the video below to see it in action:

If you're lucky enough to have gotten your hands on a K-Mix and have a Leap Motion controller, plug them in and enjoy touchless mixing!

Try the demo:

Play around with the app in fullscreen here https://files.keithmcmillen.com/blog/webmidi/k-mix/leapmotion-mixer/

The source code for this project is available here: https://github.com/keithmcmilleninstruments/k-mix-api-leapmotion-mixer

Published on
10 min read