syncing in the hair

Since I work with motion graphics, I’ve been experimenting a lot with synchronization of audio and video. I guess my inspiration of all time is the videoclip for Autechre’s Gantz Graf; after I saw it for the first time I felt I would have researched more in depth, for similar techniques.
Much was done since then, by many; I could present a huge variety of brilliant examples, but the list would be too long, and I would leave too many out of it. Then I won’t, and I’ll just talk of what I did to bring some of the knowledge I had, into the 3d world.

Here’s the video I made as a first test:

What follows is a brief recap of what I did to produce the result, and what I learnt from the experience.

A few days ago I found a great article by Satya Meka, about ways to reproduce some of the great features offered by Trapcode’s Sound Keys plugin.
I chose to start from that solution, both because it’s definitely more flexible than Sound Keys and because, being based on After Effects’ out of the box tools, it could have been managed through scripts and freely developed in future experiments.
In this very first case, I just used Satya’s preset and developed a simple script to write an XML file with normalized data for each instrument. This step ran smooth, as I already had separate tracks which were only to be cleaned, cutting out noise and other instruments’ interference.

Here’s the ExtendScript (it works by selecting the layers created by Satya’s preset, from which you want to export the audio data):

var soundData = new Array();
for (var i=1; i<=app.project.numItems; ++i) {
	var curItem = app.project.item(i);
	if (curItem instanceof CompItem && curItem.selectedLayers.length>0) {
		for(var j=0; j<curItem.selectedLayers.length; j++){
			var curLayer = curItem.selectedLayers[j];
			var curLayerName = curLayer.name;
			var curDuration = (curLayer.outPoint - curLayer.inPoint)*25;
			var from = curLayer.startTime;
			var upTo = curLayer.startTime + curDuration;
			soundData[curLayerName] = new Array;
			var minValue = null;
			var maxValue = null;
			for(var k=0; k<curDuration; k++){
				// pick values frame by frame from the "Final Range Value" effect created by Satya's preset, and populate the array
				var curValue = curLayer.effect("Final Range Value")("Slider").valueAtTime(from+(k/25),false)
				soundData[curLayerName][k] = curValue;
				if(!maxValue || curValue>maxValue){ maxValue = curValue; }
			}
			soundData[curLayerName]['max'] = maxValue;
		}
	}
}

proj = app.project.file.name.split(".");
todayDate = Date(0).split(" ");
date = todayDate[1] + "_" + todayDate[2] + "_" + todayDate[3]

myFileName = "syncaudio.xml";
file = new File(myFileName);
myFile = file.saveDlg("Save file's key infos", myFileName, "TEXT xml");

if (myFile != null) {
	myFile.lineFeed = "Windows";
	myFile.open("w");
	myFile.writeln('<project id="' + proj[0] + '" date="' + date + '">');
	for(i in soundData){
		myFile.writeln('	<track id="' + i + '">');
		for(j in soundData[i]){
			// check the max value for the track, normalize, and write to XML file
			var  curMax = soundData[i]['max'];
			if(j!='max'){
				myFile.writeln('		<value time="' + j + '">' + soundData[i][j]/curMax + '</value>');
			}
		}
		myFile.writeln('	</track>');
	}
	myFile.writeln('</project>');
	myFile.close();
}

What follows is an example of the output XML structure:

	<track id="sax">
		0.03571428510845
		0.03571428510845
		0.01785714255423
		0
		0
		0
		0
		0
		0
		0
		0
		0
		0.0535714297831
		0.0535714297831
		0.03571428510845
	</track>
	<track id="drums">
		0.13809523619965
		0.12380952398185
		0.13333333212705
		0.07619047420872
		0.01428571447958
		0
		0
		0
		0
		0.13809523619965
		0.12380952398185
		0.10476189864432
		0.04761904524956
		0
		0
	</track>

And here’s the Python script to run from Maya, which creates a locator for each instrument and a bunch of nodes for the procedural maps to apply to the fur attributes:

steps = 25
stripFade = 0.2

import maya.cmds as cmd
from xml.etree import ElementTree as ET

tree = ET.parse('C:/path/to/syncaudio.xml')
project = tree.getroot()
timelineEnd = cmd.playbackOptions(q=1,max=1)

def setRampEntry(name,i,pos,r,g,b):
	cmd.setAttr((name+'.colorEntryList[%d].position')%i,pos)
	cmd.setAttr((name+'.colorEntryList[%d].colorR')%i,r)
	cmd.setAttr((name+'.colorEntryList[%d].colorG')%i,g)
	cmd.setAttr((name+'.colorEntryList[%d].colorB')%i,b)

counter = 0
for instrument in project:
	lastFrame = len(instrument) - 1
	if lastFrame>timelineEnd:
		timelineEnd = cmd.playbackOptions(e=1,max=lastFrame)
	name = instrument.attrib.get("id")
	print name
	nullInst = cmd.spaceLocator(n=name)
	cmd.setAttr(name+".translateX",counter)
	lastValue = -1
	lastMod = False
	for value in instrument:
		frame = int(value.attrib.get("time"))
		index = float(value.text)
		if index!=lastValue:
			if lastMod==False:
				cmd.setKeyframe(name, at='translateY', v=lastValue, t=(frame-1))
			cmd.setKeyframe(name, at='translateY', v=index, t=frame)
			lastMod = True
		else:
			lastMod = False
		lastValue = index
	cmd.shadingNode('ramp',at=1,n=name+'Ramp')
	setRampEntry(name+'Ramp',0,0,1,1,1)
	for i in range(1,steps+1):
		cmd.setAttr((name + 'Ramp.colorEntryList[%d].position')%i,(float(i)/steps*0.9)+0.1)
		exprString = "float $t=time1.outTime-%d"%(i) + "; \r\nfloat $n=`getAttr -t $t " + name + ".translateY`; \r\nfloat $v=(1-$n)+($n*%f)"%(float(i)/steps*0.9) + "; \r\ncolorR=$v; \r\ncolorG=$v; \r\ncolorB=$v;"
		cmd.expression(o=(name+'Ramp.colorEntryList[%d]')%i, s=exprString)
	cmd.shadingNode('remapValue',au=1,n=name+'Remap')
	cmd.shadingNode('ramp',at=1,n=name+'Alpha')
	cmd.setAttr(name+"Alpha.type",1)
	cmd.connectAttr(name+"Alpha.outAlpha", name+"Remap.inputValue")
	cmd.connectAttr(name+"Ramp.outColor", name+"Remap.color[0].color_Color")
	stripLen = float(1)/len(project)
	if counter!=0:
		setRampEntry(name+'Alpha',0,(stripLen*counter)-(stripLen*stripFade*3),1,1,1)
	setRampEntry(name+'Alpha',1,(stripLen*counter)+(stripLen*stripFade),0,0,0)
	setRampEntry(name+'Alpha',2,(stripLen*(counter+1))-(stripLen*stripFade),0,0,0)
	if counter!=(len(project)-1):
		setRampEntry(name+'Alpha',3,(stripLen*(counter+1))+(stripLen*stripFade*3),1,1,1)
	counter += 1

Finally, here you are a few tips, regarding what I found and learnt by this experiment:

// composite width height "file_name" file_extension destination start end step padding autoload
composite 256 256 "syncaudio_map" iff images 0 2970 1 4 1;