syncing in the hair

Since I work with motion graphics, I’ve been experimenting a lot with synchronization of audio and video. I guess my inspiration of all time is the videoclip for Autechre’s Gantz Graf; after I saw it for the first time I felt I would have researched more in depth, for similar techniques.
Much was done since then, by many; I could present a huge variety of brilliant examples, but the list would be too long, and I would leave too many out of it. Then I won’t, and I’ll just talk of what I did to bring some of the knowledge I had, into the 3d world.

Here’s the video I made as a first test:

What follows is a brief recap of what I did to produce the result, and what I learnt from the experience.

A few days ago I found a great article by Satya Meka, about ways to reproduce some of the great features offered by Trapcode’s Sound Keys plugin.
I chose to start from that solution, both because it’s definitely more flexible than Sound Keys and because, being based on After Effects’ out of the box tools, it could have been managed through scripts and freely developed in future experiments.
In this very first case, I just used Satya’s preset and developed a simple script to write an XML file with normalized data for each instrument. This step ran smooth, as I already had separate tracks which were only to be cleaned, cutting out noise and other instruments’ interference.

Here’s the ExtendScript (it works by selecting the layers created by Satya’s preset, from which you want to export the audio data):

var soundData = new Array();
for (var i=1; i<=app.project.numItems; ++i) {
	var curItem = app.project.item(i);
	if (curItem instanceof CompItem && curItem.selectedLayers.length>0) {
		for(var j=0; j<curItem.selectedLayers.length; j++){
			var curLayer = curItem.selectedLayers[j];
			var curLayerName = curLayer.name;
			var curDuration = (curLayer.outPoint - curLayer.inPoint)*25;
			var from = curLayer.startTime;
			var upTo = curLayer.startTime + curDuration;
			soundData[curLayerName] = new Array;
			var minValue = null;
			var maxValue = null;
			for(var k=0; k<curDuration; k++){
				// pick values frame by frame from the "Final Range Value" effect created by Satya's preset, and populate the array
				var curValue = curLayer.effect("Final Range Value")("Slider").valueAtTime(from+(k/25),false)
				soundData[curLayerName][k] = curValue;
				if(!maxValue || curValue>maxValue){ maxValue = curValue; }
			}
			soundData[curLayerName]['max'] = maxValue;
		}
	}
}

proj = app.project.file.name.split(".");
todayDate = Date(0).split(" ");
date = todayDate[1] + "_" + todayDate[2] + "_" + todayDate[3]

myFileName = "syncaudio.xml";
file = new File(myFileName);
myFile = file.saveDlg("Save file's key infos", myFileName, "TEXT xml");

if (myFile != null) {
	myFile.lineFeed = "Windows";
	myFile.open("w");
	myFile.writeln('<project id="' + proj[0] + '" date="' + date + '">');
	for(i in soundData){
		myFile.writeln('	<track id="' + i + '">');
		for(j in soundData[i]){
			// check the max value for the track, normalize, and write to XML file
			var  curMax = soundData[i]['max'];
			if(j!='max'){
				myFile.writeln('		<value time="' + j + '">' + soundData[i][j]/curMax + '</value>');
			}
		}
		myFile.writeln('	</track>');
	}
	myFile.writeln('</project>');
	myFile.close();
}

What follows is an example of the output XML structure:


	<track id="sax">
		0.03571428510845
		0.03571428510845
		0.01785714255423
		0
		0
		0
		0
		0
		0
		0
		0
		0
		0.0535714297831
		0.0535714297831
		0.03571428510845
	</track>
	<track id="drums">
		0.13809523619965
		0.12380952398185
		0.13333333212705
		0.07619047420872
		0.01428571447958
		0
		0
		0
		0
		0.13809523619965
		0.12380952398185
		0.10476189864432
		0.04761904524956
		0
		0
	</track>

And here’s the Python script to run from Maya, which creates a locator for each instrument and a bunch of nodes for the procedural maps to apply to the fur attributes:

steps = 25
stripFade = 0.2

import maya.cmds as cmd
from xml.etree import ElementTree as ET
 
tree = ET.parse('C:/path/to/syncaudio.xml')
project = tree.getroot()
timelineEnd = cmd.playbackOptions(q=1,max=1)

def setRampEntry(name,i,pos,r,g,b):
	cmd.setAttr((name+'.colorEntryList[%d].position')%i,pos)
	cmd.setAttr((name+'.colorEntryList[%d].colorR')%i,r)
	cmd.setAttr((name+'.colorEntryList[%d].colorG')%i,g)
	cmd.setAttr((name+'.colorEntryList[%d].colorB')%i,b)

counter = 0
for instrument in project:
	lastFrame = len(instrument) - 1
	if lastFrame>timelineEnd:
		timelineEnd = cmd.playbackOptions(e=1,max=lastFrame)
	name = instrument.attrib.get("id")
	print name	
	nullInst = cmd.spaceLocator(n=name)
	cmd.setAttr(name+".translateX",counter)
	lastValue = -1
	lastMod = False
	for value in instrument:
		frame = int(value.attrib.get("time"))
		index = float(value.text)
		if index!=lastValue:
			if lastMod==False:
				cmd.setKeyframe(name, at='translateY', v=lastValue, t=(frame-1))
			cmd.setKeyframe(name, at='translateY', v=index, t=frame)
			lastMod = True
		else:
			lastMod = False
		lastValue = index
	cmd.shadingNode('ramp',at=1,n=name+'Ramp')
	setRampEntry(name+'Ramp',0,0,1,1,1)
	for i in range(1,steps+1):
		cmd.setAttr((name + 'Ramp.colorEntryList[%d].position')%i,(float(i)/steps*0.9)+0.1)
		exprString = "float $t=time1.outTime-%d"%(i) + "; \r\nfloat $n=`getAttr -t $t " + name + ".translateY`; \r\nfloat $v=(1-$n)+($n*%f)"%(float(i)/steps*0.9) + "; \r\ncolorR=$v; \r\ncolorG=$v; \r\ncolorB=$v;"
		cmd.expression(o=(name+'Ramp.colorEntryList[%d]')%i, s=exprString)
	cmd.shadingNode('remapValue',au=1,n=name+'Remap')
	cmd.shadingNode('ramp',at=1,n=name+'Alpha')
	cmd.setAttr(name+"Alpha.type",1)
	cmd.connectAttr(name+"Alpha.outAlpha", name+"Remap.inputValue")
	cmd.connectAttr(name+"Ramp.outColor", name+"Remap.color[0].color_Color")
	stripLen = float(1)/len(project)
	if counter!=0:
		setRampEntry(name+'Alpha',0,(stripLen*counter)-(stripLen*stripFade*3),1,1,1)
	setRampEntry(name+'Alpha',1,(stripLen*counter)+(stripLen*stripFade),0,0,0)
	setRampEntry(name+'Alpha',2,(stripLen*(counter+1))-(stripLen*stripFade),0,0,0)
	if counter!=(len(project)-1):
		setRampEntry(name+'Alpha',3,(stripLen*(counter+1))+(stripLen*stripFade*3),1,1,1)
	counter += 1

Finally, here you are a few tips, regarding what I found and learnt by this experiment:

  • Maya’s Render Texture Range command, for exporting a group of nodes into a texture, has a limit to frame 1024 both for Start Frame and End Frame, at least with Maya 2009 version; the workaround to this, suggested by my nerdy friend Alan Stanzione, was to export the nodes via command line; here’s the command to run after selecting the desired node:
    // composite width height "file_name" file_extension destination start end step padding autoload
    composite 256 256 "syncaudio_map" iff images 0 2970 1 4 1;
  • when speaking with grooming TDs, 3D generalists and enthusiasts, the common opinion about Maya’s fur is pretty bad, and I make no exception; I had several issues to face, which are mostly related to the fact that the plugin was never really thought to be optimized in any kind of pipeline. What I would report here is a problem I had while baking animated attributes’ maps: no way to manage the resolution; changing the Map Width and Map Height attributes within the Fur Description just didn’t make any difference: Maya would always export maps with a resolution of 1024×1024. No idea which was the reason.
    In this case the only solution I found was the most brutal I could come out with, and yet it was much more effective than letting Maya bake the textures at the maximum res, for 7 different attributes, for a length of 2971 frames: I faked the baking process by opening XnView, batch renaming the files with the proper template, and copying them in the furAttrMap folder.
    I’m still waiting for somebody to tell me a wiser and cleaner way to deal with such a dumbish issue.
  • One thought on “syncing in the hair

    1. Pingback: Tweets that mention syncing in the hair | CGies -- Topsy.com

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>