Acko.net

Yak Shading

2015-09-27T00:00:00+02:00

Data-Driven Geometry

MathBox primitives need to take arbitrary data, transform it on the fly, and render it as styled geometry based on their attributes. Done as much as possible on the graphics hardware.

Three.js can render points, lines, triangles, but only with a few predetermined strategies. The alternative is to write your own vertex and fragment shader and do everything from scratch. Each new use case means a new ShaderMaterial with its own properties, so called uniforms. If the stock geometry doesn't suffice, you can make your own triangles by filling a raw BufferGeometry and assign custom per-vertex attributes. Essentially, to leverage GPU computation with Three.js—most engines, really—you have to ignore most of it.

Virtual Geometry

Shader computations are mainly rote transforms. For example, if you want to draw a line between two points, you'll have to make a long rectangle, made out of two triangles. But this simple idea gets complicated quickly once you add corner joins, depth scaling, 3D clipping, and so on. Doing this to an entire data set at once is what GPUs are made for, through vertex shaders which transform points.

Vertex shaders can only do 1-to-1 mappings. This isn't a problem by itself. You can use a gather approach to do N-to-1 mapping, where all the necessary data is pre-arranged into attribute arrays, with the data repeated and interleaved per vertex as necessary.

The proper tool for this is a geometry shader: a program that creates new geometry by N-to-M mapping of data, like making triangles out of points. WebGL doesn't support geometry shaders, won't any time soon, but you can emulate them with texture samplers. A texture image is just a big typed array, and you have random access unlike vertex attributes.

The original geometry acts only as a template, directing the shader's real data lookups. You lose some performance this way, but it's not too bad. Any procedural sampling pattern works, drawing 1 shape or 10,000. As textures can be rendered to, not just from, this also enables transform feedback, using the result of one pass to create new geometry in another.

All geometry rendered this way is 100% static as far as Three.js is concerned. New values are uploaded directly to GPU memory just before the rendering starts. The only gotcha is handling variable size input, because reallocation is costly. Pre-allocating a larger texture is easy, but clipping off the excess geometry in an O(1) fashion on the JS side is hard. In most cases there's the work around of dynamically generating degenerate triangles in a shader, which collapse down to invisible edges or points. This way, MathBox can accept variable sized arrays in multiple dimensions and will do its best to minimize disruption. If attribute instancing was more standard in WebGL, this wouldn't be such an issue, but as it stands, the workarounds are very necessary.

Vertex Party

If you squint very hard it looks a bit like React for live geometry. Except instead of a diffing algorithm, there's a few events, some texture uploads, a handful of draw calls and then an idle CPU. It's ideal for drawing thousands of things that look similar and follow the same rules. It can handle not just basic GL primitives like lines or triangles, but higher level shapes like 3D arrows or sprites.

My first prototype of this was my last christmas demo. It was messy and tedious to make, especially the shaders, but it performed excellently: the final scene renders ~200,000 triangles. Despite being a layer around Three.js … around WebGL … around OpenGL … around a driver … around a GPU … performance has far exceeded my expectations. Even complex scenes run great on my Android phone, easily 10x faster than MathBox 1, in some cases more like 1000x.

Of course compared to cutting edge DirectX or OpenCL (not a typo), this is still very limited. In today's GPUs, the charade of attributes, geometries, vertices and samples has mostly been stripped away. What remains is buffers and massive operations on them, exposed raw in new APIs like AMD's Mantle and iOS's Metal. My vertex trickery acts like a polyfill, virtualizing WebGL's capabilities to bring them closer to the present. It goes a bit beyond what geometry shaders can provide, but still lacks many useful things like atomic append queues or stream compaction.

For large geometries, the set up cost can be noticeable though. Shader compilation time also grows with transform complexity, doubly so on Windows where shaders are recompiled to HLSL / Direct3D. This makes drawing ops the heaviest MathBox primitives to spawn and reallocate. You could call this the MathBox version of the dreaded 'paint' of HTML. Once warmed up though, most other properties can be animated instantly, including the data being displayed: this is the opposite of how HTML works. Hence you can mostly spawn things ahead of time, revealing and hiding objects as needed, with minimal overhead and jank at runtime.

This all relies on carefully constructed shaders which have to be wired up in all their individual permutations. This needed to be solved programmatically, which is where we go last.

ShaderGraph 2

2015-09-27T00:00:00+02:00

Functional GLSL

For MathBox 1, I already needed to generate GL shaders programmatically. So I built ShaderGraph. You gave it snippets of GLSL code, each with a function inside. It would connect them for you, matching up the inputs and outputs. It supported directed graphs of calls with splits and joins, which were compiled down into a single shader. To help build up the graph progressively, it came with a simple chainable factory API.

It worked despite being several steps short of being a real compiler and having gaps in its functionality. It also committed the cardinal sin of regex code parsing, and hence accepted only a small subset of GLSL. All in all it was a bit of a happy mess, weaving vertex and fragment shaders together in a very ad-hoc fashion. Each snippet could only appear once in a shader, as it was still just a dumb code concatenator. I needed a proper way to compose shaders.

Select a node to view its code

Instanced Data Flow

Enter ShaderGraph 2. It's a total rewrite using Chris Dickinson's bona fide glsl-parser. It still parses snippets and connects them into a directed graph to be compiled. But a snippet is now a full GLSL program whose main() function can have open inputs and outputs. What's more, it now also links code in the proper sense of the word: linking up module entry points as callbacks.

Basically, snippets can now have inputs and outputs that are themselves functions. These connections don't obey the typical data flow of a directed graph and instead are for function calls. A callback connection provides a path along which calls are made and values are returned.

Snippets can be instanced multiple times, including their uniforms, attributes and varyings (if requested). Uniforms are bound to Three.js-style registers as you build the graph incrementally. So it's a module system, sort of, which enables functional shader building. Using callbacks as micro-interfaces feels very natural in practice, especially with bound parameters. You can decorate existing functions, e.g. turning a texture sampler into a convolution filter.

// Build shader graph
var shader = shadergraph.shader();
shader
  .callback()
    .pipe('sampleColor')
    .fan()
      .pipe('sepiaColor')
    .next()
      .pipe('invertColor')
    .join()
    .pipe('combineColors')
  .join()
  .pipe('convolveColor');

GLSL Composer

If you know GLSL, you can write ShaderGraph snippets: there is no extra syntax, you just add inputs and outputs to your main() function. You can use in/out/inout qualifiers or return a value. If there's no main function, the last defined function is exported.

vec3 callback(vec3 arg1, vec3 arg2);

To create a callback input in a snippet, you declare a function prototype in GLSL without a body. The function name and signature is used to create the outlet.

To create a callback output, you use the factory API. You can .require() a snippet directly, or bundle up a subgraph with .callback().….join(). In the latter case, the function signature includes all unconnected inputs and outputs inside. Outlets are auto-matched by name, type and order, with the semantics from v1 cleaned up.

Building basic pipes is easy: .pipe(…).pipe(…).…, passing in a snippet or factory. For forked graphs, you can .fan() (1-to-N) or .split() (N-to-N), use .next() to begin a new branch, and then .join() at the end. There's a few other operations, nothing crazy.

var v = shadergraph.shader();

// Graphs generated elsewhere
v.pipe(vertexColor(color, mask));
v.require(vertexPosition(position, material, map, 2, stpq));

v.pipe('line.position',    uniforms, defs);
v.pipe('project.position', uniforms);

By connecting pairs you create a functional data flow that compiles down to vanilla GLSL. It's not functional programming in GLSL, it just enables useful run-time assembly patterns, letting the snippets do the heavy lifting the old fashioned way.

As GPUs are massively parallel pure function applicators, the resulting mega-shaders are a great fit.

`$ cat *.glsl | magic`

The process still comes down to concatenating the code in a clever way, with global symbols namespaced to be unique. Function bodies are generated to call snippets in the right order, and the callbacks are linked. In the trivial case it links a callback by #defineing the two symbols to be the same. It can also impedance match compatible signatures like void main(in float, out vec2) and vec2 main(float) by inserting an intermediate call.

precision highp float;
precision highp int;
uniform mat4 modelMatrix;
uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;
uniform mat3 normalMatrix;
uniform vec3 cameraPosition;
#define _sn_191_getPosition _pg_103_
#define _sn_190_getPosition _pg_102_
#define _sn_189_getSample _pg_100_
#define _pg_99_ _sn_185_warpVertex
#define _pg_103_ _sn_190_getMeshPosition
#define _pg_100_ _sn_188_getTransitionSDFMask
#define _pg_101_ _sn_189_maskLevel
vec2 _sn_180_truncateVec(vec4 v) { return v.xy; }
uniform vec2 _sn_181_dataResolution;
uniform vec2 _sn_181_dataPointer;

vec2 _sn_181_map2DData(vec2 xy) {
  return fract((xy + _sn_181_dataPointer) * _sn_181_dataResolution);
}

uniform sampler2D _sn_182_dataTexture;

vec4 _sn_182_sample2D(vec2 uv) {
  return texture2D(_sn_182_dataTexture, uv);
}

vec4 _sn_183_swizzle(vec4 xyzw) {
  return vec4(xyzw.x, xyzw.w, 0.0, 0.0);
}
uniform float _sn_184_polarBend;
uniform float _sn_184_polarFocus;
uniform float _sn_184_polarAspect;
uniform float _sn_184_polarHelix;

uniform mat4 _sn_184_viewMatrix;

vec4 _sn_184_getPolarPosition(vec4 position, inout vec4 stpq) {
  if (_sn_184_polarBend > 0.0) {

    if (_sn_184_polarBend < 0.001) {
      
      
      
      
      vec2 pb = position.xy * _sn_184_polarBend;
      float ppbbx = pb.x * pb.x;
      return _sn_184_viewMatrix * vec4(
        position.x * (1.0 - _sn_184_polarBend + (pb.y * _sn_184_polarAspect)),
        position.y * (1.0 - .5 * ppbbx) - (.5 * ppbbx) * _sn_184_polarFocus / _sn_184_polarAspect,
        position.z + position.x * _sn_184_polarHelix * _sn_184_polarBend,
        1.0
      );
    }
    else {
      vec2 xy = position.xy * vec2(_sn_184_polarBend, _sn_184_polarAspect);
      float radius = _sn_184_polarFocus + xy.y;
      return _sn_184_viewMatrix * vec4(
        sin(xy.x) * radius,
        (cos(xy.x) * radius - _sn_184_polarFocus) / _sn_184_polarAspect,
        position.z + position.x * _sn_184_polarHelix * _sn_184_polarBend,
        1.0
      );
    }
  }
  else {
    return _sn_184_viewMatrix * vec4(position.xyz, 1.0);
  }
}
uniform float _sn_185_time;
uniform float _sn_185_intensity;

vec4 _sn_185_warpVertex(vec4 xyzw, inout vec4 stpq) {
  xyzw +=   0.2 * _sn_185_intensity * (sin(xyzw.yzwx * 1.91 + _sn_185_time + sin(xyzw.wxyz * 1.74 + _sn_185_time)));
  xyzw +=   0.1 * _sn_185_intensity * (sin(xyzw.yzwx * 4.03 + _sn_185_time + sin(xyzw.wxyz * 2.74 + _sn_185_time)));
  xyzw +=  0.05 * _sn_185_intensity * (sin(xyzw.yzwx * 8.39 + _sn_185_time + sin(xyzw.wxyz * 4.18 + _sn_185_time)));
  xyzw += 0.025 * _sn_185_intensity * (sin(xyzw.yzwx * 15.1 + _sn_185_time + sin(xyzw.wxyz * 9.18 + _sn_185_time)));

  return xyzw;
}



vec4 _sn_186_getViewPosition(vec4 position, inout vec4 stpq) {
  return (viewMatrix * vec4(position.xyz, 1.0));
}

vec3 _sn_187_getRootPosition(vec4 position, in vec4 stpq) {
  return position.xyz;
}
vec3 _pg_102_(vec4 _io_510_v, in vec4 _io_519_stpq) {
  vec2 _io_509_return;
  vec2 _io_511_return;
  vec4 _io_513_return;
  vec4 _io_515_return;
  vec4 _io_517_return;
  vec4 _io_520_stpq;
  vec4 _io_527_return;
  vec4 _io_528_stpq;
  vec4 _io_529_return;
  vec4 _io_532_stpq;
  vec3 _io_533_return;

  _io_509_return = _sn_180_truncateVec(_io_510_v);
  _io_511_return = _sn_181_map2DData(_io_509_return);
  _io_513_return = _sn_182_sample2D(_io_511_return);
  _io_515_return = _sn_183_swizzle(_io_513_return);
  _io_520_stpq = _io_519_stpq;
  _io_517_return = _sn_184_getPolarPosition(_io_515_return, _io_520_stpq);
  _io_528_stpq = _io_520_stpq;
  _io_527_return = _pg_99_(_io_517_return, _io_528_stpq);
  _io_532_stpq = _io_528_stpq;
  _io_529_return = _sn_186_getViewPosition(_io_527_return, _io_532_stpq);
  _io_533_return = _sn_187_getRootPosition(_io_529_return, _io_532_stpq);
  return _io_533_return;
}
uniform vec4 _sn_190_geometryResolution;

#ifdef POSITION_STPQ
varying vec4 vSTPQ;
#endif
#ifdef POSITION_U
varying float vU;
#endif
#ifdef POSITION_UV
varying vec2 vUV;
#endif
#ifdef POSITION_UVW
varying vec3 vUVW;
#endif
#ifdef POSITION_UVWO
varying vec4 vUVWO;
#endif


vec3 _sn_190_getMeshPosition(vec4 xyzw, float canonical) {
  vec4 stpq = xyzw * _sn_190_geometryResolution;
  vec3 xyz = _sn_190_getPosition(xyzw, stpq);

  #ifdef POSITION_MAP
  if (canonical > 0.5) {
    #ifdef POSITION_STPQ
    vSTPQ = stpq;
    #endif
    #ifdef POSITION_U
    vU = stpq.x;
    #endif
    #ifdef POSITION_UV
    vUV = stpq.xy;
    #endif
    #ifdef POSITION_UVW
    vUVW = stpq.xyz;
    #endif
    #ifdef POSITION_UVWO
    vUVWO = stpq;
    #endif
  }
  #endif
  return xyz;
}

uniform float _sn_188_transitionEnter;
uniform float _sn_188_transitionExit;
uniform vec4  _sn_188_transitionScale;
uniform vec4  _sn_188_transitionBias;
uniform float _sn_188_transitionSkew;
uniform float _sn_188_transitionActive;

float _sn_188_getTransitionSDFMask(vec4 stpq) {
  if (_sn_188_transitionActive < 0.5) return 1.0;

  float enter   = _sn_188_transitionEnter;
  float exit    = _sn_188_transitionExit;
  float skew    = _sn_188_transitionSkew;
  vec4  scale   = _sn_188_transitionScale;
  vec4  bias    = _sn_188_transitionBias;

  float factor  = 1.0 + skew;
  float offset  = dot(vec4(1.0), stpq * scale + bias);

  vec2 d = vec2(enter, exit) * factor + vec2(-offset, offset - skew);
  if (exit  == 1.0) return d.x;
  if (enter == 1.0) return d.y;
  return min(d.x, d.y);
}
uniform float _sn_191_worldUnit;
uniform float _sn_191_lineWidth;
uniform float _sn_191_lineDepth;
uniform float _sn_191_focusDepth;

uniform vec4 _sn_191_geometryClip;
attribute vec2 line;
attribute vec4 position4;

#ifdef LINE_PROXIMITY
uniform float _sn_191_lineProximity;
varying float vClipProximity;
#endif

#ifdef LINE_STROKE
varying float vClipStrokeWidth;
varying float vClipStrokeIndex;
varying vec3  vClipStrokeEven;
varying vec3  vClipStrokeOdd;
varying vec3  vClipStrokePosition;
#endif


#ifdef LINE_CLIP
uniform float _sn_191_clipRange;
uniform vec2  _sn_191_clipStyle;
uniform float _sn_191_clipSpace;

attribute vec2 strip;

varying vec2 vClipEnds;

void _sn_191_clipEnds(vec4 xyzw, vec3 center, vec3 pos) {

  
  vec4 xyzwE = vec4(strip.y, xyzw.yzw);
  vec3 end   = _sn_191_getPosition(xyzwE, 0.0);

  
  vec4 xyzwS = vec4(strip.x, xyzw.yzw);
  vec3 start = _sn_191_getPosition(xyzwS, 0.0);

  
  vec3 diff = end - start;
  float l = length(diff) * _sn_191_clipSpace;

  
  float arrowSize = 1.25 * _sn_191_clipRange * _sn_191_lineWidth * _sn_191_worldUnit;

  vClipEnds = vec2(1.0);

  if (_sn_191_clipStyle.y > 0.0) {
    
    float depth = _sn_191_focusDepth;
    if (_sn_191_lineDepth < 1.0) {
      float z = max(0.00001, -end.z);
      depth = mix(z, _sn_191_focusDepth, _sn_191_lineDepth);
    }
    
    
    float size = arrowSize * depth;

    
    
    float mini = clamp(1.0 - l / size * .333, 0.0, 1.0);
    float scale = 1.0 - mini * mini * mini; 
    float invrange = 1.0 / (size * scale);
  
    
    diff = normalize(end - center);
    float d = dot(end - pos, diff);
    vClipEnds.x = d * invrange - 1.0;
  }

  if (_sn_191_clipStyle.x > 0.0) {
    
    float depth = _sn_191_focusDepth;
    if (_sn_191_lineDepth < 1.0) {
      float z = max(0.00001, -start.z);
      depth = mix(z, _sn_191_focusDepth, _sn_191_lineDepth);
    }
    
    
    float size = arrowSize * depth;

    
    
    float mini = clamp(1.0 - l / size * .333, 0.0, 1.0);
    float scale = 1.0 - mini * mini * mini; 
    float invrange = 1.0 / (size * scale);
  
    
    diff = normalize(center - start);
    float d = dot(pos - start, diff);
    vClipEnds.y = d * invrange - 1.0;
  }


}
#endif

const float _sn_191_epsilon = 1e-5;
void _sn_191_fixCenter(vec3 left, inout vec3 center, vec3 right) {
  if (center.z >= 0.0) {
    if (left.z < 0.0) {
      float d = (center.z - _sn_191_epsilon) / (center.z - left.z);
      center = mix(center, left, d);
    }
    else if (right.z < 0.0) {
      float d = (center.z - _sn_191_epsilon) / (center.z - right.z);
      center = mix(center, right, d);
    }
  }
}


void _sn_191_getLineGeometry(vec4 xyzw, float edge, out vec3 left, out vec3 center, out vec3 right) {
  vec4 delta = vec4(1.0, 0.0, 0.0, 0.0);

  center =                 _sn_191_getPosition(xyzw, 1.0);
  left   = (edge > -0.5) ? _sn_191_getPosition(xyzw - delta, 0.0) : center;
  right  = (edge < 0.5)  ? _sn_191_getPosition(xyzw + delta, 0.0) : center;
}

vec3 _sn_191_getLineJoin(float edge, bool odd, vec3 left, vec3 center, vec3 right, float width) {
  vec2 join = vec2(1.0, 0.0);

  _sn_191_fixCenter(left, center, right);

  vec4 a = vec4(left.xy, right.xy);
  vec4 b = a / vec4(left.zz, right.zz);

  vec2 l = b.xy;
  vec2 r = b.zw;
  vec2 c = center.xy / center.z;

  vec4 d = vec4(l, c) - vec4(c, r);
  float l1 = dot(d.xy, d.xy);
  float l2 = dot(d.zw, d.zw);

  if (l1 + l2 > 0.0) {
    
    if (edge > 0.5 || l2 == 0.0) {
      vec2 nl = normalize(d.xy);
      vec2 tl = vec2(nl.y, -nl.x);

#ifdef LINE_PROXIMITY
      vClipProximity = 1.0;
#endif

#ifdef LINE_STROKE
      vClipStrokeEven = vClipStrokeOdd = normalize(left - center);
#endif
      join = tl;
    }
    else if (edge < -0.5 || l1 == 0.0) {
      vec2 nr = normalize(d.zw);
      vec2 tr = vec2(nr.y, -nr.x);

#ifdef LINE_PROXIMITY
      vClipProximity = 1.0;
#endif

#ifdef LINE_STROKE
      vClipStrokeEven = vClipStrokeOdd = normalize(center - right);
#endif
      join = tr;
    }
    else {
      
      float lmin2 = min(l1, l2) / (width * width);

      
#ifdef LINE_PROXIMITY
      float lr     = l1 / l2;
      float rl     = l2 / l1;
      float ratio  = max(lr, rl);
      float thresh = _sn_191_lineProximity + 1.0;
      vClipProximity = (ratio > thresh * thresh) ? 1.0 : 0.0;
#endif
      
      
      vec2 nl = normalize(d.xy);
      vec2 nr = normalize(d.zw);

      vec2 tl = vec2(nl.y, -nl.x);
      vec2 tr = vec2(nr.y, -nr.x);

#ifdef LINE_PROXIMITY
      
      vec2 tc = normalize(mix(tl, tr, l1/(l1+l2)));
#else
      
      vec2 tc = normalize(tl + tr);
#endif
    
      float cosA   = dot(nl, tc);
      float sinA   = max(0.1, abs(dot(tl, tc)));
      float factor = cosA / sinA;
      float scale  = sqrt(1.0 + min(lmin2, factor * factor));

#ifdef LINE_STROKE
      vec3 stroke1 = normalize(left - center);
      vec3 stroke2 = normalize(center - right);

      if (odd) {
        vClipStrokeEven = stroke1;
        vClipStrokeOdd  = stroke2;
      }
      else {
        vClipStrokeEven = stroke2;
        vClipStrokeOdd  = stroke1;
      }
#endif
      join = tc * scale;
    }
    return vec3(join, 0.0);
  }
  else {
    return vec3(0.0);
  }

}

vec3 _sn_191_getLinePosition() {
  vec3 left, center, right, join;

  float edge = line.x;
  float offset = line.y;

  vec4 p = min(_sn_191_geometryClip, position4);
  edge += max(0.0, position4.x - _sn_191_geometryClip.x);

  
  _sn_191_getLineGeometry(p, edge, left, center, right);

#ifdef LINE_STROKE
  
  vClipStrokePosition = center;
  vClipStrokeIndex = p.x;
  bool odd = mod(p.x, 2.0) >= 1.0;
#else
  bool odd = true;
#endif

  
  float width = _sn_191_lineWidth * 0.5;

  float depth = _sn_191_focusDepth;
  if (_sn_191_lineDepth < 1.0) {
    
    float z = max(0.00001, -center.z);
    depth = mix(z, _sn_191_focusDepth, _sn_191_lineDepth);
  }
  width *= depth;

  
  width *= _sn_191_worldUnit;

  join = _sn_191_getLineJoin(edge, odd, left, center, right, width);

#ifdef LINE_STROKE
  vClipStrokeWidth = width;
#endif
  
  vec3 pos = center + join * offset * width;

#ifdef LINE_CLIP
  _sn_191_clipEnds(p, center, pos);
#endif

  return pos;
}

uniform vec4 _sn_189_geometryResolution;
uniform vec4 _sn_189_geometryClip;
varying float vMask;


void _sn_189_maskLevel() {
  vec4 p = min(_sn_189_geometryClip, position4);
  vMask = _sn_189_getSample(p * _sn_189_geometryResolution);
}

uniform float _sn_192_styleZBias;
uniform float _sn_192_styleZIndex;

void _sn_192_setPosition(vec3 position) {
  vec4 pos = projectionMatrix * vec4(position, 1.0);

  
  float bias  = (1.0 - _sn_192_styleZBias / 32768.0);
  pos.z *= bias;
  
  
  if (_sn_192_styleZIndex > 0.0) {
    float z = pos.z / pos.w;
    pos.z = ((z + 1.0) / (_sn_192_styleZIndex + 1.0) - 1.0) * pos.w;
  }
  
  gl_Position = pos;
}
void main() {
  vec3 _io_546_return;

  _io_546_return = _sn_191_getLinePosition();
  _sn_192_setPosition(_io_546_return);
  _pg_101_();
}

It still does guarded regex manipulation of code too, but those manipulations are now derived from a proper syntax tree. GLSL doesn't have strings and its scope is simple, so this is unusually safe. I'm sure you can still trip it up somehow, but it's worth it for speed. I'm seeing assembly times of ~10-30ms cold, 2-4ms warm, but it depends entirely on the particular shaders.

The assembly process is now properly recursive. Unassembled shaders can be used in factory form, standing in for snippets. Completed graphs form stand-alone programs with no open inputs or outputs. The result can be turned straight into a Three.js ShaderMaterial, but there is no strict Three dependency. It's just a dictionary with code and a list of uniforms, attributes and varyings. Unlike before, building a combined vertex/fragment program is now merely syntactic sugar for a pair of separate graphs.

As it's run-time, you can slot in user-defined or code-generated GLSL just the same. Shaders are fetched by name or passed as inline code, mixed freely as needed. You supply the dictionary or lookup method. You could bundle your GLSL into JS with a build step or include embedded



PowerPoint Must Die








  "I think a lot of mathematics is really about how you understand things in your head. It's people that did mathematics, we're not just general purpose machines, we're people. We see things, we feel things, we think of things. A lot of what I have done in my mathematical career has had to do with finding new ways to build models, to see things, do computations. Really get a feel for stuff.


It may seem unimportant, but when I started out people drew pictures of 3-manifolds one way and I started drawing them a different way. People drew pictures of surfaces one way and I started drawing them a different way. There's something significant about how the representation in your head profoundly changes how you think.


It's very hard to do a brain dump. Very hard to do that. But I'm still going to try to do something to give a feel for 3-manifolds. Words are one thing, we can talk about geometric structures. There are many precise mathematical words that could be used, but they don't automatically convey a feeling for it. I probably can't convey a feeling for it either, but I want to try."
– William Thurston, The Mystery of 3-Manifolds (Video)






How do you convince web developers—heck, people in general—to care about math? This was the challenge underlying Making Things With Maths, a talk I gave three years ago. I didn't know either, I just knew why I liked this stuff: demoscene, games, simulation, physics, VR, … It had little to do with what passed for mathematics in my own engineering education. There we were served only eyesore PowerPoints or handwritten overhead transparencies, with simplified graphs, abstract flowcharts and rote formulas, available on black and white photocopies.

Smart people who were supposed to teach us about technology seemed unable to teach us with technology. Fixing this felt like a huge challenge where I'd have to start from scratch. This is why the focus was entirely on showing rather than telling, and why MathBox 1 was born. It's how this stuff looks and feels in my head, and how I got my degree: by translating formulas into mental pictures, which I could replay and reason about on demand.

PowerPoint Syndrome

Initially I used MathBox like an embedded image or video: compact diagrams, each a point or two in a presentation. My style quickly shifted though. I kept on finding ways to transform from one visualization to another. Not for show, but to reveal the similarities and relationships underneath. MathBox encouraged me to animate things correctly, leveraging the actual models themselves, instead of doing a visual morph from A to B. Each animation became a continuous stream of valid examples, a quality both captivating and revealing.






    




For instance, How to Fold a Julia Fractal is filled with animations of complex exponentials, right from the get go. This way I avoid the scare that ($ e^{i\pi} $) is a meaningful expression; symbology and tau-tology never have a chance to obscure geometrical workings. Instead a web page that casually demonstrates conformal mapping and complex differential equations got 340,000 visits. Despite spotty web browser support and excluding all mobile phones for years.




  
  Meanwhile academics voluntarily published their writings behind a $42 per PDF paywall, the colossal idiots.




The next talk, Making WebGL Dance, contained elaborate long takes worthy of an Alfonso Cuarón film, with only 3 separate shots for the bulk of a 30 minute talk. The lesson seemed obvious: the slides shouldn't have graphics in them, rather, the graphics should have slides in them. The diagnosis of PowerPoint syndrome is then the constant trashing of context from one slide to the next. A traditional blackboard doesn't have this problem: you build up diagrams slowly, by hand, across a large surface, erasing selectively and only when you run out of space.







It's not just about permanence and progression though, it's also about leveraging our natural understanding of shape, scale, color and motion. Think of how a toddler learns to interact with the world: poke, grab, chew, spit, smash. Which evolves into run, jump, fall, get back up again. Humans are naturals at taking multiple cases of "If I do this, that will happen" and turning it into a consistent, functional model of how things work. We learn language by bootstrapping random jibberish into situational meaning, converging on a shared protocol.

That said, I find the usual descriptions of how people experience language and thought foreign. Instead, when Temple Grandin speaks about visual thinking, I nod vigorously. Thought to me is analog concepts and sensory memories, remixed with visual and other simulations. It builds off the quantities and qualities present in spatial and temporal notions, which appear built-in to us.





Speech and writing is then a program designed to reconstruct particular thoughts in a compatible brain. There are a multitude of evolving languages, they can be used elegantly, bluntly, incomprehensibly, but the desired output remains the same. In my talks, armed with weapons-grade C2-continuous animations, it is easy to transcode my film reel into words, because the slides run themselves. The string of concepts already hangs in the air, I only add the missing grammar that links them up. This is a puzzle our brains are so good at solving, we usually do it without thinking.

Language is the ability of thoughts to compute their own source code.

(It's not proof, I just supply pudding.)




  
  Tip: Powerpoint remotes are 4-key USB keyboards with PageUp/PageDown, F5 and . keys.

Comes with dongle.





  


  



  I sketch rough thumbnails, then start animating until I hit a dead end. Then start another one. Titles and overlays always come last.




Manifold Dreams

I don't say all this to up my Rain Man cred, but to lay to rest the recurring question of where my work comes from. I translate the pictures in my head to HD, in order to learn from and refine the view. As I did with quaternions: I struggled to grok the hypersphere, it wouldn't fit together right. So I wrote the code to trace out geodesics in color and fly around in it, and suddenly the twisting made sense. Hence my entire tutorial was built to replicate the same discovery process I went through myself.







  
    

    For visualizing the 4D hypersphere, quaternions are a natural fit.
They reveal their underlying cyclic symmetry under 4D stereographic projection.
  





There was one big problem: scenes now consisted of diagrams of diagrams, which meant working around MathBox more than with it. Performance issues arose as complexity grew. Above all there was a total lack of composability in the components. None of this could be fixed without ripping out significant pieces, so doing it incrementally seemed futile. I started from scratch and set off to reinvent all the wheels.

$$ \text{MathBox}^2 = \int_1^2 \text{code}(v) dv $$

MathBox 2 was inevitably going to suffer second-system syndrome, parts would be overengineered. Rather than fight it, I embraced it and effectively wrote a strange vector GPU driver in CoffeeScript. (Such is life, this is a blueprint meant to be simplified and made obsolete over time, not expanded upon.) It's a freight train straight to the heart of a graphics card, combining low-level and high-level in a way that feels novel 🐴 when you use it, squeezing 🐴 through a very small opening.

What was tedious before, now falls out naturally. If I format the scene above as XML/JSX, it becomes:




  


  
  
  
  
    
    
      
      
      
      
        
        
        
        

        
        
        
        

        
        
        
        
      
    
  







  
In order to make these pieces behave, a bunch of additional attributes are applied, most of which are strings or values, some of which are functions/code, either JavaScript or GLSL:




  

id="1" scale={300}>
  id="2" proxy={true} position={[0, 0, 3]} />
  id="3" speed={1/4}>
    id="4" bend={1}>
      id="5" code="
uniform float cos1;
uniform float sin1;
uniform float cos2;
uniform float sin2;
uniform float cos3;
uniform float sin3;
uniform float cos4;
uniform float sin4;

vec4 getRotate4D(vec4 xyzw, inout vec4 stpq) {
  xyzw.xy = xyzw.xy * mat2(cos1, sin1, -sin1, cos1);
  xyzw.zw = xyzw.zw * mat2(cos2, sin2, -sin2, cos2);
  xyzw.xz = xyzw.xz * mat2(cos3, sin3, -sin3, cos3);
  xyzw.yw = xyzw.yw * mat2(cos4, sin4, -sin4, cos4);

  return xyzw;
}"
 cos1=>{(t) => Math.cos(t * .111)} sin1=>{(t) => Math.sin(t * .111)} cos2=>{(t) => Math.cos(t * .151 + 1)} sin2=>{(t) => Math.sin(t * .151 + 1)} cos3=>{(t) => Math.cos(t * .071 + Math.sin(t * .081))} sin3=>{(t) => Math.sin(t * .071 + Math.sin(t * .081))} cos4=>{(t) => Math.cos(t * .053 + Math.sin(t * .066) + 1)} sin4=>{(t) => Math.sin(t * .053 + Math.sin(t * .066) + 1)} />
      id="6">
        id="7" rangeX={[-π/2, π/2]} rangeY={[0, τ]} width={129} height={65} expr={(emit, θ, ϕ, i, j) => {
        q1.set(0, 0, Math.sin(θ), Math.cos(θ));
        q2.set(0, Math.sin(ϕ), 0, Math.cos(ϕ));
        q1.multiply(q2);
        emit(q1.x, q1.y, q1.z, q1.w);
      }} live={false} channels={4} />
        id="8" color="#3090FF" />
        id="9" rangeX={[-π/2, π/2]} rangeY={[0, τ]} width={129} height={65} expr={(emit, θ, ϕ, i, j) => {
        q1.set(0, Math.sin(θ), 0, Math.cos(θ));
        q2.set(Math.sin(ϕ), 0, 0, Math.cos(ϕ));
        q1.multiply(q2);
        emit(q1.x, q1.y, q1.z, q1.w);
      }} live={false} channels={4} />
        id="10" color="#20A000" />
        id="11" rangeX={[-π/2, π/2]} rangeY={[0, τ]} width={129} height={65} expr={(emit, θ, ϕ, i, j) => {
        q1.set(Math.sin(θ), 0, 0, Math.cos(θ));
        q2.set(0, 0, Math.sin(ϕ), Math.cos(ϕ));
        q1.multiply(q2);
        emit(q1.x, q1.y, q1.z, q1.w);
      }} live={false} channels={4} />
        id="12" color="#DF2000" />
      >
    >
  >
>





Phew. That's how you make a 4D diagram with Hopf fibration as far as the eye can see. Except it's not actually JSX, that's just me and my pretty-printer pretending.

Geometry Streaming

The key is the data itself. It's an array of points mostly, but how that data is laid out and interpreted determines how useful it can be.

Most basic primitives come in fixed size chunks. Particles are single points, lines have two points, triangles have three points. Polygons and polylines have N points. So it made sense to have a tuple of N points be the basic logical unit. You can think in logical pieces of geometry, rather than raw points or individual triangles, unlike GL.

Each primitive maps over data in a standard way. Feed an array of points to a line, you get a polyline. Feed a matrix of points to a surface and you get a grid mesh. Simple. But feed a voxel to a vector, and you get a 3D vector field. The general idea is that drawing 1 of something should be as easy as drawing 100×100×100.





  
    
  







This is particularly useful for custom data expressions, which stream in live or procedural data. They now receive an emit(x, y, z, w) function, for emitting a 4-vector like XYZW or RGBA. This is little more than an inlineable call to fill a floatArray[i++] = x, quite a lot faster than returning an array or object.







  
    
  

mathbox
  .interval({
    expr: function (emit, x, i, t) {
      y = Math.sin(x + t);
      emit(x,  y);
      emit(x, -y);
    },
    width:   64,
    items:    2,
    channels: 2,
  })
  .vector({
    color: 0x3090FF,
    width: 3,
    start: true,
  });




Emitting 64 2D vectors on an interval, 2 points each.

More importantly it lets you emit N points in one iteration, which makes the JS expressions themselves feel like geometry shaders. The result feeds into one or more styled drawing ops. The number of emit calls has to be constant, but you can always knock out or mask the excess geometry.





  emit = switch channels
  when 1 then (x) ->
    array[i++] = x
    ++j
    return

  when 2 then (x, y) ->
    array[i++] = x
    array[i++] = y
    ++j
    return

  when 3 then (x, y, z) ->
    array[i++] = x
    array[i++] = y
    array[i++] = z
    ++j
    return

  when 4 then (x, y, z, w) ->
    array[i++] = x
    array[i++] = y
    array[i++] = z
    array[i++] = w
    ++j
    return

  
  
  Both the expression and emitter will be inlined into the stream's iteration loop.





  consume = switch channels
  when 1 then (emit) ->
    emit array[i++]
    ++j
    return

  when 2 then (emit) ->
    emit array[i++], array[i++]
    ++j
    return

  when 3 then (emit) ->
    emit array[i++], array[i++], array[i++]
    ++j
    return

  when 4 then (emit) ->
    emit array[i++], array[i++], array[i++], array[i++]
    ++j
    return

  
    
  Closures of Hanoi





(4-in-1)²

GPUs can operate on 4×1 vectors and 4×4 matrices, so working with 4D values is natural. Values can also be referenced by 4D indices. With one dimension reserved for the tuples, that leaves us 3 dimensions XYZ. Hence MathBox arrays are 3+1D. This is for width, height, depth, while the tuple dimension is called items. It does what it says on the tin, creating 1D W, 2D W×H and 3D W×H×D arrays of tuples. Each tuple is made of N vectors of up to 4 channels each.

Thanks to cyclic buffers and partial updates, history also comes baked in. You can use a spare dimension as a free time axis, retaining samples on the go. You can .set('history', N) to record a short log of a whole array over time, indefinitely.

All of this is modular: a data source is something that can be sampled by a 4D pointer from GLSL. Underneath, arrays end up packed into a regular 2D float texture, with "items × width" horizontally and "height × depth" vertically. Each 'pixel' holds a 1/2/3/4D point.

Mapping a 4D 'pointer' to the real 2D UV coordinates is just arithmetic, and so are operators like transpose and repeat. You just swap the XY indices and tell everyone downstream that it's now this big instead. They can't tell the difference.

You can create giant procedural arrays this way, including across rectangular texture size limits, as none of them actually exist except as transient values deep inside a GPU core. Until you materialize them by rendering to a texture using the memo primitive. Add in operators like interpolation and convolution and it's a pretty neat real-time finishing kit for data.





  
    
  













Continued in Part 2.



MathBox²
2015-09-27T00:00:00+02:00








Part 2







Continued from Part 1.

I-Can't-Believe-It's-Not-React

Underneath sits a large codebase driving it, 200+ files in JS/CS alone, not including dependencies that aren't my own. Much of it is infrastructure necessary to pull off certain tricks consistently: you can draw 2.5D lines with grace, render arbitrary Unicode text in GL, sync HTML to GPU-computed geometry, and do all this with GLSL code composed on the fly, including your own. Nobody needs all of this.





  
    
  





    
    LaTeX HTML with KaTeX.
    Plain HTML DOM.
    Virtual HTML with DOM diff.
    Live Signed Distance Fields for GL.
    
  




These wildly different strategies are actually all abstracted into the DOM path or the GL path, with a quacks-like-React component as the glue in the DOM path.






  
  
  
  
      
      
      
      
      
      
      
      
      
      
      
      

      ...      

      
      
      
      
      
      
      
      
      
      
      
      
      
      

    
  






// Define VDOM handler to clone real DOM elements
var clone = MathBox.DOM.createClass({
  render: function (el, props, children) {
    var element = children.cloneNode(true);
    return element;
  },
});

// Define VDOM handler to format 'latex' into an HTML span
var latex = MathBox.DOM.createClass({
  render: function (el) {
    this.props.innerHTML = katex.renderToString(this.children);
    return el('span', this.props);
  }
});



The interface with the HTML DOM.

Appearances deceive however,
as MathBox's own DOM is an entirely different beast.





Most of the code is for initialization only, building up a reactive machine by combining components. Once assembled it lets the GPU do most of the crunching, while relying on the JS VM to inline and optimize the chewy outside.

At the top level, MathBox is plain old JavaScript, used like this:




  /* Easy Mode */

// Bootstrap MathBox and Three.js
var mathbox = mathBox();
if (mathbox.fallback) { throw Error("No WebGL support.") };

// Make MathBox primitives
var view =
  mathbox
  .set({
    scale: null,
  })
  .camera({
    proxy: true,
    position: [0, 0, 3]
  })
  .polar({
    range: [[-2, 2], [-1, 1], [-1, 1]],
    scale: [2, 1, 1],
    bend: .25
  });

view.interval({
  width: 48,
  expr: function (emit, x, i, t) {
    // Emit sine wave
    y = Math.sin(x + t / 4) * .5 + .75;
    emit(x, y);
  },
  channels: 2,
})
.line({
  color: 0x30C0FF,
  width: 16,
})
.resample({
  width: 8,
})
.point({
  color: 0x30C0FF,
  size: 60,
})
.html({
  width:  8,
  expr: function (emit, el, i, j, k, l, t) {
    // Emit random latex
    var color = ['#30D0FF','#30A0FF'][i%2];
    var a = Math.round(t + i) % symbols.length;
    var b = Math.round(Math.sqrt(t * t + Math.sin(t + i * i) + 5));
    emit(el(latex, {style: {color: color}},
      '\\sqrt{\\text{LaTeX} + '+(i + b)+' \\pi^{'+symbols[a]+'}}'));
  },
})
.dom({
  snap: false,
  offset: [0, 32],
  depth: 0,
  zoom: 1,
  outline: 2,
  size: 20,
});

// ...






  
    

    CSS 3D rims included.
  





  
    

    The default fallback message.
  




  
The WebGL Canvas bootstrapper is a separate piece though, it's wrapping Threestrap, the little non-framework that could. It lets you spawn a fully functioning GL canvas in one exceedingly configurable call. It takes care of browser support, resizing with retina, CSS alignment, warm-up and more.

If you prefer instead, you can spawn a bare MathBox context in a Three.js scene of your choosing, but you'll have to babysit it:




  /* Simple Mode */

// Vanilla Three.js
var renderer = new THREE.WebGLRenderer();
var scene = new THREE.Scene();
var camera = new THREE.PerspectiveCamera(60, WIDTH / HEIGHT, .01, 1000);

// Insert into document
document.body.appendChild(renderer.domElement);

// MathBox context
var context = new MathBox.Context(renderer, scene, camera).init();
var mathbox = context.api;

// Set size
renderer.setSize(WIDTH, HEIGHT);
context.resize({ viewWidth: WIDTH, viewHeight: HEIGHT });

// Place camera and set background
camera.position.set(0, 0, 3);
renderer.setClearColor(new THREE.Color(0xFFFFFF), 1.0);

// MathBox elements
view = mathbox
.set({
  focus: 3,
})
.cartesian({
  range: [[-2, 2], [-1, 1], [-1, 1]],
  scale: [2, 1, 1],
});
// ...

// Render frames
var frame = function () {
  requestAnimationFrame(frame);
  context.frame();
  renderer.render(scene, camera);
};
requestAnimationFrame(frame);




Despite looking like a monolith, it really isn't, it was merely a matter of convenience and sanity to not decouple it more until its shape had stabilized. Minimal builds are, for now, left as an exercise to the reader. I've split up the thinking and design into several articles, mirroring the architecture. However, you don't need to know all this to use MathBox 2, they are for people who want to know the how and why... the document model, the geometry core and the shader assembly.

In putting it all together, the devil's in the details of course. Depending on your imagination, it's either much more or much less powerful than you want. There's still far too much to cover: slideshows, keyframe tracks, fov-calibrated units, z-indexes, atlas retexting, … Most of this is unsurprising in that it all works. You can define a keyframe interpolation between two value or emitter expressions, and watch the smoothly lerped data go. Animation tracks are tied to triggers like clocks and slides, which lets them fit naturally in presentations.








  
  index={1}>
    
    steps={2}>
      
      
        
        
        
        
        
        script={[
            {props: {color: 'red'}},
            {props: {color: 'blue'}},
          ]} >
      
    
    
    
      
      
        
        
        
        script={[
                       {props: {expr: (emit, x) => emit(x, Math.sin(x))}},
                       {props: {expr: (emit, x) => emit(x, Math.tan(x))}},
                      ]} />
        
        

        
        
        
      
    
    
    
      
      
        
        
        
        
      
    
  






One More Thing…



Images are data. So is audio. That means MathBox 2 is Winamp AVS, Milkdrop and the mythical Fridge all rolled into one. You can replicate your everyday trippy music visualizer with two operators: render-to-texture (rtt) and compose. It acts as an embedded scene, rendering all of its children to an off-screen image, while Compose renders a full-screen pass. This is where the model's expressiveness shines.

Milkdrop equals mathbox.rtt(…).compose(…).….end().compose(…), that is, an image feeding back into itself, but also rendered to the screen. The necessary double buffering and swaps are abstracted away. Drop in shapes and shaders, add transforms, nest as you like. RTTs have a history parameter like arrays, so Turing patterns, self-propagating hypno spirals, and other cool partial diffy eqs are a shader away.







  
    
  






  

id="1" scale={720}>
  id="render" minFilter="nearest" magFilter="nearest" type="unsignedByte">
    id="3" lookAt={[0, 0, 0]} position=>{(t) => { return [Math.cos(t) * 3, 0, Math.sin(t) * 3] }} />
    id="4" range={[[-2, 2], [-1, 1], [-1, 1]]} scale={[2, 1, 1]}>
      id="5" scale={[7/10, 7/10, 7/10]}>
        id="6" width={5} divideX={2} divideY={2} zBias={10} opacity={1/4} color={16768992} />
      >
    >
  >
  id="rtt1" history={4} type="unsignedByte" minFilter="linear" magFilter="linear">
    id="8" code="
uniform vec3 dataResolution;
uniform vec3 dataSize;
uniform float cosine;
uniform float sine;
vec4 getSample(vec3 xyz);
vec4 getFramesSample(vec3 xyz) {
  vec2 pos = xyz.xy * dataResolution.xy - .5;
  pos = ((pos * dataSize.xy) * mat2(cosine, sine, -sine, cosine) * .999) / dataSize.xy;
  xyz.xy = (pos + .5) * dataSize.xy;
  vec4 c = getSample(xyz + vec3( 0.0, 0.0, 1.0));
  vec3 t = getSample(xyz + vec3( 0.0, 1.5, 0.0)).xyz;
  vec3 b = getSample(xyz + vec3( 0.0,-1.5, 0.0)).xyz;
  vec3 l = getSample(xyz + vec3(-1.5, 0.0, 0.0)).xyz;
  vec3 r = getSample(xyz + vec3( 1.5, 0.0, 0.0)).xyz;
  return vec4((t + b + l + r) / 2.0 - c.xyz, c.w);
}"
 cosine=>{(t) => Math.cos(Math.sin(t * .2) * .005)} sine=>{(t) => Math.sin(Math.sin(t * .2) * .005)} />
    id="resample1" indices={3} channels={4} />
    id="10" />
    id="11" source="#render" blending="add" />
  >
  id="12" minFilter="linear" magFilter="linear" type="unsignedByte">
    id="colormap" code="
uniform float modulate1;
uniform float modulate2;
uniform float modulate3;
uniform float modulate4;
vec4 getSample(vec3 xyz);
vec4 getFramesSample(vec3 xyz) {
  vec4 color = (
    getSample(xyz) +
    getSample(xyz + vec3(0.0, 0.0, 1.0)) +
    getSample(xyz + vec3(0.0, 0.0, 2.0)) +
    getSample(xyz + vec3(0.0, 0.0, 3.0))
  ) / 4.0;
  color = color * color * color * 1.15;
  float v = color.x + color.y + color.z;
  vec3 c = vec3(v*v + color.x * .2, v*v, v*v*v + color.z) * .333;
  c = mix(c, mix(sqrt(c.yzx * c), c.zxy, modulate1), modulate2);
  c = mix(c, mix(c.yzx, c.zxy, modulate1), modulate2);
  c = mix(c, mix(abs(sin(c.yxz * 2.0)), c.zyx, modulate3), modulate4);
  return vec4(c, 1.0);
}"
 modulate1=>{(t) => Math.cos(t * .417) * .5 + .5} modulate2=>{(t) => Math.cos(t * .617 + Math.sin(t * .133)) * .5 + .5} modulate3=>{(t) => Math.cos(t * .217 + 2.0) * .5 + .5} modulate4=>{(t) => Math.cos(t * .117 + 3.0 + Math.sin(t * .133)) * .5 + .5} />
    id="resample2" source="#rtt1" indices={3} channels={4} />
    id="15" />
  >
  id="16" code="
vec4 getSample(vec2 xy);
vec4 getFramesSample(vec2 xy) {
  return getSample(xy + vec2(0.5, 0.5));
}"
 />
  id="resample3" indices={2} channels={4} />
  id="18" source="#resample2" />
>







The difference compared to AVS is that the .rtt() is inert to its container by default. Until you add on a .compose() pass, it's just a dangling data source. Meanwhile the .compose() op offers the necessary  GL blend modes, opacity and color tints through style properties. Document order defines drawing order, so the decomposition into render passes is direct. On top, zOrder can be overridden (drawing order), as can zIndex (2D stacking order) and zBias (3D stacking order).

You can nest effects and compose shaders to create recursive visualizations, sampling from themselves or each other:







  
    
  
  
  
  
  Open in New Window
(Burn that fillrate, baby.)
  
  Bonus: Endless Visualizer.





  
id="1" scale={720}>
  id="2" proxy={true} position={[3/10, 1/10, 2]} />
  id="3">
    id="audioTime" data={[]} width={1024} channels={1} />
    id="audioFreq" data={[]} width={512} channels={1} />
  >
  id="render" width={256} height={144} type="unsignedByte" minFilter="nearest" magFilter="nearest">
    id="7" position={[0, 0, 5/2]} />
    id="8">
      id="9" source="#audioTime" order="yx" />
      id="10" width={[861/250, 0, 0, 0]} />
      id="11" code="
vec4 getSample(vec4 xyzw);
vec4 getColor(vec4 xyzw) {
  float h = getSample(xyzw).y;
  return vec4(vec3(h), 1.0);
}"
 />
      id="12" />
      id="13" scale={[1, 3/4, 1]}>
        id="14" points="<<" colors="<" width={5} color={16777215} opacity={2/5} blending="add" />
      >
    >
    id="15" range={[[-2, 2], [-1, 1], [-1, 1]]} scale={[1/2, 1/4, 1/4]} quaternion=>{(t) => {
          c = Math.cos(t / 3);
          s = Math.sin(t / 3);
          c2 = Math.cos(t / 8.71);
          s2 = Math.sin(t / 8.71);
          return [s * s2, s * c2, .2, c];
        }}>
      id="16" divideX={4} divideY={4} zBias={10} opacity={1/10} color={16768992} width={6} />
    >
  >
  id="rtt1" history={4} width={256} height={144} type="unsignedByte">
    id="18" code="#map-rotate" />
    id="resample1" indices={3} channels={4} />
    id="20" color="#ffffff" zWrite={false} />
    id="21" source="#render" blending="add" color="#ffffff" zWrite={false} />
  >
  id="rtt2" width={256} height={144} type="float">
    id="23" position={[0, 0, 5/2]} />
    id="24" seek=>{(t) => audio ? audio.currentTime : t}>
      id="25" code="#map-temporal-blur" time=>{(t) => t * 16.0} modulate=>{(t) => {
            var bang = ((t > 69.229311)  && (t < 88.922656)) ||
                       ((t > 88.922656)  && (t < 148.9143)) ||
                       ((t > 148.9143)   && (t < 158.2)) ||
                       ((t > 168.284427) && (t < 188.00)) ? 1 : 0;
            if ((t > 88.922656)  && (t < 148.9143)) {
              bang *= .5 + .45 * Math.cos(t / 3);
            }
            if ((t > 168)) {
              bang *= .85 + .15 * Math.cos(t);
            }
            modulate = modulate + (bang - modulate) * .1;
            return modulate;
          }} pattern=>{(t) => {
            var bang = ((t > 88.922656) && (t < 148.9143));
            pattern = pattern + (bang - pattern) * .1;
            if ((t > 168)) {
              pattern = .5 + .4 * Math.cos(t * 2.311);
            }
            return pattern;
          }} warp=>{(t) => {
            var bang = (t > 148.9143);
            if ((t > 168)) {
              warp *= 1 + .5 * Math.cos(t * .556);
            }
            if ((t > 148.2) && (t < 158.2)) warp = warp + .75 + .25 * Math.cos((t - 158.2));
            warp = warp + (bang - warp) * .1;
            return warp;
          }} shift=>{(t) => {
            var bang = (t > 168) ? Math.max(0, Math.min(1, .1 * (t - 168))) : 0;
            bang *= .75 + .25 * Math.cos(t * .731);
            warp = warp + (bang - warp) * .1;
            return warp;
          }} />
      id="resample2" source="#rtt1" indices={3} channels={4} />
      id="27" color="#fff" zWrite={false} />
    >
    id="28" scale={[1, 1/4, 1]}>
      id="29" source="#audioTime" order="yx" />
      id="30" width={[861/250, 0, 0, 0]} />
      id="31" code="
vec4 getSample(vec4 xyzw);
vec4 getColor(vec4 xyzw) {
  float h = getSample(xyzw).y;
  return vec4(vec3(h) * .2, 1.0);
}"
 />
      id="32" />
      id="33" points="<<" colors="<" width={50} color={16777215} opacity={1} blending="add" />
    >
  >
  id="34" width={129} height={73} />
  id="lerp" depth={2} />
  id="36" code="#map-xy-to-xyz" />
  id="37" indices={3} channels={3} />
  id="transpose" order="xywz" />
  id="color" source="#lerp" order="xywz" />
  id="40" seek=>{(t) => audio ? audio.currentTime : t}>
    id="disco" speed=>{(t) => {
        var bang = ((t > 69.329311)  && (t < 89.122656)) ||
                   ((t > 148.9143)   && (t < 158.0)) ||
                   ((t > 168.284427) && (t < 188.077772));
        return bang ? 1 : .2;
      }}>
      id="42" code="#map-z-to-color" modulate1=>{(t) => Math.cos((t + 1) * .417) * .5 + .5} modulate2=>{(t) => Math.cos((t + 1) * .617 + Math.sin(t * .133)) * .5 + .5} modulate3=>{(t) => Math.cos((t + 1) * .217 + 2.0) * .5 + .5} modulate4=>{(t) => Math.cos((t + 1) * .117 + 3.0 + Math.sin(t * .133)) * .5 + .5} />
      id="color1" source="#lerp" indices={2} channels={4} />
      id="44" code="#map-z-to-color-2" modulate1=>{(t) => Math.cos((t + 1) * .417) * .5 + .5} modulate2=>{(t) => Math.cos((t + 1) * .617 + Math.sin(t * .133)) * .5 + .5} modulate3=>{(t) => Math.cos((t + 1) * .217 + 2.0) * .5 + .5} modulate4=>{(t) => Math.cos((t + 1) * .117 + 3.0 + Math.sin(t * .133)) * .5 + .5} />
      id="color2" source="#lerp" indices={2} channels={4} />
    >
    id="46" range={[[-1.7788, 1.7788], [-1, 1], [-1, 1]]} scale={[16/9, 1, 1]} quaternion=>{(t) => {
        t = t / 3;
        c = Math.cos(t / 4);
        s = Math.sin(t / 4);
        c2 = Math.cos(t / 11.71) * 1.71;
        s2 = Math.sin(t / 11.71) * 1.71;
        return [s * s2, s * c2, -.2, c];
      }}>
      id="47" source="#transpose" width={33} height={19} />
      id="48" source="#color2" width={33} height={19} />
      id="49">
        id="50" points="<<" colors="<" color="#ffffff" width={2} zBias={5} />
      >
      id="51" script={{19: {position: [0, 0, 0]}, 39: {position: [0, 0, 2]}, 57: {position: [0, 0, 0]}}} />
      id="52" source="<<" order="yxzw" />
      id="53" source="<<" order="yxzw" />
      id="54">
        id="55" points="<<" colors="<" color="#ffffff" width={2} zBias={5} />
      >
      id="56" script={{19: {position: [0, 0, 0]}, 39: {position: [0, 0, -2]}, 57: {position: [0, 0, 0]}}} />
      id="57">
        id="58" points="<<" colors="<" color="#ffffff" size={10} zBias={5} zOrder={1} blending="add" zWrite={false} />
      >
      id="59" script={{19: {position: [0, 0, 0]}, 39: {position: [0, 0, -1]}, 57: {position: [0, 0, 0]}}} />
      id="60">
        id="61" points="#transpose" colors="#color2" color="#ffffff" size={5} zBias={5} zOrder={1} blending="add" zWrite={false} />
      >
      id="62" script={{9: {position: [0, 0, 0]}, 39: {position: [0, 0, 1]}, 57: {position: [0, 0, 0]}}} />
      id="63" points="#transpose" colors="#color1" color="#ffffff" start={false} end={false} width={40} opacity={3/100} blending="add" zWrite={false} zOrder={-2} />
    >
  >
>
  






Full Stack

What's left is basically kicking the tires and fixing the blind spots. As such this is not MathBox 2.0, this is MathBox 2 Alpha 1. It's still rough in the compatibility department, easily letting you exceed GL limits satisfied by only 70-80% of WebGL implementations in the wild, without warning. My own goal for public release was to be able to make another one of those presentations with it, only this time, 100% idiosyncratic MathBox. Result, The Pixel Factory.

Some people have assumed this talk was another tour-de-force of multi-week autism, but in fact, rebuilding my old slides for v2 was easy and obvious. The RGBA subpixels and their labels are animated lambdas and GLSL. The multi-samples, the depth buffer columns, the tangents and normals, same thing. JavaScript twiddles the knobs while the GPU visualizes the visualizer, and in doing so, itself.

Here the code be.

To give it a whirl in your browser, open the JSBin Sandbox. There is a quick start introduction and a list of legos.


  MathBox² - PowerPoint Must Die
  A DOM for Robots - Modelling Live Data
  Yak Shading - Data Driven Geometry
  ShaderGraph 2 - Functional GLSL







A DOM for Robots
2015-09-27T00:00:00+02:00








Modelling Live Data

I want to render live 3D graphics based on a declarative data model. That means a choice of shapes and transforms, as well as data sources and formats. I also want to combine them and make live changes. Which sounds kind of DOMmy.




  




3D engines don't have Document Object Models though, they have scene graphs and render trees: minimal data structures optimized for rendering output. In Three.js, each tree node is a JS object with properties and children like itself. Composition only exists in a limited form, with a parent's matrix and visibility combining with that of its children. There is no fancy data binding: the renderer loops over the visible tree leaves every frame, passing in values directly to GL calls. Any geometry is uploaded once to GPU memory and cached. If you put in new parameters or data, it will be used to produce the next frame automatically, aside from a needsUpdate bit here and there for performance reasons.

So Three.js is a thin retained mode layer on top of an immediate mode API. It makes it trivial to draw the same thing over and over again in various configurations. That won't do, I want to draw dynamic things with the same ease. I need a richer model, which means wrapping another retained mode layer around it. That could mean observables, data binding, tree diffing, immutable data, and all the other fun stuff nobody can agree on.

However I mostly feed data in and many parameters will end up as shader properties. These are passed to Three as a dictionary of { type: '…', value: x } objects, each holding a single parameter. Any code that holds a reference to the dictionary will see the same value, as such it acts as a register: you can share it, transparently binding one value to N shaders. This way a single .set('color', 'blue') call on the fringes can instantly affect data structures deep inside the WebGLRenderer, without actually cascading through.





  
    
  




  




I applied this to build a view tree which retains this property, storing all attributes as shareable registers. The Three.js scene graph is reduced to a single layer of THREE.Mesh objects, flattening the hierarchy. Rather than clumsy CSS3D divs which encode matrices as strings, there's binary arrays, GLSL shaders, and highly optimizable JS lambdas.

As long as you don't go overboard with the numbers, it runs fine even on mobile.




  

id="1" scale={600} focus={3}>
  id="2" proxy={true} position={[0, 0, 3]} />
  id="3" code="
uniform float time;
uniform float intensity;

vec4 warpVertex(vec4 xyzw, inout vec4 stpq) {
  xyzw +=   0.2 * intensity * (sin(xyzw.yzwx * 1.91 + time + sin(xyzw.wxyz * 1.74 + time)));
  xyzw +=   0.1 * intensity * (sin(xyzw.yzwx * 4.03 + time + sin(xyzw.wxyz * 2.74 + time)));
  xyzw +=  0.05 * intensity * (sin(xyzw.yzwx * 8.39 + time + sin(xyzw.wxyz * 4.18 + time)));
  xyzw += 0.025 * intensity * (sin(xyzw.yzwx * 15.1 + time + sin(xyzw.wxyz * 9.18 + time)));

  return xyzw;
}"
 time=>{(t) => t / 4} intensity=>{(t) => {
        t = t / 4;
        intensity = .5 + .5 * Math.cos(t / 3);
        intensity = 1.0 - Math.pow(intensity, 4);
        return intensity * 2.5;
      }} />
  id="4" stagger={[10, 0, 0, 0]} enter=>{(t) => 1.0 - Math.pow(1.0 - Math.min(1,  (1 + pingpong(t))*2), 2)} exit=>{(t) => 1.0 - Math.pow(1.0 - Math.min(1,  (1 - pingpong(t))*2), 2)}>
    id="5" pass="view">
      id="6" bend={1/4} range={[[-π, π], [0, 1], [-1, 1]]} scale={[2, 1, 1]}>
        id="7" position={[0, 1/2, 0]}>
          id="8" detail={512} />
          id="9" divide={10} unit={π} base={2} />
          id="10" width={3} classes=["foo", "bar"] />
          id="11" divide={5} unit={π} base={2} />
          id="12" expr={(x) => {
        return x ? (x / π).toPrecision(2) + 'π' : 0
      }} />
          id="13" depth={1/2} zIndex={1} />
        >
        id="14" axis={2} detail={128} crossed={true} />
        id="15" position={[π/2, 0, 0]}>
          id="16" axis={2} detail={128} crossed={true} />
        >
        id="17" position={[-π/2, 0, 0]}>
          id="18" axis={2} detail={128} crossed={true} />
        >
        id="19" divideX={40} detailX={512} divideY={20} detailY={128} width={1} opacity={1/2} unitX={π} baseX={2} zBias={-5} />
        id="20" width={512} expr={(emit, x, i, t) => {
        emit(x, .5 + .25 * Math.sin(x + t) + .25 * Math.sin(x * 1.91 + t * 1.81));
      }} channels={2} />
        id="21" width={5} />
        id="22" pace={10} loop={true} to={3} script=[[{color: "rgb(48, 144, 255)"}], [{color: "rgb(100, 180, 60)"}], [{color: "rgb(240, 20, 40)"}], [{color: "rgb(48, 144, 255)"}]] />
      >
    >
  >
>  
  
  
    
  Note: The JSX is a lie, you define nodes in pure JS.




Keep it Simple

From afar there's a tree of nodes, similar to SVG tags. This is the MathBox library of vector primitives. The basic shapes are all there: points, lines, faces, vectors, surfaces, etc. These nodes are placed inside a shallow hierarchy of views and transforms.

However none of the shapes draw anything by themselves. They only know how to draw data supplied by a linked source. Data can be an array (static or live), a procedural source, custom JS / GLSL code, etc. This is further augmented by data operators which can be sandwiched between source and shape, forming automatic pipelines between siblings.

The current set of components looks like this:









  
  Base
  
    Group
    Inherit
    Root
    Unit
  

  Camera
  
    Camera
  
  

  
  Draw
  
    Axis
    Face
    Grid
    Line
    Point
    Strip
    Surface
    Ticks
    Vector
  
  

  
  Data
  
    Area
    Array
    Interval
    Matrix
    Scale
    Volume
    Voxel
  
  

  
  Operator
  
    Grow
    Join
    Lerp
    Memo
    Resample
    Repeat
    Slice
    Split
    Spread
    Swizzle
    Transpose
  
  

  

  
  Overlay
  
    DOM
    HTML
  

  Present
  
    Move
    Play
    Present
    Reveal
    Slide
    Step
  
  

  
  RTT
  
    Compose
    RTT
  

  Shader
  
    Shader
  
  

  
  Text
  
    Format
    Label
    Text
    Retext
  

  Time
  
    Clock
    Now
  
  

  
  Transform
  
    Fragment
    Layer
    Transform
    Transform4
    Vertex
  
  

  
  View
  
    Cartesian
    Cartesian4
    Polar
    Spherical
    Stereographic
    Stereographic4
    View
  
  









To make you feel at home, nodes have an id and classes, and you can use CSS selectors to identify them. Nodes link up with preceding siblings and parents by default, but you can select any node in the tree. This allows for arbitrary graphs, including feedback loops. However all of this is optional: you can also pass in direct node objects or MathBox's own jQuery-like selections. What it doesn't have is a notion of detached document fragments: nodes are immediately inserted on creation.

A node's attributes can be .get() and .set(), though there is also a read-only .props dictionary for fashionable reasons. The values are strongly typed as Three.js colors, vectors, matrices, … but accept e.g. CSS colors and ordinary arrays too. The values are normalized for immediate use, the original values are preserved on the side for printing and serialization.



What's unique is the emphasis on time. First, properties can be directly bound to time-dependent expressions, on creation or afterwards. Second, clocks are primitives on their own. This allows for nested timelines, on-demand bullet time, fast forwards and more. It even supports limited time travel, evaluating an expression several frames in the past. This can be used to ensure consistent 60 fps data logging through janky updates, useful for all sorts of things. It's exposed publicly as .bind(key, expr) and .evaluate(key, time) per node. It's also dogfood for declarative animation tracks. The primitives clock/now provide timing, while step and play handle keyframes on tracks.

This is definitely a DOM, but it has only basic features in common with the HTML DOM and does much less. Most of the magic comes from the components themselves. There's no cascade of styles to inherit. Children compose with a parent, they do not inherit from it, only caring about their own attributes. The namespace is clean, with no weird combo styles à la CSS. As much as possible, attributes are unique orthogonal knobs you can turn freely.

Model-View-Projection

On the inside I separate the generic data model from the type-specific View Controller attached to it. The controller's job is to create and manage Three.js objects to display the node (if any). Because a data source and a visible shape have very little in common, the nodes and their controllers are blank slates built and organized around named traits. Each trait is a data mix-in, with associated attributes and helpers for common behavior. Primitives with the same traits can be expected to work the same, as their public facing models are identical.

Controllers can traverse the graph to find each other by matching traits, listening for events and making calls in response. This way only specific events will cascade through cause and effect, often skipping large parts of the hierarchy. The only way to do a "global style recalculation" would be to send a forced change event to every single controller, and there's never a reason to do so.

The controller lifecycle is deliberately kept simple: make(), made(), change(…), unmake(), unmade(). When a model changes, its controller either updates in place, or rebuilds itself, doing an unmake/make cycle. The change handler is invoked on creation as well, to encourage stateless updates. It affords live editing of anything, without having to micro-optimize every possible change scenario. Controllers can also watch bound selectors, retargeting if their matched set changes. This lets primitives link up with elements that have yet to be inserted.

Unlike HTML, the DOM is not forced to contain a render tree as well. Only some of the leaf nodes have styles and create renderables. Siblings and parents are called upon to help, but the effects don't have to be strictly hierarchical. For example, a visual effect can wrap a single leaf but still be applied after all its parents, as transformations are collected and composed in passes.


It'll Do

The result is not so much a document model as it is a computational model inside a presentational model. You can feed it finalized data and draw it directly… or you can build new models within it and reproject them live. Memoization enables feedback and meta-visualization. The line between data viz and demo scene is rarely this blurry.

Here, the notion of a computed style has little meaning. Any value will end up being transformed and processed in arbitrary ways down the pipe. As I've tried to explain before, the kinds of things people do with getComputedStyle() and getClientBoundingRect() are better achieved by having an extensible layout model, one that affords custom constraints and composition on an equal footing. To do otherwise is to admit defeat and embrace a leaky abstraction by design.

The shallow hierarchy with composition between siblings is particularly appealing to me, even if I realize it introduces non-traditional semantics more reminiscent of a command-line. It acts as both a jQuery-style chainable API, and a minimal document model. If it offends your sensibilities, you could always defuse the magic by explicitly wiring up every relationship. In case of confusion, .inspect() will log syntax highlighted JSX, while .debug() will draw the underlying shader graphs.

I've defined a good set of basic primitives and iterated on them a few times. But how to implement it, when WebGL doesn't even fully cover OpenGL ES 2?



  MathBox² - PowerPoint Must Die
  A DOM for Robots - Modelling Live Data
  Yak Shading - Data Driven Geometry
  ShaderGraph 2 - Functional GLSL






Shadow DOM
2014-03-24T00:00:00+01:00


SVG, CSS, React and Angular







For a while now I've been working on MathBox 2. I want to have an environment where you take a bunch of mathematical legos, bind them to data models, draw them, and modify them interactively at scale. Preferably in a web browser.

Unfortunately HTML is crufty, CSS is annoying and the DOM's unwieldy. Hence we now have libraries like React. It creates its own virtual DOM just to be able to manipulate the real one—the Agile Bureaucracy design pattern.

The more we can avoid the DOM, the better. But why? And can we fix it?






  





  













  
Dangling Nodes

Take SVG. Each XML tag is a graphical shape or instruction. Like all XML, the data has to be serialized into tags with attributes made of strings. Large data sets turn into long string attributes to be parsed. Large collections of stuff turn into many separate tags to be iterated over. Neither is really desirable.

It only represents basic operations, so all serious prep work has to be done by the user up front. This is what D3 is used for, generating and managing more complex mappings for you.

When you put SVG into HTML, each element becomes a full DOM node. A simple  with attributes is now a colossal binding between HTML, JS, CSS and native. It's a JavaScript object that pretends to be an XML tag, embedded inside a layout model that takes years to understand fully.

Its namespace mixes metadata with page layout, getters and setters with plain properties, native methods with JS, string shorthands with nested objects, and so on. Guess how many properties the DOM Node Object actually has in total? We'll be generous and count style as one.

A hundred is not even close. A plain 
 doesn't fare much better. Just serializing a chunk of DOM back into its constituent XML is a tricky task once you get into fun stuff like namespaces. Nothing in the DOM is as simple as JSON.stringify. Why does my polygon have a base URI?

We have all these awesome dev tools now, yet we're using them to teach a terrible model to people who don't know any better.




  




DOM Shader

In contrast, there's Angular. I like it because they've pulled off a very neat trick: convincing people to adopt a whole new DOM by disguising it as HTML.



  
    
      {{phone.name}}
      {{phone.snippet}}
    
  





When you use  or , you're creating a controller and a scope, entirely separate from the DOM, with their own rules and chain of inheritance. The pseudo-HTML in the source code is merely an initial definition, most of it inert to the browser. Angular parses it out and replaces much of it.

Like React, the browser's live DOM is subsumed and used as a sort of render tree, a generic canvas to be cleverly manipulated to match a given set of views. The real view tree hides in the shadows of JS, where controllers operate on scopes. They only use the DOM to find each other on creation, and then communicate directly. The DOM is mostly there to trigger events, do layout and look pretty. Form controls are the one exception.




  




It's a bad fit because the DOM was built for text markup and there's tons of baggage in the form of inline spans, floats, alignment, indentation, etc. Most of these are layout systems disguised as typography, of which CSS now has several.

The whole idea of cascading styles is suspect. In reality, most styles don't actually cascade: paddings and backgrounds are set on individual elements. The inherited ones are almost all about typography: font styles, text justification, writing direction, word wrap, etc.

Think of it this way: why should a table have a font size? Only the text inside the table can have a font size, the table is just a box with layout that contains other boxes. Why don't we write table text { size: 16px; } instead of table { font-size: 16px; }? Text nodes exist today.

Well because that's how HTML's  tag worked. Instead of just making a selector for text nodes, they gave all the other elements font properties. They didn't get rid of font tags, they made them invisible and put one inside each DOM node.







  
    Hello World
    Welcome to the future.
  




Unreasonable Behavior

It was decided the world would be made of block and inline elements—divs and spans—and they saw that it was good, until someone came along and said, hey, so what about my table?


  
    Forever
    Alone
  




This 
 can't be replicated with CSS 1. Tables require a particular arrangement of children and apply their own box model. It's a directive posing amongst generic markup, just like Angular.
CSS has never been able to deliver on the promise of turning semantic HTML into arbitrary layout. We've always been forced to add extra divs or classes. These are really just attachment points for independent behaviors.
Purists see these as a taint upon otherwise pristine HTML, even though I've never seen someone close a website because the markup was messy. Not all HTML should be semantic. Rather, HTML stripped of its non-semantic parts should remain meaningful to robots.
CSS 2's solution was instead to make 






 invisible too, to go with the invisible , ,  and  tags which we pretended we didn't have. Watch:



  17.2.1 Anonymous table objects



  […] Any table element will automatically generate necessary anonymous table objects around itself, consisting of at least three nested objects corresponding to a 'table'/'inline-table' element, a 'table-row' element, and a 'table-cell' element. Missing elements generate anonymous objects (e.g., anonymous boxes in visual table layout) according to the following rules […]




  .grid {
  display: table;
}
.grid > ul {
  display: table-row;
}
.grid > ul > li {
  display: table-cell;
}



This is called Not Using Tables.

 
  
Without typographical styles, block elements start to look very different. They're styled boxes with implied layout constraints. They stack vertically, expand horizontally and shrink wrap vertically. Floated blocks are boxes that stack horizontally, and shrink wrap both ways. Tables are grids of boxes that are locked together.


Just think how much simpler CSS would be if boxes had box styles and text had text styles, instead of all of them having both. Besides, block margins and paddings don't even work the same on inline elements, there's a whole new layout behavior there.

So we do have two kinds of objects, text and boxes, but several different ways of combining them into layout: inline, stacked, nested, absolute, relative, fixed, floated, flex or table. We have optional behaviors like scrollable, draggable, clipped or overflowing.

They're spread across display, position, float and more, only meaningful in some combinations. And presence is mixed in there too. As a result, you can't unhide an element without knowing its display model. This is a giant red flag.

Thinking with Portals
  
It should further raise eyebrows that the binary world of inline and block now also includes a hybrid called inline-block.



You generally don't need to embed a contact form–or all of Gmail—in the middle of mixed English/Hebrew poetry shaped like a bird. You just don't. To attach something to flowing text, you should insert an anchor point instead and add floating constraints. Links are called anchor tags for a reason. Why did we forget this?

Don't shove your entire widget right between the words. You'd inherit styles, match new selectors and bubble all your events up through the text just for the sake of binding a pair of (x, y) coordinates.

Heck, pointer events, cursors, hover states... these are for interactive elements only. Why isn't that optional, so mouse events wouldn't need to bubble up through inert markup? This would completely avoid the mouseover vs mouseenter problem. What is the point of putting a resize cursor on something that is dead without JavaScript? Pointer events shouldn't fire on inert children, and inert parents shouldn't care about interactive children. It's about boundaries, not hierarchy.



Things like SVG are better used as image tags instead of embedded trees, just slotting into place while ignoring their surroundings. They do need their own tree structure, but there is little reason to graft it onto HTML/CSS, inheriting original sin. The nodes have too little in common. At most you can share the models, not the controllers.

We should be able to manipulate them from the outside, like a , but define and load them declaratively, like an image tag.

For that matter, MathML should really be a single inline text tag, optimized for math, not a bunch of tags. Regular text spans are not just "plain text". They are trimmed, joined, bidirectionalized, word wrapped and ellipsified before display. It's a separate embedded layout model that makes up the true, invisible 
 tag. A tag that HTML1 actually sort of got right: as an operator.

We create JavaScript with code, not as abstract syntax trees. Why should I build articles and embedded languages out of enormously nested trees, instead of just typing them out and adding some anchor tags around specific interesting parts? The DOM already inserts invisible text nodes everywhere. We didn't need to wrap all our words in  tags by hand just to embiggen one of them. The mutant tree on the right could just look like this:
  
 $x = (-b ± √(b^2 - 4 a c)) / 2a$ 


 $x = (-b ± √(b^2 - 4 a c)) / 2a$ 


Wasn't HTML5 supposed to match how people write it? LaTeX exists.

And which is easier: defining a hairy new category of pseudo-elements like :first-letter and :first-line… or just telling people to wrap their first letter in a span if they really want to make it giant? It was ridiculous to have this feature in a spec that didn't include tables.

The :first-line problem should be solved differently: you define two separate blocks inside a directive, to spread markup across two children with a content binding. It's no different from flowing text across lines and columns.



  
  x
  =
  
    
      
        -
        b
      
      ±
      
        
          
            b
            2
          
          -
          
            4
            ⁢
            a
            ⁢
            c
          
        
      
    
    
      2
      ⁢
      a
    
  




This is the first example in the MathML spec. Really. "Invisible times".



  
  
  Hello New World




Would this really be insane?



The Boxed Model

CSS got it wrong and we're now suffering the consequences. The HTML feature that was ignored in CSS 1 was the thing they should've focused on: tables, which were directives that generated layout. It set us on a path of trying to fake them by piggybacking on supposedly semantic elements, like lipstick on a div. Really we were pigeonholing non-linear layout as a nested styling problem.

Semantic content was a false spectre on the document level. Making our menus out of 
 and  tags did not help impaired users skip to the main article. Adding roman numerals for lists did not help us number our headers and chapters automatically.

View and render trees are supposed to be simple and transparent data structures, the model for and result of layout. This is why absolute positioning is a performance win for mobile: it avoids creating invisible dynamic constraints between things that rarely change. Styles are orthogonal to that, they merely define the shape, not where it goes.

Flash had its flaws, but it worked 15 years ago. Shoving raw SVG or MathML into the DOM—or god forbid XML3D–is a terrible idea. It's like there's an entire class of developers who've now forgotten how fast computers actually are and how memory is supposed to work. A stringly typed kitchen sink is not it.

So I frown when I see people excited about SVG in the browser in the year 2014, making polygons out of CSS 3D or driving divs with React. Yes I know, it's fun and it does work. And Angular shows the web component approach has merit. But we need a way out.

CSS should be limited to style and typography. We can define a real layout system next to it rather than on top of it. The two can combine in something that still includes semantic HTML fragments, but wraps layout as a first class citizen. We shouldn't be afraid to embrace a modular web page made of isolated sections, connected by reference instead of hierarchy.

Not my problem though, I can make better SVGs with WebGL in the meantime. But one can dream.


On Asm.js
2013-11-27T00:00:00+01:00


Ending The Ice Age of JavaScript

The demo is striking: Unreal Engine, running live in a browser, powered by Mozilla's Asm.js, offering near native performance with hardware accelerated graphics. A gasp here, a wow there, everyone is surely impressed. And look ma', no plug-ins: just JavaScript, Web Audio and WebGL! Just don't mind the 10 second hang on load as it's Preparing JavaScript.

When I heard of it, it sounded great: a highly optimizable subset of JS, that can be dropped seamlessly into any existing code. It reminded me of Winamp's charming evallib compiler, used in AVS and Milkdrop, which only did floating point expressions. It spawned a whole subculture of visual hackery based on little more than a dozen or so math functions and some clever graphics routines. It showed the power of being able to turn scripts into optimal machine code on the fly, and having a multimedia platform at your disposal while doing so.






    




But that's not what Asm.js is for at all. It's not for people, it's a compiler target, a way of converting non-JS code into a form that browsers handle well. Its design is based on how JavaScript handles numbers: as 64-bit doubles, onto which you can perform select 32-bit integer operations. As such, Asm.js seems like a sweet hack similar to UTF-8, an elegant way of encoding something complicated under strong legacy constraints: typed arithmetic despite a single number type. Yet the part of me that remembers pushing pixels with MMX, that watched this web thing for more than a few years, can't help but ask exactly what it is we're trying to do here.

Asm.js deserves closer inspection for two reasons. First, it's the one "native browser VM" that doesn't massively reinvent wheels. Second, it's the only time a browser vendor's "next-gen JS" attempts have actually gotten everybody else to pay attention. But what are we transitioning into exactly?

LLVM to Asm

To understand Asm.js, you have to understand LLVM. Contrary to its name, it's not really a Virtual Machine but rather a compiler. More precisely, it's the modular core of a compiler, along with assorted tools and libraries. You plug in different front-ends to parse different languages (e.g. C or C++), and different back-ends to target different machines (e.g. Intel or ARM). LLVM can do common tasks like optimization in a platform and language agnostic way, and as such should probably be considered an open-source treasure.



LLVM architecture (source)

Per Atwood's law, it was inevitable that someone decided the back-end should be JavaScript. Thus was born emscripten, turning C into JS—or indeed, anything native into JS. Because the output is tailored to how JS VMs work, this already gets you pretty far. The trick is that native code manages its own memory, creating a stack and heap. Hence you can output JS that just manipulates pre-allocated typed arrays as much as possible, and minimizes use of the garbage collector.

This works particularly well because JavaScript VMs already cheat when it comes to numbers and number types. They receive special treatment compared to other data. You can find a good overview in value representation in JavaScript implementations. The gist is that JS VMs handle floating point and integer arithmetic separately and efficiently, with lots of low level trickery to speed up computation. Modern VMs will furthermore try to identify code that uses one type only, and emit highly optimized type-specific code with only minimal checks at the boundaries. It's this kind of code that emscripten can emit a lot of, e.g. translating the clang compiler into 48MB of JS.





function DiagModule(stdlib, foreign, heap) {
  "use asm";

  // Variable Declarations
  var sqrt = stdlib.Math.sqrt;

  // Function Declarations
  function square(x) {
      x = +x;
      return +(x*x);
  }

  function diag(x, y) {
      x = +x;
      y = +y;
      return +sqrt(square(x) + square(y));
  }

  return { diag: diag };
}

Example Asm.js code from the spec.




Which brings us to the bittersweet hack of Asm.js. Once you realize that C can run 'well' as JavaScript even when the VM has to guess and juggle, imagine how much faster it could be when the VM is in on it.

What this means in practice is a directive "use asm" in a block of tailored code, along with implicit type annotations for arguments, casts and return values. Type casts are x|0 for int, +x for double. These annotations are parsed and validated, and optimized code is emitted for the entire code block. This doesn't look bad at all. However, it also looks nothing like real Asm.js code in the wild.

Benchmarks show roughly a 1–2× slowdown compared to native code, significantly faster than normal JS. Hooray, JavaScript wins again, the web is awesome. Because of LLVM, an enormous piece of external, non-web infrastructure. Wait what?





Impostor Syndrome

When something bugs me that I can't put my finger on, there's usually a contradiction that I'm not seeing. After a few talks, articles and conversations, it seems pretty obvious: it puts JavaScript on a pedestal, even as it makes it irrelevant.

It makes JavaScript irrelevant because with LLVM's infrastructure or similar tools, practically anything can or will be compiled into JS.

But it also makes JavaScript more important, focusing the future optimization efforts of browser makers onto it, despite its ill-suited semantics.

It means JavaScript has nothing to do with it, it's just the poison we ended up with, the bitter pill we supposedly have to swallow. So when Brendan Eich says with a smile "Always bet on JavaScript", what he really means is "Always bet on legacy code" or perhaps "Always bet on politics". When you think about it, it's weird to tell JavaScript developers about Asm.js. It's not actually aimed at them.
  
Looking around, in the browser there's CoffeeScript, TypeScript and Dart. Outside, there's Python, Ruby, Go and Rust. Even CSS now has offspring. The future of the web is definitely multilingual and some people want to jump ship, they're just not sure which one will actually sail yet.

When faced with a legacy mechanism like UTF-8 or indeed Asm.js, we have to ask, is it actually necessary? In the case of UTF-8, it's a resounding yes: we need to assign unique names to things, and this name has to work with both modern and legacy software, passing through unharmed as much as possible. UTF-8 solves a bunch of problems while causing very few.

But with Asm.js, it's just a nice to have. All Asm.js code is new, there is no vault of legacy code that will stop working if we do it wrong. We can already generate functioning JS for legacy browsers, along with something new for alternative VMs. Having one .js file that does both is merely a convenience, and a dubious one at that.

Indeed, the unique appeal of Asm.js is for the browser maker who implements it first: it lets their JS VM race closer to that much desired Native line. It also turns any demo that uses Asm.js into an instant benchmark, which other vendors have to catch up with. It's a rational choice made in self interest, but also a tragedy of the commons.

Maybe that's a bit hyperbolic, but work with me here. There's a serious amount of defeatism and learned helplessness at work here, and again a contradiction. We seek to get ever closer to native performance, yet fall short by design, resigning ourselves to never quite reaching it. I can't be the only one who finds that completely bizarre, when there's laptops and phones running entirely on a web stack now?

If you look at the possible future of Asm.js, there's SIMD extensions, long ints with value objects, math extensions, support for specific JVM/CLR instructions and more. Asm.js is positioned not just as something that works today, but that leads into a bright future to boot. And yet, it all has to be shoehorned into something that is still 100% JavaScript, even as that target itself consists of moving goalposts.






  
  Part of Unreal Engine, JSified.




History Repeating

So fast forward a year or two. Firefox has completed its wishlist and Asm.js has filled its instruction set gaps. Meanwhile Chrome has continued to optimize V8. Will they have the same new language features that Firefox has? Will they officially support Asm.js? Or push Dart and PNaCl, expanding their influence through ChromeOS and Android? Your guess is as good as mine. As for IE and Safari, I'll just pencil in "behind" for now and leave it at that.

But a certain phrase comes to mind: embrace and extend. From multiple fronts.




  
  





It looks like a future where your best bet to get things running fast in a browser is to do decidedly non-web things. You compile something like C to the different flavors of web at your disposal, either papering over their strengths, or tailoring for each individually. That's not something I personally look forward to, as much as it might arouse Epic's executives and shareholders today.

Web developers wouldn't actually be working that differently. They might be using a multi-language framework like Angular, or dropping in a neat C physics library somebody cross compiled for them. I doubt they'll have a nice web-native way to run the same performance-critical code everywhere. You'll just waste some battery life because a computer pretended to be a JavaScript developer. For backwards compatibility with browsers that auto-update every few weeks. Eh?





I admit, I don't know what the post-JS low level future should look like either, but it should probably be closer to LLJS's nicely struct'ed and typed order, than either the featherweight of Asm.js or the monolithic Flash-replacement that is PNaCl.

The big problem with Asm.js isn't that it runs off script rather than bytecode, it's that the code is generated to match how JS engines work rather than how CPUs compute. At best it will be replaced with something more sensible later or just fizz out as an optimization fad. At worst it'll become the IA-32 of the web, still pretending to be an 8086 if asked to.

Looking ahead, there's computation with WebCL, advanced GLSL shaders and more on the horizon. That's a whole set of problems that can become much simpler when "browser" is a language that everyone can speak, to and from, rather than a weird write-only dialect built on a tower of Babel. We don't just need a compilation target, we need a compilation source, as well as a universal intermediate representation.

And this is really the biggest contradiction of them all. Tons of people have invested countless hours to build these VMs, these new languages, these compilers, these optimizations. Yet somehow, they all seem to agree that it is impossible for them to sit down and define the most basic glue that binds their platforms, and implement a shared baseline for their many innovations.

We really should aim higher than a language frozen after 10 days, thawing slowly over 20 years.




Making MathBox
2012-11-14T00:00:00+01:00


  Presentation-Quality Math with Three.js and WebGL






  
  A fun little graph involving rational functions on the real projective line.




  For most of my life, I've found math to be a visual experience. My math scores went from crap to great once I started playing with graphics code, found some demoscene tutorials, and realized I could reason about formulas by picturing the graphs they create. I could apply operators by learning how they morph, shift, turn and fold those graphs and create symmetries. I could remember equations and formulas more easily when I could layer on top the visual relationships they embody. I was less likely to make mistakes when I could augment the boring symbolic manipulation with a mental set of visual cross-checks.

  So, when tasked with holding a conference talk on how to make things out of math at Full Frontal—later redone at Web Directions Code—I knew the resulting presentation would have to consist of intricate visualizations as the main draw, with whatever I had to say as mere glue to hold it together.

  The problem was, I didn't know of a good tool to do so, and creating animations by hand would probably be too time consuming. With the writings of Paul Lockhart and Bret Victor firmly in mind, I also knew I wanted to start blogging more about mathematical concepts in a non-traditional way, showing the principles of calculus, analysis and algebra the way I learnt to see them in my head, rather than through the obscure symbols served up in engineering school.

  So I set out to create that tool, keeping in mind the most important lesson I've picked up as a web developer: one cannot overstate the value in being able to send someone a link and have it just work, right there. It was obvious it would have to be browser-based.

  2015 Update: MathBox has evolved, version 2 is now available.




  
    
    
    Presentation Video
(updated)
    
  



  
    
    
    Slide Deck

    (updated)
    
  




  Choose your Poison

  Now, when people think of graphs in a browser, the natural thought is vector graphics and SVG, which quickly leads to visualization powerhouse d3.js. It really is an amazing piece of tech with a vast library of useful code to accompany it. When I wrapped my head around how d3's enter/exit selections are implemented and how little it actually does to achieve so much, I was blown away. It's just so elegant and simple.

  Unfortunately, d3's core is intricately tied to the DOM through SVG and CSS. And that means ironically that d3 is not really capable of 3D. Additionally, d3 is a power tool that makes no assumptions: it is up to you to choose which visual elements and techniques to use to make your diagrams, and as such it is more like assembly language for graphs than a drop-in tool. These two were show stoppers.

  For one, manually designing layouts, grids, axes, etc. every time is tedious. You should be able to drop in a mathematical expression with as little fanfare as possible and have it come out looking right. This includes sane defaults for transitions and animations.




  




  For another, I've found that, when in doubt, adding an extra dimension always helps. The moment I finally realized that every implicit graph in N dimensions is really just a slice of an explicit one in N+1 dimensions, a ridiculous amount of things clicked together. And it took until years after studying signal processing to at long last discover the 4D picture of complex exponentiation that tied the entire thing together (projected into 3D below): it revealed the famous "magic formula" involving e, i and π to be a meaningless symbological distraction, a pinhole view of a much larger, much more beautiful structure, underpinning every Fourier and Z transform I'd ever encountered.




  
    e^iπ = -1
  
  This particular formula is not that important.

  
    e^x+iy = e^x · e^iy = e^x ∠ y
    
  
  This one is (∠ = rotate by).
Unfortunately it has a four dimensional graph.
  



  




  So, WebGL it was, because I needed 3D. Unfortunately that meant the promise of having it just work everywhere was tempered by a lack of browser support, but I would certainly hope that's something we can overcome sooner than later. Dear Apple and Microsoft: get your shit together already. Dear Firefox and Opera: your WebGL performance could be a lot better.

  Shady Dealings

  These days I don't really touch WebGL without going through Three.js first. Three.js is a wonderful, mature engine that contains tons of useful high-level components. At the same time, it also does a great job in just handling the boilerplate of WebGL while not getting in the way of doing some heavy lifting yourself.

  Rendering vector-style graphics with WebGL is not hard, certainly easier than photorealistic 3D. Primitives like lines and points are sized in absolute pixels by default, and with hardware multisampling for anti-aliasing, you get somewhat decent image quality out of it. Though, as is typical for a Web API, we're treated like children and can only cross our fingers and request anti-aliasing politely, hoping it will be available. Meanwhile native developers have full control over speed and quality and can adjust their strategy to the specific hardware's capabilities. The more things change... And then Chrome decided to disable anti-aliasing altogether due to esoteric security issues with buggy drivers. Bah.

  Now, when rendering with WebGL, you really have two options. One is to just treat it as a dumb output layer, loading or generating all your geometry in JavaScript and rendering it directly in 3D. With the speed of JS engines today, this can get you pretty far.




  The second option is to leverage the GPU's own capabilities as much as possible, doing computations in GLSL through so-called vertex and fragment shader programs. These are run for every vertex in a mesh, every pixel being drawn, and have been the main force driving innovation in real-time graphics for the past decade. With the goal of butter-smooth 60fps graphical goodness, this seemed like the better choice.

  Unfortunately, GLSL shaders are rather monolithic things. While you do have the ability to create subroutines, every shader still has to be a stand-alone program with its own main() function. This means you either need to include a shader for every possible combination of operations, or generate shader code dynamically by concatenating pre-made snippets or using #ifdef switches to knock them out. This is the approach taken by Three.js, which results in some very hairy code that is neither easy to read nor easy to maintain.

  Having made a prototype, I knew I wanted to show continuous transitions between various coordinate systems (e.g. polar and spherical), knew I needed to render shaded and unshaded geometry, and knew I would need to slot in specific snippets for things like point sprites, bezier curves/surfaces, dynamic tick marks, and more. Sorting this all out Three.js-style would be a nightmare.





uniform sampler2D texture;
varying vec2 vUV;

void main() {
  gl_FragColor = texture2D(texture, vUV);
}


A pixel or fragment shader that looks up a pixel's color in a texture.


uniform mat4 projectionMatrix;
uniform mat4 modelViewMatrix;
attribute vec4 position;
attribute vec2 uv;
varying vec2 vUV;

void main() {
  vUV = uv;

  gl_Position = projectionMatrix
              * modelViewMatrix
              * position;
}



A vertex shader that projects a 3D position into 2D by applying two matrices. It also provides UV coordinates for the texture look up.






var graph =
  factory
    .snippet('split')
    .group()
      .snippet('top')
    .next()
      .snippet('middle')
    .next()
      .snippet('bottom')
    .combine()
    .snippet('join')
    .end();


ShaderGraph's factory API lets you build shader chains with very little hassle. In this case, the names refer to IDs of <script> tags in the source.





  So I wrote a library to solve that problem, called ShaderGraph.js. It is best described as a smart code-concatenator, a few steps short of writing a full blown compiler. You feed it snippets of GLSL code, each with one or more inputs and outputs, and these get parsed and turned into lego-like building blocks. Each input/output becomes an outlet, and outlets are wired up in a typical dataflow style. Given a graph of connected snippets, it can be compiled back into a program by assembling the subroutines, assigning intermediate variables and constructing an appropriate main() function to invoke them. It also exports a list of all external variables, i.e. GLSL uniforms and attributes, so you can control the program's behavior easily.

  If I'd stopped there however, I'd have just replaced the act of manual code writing with that of manually wiring graphs. So I applied the principle of convention-over-configuration instead: you tell ShaderGraph to connect two snippets, and it will automatically match up outlets by name and type. This is augmented by a chainable factory API, which allows you to pass a partially built graph around. It allows different classes to work together to build shaders, each inserting their own snippets into the processing chain.
  
For example, to render a Bezier surface, the vertex shader is composed of: cubic interpolation, viewport transform (position + tangents), normal calculation and lighting. When transforming to e.g. a polar viewport, the surface normals are seamlessly recalculated. It really works like magic and I can't wait to use this in my next WebGL projects.




  
  Viewports, Primitives and Renderables

  At its core, Three.js matches pretty directly with WebGL. You can insert objects such as a Mesh, Line or ParticleSystem into your scene, which invokes a specific GL drawing command with high efficiency. As such, I certainly didn't want to reinvent the wheel.

  Hence, MathBox is set up as a sort of scene-manager-within-a-scene-manager. It's a little sandbox that speaks the language of math, allowing you to insert various primitives like curves, vectors, axes and grids. Each of these primitives then instantiates one or more renderables, which simply wrap a native Three.js object and its associated ShaderGraph material. Thus, once instantiated, MathBox gets out of the way and Three.js does the heavy lifting as normal. You can even insert multiple mathboxen into a Three.js scene if you like, mixed in with other objects.



  For example, a vector primitive is rendered as an arrow: it consists of a shaft and an arrowhead, realized as a line segment and a cone. An axis primitive is an arrow as well, but it also has tick marks (specially transformed line segments), and is positioned implicitly just by specifying the axis' direction rather than a start and end point.

  To render curves and surfaces, you can either specify an array of data points or a live expression to be evaluated at every point. This turned out to be essential for the kinds of intricate visualizations I wanted to show, my slides being driven by timed clocks, shared arrays of data points, and live formulas and interpolations. I even fed in data from a physics engine, and it worked perfectly.

  This is all tied together through Viewport objects, which define a specific mapping from a mathematical coordinate space into the 3D world space of Three.js. For example, the default cartesian viewport has the range [–1, 1] in the X, Y and Z directions. Altering the viewport's extents will shift and scale anything rendered within, as well as reflow grids and tick marks on each axis.

  There are two more sophisticated viewport types, polar and spherical, which each apply the relevant coordinate transform, and can transition smoothly to and from cartesian. More viewport types can be added, all that is required is to define an appropriate transformation in JavaScript and GLSL. That said, defining a seamless transition to and from cartesian space is not always easy, particularly if you want to preserve the aspect-ratio through the entire process.

  Interpolate all the things!

  Finally, I had to tackle the problem of animation, keeping in mind a tip I learnt from the ever so mindbending Vihart: "If I can draw the point of a sentence, I don't actually need to say the sentence." This applies doubly so for animation: every time you replace a "before" and "after" with a smooth transition, your audience implicitly understands the change rather than having to go look for it.

  Hence, each primitive can be fully animated. Each has a set of options (controlling behavior) and styles (controlling GLSL shaders), and there is a universal animator that can interpolate between arbitrary data types in a smart fashion.

  For example, given a viewport with the XYZ range [[–1, 1], [–1, 1], [–1, 1]], you can tell it to animate to [[0, 2], [0, 1], [–3, 3]], and it just works. The animator will recursively animate each subarray's elements, and any dependent objects like grids and axes will reflow to match the intermediate values. This works for colors, vectors and matrices too. In case of live curves with custom expressions, the animator will invoke both the old and the new, and interpolate between the results.




  
    
      
    
    
      
      
      
      
      
      
      
    
  



  However, executing animations manually in code is tedious, particularly in a presentation, where you want to be able to step forward and backward. So I added a Director class whose job it is to coordinate things. All you do is feed it a script of steps (add this object, animate that object). Then, as it applies them, it remembers the previous state of each object and generates an automatic rollback script. It also contains logic to detect rapid navigation, and will hurry up animations appropriately. This avoids that agonizing situation of watching someone skip through their slide deck, playing the same cheesy PowerPoint transitions over and over again.

  Presenting Naturally
  
  With MathBox's core working, it was time to build my slides for the conference. After a quick survey, I quickly settled on deck.js as an HTML5 slidedeck solution that was clean and flexible enough for my purposes. However, while MathBox can be spawned inside any DOM element, it wouldn't work to insert a dozen live WebGL canvases into the presentation. The entire thing would grind to a halt or at least become very choppy.
  
  So instead, I integrated each MathBox graphic as an IFRAME, and added some logic that only loads each IFRAME one slide before it's needed, and unloads it one slide after it's gone off screen. To sync up with the main presentation, all deck.js navigation events were forwarded into each active IFRAME using window.postMessage. With the MathBox Director running inside, this was very easy to do, and meant that I could skip around freely during the talk, without any worries of desynchronization between MathBox and the associated HTML5 overlays.
  
  In fact, I applied a similar principle to this post. To avoid rendering all diagrams simultaneously and spinning up laptop fans more than necessary, each MathBox IFRAME is started as it scrolls into view and stopped once it's gone.
  
  I've also found that having a handheld clicker makes a huge difference while speaking—as it allows you to gesture freely and move around. So, I grabbed the infrared remote code from VLC and built a simple bridge from to Cocoa to Node.js to WebSocket to allow the remote to work in a browser. It's a shame Apple's decided to discontinue IR ports on their laptops. I guess I'll have to come up with a BlueTooth-based solution when I upgrade my hardware.

  Towards MathBox 1.0

  In its current state, MathBox is still a bit rough. The selection of primitives and viewports is limited, and only includes the ones I needed for my presentation. That said, it is obvious you can already do quite a lot with it, and I couldn't have been happier to hear that all this effort had the desired response at the conference. I wasn't 100% sure whether other people would have the same a-ha moments that I've had, but I'm convinced more than ever that seeing math in motion is essential for honing our intuition about it. MathBox not only makes animated diagrams much easier to make and share, but it also opens the door to making them interactive in the future.

  I plan to continue to evolve MathBox as needed by using it on this site and addressing gaps that come up, though I've already identified a couple of sore points:

  
  I used tQuery as a boilerplate and because I liked the idea of having a chainable API for this. However, this also means it's currently running off an outdated version of Three.js. I need to look into updating and/or dropping tQuery.
MathBox has been updated to Three.js r53.
  Numeric or text labels are completely unsupported. It should be possible to use my CSS3D renderer for Three.js to layer on beautifully typeset MathJax formulas, positioning them correctly in 3D on top of the WebGL render.
I've added labeling for axes. I've integrated MathJax, but it's tricky because the typesetting is painfully slow in the middle of a 60fps render. But it's automatically used if MathJax is present.
  All styles have to be specified on a per-object basis. Some form of stylesheet, default styles or class mechanism to allow re-use seems like an obvious next step.
  There are undoubtedly memory leaks, as I was focused first and foremost on getting it to work.
  Expressions that don't change frame-to-frame are still continuously re-evaluated, which is wasteful. There is a live: false flag you can set on objects, but it triggers a few bugs here and there.
  There needs to be a predictable, built-in way of running a clock per slide to sync custom expressions off of. In my presentation I used a hack of clocks that start once first invoked, but this lacks repeatability.
I added a director.clock() method that gives you a clock per slide. 
  

  Finally, it doesn't take much imagination to imagine a MathBox Editor that would allow you to build diagrams visually rather than having to use code like I did. However, that's a can of worms I'm not going to open by myself, especially because the API is already quite straightforward to use, and the library itself is still a bit in flux. Perhaps this could be done as an extension of the Three.js editor.

  You can see what MathBox is really capable of in the conference video. I invite you to play around with MathBox and see what you can make it do. Contributions are welcome, and the architecture is modular enough to allow its functionality to grow for quite some time.




Going Full Frontal
2012-11-11T00:00:00+01:00


  Making things with Maths






  
    
    Duke of York's Picturehouse, a gorgeous venue, and "the oldest continuously operating purpose built cinema in Britain that has retained both its original name and remains largely unaltered."
  




Last week, I had the privilege of speaking about "Making things with Maths" at Full Frontal, a tech conference hosted in a gorgeous picturehouse in the seaside town of Brighton, UK. I was nervous as hell: I hadn't attended a tech conference in ages, let alone taken the stage, and I'd never done a talk on this subject before. I'd been planning and working for months to assemble the code just to be able to show what I saw in my head—and of course scrambled to finish the week before regardless. This talk has been the number one thing on my mind for a while.

Yet two days later, I barely remember my own part in it, and find myself mulling over everything else that was said instead. It was simply too good, too provoking, not to think about.

The lovely duo of organizers, Remy and Julie Sharp, have crafted something very special. The line up was stellar: each and every speaker challenged my preconceptions with the kind of casualness that only in-depth experience can bring.





Arguments for abandoning the purity of vanilla HTML (James Pearce) were followed by a philosophical lesson on not throwing away the baby with the bathwater (John Allsopp) and I found myself agreeing wholeheartedly with both, cognitive dissonance notwithstanding. As someone who isn't a fan of the mobile app world, I had to admit I was ignorant on the difficulties of implementing offline web apps (Andrew Betts), and blissfully unaware of the absolute zoo of devices people really do try to access the web with (Anna Debenham), webological purity be damned.

We've barely scratched the surface of what browsers can do (Paul Kinlan), we need to chase the high of writing code and actually have it Just Work (Rebecca Murphey), and above all, we need to remember where it all came from (Chris Wilson), lest we repeat the mistakes of the past. If you weren't one of the lucky people who managed to snag tickets before it sold out, take some time out of your busy day to enjoy these sessions in video once they are posted online rather than just skipping through slides.

If there's one thing that stood out though, it's how little of what I heard on and off-stage is part of the daily discourse online in the tech world, in the news or on sites like HackerNews, Twitter and Reddit. We only see caricatures of these conversations. More than ever, I'm convinced I need to filter out these echochambers from my thoughts and seek out more substance. Particularly, the Silicon Valley-centric TechCrunch-driven worship of runaway success adds nothing, and only holds us back. It makes people think they need to chase something that only ever happens by accident, and diverts attention away from rolling up your sleeves and doing what actually needs to be done. This was emphasized all the more by the fact that the venue had no wi-fi, which meant everyone had their eyes away from their screens for a change.




To seal the deal, the conference was flanked by in-depth workshops, the obvious drinks and social gatherings and even a NodeCopter hackathon, where we made quadcopters do crazy things with nothing more than JavaScript. I only wish I'd been more rested and less jetlagged so I could've spoken to more folks the past few days.

Thank you to the organizers and volunteers, to the event crew, to my fellow speakers, to the people who travelled from near and far to listen, and to whomever decided to stick those funky legs on top of the cinema. They heralded quite literally that things were about to be turned upside down, and the event certainly delivered on that.

Video and Slides




  
    The Legs
  



  
    
    
    Conference Video
    
  



  
    
    
    Slide Deck
    
  




You can read more about MathBox in the follow-up blog post.
The adventurous can go see how the sausage was made and check out the code for MathBox, the library I wrote to make it happen, as well as the HTML5 slide deck.





Introducing Facing.me
2012-04-25T00:00:00+02:00


  A unique way to meet people






  
    
  





We've been sending out whispers for a while now, but it's finally out: a new web site called Facing.me. Coded and designed by Michael Holly, Ross Howard-Jones and myself, it promises a unique way to meet people online. This would be the point where the obvious question is dropped: wait, what… you built a dating site?

Sort of. Let me explain.

Having spent many years in the web world, we'd all gotten a bit complacent. The web has settled into its comfortable rhythms. Sites and applications can be modelled quickly and coded on your framework of choice. And nowadays, Web 2.0 cred comes baked in: clean URLs, semantic HTML, AJAX, data feeds, APIs, etc. Isn't this what we all wanted?

But the web continues to evolve, and giants are roaming the playground. Sites like Facebook and Twitter hold people's attention with surgical precision, while engines like Google answer your queries with lightning speed. Given that we've all slotted such services into our workflows and indeed lives, it seems only natural that 'indie' developers should keep up. We can't pretend that a 2000-era style web-page-with-ajax-sprinkles is the pinnacle of modern interactive design.

So we set out to try something different.




  






A Guy Walks into a Bar...

If you've managed to score an invite, the first thing you'll see is the wall of faces that loads and fills the screen. The second thing you'll notice—we hope at least—is the lack of everything else.

The metaphor we kept in mind was the idea of walking into a bar, and looking around. If you see someone you like, you can go up to them and strike up a conversation. So that's exactly what the app lets you do, through video chat. You can pan around to see more people, and just keep going. If you're looking for something specific, you can filter your view with a simple "I'm looking for…" dialog.

As you mouse around, you can see who's online, and flip open their profile. If you want to strike up a video chat, it happens right there too. If the person is online, they'll see your request immediately in a popup and can choose to accept or decline after reviewing your profile. If they're offline, they'll see your request next time they visit.

To avoid missed connections, you can 'like' people you're interested in. You'll see (and hear) a notification pop up the moment they're online. You can keep the app open in a background tab and never miss a thing.

Aside from some minor social glue and a few fun little extras for you to discover, that's it. It's our twist on a minimally viable product if you will. Studies have shown that online matching algorithms are a poor predictor for how well people mesh in person. Until you meet face-to-face, you just don't know. We think direct, spontaneous video chat is a better first step rather than endless profile matching and messaging.




  

  

  

  




Polishing Bacon

But despite its minimalism, a big aspect of Facing.me is the effort and care we put into it. Our goal was to achieve a level of polish typically reserved for premium iPhone apps and bring it into the browser. We wrapped the whole thing in a crisp design, enhanced with tasteful web fonts. But most importantly, we sought to expose the app's functionality with as little interruption as possible. To do that, we layered on plenty of transitions driven by CSS3 and JavaScript, and stream in data and content as needed.

Based on previous work in custom animations—and bacon—we refined the approach of using jQuery as an animation helper for completely custom transitions. We tell jQuery to animate placeholder properties on orphaned proxy divs, and key off those animations with per-frame code to drive the fancy stuff.




  



As a result, we can have a photo grow a picture frame as you pick it up, and then flip it around to show a person's full profile. This careful choreography involves animating about a dozen CSS properties, including borders, shadows, margins and 3D transforms, all with custom expressions and hand-tuned animation curves. Similar transitions are used for lightbox dialogs.

Throughout all of this, the animations remain eminently manageable. We can interrupt and reverse them at any point, and run multiple copies at the same time, thanks to pervasive use of view controllers. Far from being a useless tech demo, it actually enables us to craft the user experience exactly the way we like it: being able to acknowledge user intentions with intuitive feedback no matter what's going on, and firing off new events and requests without worrying about the internal state. Gone are the fragile jQuery behavior soups of old.

The one downside is that only the newer browsers—i.e. Chrome, Safari and Firefox—get to see everything the way it was intended. And actually the performance in Firefox is still a bit disappointing. IE9 users will have to be satisfied with a crude 2D approximation until IE10 comes out.





Rapid Rails and Real-Time Node

To make all this work effectively on the server-side, we used a dual-mode stack of Rails and Node.js.

The Rails side houses the app's models and controllers, and provides an API for all the client-side JavaScript to do its job. Video chats are handled through Flash and routed through its built-in peer-to-peer functionality.

The node.js component acts as a real-time presence daemon which users connect to over socket.io. It's used to drive the status notifications and to coordinate the video chats. We can exchange any sort of notifications between users with a publish-subscribe model, opening up many interesting avenues for future development.

Overall, this approach has worked out great. Rails' ActiveRecord and the stack around it allowed us to build out functionality quickly and with just the right amount of necessary baggage. We made generous use of Ruby Gems to save time while still maintaining full control.

Node.js's event-driven model adds real-time signalling with no hassle. For the few cases where node.js needs to interface with the Rails database directly, we slot in some manual SQL to take care of that. For everything else, Rails and node.js exchange signed data through the browser.




  

  




Come Take it for a Spin

Finally, we also put our heads together and made a promo video, voiced by the lovely Tina Hoang:





  





Built in our spare time by just 3 guys in a virtual garage, we're pretty proud of the end result. We'd love for you to take it for a spin, so head over to facing.me and grab yourself an invite. There's a feedback form built-in, and any suggestions are welcome.

Discuss on Google Plus.




This is Your Brain on CSS
2012-02-19T00:00:00+01:00




First things first: the CSS 3D renderer used to power this the previous site is now available on GitHub.com. However, it's still limited to only solid lines and planes. It's also limited to WebKit browsers, as Firefox's CSS 3D support just isn't quite there yet.


  But CSS 3D is not a one trick pony, and as with many things, what you get out of it depends entirely on what you put in. So here's a disembodied head made out of CSS 3D. It consists of nothing more than a bunch of images stacked up against each other, and integrates perfectly with the existing 3D parallax on this site. Click and drag to rotate, or use the slider to look inside.





  
    
      
        
      
      
    
  
  



  Making the basic effect was actually quite easy. I took an MRI from the Stanford Volume Data Archive and wrote a small script to turn it into a sheet of CSS sprites. There's one file for color, one for opacity, totalling about 2.1 MB. Both files are composited into Canvases and placed in slices into the DOM, offset forward or backwards in 3D. Then there's just some minor logic to rotate the slices in 90 degree increments to follow the camera.



  But the slices are rendered as is, and the MRI consists of boring grayscale data. Luckily, I can precompute any amount of shaders and effects I want and just bake them into the slices. I geeked out by applying fake specular lighting, for that 'fresh meat' look, and volumetric obscurance to enhance the sense of depth on the inside. I changed the palette to gory colors based on local density, giving the impression of flesh and bone knitting itself together. Creepy, but cool.



  I wrapped it in a custom widget, using straight up CSS rather than Three.js this time. I've wanted to play with Tangle.js, so I used that to hook up the camera controls and slider. That's pretty much it. In an ideal world, the jarring transition when rotating would be covered up by a nice transition, but the browsers don't like it.










Making Love to WebKit
2012-01-09T00:00:00+01:00

  Parallax, GPUs and Technofetishism

  
    If the world is going to end in 2012, Acko.net will at least go out in style: I've redesigned. Those of you reading through RSS readers will want to enter through the front door in a WebKit-browser like Chrome, Safari or even an iPad.
  

  
    The last design was meant to feel spacious, the new design is spacious, thanks to generous use of CSS 3D transforms.
  

  CSS 3D vs. WebGL
  
  
    This idea started with an accidental discovery: if you put a CSS perspective on a scrollable 
, then 3D elements inside that  will retain their perspective while you scroll. This results in smooth, native parallax effects, and makes objects jump out of the page, particularly when using an analog input device with inertial scrolling.
  

  
    This raises the obvious question: how far can you take it? Of course, this only works on WebKit browsers, who currently have the only CSS 3D implementation out of beta, so it's not a viable strategy by itself yet. IE10 and Firefox will be the next browsers to offer it. There's WebGL in Chrome and Firefox that can be used to do similar things, but WebGL is its own sandbox: you can't put DOM elements in there, or use native interaction. And any amount of WebGL rendering in response to e.g. scrolling is going to involve some amount of lag. Still, I wasn't going put a lot of effort into making a CSS 3D-only design without some backup.
  

  
    That's why I actually built the whole thing on top of Three.js, mrdoob's excellent JavaScript 3D engine. Aside from providing a comprehensive standard library for 3D manipulation, it also lets you swap out the rendering component. Out of the box, it can render to a 2D canvas, a WebGL canvas, or SVG.
  




  The DOM Scenegraph
  
  
    So I augmented it with a CSS 3D renderer (GitHub). It reads out the scene and renders each object using DOM elements, shaped and transformed into the right 3D position, orientation and appearance. They sit ‘in’ the page, and the browser projects and composits them for you. Of course, this only works for simple geometric shapes like lines or rectangles, but luckily that's all I need.
  
  
  
    It would be too slow to have to render out new elements for every frame, so the CSS 3D renderer's elements persist. Moving or rotating an object involves just changing a CSS property. Same for the camera: the entire scene is wrapped in a 
 that has its own 3D transform.
  

  
    So it's VRML all over again, but this time, it actually sort of performs. With our browsers being actual 3D engines, it's not a huge leap from here to having a  tag in HTML6, can-of-worms-factor not withstanding.
  

  
    Having built a quick prototype, I was satisfied with how well it worked, particularly in Safari on OS X, where the cross-pollination from the iPhone's mature tile-based GPU renderer has clearly paid off and there is no lag at all.
  




  The DOM tree of this page. Yup, nasty.





  Previous design (Archive)
  Initial sketch
  Scene editor




  
    Design Process
  

  
    Now all that was needed was a design. Last time I drew out a manual perspective drawing in Illustrator, which was tedious, but still basically came down to designing a flat image. This time, it would have to work in 3D. I started with a quick sketch to get a feel for the perspective, now that it no longer needed to double as a flat frame for the site's content.
  
  
  
    Simple geometric shapes, parallel lines, consistent angles. Simple enough. But if real perspective was involved, I would have to place items so they would look good from multiple angles, and each would need convincing depth and shading. To do this all by hand, typing out coordinates and perpetually refreshing the page, would take forever.
  
  
  
    So instead I built a simple editor to speed up the process. It's super ghetto, and basically just exists to manipulate the colors, positions and orientations of objects in a Three.js scene. It spits out a JSON object describing them, which can then be unserialized again into a scene.
  

  
    This also helped maintain a consistent palette. The colors are built from a few base tints, brightened or darked in linear RGB—i.e. before gamma correction. This ensures even tones and allowed for easy color adjustments.
  

  
    The editor is almost entirely keyboard operated, but with its minimum amount of features I was at least able to place items in 3D, copy/paste objects and see it from any angle or position I wanted. To 'save', I just copied the output into a .JS file, where I could make manual tweaks too if necessary.
  

  
    As for the actual site and content, I wanted to keep it much more sober. Like many others these days, I want to treat blogging more like publishing. That way I can focus on crafting each post more like an article with illustrations and asides rather than just a text blog.
  
    
  
    Hence, while there's a big party upstairs, it's all typography down below. The font of choice is Klavika, a humanist/geometric sans-serif with just the right kind of “Dutch Art Museum Signage” meets “Cyberpunk” I was looking for. The layout is a responsive multi-column grid that collapses down for smaller screens and devices. Finally, a strict vertical rhythm is enforced in the lines to keep everything nice and tidy.
  
  



  Editor
  
  
    Open editor in new window
  



  Controls
  
    Click+Drag — Orbit camera

    Enter — New object
    Space — Clone object
    Backspace — Delete object
    Tab / Shift+Tab
Cycle through objects

    WASD  QE
Move object
    Shift+WASD   QE
Resize object
    Ctrl+WASD   QE
Move camera

    [] — Lower/raise units
    ZX
Orbit distance
    T/T/U
Tag/untag/untag all
  



  She cannae take the power cap'n!
  
    307 objects later it was finished, and not a single image was used. Unfortunately, as you can see there are tons of glitches in the editor—though some objects only have one side by design, and it works a lot better in a separate window. CSS 3D was never meant to do this, and you often see incorrect depth layering and flickering. Luckily most of these are caused by the floating grid markers and aren't a problem in the final view. The rest was resolved by splitting up objects or dual layering problematic surfaces, but some minor problems remain. Also for some reason, the background 
's click areas extend beyond their visible area, causing some click layering issues that I had to work around. Text resizing in the browser also leads to breakage, though multi-touch zoom works in Safari.
  
  
  
    Performance in Safari is wonderfully smooth too, but Chrome OS X starts to lag a bit. Luckily the effects are turned off as soon as they go off screen, so any lag should be confined to the top of the page. Finally, there's also a random bug where sometimes the page will refuse to scroll if the mouse is over a 3D object, which is unfortunate, but also near-impossible to reproduce reliably.
  
  
  
    In theory the iPad would perform second, but it has its own issues. The use of page-in-page scrolling disables inertia, but this is entirely beyond my control. The other issue is that sometimes, the iPad will decide to render the page content at lower resolution, making it hard to read. I guess the CSS wizardry confuses its GPU texture management. A refresh usually fixes this.
  
  
  
    I also discovered some funny ways of abusing CSS 3D for weird effects. If you have a WebKit browser, scroll to the top and enter the Konami code for an impressionistic version of the same thing.
  
    
  
    I guess I'm now the proud owner of the first unofficial CSS 3D ‘ACID’ test. I'm eager to see how the next browser handles it. If it ends up being a silly idea in the long run, I can always just switch the output to WebGL, but for now I'm willing to run with it. I put in a universal CSS 3D detector and prefixes for all the major browsers.
  
  
  
    For non-CSS 3D browsers, I simply rendered the header into a static image. It's not as fun without the shifting perspective, but it adds its own kind of optical illusion as you scroll down.
  
  
  
    Putting it all together
  
  
  
    To power the site, I got rid of Drupal and replaced it with the nimble Jekyll. Hat tip to James Walker, who did the same thing just a few days earlier and put all the code on GitHub to learn from.
  
    
  
    I've been really impressed with Jekyll's simple workflow, and though it's all static HTML, it's a refreshing change of pace. And thanks to client-side JS, it doesn't preclude adding interactive elements at all. I can treat my site as just a database of documents retrievable over HTTP, and wrap the logic around that.
  
  
  
    So I created a nice client-side navigator that transitions between pages, using 2D transforms, which also work on Firefox. It uses the HTML5 pushState API and replaces regular links with AJAX requests. Aside from being a faster way to navigate around, it also lets me link up multiple articles in a series elegantly. When you go back to a previous screen, it literally presses the browser's back button, thus avoiding creating a long, useless history trail. You go back exactly the way you came, scrolling back to where you were, just like the real back/forward buttons do. For example, click over to my Making Worlds series of posts. You can come back right away.
  
  
  
    I didn't use any libraries or router frameworks for this, simply because I wanted to have done it all myself at least once. As it now says on my About page, quoting Feynman: "What I cannot create, I do not understand". The only way to grok the intricacies of something like browser history state, which we all use every day, is to dive in and replicate it. Otherwise, you'll just take carefully choreographed behavior for granted and your mental model will be incomplete.
  

  
    To keep code size down, I compiled a custom build of Three.js with only the parts I need. I also used YUI compressor to minify the CSS and JS. However, I don't mean to obfuscate the code: the important bits will make their way onto Github soon enough.
  
  
  Update: The CSS 3D renderer and editor are now available on GitHub.
  
  
    And Done?
  

  
    I migrated over most of the content and did some house cleaning while I was at it. Most things should be back, but further fixes will be made. I also haven't implemented any commenting solution so far, but I'll be adding it back somehow as soon as I figure something out. In the mean time, there's a Google Plus thread.
  

  
    The final result looks like something that would perhaps once unironically be labeled The Information Superhighway in a magazine from the 90s, though with less neon green. I like it.
  




On TermKit
2011-05-17T00:00:00+02:00

  
I've been administering Unix machines for many years now, and frankly, it kinda sucks. It makes me wonder, when sitting in front of a crisp, 2.3 million pixel display (i.e. a laptop) why I'm telling those pixels to draw me a computer terminal from the 80s.







And yet, that's what us tech nerds do every day. The default Unix toolchain, marked in time by the 1970 epoch, operates in a world where data is either binary or text, and text is displayed in monospace chunks. The interaction is strictly limited to a linear flow of keystrokes, always directed at only one process. And that process is capable of communicating only in short little grunts of text, perhaps coalescing into a cutesy little ASCII art imitation of things that grown-ups call "dialogs", "progress bars", "tables" and "graphs".



The Unix philosophy talks about software as a toolset, about tiny programs that can be composed seamlessly. The principles are sound, and have indeed stood the test of time. But they were implemented in a time when computing resources were orders of magnitude smaller, and computer interaction was undiscovered country.



In the meantime, we've gotten a lot better at displaying information. We've also learned a lot of lessons through the web about data interchange, network transparency, API design, and more. We know better how small tweaks in an implementation can make a world of difference in usability.



And yet the world of Unix is rife with jargon, invisible processes, traps and legacy bits. Every new adept has to pass a constant trial by fire, of not destroying their system at every opportunity it gives them.



So while I agree that having a flexible toolbox is great, in my opinion, those pieces could be built a lot better. I don't want the computer equivalent of a screwdriver and a hammer, I want a tricorder and a laser saw. TermKit is my attempt at making these better tools and addresses a couple of major pain points.



I see TermKit as an extension of what Apple did with OS X, in particular the system tools like Disk Utility and Activity Monitor. Tech stuff doesn't have to look like it comes from the Matrix.


Rich Display


It's 2011, and monospace text just doesn't cut it anymore. In the default ANSI color palette, barely any of the possible color combinations are even readable. We can't display graphs, mathematical formulas, tables, etc. We can't use the principles of modern typography to lay out information in a readable, balanced way.







So instead, I opted for a front-end built in WebKit. Programs can display anything that a browser can, including HTML5 media. The output is built out of generic widgets (lists, tables, images, files, progress bars, etc.). The goal is to offer a rich enough set for the common data types of Unix, extensible with plug-ins. The back-end streams display output to the front-end, as a series of objects and commands.



I should stress that despite WebKit it is not my intent to make HTML the lingua franca of Unix. The front-end is merely implemented in it, as it makes it instantly accessible to anyone with HTML/CSS knowledge.


Pipes


  Unix pipes are anonymous binary streams, and each process comes with at least three: Standard In, Standard Out and Standard Error. This corresponds to the typical Input > Processing > Output model, with an additional error channel. However, in actual usage, there are two very different scenarios.







One is the case of interactive usage: a human watches the program output (from Std Out) on a display, and types keystrokes to interact with it (into Std In). Another case is the data processing job: a program accepts a data stream in a particular format on Std In, and immediately outputs a related data stream on Std Out. These two can be mixed, in that a chain of piped commands can have a human at either end, though usually this implies non-interactive operation.



These two cases are shoehorned into the same pipes, but happen quite differently. Human input is spontaneous, sporadic and error prone. Data input is strictly formatted and continuous. Human output is ambiguous, spaced out and wordy. Data output is conservative and monolithic.



As a result, many Unix programs have to be careful about data. For example, many tools dynamically detect whether they are running in interactive mode, and adjust their output to be more human-friendly or computer-friendly. Other tools come with flags to request the input/output in specific formats.



This has lead to "somewhat parseable text" being the default interchange format of choice. This seems like an okay choice, until you start to factor in the biggest lesson learned on the web: there is no such thing as plain text. Text is messy. Text-based formats lie at the basis of every SQL injection, XSS exploit and encoding error. And it's in text-parsing code where you'll likely find buffer overflows.



What this means in practice is that in every context, there are some forbidden characters, either by convention or by spec. For example, no Unicode or spaces in filenames. In theory, it's perfectly fine, but in practice, there's at least one shell script on your system that would blow up if you tried. Despite the promise of text as the universal interchange format, we've been forced to impose tons of vague limits.



So how do we fix this? By separating the "data" part from the "human" part. Then we can use messy text for humans, and pure data for the machines. Enter "Data In/Out", "View In/Out".







The data pipes correspond to the classical Std pipes, with one difference: the stream is prefixed with MIME-like headers (Content-Type, Content-Length, etc). Of these, only the 'Content-Type' is required. It allows programs to know what kind of input they're receiving, and handle it graciously without sniffing. Aside from that, the data on the pipe is a raw binary stream.



The view pipes carry the display output and interaction to the front-end. Widgets and UI commands are streamed back and forth as JSON messages over the view pipes.



The real magic happens when these two are combined. The last dangling Std Out pipe of any command chain needs to go into the Terminal, to be displayed as output. But the data coming out of Data Out is not necessarily human-friendly.



Thanks to the MIME-types, we can solve this universally. TermKit contains a library of output formatters which each handle a certain type of content (text, code, images, ...). It selects the right formatter based on the Content-Type, which then generates a stream of view updates. These go over the View Out pipe and are added to the command output.







As a result, you can cat a PNG and have it just work. TermKit cat doesn't know how to process PNGs or HTML—it only guesses the MIME type based on the filename and pipes the raw data to the next process. Then the formatter sends the image to the front-end. If you cat a source code file, it gets printed with line numbers and syntax highlighting.



So where does "somewhat parseable text" fit in? It turns out to be mostly unnecessary. Commands like ls output structured data by nature, i.e. a listing of files from one or more locations. It makes sense to pipe around this data in machine-form. Output flags like ls -l become mere hints for the final display, which can toggle on-the-fly between compact and full listing.



In TermKit's case, JSON is the interchange format of choice. The Content-Type for file listings is application/json; schema=termkit.files. The schema acts as a marker to select the right output plug-in. In this case, we want the file formatter rather than the generic raw JSON formatter.







Isn't JSON data harder to work with than lines of text? Only in some ways, but parsing JSON is trivial these days in any language. Because of this, I built TermKit grep so it supports grepping JSON data recursively. This happens transparently when the input is application/json instead of text/plain. As a result ls | grep works as you'd expect it to.



To slot in traditional Unix utilities in this model, we can pipe their data as application/octet-stream to start with, and enhance specific applications with type hints and wrapper scripts.







Finally, having type annotations on pipes opens up another opportunity: it allows us to pipe in HTTP GET / POST requests almost transparently. Getting a URL becomes no different from catting a file, and both can have fancy progress bars, even when inside a pipe chain like get | grep.


Synchronous interaction


All interaction in a traditional terminal is synchronous. Only one process is interactive at a time, and each keystroke must be processed by the remote shell before it is displayed. This leads to an obvious daily frustration: SSH keystroke lag.



To fix this, TermKit is built out of a separate front-end and back-end. The front-end can run locally, controlling a back-end on a remote machine. The connection can be tunneled over SSH for security.



Architecture diagram (TK stands for TermKit)



Additionally, all display updates and queries are asynchronous. The WebKit-based HTML display is split up into component views, and the view pipes of each subprocess are routed to their own view. Vice-versa, any interactive widgets inside a view can send callback messages back to their origin process, as long as it's still running.



This also allows background processes to work without overflowing the command prompt.


String-based command line


A lot of my frustration comes from bash's arcane syntax. It has a particularly nasty variant of C-style escaping. Just go ahead and try to match a regular expression involving both types of quotes.



But at its core, a bash command is a series of tokens. Some tokens are single words, some are flags, some are quoted strings, some are modifiers (like | and >). It makes sense for the input to reflect this.







TermKit's input revolves around tokenfield.js, a new snappy widget with plenty of tricks. It can do auto-quoting, inline autocomplete, icon badges, and more. It avoids the escaping issue altogether, by always processing the command as tokens rather than text. Keys that trigger special behaviors (like a quote) can be pressed again to undo the behavior and just type one character.



The behaviors are encoded in a series of objects and regexp-based triggers, which transform and split tokens as they are typed. That means it's extensible too.


Usability


At the end of the day, Unix just has bad usability. It tricks us with unnecessary abbreviations, inconsistent arguments (-r vs -R) and nitpicky syntax. Additionally, Unix has a habit of giving you raw data, but not telling you useful facts, e.g. 'r-xr-xr-x' instead of "You can't touch this" (ba-dum tsshh).



One of the Unix principles is nobly called "Least Surprise", but in practice, from having observed new Unix users, I think it often becomes "Maximum Confusion". We should be more pro-active in nudging our users in the right direction, and our tools should be designed for maximum discoverability.



For example, I want to see the relevant part of a man page in a tooltip when I'm typing argument switches. I'd love for dangerous flags to be highlighted in red. I'd love to see regexp hints of possible patterns inline.



There's tons to be done here, but we can't do anything without modern UI abilities.


Focus and Status


With a project like TermKit, it's easy to look at the shiny exterior and think "meh", or that I'm just doing things differently for difference's sake. But to me, the real action is under the hood. With a couple of tweaks and some uncompromising spring cleaning, we can get Unix to do a lot more for us.



The current version of TermKit is just a rough alpha, and what it does is in many ways just parlour tricks compared to what it could be doing in a few months. The architecture definitely supports it.



I've worked on TermKit off and on for about a year now, so I'd love to hear feedback and ideas. Please go check out the code.



TermKit owes its existence to Node.js, Socket.IO, jQuery and WebKit. Thanks to everyone who has contributed to those projects.



Edit, a couple of quick points:



A Linux port will definitely happen, since it's built out of WebKit and Node.js. Whoever does it first gets a cookie.
TermKit is not tied to JSON except in its own internal communication channels. TermKit Pipes can be in any format, and old-school plain-text still works. JSON just happens to be very handy and very lightweight.
The current output is just a proof of concept and lacks many planned usability enhancements. There are mockups on github.
If you're going to tell me I'm stupid, please read all the other 100 comments doing so first, so we can keep this short for everyone else.



Edit, random fun:


Someone asked for AVS instead of TermKit in the comments... best I could do was JS1K with a PDF surprise:








My JS1K Demo - The Making Of
2010-08-06T00:00:00+02:00

  
If you haven't seen it yet, check out the JS1K demo contest. The goal is to do something neat in 1 kilobyte of JavaScript code.
I couldn't resist making one myself, so I pulled out my bag of tricks from my Winamp music visualization days and started coding. I'm really happy with how it turned out. And no, it won't work in Internet Explorer 8 or less.

Edit: OH SNAP! I just rewrote the demo to include volumetric light beams and still fit in 1K:





Original Version






Improved Version








Now, whenever size is an issue, the best way to make a small program is to generate all data on the fly, i.e. procedurally. This saves valuable storage space. While this might seem like a black art, often it just comes down to clever use of (high school) math. And as is often the case, the best tricks are also the simplest, as they use the least amount of code.
To illustrate this, I'm going to break down my demo and show you all the major pieces and shortcuts used. Unlike the actual 1K demo, the code snippets here will feature legible spacing and descriptive variable names.

Initialization

JS1K's rules give you a Canvas tag to work with, so the first piece of code initializes it and makes it fill the window.
From then on, it just renders frames of the demo. There are four major parts to this:

Animating the wires
Rotating and projecting the wires into the camera view
Coloring the wires
Animating the camera

All of this is done 30 times per second, using a normal setInterval timer:
setInterval(function () { ... }, 33);

Drawing Wires

The most obvious trick is that everything in the demo is drawn using only a single primitive: a line segment of varying color and stroke width. This allows the whole drawing process to be streamlined into two tight, nested loops. Each inner iteration draws a new line segment from where the previous one ended, while the outer iteration loops over the different wires.



The lines are blended additively, using the built-in 'lighten' mode, which means they can be drawn in any order. This avoids having to manually sort them back-to-front.
To simplify the perspective transformations, I use a coordinate system that places the point (0, 0) in the center of the canvas and ranges from -1 to 1 in both coordinates. This is a compact and convenient way of dealing with varying window sizes, without using up a lot of code:
with (graphics) {
  ratio = width / height;
  globalCompositeOperation = 'lighter';
  scale(width / 2 / ratio, height / 2);
  translate(ratio, 1);
  lineWidthFactor = 45 / height;
  ...
I add a correction ratio for non-square windows and calculate a reference line width lineWidthFactor for later. Here, I'm using the with construct to save valuable code space, though its use is generally discouraged. 
Then there's the two nested for loops: one iterating over the wires, and one iterating over the individual points along each wire. In pseudo-code they look like:
For (12 wires => wireIndex) {
  Begin new wire
  For (45 points along each wire => pointIndex) {
    Calculate path of point on a sphere: (x,y,z)
    Extrude outwards in swooshes: (x,y,z)
    Translate and Rotate into camera view: (x,y,z)
    Project to 2D: (x,y)
    Calculate color, width and luminance of this line: (r,g,b) (w,l)
    If (this point is in front of the camera) {
      If (the last point was visible) {
        Draw line segment from last point to (x,y)
      }
    }
    else {
      Mark this point as invisible
    }
    Mark beginning of new line segment at (x,y)
  }
}

Mathbending

To generate the wires, I start with a formula which generates a sinuous path on a sphere, using latitude/longitude. This controls the tip of each wire and looks like:
offset = time - pointIndex * 0.03 - wireIndex * 3;
longitude = cos(offset + sin(offset * 0.31)) * 2
          + sin(offset * 0.83) * 3 + offset * 0.02;
latitude = sin(offset * 0.7) - cos(3 + offset * 0.23) * 3;
This is classic procedural coding at its best: take a time-based offset and plug it into a random mish-mash of easily available functions like cosine and sine. Tweak it until it 'does the right thing'. It's a very cheap way of creating interesting, organic looking patterns.
This is more art than science, and mostly just takes practice. Any time spent with a graphical calculator will definitely pay off, as you will know better which mathematical ingredients result in which shapes or patterns. Also, there are a couple of things you can do to maximize the appeal of these formulas.
First, always include some non-linear combinations of operators, e.g. nesting the sin() inside the cos() call. Combined, they are more interesting than when one is merely overlaid on the other. In this case, it turns regular oscillations into time-varying frequencies.
Second, always scale different wave periods using prime numbers. Because primes have no factors in common, this ensures that it takes a very long time before there is a perfect repetition of all the individual periods. Mathematically, the least common multiple of the chosen periods is huge (414253 units ~ 4.8 hours). Plotting the longitude/latitude for offset = 0..600 you get:

The graph looks like a random tangled curve, with no apparent structure, which makes for motions that never seem to repeat. If however, you reduce each constant to only a single significant digit (e.g. 0.31 -> 0.3, 0.83 -> 0.8), then suddenly repetition becomes apparent:

This is because the least common multiple has dropped to 84 units ~ 3.5 seconds. Note that both formulas have the same code complexity, but radically different results. This is why all procedural coding involves some degree of creative fine tuning.
Extrusion

Given the formula for the tip of each wire, I can generate the rest of the wire by sweeping its tail behind it, delayed in time. This is why pointIndex appears as a negative in the formula for offset above. At the same time, I move the points outwards to create long tails.
I also need to convert from lat/long to regular 3D XYZ, which is done using the spherical coordinate transform:
Source: Wikipedia
distance = f.sqrt(pointIndex+.2);
x = cos(longitude) * cos(latitude) * distance;
y = sin(longitude) * cos(latitude) * distance;
z = sin(latitude) * distance;
You might notice that rather than making distance a straight up function of the length pointIndex along the wire, I applied a square root. This is another one of those procedural tricks that seems arbitrary, but actually serves an important visual purpose. This is what the square root looks like (solid curve):

The dotted curve is the square root's derivative, i.e. it indicates the slope of the solid curve. Because the slope goes down with increasing distance, this trick has the effect of slowing down the outward motion of the wires the further they get. In practice, this means the wires are more tense in the middle, and more slack on the outside. It adds just enough faux-physics to make the effect visually appealing.
Rotation and Projection

Once I have absolute 3D coordinates for a point on a wire, I have to render it from the camera's point of view. This is done by moving the origin to the camera's position (X,Y,Z), and applying two rotations: one around the vertical (yaw) and one around the horizontal (pitch). It's like spinning on a wheely chair, while tilting your head up/down.
x -= X; y -= Y; z -= Z;

x2 = x * cos(yaw) + z * sin(yaw);
y2 = y;
z2 = z * cos(yaw) - x * sin(yaw);

x3 = x2;
y3 = y2 * cos(pitch) + z2 * sin(pitch);
z3 = z2 * cos(pitch) - y2 * sin(pitch);
The camera-relative coordinates are projected in perspective by dividing by Z — the further away an object, the smaller it is. Lines with negative Z are behind the camera and shouldn't be drawn. The width of the line is also scaled proportional to distance, and the first line segment of each wire is drawn thicker, so it looks like a plug of some kind:
plug = !pointIndex;
lineWidth = lineWidthFactor * (2 + plug) / z3;
x = x3 / z3;
y = y3 / z3;

lineTo(x, y);
if (z3 > 0.1) {
  if (lastPointVisible) {
    stroke();
  }
  else {
    lastPointVisible = true;
  }
}
else {
  lastPointVisible = false;
}
beginPath();
moveTo(x, y);
There's a subtle optimization hiding in plain sight here. Because I'm drawing lines, I need both the previous and the current point's coordinates at each step. To avoid using variables for these, I place the moveTo command at the end of the loop rather than at the beginning. As a result, the previous coordinates are remembered transparently into the next iteration, and all I need to do is make sure the first call to stroke() doesn't happen.

Coloring
Each line segment also needs an appropriate coloring. Again, I used some trial and error to find a simple formula that works well. It uses a sine wave to rotate overall luminance in and out of the (Red, Green, Blue) channels in a deliberately skewed fashion, and shifts the R component slowly over time. This results in a nice varied palette that isn't overly saturated.
pulse = max(0, sin(time * 6 - pointIndex / 8) - 0.95) * 70;
luminance = round(45 - pointIndex) * (1 + plug + pulse);
strokeStyle='rgb(' + 
  round(luminance * (sin(plug + wireIndex + time * 0.15) + 1)) + ',' + 
  round(luminance * (plug + sin(wireIndex - 1) + 1)) + ',' +
  round(luminance * (plug + sin(wireIndex - 1.3) + 1)) +
  ')';
Here, pulse causes bright pulses to run across the wires. I start with a regular sine wave over the length of the wire, but truncate off everything but the last 5% of each crest to turn it into a sparse pulse train:

Camera Motion

With the main visual in place, almost all my code budget is gone, leaving very little room for the camera. I need a simple way to create consistent motion of the camera's X, Y and Z coordinates. So, I use a neat low-tech trick: repeated interpolation. It looks like this:
sample += (target - sample) * fraction
target is set to a random value. Then, every frame, sample is moved a certain fraction towards it (e.g. 0.1). This turns sample into a smoothed version of target. Technically, this is a one-pole low-pass filter.
This works even better when you apply it twice in a row, with an intermediate value being interpolated as well:
intermediate += (target - intermediate) * fraction
sample += (intermediate - sample) * fraction
A sample run with target being changed at random might look like this:

You can see that with each interpolation pass, more discontinuities get filtered out. First, jumps are turned into kinks. Then, those are smoothed out into nice bumps.
In my demo, this principle is applied separately to the camera's X, Y and Z positions. Every ~2.5 seconds a new target position is chosen:
if (frames++ > 70) {
  Xt = random() * 18 - 9;
  Yt = random() * 18 - 9;
  Zt = random() * 18 - 9;
  frames = 0;
}

function interpolate(a,b) {
  return a + (b-a) * 0.04;
}

Xi = interpolate(Xi, Xt);
Yi = interpolate(Yi, Yt);
Zi = interpolate(Zi, Zt);

X  = interpolate(X,  Xi);
Y  = interpolate(Y,  Yi);
Z  = interpolate(Z,  Zi);
The resulting path is completely smooth and feels quite dynamic.
Camera Rotation

The final piece is orienting the camera properly. The simplest solution would be to point the camera straight at the center of the object, by calculating the appropriate pitch and yaw directly off the camera's position (X,Y,Z):
yaw   = atan2(Z, -X);
pitch = atan2(Y, sqrt(X * X + Z * Z));
However, this gives the demo a very static, artificial appearance. What's better is making the camera point in the right direction, but with just enough freedom to pan around a bit.
Unfortunately, the 1K limit is unforgiving, and I don't have any space to waste on more 'magic' formulas or interpolations. So instead, I cheat by replacing the formulas above with:
yaw   = atan2(Z, -X * 2);
pitch = atan2(Y * 2, sqrt(X * X + Z * Z));
By multiplying X and Y by 2 strategically, the formula is 'wrong', but the error is limited to about 45 degrees and varies smoothly. Essentially, I gave the camera a lazy eye, and got the perfect dynamic motion with only 4 bytes extra!
Addendum

After seeing the other demos in the contest, I wasn't so sure about my entry, so I started working on a version 2. The main difference is the addition of glowy light beams around the object.
As you might suspect, I'm cheating massively here: rather than do physically correct light scattering calculations, I'm just using a 2D effect. Thankfully it comes out looking great.
Essentially, I take the rendered image, and process it in a second Canvas that is hidden. This new image is then layered on the original.
I take the image and repeatedly blend it with a zoomed out copy of itself. With every pass the number of copies doubles, and the zoom factor is squared every time. After 3 passes, the image has been smeared out into an 8 = 2³ 'tap' radial blur. I lock the zooming to the center of the 3D object. This makes the beams look like they're part of the 3D world rather than drawn on later.
For additional speed, the beam image is processed at half the resolution. As a side effect, the scaling down acts like a slight blur filter for the beams.
Unfortunately, this effect was not very compact, as it required a lot of drawing mode changes and context switches. I had no room for it  in the source code.
So, I had to squeeze out some more room in the original. First, I simplified the various formulas to the bare minimum required for interesting visuals. I replaced the camera code with a much simpler one, and started aggressively shaving off every single byte I could find. Then I got creative, and ended up recreating the secondary canvas every frame just to avoid switching back its state to the default.
Eventually, after a lot of bit twiddling, a version came out that was 1024 bytes long. I had to do a lot of unholy things to get it to fit, but I think the end result is worth it ;). 
Closing Thoughts

I've long been a fan of the demo scene, and fondly remember Second Reality in 1993 as my introduction to the genre. Since then, I've always looked at math as a tool to be mastered and wielded rather than subject matter to be absorbed.
With this blog post, I hope to inspire you to take the plunge and see where some simple formulas can take you.




JavaScript audio synthesis with HTML 5
2009-08-12T00:00:00+02:00

    



  HTML5 gives us a couple new toys to play with, such as  and  tags. On the visual side, we've already seen live green-screening with Canvas and JS, and in terms of audio there's been several JS drum machines already. But the question I was interested in was: can you use JavaScript to stream live data into these media tags?



Enter the JavaScript audio synth. It generates a handful of samples using very basic time-domain synthesis, wraps them up in a WAVE file header and embeds them in  tags using base64-encoded data URIs. Each sample is then triggered using timers to play the drum pattern. It's quite simple to do and runs fast enough in HTML5 capable browsers to be unnoticeable. Yes, it sounds tinny, but that's just because I'm too lazy to design proper filters for toys like this.

Unfortunately, while the synthesis is fast enough to run real-time, you can't actually use it for a full live audio stream, as there is no way to queue up chunks of synthesized audio for seamless playback. I tried triggering multiple  tags in parallel to address this, but that didn't work either.



My final attempt was to generate tons of periodic audio loops only a couple of ms long, and to play them back with looping turned on while altering each tag's volume in real time, hence doing a sort of additive wavetable synthesis. Unfortunately, looping is not a fully supported feature, and the only browser I found that does it (Safari) doesn't loop seamlessly at all.



All in all, my first brush with the  tag was a major disappointment. The  tag's high-level approach leads to similar limitations, but they are offset by the flexibility and power of the 




Projective Texturing with Canvas
2008-11-11T00:00:00+01:00


Update: People keep asking me to use this code, apparently unaware this has been made obsolete by CSS 3D. So it's gone now.
  
The Canvas tag's popularity is slowly increasing around the web. I've seen big sites use it for image rotation, graph plotting, reflection effects and much more.



However, Canvas is still limited to 2D: its drawing operations can only do typical vector graphics with so-called affine transformations, i.e. scaling, rotating, skewing and translation. Though there have been some efforts to try and add a 3D context to Canvas, these efforts are still experimental and only available for a minority of browsers through plug-ins.



So when my colleague Ross asked me if we could build a Cover Flow-like widget with JavaScript, my initial reaction was no... but really, that's just a cop out. All you need are textured rectangles drawn in a convincing perspective: a much simpler scenario than full blown 3D.








Perspective views are described by so-called projective transforms, which Canvas2D does not support. However, it does support arbitrary clipping masks as well as affine transforms of both entire and partial images. These can be used to do a fake projective transform: you cut up your textured surface into a bunch of smaller patches (which are almost-affine) and render each with a normal affine transform. Of course you need to place the patches just right, so as to cover any possible gaps. As long as the divisions are small enough, this looks convincingly 3D.



So some hacking later, I have a working projective transform renderer in JavaScript. The algorithm uses adaptive subdivision to maintain quality and can be tuned for detail or performance. At its core it's really just a lot of linear algebra, though I did have to add a bunch of tweaks to make it look seamless due to annoying aliasing effects.



Unfortunately Safari seems to be the only browser that can render it at an acceptable speed, so this technique is just a curiosity for now. The current code was mostly written for readability rather than performance though, so it's possible it could be optimized to a more useful state. Feel free to browse the code.



A real 3D Canvas in the browser would obviously rock, but you can still do some nifty things if you know the right tricks...


Abusing jQuery.animate for fun and profit (and bacon)
2008-09-22T00:00:00+02:00
The days of static UIs that only have jarring transitions between pages are pretty much over. With frameworks like CoreAnimation or jQuery, it's easy to add useful animations to applications and webpages. In the case of jQuery, you can easily animate any CSS property, and you get free work-arounds for browser bugs to boot. You can run multiple animations (of arbitrary duration) at the same time, queue animations and even animate complex properties like colors or clipping rectangles.





But what if you want to go beyond mere CSS? You might have a custom widget that is drawn using , whose contents are controlled by internal variables; maybe you're using 3D transformations to scale and position images on a page, and simple 2D tweening just doesn't cut it.



In that case, it would seem you are out of luck: jQuery's .animate() method can only be applied to a collection of DOM elements, and relies heavily on the browser's own semantics for processing CSS values and their units. However thanks to JavaScript's flexibility and jQuery's architecture, we can work around this, and re-use jQuery's excellent animation core for our own nefarious purposes.


Hackity hack hack


First, we need an object to store all the variables we wish to animate. We use an anonymous 
 outside of the main document, so that jQuery's DOM calls still work on it. We simply add our own properties to it:



var vars = $.extend($('
')[0], {

  foo: 1,

  bar: 2,



  customAnimate: true,

  updated: true

});





In this case, our properties are foo and bar. We also set customAnimate and updated to identify this object (see below).



Next we need to override jQuery's default step function, which gets called for every step of an animation, and applies new values to an element's CSS properties.



   // jQuery.fx.step._default

    _default: function(fx) {

      fx.elem.style[fx.prop] = fx.now + fx.unit;

    }





We can replace it using the following snippet:



var $_fx_step_default = $.fx.step._default;

$.fx.step._default = function (fx) {

  if (!fx.elem.customAnimate) return $_fx_step_default(fx);

  fx.elem[fx.prop] = fx.now;

  fx.elem.updated = true;

};





With the new step function, jQuery will check for the presence of a customAnimate property on any element it is animating. If present, it will assign the (unit-less) value to element.property rather than element.style.property and mark the element by setting element.updated to true.



Now we're ready to animate, using the normal $.animate syntax:



$(vars).animate({ foo: 5, bar: 10 }, { duration: 1000 });





The values vars.foo and vars.bar will now smoothly change over time. You can use any of jQuery's animation abilities as usual.



Now what about that updated variable? Well to actually use these animated values, you will need some kind of timer or step callback to read them back and draw them on the page. If you're using , you need to redraw your entire widget for every change, but you don't want to be wasting CPU time by constantly refreshing it. Furthermore, if you're running multiple animations at the same time, you'll want to aggregate all your property changes into a single redraw per frame. This is easy with the updated property and a simple timer:



setInterval(function () {

  if (!vars.updated) return;

  vars.updated = false;

  

  drawWidget();

}, 30);





Now your widget will only refresh itself when its values are changed by the animation step function we defined earlier, and very few CPU cycles are wasted. As a plus, you can render updates as fast or as slow as you want, without affecting the duration of your animations.


Demo


I whipped up a quick demo which renders a cloud of bacon using . All the motion in the demo is created through $.animate(), with a bunch of animations running at once.



This demo will not work in Internet Explorer, and has only been tested in Firefox 3 and Safari 3.



This is a rather esoteric example, but there are plenty of useful ways to apply this technique. I've used it to implement smooth, beautiful, usable widgets. You can combine multiple motion and opacity animations triggered by clicks and hovers without issues.

 
Final notes


While this technique works great, there is one big caveat. You should avoid animating any property that exists in CSS ('float', 'display', 'opacity', ...) because these have unexpected side effects depending on the browser.



There are also a couple of weaknesses:



  jQuery does not support continued easing. That is, when you override an animation that is already in progress, the variable being animated will instantly stop and restart from its current position. The rate of change is not continuous between the two animations.
  Animating angles is tricky. E.g. when animating from 350˚ to 0˚, you want it to animate across 10˚ and not the long way around. This requires manual correction.



And obviously, it would be cleaner if jQuery's animation core was refactored to separate out the CSS-specific code instead...





Welcome to the World of Tomorrow!
2008-07-20T00:00:00+02:00
(with apologies to Matt Groening)



After about two years, it's time for another make-over of my site.



My last design had a relatively quirky look, with a bold red/yellow theme built from various irregular vector shapes. The idea was to step away from the typical mold of rectangular aligned frames on a page. I tried to incorporate some elements of perspective into the page composition, but it ended up being a relatively flat, geometrical theme.



This time I wanted to work on the depth aspect and try to create something that feels spacious. To do this, I based the entire redesign on a two-point perspective. While the content itself is normal 2D markup, it sits in a 3D frame.









The header image is a regular illustration file (which is 100% manual vector work) and the content is typical HTML/CSS. However there is a twist: the perspective from the header is continued into the content with some simple 3D decorations, created on-demand with Canvas tags and JavaScript (highlight canvases, check out the footer).



While this perspective works perfectly near the top, the further down you go, the more vertically stretched the shapes get and it ends up looking weird. To compromise, the projection actually gets more and more isometric the further down you go. This creates an interesting effect when scrolling down.



The design also uses various CSS3 methods (@font-face, text-shadow, box-shadow) throughout, and uses sIFR 3 as a fallback for the headline font. Unfortunately CSS3 is still mostly unsupported in the browserscape, so only Safari 3.1 users get the luxury combo of pretty, fast and no Flash. Everyone else will have to suffer through hacks.



As a total surprise, the canvas-rocket-science trickery even works in IE6 thanks to Google's ExplorerCanvas library.



I'll probably be tweaking it a bit more in the days to come, but feedback is appreciated.





ComicJuice gets even better
2007-03-09T00:00:00+01:00
I finished some more tweaks to ComicJuice:




IE6 and 7 are now supported, thanks to the amazing ExplorerCanvas by Google. It emulates the  in IE, meaning that client-side scriptable vector graphics are now available on all the major browsers (IE, Firefox, Safari, Opera). I doubt Konqueror will be far behind.



This opens up some cool abilities, like dynamic in-page graphs, mini-widgets (sliders, dials, maps, ...) and even pure JS games. There's a bunch of examples linked on Wikipedia (though most don't use ExplorerCanvas yet).


I added support for uploading your own images rather than using pictures on the web. It uses a customized and themed version of core's JS uploader.



I improved the clipping of speech bubbles so there should be less useless whitespace around comics, especially when embedding them.





Announcing.... ComicJuice!
2007-03-06T00:00:00+01:00

  
Announcing.... ComicJuice!
  
I'm proud to announce the start of ComicJuice, a web 2.0 social mashup tool that lets you create comics in your browser and share them with others.



Update: Now with Internet Explorer support! Thanks to Google's ExplorerCanvas. Viewing comics works in IE6 and 7, while editing still requires IE7.







The crazy part is that I started working on this only friday evening (that's 4 days ago). Once I had the initial idea and a rough plan, I simply couldn't not code it.



A lot of jQuery and JavaScript later, with some 


You can also embed comics with iframes, and copy/pastable code is provided. Like this lame example:


(No longer available)


I figured a Web 2.0 mash-up would not be complete without a fitting design to go in, so I designed icons, sliders and toolbars for the editor, as well as a theme for the website. The theme is a Garland knock-off: I guess I'm proving myself wrong that it's a bad base theme. It's actually quite good as it has fluid/fixed 1-3 column layouts in it.



I'm curious to see if ComicJuice takes off and what people do with it. It was a blast to code in any case. Check it out.

Acko.net

Yak Shading

Data-Driven Geometry

Virtual Geometry

Vertex Party

Shader­Graph 2

Functional GLSL

Instanced Data Flow

GLSL Composer

$ cat *.glsl | magic

PowerPoint Must Die

PowerPoint Syndrome

Manifold Dreams

Geometry Streaming

(4-in-1)²

MathBox²

Part 2

I-Can't-Believe-It's-Not-React

One More Thing…

Full Stack

A DOM for Robots

Modelling Live Data

Keep it Simple

Base

Camera

Draw

Data

Operator

Overlay

Present

RTT

Shader

Text

Time

Transform

View

Model-View-Projection

It'll Do

Shadow DOM

SVG, CSS, React and Angular

Dangling Nodes

DOM Shader

Hello World

Unreasonable Behavior

Thinking with Portals

The Boxed Model

On Asm.js

Ending The Ice Age of JavaScript

LLVM to Asm

Impostor Syndrome

History Repeating

Making MathBox

Presentation-Quality Math with Three.js and WebGL

Choose your Poison

Shady Dealings

Viewports, Primitives and Renderables

Interpolate all the things!

Presenting Naturally

Towards MathBox 1.0

Going Full Frontal

Making things with Maths

Video and Slides

Introducing Facing.me

A unique way to meet people

A Guy Walks into a Bar...

Polishing Bacon

Rapid Rails and Real-Time Node

Come Take it for a Spin

This is Your Brain on CSS

Making Love to WebKit

Parallax, GPUs and Technofetishism

CSS 3D vs. WebGL

The DOM Scenegraph

Design Process

Editor

Controls

She cannae take the power cap'n!

Putting it all together

And Done?

On TermKit

ShaderGraph 2

`$ cat *.glsl | magic`