Deep learning networks
using WebGL

University of Pennsylvania

Xavier Bourry

Deep learning using WebGL

  • What is a deep learning network,
  • Why WebGL, and how to implement,
  • Optimize WebGL GPGPU,
  • Hardware difficulties.

Deep learning network

Crédit : Oolution Technologies
Entrée →
→ Sortie

Why WebGL for this ?

Cons of GPU processing

  • Feedforward or backpropagation → sequential processing by layer,
  • Layer processing → per neuron parallelization,
  • image/video as input → many inputs ! And already on the GPU,
  • layer parameter = texture (synaptic weights, connectivity, input summed, ...),
  • Already used in production (Tesla, GPU server side, ...).

Why WebGL in particular

  • Independant of the hardware, run in the browser, as mobile app, without installing,
  • WebCL seems to be dead,
  • Compute shaders might be implemented in WebGL<X>, X>2,
  • Server processing → communication latency, not scalable,
  • Webassembly/JS improvements : will never catch up because of misfit between NN architecture and CPU processing.

Shaders use

    Vertex shader
    → draw 2 triangles
    Fragment shader
    → pixel per pixel operations

Optimize WebGL GPGPU

All on the GPU

What happens in GPU stays in GPU.
  • Avoid gl.readPixels(), canvas.toDataURL()
  • There is always a solution to not use the CPU,
  • Synchronization CPU/GPU → can be slooooooow...

Read FLOAT/HALF FLOAT TEXTURES

  • No GL.readPixels() with FLOAT/HALF FLOAT textures,
  • → gl_FragColor encodes 1 float in its RGBA channels,
  • → 4 renderings, for RGBA values.

At the training

  • Dataset as texture atlas
  • Output data :
    JSON → Float32Array
    → Floating texture,
  • Load everything on the GPU once and for all.

The context

  • GL = canvas.getContext("webgl",
    {antialias: false,
    preserveDrawingBuffer: true});
  • GL.disable(GL.DEPTH_TEST);
    GL.disable(GL.SCISSOR_TEST);
    GL.disable(GL.DITHER);

Floats precision

  • GL.FLOAT → 32 bits
  • GL.HALF_FLOAT → 16 bits, enough for deep learning,
  • GL.UNSIGNED_BYTE → 8 bits, not enough,
  • Empirical result : at least 11 bits for deep learning

Floating point specials

  • highp float → 32 bits

  • → Fmax = 3.402823 × 1038

Floating point specials

Activation function : sigmoid
y=1/(1+exp(-x));
If  x = -100,
y ≈ 3.7*10-44 ≈ 0

But
exp(-x) = exp(100)
        ≈ 2.7*1043
        >> Fmax !!!

→outputs a floating point special, +Inf

Avoid this

  • avoid functions using exp(),
  • majorate,
  • Use L1 and L2 regularization (w←d*w with d<1)

Hardware difficulties

GPGPU required capabilities

Required :

  • Using float or half float textures,
  • Do render to texture with them,

Always better :

  • Use mipmap on float/half float textures.

WebGL 1 limitations

  • Requires OES_TEXTURE_FLOAT or OES_TEXTURE_HALF_FLOAT extension,
  • Better to have OES_TEXTURE_FLOAT_LINEAR or OES_TEXTURE_HALF_FLOAT_LINEAR,
  • Also ask for EXT_color_buffer_float and EXT_color_buffer_half_float,
  • Test if you can mipmap on float/half float textures,
  • Test if you can do Render to Texture on float/half float textures.

WebGL 2 limitations

  • Float textures and half float texture are included !
  • Render to texture with half float texture is included !
  • →you can always do Deep Learning with WebGL2 !!!
  • Still requires to enable OES_TEXTURE_FLOAT_LINEAR or OES_TEXTURE_HALF_FLOAT_LINEAR for using mipmapping on float/half float textures.

HALF_FLOAT texture init

OES_TEXTURE_HALF_FLOAT → you can use half float texture.

Problem: initialization from an array : there is no JS Float16Array() type.

  • JS float array → JS Uint16Array() using binary float format,
  • If not working (ipad) : create a floating point texture, fill it from a JS Float32Array(), then copy it to the half float texture.

References

Questions ?


xavier@jeeliz.com

Career advices

Main experience : Freelance developer/trainer. Advantages :

  • You see many different environments, different people and needs,
  • You choose your projects,
  • You manage your time.

Bootstrap

  • Start with an easy technology, required everywhere,
  • First use platforms like Linkedin or Hopwork/Malt,
  • Deal with a good accountant. Paperwork can kill you !
  • Do free content with a real added value :
    • tutorials,
    • demonstrations & open-source code,
    • events,
    • blog posts,
  • Report all your time & be honest. You will got cheated, and you will learn from it.

Manage your project portfolio

  • Training sessions & consulting,
  • Time paid jobs in large companies,
  • Task paid jobs in small companies,
  • Your own projects.

A good position to startup

  • You earn money so you can start progressively,
  • You see people needs in your field,
  • You will meet you next cofounder,
  • And your next clients !

Questions ?


xavier@jeeliz.com