Why Real-time Summarization Matters
Imagine watching a two‑hour webinar and needing a five‑minute recap within seconds. In 2023, 68% of professionals reported that they skim video content because they lack quick summaries. Delivering a concise overview directly in the browser eliminates the latency of server‑side processing and protects user privacy.
Core Technologies: JavaScript, WebGPU, Whisper
JavaScript provides the glue that runs in every modern browser. WebGPU, the successor to WebGL, exposes low‑level compute shaders that can accelerate neural inference on the GPU without native extensions. OpenAI’s Whisper model, originally released in 2022, offers state‑of‑the‑art speech‑to‑text conversion and can be fine‑tuned for summarization pipelines. Combining these three creates a fully on‑device AI pipeline that runs in real time.
Preparing the Browser Environment
First, verify that the target browser supports WebGPU. As of Chrome 113 and Edge 113, the feature is stable behind the "gpu" flag. The following check throws a clear error if the API is unavailable:
if (!('gpu' in navigator)) { throw new Error('WebGPU not supported in this browser'); }Next, allocate a canvas element that will host the video and a hidden offscreen canvas for frame extraction. The video element must have the "playsinline" attribute to avoid fullscreen restrictions on mobile devices.
Loading Whisper with WebGPU
Whisper can be compiled to WebAssembly with a WebGPU backend using the whisper.cpp project. After building the WASM module, load it asynchronously:
const response = await fetch('whisper.wasm'); const bytes = await response.arrayBuffer(); const wasmModule = await WebAssembly.instantiate(bytes, { env: { /* GPU bindings */ } }); const whisper = new wasmModule.instance.exports.Whisper();The env object must expose gpuDevice and memory buffers. A minimal wrapper looks like this:
async function initWhisper() { const adapter = await navigator.gpu.requestAdapter(); const device = await adapter.requestDevice(); const env = { gpuDevice: device, memory: new WebAssembly.Memory({initial:256, maximum:512}) }; const {instance} = await WebAssembly.instantiateStreaming(fetch('whisper.wasm'), {env}); return new instance.exports.Whisper(); }Processing Video Frames
Extracting audio directly from the video element is more efficient than decoding the entire stream. The Web Audio API can pipe the MediaElementAudioSourceNode into a ScriptProcessorNode that buffers 30 ms chunks. Each chunk is then fed to Whisper:
const audioCtx = new AudioContext(); const source = audioCtx.createMediaElementSource(video); const processor = audioCtx.createScriptProcessor(4096, 1, 1); source.connect(processor).connect(audioCtx.destination); processor.onaudioprocess = e => { const samples = e.inputBuffer.getChannelData(0); whisper.processAudio(samples); };While audio drives transcription, visual cues improve summarization quality. Use an offscreen canvas to capture a frame every second:
const offscreen = new OffscreenCanvas(video.videoWidth, video.videoHeight); const ctx = offscreen.getContext('2d'); setInterval(() => { ctx.drawImage(video, 0, 0); const imageData = ctx.getImageData(0,0,offscreen.width,offscreen.height); // optional: pass to a vision model }Generating the Summary
Whisper returns timestamps and transcribed text. A lightweight JavaScript summarizer can apply extractive techniques such as TextRank. For a demo, the following function selects the three highest‑scoring sentences based on word frequency:
function summarize(transcript) { const sentences = transcript.match(/[^.!?]+[.!?]/g) || []; const wordFreq = {}; transcript.toLowerCase().replace(/[^a-z\s]/g,'').split(/\s+/).forEach(w=>{if(w){wordFreq[w]=(wordFreq[w]||0)+1;}}); const scores = sentences.map(s=>{ const words=s.toLowerCase().match(/\b\w+\b/g)||[]; return words.reduce((sum,w)=>sum+ (wordFreq[w]||0),0);}); const topIdx = scores.map((s,i)=>[s,i]).sort((a,b)=>b[0]-a[0]).slice(0,3).map(p=>p[1]); return topIdx.map(i=>sentences[i].trim()).join(' '); }The final summary appears instantly below the video player, giving users a readable paragraph that reflects the most important spoken content.
Performance Tips and Limitations
Real‑time inference on a mid‑range laptop GPU (e.g., Intel Iris Xe) typically processes 30 ms of audio in 12 ms, leaving headroom for UI updates. To stay within the 60 fps budget, limit visual frame extraction to one per second and reuse the same GPU buffers. Memory consumption peaks at ~150 MB for the base Whisper model; consider the tiny variant (≈40 MB) for mobile browsers.
Because the entire pipeline runs client‑side, users retain control over their media. However, browsers impose a 2‑minute limit on continuous WebGPU compute dispatches for power‑saving reasons. Splitting the audio stream into short batches circumvents this restriction.
Conclusion
By harnessing JavaScript, WebGPU, and the Whisper model, developers can deliver instant video summarization without sending raw media to a server. The approach respects privacy, reduces latency, and scales with the user’s hardware. Implement the steps above, experiment with different Whisper sizes, and watch your web app turn hours of video into bite‑size knowledge in seconds.
Sources
- WebGPU Specification – w3.org
- Whisper.cpp GitHub Repository – ggerganov/whisper.cpp
- MDN Web Docs – Web Audio API
Author: Mahmut Sarıkaya — sarikayadev.com
