← All posts

Building a Pro-Tier Video Streaming Server: HLS, AI Subtitles, and Interactive Thumbnails

·10 min read·Read on Medium

Beyond the "Play" Button

We spend a third of our lives staring at streaming platforms, but we rarely look under the hood. Most developers think "video streaming" is just a <video> tag pointing to an .mp4 file. I thought so too, until I tried to build my own. I wanted to see if I could recreate the Netflix experience from scratch — no heavy frameworks, no paid cloud APIs, just raw engineering. I ended up building an autonomous pipeline that handles HLS segmentation, local AI inference, and complex coordinate math, all running on a bare-bones Node.js engine.

The Tech Stack

Level 1: The "Naive" Static Server

In the beginning, my goal was simple: get a video to show up in a browser. I wrote a bare-bones Node.js server using the native http and fs modules to serve a raw .mp4 file.

The Initial Implementation

My first attempt was straightforward: use fs.createReadStream to pipe the video file directly into the HTTP response. It worked perfectly on my local machine. However, the moment I tested it over a real-world connection (a 4G hotspot), the experience fell apart.

I discovered three glaring flaws:

  1. The Buffering Trap: Browsers are "greedy." They try to download the entire file as fast as possible. For a large video, the user might wait a minute just for the first frame to appear.
  2. The "No Seeking" Problem: By default, a standard stream doesn't allow users to skip ahead. If they click the 5-minute mark, nothing happens because the server doesn't know how to jump to that specific byte.
  3. Data Hunger: I was serving a high-bitrate file. There was no way to lower the quality for a user on a weak connection.

The "Failed" Fix: Manual Byte-Range Handling

To solve the seeking issue, I had to implement HTTP 206 Partial Content logic. This allows the browser to request specific "ranges" of the video rather than the whole file at once.

I used a CHUNK_SIZE of 1MB to throttle the server. This acted as a "safety valve," ensuring that my server only pulled 1MB into memory at a time before piping it to the user.

const server = http.createServer((req, res) => {
  const videoPath = './storage/video.mp4';
  const range = req.headers.range;

  // Level 1: Handling "Seeking" via Manual Byte-Ranges
  if (!range) {
    res.writeHead(200, { 'Content-Type': 'video/mp4' });
    fs.createReadStream(videoPath).pipe(res);
  } else {
    const videoSize = fs.statSync(videoPath).size;
    const CHUNK_SIZE = 10 ** 6; // 1MB chunks
    const start = Number(range.replace(/\D/g, ""));
    const end = Math.min(start + CHUNK_SIZE, videoSize - 1);
    const contentLength = (end - start) + 1;
    const headers = {
      "Content-Range": `bytes ${start}-${end}/${videoSize}`,
      "Accept-Ranges": "bytes",
      "Content-Length": contentLength,
      "Content-Type": "video/mp4",
    };
    res.writeHead(206, headers);
    fs.createReadStream(videoPath, { start, end }).pipe(res);
  }
});

I successfully implemented seeking, but I hadn't solved the data problem. I was still trying to send a 1080p 'heavy rock' when the user's connection only had room for 'pebbles.' My server was smart enough to send bytes, but it wasn't smart enough to understand the user's struggle.

The Reality Check: Micro-managing a Disaster

While the code above technically enabled seeking, it was an engineering dead end.

I realized I was trying to make a static file act like a pro-tier stream. To solve this, I needed to stop slicing the delivery and start slicing the content itself.

Level 2: The HLS Pivot (Adaptive Bitrate Magic)

The breakthrough happened when I stopped trying to make a single .mp4 smarter and moved to HLS (HTTP Live Streaming). This is the industry standard protocol that enables the same smooth, adaptive playback experiences we see on platforms like YouTube and Netflix.

Instead of serving one massive file, HLS slices the video into hundreds of tiny, 4-second segments (.ts files) and creates a master index — the .m3u8 playlist that acts as a "map" for the player.

The Transformation: Enter FFmpeg

To handle the heavy lifting, I integrated FFmpeg into my pipeline. I wrote a script to automate the conversion, ensuring that every raw upload is instantly "segmentized" for the web.

ffmpeg -i video.mp4 -c:v libx264 -c:a aac -hls_time 4 -hls_list_size 0 -f hls playlist.m3u8

Why This Changed Everything

Because the video was already segmented by FFmpeg, my Node.js server no longer needed to calculate bytes. Its only job was to act as a high-speed traffic controller, delivering the right files with the correct MIME headers so the browser knew how to play them.

// Pure Node.js HLS Dispatcher
if (req.url.startsWith("/hls/")) {
  const relativePath = req.url.replace("/hls/", "").split("?")[0];
  const filePath = path.join(hlsFolder, relativePath);

  let contentType = "application/octet-stream";
  if (filePath.endsWith(".m3u8")) contentType = "application/vnd.apple.mpegurl";
  if (filePath.endsWith(".ts"))   contentType = "video/MP2T";
  if (filePath.endsWith(".vtt"))  contentType = "text/vtt";

  res.writeHead(200, { "Content-Type": contentType });
  fs.createReadStream(filePath).pipe(res);
}

By moving the complexity from the Request (Level 1) to the Pre-processing (Level 2), I finally had a stable streaming foundation.

Level 3: The Autonomous Pipeline (AI & Multi-Bitrate)

With the infrastructure proven, I needed a way to process raw videos into a professional format automatically. I didn't want just one resolution — I wanted Adaptive Bitrate Streaming (ABR) and AI-generated accessibility.

Giving the Server "Ears" with Xenova Whisper

Instead of paying for cloud transcription, I integrated the Whisper-tiny model. Using the @xenova/transformers library, I built a pipeline that:

  1. Extracts the audio as a 16kHz WAV file.
  2. Runs local AI inference to "listen" to the speech.
  3. Formats the output into a valid .vtt subtitle file.

The Multi-Bitrate Chop

A pro-tier stream needs to be adaptive. I upgraded the video processing to create a Bitrate Ladder — generating 360p, 720p, and 1080p versions so the player can automatically switch quality based on network speed.

// Level 3: Multi-Bitrate Transcoding Logic
export const transcode = (inputFile, outputFolder, resolution, variantName, bitrate) => {
  return new Promise((resolve, reject) => {
    console.log(`Starting ${resolution} transcode...`);
    const variantFolder = path.join(outputFolder, variantName);
    ensureDir(variantFolder);

    ffmpeg(inputFile)
      .format("hls")
      .videoCodec("libx264")
      .audioCodec("aac")
      .size(resolution)
      .videoBitrate(bitrate)
      .autoPad(true, "black") // Handles vertical videos without distortion
      .outputOptions([
        "-hls_time 4",                 // 4-second segments for snappy seeking
        "-hls_list_size 0",            // Ensure all segments are listed (VOD)
        "-hls_segment_filename", path.join(variantFolder, "segment_%03d.ts"),
        "-force_key_frames expr:gte(t,n_forced*4)",
        "-sc_threshold 0",             // Disable scene-change cuts
        "-g 48",                       // GOP size: critical for synchronization
        "-keyint_min 48",
      ])
      .save(path.join(variantFolder, "playlist.m3u8"))
      .on("end", () => resolve())
      .on("error", (err) => reject(err));
  });
};

Notice the -g 48 and -sc_threshold 0 flags. These are critical — they ensure that the 360p and 720p segments "cut" at the exact same millisecond. Without this, the video would glitch when switching resolutions.

Level 4: The Frontend (The "Eyes")

The backend was a powerhouse of automation, but a pro-tier service is nothing without a polished interface.

The Hls.js Integration

Standard HTML5 <video> tags do not support .m3u8 playlists natively in most desktop browsers. Hls.js acts as a lightweight bridge, transmuxing the HLS segments into something the browser can understand on the fly.

const video = document.getElementById('video');
if (Hls.isSupported()) {
  const hls = new Hls();
  hls.loadSource('/hls/master.m3u8'); // Our Multi-Bitrate Master Playlist
  hls.attachMedia(video);
}

Giving the User the Steering Wheel: Quality Selection

While Hls.js is great at automatically choosing the best resolution, a professional player needs to give the user manual control. I tapped into the hls.levels array, which is automatically populated when the browser reads the master.m3u8 manifest.

hls.on(Hls.Events.MANIFEST_PARSED, function () {
  const selector = document.getElementById('qualitySelect');

  hls.levels.forEach((level, index) => {
    const option = document.createElement('option');
    option.value = index;
    option.text = `${level.height}p`;
    selector.appendChild(option);
  });
});

document.getElementById('qualitySelect').onchange = (e) => {
  hls.currentLevel = parseInt(e.target.value);
};

The "Netflix" Touch: Sprite Sheet Previews

The most challenging part of the UI was the scrubbing preview. Fetching hundreds of tiny images as the user hovers would kill server performance and cause laggy previews. Instead, I used FFmpeg's complex "tile" filter to extract frames and stitch them into a single 16-column "Mega-Image" sprite sheet in one pass — reducing network overhead from 150 requests to just one.

To synchronize timestamps to sprite coordinates, I wrote a Metadata VTT file where each entry contains a Media Fragment URI pointing to the exact xywh crop position in the sprite:

// Level 4: Mapping Time to Space
for (let i = 0; i < timestamps.length; i++) {
  const start = timestamps[i];
  const end = timestamps[i + 1] || videoDuration;
  const col = i % 16;
  const row = Math.floor(i / 16);
  const x = col * 160;
  const y = row * 90;
  vttContent += `${start} --> ${end}\nthumbnails.jpg#xywh=${x},${y},160,90\n\n`;
}

As the user hovers over the seek bar, the JS calculates the current "tile" and slides the background of a preview window — with zero extra network requests.

Final Thoughts: The Engineering Behind the Play Button

When I started at Level 1, I thought streaming was just about moving a file from Point A to Point B. I quickly learned that "simple" is often the enemy of "scalable."

We often take that "Play" button for granted. But behind every smooth seek, every auto-generated subtitle, and every instant quality switch, there is a mountain of hidden logic. By building this from scratch, I didn't just build a video player — I built a system that respects the user's bandwidth, the server's memory, and the reality of the modern web.

The full source code is on GitHub.