The emerging HTML5 specification lifts video playback out of the generic <object> element and into specialized <video> handlers. Explicit markup for audio and video places elevates moving pictures to a similar native rendering capacity as <img> markup we are used to but with more fine-grained details about underlying formats and compression available before loading. In this post I will dive into implementation details of HTML5 video based on currently available consuming agents and outline some of the nuances of preparing media for playback.

1. Inside the video element
   1. Browser workflow
   2. JavaScript-based workflow
2. Implementation nuances
3. Player UIs
4. HTML5 video and Flash
5. Summary

Inside the video element

The video element is the top-level element of a cascading element set designed to handle graceful degradation across a wide array of HTML rendering engines. If a web browser or other consuming agent unpacks the DOM and does not understand what you have described it should process child elements until something makes sense or it reaches the end of your element tree.

Look complicated? It is! The static markup above describes six possible video interactions with the web browser. Three different source videos are described in HTML5 markup: a MP4 file container with a H.264 video track using the High profile Level 3 and low-complexity AAC audio (suitable for desktops); a MP4 file container with a H.264 video track using the Baseline profile and low-complexity AAC audio (suitable for mobile phones); an Ogg file container with a Theora video track and a Vorbis audio track. I will dive deeper into file format and codec nuances in a separate post. If the HTML5 video markup fails, or none of the three specified source videos are compatible with the consuming agent the markup falls back to double-baked markup for the Flash Player plugin. If HTML5 video fails and Flash embedding fails the markup includes simple information about the video and an animated GIF preview.

Behind the scenes the web browser is converting your markup string into its own set of mapped elements, passing off to the appropriate handler, and adjusting page layout based on its new discoveries.

Browser workflow

1. Read the markup string.
2. Build an element tree.
3. Find a <video> element.
4. I know how to process a <video> element. Map defined attributes.
   1. Found width and height attributes. Prepare the page layout for new content.
   2. The controls attribute is present and I know how to process the attribute. The publisher would like to use the default playback UI built-in to my video handler.
   3. Found a src attribute. Try to load the referenced resource. Similar handling to an <img> src.
   4. Found an autobuffer attribute and I know how to process the attribute. Start buffering the movie resource before the viewer initiates playback.
   5. Found a poster attribute and I know how to process such an attribute. The publisher would like to show a poster frame image inside the video object dimensions before the viewer initiates playback.
   6. The src attribute is either undefined, unavailable, or incompatible. Continue parsing child elements for a better content match.
      1. Found a source element and I know how to process such an element.
         1. The type attribute value references an Internet media type I recognize and support for Internet video. It’s possible I might be able to read the file format and unpack the video container after download.
            1. A codecs parameter is specified within the type attribute, defining the video codec and audio codec needed to decode the container’s video and audio tracks respectively.
         2. The src attribute exists. Queue the referenced resource for network loading after the viewer initiates playback, or immediately if autobuffer was specified in the video element.
      2. No suitable source element found. Continue searching.
5. Found an <object> element with an object handler specified using the classid attribute. My name is most likely Trident/IE.
   1. The classid attribute value matches a plugin installed on the viewer’s computer: Adobe Flash Player.
   2. The version of Flash Player currently installed on the viewer’s computer is less than the minimum specified value in the codebase attribute.
      1. Attempt to download and install a new Flash Player ActiveX control at or above version 9.0.115 “MovieStar.” The specified Flash Player version is capable of handling a MP4 video container with an H.264 video track and AAC audio track.
      2. Stop processing the video object; reload later.
   3. A <param> element exists with a name attribute of movie and a resource location declared in the value attribute.
   4. Display text specified in the object’s standby attribute value while I attempt to load the Adobe Flash browser plugin and its SWF file interpreter. Pass the specified param element key-value pairs into the Flash interpreter as well as the FlashVars query parameter describing dynamic values interpreted by the SWF at runtime.
6. I don’t care about conditional comment blocks targeting Trident/IE or such a conditional evaluates as true.
7. Found an object element with an object handler specified using the type attribute.
   1. The type attribute specifies an Internet media type connected to a known plugin registered in the plugin system (most likely NPAPI).
   2. The data attribute exists and specifies a valid resource.
   3. Display text specified in the object’s standby attribute value while I attempt to load the Adobe Flash browser plugin and its SWF file interpreter. Pass the specified param element key-value pairs into the Flash interpreter as well as the FlashVars query parameter describing dynamic values interpreted by the SWF at runtime.
8. No acceptable video player found. Display an animated GIF preview of the movie. Let the viewer know they are missing out on the full content experience.
9. Acceptable movie found and queued.
   1. Attempt to progressively download or stream the specified video element.
      1. Does the Content-Type returned by the server match our expected value(s)?
      2. Does the server accept downloading individual pieces of a file at a time (Accept-Ranges)?
      3. Did the resource return a X-Content-Duration header specifying expected playback length in seconds?
   2. Send downloaded video pieces to the video decoder for decompression.
   3. Initiate a playback buffer.
   4. Fire events related to the final loaded stage of the process.

Yes, I have over simplified.

JavaScript-based workflow

It is possible to test playback capabilities of the browser and its related plugins through JavaScript (if JavaScript is available on the page of course). If you are considering supporting HTML5 video at some point in the future but are curious how many of your visitors could support the new playback method you could track analytic events today to influence your product roll-out months down the road.

Video element support

Test the current consuming agent’s support for the <video> element by declaring a new DOM object and evaluating the browser’s default handlers. If the created DOM object contains functions present in a default HTMLVideoElement or HTMLMediaElement interface we know the consuming agent applied special handling to our video element declaration and likely supports HTML5 video.

Individual codec support

Testing support for the video element is only the first step. We also need to check playback support for the specific video and audio codecs used in our source videos. The canPlayType method returns the likelihood a given file container, video codec and audio codec are supported by the consuming agent.

Detect Flash

Flash Player 9.0.115 and above is required to play MP4 file containers with H.264 video and AAC audio. The Flash Player detection kit provides client-side detection libraries and automatic upgrade capability for site visitors not already using the latest version of Flash.

Check for an ActiveXObject of ShockwaveFlash.ShockwaveFlash.10 or ShockwaveFlash.ShockwaveFlash.9 and compare the full version string.

In a NPAPI plugin environment check the navigator.mimeTypes array for the key “application/x-shockwave-flash,” verify the associated plugin is enabled, and parse the version number from the plugin’s description string.

DOM insertion

Once your script has determined the best available video playback method you can insert the appropriate markup using a subcomponent of the markup used above.

Implementation nuances

In the static markup method of describing content the consuming agent cycles through possible <source> elements one at a time in search of a suitable match. In my testing on mobile WebKit (iPhone OS) this test cycle removes the poster frame image described in the <video> element and instead places a broken video image inside the element dimensions instead. If a later <source> element matches a generic playback image is added to the element. Source element cycling is the new flash of unstyled content for the HTML5 video world.

The dynamic insertion method relies on the canPlayType method and its return values of “probably” or “maybe.” Maybe is not good enough for my needs if I have a Flash fallback option, but if you are in a constrained playback environment such as low-power mobile devices then acting on a response of “maybe” is better than nothing. Just be sure to send along some alternate HTML as a failure fallback.

Player UIs

Each web browser supporting HTML5 video uses its own backing software to power the video playback experience. Chromium and Google Chrome use a specially patched version of FFmpeg. QTWebKit uses Phonon. Layer on top platform-specific video acceleration, UI, and handling and you will see a variety of final UIs across browsers and platforms. Including the controls in your <video> element is the quickest path to launch but you will give up control over interactions.

If a web browser supports HTML5 video it almost certainly supports native vector graphics as well. It’s possible to craft your own UI with supported JavaScript methods triggering play, pause, and final frame handling in the native video handler.

HTML5 video and Flash

Flash is the dominant method of video playback on the web today. Native browser support of HTML5 video and business excitement to reach low-power devices such as the iPhone provide compelling reasons to offer content using HTML5 video markup. Flash supports progressively loading MP4 files with H.264 video and AAC audio since 2007. Flash Player 10.1, expected in the next few months, speeds up playback with less resources thanks to specialized GPU handling and more efficient code. HTML5 video and Flash playback solutions will need to co-exist for maximum reach (that’s the reason you are using Flash in the first place).

Playback is only one component of the total video experience. You will need to develop analytics and advertising capabilities to match or exceed your current Flash experience. Advertisers don’t publish interactive advertisements in <canvas>. The high-CPM pre-roll and post-roll video advertisements we see today are based on a Flash ecosystem built up over the years. HTML5 video and your money maker of choice will need to find a way to co-exist (banner and text advertisements still work well) and drive your development budget. I expect to see better JavaScript libraries from the open-source community as well as advertising networks solve some of the problem in the near future, just like a suite of XHR handlers popped up once Ajax started to take off.

Summary

HTML5 video has arrived and is deployed across a wide enough user base for sites and developers to stand up and pay attention. File support and markup varies by browser and there is currently no native support in Internet Explorer. Developers are excited to take advantage of the performance gains of native video handlers and reach new audiences in the smartphone market. If you are thinking of getting implementing HTML5 video in the future it’s possible to start measuring your audience’s playback compatibility today so you at least know your deploy targets.