Yahoo! Open Strategy launch

Yahoo! Open Strategy sign

On Tuesday Yahoo! launched its Open Strategy, exposing Yahoo! account data and social connections to third-party developers. Yahoo! Open Strategy is the third pillar of faith announced by CEO Jerry Yang last year during the company’s rebirth. Y!OS is the new glue connecting the next versions of Yahoo!’s own properties and will eventually power more relevant advertising across the network. In this post I will provide an overview of the new Yahoo! services and its impact on both Yahoo! and third-party developers. Yahoo! Open Strategy is one of the keynote presentations at Widget Summit next week.

The Yahoo! Open Strategy first and foremost unites Yahoo!’s own product fiefdoms into a common set of interchangeable components. The new platform ties together social features and rich content units across Yahoo! properties in much the same way as YUI abstracts a common set of JavaScript and CSS interactions across Yahoo!. The company announced plans to integrate the Y!OS platform into a new version of My Yahoo! and Yahoo! Mail over the next year.

Yahoo!, like many portals, receives most of its user activity from mail and IM networks. These are the explicit social networks, tracking who we communicate with every day and at what frequency. Yahoo! Mail, Yahoo! Messenger, and Yahoo! Mail represent some of the largest front-doors on the Web across a very international audience. Yahoo! is building upon these established strong relationships to construct a social platform receiving new, relevant inputs every day. Y!OS is a partial answer to Yahoo!’s build vs. buy decision it faced in September 2006 when Yahoo! reportedly offered to buy Facebook for $1 billion (about $400 million at today’s share price).

A new Yahoo! Profile

Yahoo! Profile

The new Yahoo! profile pages collect more information on every Yahoo! user in an interface that is easy to discover and share. Yahoo! previously buried this data inside user account preferences and really didn’t provide much motivation to keep profile data up-to-date. The new Yahoo! profile is at the center of the Yahoo! Social experience, motivating participating users to better describe themselves to connected friends and services.

Exposing this profile data across Yahoo! applications ultimately leads to richer ad-mining for Yahoo! and its partners.

Social APIs

Yahoo! tracks explicit and implicit user connections through its Contacts platform and Social Directory. Messenger buddies, address book entries, and direct communications all influence the Yahoo! social platform. Yahoo! social features are similar to what publishers have come to expect inside a Facebook social graph or combined news feed but adds live user presence (online/offline) to the mix.

Yahoo! Application Platform

Yahoo! Application Platform small view sample

The Yahoo! Application Platform (YAP) adds third-party content to Yahoo! properties through a full-page editing canvas and small widgets. The Yahoo! Application Platform is build on OpenSocial 0.8 JavaScript APIs combined with Yahoo!’s own markup language similar to a server-side include and FBML.

The YAP widget view is restricted to static content for speed. Developers can’t use JavaScript, PHP, or other languages within a Yahoo! widget and are not allowed to display any advertising or promotions. These heavy restrictions strip the widget down to its essentials for the sake of speed and scalability. The no-advertising and no-promotions clause may cause companies to rethink content monetization on the Yahoo! platform.

Summary

Cody Simms and Neal Sample

The Yahoo! Open Strategy 1.0 release is just the beginning of Yahoo!’s rewiring. The platform really gets interesting once Yahoo! flips on My Yahoo! and Yahoo! Mail support, enabling 500 million worldwide users to access new content. Yahoo! already owns a compelling notification platform in Messenger and Mail, which should drive the virality of applications through invitations and updates that actually reach their intended recipient. Yahoo! has assembled a small team of ex-BEA middleware experts behind the scenes to drive new Java-based platform apps across the company and around the world. The Y!OS launch this week is a good start and I’m looking forward to Yahoo! delivering on its vision.

Better Design Through Code

Every day our web applications ignore useful visitor data. We respond to single request based on a domain and a path without listening to the capabilities, location, preferences, and favorite interactions of our visitors and their requesting agent. A few weeks ago I challenged a room full of designers at PARC to rethink what’s possible on the Web and rely on adaptive programming techniques to serve the right content to the right audience at the right time. I titled the 50-minute talk “Better Design Through Code” and walk through latent capabilities of servers and browsers ready and waiting to deliver personalized, adaptive content to unique Web visitors.

I prefer recorded presentations to static shared slideshows. Each movie has 3GPP timed text chapters indexed by slide if you would like to jump ahead to a particular part of the presentation. The whole process is very experimental yet an interesting way to reach new audiences.

Classify incoming requests

Incoming requests contain more than a domain and a path. Servers can listen to full request data and segment your audience based on key factors such as preferred language, browser capabilities, or requesting device such as a TV or mobile phone. Listening for key navigation clues reduces visitor input and delivers the best content possible quickly and easily.

Location filters

Broad data options can be quickly narrowed through location-based targeting. Web sites can store simple lookup tables to identify the location of their audience at various confidence levels as broad as a home country or as specific as a postal code. New data-driven location services such as Gears or Loki offer even greater location precision by searching the local network for mapped devices on your local network, within radio range, or even receiving signals from GPS satellites.

Detecting installed software

Software installed on our computers leave browser-addressable footprints in the form of MIME and URL schemes meant to connect our browsers, webpage embeds, or downloaded files with the appropriate installed application. We can detect installed software on the requesting visitor’s machine by testing these known MIME footprints and establishing connections between our web application and the best possible handler on the client. Want to send a photo RSS feed to iPhoto without confusing your users with technical jargon? Test it. Need to communicate an physical address or seamlessly hand off a podcast subscription? Identify tethered GPS or music players on your visitors’ machines and dynamically create links to desktop-addressable software from within your webpage.

Detect favorite websites

The final part of my presentation focused on identifying the favorite websites and web services of a visiting user to improve site content. Browsers leave a history trail to help us quickly navigate to our favorite resources and identify previously viewed content. We can connect our audience to the web applications and services they care about by testing websites of interest against the current browser history and displaying the best activity prompts to each unique visitor.

Summary

Every time a web page loads we throw out potentially useful data. With just a little effort we can thrill our users with custom, adaptive experiences based on their unique computing and personality profiles for increased engagement and conversions. This presentation outlines some of the reasonably easy methods of customization available to site owners seeking more intelligent methods of visitor interaction through smart server- and client-side applications.

Inside the iPhone App Store acceptance process

Apple’s iPhone OS App Store is a little over two months old and already the focus of both hype and fear among members of the press. KPCB has already invested more than $30 million through its iPhone-specific fund. Established companies are writing iPhone applications for the first time. A few applications have been banned, as expected with most platforms. Apple’s relative secrecy regarding the iPhone platform and distribution policies have caused market uncertainties in need of some further clarity. In this post I will examine the iPhone OS 2.0 platform and the iPhone App Store from the point of view of Apple and other hosted storefront providers.

As I write this post there are over 3400 applications available from the iTunes App Store. 90% of those apps are available for both iPhone and iPod touch. 76% of App Store listings require payment ($1 or more). Developers may distribute an iPhone OS 2.0 application directly to handsets using their own infrastructure or distribute through Apple’s App Store built-in to iTunes and iPhone OS.

  1. Inside the App Store
  2. Podcaster rejection
    1. Submitting to App Store
  3. Ad-Hoc Distribution
  4. Summary

Inside the App Store

Apple’s iPhone App Store connects developers large and small to millions of iPhone OS devices around the world. Apple handles payment processing, international business licenses, distribution, marketing, and delivery of applications to iPhone and iPod Touch devices over WiFi, cellular data, and tethered experiences. App developers may list items for sale in Apple’s App Store in exchange for a consignment fee of 30%.

Carrying third-party content inside your store and on your devices does carry some risk. Developers pay to access the iPhone developer program, sign their applications with unique keys, and assume some of the support burden for their applications. There are a few obvious reasons why a platform such as iPhone might choose not to carry an application in its storefront:

  • Chargebacks. Buyers frequently return your product for reasons including buyer’s remorse or just receiving a different product than they expected. The “I Am Rich” $1000 iPhone app carries a heavy chargeback risk.
  • Insufficient differentiation. App authors should be able to submit an application to App Store and expect there won’t be a knock-off product sold directly alongside. Open-source applications can swap out an application title and submit the app as their own without adding new functionality.
  • Misleading marketing, including trademarks. Don’t misrepresent yourself or your product or cause obvious confusion.
  • Horrible customer experience. Apple will recommend interface designers who can assist you with visual aspects of your application. Long load times or heavy resource utilization might will make both you and the platform look bad.

Podcaster rejection

The Podcaster application was initially rejected for redistribution through Apple’s iPhone App. I downloaded a copy of the app, version 1.0.9b, directly from the developer, installed it on my iPhone, and reviewed the app from the point of view of the Apple based on existing publications and guidelines.

Podcaster navigation screen

Navigating the tab bar to content sections such as newest podcasts or featured podcasts brings up a splash advising 3-5 minute wait times before content is returned. Red flag. My previous search input does not disappear when I change navigation modes.

Podcaster for iPhone add podcast confirmation

The developer used a single button action sheet with a vibration action to confirm each new podcast download. Apple adds a badge to the Downloads tab for a similar action in iTunes. The user has to dismiss a confirmation sheet for every new addition.

Podcaster iPhone application iPod application comparison

Pictured above is a view of the same podcast in Podcaster and iPod. Duplicate functionality when tethered but Podcaster does provide over-the-air updates direct from the handset. Podcaster has some obvious errors with text display in the regular table view but the display does work. Displaying the sound disclosure indicator on every row is a bit overkill: Apple only includes this disclosure indicator when the audio file is currently playing. Small fit and polish issue.

Submitting to App Store

Podcaster submitted the first release candidate of their iPhone application to iTunes Connect, App Store’s web-based management tool, on August 14. Application release notes are available through the developer’s Twitter feed. The version submitted to Apple in mid-August had problems downloading and playing many podcasts including FeedBurner’s redirect URLs or large downloads (likely a Range issue). On September 12 Podcaster heard back from Apple regarding their App Store submission. The $4.95 application was not accepted for distribution through App because it it duplicates the functionality of the Podcast section of iTunes.

The developers resubmitted Podcaster to iTunes Connect on September 13 with a new description before launching a media campaign and ad-hoc distribution.

Ad-Hoc Distribution

Apple iPhone App Store Ad Hoc distribution design

Apple designed Ad-Hoc distribution for direct distribution of iPhone OS applications. Each application build is limited to 100 provisioned handsets. Ad-Hoc distribution is a primary distribution mechanism for beta testers, classrooms, workshops, or corporate environments that do not desire worldwide distribution through App Store.

As of yesterday afternoon Podcaster had provisioned 1130 devices for distribution across 12 different copies of the application hosted on Google Code. Each new uploaded build included up to new 100 authorized devices after the publisher received payment via PayPal. It’s stretching the Ad-Hoc distribution model a bit but the application may have collected approximately $11,000 over the weekend through suggested donations of $10 per handset. At the time of writing Apple has not pulled the application or developer certificates from their central certificate authority.

Summary

It’s not always clear what Apple is up to but there are legitimate reasons to not carry an item for sale in your store. iPhone OS will continue to be a popular development platform even if Apple is overwhelmed with interest and developer support. The best developer information comes from within the confines of ADC, WWDC, and iPhone developer programs, and other privileged access forums restricted by Apple and its partners.

The story behind Google Chrome

Ben Goodger and Google Chrome

Google released its second web browser yesterday afternoon, adding additional headroom for web applications stretching the limits of what it’s possible to accomplish within a web browser. The Google Chrome team assembled domain experts in various fields over the past six years, both through direct hires and acquisitions, to create a new browser and its critical components from scratch. GMail and Google Maps pushed the Web to its limits, taking advantage of browser technologies invented in Redmond but left dormant for far too long. Contributing to Firefox’s core, writing browser extensions, and championing HTML could only take the $150 billion company so far: they needed to own the full browser to push their Web efforts forward at full speed.

  1. Growing Frustrations
  2. Acquisition Boost
  3. A New Browser from Scratch
  4. Rev your JavaScript Engines
  5. Meet the Team
  6. Summary

Growing Frustrations

Brian Rakowski joined Google in July 2002 as the company’s first associate product manager. His first assignment? Launch GMail with features and responsiveness to rival desktop mail clients. Gmail tapped into relatively dormant browser features such as XMLHttpRequest, sockets, prefetch, and more to create a web applications stretching the limits of what was possible inside web browsers of 2004. Today’s Gmail continues to run into a browser’s limits, setting minimum requirements of Internet Explorer 7+ and Firefox 2+. Google web apps teams such as Maps and Mail continually bump their heads against the latest capabilities of web browsers and in some cases invent their own runtimes.

Ian Hickson first learned the inner workings of web browsers while an intern at Netscape. After working on Opera for a few years and creating tests for Firefox Ian joined Google to continue his work on new browser features. HTML5 and browser compliance “acid” tests are significant attempts by Ian and others to redefine Web browsers through specs, test, and implementations but until now Google could only offer development help and browser extensions such as Gears to accelerate browser capabilities.

Google extended what it could not immediately add to the browser core. Gears for new application functionality on multiple browsers. Browser Sync to synchronize browser settings and data across multiple computers. Safe Browsing to create more web trust. Teams from each of these extensions are now working on Google Chrome.

Acquisition Boost

Google released its first official Web browser on August 18, 2008 with the beta release of the Android mobile operating system. Google acquired Android in August 2005 to establish a foot-hold on the fastest growing computer (and Web) market: mobile handsets. Android highlights Google’s web properties through its WebKit-based browser and dependent applications. Google acquired Ottawa-based Reqwireless and its mobile web browser in the summer of 2005 to team up with the Android team on its web interface. Web views are an integral part of Android and Google Chrome shares much of Android’s code, including its graphics engine.

Google Chrome and Android both take advantage of the Skia vector graphics library developed by a small company in North Carolina Google acquired in 2005. The Skia team formerly worked on Openwave’s popular mobile browser’s graphics engine. Google Chrome browser includes Skia graphics engine ports for Windows, Mac, and Linux.

Google acquired application security company GreenBorder in May 2007. GreenBorder technology automatically sandboxes web code and network traffic by creating a bridge between applications. The GreenBorder technology isolated Internet Explorer or Firefox instances into a “sandbox” inside virtual machine instances. These sandboxes form the code isolation layers of Google Chrome, protecting other tabs and the parent operating system from the code executing on each web page.

A New Browser from Scratch

Ben Goodger, Google Chrome’s tech lead, is best known for assembling the Firefox web browser out of Mozilla’s SeaMonkey application suite. Manticore, Camino, and later Firefox were all attempts in 2001 to rethink the Web browser for the modern age. Browsing took center stage away from a communications suite, user interfaces reimagined for Web efficiency, and (some) legacy cruft tossed to the side. Google hired Ben in 2005 to strengthen its own browser contributions and eventually fully rearchitect a web browser for the modern Web.

Google hired top Firefox developers in 2005 and 2006 such as Darin Fisher, Pam Greene, and Brian Ryner. In Spring 2006 the team began work on a new browser prototype built on top of WebKit designed for broadband-connected, always-on, web applications such as Gmail or Google Maps. Could the browser experts give web apps some breathing room?

Modern computers feature multi-core multi-gigahertz CPUs, gigabytes of memory, megabits of bandwidth, and bulky hard drives. Our web browsers should separate browser tabs into their own processes, multi-thread all communications with the operating system, boost cache sizes, and not be afraid to command more bandwidth when available. Internet Explorer 8, Firefox 3.1, and Apple Safari are taking fresh approaches to web browsers for modern machines but Google Chrome has the advantage of a fresh start to achieve some features not currently possible in other browser architectures.

Features such as tab-isolation and task monitoring are difficult tasks to add inside an existing browser architecture of shared run-times and window models (as John Resig mentioned). Internet Explorer 8’s Loosely Coupled IE partially abstracts browser tab instances and the industry is generally headed in this direction.

Web application-specific resource monitoring should motivate more websites to reduce their browser bloat now that they’ve been identified. Individual users can also compare web application resource usage directly with their desktop counterparts.

Rev your JavaScript Engines

Lars Bak and his team in Århus, Denmark have spent many years writing virtual machines: the run-times that translate programming code into machine code. Lars wrote Sun’s Java VM, HotSpot, and later slimmed down the VM for J2ME (CLDC HI project Monty). A few years ago Lars and his team in Denmark began work on a new interpreted JavaScript engine optimized for x86 and ARM architectures.

The V8 engine is specifically tuned for recursive JavaScript tasks, optimizing commonly used components of your application. V8 is multi-threaded, opening up new parallel processing on multiple computing cores. V8 guesses how you might use your JavaScript code, and backtracks over any faulty assumptions. It’s just one of the new engines we’ll see inside our web browsers by the end of 2008.

Google Chrome could have used the same JavaScript interpreter as its WebKit rendering engine (JavaScriptCore, SquirrelFish) but the team had an opportunity, and the funding, to rewrite an interpreter from scratch for desktop and mobile runtimes.

The V8 engine enables new feature sets for Google’s web applications such as Gmail and Google Maps. Web application developers avoid adding features that visibly slow down browsers or cause processing pauses in your application experience. New speed in new areas adds functionality to existing apps. Google programmers should create more efficient code, tested against multiple interpreters, and optimized for modern computers as a result of V8. Even if Google Chrome gains no significant browser market share I still expect it will be the best single-site browser for Google web applications.

Google Chrome adds additional JavaScript functionality through Gears. Gears is bundled with every Chrome install, adding new features to the web browser faster than previous plugins. The Gears libraries include support for new local cache structures, local databases, location data, background tasks, and file handling. Chrome boosts the available Gears footprint for web developers, including Google’s own apps such as Google Reader and Google Docs (and my blog). The current Gears code included in Chrome replicates V8 and sqlite code already present in the browser, a bolt-on that will hopefully be integrated in the near future.

Chrome, V8, and Gears will be a new testing ground for Google’s HTML5 efforts, winning a new seat at the table as an implementor with upstream standards groups such as W3C.

Meet the Team

Questions with the Google Chrome leads

I am tracking at least 20 people involved in the Google Chrome project across Google. I’m sure Chromium commit logs will reveal even more (update: more complete list here), but below is a quick summary of Chrome staff.

Brian Rakowski, Lead Product Manager
Brian was Google’s first associate product manager in 2002, assigned to Gmail. He later worked on the Google Browser Sync Firefox plugin.
Ben Goodger, Software Engineer
Ben is the former Firefox 1.0 project lead. He also authored the Firefox extensions system. He joined Google as 2005.
Mike Pinkerton, Technical Lead
Mike is one of the Google team members responsible for bringing Chrome to the Mac. Mike worked at Netscape and later on the Gecko-powered AOL client before co-founding the Camino project. Mike joined Google in September 2005 and continues to lead Camino development.
Darin Fisher, Software Engineer
Darin was a frequent contributor to the Firefox codebase. He specialized in network libraries, cookies and permissions, and the Netscape Portable Runtime. Darin joined Google in 2005.
Lars Bak, Software Engineer, V8
Lars was the core developer on Java HotSpot VM and Monty VM in J2ME for Sun. He co-founded object-oriented VM companies for embedded devices before joining Google. Lars worked on V8 from a farm in Århus, Denmark before moving the team to university offices.
Kasper Lund, Software Engineer, V8
Kasper shares a long history with Lars Bak working on virtual machines.
Brian Ryner, Software Engineer
Brian is a former contributor to Firefox where he added mousewheel support, tweaked the Gecko rendering engine core, password management, and Linux installers.
Pam Greene, Software Engineer
Pam is a long time Firefox contributor. She added OpenSearch to the browser and contributed to full-text search in Places/AwesomeBar.
Ian Fette, Product Manager
Ian is a former Firefox contributor who worked on anti-phishing, anti-malware, spelling correction, and the Safe Browsing API.
Arnaud Weber, Software Engineer
Arnaud is a former Director of Research and Development at Netscape and Borland before joining Google to work on a “secret project” in September 2006.
Brett Wilson, Software Engineer
Brett formerly worked on the Google Toolbar. He contributed to Firefox history and bookmarks functionality.
Mike Belshe, Software Engineer
Mike helped write an Outlook add-on called Chrome for Lookout Software before being acquired by Microsoft. Mike also formerly worked at Netscape and Good Technology.
Huan Ren, Software Engineer
Huan works on network flow control, negotiating browser interactions with network resources. Huan formerly worked at Microsoft.
Erik Kay, Software Engineer
Erik formerly worked on the AvantGo browser, Qurb anti-spam software for Outlook and Outlook Express.
Glen Murphy, Software Engineer
Glen specializes in user interface design. He previously worked on user interface. Firefox extensions. Google Browser Sync, Google Blog Search
Evan Martin, Software Engineer
Evan writes automated testing tools for Chrome and the Web.
John Abd-El-Malek, Software Engineer
John is part of the Windows specialist team at Google bringing Google Desktop, Google Talk , and Breakpad onto Windows XP and Windows Vista.
Amanda Walker, Software Engineer
Amanda is one of the people responsible for Chrome’s upcoming Mac version.
Mark Mentovai, Software Engineer
Mark was heavily involved in moving Firefox for Mac to its current Intel-based architecture. He has worked on the Breakpad project and many levels of Chrome’s code.
Carlos Pizano, Software Engineer
Carlos formerly worked on GreenBorder and continues to work on Chrome sandboxing.
Mark Larson, Program Manager
Mark is also formerly of GreenBorder and its sandboxing specialties.
Aaron Boodman, Software Engineer, Gears
Aaron improves user experience with JavaScript. He’s best known for his work on Gmail, Greasemonkey, and Gears.

Summary

Google Chrome original logo

Google’s business depends on the speed and availability of Web access to search, advertising, and applications. Chrome is Google’s second attempt to better control the front-door to its content with full applications optimized for its heavy apps. Google Chrome builds on top of the work of Android by adding individual applications to already popular operating systems. Google has flirted with the idea of its own web browser for many years but has only recently released working implementations of its own full browser applications.

Android, Chrome, and Gears will continue to grow in unison and extend individual pieces into established operating systems. Google is building a new suite of application extraction layers that should have strong leverage across Windows, Mac, and Linux to directly control the company’s destiny on these platforms.

It’s an exciting time for new browser technologies as Internet Explorer, Firefox, and WebKit each compete over standards implementations and performance. Officially adding Google Chrome to the browser space only strengthens Google’s position strengthening the future web and delivers strong single site browsing experiences for their core web applications.

Internet Explorer 8 Search Suggestions

Microsoft released a second beta of its upcoming Internet Explorer 8 browser yesterday afternoon. The new browser will reach full release by the end of the year, changing the way most Windows users view the Web. There are many new features of IE 8 for web developers, including completely new ways to light up the browser chrome. Microsoft has extended the OpenSearch protocol with a new search suggestions data formats expressed XML or JSON. The new format will display real-time search results, summaries, images, and even search result classifications inside the browser chrome for any site owner supplying the appropriate format. In this post I’ll teach you how to add search suggestions to your OpenSearch description document for instant search suggestions in IE8.

  1. Search Suggestions
  2. Suggestions Format
    1. Quick element definitions
  3. Summary
Internet Explorer 8 instant search

Internet Explorer’s Instant Search provides suggestions based on text already entered into the search box. The functionality is very similar to Google Suggest and its JavaScript feed expanded for multiple categories, graphics, and short descriptions.

The new format covers search suggestions and not necessarily search results. Syndicated search results should continue to be exposed as a Url element of attribute type of Atom or RSS.

Search suggestion data files are exposed through a new MIME type referenced in a OpenSearch descriptor’s Url element: application/x-suggestions+xml.

<Url type=”application/x-suggestions+xml” template=”http://example.org/suggest?q={searchTerms}” />

Suggestions URLs should follow the same fill-in-the-blank parameters defined by OpenSearch such as result count or language scope.

Suggestions Format

How do you make your site light up with pretty pictures in the search box? Just add XML descriptors of possible intended searches.

<?xml version="1.0"?>
<SearchSuggestion>
  <Query>ajax</Query>
  <Section>
    <Separator title="Web Development"/>
    <Item>
      <Text>AJAX Developer Center</Text>
      <Description>Asynchronous JavaScript and XML</Description>
      <Url>http://developer.mozilla.org/En/AJAX</Url>
      <Image source="http://example.org/ajax.jpg" alt="AJAX Web Development"
             height="50" width="50" align="middle"/>
    </Item>
    <Separator title="Soccer"/>
    <Item>
      <Text>AFC Ajax</Text>
      <Description>Amsterdamsche Football Club Ajax</Description>
      <Url>http://english.ajax.nl/</Url>
      <Image source="http://example.org/afcajax.jpg" alt="AFC Ajax"
             height="50" width="50" align="middle"/>
    </Item>
    ...
  </Section>
</SearchSuggestion>

In the example above I provided search suggestions broken into two sections with Separators: web development and soccer. Each section has a list of Items defining title text, a short summary, and a related image.

Internet Explorer 8 also supports results in JSON format if you prefer.

Quick element definitions

Separator
A distinct grouping of your search result set. Correlates well with site categories.
Item
An individual result wrapper.
Text
Your result title. Internet Explorer will highlight text in your result title matching the text entered in the search box.
Description
A short summary of the search result. Similar to a HTML meta description or search result snippets.
Image
An image you would like to display alongside the search result. Internet Explorer 8 will pass along height and widths in the drop-down configuration — max width, row height, and section height — if you setup a few extra parameters in your OpenSearch description. The image should be relatively small to fit alongside a search result title and description (~75px).

Summary

Internet Explorer 8 opens up more of the browser chrome to user search customizations including instant search suggestions. Webmasters can enhance their search results for supporting web browsers with suggested terms or results served directly within prime browser real estate. Images, highlighted text, and short summary will help your results stand out and should drive increased search usage by loyal customers.

Google crawls web forms and I expect their search team will look for signals described on the page such as OpenSearch or site suggest to make an educated guess about the best ways to probe your site for deep results. Adding search suggest markups to your site could also help machines discover deeper content within your site among popular keywords.

Internet Explorer 8 just released beta 2 and these features are not frozen. Search suggestion XML are a good feature to track for sites with deep content and engaged users.

Intel and Yahoo! announce Widget Channel for HDTV

Flickr on Widget Channel

The Internet is coming to your TV, reclaiming your split attention span from the other gadgets around the house. Intel announced its latest effort to power your living room yesterday with new media processors, reference designs, and software stacks that may eventually find their way into the cable boxes, Blu-ray players, and home media centers of 2010. Intel partnered with Yahoo! to deliver Internet-connected widgets, advertising, and content to potential partners with a software stack branded The Widget Channel. Yahoo! spent about two years customizing Yahoo! Widget Engine for high-definition televisions and hardware-accelerated graphics displays. Yahoo! will pitch its widget engine for televisions, display advertising integration, and customizable widget gallery to cable operators, television manufacturers, and other major consumer electronics companies as Yahoo! seeks a prominent role in what it calls the “Cinematic Internet.”

I visited the Intel Developer Forum in San Francisco yesterday for a first-hand look at the new prototype widgets platform. I was lucky enough to bump into Eric Kim, Intel’s SVP and General Manager of the Digital Home group, and recorded a 8-minute overview of Widget Channel. I’ve embedded the walk-through below. High-resolution snapshots of individual Widget Channel widgets are available on my Flickr account.

  1. Yahoo! Widgets on TV
  2. From reference design to reality
  3. Picking apart the pieces
  4. Summary

Yahoo! Widgets on TV

Widget developers may be already familiar with the Yahoo! Widget Engine, also known as Konfabulator. This desktop engine started out on the Mac, ported to Windows, and now runs inside the Linux-based Intel TV platform. The Widget Engine team is a part of Yahoo!’s Connected Life division which also includes Yahoo! Mobile — powered by Blueprint widgets — and DVR acquisition Meedio. Yahoo! has tied its data APIs to TiVo and Windows Media Center in the past, accessible as a full-screen application after a deep dive through the device’s navigation options. The Widget Channel and its alpha-blended snippet dock complementing the main viewing experience of your TV is a radical departure from past Yahoo! partnerships in the space, an obvious result of designing an experience from the ground-up instead of bolting onto other vendor’s solutions.

Widget developers can build new widgets for Widget Channel using most of the same resource bundles and runtimes used on the desktop Konfabulator engine. XML manifest files define widget metadata, preferences, and screen UI. JavaScript powers on-screen interactions and dynamic data. Yahoo! Widget Engine includes a WebKit run-time, which will hopefully be ported by Yahoo! and Intel to support hardware accelerated CSS and other nice features on the new software stack.

Widgets written for the new Yahoo! Widget Engine for TV must conform to four major UI modes: snippet content on the bottom dock, sidebar content, full-screen display, and background processing. Docked snippets are more than just minimized widgets: viewers can cycle through multiple snippets inside a single widget such as weather in various cities or a sports scores. Most sidebar displays mocked-up by Yahoo! used an accordion design pattern to collapse multiple content sections inside a small space. Full-screen experiences match the full-screen designs of the Web. The demo widget for Flickr uses their newly redesigned Flash slideshow display for a familiar look and feel from desktop to living room.

I couldn’t get a solid answer from Intel regarding how tightly Widget Channel was tied to Yahoo!’s Widget Engine. Intel wants to sell its new consumer electronics system-on-a-chip, the media processor CE 3100, far and wide with or without the Yahoo! engine. I expect the underlying platform contains a native widget layer and programming environment with tighter integration but more programming complexity than Yahoo!’s engine in much the same way NVIDIA Preface bolsters its platform offering with a Windows Sideshow gadgets run-time.

From reference design to reality

Gigabyte MD300 DVP rear

Yesterday’s announcement from Intel and Yahoo! is merely a reference design showing off what both companies hope is the future of Internet-enabled consumer electronics. The Yahoo! Widget Engine for TV still needs a lot of work and there are currently no shipping products implementing the hardware and software stack demonstrated yesterday. Cable companies, television manufacturers, and other consumer electronics companies will evaluate the stack over the next year for possible inclusion in products shipping next decade.

Intel is trying to displace consumer electronics chipsets already in production from IBM and NVIDIA. Sony, Toshiba, and IBM worked together to create the Cell multiprocessor already powering the PlayStation 3 and built-in to the next generation of televisions. NVIDIA chipsets are inside cable boxes from Scientific Atlanta and others. The upcoming Java-based tru2way cable software platform is already under active development by consumer electronics companies and software vendors. Widget Channel could operate as an additional layer on top of tru2way, as mentioned in Comcast’s press release yesterday. The word “Yahoo” does not appear in the Comcast press release and Comcast has only announced their intent to evaluate the new reference design against their own Java-based offerings in 2009.

Picking apart the pieces

Widget Channel is a Linux-based operating system with platform software and middleware provided by channel partners. The Widget Engine is one of the available software options on the device. Carrier-specific back-end services including reporting, storage, security, and developer certificate verification reside within the carrier network with possible add-ons such as display advertising powered by Yahoo! or others. Each Widget Channel implementation can choose its own Widget Gallery service and white-listed widgets and possibly receive extra content from a compatible widget gallery offering served by Yahoo!.

Yahoo! has a good opportunity to serve display advertisements, sell premium widget placements and certifications, and promote its own content within each Widget Channel deployment. It may be possible for Yahoo! and Google to program their own compatible widget and advertising layers on top of the base Intel platform to replace or compete with Yahoo!. Major features such as contextual widgets and advertising layers have yet to be developed for the platform, leaving new opportunities for other Internet companies to step in with their own swappable components inside the software stack.

Summary

Yahoo! Widget Gallery home screen

The Intel Widget Channel provides a peek inside the connected future of our living rooms. Consumer electronics companies and large carriers from the cable and satellite industries want to participate in the premium content offerings available through Internet-connected electronics and new software stacks from chip vendors could help bootstrap new services. Intel wants to sell more chips, Yahoo! wants to serve more ads, and cable companies want to boost subscription revenues without major investments in new infrastructure.

Intel admits interactive services on the television has been a popular goal over the last 10 years but without measured success. [T]he rate of adoption has so far been disappointing, studies show that consumers remain receptive to the concept. New levels of broadband penetration combined with high-definition viewing could change consumer adoption but we are still a few years out from real adoptable implementations. We’ll have to wait and see what hardware and services are announced in 2009 before spending too much development time against a reference design.

Writing Flash for search engines

Adobe Flash logo On June 30 Google and Adobe announced a new indexer optimized for Flash (SWF) discovered by its web crawlers. The new partnership takes advantage of a server-side Flash player optimized for a search engine indexing environment and unidirectional text (e.g. no Hebrew or Arabic). Search engines previously discovered the location of a SWF file on the Web and perhaps indexed its metadata but did not take a deep look inside its binary content. Last month’s announcement was a big change for both Adobe and major search engines as it is now possible to run a very GUI-based Flash file at the command line and interpret both its text content and interaction opportunities. In this post I will walk through what we currently know about the search engine Flash runtime and how it affects search engine optimization in Flash.

Build for a blind, deaf user

Search engine indexers are blind and deaf. They open a file, examine its contents, and try to deduce meaning through your page structure and its content. A web page designed for screen readers will also expose more content to search engines not evaluating your page’s full render state of content, layout, and interactions.

Search engines utilizing Flash player indexing are still restricted to this screen reader approach. Accessible Flash applications complete with names, labels, reading order and XMP should continue to be more search engine friendly than other SWF files on the Web. Google’s tips for creating accessible, crawlable sites still apply, but in a new Flash context.

The server-side Flash Player

If we want to understand how search engines such as Google might interpret Flash content we’ll first need to take a look at the Flash Player itself. Adobe provides little details in its official SWF searchability FAQ but we can infer a few implementation details. How would you rewrite Flash Player for server-side indexing of SWF content?

The search engine Flash Player is likely a scaled-down, secure version optimized for machine readers. Strip out video, audio, fonts, and file system access. The server side Flash player should open a binary SWF file, pull out the functionality it understands, and create a data tree of all possible actions. These features are actually quite similar to a screen reader interface, but Adobe is instead targeting a Linux-based headless runtime. I believe the guts of the Flash Player for servers is built using the same accessibility abstraction layer Adobe currently uses for Windows and could extend to platform-level binds Mac and Linux desktops.

The Adobe Flash Player creates a list of objects on the screen at each render and records this list into an accessible data tree (according to a 2005 white paper by Bob Regan). This data tree is updated with each change in the application state, allowing any application listening in to update an object model of clickable buttons, labels, and links.

Adobe interfaces with OS-level accessibility frameworks on Windows currently and could extend this model to every major desktop platform. The Windows version of Flash Player binds to Microsoft Active Accessibility. Mac versions of Flash could bind to Universal Access. On GNOME the player could bind to the Assistive Technology Service Provider Interface (at-spi). A server-side version of Flash likely builds upon this same abstracted accessibility object model, passing screen objects to the search engine indexer for further interpretation or interaction.

Windows Live Search was noticeably missing from the server-side Flash player announcement for search engines. It’s possible Adobe has developed a server-side Flash Player for Linux that is not yet compatible with the Windows Server environment of Microsoft’s Windows Live Search.

Accessing deep content

Googlebot can fill out forms, click buttons, and navigate deep within your site. Clickable Flash objects will likely behave the same way, exposing new content paths for Googlebot within your larger SWF. Flash websites can help ensure deep indexing of SWF content by adding individual SWF fragments to their sitemap. Reading order will likely play a roll in selecting important content on your page, and I expect Googlebot may follow the first item in your reading order sooner than the last.

Googlebot still throws out references to a anchor name fragment in the URL (e.g. #section=menu) and this announcement does not change the general behavior of Google’s URL storage and analysis.

Do Flash versions matter?

Emperor Tamarin monkey

The official announcement from Google and Adobe makes it seem like all Flash is now universally indexed regardless of your Flash version but I think that’s bogus. If a search engine wanted to index JavaScript they might run Rhino on the server and interpret results. If you wanted to build an advanced interpreter of Flash content you might use Tamarin or its derivatives, an AVM2 (Flash 9+) virtual machine. I believe AVM2-compatible SWF files will enjoy better search exposure than binaries built for the older AVM. I can’t prove it; just a hunch.

Dynamic object insertion

Googlebot will detect common JavaScript libraries such as SWFObject used to dynamically insert Flash content at page load. Publishers can back up the dynamic insertion JavaScript with a noscript element just in case Google doesn’t discover your dynamic insertion. Sticking with standard dynamic insertion libraries will help ensure your content is discovered through expected behaviors.

Summary

The new search version of Flash Player opens the binary SWF format to interpretation by text-focused search engines. Flash developers can take additional steps to package SWF content for accessibility and search discoverability. Developing for modern virtual machines, adding accessibility hooks, and wrapping your SWF in XMP.

Google App Engine optimizations

Google App Engine

I have developed a few web applications powered by Google App Engine since its launch in May. It has been a fairly easy transition from my traditional programming in Python and Django backed by MySQL to the distributed App Engine environment, Bigtable, and the limitations of each. I have learned a few App Engine best practices over over the past month and would like to share some best practices for App Engine development gained mostly through trial and error. In this post I will share data optimization tips for Google’s hosted Bigtable instance, reduce the errors and resource usage of your application, and add a few steps to your deployment checklist.

Key-based lookups

I program Django applications referenced by a set of short unique object labels named slugs. A slug column is uniquely queried across a model and easily indexed for fast scans. In the Bigtable world of Google App Engine slugs are optimally stored as a model’s key name. Key names are limited to 500 bytes and must be unique across your defined entity. This unique key lookup directly copies the entity into memory without needing to scan an entire distributed hashtable.

Entity key names provide very fast lookups for developers who like to plan ahead. You cannot alter the key name once it’s set and it cannot start with a number or underscores. If you can accept these limitations within your code you’ll experience an even snappier reads from your data store.

Reduce indexed columns

It’s tempting to choose a Datastore property by its input helper or based on names similar to a SQL equivalent. So what’s the difference between a short String and Text? An index.

According to Guido, a 300 byte string stored as Text is the same size as String but without an index. If you have a short string you never query or sort you’ll optimize your data queries if it’s stored as Text.

Define a favicon

App Engine developers should define favicon.ico, robots.txt, and other frequently requested file paths. Google App Engine logs frequent errors inside your administrative console if it has to hunt for your icon with every browser request.

Define the location of your static favicon file directly from app.yaml for fast response times:

- url: /favicon.ico
  static_files: static/favicon.ico
  upload: static/favicon.ico

You should follow a similar pattern for robots.txt and optionally the verification files from Google Webmaster Tools, Yahoo! Site Explorer, and Windows Live Search.

Define default 400 and 500 response templates

Your site is not perfect. Visitors will inevitably request pages that do not exist or generate an internal server error. Your site should define default templates for 404 and 500 status codes or risk displaying whatever is sitting on Google’s NetScaler.

Google App Engine default 500 page

The screenshot above shows an error page of an App Engine application without a defined 500 handler. A link on the page suggests a visit to Google’s support website where your visitors will find no support options of interest.

Django developers should define 404.html and 500.html in your app‘s templates directory. Django will load and render each file for the default page_not_found and server_error views respectively.

Deploy and request

Developers should prime Google’s distributed server networks by issuing requests for key URLs a few minutes after deploy. These automated requests trigger your memcache storage and distribute your app instance across Google’s distributed servers. The first request requires more CPU cycles and memory than subsequent requests as Google tries to prioritize active application instances and their versions. You can speed things up by always issuing one or more requests after a successful deploy.

This process is not unlike flushing and re-populating CDN PoPs with new content from your origin server or propagating dynamic handlers across your front-end cluster. It’s best to kick off the process early and have the latest version of your content waiting for new visitors on subsequent requests.

Summary

Google App Engine simplifies the scaling process but is not a magic cloud that will erase all latency and resource usage issues in your app. App Engine requires new approaches to data storage, data latency, and resource requirements in a metered and opaque environment. Hopefully my trials and experience will speed up your App Engine web apps as you create new services in the cloud.

Announcing Widget Summit 2008

Widget Summit logo

I am hosting a my third annual Widget Summit conference November 3rd and 4th at Hotel Nikko in San Francisco. The two-day widget event will once again educate and connect a a widget ecosystem of publishers, toolmakers, developers, and service providers across a variety of platforms including desktop, mobile, web, and social networks. I enjoy taking a look beyond the hype with a sold-out audience interested in building better syndicated content experience through distributed widgets.

The widget industry is constantly evolving as publishers extend their reach beyond their web address and into remote locations already bustling with activity. The popularity of a single site pales in comparison to the aggregate crowds gathered in front of their Windows Vista desktops, iPhones, or My Yahoo! homepages. In the past year we’ve seen new context added to our widget environments connecting us to the location, friend list, or shared application of our widget community wherever they may interact with our content. Today’s smartest widgets enjoy a close bind with their parent platform’s features, regularly poll their home base for relevant updates, and reach new audiences through targeted and integrated content interactions.

At my first widget conference in 2006 we struggled with the name “widget” and this new distribution network most people interpreted as a Flash badge on MySpace. Last year iPhone web applications and the social canvas of Facebook was all the rage, with new opportunities in the enterprise slowly emerging through the rollout of Windows Vista and personal information dashboards powered by software as a service offerings from established consumer brands such as Google and Netvibes.

A lot has changed in the widget space in the 8 months since the last Widget Summit. Widgets are going mainstream, with the startup valuations and press coverage to match. Somewhere among the fog of hype are useful opportunities to reach targeted audiences on their platform of choice. Let’s take a look at some of the big changes we’ve seen since October 2007.

  • New collaborative technologies such as OpenSocial and its open-source reference container Apache Shindig are quickly creating new widget environments at companies that could not afford to create their own implementations from scratch. MySpace, Orkut, Hi5, LinkedIn, and Yahoo! have all committed to a standard set of widget APIs.
  • The Facebook platform is in the middle of its first big changes since its 2.0 release in May 2007. Shifting concepts of profile display, authoring, and member interaction will require new upgrades or fresh opportunities for completely new applications.
  • The iPhone continues to spark interest in mobile web app development based on single-browser environments. iPhone 2.0 will put smartphones in the hands of a worldwide audience for about the price of a ubiquitous iPod and hopefully expand mobile data opportunities.
  • Advertising networks have created separate product offerings specifically focused on widgets. DoubleClick syndicates and tracks widgets through its DART platform. AOL’s Platform-A recently announced widget-specific advertising and sponsorship powered by TACODA’s trail of cookie bounties.
  • The enterprise continues to adopt software as a service and widgets are no exception. Google, IBM, and Microsoft are extending their hosted software into large companies and bundling the latest widget technologies inside an integrated package.
  • Consumer electronics ship with widgets built-in. Your next car, GPS unit, television, or alarm clock may contain customized widget content.

These are just a few of the large trends creating new opportunities for publishers extending the reach of their content through widgets. We’ll cover all the major widget platforms and opportunities at this year’s Widget Summit, providing the business sense and development basics to kick off your new widget initiatives in 2009.

You may have noticed this blog grow quiet over the past few months as I rebuilt the conference software behind Widget Summit and aligned the many business details needed to create the best possible experience. In the next week I’ll share some of the technical details behind my new sites and services.

Registration for Widget Summit is now open with early bird pricing of $795 for the two-day conference in downtown San Francisco on November 3rd and 4th (the Monday and Tuesday before Web 2.0 Summit). I hope you can join us for what should be our best conference yet!

Customizing conference speeches for your audience

Speaking at a conference can be a hit-or-miss event. Next week I will take the stage at Web 2.0 Expo for a three-hour workshop on Web 2.0 Best Practices: expressive HTML, feed syndication, and widgets. Delivering technical content longer than The Godfather is an intimidating yet worthy challenge. I like to tailor my talks for each audience, dive deep when given the opportunity, and connect with new smart people.

Over the past few weeks new conversations have emerged regarding how conferences must change to better suit their audience. As a conference producer, conference speaker, and attendee I have many opinions on running a great show but today’s post will focus on speakers. In this post I will share three speaking tips that keep coming up in my conversations with other speakers in the industry.

  1. Gauge audience skill levels
  2. Prepare more content than needed
  3. Avoid card collectors

Gauge audience skill levels

I like to address audiences with an intermediate to advanced knowledge of web development, content syndication, and widget platforms. I am never quite sure how much my audience already knows and how quickly I can move past the basic bits of knowledge about a particular product or technology. I typically begin a longer presentation with a few technical questions for the audience to set the pace and depth of my talk.

At last year’s Web 2.0 Expo I decided to gauge my audience’s experience with XML and syndication basics by a show of hands. I exposed the following bullet points one-by-one with rising levels of difficulty.

Does this scare you?

  1. & vs. &amp;
  2. 2007-04-17T16:50:00-07:00
  3. HTTP status codes: 200, 304, 410

I was pleasantly surprised by my audience’s reaction to these questions. Only a few people in the audience admitted to not knowing the difference between an escaped and unescaped characters and the ampersand entity reference. A few more were unable to decipher an ISO 8601 date and time. Approximately 10% of the room knew the difference between Found, Not Modified, and Gone HTTP status codes.

Prepare more content than needed

I typically throw out 20% of my presentation based on the skill level of my audience and unforeseen time limitations. Throwing out my carefully-prepared slides was a big mental leap but it allows me to refocus my message on-the-fly to better match the conference, its topics, and its attendees.

Armed with my on-the-fly audience demographics from my earlier questions I may quickly skim over basics on my way to more advanced content. I may skip a topic already over-covered during previous sessions. Quickly flashing more advanced slides on screen on my way to my final presentation slide may also prompt conversations after my talk with more advanced members of the audience curious to hear even more.

I prepared a 10-minute talk for last year’s Web 2.0 Summit. I did not realize the organizers start the timer for your talk when the conference chair takes the stage for introductions, not when you reach the podium. John Battelle provided a nice introduction but my presentation was suddenly cut to 8.5 minutes instead of the prepared 10. I stuck to the basics for the Cx0 crowd and threw out the final 20% of my presentation.

Avoid card collectors

Some conference attendees are business card collectors. They don’t actually engage in conversation or ask questions on site but will come up to the stage to collect a new slip of paper from every session, perhaps for a more itemized expense report or a vast spam database.

After my presentation I like to stick around and answer 1-on-1 questions with session attendees. I place a small stack of business cards on one end of the stage for easy self-service while I continue to engage members of the audience 1-on-1. The conversationalist crowd is a bit thinner and may invite new participants.

Summary

Speaker content can and should adapt to the audience. Conference organizers should help their speakers better understand audience composition, but it’s also possible for a speakers to step up and deliver a stellar individual performance.