But isn't it all data?  Although web content certainly is an important type of information available on a web site, Data needs to be treated differently.  Here I'm talking about Data with a capital D -- I thought the Wikipedia description was good: "Data refers to a collection of organized information, usually the results of experience, observation or experiment, or a set of premises. This may consist of numbers, words, or images, particularly as measurements or observations of a set of variables."  Here are some of the ways that Data is different:

1. People expect Data to be available in different Formats.
2. Users want to manipulate the Data.
3. You don't totally control your Data, since it is available in different Channels.

There are several implications of this including:

• Formats.  You may wish to standardize the formats that your data is available in.  Is all your data always available in csv (if that's what you standardize on)?  This includes both the formats themselves (Excel, Stata, etc) and also the method by which the data is requested.  For instance, is there one place that users can directly get all your data?  Directly doesn't mean some thin layer with links to databases each doing its own thing.  An example consistent format would be a web service with a published set of parameters by which the data could be requested.  Ideally, all the institutions data would be available from this one web service.
• Manipulation.  Sometimes people just want to see your data, but usually they will want to manipulate the data.  By providing your data in consistent formats, then it will be easier for your users to utilize your data.  Other users will expect that *you* provide the tools for manipulation of your own data. 
• Channels.  Ideally you will work to directly feed data to primary channels.  For instance, if you feed data directly to services like Swivel then you both get more use out of your data and also can ensure your data is available in its highest quality (not watered down by other people copying and pasting your data, for example). 

Since posting Enabling the Interaction Publisher, I've done some more research on mashup builders.   I found a lot of excellent resources (see end of this post for a list), but felt that a summary of what's out there would be helpful (this is what I would have liked to see when I researched this).  Although lists of tools exist, I wasn't getting a sense of the overall space.  Note that I believe that mashup editors are only part of enabling the interaction publisher: the other parts being standardized access to data (particularly all data from one institution) and deeply embedding these types of tools in content management systems (for instance, using topic-driven templates already in your CMS to drive mashups that are driven by those topics). 

Some of the particular variables that are probably most relevant in what tool you should use for your particular application / situation:

• Is the expectation that you will quickly have a mashup, or that you are building out an infrastructure for your institution?  Does this need to be behind your firewall?  Do you need guaranteed uptime?  Basically, do you need this to be hosted (quick and easy, get-me-started-now) or will you build out and manage the infrastructure?
• Do you want the end result of your efforts to be another data feed, a map, or other types of user interactions like data grids?  What should the output be?
• What types of inputs do you need to pull: structured (if so, generic XML, or just RSS/atom specifically), unstructured (like web pages), or direct database connections?
• How are the mashups built?  Do you need totally non-technical people to create, or do you just need to support your power users?  What is the mashup building environment for building the feeds (create-a-flowchart like Yahoo Pipes and Microsoft Popfly?)? 
• What are the browser requirements for people to use / consume your mashups (for instance Microsoft Popfly requires both Silverlight and recent versions of just Firefox and IE)?  
• Can a mashup be embedded into an existing web page? I didn't find enough useful information on this to fill this out meaningfully in the table below, so perhaps this is something to add in the future.

Although I was hoping to put together a fuller spreadsheet of all the tools out there, I selected a subset that was either easy to start using or had very clear documentation (I used all to at least some extent but AquaLogic Pages).  At any rate, here's a brief (and admittedly incomplete) table comparing different mashup builders along the criteria listed above (please comment with any corrections/additions):

I found the easiest to use were Apatar, StrikeIron SOA Express for Excel, and Yahoo! Pipes (if you just want to play with something to get a feel for mashup editors, I'd recommend starting with these), although each is entirely different.  Although QEDwiki has a great intro video and appears to be able to do a lot, I got the least far actually using it.  Although I'm sure that someone that knows Dapper inside and out could create a feed from scraping pages, in practice I didn't manage to get it to do the two tests I tried (for example, in trying to scrape country pages from the World Bank site, it wouldn't let me since the different pages were coming from different domains although they were driven from the same CMS).  Popfly seemed interesting, but doesn't appear to be geared toward the enterprise and has very specific browser requirements.  That said, these were just initial impressions (and initial ease of use may be irrelevant for a particular application) -- the main purpose of this post was to put together the matrix above just getting a feel for the current overall state / scope of mashup editors. 

Here are some excellent resources on mashup editors:

• blog post: Mashups: Mashup Maker Smackdown
• Enterprise Web 2.0 blog
• Mashups category on Naked Open Source blog
• blog post: Why Mashups = (REST + ‘Traditional SOA’) * Web 2.0
• blog post: Mashup Business Models
• Programmable Web site

I've been thinking about and researching how an institution can share its data, documents, and other content.  Obviously your data and content is already exposed via the web, but providing the data in a more structured way allows more users (both internal and external) to manipulate the data in interesting ways, for example in mashups.  There seem to be a few ways to share data from an enterprise with a lot of content:

• Straight RSS/Atom.  Although straight RSS/Atom (with no custom extensions / namespaces) may not be that interesting, it's obviously a useful way to get your content out there.  Typically straight RSS/Atom is fairly time-based and might in effect show some history (news items like "John goes to work" and then "John goes home") rather than some state (like "John is now home"). 

• Common repositories / services such as Swivel and StrikeIron.   Rather than exposing your data/content directly to the outside world from your site/servers, you can use an intermediary.  Swivel allows users to create their own graphs on fata from either official sources or any user-supplied data.  StrikeIron is built into mashup editors like QEDwiki, and also has built an extension to Excel to call their services.  You probably would want to provide data to these services through an API of your own, but you could get started with Swivel for example by directly uploading the data. 
• Specialized XML formats for particular types of content.  Examples include OpenSearch for search results and SDMX for statistical data.  These specific XML formats both allow a level of sophistication for people specializing in your type of content and allows tools built for this type of data to consume it.  This fits in with the following item, which, for historical reasons may or may not be XML-based. 
  
• Institution-to-institution services.  Sometimes you need to provide a point-to-point interface with another institution.  In that case, you may need to support all sorts of unusual formats and delivery mechanisms.  Hopefully you could leverage your various systems' web services to just transform the data into the formats you need. 
• A common API that your institution follows across all types of content.   This one is the most interesting to me and one that I alluded to in my previous post on interaction publishing.  Especially if your institution has various repositories, one possible approach would be to slap up a page that has links to the different instructions for referencing each.  But, to make access as easy as possible, a common API with consistent parameters that can be queried against all systems would be preferable (for instance, queries such as "give me all your documents and data on Chad" via url requests like http://xml.example-domain.com/apis/type=docs,data&country=td).  Potentially the returned XML could be in a simple format such as RSS extended with a custom namespace (so that other tools such as Yahoo Pipes, and even feedreaders, could easily consume the data).
• Microformats.  Probably most useful to future browsers or other tools like the Firefox Operator extension (or for services that crawl sites such as Google), microformats allow you to just change your existing HTML a bit to expose very common types of data like address and calendar events.  For example, instead of your HTML having "100 Main Street, Anytown, USA" it would be marked up as "<div class="adr"> <div class="street-address">100 Main Street</div>, <span class="locality">Anytown</span>, <div class="country-name">USA</div></div>" and then define the CSS to show it as you wish.  For example (with sloppy CSS):
  
  100 Main Street

  Anytown,
  

  U.S.A.

  See how this page appears in Firefox Operator (also notice the tagspaces):

Sometimes you need to pull content from multiple systems into a single page, and you want to pull from both systems based on some metadata, perhaps by topic.  For instance, let's say you have a site that you want to pull data from a document repository and a news archive, and you want the the user to use a pulldown to select the topic they want to filter the content by (for example, by "Politics", "Entertainment", "Travel", "Europe", and other topics).  Sometimes out of the box the two systems will share the same list of topics, but more frequently than not they will not.

One deceptively simple approach when systems do not share the same list of topics is to have some sort of mapping between the taxonomies of the two systems (for instance, "Travel" = "Vacations", "Politics" = "Domestic Politics", "Europe" = "EU", etc).  I fairly frequently hear something like this when discussing integration between different systems: "We have a mapping between these topics, so there shouldn't be any problem."  But just because you have a mapping doesn't mean that it will be satisfactory for combining information from multiple systems.  I thought it would be helpful to think through the issues some and write out some examples.

One taxonomy's controlled vocabulary being more specific than another

Let's say you've got some content in two systems that you want to pull into one page.  Perhaps you want to find out all the fathers in both systems.  If the taxonomies available were the following (and you didn't have other metadata on gender, for example), then you could not do this:

A simple and meaningful mapping between the two would be something like this (allowing you to find all the people across systems that are a parent, for example):

Father or Mother - > Parent
Sister or Brother - > Sibling

Note that the other direction makes no sense (it's tough to be both a sister and brother, and you wouldn't know which to pick when translating between systems).  So, although you may have a mapping between the systems, it does NOT neccessarily enable the types of queries you want to do.

A slightly more realistic example

Of course that was a simplified example to illustrate the point, and you usually have overlapping, something like the following (still a forced example though):

Let's say these two systems had a selection of companies tagged to these controlled vocabularies.  What kinds of queries would probably be meaningful?

• All companies in Silicon Valley
•  All companies in East Bay
• All companies in Marin County
• All companies in the San Francisco Bay area

Obviously, you couldn't query on cities since system two has virtually no cities.  But what about San Francisco?  Isn't that on both lists?  Although at first blush it may seem that you could find all companies in San Francisco across both systems, looking at the list more carefully it becomes apparent that they almost certainly have different meanings: the first taxonomy only has the broad San Francisco Bay Area and then cities, and the second taxonomy is just listing areas within the San Francisco Bay area.  So San Fancisco in system 2 probably includes San Francisco proper as well as, for example, South San Francisco.  So you can do this query (but *not* query on all companies in San Francisco):

• All companies in San Francisco and the immediate area (including South San Francisco)

Part of the issue is that often you have much larger taxonomies that are more difficult to analyze (for example, for a taxonomy that includes all cities in California, or the US).  It would be very difficult to go through and determine the meaning of the different values of the taxonomy.

What to do?

In practice, you probably won't be able to deeply analyze all the mappings between your systems, so you'll have a mapping but might only have a feel for how good it is (and in what direction).  Perhaps the most dangerous mapping, and one that is hopefully fairly easy to identify, is from a more general taxonomy to a more specific one (the first example above) and should be avoided entirely.  Of course if the systems do not even having taxonomies that are close, then this will be obvious and require changes to at least one of the source systems.   The second example above (overlapping but not quite lining up) might a type of taxonomy matching that's not be that bad but just require documentation/labeling (just use "San Francisco and South San Francisco" in a pulldown to select areas of the Bay Area) and careful design (obviously don't allow the user to select a city if you need data from both systems, or you could clearly show in the results that the information is just from System 1).   But figuring out the relationships between the taxonomies might take a lot of work.  Some potential general approaches: be careful and a) where possible try to access globally well-understood and clear values (like zip codes, lat/long, ISO country codes, etc) rather than fall into the trap of just trying to use two taxonomies since they're called the same thing (this is probably easier in something like a location than a topic), b) force all systems to tag to a neutral reference source (this could, with a lot of work in defining rules, be automated with something like Teragram for example), or c) seek out a metadata expert since they have some best practices of mapping between flat or networked taxonomies.  Also, when you are designing a system in the first place (even before being faced with a new integration), try if possible to use metadata for your content to well-understood values (especially easy for geographic tagging).

Very large sites supporting a large number of units/stakeholders can easily turn into a hodge-podge of styles, user interface elements, and quality. One of the toughest discussions with clients, however, is why they can't do more customization (even if one of the core requirements of the system is to help enforce standardization). What are some of the reasons *not* to standardize:

• specific business needs of different groups (not to be confused with a group just wanting to differentiate itself somehow, for instance with a different look, that does not help the web visitor at all)
• professional development (for instance a developer might be interesting to do a mashup)
• personal expression (liking particular colors for example)
• experimentation (don't know in advance what's going to "stick," so try a variety of things)

In my opinion, the first and last reasons are the most compelling (and the third not being a good reason at all for an enterprise-wide system), although one of the problems with experimentation is the frequent expectation that an experiment could quickly be rolled into the normal standardized platform (that's probably a post on its own!). Here are some reasons *to* standardize:

• consistent brand for the user ("am I still on the same site? Is this high quality content?")
• consistent UI for the user ("do I know how to use the site?")
• better support for new site admins or transition of support between sites
• single sign-on. It's confusing for a user to have various accounts with the same institution.
• standard statistics. Different statistics packages can have entirely different ways of counting something as basic as a page view. Standardizing no a statistics package can help ensure you're comparing apples to apples in your web analysis.
• better search. If everyone does their own thing, then there may be more fragmented information which would mean search results aren't as good.
• stability / support. As anyone who works with software/systems knows, the more functionality or special customization you put into a system, the more effort it takes to maintain it. Also, the system will probably be less stable. This one is also very tough to discuss with a client (and another probable future blog post) since they tend to only see their particular need.

Some possible methods of standardization:

• Governance. There needs to be a group with the power and influence to say "no" to requests that undermine the quality of the user experience of the site at large. This ideally is not the technology group since there would appear to be a conflict of interest.
• Clearly define exactly what is inside the standard and what is outside.
• Technology. The content management system used to manage the site can be set up such that users can only make changes that comply with the standard.
• The right level of customization. Standardization shouldn't be an excuse to totally control every aspect of everyone's sites or to not allow any innovation.
• Hooks into core shared functionalty. You may decide that a single sign on for users of your site is desirable. If so, then perhaps the system could be set up with an API such that tools developed and commissioned by other groups could work with the core functionality.
• Standardized access to data. Ideally, you could define a standard method of each system exposing its core data, that even people outside the institution could utilize for mashups, etc. By providing the data in a simple XML API, this could facilitate both internal and external usage of data.
• Another potential approach is to have separate branding for the official, blessed content and for the organization-centric content. For instance, you may have multiple units in your institution all looking at the topic of taxes. Ideally you would have one official web site that makes sense of your institution's view of taxes overall, and preferably this would pull information from all the units. The various units still may want their own site, but this is less useful for the end user -- so perhaps these units could have their own sites branded differently (and perhaps all requiring a standard link back to the official site) to clearly indicate it is the view of a particular unit with your institution.

Of course, all of these are easier said than done when trying to get a large number of units into the same system, but perhaps some of these could be initiated even after a large suite of sites have been implemented in a central content management system.