Especially as a content management system grows to have a large amount of content, it would be nice if you could do structured link checking. One of the problems with link checking in general is what to do with the reports once you get them. Of course, for a very small site you can easily scan an entire site with tools like LinkScan ($) and Xenu Linksleuth (free, but ads are put in the reports) or even monitor 404 requests and use single page tools like the LinkChecker Firefox extension. But with large sites you can end up with reports that are hard to know where to even start fixing links. This is especially true for CMS-driven sites: the same bad link may appear in only one piece of content that is displayed throughout the site. Or you could wind up linking from lots of content items to a url (possibly outside your control) that changes.

I envision getting a report with a list of the bad links, where a user (with appropriate global rights) could indicate the correct new link which would get reflected in all content items (or left menus or other components surrounding the content) that used that link. This list could be prioritized by the cumulative page views that contained that bad link, or by the number of pages that contained that link. Another approach might be to provide a prioritized list of content items that have bad links (preferably directly linkable to edit mode of that content item. At any rate, note that we're not talking about pages here but content items or links -- the user can quickly take action that will correct links on multiple pages. A long list of pages (specific urls) with bad links are confusing, but, more importantly, aren't as quickly actionable.

Here is how normal link checking reports look and how more useful reports might look:

Before / Existing Reports (where do you start with a report like this, where content items may drive multiple pages?)	Report indicating bad links where the user can immediately correct them (and apply the correction everywhere)	Report indicating which content items have the bad links(content items linkable to edit them directly)
http://badlinkone.com is referenced on http://example-site.com/page1, http://example-site.com/page35, and http://example-site.com/page102 http://badlinktwo.com is referenced on http://example-site.com/page1, http://example-site.com/page1023, http://example-site.com/page2439, http://example-site.com/page5192 Etc.	http://badlinkone.com is bad. Change everywhere now. http://badlinktwo.com is bad. Change everywhere now. Etc.	Bad link http://badlinkone.com is referenced on Content32 and Content507. Bad link http://badlinketwo.com is referenced on Content32 and Content209. Etc.

One possible way to implement this is to change all the urls into some logical link in your CMS. Assuming your CMS stores straight HTML rather than a more structured format, then any url the user enters could be changed to a macro (if the user could put in a hard link directly into the HTML without the system changing it, even if there was an option for creating a logical link, most users would probably just skip the logic linking). For example if the user put in this HTML:

<a href="http://hobbsontech.com>Hobbs On Tech</a> then the system would replace it with !link(123,"Hobbs On Tech")and put in its link repository that link 123 was http://hobbsontech.com. When the page was generated then the correct link could be replaced in the HTML (so of course the end user's browser should never see the "123" in the HTML). If the page linked to was in your CMS, then the macro could be different and just indicate the unique key for the content item being pointed to (this would depend on whether the context that the content appeared in was relevant). For example: !cms_item(123,"Hobbs On Tech")

Related items that a link repository might help with:

• Reporting on content use. A link repository would allow other interesting reporting, such as the most linked-to content items in your repository.
• Easily move content. In some cases, it may be easier to move content if you had a link repository. For instance, you may sometimes need to restructure your site resulting in the links changing. With a link repository, you could automatically change all the links so that the move did not result in broken links (of course this would work best for intranet sites where there were limited links outside your control to your content).

Of course, this would add complexity (and possible failure points) to a CMS. Do you think it would be worth it?

New sites with dynamic, interactive functionality using data from different sources and allowing the user to interact with the data are exciting to see (examples: geo.worldbank.org and carma.org). But how do we unleash this functionality so that non-programmers can create interaction like this? We have content management systems that allow more people to easily add content to sites. But I think we should be driving toward an environment where users can a) take data from a variety of sources and b) create interactive sites based on this data. Maps are the most prominent example, but interactive tables are also important. Let's review where we are now:

• We have sites already applying Google maps and other interactive functionality to various data sources (examples above).
• Programmers have resources/examples/documentation for creating these types of sites (see Programmable Web for example).
• Various APIs have been exposed for interacting and using data (examples).
• We have tools like Yahoo Pipes that allow advanced users (probably not needing full-blown programmer skills) to create mashups. That said Yahoo Pipes is now focused on consuming/dealing with RSS feeds (the Fetch Data Module is supposed to more general XML, I had problems getting it to do so -- if you look at examples using DC crime data, you see it's RSS with some customization). In addition, this is a hosted solution, so you're at the mercy of Yahoo if you host a mashup with them (I noted Yahoo Pipes having problems accessing feeds intermittently even in my brief testing).
• There are probably other similar examples of specialized tools, but I know of Swivel, which allows you to create your own graphs of data.

Here are the types of interactive functionality that I think we should be allowing non-programmers (let's call these folks "Interaction Publisher", riffing off the role of "Content Publisher") to create:

• Interactive data tables. Interaction Publisher should be able to point at one (or multiple) data source, and indicate which columns/attributes to display in a table. The Interaction Publisher should also indicate which attributes should be selectable (in pulldowns for example) be the end user. Of course some theming / design and annotation should be possible.
• Interactive maps. Interaction Publisher should be able to point at a data source, the attributes containing the locations, and what data to show for each location (along with the extent of the default map and formatting). Also, please can we get rid of the points / waypoints / circles that indicate arbitrary points that are used to indicate data for a large area (for example, a pointer to the capital for a country), and instead highlight the whole area (for example, the whole country). Ideally the Interaction Publisher will be able to indicate further interaction with the map (for example, displaying different layers of a map -- if not full-blown layers, then at least indicating different sets of waypoints to display).
• Custom data. The Interaction Publisher should also be able to easily publish their own data/content, and pull their data into an interactive feature (for instance, this could even be a simple search on a little database / resource center the user has). An extension of this would be including some mechanism for overriding other data sources data points (of course this should somehow be indicated on the map/table so it isn't misleading).
• Wizard-like functionality. The Interaction Publisher should not have to resort to XPATH, XSL, or programming in PHP / Perl / whatever.

Sounds nice -- but how would this be possible? One possible step is for institutions to expose their data in a consistent manner (at least each institution exposing its own data in consistently). This would involve something of a meta-API, where you are consistent about:

• Attributes that can be queried. Perhaps the list would be just topics and countries, for example. The topics lists should be something that the outside world will understand rather than an organization-centric list. If you have multiple topics lists, then it would be preferable if all systems were moved to a single topics list (even if that meant two topics lists per system).
• Simplicity and consistency in APIs. Perhaps all your XML APIs are at http://xml.example-domain.com/apis/ (with an html page just listing all the APIs there) and then APIs to different systems like http://xml.example-domain.com/api/documents and http://xml.example-domain.com/api/web with example calls like http://xml.example-domain.com/api/web/api-version=1&topic=agriculture.
• Consistent exposure of non-standard attributes. The issue of consistent query parameters was covered above -- this means that all systems are queried on the same parameters. But of course some systems will need to provide other attributes (such as, say, "Population"). This could be done in a custom namespace in RSS as the DC crime data (see xml) does in its Atom feed (which Yahoo Pipes, for example, can consume). This could be documented, and the consumer of the data could handle this.
• Custom databases would also preferably comply. Perhaps there could be an http://xml.example-domain.com/api/core/ for institutionally, centrally supported repositories and http://xml.example-domain.com/api/special/ for one-off databases. This would still allow easy access of data by Interaction Publishers.

Some potential ways of inching toward the goal of the non-developer Interaction Designer easily being able to publish dynamic, interactive features would be:

• Start by using javascript libraries. There are several javascript libraries out there (examples: Dojo, mootools, Prototype / Scriptalicious), but most seem to be too low-level (concentrating on opening/closing panels, transitions, and the like) to be useful for interactive data features. Possibly a library that has higher level features including interactive table such as EXT JS could be used as a first step. It would require touching some code, but perhaps a CMS, for example, could include in its documentation with code snippets indicating what needs to be replaced (for example, where to put in the url to the source XML).
• Create some simple wizards in CMSes. So that we aren't relying on, for example, Yahoo Pipes for hosting our interaction, we may wish to start including simple wizards in our CMSes. For example, one could be for interactive tables that just had one data source and three columns.
• Push for stronger hosted interactive feature builders. For example, Yahoo Pipes perhaps could include some of the features mentioned in this email (for example, a tool for creating interactive maps, or a tool for creating a pulldown of options to drive a Google map.

Here's a little chart displaying some of the ideas in this post (also see pdf version):

I'd really like your comments on this post. Specifically:

• Is the role of Interaction Publisher important?
• How could we enable this role?
• What ideas above do you think would work and which would not work?
• Is their a need for a separate generic standard XML from RSS feeds, or should an institution's RSS just be extended to include custom portions?

 

There are already various sites comparing features of content management systems (for example the CMS matrix), so this post aims to help set a framework for selecting a Content Management System (CMS). Aside from standard things to keep in mind when selecting a technology, there are some particularly important items for setting the tone of your CMS selection:

• Standardization / Governance. Is one of your objectives to standardize the look and feel of your site, or to try to ensure there's a consistent quality across your site? If so, then before you start moving into the new system then deciding who will make the decision of what goes up and how the decisions will get made is important. Sure, an advantage of a CMS is that anyone can publish, but this can lead to inconsistent quality. I'm not just talking about how workflow: for instance, who makes the call about adding a whole new site section?
• Stakeholder buy-in of objectives. This one is of course part of any technology decision, but some key factors in deciding about a CMS are: a) if you've decided to standardize aspects of your site, make sure everyone is bought in (otherwise people will try whatever they can to get out of the standard), b) if people's jobs are going to change (for instance, people that are doing hands-on HTML coding may not be doing that anymore), then is everyone clear on this?
• Envision key use cases. After you're in the middle of migrating your content/systems, you may lose sight of why you undertook this in the first place. Laying out key use cases in advance allows you to both not loose sight of the goals and also let's you more easily claim victory. Key use cases might be something like "Will be able to allow any staff member to publish a piece of content, resulting in it automatically appearing on the home page as well as the relevant country page, and also appears in country's RSS feed and email alerts". Of course, you also should list key use cases that you don't want to go away like "Compare statistics across different areas of the site in a consistent manner."
• Make sure everyone understands the complexity of a move to a new system. See this post that lists some of the complexity.

These are some of the particular factors to consider when selecting a CMS:

• Tagging. For a large institution, you may have issues keeping consistent quality in your tagging (and you may wish to consider an automated concept extraction tool to help in the tagging). At any rate, you will want to think about a method of tagging that will work for everyone (and ensure that your system will support this).
• multilingual/internationalization support. See this page that describes different levels of multilingual support. Some more advanced types of features to consider are Administrative Title and Interleaving Languages.
• distributed or centralized content entry input. This relates to the issue of standardization above.
• community/support.
• multiple site support. If you need to have multiple sites, what kind of functionality do you need? For instance, does content need to flow between sites? Do the different sites need to enforce a consistent look/brand?
• integration with other systems or all-in-one. A key decision will be how you are going to integrate with other systems, and, if integration is not as important (for instance for a smaller organization), then ensure that your solution supports the different functionalities you need.

With everything on the web moving so fast now (who knows when Web 3.0 will be the next thing we're all moving to), consider moving to a CMS environment that will allow quick innovation and new functionality. Some specific approaches to this:

• Try to pick a CMS that is innovating quickly. Of course, what you really want is to pick what CMS will be a winner in the future, but the best we can do now is pick a CMS that is quickly adding new features. Looking at lists like Joomla's extensions page for any CMS that you're interested in should help with this. Of course, it needs to be easy to add any new modules/extensions when they are released.
• ease of upgrading to new versions of the core CMS. Obviously hosted, SaaS solutions have an advantage here.
• ease of writing your own new functionality. Would the CMS allow your team to program (in some lightweight language like PHP for example) their own new functionality? If you don't have the skillset, is there a pool of developers outside your organization who could help? Is there useful documentation on how to write your own new functionality?
• support to expose/share data. We have RSS as a mainstream feed now, but what about richer XML exposed for more structured data? More and more, we'll need to support people outside our organizations utilizing our content/data to write functionality on their own sites, combining your data with other organizations' data.
• integration with outside systems. If a CMS already has integration with other types of systems (for instance, stats, newsletters, email alerts, membership databases, etc), then it may be easier to move to future leaders in these different spaces.

You know when it's time to move into a new house or apartment, when you look at the stuff you need to move and think "Why in the world do I have this bread machine? I haven't used this in years and I forgot I even had it." Or you dread moving your old clunker of a TV, thinking of the new fancy flat-panel TVs? Well, it's the same thing with migrating to a new system, for instance into a new content management system. Only it's harder. When you're moving and you're pressed for time, you may just start tossing stuff into boxes to be moved, even when you know you don't totally want all the stuff (one reason: you'll need to negotiate with a spouse about getting rid of something, and there's no time for that). This isn't that big a deal, since it's just moving more of the same stuff. Or, if you have a huge sectional couch that won't fit in your new place, then perhaps you can just sell it to the next homeowner. When you're moving content, you have all sorts of extra things to think about including:

1. It's not just content. Content on a site doesn't just live in some abstract ether, but it is linked into a larger site context. This includes left navigation, headers, footers, and special site behaviors. Of course moving the site context of a simple site like hobbsontech.com would be relatively easy to move (re-creating the the menus, configuring the overall style, etc), but the more sites you have, the more there would be to do. This is especially relevant for sites with a lot of custom dynamic functionality. For instance, if you have comments on your current site's content, then you'd have to figure out how to embed it in the new framework (or just leave it behind). Chances are you have a lot of functionality distributed throughout your site that may even be hard to inventory.
2. Metadata and taxonomies. You may have to re-create taxonomies in another system, and there may be incompatabilities you have to work through.
3. Internal references to other pieces of content. Your content probably refers to itself (for instance, a press release may refer to your product description page). This somehow has to be reflected in a new system.
4. Structured content. You may have structured content (for instance, a document that has multiple chapters), which you'll need to figure out how to handle in the new system.
5. Outside references to your content. Other sites, as well as search engines, will have links to your content. You'll need to have some strategy to deal with the links from external sites.

In the end, a lot of this has to do with the web of information that's involved in the content of a web site. And this isn't counting the types of technical issues that would come up with any technical migration (differences in size limits for fields, encoding differences, etc.). Of course there's the issue of why you even have all this stuff to move in the first place (and the more stuff you have the more hassle it is to move). This blog entry has focused on why it's difficult to move all this content, but of course one of the morals of the story is to have less stuff in the first place. In the case of the web this would involve better governance of what goes on the web, and clearly defining what the focus of your web site should be. Hopefully, just like when moving houses, any discussion of moving content would also include discussing what stuff you need in the first place. Unlike houses, having extra or duplicate stuff doesn't just inconvenience you but it is a disservice to your users. I'll leave the issue of the old TV and desiring the new flat panel to a future post (on survivorship bias).

Now that I've been using Drupal for a month (Drupal is driving this site), I thought it would be a good time to write up my impressions of this open source CMS. Obviously this is before I know the tool in depth, and also before I become jaded or a zealot. For reference, over the past seven years I've worked with a couple custom CMS systems, both driving very large and small web sites, and I also have used Wordpress (not a CMS) for two other personal blogs. Also, my emphasis using Drupal for this site has been as a blog so I haven't fully explored all the CMS features. I really like Drupal for many reasons including:

• Very easy to add features to a site. You download a module, set some parameters, and then you have a new feature (instructions). Examples of useful features that I've added easily to this site: feedburner redirects (Feedburner module), CAPTCHA checking for form submission to block robots (CAPTCHA module), email forwarding posts (Forward module), SEO and human-friendly urls (pathauto module), links to digg and other services (Service Links module), and full name listing as author for blog posts (Authorship module). That's not counting the really useful core modules that I've enabled for the site like the ability to pull in / aggregate other feeds (Aggregator module), comments (Comments module), automatically pinging services such as Technorati (Ping module), search (Search module), some web access statistics (Statistics module), the ability to create different taxonomies/categorizations (Taxonomy module), and the ability to automatically have features of the site turned off under high load (Throttle module).
• Built-in performance and throttling. You can set how aggressively you cache, and selectively set which features get turned off under higher load.
• Upcoming features. Since Drupal has gained some momentum, one feels the inevitability of new features being added as time goes on. Also, since it looks to be easy to add new modules, you could add a module yourself if you wanted.
• Nice modularization. I haven't read the documentation on how to develop modules, but, just seeing how modules work when installing them, the pluggability of modules seems very nice. Once installed/enabled, modules aren't just stovepipes. For example, when in admin mode the CAPTCHA module shows a message next to every form on your site asking if you want CAPTCHA there. Also, it appears that modules can easily write to the log screen, and are all controlled from the same core administration screen. When writing a post, the different options are embedded right on one page (for instance, the pathauto module automatically indicates, and lets you override, the alias it plans on giving your post).
• Flexible themes. I'm using the Garland theme, and I liked how you can set your own colors and other things are parameterized like your logo.
• Multiple taxonomies. You can create easily create your own category lists.

Perhaps above everything else, it just seems that the details have been done very nicely in Drupal. Overall a site in Drupal appears to be easy to administer. These are some items that weren't as smooth: a) didn't get the Backup module to work quickly enough (was faster just to use scripts to do it rather than get the module to work), b) still don't fully understand the file upload/download environment (especially for counting the downloads), c) by default you're in raw html editing (yes, you can install a web-based HTML editor, but the ones I tried so far don't seem very useful), and d) getting the transparent logo needed for the Garland theme working quickly. A note to people currently working in a large, complex enterprise content management environment: I highly recommend playing with Drupal to get the creative juices flowing and also to work in a less constrained environment. Also, it's nice to work from a clean slate on a new site (although I've already bumped into a place where I wish I had set up the site differently in the first place). But of course working in an enterprise environment has a host of other requirements that have to be dealt with (for example standardized look across sites, security, single sign on, integration with internal repositories, existing systems, and standardized administration).