SharePoint 2010 introduced managed metadata for the first time – the Managed Metadata Service (MMS). You can create a hierarchy of terms (metadata) and then use those terms to classify content stored in SharePoint. If you decide to rename a term, all items classified will be updated to reflect the new term. That’s the managed bit.

For an overview, please read an earlier blog post: SharePoint Managed Metadata Overview (Nov 2010)

Whilst the MMS is a great addition to SharePoint, it doesn’t cover all taxonomy requirements. This post will briefly explain the different types of taxonomy and what SharePoint can and can’t do to implement them.

The short version:

The SharePoint MMS can create taxonomies with a single hierarchy or multiple hierarchies using term sets. It can also be used for folksonomies by using a keywords list of tags. It can nearly do polyhierarchies, by reusing terms across term sets, but with limited uses.  If you have a deep taxonomy hierarchy to implement, you may need to add FAST to your SharePoint deployment.

The details:

Types of taxonomy

A taxonomy is a hierarchical form of classification. You define a hierarchy of metadata terms that can then be used to classify and describe stuff.

There are three main types of taxononomy:

• Single hierarchy
• Multiple hierarchies
• Polyhierarchies

And then there’s the informal rebel, the folksonomy.

Single Hierarchy

The simplest of taxonomies – a single hierarchy containing all the terms you plan to use. In this example, I’ve started with Flowers. That means the only things I’m going to classify are flowers. If I want more in a single hierarchy, then flowers would be a sub-class with a parent – e.g. plants, which may have a parent – organic material, that may have other sub-classes like mammals, which would have a sub-class for primates etc.  A single hierarchy is simple in theory but quickly becomes complex as you try to organise all possible terms.

Which is why, unless you are a library or garden centre, you end up with…

Multiple (Faceted) Hierarchies

In the digital world, we don’t need items to be located in only one place. We can use multiple hierarchies to describe them.  I might be looking for tulips. I might be looking for anything with red petals. I might want flowers that open earlier or later in the year for my garden.  Thanks to multiple hierarchies, I can find what I want without having to know where to look first.

Polyhierarchies

The most complex of taxonomies – a polyhierarchy is where a child or sub-class has two or more parents instead of just one.  Jasmine can be in the form of a shrub (small bush) or a vine (tall climbing plant).  Instead of being listed twice, the word is listed just once and linked to the two different parents.

Folksonomy

A folksonomy is the informal version of a taxonomy. It is simply the use of tags to describe objects. An object may have many tags. A tag may be reused for many objects.  There is no hierarchy – all tags are equal and there is no relationship between them.  You don’t define the tags first. You create them as you go. Once a person has created a tag, it’s added to the list.

A folksonomy can seem chaotic and confusing. Its success on the Internet, on sites such as Flickr, has been because it is a lot easier and quicker to simply tag items than have to select from a pre-defined hierarchy that may not match your vocabulary (imagine if plants were described using their Latin names).

SharePoint and Managed Metadata

SharePoint’s MMS can be used for both taxonomies and folksonomies.  Within Central Administration (and can also be accessed via Site Collection Administration) is the Term Store Management tool.

Keywords

SharePoint’s folksonomy is a group called Keywords:

When any item is tagged, either using the Enterprise Keywords column available in lists and libraries or by tagging site pages, the tag is added to the Keywords group (1. above).  As you start to type a tag, SharePoint will automatically suggest matching tags as you type (2.), to avoid having multiple spellings of the same word.  You can manage your keywords (3.) If you want to clear out unnecessary tags, you simply delete them.  If you also have a taxonomy, you can move popular keywords into the term set and make it part of a formal hierarchy. This is a great way to develop an effective taxonomy rather than a taxonomy filled with unused terms.

Term Sets

SharePoint uses term sets to create taxonomy hierarchies, with the following levels:

You create a group to contain one or more term sets.  A group with a single term set would be a single hierarchy. A group with multiple term sets would be a multiple or faceted hierarchy.

Within a term set, you create terms. Each term can itself have terms. In the image above, we have a group called Taxonomy Examples (created for this post). The term set is called Colours. The first level of terms contains Blue. Beneath blue is a second level of terms: Royal Blue, Cambridge Blue and Navy Blue.

Terms can be re-used across term sets, which sort of (but not quite) enables polyhierarchies.

In the image above (click on it to view in larger detail), I have a group called Plants. It contains term sets for Shrubs and Vines. The Shrubs term set contains a term called Jasmine. The term has been re-used and linked to the term set Vines.  If you look at the right side of the page,  displaying the term properties for Jasmine, in the box ‘Member of’ we can see the term is linked to two term sets. If I rename Jasmine, both term sets will be updated.

To create this polyhierarchy, the group Plants is the class, and the term sets are the first sub-classes. It has to be done this way because you cannot re-use a term within the same term set.  But this creates a challenge because only the built-in Enterprise Keywords column allows users to pick terms across groups and term sets. Managed Metadata columns have to point to a specific term set.

Whilst you cannot re-use a term in the same term set, there is nothing to stop you having duplicate terms, i.e. two or more terms with the same name. The downside is that you are creating extra terms, each with its own properties, which is extra work to manage and can be confusing for people.  But duplicates enable you to use a single term set that works better with the Managed Metadata column.

It’s probably easier to demonstrate:

The image above on the left is displaying the term store within the MMS. At 1. is the polyhierarchy example: A group for Plants with term sets for Shrubs and Vines, each linked to a single term, Jasmine (it has a slightly different icon to show it is linked to more than one term set).  At 2. I have created the alternative scenario, a single hierarchy with duplicate terms. This time, Plants is the term set with first-level terms for Shrubs and Vines. Each has a second-level term called Jas.

The MMS is used in two ways within SharePoint lists and libraries:

• Enable the built-in Enterprise Keywords column, which points to the entire term store (all groups and term sets).  It is always a mult-value field (people can enter one or more tags) and if the words entered are not already in a term set or the keywords group, they will be added to the keywords group. (Side note: the label ‘Enterprise Keywords’ can be confusing since it points to all groups in the MMS, not just the Keywords group).
• Create columns of the type ‘Managed Metadata’ which must be pointed to a single term set. However you can then specify if the column is to contain only a single value versus multiple values, and whether or not people can create and add their own tags or must choose from the list provided.

The top-right image is displaying the properties form for an item in a document library, with the built-in Enterprise Keywords column (3.) enabled. As I type in Jas, three suggestions are offered – Jas in the term set Plants, under the term Shrubs; the duplicate Jas in the same term set Plants, under the term Vines; and, Jasmine from the group Plants, displayed once but showing both term sets that it is linked to. In this image, the correct method would be to use the Polyhierarchy (from 1.) so that I can classify the document as about Jasmine, regardless that it is both a shrub and a vine.

The bottom-right image is displaying the properties for a Managed Metadata column (4.). When you create a column of the type Managed Metadata, you have to point it to a single term set.  This is where the polyhierarchy (1. in the image above) fails in SharePoint because I can’t point the column to the group Plants, I have to pick one of the term sets within the group, either Shrubs or Vines. I don’t want separate columns for each sub-class within plants. My alternative approach (2.) with duplicate values does work because Plants is the term set. But then how do I decide which Jas to pick to classify my document?

And there’s another gotcha.  One of the most useful reasons for implementing managed metadata is to refine search results.

The image above is displaying a standard SharePoint search results page. On the left side of the page are the search refiners.  I have two term sets listed – Department and Products.  Under Products, I can refine results by SharePoint, Content and Apps.  Here’s the rub. My term set hierarchy is as follows:

• Term set: Products
  • First-level term: SharePoint
    • Second-level term: Content
    • Second-level term: Apps

The standard search refiners in SharePoint do not display a hierarchy under the term set, it is simply Term Set as the heading, and all terms listed beneath, regardless of their position in the term set hierarchy.  A deep hierarchy within a term set does not work well with the standard search refiners, they are better suited to flat hierarchies – term set as the class with just one sub-class of terms.  In this example, I should either keep the term set Products but have just one level of terms – the product names (SharePoint, Office etc.) or make the product the term set if I want to classify by feature, e.g. term set: SharePoint, terms: Content, Apps etc.  The alternative is to upgrade my SharePoint deployment to include FAST, which is Microsoft’s advanced search and taxonomy tool. FAST includes the ability to configure your search refiners to match your taxonomy hierarchy.

In conclusion, the Managed Metadata Service enables you to create all three types of taxonomy: single hierarchy, multiple hierarchies and polyhierarchies, as well as the folksonomy of keyword tags.  However, you can only use all these methods with the built-in Enterprise Keywords column, which cannot be locked down if you want to restrict what words can be used to classify content.  The Managed Metadata column can be locked down but only works with a single hierarchy, meaning you either have a deep single hierarchy or have to create separate columns for each term set.  Search refiners work best with multiple (and flat) hierarchies, they do not work well with a single deep hierarchy.

To design the MMS effectively for search, you have three choices:

1. Flatten your taxonomy:

• Use multiple term sets, each with a list of terms – i.e the term set is the class, the terms are a single sub-class. Do not have a hierarchy deeper than that.
• If using the Managed Metadata column, you will need a column per term set.  Essential if you need to restrict people to picking from your defined lists of terms.  Aim for fewer term sets. This will likely mean not going to deep with your taxonomy (in my Plants example, I’d need to either drop the final level – Jasmine, or drop the sub-clases of shrubs vs vines vs flowers, and just have a term set of Plants with terms for the type: Tulips, Roses, Jasmine etc.
• Use the Enterprise Keywords column when you don’t mind people adding their own tags (will be added to the Keywords group). Matching terms will be suggested to help avoid unnecessary duplicates.

2. Upgrade your deployment to FAST

If your taxonomy is too important to flatten but you still want to use search refiners (you should, reasons for classifying content usually include making information easier to find), you need to consider whether or not to configure FAST in your SharePoint deployment.  FAST includes advanced search refiners that include displaying a deeper taxonomy hierarchy.  It also includes the ability to auto-classify. But may involve additional licenses so you need to factor the cost into your project.

3. Use a specialist add-on

The third option is to use an alternative (non-Microsoft) taxonomy solution that can be added to your SharePoint deployment.  You are likely to also require an auto-classification tool.

If you are immediately thinking that option 1 is not good enough for your taxonomy needs, before heading to the second or third option consider if your taxonomy is good enough for your organisation’s needs.  Folksonomies have succeeded where taxonomies have failed because of their ease of use.  It’s no use having a deep and detailed taxonomy hierarchy if people make mistakes, choose the defaults or just click the first in the list to get through classifying their information. And auto-classifiers are a long way from perfect.

No metadata is better than bad metadata.

Related blog posts

• SharePoint Managed Metadata Overview

SharePoint has always had some form of metadata capabilities. However the latest version includes the first attempt at managing metadata. This post is a brief introduction and overview.

First a quick (and very basic) primer on what is metadata. It is simply data about data – the stuff that describes a document (or any digital artefact). The title, author and date modified are all metadata properties. You can define your own properties such as category (what’s it about) or status (public, confidential). Metadata typically comes in two forms: a hierarchy of terms known as a taxonomy or a mesh of terms known as a folksonomy. A taxonomy is good for controlled classification and organising related terms – for example, a list of departments and teams within departments to identify ownership of documents. A folksonomy is good for flexible classification – letting people choose their own tags to describe the contents of a document. Folksonomies can lack the consistency of formal hierarchies but enable a wider variety of unexpected terms to be used.

Keeping the definitions simple, metadata serves two purposes: 1. Managing content – such as identifying when documents have expired and moving them to an archive or reviewing all confidential items; 2. Finding content – ‘Show me all posts written about SharePoint’ for example. If metadata doesn’t make content easier to manage and/or easier to find, question why you need it.

SharePoint 2010 works with four different types of metadata:

The first two are also available in all previous versions and editions of SharePoint. The second two are new features in SharePoint 2010 (and only available in the Server editions, they are not included with the SharePoint Foundation Service that is included with Windows Server )

Document Properties

…are actually nothing to do with SharePoint. They are managed and stored within documents themselves. Some properties are common to most file formats: title, author, date created, date modified and document size. Others are unique to different types of content, such as duration time for a media or audio file. Whilst the properties are part of the document, SharePoint can make use of them. Properties within Office files (Word, Excel, PowerPoint) will be automatically promoted into SharePoint columns and indexed. Unfortunately, the same cannot be said for other file formats such as PDFs.

For most applications, you can access and view/edit the properties from the File menu. A common search gotcha is creating documents from existing files instead of blank templates. Certain properties, such as title and author, are automatically populated when a document is first saved from a template. But these properties do not automatically update if someone uses an existing document and saves it under a new name. It means title and author can have little relevance to the actual content of the document.

List/library Properties

…go by many names, columns being the one used in the SharePoint user interface (UI). Columns are used in SharePoint lists and libraries to store all the different properties used to describe a document, including any extracted from the document itself. SharePoint provides a range of pre-configured columns. Or you can create your own using the many different types of column available. You can even create calculated columns, such as setting an expiry or review date 12 months after the date last modified. Because the properties are managed within SharePoint, they can be applied to any digital item stored in SharePoint.

The image above shows a SharePoint library containing a PDF file, a Word document and a Windows Media file. Using SharePoint columns, the same properties can be applied to all types of files. The added feature you get with Office is that you can access the SharePoint properties from within Office as well as the browser, as shown in the image below:

Managed Metadata

…is a new feature in SharePoint 2010. It runs as a service just like search, user profiles and other features that benefit from being managed centrally. The Managed Metadata Service (MMS) enables you to create hierarchical term sets that people can then pick from to classify documents.

The image above shows three term sets under the group heading Organisation – Department, Month and Product, with the term set for ‘Product’ expanded. ‘SharePoint’ is selected and on the right-side of the image, you can see the default value and other labels that people might try to use. If anyone were to enter ‘Tahoe’ or ‘SP2010′, it will automatically update to display the default value of ‘SharePoint’. You can also add language-specific variations. If you decide to change the default value, all items currently classified with that term will be automatically updated. This is very different behaviour to normal list/library columns within SharePoint – if you change the settings for a normal column, all existing items will keep their original properties until someone manually updates them.

Managed metadata is also used by the search service, which uses the term sets to help refine search results

The image above shows search results with the ability to refine by Department and Product – these are both based on the corresponding term sets in the Managed Metadata Service.

When it comes to applying managed metadata, it’s back to the SharePoint lists and libraries. Two new types of column are available:

• Enterprise keywords column
• Managed metadata column

Enterprise Keywords column

The Enterprise keywords column is enabled in the list/library settings:

Note: you can also choose whether or not properties populated using this column will appear as part of a user’s social tag cloud (more on that later). The Enterprise Keywords column allows users to enter any words to classify a document or item. Autocomplete will suggest available metadata properties and tags (social tagging is coming up shortly) and a user can enter as many values as they want.

In the image above, entering the letters ‘sh’ results in two suggestions. The first is from the Product term set. The second is a tag that someone else has used. All unmanaged tags are added to a single term set called ‘Keywords’. I could choose one of those suggestions or enter a new tag that will then be added to the Keywords.

Managed Metadata column

When you create a column in SharePoint 2010 you can choose a new type – Managed metadata. You’ll get a warning that it is not compatible with previous versions of Office. For the column settings, you can point it to any term set and users will then be able to pick from that list.

Social Tagging

In reality, this is part of the managed metadata service (MMS). Microsoft documents and SharePoint user tend to use the phrase ‘keywords’ for both the formal hierarchical term sets and the informal tags which can be confusing.

Social tagging enables anyone to classify an item using any word. The first time it is used it is added to a list of keywords in the MMS. After that, it will be suggested whenever someone starts to type a word beginning with the same letters. Just about anything can be tagged in SharePoint including sites and pages as well as items in lists and libraries.

The image above shows the default header for all sites and pages in SharePoint 2010. You can use the simple ‘I Like It’ tag or you can click ‘Tags & Notes’ and enter your own tags as well as notes about the page. These tags are organised in a tag cloud on the user’s Profile page which enables everyone to see what other people have been tagging. You can even tag external web sites.

The image above shows a (very small) tag cloud on The Administrator’s profile page.

To use social tagging in lists and libraries, you have to enable the Enterprise Keywords column in the list/library settings (already mentioned earlier in this post). You have the option to choose whether or not the column will be used to populate social tag clouds.

Social tags are managed in the same place as your hierarchical term sets – the Managed Metadata service. They are organised as a flat list under a single System term set called ‘Keywords’. The benefit of having both hierarchical term sets and informal tag sets managed together is that you can promote tags into your hierarchical term sets if it becomes evident people are using different words to the ones you thought they would use.

In the image above, I have clicked on Customer Support and have the option to move or delete the keyword. If I select Move, I can then pick a location in one of my hierarchical term sets (such as Department). Once moved, you can also merge terms together, for example if Customer Support were to replace Customer Services.

Managed Metadata Issues

There are a couple of caveats to be aware of currently with managed metadata – both the hierarchical term sets and social tagging (keywords). They only work within the browser and Office 2010. If you are using an older version of Office, including 2007 regardless of using the new file formats, you will not be able to edit these properties from within Office. Secondly, they do not work in Datasheet view in the browser which means there is no easy way to bulk classify items, you have to edit them individually. This is a huge pain if you are migrating documents from other sources including older SharePoint sites. Workarounds are limited. You can set the default value but need to be organised with how you import your documents or consider using 3rd party tools. Whilst the Managed Metadata and Enterprise Keywords columns appear within the SharePoint list or library, their contents are managed in a separate database. The downside is some loss of functionality within Office and the SharePoint list/library compared to normal column types. The benefit is being able to automatically update all document properties centrally and being able to use the term sets as search refiners/filters.

The other gotcha is the term set hierarchy and search results. If you scroll back up and look at the Search Results image, in the sidebar on the left you will see my two term sets – Department and Products, each listing terms beneath them. Click on these terms and you can refine search results to display only content tagged with these terms. So far, so great. However, notice that there is no hierarchy beneath the term set.  My term set Products has first-level terms SharePoint (and others not showing, SQL and Office). The term SharePoint has second-level terms including Apps and Content. But this hierarchy is lost in the search refiners, all terms in the Term Set are displayed as one level. So if you are using managed metadata primarily to help improve search results, keep your hierarchy as flat as possible, ideally just Term Set – Terms.

[Update 15th Nov 2010] This post has also been published on EndUserSharePoint – http://www.endusersharepoint.com/2010/11/15/sharepoint-managed-metadata-overview/

[Update 1st Sep 2011] Related blog post publishing highlighting taxonomy limits – http://www.sharepointsharon.com/2011/09/sharepoint-2010-taxonomy-limits/

Related blog posts

• SharePoint 2010 and Taxonomy Limits
• Embedding Office documents in SharePoint
• Classifying content in SharePoint
• Taxonomy in MOSS
• Rethinking the file plan

This is a follow on post to Taxonomy in MOSS (SharePoint Server 2007). Not everyone knows that you can manage columns at the site collection level and then re-use them across libraries. Also, whilst SharePoint doesn’t really do taxonomy management, you can use lists to perform some lightweight management, such as controlling what values are used within metadata columns. Here’s how to set it all up. Note: if you have multiple site collections, you would need to repeat this process for each site collection. You will need to be a Site Administrator to perform the following steps.

Scenario: We want all documents created or uploaded into any library within the site collection to be classified by Business Unit. To ensure the data entered is consistent, users will be presented with a menu from which to choose the appropriate Business Unit. The menu will be populated with values stored in a SharePoint list. Whenever the SharePoint list is updated (i.e. to add/remove/rename business units), the menu will automatically display the changes. The image below shows the basic architecture:

Image 1: outline architecture

Step 1: Create your taxonomy lists.

Because we are going to create a column that looks up values held in a SharePoint list, first we need to create the SharePoint list. You need to create the list(s) in the top-level site of the site collection (Joining Dots in this example).In the image below, I have created a list called ‘Business Unit’. If I want to add an item to the list, I simply click New and enter the title of another business unit:

Image 2: A SharePoint List

Step 2: Create the site column.

The next step is to create the site column that will look up the values in the SharePoint list. Click the Site Actions button at the top-right of the SharePoint page and click ‘Site Settings’. (Hint: If you can’t see the Site Actions button, you don’t have the required permissions – you need to be a Site Administrator). It is important to navigate to the top-level site. On the Site Settings page, view the options under ‘Site Collection Administration’ (circled in red in the image below). If you don’t see the list of options, you should see a single link ‘Go to top level site settings’. Click on it.

Image 3: SharePoint Site Collection Administration

Assuming you are at the top level site, under Galleries, click Site columns. You will be presented with a list of the existing site columns. Click ‘Create’ to create a new one and you will be presented with a page like the one below (the red arrows are mine):

Image 4: Create a SharePoint Site Column

• Give the column a name (in this example, ‘Business Unit’).
• For the type of column, select ‘Lookup (information already on this site)’
• Under Group, for the first time, select New group and give it a name (in this example, ‘Our Taxonomy). After that, use the same group. Makes it easy to locate your taxonomy columns
• Under Additional Column Settings, choose if you want the column to be mandatory or not (‘Require that the column contains information’) and configure the look-up:
  • Under ‘Get information from:’, select the SharePoint list (in this example, ‘Business Unit’).
  • Under ‘In this column:’, select the column within the list that contains the values you want to use in this column. (In this example, it is ‘Title’. You can see the column label on display in image 2)
• Click OK to create the column

Step 3: Configure a document library to use the site column

Navigate to a document library where you want to use this site column. In this example, we have a sub-site called ‘Library’ containing a document library called ‘Documents’ (yes, in hindsight, I could have used better names to avoid confusion).

Image 5: A SharePoint document library

In the document library, click the Settings button and do NOT choose the obvious option of ‘Create Column’. Instead, select ‘Document Library Settings and you will be presented with the screen shown below:

Image 6: SharePoint Document Library Settings

The clue is circled in red again. Click ‘Add from existing site columns’ and you will be presented with the following screen:

Image 7: Add existing site column

SharePoint has a lot of built-in columns and groups. That is why it helps to use your own group names to organise your own site columns. In this example, the group is called ‘Our Taxonomy’ and that filters the available site columns to the one and only ‘Business Unit’. Select the column and click Add. Make sure the ‘Add to default view’ check box is selected and click OK.

Back in the document library itself, this time click Upload to add a document to the library:

Image 8: Upload a document

After selecting your document and clicking OK, you will be presented with the form to update any properties (values to be entered into columns):

Image 9: Classify the document

As shown in image 9, the user is presented with a dropdown menu for Business Unit. The list of values in the menu come from the SharePoint list created in step 1. Hey presto. We’re done!

Now, it just wouldn’t be natural to write a SharePoint post without highlighting at least one gotcha to watch out for… Here are a couple of limitations to be aware of::

1. Each time you classify a document, you select and insert a value from the menu. If you look at image 9, there are two similar business units – Accounts and Finance. If we decide that we do not need Accounts, we can delete it from the Business Unit list and it will automatically disappear from the menu, across every document library that references the Business Unit list. However, any documents that have already been classified as ‘Accounts’ will still show that value in their properties, even though it is no longer available.
2. This approach is the best way to ensure consistency in your columns across your sites. However, there is no way to prevent people from creating their own columns at the document library level (see image 5), beyond restricting permissions to prevent access to all document library settings and/or providing good user training.

Filed in library under: SharePoint

Technorati tags: SharePoint; SharePoint 2007; MOSS 2007

On 21st May 2008 I presented to an audience of taxonomy professionals within the UK public sector. The last session of the day, I had 30 minutes to present on “Taxonomy within Microsoft Office SharePoint Server 2007 (MOSS): Lessons learned from real-world deployments”.

My aim was to briefly explain what MOSS can and cannot do with taxonomy and provide a few tips on how to leverage MOSS taxonomy features to improve information findability. The session generated quite a bit of note-taking and debate. Here are the slides:

Key messages from the presentation:

• MOSS uses elements of taxonomy to improve search and navigation. The core feature is ‘columns’, used for metadata. Case study: a tag-driven user interface created for the New Zealand Ministry of Transport. A great end result but a lot of effort required to implement and maintain
• MOSS does not (yet) provide taxonomy management tools. Taxonomy management is about defining and managing schema(s), and classifying content agains those schemas
• Taxonomy is not the holy grail. Schemas need to continually evolve to be effective. Often there is a disconnect between the language used by those creating the schema and those looking for information that the schema is for. This perhaps explains why folksonomies have achieved more success than official taxonomies, but…
• User tagging is less accurate or consistent than automatic classification. Comment from Google founder Sergey Brin: Semantics and tagging are great as long as computers are doing it [not people].” Automatic classification is by no means perfect either. Accuracy rarely exceeds 70% – lots of development going on to improve this
• 4 tips to improve the use and value of taxonomy within MOSS today:
1. Where possible, define columns at the site collection level, not per library. Do it per library and each instance will be treated as separate crawled property in the index. By managing per site collection, you can also control what values can be entered, improving consistency across sites and libraries
2. Avoid using sites and sub-sites to mimic file structures (popular when creating file plans). One of the relevance algorithms is URL depth. The deeper the URL, the less relevant and you don’t want empty sites returned in search results. Alternative approach: create a link-driven UI to mimic the file plan but apply it using columns and store content in as few sites as possible
3. Check out your sources. When indexing content, if one source has a lot more metadata than others, it can dominate search results. A common issue for mergers and acquistions, or re-orgs within government. Solution: split the index and/or use federated search
4. Maximise the effectiveness of automatic metadata, such as titles and descriptions. Avoid bland document titles (e.g. ‘meeting notes’ x 50) and irrelevant link titles (‘Click here’ versus a title that describes where the link takes you)
• Most likely scenarios to want to go beyond MOSS are concept-driven search and automatic classification. You can use bespoke code and lightweight tools like the Faceted Search tool on Codeplex. But it is usually better to engage a partner.
• Final case study: legal firm – lots of taxonomy but just getting search up and running was a big win. People found it easier to find information using basic search than the formal navigation structures created by file plans…

To download a copy of the presentation: MOSSTIPS-May08.pdf (2.7Mb) <--Note I haven't used 'Click here' :-)

Technorati tags: MOSS 2007 SharePoint Taxonomy Tagging Information Architecture

Perhaps one of the loudest unspoken messages from the SharePoint conference held recently in Seattle was the need for information architects and managers to work more closely with their user interface (UI) and technology-focused counterparts. Thanks to the Internet, we are unlikely to see a downturn in the market for digital information in the foreseeable future. But the methods used to classify, manage and access information are still dominated by techniques taken from the physical world of information – paper and its storage methods: micro (books) and macro (libraries).

Let’s pick on ‘The Fileplan’

A common scenario I see in organisations, especially government ones, is the use of a fileplan to store and access content. Here’s the definition of a fileplan, courtesy of ‘Developing a Fileplan for Local Government‘ (UK) (My comments in brackets):

¨The fileplan will be a hierarchical structure of classes starting with a number of broad functional categories. These categories will be sub-divided and perhaps divided again until folders are created at the lowest level. These folders, confusingly called files in paper record management systems (hence the term ‘fileplan’), are the containers in which either paper records or electronic documents are stored.¨

And why do we need fileplans

¨An important purpose of the fileplan is to link the documents and records to an appropriate retention schedule.¨

Really? Just how many different retention schedules does an organisation need to have? One per lowest-level folder? I doubt that. Let’s create a (very) simple fileplan: Geography – Business Unit – Activity

Taking 3 geographies, 3 business units and 3 activities. These are the folders you end up with:

• UK/finance/budget/
• UK/finance/managementaccounts/
• UK/finance/projects/
• UK/IT/operations/
• UK/IT/procedures/
• UK/IT/projects/
• UK/Sales/campaigns
• UK/Sales/products
• UK/Sales/projects
• France/finance/budget/
• France/finance/managementaccounts/
• France/finance/projects/
• France/IT/operations/
• France/IT/procedures/
• France/IT/projects/
• France/Sales/campaigns/
• France/Sales/products/
• France/Sales/projects/
• Germany/finance/budget/
• Germany/finance/managementaccounts/
• Germany/finance/projects/
• Germany/IT/operations/
• Germany/IT/procedures/
• Germany/IT/projects/
• Germany/Sales/campaigns
• Germany/Sales/products
• Germany/Sales/projects

So we have 27 different locations to cover 3 geographies with 3 departments and 3 activities. Now scale this up for your organisation. How many different folders do you end up with?

The ultimate killer with this scenario? There isn’t any content in the first 2 levels of the hierarchy. You’ve got to navigate through 3 levels before you can even start to find what you are looking for. This is because a librarian approach is used for storing and locating information:

Go upstairs, ‘Technology’ section is on the left, you’ll find ‘Computing’ about halfway along. Third shelf up is ‘Programming Languages’, books organised alphabetically by author…

In the physical world, we can’t do a ‘Beam me up, Scotty!‘ and magically arrive at the shelf containing the book containing the page(s) we want. But in the digital world, we can. If fans of the fileplan designed Google’s navigation, it might look something like this:

And they probably wouldn’t include the search box on the first two pages. Fortunately for everyone who uses the Internet to search for information, Google took the ‘Beam me up, Scotty!’ approach.

The fileplan approach causes problems for everyone. Authors have to find ‘the right’ location to put their stuff. If they are working on anything remotely ambiguous, it is unlikely there will be one clear option. That’s why everyone ends up defaulting to the ‘projects’ folder (‘miscellaneous’ is another popular destination). Search engines that use URL depth algorithms (such as PageRank) struggle to identify relevant content – is the folder ‘Finance’ more important than a document called ‘Finance’ that is two levels deeper in the hierarchy buried under Projects/Miscellaneous? If someone is searching for documents about France, are documents located in the France folder hierarchy more important than documents containing references to France that have been stored in the UK hierarchy? Authors don’t know where to put their stuff, and searchers can’t find it. What about those all important retention schedules? They might be different for different geographies (governments don’t seem to agree or standardise on anything much, globally) but then what? Do all of Finance docs have a different retention schedule to all of IT? Within Finance, do different teams have different retention schedules? (Quite possibly – certain financial documents need storing for specific periods of time). Current solution? Sub-divide and conquer, i.e. create yet another level of abstraction in the fileplan… I have seen solutions where users have to navigate through 6 levels before reaching a folder that contains any content.

So what’s the alternative?

Perhaps a better question would be ‘what’s an alternative?’ The desire to find one optimal solution is what trips up most information system designs. Here are some of my emerging thoughts. If you’ve got an opinion, please contribute in the comments because I certainly don’t have all the answers.

Step 1: Stop thinking physically and start thinking digitally

There are two fundamental problems with the fileplan. First, it originates from the constraints enforced by physical technologies. A paper document must exist somewhere and you don’t want to have to create 100 copies to cover all retrieval possibilities – it’s expensive and time-consuming. Instead, all roads lead to one location… and it’s upstairs, third cabinet on the right, second drawer down, filed by case title. This approach creates the second problem – because content is managed in one place, that one place – the fileplan – must cover all purposes, i.e. storage, updates, retention schedule, findability and access. Physical limits required you to think this way. But those limits are removed when you switch to digital methods. What we need are multiple file plans, each suited to a specific purpose.

Information specialists can help identify the different purposes and different ‘file plans’ required. Technologists need to help create solutions that make it as easy as possible (i.e. minimal effort required) for authors and searchers to work with information and ‘fileplans’. And user interface specialists need to remind everyone about what happens when you create mandatory metadata fields and put the search box in the wrong place on the page…

Digital storage of content should be logical to the creators, because authors ultimately decide where they save their documents. Trying to force them into a rigid navigation hierarchy designed by somebody else just means everything gets saved in ‘miscellaneous’. Don’t aim for a perfect solution. Instead, provide guidance about where ‘stuff’ should go. Areas for personal ‘stuff’, team ‘stuff’, community sites, collaborative work spaces, ‘best practices’ sites. Ideally, you still want to stick to one location. Not because of any resource constraints but rather to avoid unnecessary duplication that can cause confusion. If an item of content needs to appear ‘somewhere else’ then it should be a link rather than a copy, unless a copy is required to fit a different scenario (e.g. publishing a copy of a case study up onto a public web site, but keeping the original held in a location that can only be edited by authors)

To improve relevance of search results, thesauri and controlled vocabularies can help bridge the language barriers between authors and readers. A new starter might be looking for the ‘employee manual’. What they don’t know is what they are actually looking for is the ‘corporate handbook’ or ‘human remains guide’ that may contain the words ‘employee’ and ‘manual’ but never together in the same sentence. The majority of search frustrations come from information seekers using a different language to the one used by the authors of the information they seek. Creating relationships between different terms can dramatically improve relevance of search results. Creating tailored results pages (a mix of organic search results and manufactured links) can overcome differences in terminology and improve future search behaviour.

And the elephant in the file system – retention schedules. First identify what retention schedules you require to comply with industry regulations and to manage legal risk. And do they apply to all content or only certain content? (I doubt many government organisations have kept junk paper mail for 30 years.) And at what point do they need to be applied? From the minute somebody opens a word processor tool and starts typing, or at the point when a document becomes finalised? This is the area that needs most coordination between information specialists and technologists. As we start to move to XML file formats, life could potentially become so much easier for everyone. For example, running scripts to automatically track documents for certain words that give a high probability that the document should be treated as a record and moved from a community discussion forum to the archive. Automatically inserting codes that enable rapid retrieval of content to comply with a legal request but that have no effect on relevance for regular searches.

On the Internet, Google introduced a tag ‘nofollow’ that could be applied to links to prevent the link improving a page’s relevance rank. (PageRank works by determining relevance based on the number of incoming links to a page. If you want to link to a page so that people can look at it but you don’t want the page to benefit from the link in search results, you can insert ‘nofollow’). Maybe Enterprise Search solutions need a similar method. Different indicators for metadata that helps describe content for searches versus metadata that organises content for retention schedules versus metadata that helps authors remember where they left their stuff. And again, XML formats ought to make it possible to automatically insert the appropriate values without requiring the author to figure out what’s needed. The ultimate goal would be to automatically insert sufficient information within individual content items so that requirements are met regardless of where the content is stored or moved to. I email an image to someone and its embedded metadata includes its fileplan(s).

There are lots of ways that technology could be used to improve information management and findability, to meet all the different scenarios demanded by different requirements. But to achieve them requires closer interaction between people making the policies regarding how information is managed, people creating the so-called ‘technology-agnostic’ (in reality it is ‘technology-vendor-agnostic’) file plans to satisfy those policies and the technology vendors creating solutions used to create, store and access the content being created that have to cope with the fileplans and the policies.

The information industry has to move on from the library view of there being only one fileplan. Lessons can be learned from the food industry. There was a time when there was only one type of spaghetti sauce. In the TED talk below, Malcolm Gladwell explains how the food industry discovered the benefits from offering many different types of spaghetti sauce (and why you can’t rely on focus groups to tell you what they want – another dilemma when designing information systems):

Direct link to TED talk (in case video doesn’t load here)

There is a great quote within the above talk:

¨When we pursue universal principles in food, we aren’t just making an error, we are actually doing ourselves a massive disservice¨

You could replace the word ‘food’ with ‘information’. It’s not just the fileplan that needs rethinking…

On Microsoft’s Channel 9 network, there is an interesting podcast called ‘Just Enough Architecture‘, where the interviewee provides some good recommendations about the balance between how much architecture you need versus just getting on and writing software that does something useful.

The same debate could be applied to taxonomy, specifically the use of metadata properties to classify content.

For some reason, most companies who decide they want to improve how content is classified seem to want extreme taxonomy, swinging from not-enough taxonomy to too-much. The mantra may sound somewhat familiar:

One taxonomy to rule them all, one taxonomy to find them, one taxonomy to bring them all and, in the records management store, define them

Often starting with none at all (i.e. content is organised informally and inconsistently using folders), the desire is to create a single corporate taxonomy to classify everything (using a hierarchical structure of metadata terms). An inordinate amount of time is then spent defining and agreeing the perfect taxonomy (for some reason, many seem to settle on about 10,000 terms). Several months later, heads are being scratched as people try to figure out just how they are going to implement the taxonomy. Do they classify existing content or only apply it to new stuff? Do they have specific roles dedicated to classifying the content, rely on the content owners to do it, or look at automated classification tools. Do they put rules in place to force people to classify content and store it in specific locations that are ‘taxonomy-aware’. How do they prevent people bypassing the system, those who figure they can still get their work done by switching to a wiki or a Groove workspace or a MySpace site or a Twitter conversation? How do they validate the taxonomy and check that people are classifying correctly? What do they do if people aren’t classifying correctly, who don’t understand the hierarchy or have different meanings for the terms in use? What started out as a simple idea to improve the findability of information becomes a huge burden to maintain with questionable benefits, given there are so many opportunities for classification to go wrong.

This dilemma reveals two flaws that make implementing a taxonomy so difficult. The first is the desire to treat taxonomy as a discrete project rather than an organic one. Collaboration and knowledge management projects often share this fate. Making taxonomy a discrete project usually means tackling it all in one go from a technology perspective and then handing it over to the business to run ‘as is’ for ever more (i.e. until the next technology upgrade). Such projects end up looking like that old cliché – attempting to eat an elephant whole. The project team tries to create a perfect design that will deliver all identified requirements (and the business, knowing this could be their one chance for improved tools, delivers a loooooong list of requirements), implements a solution and then moves on to the next project. As the solution is used, the business finds flaws in their requirements or discover new ways of working enabled by the technology, but it is too late to get the solution changed. The project is closed, the budget spent.

An alternative approach is to treat taxonomy as an organic project or, for those who prefer corporate-speak, a continuous-improvement programme. Instead of planning to create and deploy the perfect taxonomy, concentrate on ‘just enough taxonomy’. A good starting point is to find out why taxonomy is needed in the first place. If it is to make it easier for people to find information, first document the specific problems being experienced. Solve those problems as simply as possible, test them and gather feedback. If successful, people will raise the bar on what they consider good findability, generating new demands waiting for IT to solve, and so the cycle continues.

The following is a simple example using a fictitious company.

Current situation: Most information is stored in folders on file shares and shared via email. There is an intranet that is primarily static content published by a few authors. The IT department has been authorised to deploy Microsoft Office SharePoint Server 2007 (MOSS)

General problem: Nobody can find what they are looking for (resist temptation to sing U2 song at this moment…)

Specific problems: Difficult to find information from recently completed projects that could be re-used in future projects; Difficult to differentiate between high quality re-usable project information versus low quality or irrelevant project information; Difficult to find all available documents for a specific customer (contracts, internal notes, project files)

Possible solution: Deploy a search engine to index all file folders and the intranet. Move all project information to a central location. Within the search engine, create a scope (or collection) for the project information location. Users will then be able to perform search queries that will return only project information within the results. Using ‘date modified’ as the sorting order will locate information from the most recent projects. Create a central location for storing top-rated ‘best practice’ project information. Set-up a team of subject matter experts to work with project teams and promote documents as ‘best practice’. The Best Practices store can be given high visibility throughout the intranet and promoted as high relevance for search queries.

Now that is a very brief answer outlining one possible solution. But the solution is relatively simple to implement and should offer immediate (and measurable) improvements based on feedback regarding the problems people are experiencing. There were two red herrings in the requirements that could have resulted in a very different, more complex, solution: 1. That MOSS was going to be the technology; and 2. The need to find documents for a specific customer. When you have chosen a technology, there is always the temptation to widen the project scope. MOSS has all sorts of features that can help improve information management and the starting point is often to replace an old crusty static intranet. But the highlighted problems did not mention any concerns about the intranet. That’s not to say those concerns do not exist, but they are a different problem and not the priority for this project. The second red herring is a classic. When people want to be able to find information based on certain parameters, such as all documents connected to a specific customer, there is the temptation to implement a corporate-wide taxonomy and start classifying all content, starting with the metadata property ‘customer name’. But documents about a specific customer will likely contain the customer’s name. In this scenario, the simplest solution is to create a central index and provide the ability for users to search for documents containing a given customer’s name. If that fails to improve the situation then you may need to consider more drastic measures.

Rejecting the large-scale information management project in favour of small chunks of continuous ‘just enough’ improvement is not an easy approach to take. The idea of having a centralised, classified and managed store of content, where you can look up information based on any parameter and receive perfect results, continues to be an attractive one with lots of benefits to the business – both value-oriented (i.e. helping people discover information to do their job) and cost-oriented (i.e. managing what people do with information – compliance checks and the like). But a perfectly classified store of content is a utopia. Trying to achieve it can result in creating systems that are harder to use and difficult to maintain when the goal is supposed to be to make them easier.

I mentioned that the common approach to implementing taxonomy has two flaws. The first has been discussed here – how to create just enough taxonomy. The second flaw is the desire to create a single universal taxonomy that can be applied to everything. I’ll tackle that challenge in a separate post (a.k.a this post is already too long…)

Reference: Just Enough Architecture (MSDN Channel 9). Highly recommended. There are plenty of similarities between software architecture and information architecture (of which taxonomy is subset). Don’t be put off by the techie speak, it debates the pro’s and con’s of formal processes and informal uses, and includes some great non-technical examples for how to find a balance.

Recent related posts:

• Metacrap
• Why taxonomy fails
• When taxonomy fails

Technorati tags: Taxonomy, Tagging, Information Architecture