Scenario: We want all documents created or uploaded into any library within the site collection to be classified by Business Unit. To ensure the data entered is consistent, users will be presented with a menu from which to choose the appropriate Business Unit. The menu will be populated with values stored in a SharePoint list. Whenever the SharePoint list is updated (i.e. to add/remove/rename business units), the menu will automatically display the changes. The image below shows the basic architecture:

Image 1: outline architecture

Step 1: Create your taxonomy lists.

Because we are going to create a column that looks up values held in a SharePoint list, first we need to create the SharePoint list. You need to create the list(s) in the top-level site of the site collection (Joining Dots in this example).In the image below, I have created a list called ‘Business Unit’. If I want to add an item to the list, I simply click New and enter the title of another business unit:

Image 2: A SharePoint List

Step 2: Create the site column.

The next step is to create the site column that will look up the values in the SharePoint list. Click the Site Actions button at the top-right of the SharePoint page and click ‘Site Settings’. (Hint: If you can’t see the Site Actions button, you don’t have the required permissions – you need to be a Site Administrator). It is important to navigate to the top-level site. On the Site Settings page, view the options under ‘Site Collection Administration’ (circled in red in the image below). If you don’t see the list of options, you should see a single link ‘Go to top level site settings’. Click on it.

Image 3: SharePoint Site Collection Administration

Assuming you are at the top level site, under Galleries, click Site columns. You will be presented with a list of the existing site columns. Click ‘Create’ to create a new one and you will be presented with a page like the one below (the red arrows are mine):

Image 4: Create a SharePoint Site Column

• Give the column a name (in this example, ‘Business Unit’).
• For the type of column, select ‘Lookup (information already on this site)’
• Under Group, for the first time, select New group and give it a name (in this example, ‘Our Taxonomy). After that, use the same group. Makes it easy to locate your taxonomy columns
• Under Additional Column Settings, choose if you want the column to be mandatory or not (‘Require that the column contains information’) and configure the look-up:
  • Under ‘Get information from:’, select the SharePoint list (in this example, ‘Business Unit’).
  • Under ‘In this column:’, select the column within the list that contains the values you want to use in this column. (In this example, it is ‘Title’. You can see the column label on display in image 2)
• Click OK to create the column

Step 3: Configure a document library to use the site column

Navigate to a document library where you want to use this site column. In this example, we have a sub-site called ‘Library’ containing a document library called ‘Documents’ (yes, in hindsight, I could have used better names to avoid confusion).

Image 5: A SharePoint document library

In the document library, click the Settings button and do NOT choose the obvious option of ‘Create Column’. Instead, select ‘Document Library Settings and you will be presented with the screen shown below:

Image 6: SharePoint Document Library Settings

The clue is circled in red again. Click ‘Add from existing site columns’ and you will be presented with the following screen:

Image 7: Add existing site column

SharePoint has a lot of built-in columns and groups. That is why it helps to use your own group names to organise your own site columns. In this example, the group is called ‘Our Taxonomy’ and that filters the available site columns to the one and only ‘Business Unit’. Select the column and click Add. Make sure the ‘Add to default view’ check box is selected and click OK.

Back in the document library itself, this time click Upload to add a document to the library:

Image 8: Upload a document

After selecting your document and clicking OK, you will be presented with the form to update any properties (values to be entered into columns):

Image 9: Classify the document

As shown in image 9, the user is presented with a dropdown menu for Business Unit. The list of values in the menu come from the SharePoint list created in step 1. Hey presto. We’re done!

Now, it just wouldn’t be natural to write a SharePoint post without highlighting at least one gotcha to watch out for… Here are a couple of limitations to be aware of::

1. Each time you classify a document, you select and insert a value from the menu. If you look at image 9, there are two similar business units – Accounts and Finance. If we decide that we do not need Accounts, we can delete it from the Business Unit list and it will automatically disappear from the menu, across every document library that references the Business Unit list. However, any documents that have already been classified as ‘Accounts’ will still show that value in their properties, even though it is no longer available.
2. This approach is the best way to ensure consistency in your columns across your sites. However, there is no way to prevent people from creating their own columns at the document library level (see image 5), beyond restricting permissions to prevent access to all document library settings and/or providing good user training.

Filed in library under: SharePoint

Technorati tags: SharePoint; SharePoint 2007; MOSS 2007

My aim was to briefly explain what MOSS can and cannot do with taxonomy and provide a few tips on how to leverage MOSS taxonomy features to improve information findability. The session generated quite a bit of note-taking and debate. Here are the slides:

Key messages from the presentation:

• MOSS uses elements of taxonomy to improve search and navigation. The core feature is ‘columns’, used for metadata. Case study: a tag-driven user interface created for the New Zealand Ministry of Transport. A great end result but a lot of effort required to implement and maintain
• MOSS does not (yet) provide taxonomy management tools. Taxonomy management is about defining and managing schema(s), and classifying content agains those schemas
• Taxonomy is not the holy grail. Schemas need to continually evolve to be effective. Often there is a disconnect between the language used by those creating the schema and those looking for information that the schema is for. This perhaps explains why folksonomies have achieved more success than official taxonomies, but…
• User tagging is less accurate or consistent than automatic classification. Comment from Google founder Sergey Brin: Semantics and tagging are great as long as computers are doing it [not people].” Automatic classification is by no means perfect either. Accuracy rarely exceeds 70% – lots of development going on to improve this
• 4 tips to improve the use and value of taxonomy within MOSS today:
1. Where possible, define columns at the site collection level, not per library. Do it per library and each instance will be treated as separate crawled property in the index. By managing per site collection, you can also control what values can be entered, improving consistency across sites and libraries
2. Avoid using sites and sub-sites to mimic file structures (popular when creating file plans). One of the relevance algorithms is URL depth. The deeper the URL, the less relevant and you don’t want empty sites returned in search results. Alternative approach: create a link-driven UI to mimic the file plan but apply it using columns and store content in as few sites as possible
3. Check out your sources. When indexing content, if one source has a lot more metadata than others, it can dominate search results. A common issue for mergers and acquistions, or re-orgs within government. Solution: split the index and/or use federated search
4. Maximise the effectiveness of automatic metadata, such as titles and descriptions. Avoid bland document titles (e.g. ‘meeting notes’ x 50) and irrelevant link titles (‘Click here’ versus a title that describes where the link takes you)
• Most likely scenarios to want to go beyond MOSS are concept-driven search and automatic classification. You can use bespoke code and lightweight tools like the Faceted Search tool on Codeplex. But it is usually better to engage a partner.
• Final case study: legal firm – lots of taxonomy but just getting search up and running was a big win. People found it easier to find information using basic search than the formal navigation structures created by file plans…

To download a copy of the presentation: MOSSTIPS-May08.pdf (2.7Mb) <--Note I haven't used 'Click here' :-)

Technorati tags: MOSS 2007 SharePoint Taxonomy Tagging Information Architecture

Let’s pick on ‘The Fileplan’

A common scenario I see in organisations, especially government ones, is the use of a fileplan to store and access content. Here’s the definition of a fileplan, courtesy of ‘Developing a Fileplan for Local Government‘ (UK) (My comments in brackets):

¨The fileplan will be a hierarchical structure of classes starting with a number of broad functional categories. These categories will be sub-divided and perhaps divided again until folders are created at the lowest level. These folders, confusingly called files in paper record management systems (hence the term ‘fileplan’), are the containers in which either paper records or electronic documents are stored.¨

And why do we need fileplans

¨An important purpose of the fileplan is to link the documents and records to an appropriate retention schedule.¨

Really? Just how many different retention schedules does an organisation need to have? One per lowest-level folder? I doubt that. Let’s create a (very) simple fileplan: Geography – Business Unit – Activity

Taking 3 geographies, 3 business units and 3 activities. These are the folders you end up with:

• UK/finance/budget/
• UK/finance/managementaccounts/
• UK/finance/projects/
• UK/IT/operations/
• UK/IT/procedures/
• UK/IT/projects/
• UK/Sales/campaigns
• UK/Sales/products
• UK/Sales/projects
• France/finance/budget/
• France/finance/managementaccounts/
• France/finance/projects/
• France/IT/operations/
• France/IT/procedures/
• France/IT/projects/
• France/Sales/campaigns/
• France/Sales/products/
• France/Sales/projects/
• Germany/finance/budget/
• Germany/finance/managementaccounts/
• Germany/finance/projects/
• Germany/IT/operations/
• Germany/IT/procedures/
• Germany/IT/projects/
• Germany/Sales/campaigns
• Germany/Sales/products
• Germany/Sales/projects

So we have 27 different locations to cover 3 geographies with 3 departments and 3 activities. Now scale this up for your organisation. How many different folders do you end up with?

The ultimate killer with this scenario? There isn’t any content in the first 2 levels of the hierarchy. You’ve got to navigate through 3 levels before you can even start to find what you are looking for. This is because a librarian approach is used for storing and locating information:

Go upstairs, ‘Technology’ section is on the left, you’ll find ‘Computing’ about halfway along. Third shelf up is ‘Programming Languages’, books organised alphabetically by author…

In the physical world, we can’t do a ‘Beam me up, Scotty!‘ and magically arrive at the shelf containing the book containing the page(s) we want. But in the digital world, we can. If fans of the fileplan designed Google’s navigation, it might look something like this:

And they probably wouldn’t include the search box on the first two pages. Fortunately for everyone who uses the Internet to search for information, Google took the ‘Beam me up, Scotty!’ approach.

The fileplan approach causes problems for everyone. Authors have to find ‘the right’ location to put their stuff. If they are working on anything remotely ambiguous, it is unlikely there will be one clear option. That’s why everyone ends up defaulting to the ‘projects’ folder (‘miscellaneous’ is another popular destination). Search engines that use URL depth algorithms (such as PageRank) struggle to identify relevant content – is the folder ‘Finance’ more important than a document called ‘Finance’ that is two levels deeper in the hierarchy buried under Projects/Miscellaneous? If someone is searching for documents about France, are documents located in the France folder hierarchy more important than documents containing references to France that have been stored in the UK hierarchy? Authors don’t know where to put their stuff, and searchers can’t find it. What about those all important retention schedules? They might be different for different geographies (governments don’t seem to agree or standardise on anything much, globally) but then what? Do all of Finance docs have a different retention schedule to all of IT? Within Finance, do different teams have different retention schedules? (Quite possibly – certain financial documents need storing for specific periods of time). Current solution? Sub-divide and conquer, i.e. create yet another level of abstraction in the fileplan… I have seen solutions where users have to navigate through 6 levels before reaching a folder that contains any content.

So what’s the alternative?

Perhaps a better question would be ‘what’s an alternative?’ The desire to find one optimal solution is what trips up most information system designs. Here are some of my emerging thoughts. If you’ve got an opinion, please contribute in the comments because I certainly don’t have all the answers.

Step 1: Stop thinking physically and start thinking digitally

There are two fundamental problems with the fileplan. First, it originates from the constraints enforced by physical technologies. A paper document must exist somewhere and you don’t want to have to create 100 copies to cover all retrieval possibilities – it’s expensive and time-consuming. Instead, all roads lead to one location… and it’s upstairs, third cabinet on the right, second drawer down, filed by case title. This approach creates the second problem – because content is managed in one place, that one place – the fileplan – must cover all purposes, i.e. storage, updates, retention schedule, findability and access. Physical limits required you to think this way. But those limits are removed when you switch to digital methods. What we need are multiple file plans, each suited to a specific purpose.

Information specialists can help identify the different purposes and different ‘file plans’ required. Technologists need to help create solutions that make it as easy as possible (i.e. minimal effort required) for authors and searchers to work with information and ‘fileplans’. And user interface specialists need to remind everyone about what happens when you create mandatory metadata fields and put the search box in the wrong place on the page…

Digital storage of content should be logical to the creators, because authors ultimately decide where they save their documents. Trying to force them into a rigid navigation hierarchy designed by somebody else just means everything gets saved in ‘miscellaneous’. Don’t aim for a perfect solution. Instead, provide guidance about where ‘stuff’ should go. Areas for personal ‘stuff’, team ‘stuff’, community sites, collaborative work spaces, ‘best practices’ sites. Ideally, you still want to stick to one location. Not because of any resource constraints but rather to avoid unnecessary duplication that can cause confusion. If an item of content needs to appear ‘somewhere else’ then it should be a link rather than a copy, unless a copy is required to fit a different scenario (e.g. publishing a copy of a case study up onto a public web site, but keeping the original held in a location that can only be edited by authors)

To improve relevance of search results, thesauri and controlled vocabularies can help bridge the language barriers between authors and readers. A new starter might be looking for the ‘employee manual’. What they don’t know is what they are actually looking for is the ‘corporate handbook’ or ‘human remains guide’ that may contain the words ‘employee’ and ‘manual’ but never together in the same sentence. The majority of search frustrations come from information seekers using a different language to the one used by the authors of the information they seek. Creating relationships between different terms can dramatically improve relevance of search results. Creating tailored results pages (a mix of organic search results and manufactured links) can overcome differences in terminology and improve future search behaviour.

And the elephant in the file system – retention schedules. First identify what retention schedules you require to comply with industry regulations and to manage legal risk. And do they apply to all content or only certain content? (I doubt many government organisations have kept junk paper mail for 30 years.) And at what point do they need to be applied? From the minute somebody opens a word processor tool and starts typing, or at the point when a document becomes finalised? This is the area that needs most coordination between information specialists and technologists. As we start to move to XML file formats, life could potentially become so much easier for everyone. For example, running scripts to automatically track documents for certain words that give a high probability that the document should be treated as a record and moved from a community discussion forum to the archive. Automatically inserting codes that enable rapid retrieval of content to comply with a legal request but that have no effect on relevance for regular searches.

On the Internet, Google introduced a tag ‘nofollow’ that could be applied to links to prevent the link improving a page’s relevance rank. (PageRank works by determining relevance based on the number of incoming links to a page. If you want to link to a page so that people can look at it but you don’t want the page to benefit from the link in search results, you can insert ‘nofollow’). Maybe Enterprise Search solutions need a similar method. Different indicators for metadata that helps describe content for searches versus metadata that organises content for retention schedules versus metadata that helps authors remember where they left their stuff. And again, XML formats ought to make it possible to automatically insert the appropriate values without requiring the author to figure out what’s needed. The ultimate goal would be to automatically insert sufficient information within individual content items so that requirements are met regardless of where the content is stored or moved to. I email an image to someone and its embedded metadata includes its fileplan(s).

There are lots of ways that technology could be used to improve information management and findability, to meet all the different scenarios demanded by different requirements. But to achieve them requires closer interaction between people making the policies regarding how information is managed, people creating the so-called ‘technology-agnostic’ (in reality it is ‘technology-vendor-agnostic’) file plans to satisfy those policies and the technology vendors creating solutions used to create, store and access the content being created that have to cope with the fileplans and the policies.

The information industry has to move on from the library view of there being only one fileplan. Lessons can be learned from the food industry. There was a time when there was only one type of spaghetti sauce. In the TED talk below, Malcolm Gladwell explains how the food industry discovered the benefits from offering many different types of spaghetti sauce (and why you can’t rely on focus groups to tell you what they want – another dilemma when designing information systems):

Direct link to TED talk (in case video doesn’t load here)

There is a great quote within the above talk:

¨When we pursue universal principles in food, we aren’t just making an error, we are actually doing ourselves a massive disservice¨

You could replace the word ‘food’ with ‘information’. It’s not just the fileplan that needs rethinking…

The same debate could be applied to taxonomy, specifically the use of metadata properties to classify content.

For some reason, most companies who decide they want to improve how content is classified seem to want extreme taxonomy, swinging from not-enough taxonomy to too-much. The mantra may sound somewhat familiar:

One taxonomy to rule them all, one taxonomy to find them, one taxonomy to bring them all and, in the records management store, define them

Often starting with none at all (i.e. content is organised informally and inconsistently using folders), the desire is to create a single corporate taxonomy to classify everything (using a hierarchical structure of metadata terms). An inordinate amount of time is then spent defining and agreeing the perfect taxonomy (for some reason, many seem to settle on about 10,000 terms). Several months later, heads are being scratched as people try to figure out just how they are going to implement the taxonomy. Do they classify existing content or only apply it to new stuff? Do they have specific roles dedicated to classifying the content, rely on the content owners to do it, or look at automated classification tools. Do they put rules in place to force people to classify content and store it in specific locations that are ‘taxonomy-aware’. How do they prevent people bypassing the system, those who figure they can still get their work done by switching to a wiki or a Groove workspace or a MySpace site or a Twitter conversation? How do they validate the taxonomy and check that people are classifying correctly? What do they do if people aren’t classifying correctly, who don’t understand the hierarchy or have different meanings for the terms in use? What started out as a simple idea to improve the findability of information becomes a huge burden to maintain with questionable benefits, given there are so many opportunities for classification to go wrong.

This dilemma reveals two flaws that make implementing a taxonomy so difficult. The first is the desire to treat taxonomy as a discrete project rather than an organic one. Collaboration and knowledge management projects often share this fate. Making taxonomy a discrete project usually means tackling it all in one go from a technology perspective and then handing it over to the business to run ‘as is’ for ever more (i.e. until the next technology upgrade). Such projects end up looking like that old cliché – attempting to eat an elephant whole. The project team tries to create a perfect design that will deliver all identified requirements (and the business, knowing this could be their one chance for improved tools, delivers a loooooong list of requirements), implements a solution and then moves on to the next project. As the solution is used, the business finds flaws in their requirements or discover new ways of working enabled by the technology, but it is too late to get the solution changed. The project is closed, the budget spent.

An alternative approach is to treat taxonomy as an organic project or, for those who prefer corporate-speak, a continuous-improvement programme. Instead of planning to create and deploy the perfect taxonomy, concentrate on ‘just enough taxonomy’. A good starting point is to find out why taxonomy is needed in the first place. If it is to make it easier for people to find information, first document the specific problems being experienced. Solve those problems as simply as possible, test them and gather feedback. If successful, people will raise the bar on what they consider good findability, generating new demands waiting for IT to solve, and so the cycle continues.

The following is a simple example using a fictitious company.

Current situation: Most information is stored in folders on file shares and shared via email. There is an intranet that is primarily static content published by a few authors. The IT department has been authorised to deploy Microsoft Office SharePoint Server 2007 (MOSS)

General problem: Nobody can find what they are looking for (resist temptation to sing U2 song at this moment…)

Specific problems: Difficult to find information from recently completed projects that could be re-used in future projects; Difficult to differentiate between high quality re-usable project information versus low quality or irrelevant project information; Difficult to find all available documents for a specific customer (contracts, internal notes, project files)

Possible solution: Deploy a search engine to index all file folders and the intranet. Move all project information to a central location. Within the search engine, create a scope (or collection) for the project information location. Users will then be able to perform search queries that will return only project information within the results. Using ‘date modified’ as the sorting order will locate information from the most recent projects. Create a central location for storing top-rated ‘best practice’ project information. Set-up a team of subject matter experts to work with project teams and promote documents as ‘best practice’. The Best Practices store can be given high visibility throughout the intranet and promoted as high relevance for search queries.

Now that is a very brief answer outlining one possible solution. But the solution is relatively simple to implement and should offer immediate (and measurable) improvements based on feedback regarding the problems people are experiencing. There were two red herrings in the requirements that could have resulted in a very different, more complex, solution: 1. That MOSS was going to be the technology; and 2. The need to find documents for a specific customer. When you have chosen a technology, there is always the temptation to widen the project scope. MOSS has all sorts of features that can help improve information management and the starting point is often to replace an old crusty static intranet. But the highlighted problems did not mention any concerns about the intranet. That’s not to say those concerns do not exist, but they are a different problem and not the priority for this project. The second red herring is a classic. When people want to be able to find information based on certain parameters, such as all documents connected to a specific customer, there is the temptation to implement a corporate-wide taxonomy and start classifying all content, starting with the metadata property ‘customer name’. But documents about a specific customer will likely contain the customer’s name. In this scenario, the simplest solution is to create a central index and provide the ability for users to search for documents containing a given customer’s name. If that fails to improve the situation then you may need to consider more drastic measures.

Rejecting the large-scale information management project in favour of small chunks of continuous ‘just enough’ improvement is not an easy approach to take. The idea of having a centralised, classified and managed store of content, where you can look up information based on any parameter and receive perfect results, continues to be an attractive one with lots of benefits to the business – both value-oriented (i.e. helping people discover information to do their job) and cost-oriented (i.e. managing what people do with information – compliance checks and the like). But a perfectly classified store of content is a utopia. Trying to achieve it can result in creating systems that are harder to use and difficult to maintain when the goal is supposed to be to make them easier.

I mentioned that the common approach to implementing taxonomy has two flaws. The first has been discussed here – how to create just enough taxonomy. The second flaw is the desire to create a single universal taxonomy that can be applied to everything. I’ll tackle that challenge in a separate post (a.k.a this post is already too long…)

Reference: Just Enough Architecture (MSDN Channel 9). Highly recommended. There are plenty of similarities between software architecture and information architecture (of which taxonomy is subset). Don’t be put off by the techie speak, it debates the pro’s and con’s of formal processes and informal uses, and includes some great non-technical examples for how to find a balance.

Recent related posts:

• Metacrap
• Why taxonomy fails
• When taxonomy fails

Technorati tags: Taxonomy, Tagging, Information Architecture

Sooo, this blog has been a little quieter than planned recently, due to other activities taking priority. But hopefully it will be back on track during January.

A quick post for starters.

I’ve always been more than a little bit interested in the search capabilities within SharePoint, ever since Microsoft introduced probabilistic ranking in SharePoint Portal Server 2001. I can still do a pretty mean explanation of how the Okapi algorithm ranks search results and compare it to PageRank.

Anyways, there is a useful Microsoft blog specialising in the search stuff – Mike Taghizadeh. He’s just written a couple of articles on word stemming. Word stemming helps determine the documents returned when you enter a search query. Mike talks all about it, so here is the short version.

• When you submit a query in SharePoint, the query is broken into individual words. For example, the query “securing the database” would be broken down into “securing”, “the”, and “database”
• Noise words can be eliminated, i.e. common words such as “and”, “the”, “or”, that are unlikely to influence results. In this example, “the” would be dropped from the query
• The query words can then be stemmed for variations. For example, a query for “security” could be expanded to include documents that refer to “securing”, “securely” and so on
• The query words will also be compared against the thesaurus. The thesaurus is customisable and very useful for words with domain-specific alternatives or abbreviations. You can choose to expand queries (e.g. expand “PMB” to also search for “Purple Medium Board”) or replace queries (e.g. replace “ie” with “Instant Everywhere” – “ie” will return just about every document in an English-language index).

In his post, Mike mentions that word stemming is turned off by default. I’ve just checked on my demo laptop and he’s right. If you want to turn on word stemming, here’s how:

• Go to the search page, enter any old query to return the search results page
• Under Site Actions, select ‘Edit page’
• Locate the ‘Search Core Results’ web part (usually in the bottom zone)
• From the Edit button, select ‘Modify shared web part’
• In the tool bar on the right hand side, under ‘Results Query Options’, check the box labeled ‘Enable Search Term Stemming’

And hey presto, it’s switched on.

Now, before you go automatically enabling the feature, despite it seeming obvious to use it, be warned. Word stemming can affect the relevance of your search query. If some terms have lots of stemming and others have none, one word may now dominate results even if it isn’t the priority in the context of what you are looking for. Stemming can also negatively affect performance – there will be a delay whilst expanding the search query to include stemming, and a larger set of results will be returned.

Technorati tags: SharePoint, SharePoint 2007, MOSS 2007