I’ve written before on an approach called Content Value Analysis (CVA) and I’ve now produced the detail of how this might be done in practice. Chiara Fox of Adaptive Path in her presentation on Content Analysis came closest to this approach when she talked about Content Audits at Euro IA 2008.

However there is a crucial difference between our approaches – the use of statistical techniques and documented heuristics. No don’t reach for that mouse yet! What I’m suggesting is not difficult, will give great substance to your final analysis and will probably save you a lot of work.

When trying to tell an intranet manager or a content owner the hard truth (e.g. their content sucks) it can sometimes result in a ‘ yes it does – no it doesn’t ‘ sort of debate that usually goes nowhere. By introducing random sampling and using a statistically calculated sample size it will be possible to state with absolute confidence that -

- The results have a very high probability that they are representative of all the pages in the site or sub-site being assessed

- All sample pages were arrived at randomly and the process was free of any bias

In order to achieve this you need three things -

Sample size – Using the correct sample size means that you will assess the minimum mumber of pages to ensure that you can be confident in your result whilst also ensuring that you don’t waste time and effort assessing more pages than you should. Assessing more pages than the sample size will not improve the accuracy of the result.

Sample size can be quickly calculated using a statistic sampling table. The one I have pointed to is used by the US military for assessing deliveries of parts in ‘lots’ or ‘batches’. I would suggest using an AQL of 0.25% (Acceptable Quality Level is the number of parts you would accept as being defective but in our case it means that at worst there would be a 0.25% or 1 chance in 400 that we were wrong). In this case a web site containing 1,000 pages would require a sample size of 75 and a web site containing 15,000 pages would require a sample size of 135. As the ‘lot’ or size of the web site is presented in ranges the number of pages can usually be approximated.

If there is little time or there is a need to benchmark a lot of sites a smaller sample size might be selected e.g. 2.5% meaning that the sample size for a web site of 15,000 pages would be 35. The downside of this is that, at worst, there is a probability that you might be wrong of 1 in 40. However if all you are looking for is a ‘ball park’ estimate this ‘CVA lite’ approach might be of value.

Random selection of pages – This took a bit of thought and only when I remembered using my old calculator to generate random numbers years ago did the penny drop. All you need is a set of random numbers (I used random.org). Ring sets of numbers randomly into groups, as many in each group as you like (make sure that there are some single numbers in there too). Simply use the random numbers to select links starting from the home page until you arrive at the last number in the group (you can set the maximum number generated so if you know the most nav links is 8 then set the limit to 8). If the first number is 7 select the seventh link in the navigation bar, then if the next number is 2 select the second link on that page etc. until you arrive at the last number in the group. This will be the page you will assess. If you find that a particular set of numbers do not work simply go to the next group and try again but my approach would be to include contextual links also if they are used . Once the page has been assessed carry on going deeper into the site until you finally can go no further. Then return to the home page and start again. Carry on until the sample has been completed.

Heuristics  So OK, you now know many pages you need to look at and you can also select them randomly but once you arrive at a page what then? You need to document the heuristics that you are going to use and ensure that the intranet manager or content owner also get a copy before the assessment so that they can’t argue afterwards. My suggestions are as follows -

No content value = 0

Good content value = 1 

Obsolete content = 0

Content outside review date = 0

Irrelevant content = 0

Incomplete content = 0 to 0.5 depending

Content not reflecting the link title – 0 to 0.5 depending

Context of page not clear (when assessing it will be like coming from a search page – Is it clear where you are? Does the content make sense?) = 0 to 0.5 depending

Over usage of jargon and acronyms = 0.5

Bad grammar and spelling = 0.5

Once you have assessed a page any score of zero would give an overall score of 0 for the page and any two 0.5 scores would also equal a zero score.

You will also need to record each page visited and the details of the assessment (I’m putting together a form that I am intending to use myself, if you want a copy please post a comment). Then you need to analyse the results and look for patterns. This part is vital. Do it well and it will provide incontrovertable evidence of the content value.  You can go down the route of calculating the standard deviation , which will give you a predicted range of results, but in most cases an average figure presented as a percentage will be more than sufficient. Percentages are great – if you can say that on average 48.9% of a site’s content sucks then you are more likely to be believed.

Ensure that the results are presented in easy to read graphics so everyone can understand the analysis. Strangely most managers I’ve worked with don’t question what I’m telling them if I present the information slickly as a graph or bar chart.

This may seem a lot of work but it is much easier than carrying out a content inventory and for a couple of days work you will get an extremely accurate snapshop of the content value of a site. A snapshot that will be very hard to argue against.

(Thanks to ‘Here’s Kate’ Kate Andrews Flickr photo as a metaphor of how most intranets might look in real life!)

CVA Flowchart  content_value_analysis_flowchart1

It might go something like this -

- Sampling would have to represent a fair percentage of the content and samples would have to be taken from several areas of a site or sub-site. Areas must be selected without bias. In an intranet with 1,000 pages a fair sample might be at a minimum 5%, that is 50 pages, or you could use statistical tables for sample sizes.

- Each page would be examined and scored for its value against a set of heuristics e.g. if content is obsolete, outside of a review date, incorrect, incomplete or irrelevant the score would be 0. If there is some value on the page it could be marked as 0.5 and if the page content generally has value the score would be 1. Perhaps screen dumps of representative non-value adding pages might form part of a results document. 

- The scores would then be aggregated and presented as a percentage e.g. if after appraisal of our sample 50 pages the score is 28.5, then we could present the site as providing only 57% of the possible content value to its users

This might encourage content owners to improve their scores by ensuring that non-value adding pages are deleted, reviewed or that good content is added. I think that the trick in making this approach work will be in defining the heuristics and finding a way of removing any bias from the sample.

Using this approach (I call it ‘Content Value Analysis’ as it has to be called something) it should be possible to benchmark sites and parts of sites against each other by using the same heuristics and, if done periodically, to chart improvement trends in the value a site offers its users.

Hopefully this approach will provide intranet managers with a tool that allows them to assess content in a structured way, produce a number that will accurately reflect the state of the content within the site evaluated and then to repeat the process over time to ensure that the intranet is going in the right direction - towards the lean intranet. Results can then be presented graphically as bar charts or graphs - a form of information that most senior managers are comfortable with and understand.

For a more complete description of the methodology see Part 2

I write about content centred design in my article as it applies to content sets such as ISO management system standards in intranets. Content is what an intranet should be all about. Intranets should be honest, minimalist systems for connecting users and content without the flim-flam you can encounter in some internet sites. Content must be current, relevant and complete, thats a given, but there is more. Content must be viewed as a stakeholder.

Content has been developed by a human to speak to other humans therefore it must be recognised that content has a structure and a purpose which is independent of the user or web designer. This fact is probably more obvious when you are looking at intranet design as an organisation’s policies, procedures, guidance, technical information etc are obviously valuable company assets. I’ve been thinking as to whether this is also the case in the larger world of web design and the internet.

There is a striking case where web design has run roughshod over the structure and purpose of content and that is in online dictionaries. Dr. Johnson’s dictionary of 1755 was an alphabetical dictionary.  The structure of the content allowed users to access entries so long as they knew the first letter or couple of letters. Most of the online dictionaries I have seen have disregarded this aspect altogether and consist of a simple search box with lots of adverts. While the search box is fine if you know the word and its exact spelling, it is absolutely useless in other cases.

If I wanted to know that word that begins ‘per’ and means a way of looking at things, how does a search box help me find the word ‘perspective’? The alphabetical dictionary has been around for a long time and with good reason – it works. Any design of an online dictionary that ignores this fact makes for a poorer user experience. As well as a search box there should always be the option to view the content in the way it was originally structured – alphabetically.

But if I’m honest opening up a dictionary at a random page and reading down a list of words connected only by the fact that they happen to have some of the same letters is a real pleasure for me. It is serendipity in action when you find a new and beautiful word. Unfortunately a pleasure denied me and others by most online versions of the dictionary.