Content Value Analysis for Intranets Part 2 – A methodology

October 8, 2008

content2I’ve written before on an approach called Content Value Analysis (CVA) and I’ve now produced the detail of how this might be done in practice. Chiara Fox of Adaptive Path in her presentation on Content Analysis came closest to this approach when she talked about Content Audits at Euro IA 2008.

However there is a crucial difference between our approaches – the use of statistical techniques and documented heuristics. No don’t reach for that mouse yet! What I’m suggesting is not difficult, will give great substance to your final analysis and will probably save you a lot of work.

When trying to tell an intranet manager or a content owner the hard truth (e.g. their content sucks) it can sometimes result in a ‘ yes it does – no it doesn’t ‘ sort of debate that usually goes nowhere. By introducing random sampling and using a statistically calculated sample size it will be possible to state with absolute confidence that –

– The results have a very high probability that they are representative of all the pages in the site or sub-site being assessed

– All sample pages were arrived at randomly and the process was free of any bias

In order to achieve this you need three things –

Sample size – Using the correct sample size means that you will assess the minimum mumber of pages to ensure that you can be confident in your result whilst also ensuring that you don’t waste time and effort assessing more pages than you should. Assessing more pages than the sample size will not improve the accuracy of the result.

Sample size can be quickly calculated using a statistic sampling table. The one I have pointed to is used by the US military for assessing deliveries of parts in ‘lots’ or ‘batches’. I would suggest using an AQL of 0.25% (Acceptable Quality Level is the number of parts you would accept as being defective but in our case it means that at worst there would be a 0.25% or 1 chance in 400 that we were wrong). In this case a web site containing 1,000 pages would require a sample size of 75 and a web site containing 15,000 pages would require a sample size of 135. As the ‘lot’ or size of the web site is presented in ranges the number of pages can usually be approximated.

If there is little time or there is a need to benchmark a lot of sites a smaller sample size might be selected e.g. 2.5% meaning that the sample size for a web site of 15,000 pages would be 35. The downside of this is that, at worst, there is a probability that you might be wrong of 1 in 40. However if all you are looking for is a ‘ball park’ estimate this ‘CVA lite’ approach might be of value.

Random selection of pages – This took a bit of thought and only when I remembered using my old calculator to generate random numbers years ago did the penny drop. All you need is a set of random numbers (I used random.org). Ring sets of numbers randomly into groups, as many in each group as you like (make sure that there are some single numbers in there too). Simply use the random numbers to select links starting from the home page until you arrive at the last number in the group (you can set the maximum number generated so if you know the most nav links is 8 then set the limit to 8). If the first number is 7 select the seventh link in the navigation bar, then if the next number is 2 select the second link on that page etc. until you arrive at the last number in the group. This will be the page you will assess. If you find that a particular set of numbers do not work simply go to the next group and try again but my approach would be to include contextual links also if they are used . Once the page has been assessed carry on going deeper into the site until you finally can go no further. Then return to the home page and start again. Carry on until the sample has been completed.

Heuristics  So OK, you now know many pages you need to look at and you can also select them randomly but once you arrive at a page what then? You need to document the heuristics that you are going to use and ensure that the intranet manager or content owner also get a copy before the assessment so that they can’t argue afterwards. My suggestions are as follows –

No content value = 0

Good content value = 1 

Obsolete content = 0

Content outside review date = 0

Irrelevant content = 0

Incomplete content = 0 to 0.5 depending

Content not reflecting the link title – 0 to 0.5 depending

Context of page not clear (when assessing it will be like coming from a search page – Is it clear where you are? Does the content make sense?) = 0 to 0.5 depending

Over usage of jargon and acronyms = 0.5

Bad grammar and spelling = 0.5

Once you have assessed a page any score of zero would give an overall score of 0 for the page and any two 0.5 scores would also equal a zero score.

You will also need to record each page visited and the details of the assessment (I’m putting together a form that I am intending to use myself, if you want a copy please post a comment). Then you need to analyse the results and look for patterns. This part is vital. Do it well and it will provide incontrovertable evidence of the content value.  You can go down the route of calculating the standard deviation , which will give you a predicted range of results, but in most cases an average figure presented as a percentage will be more than sufficient. Percentages are great – if you can say that on average 48.9% of a site’s content sucks then you are more likely to be believed.

Ensure that the results are presented in easy to read graphics so everyone can understand the analysis. Strangely most managers I’ve worked with don’t question what I’m telling them if I present the information slickly as a graph or bar chart.

This may seem a lot of work but it is much easier than carrying out a content inventory and for a couple of days work you will get an extremely accurate snapshop of the content value of a site. A snapshot that will be very hard to argue against.

CVA Flowchart  content_value_analysis_flowchart1

You can also download the Excel CVA form from the Information Architecture Institute’s Tools site

(Thanks to ‘Here’s Kate’ Kate Andrews Flickr photo as a metaphor of how most intranets might look in real life!)

31 Responses to “Content Value Analysis for Intranets Part 2 – A methodology”

  1. ewa Says:

    i like the approach – seems to be providing arguments hard to disagree with it…. would like to see the form you put together for asssessment if that’s ok. Rgrds. Ewa

  2. patrick c walsh Says:

    Ewa,
    Thanks for your comment. I’ll upload the form once its finished

  3. Pat Says:

    Patrick, I think I can see where you’re going with this, and I commend you on innovating the humble content audit🙂

    But maybe an example or diagram would help (as you suggest things are much easier to comprehend when presented in a chart or diagram). At the moment it feels a bit abstract and academic.

    Oh and seeing the form or spreadsheet you use to record the results would help too. I eagerly await that.

  4. patrick c walsh Says:

    Pat,
    Thanks for your comment. Good idea about the diagram and I will try and get something up for the form in the next few days.

  5. patrick c walsh Says:

    I’ve now attached the flowchart and will try and get the spreadsheet up soon

  6. patrick c walsh Says:

    I have now raised a Content Value Analysis Record sheet but, as it is in Excel the blog won’t allow me to upload it. If anyone wants a copy please post a comment.


  7. I’d love to see the spreadsheet if you are stilling sending it out. Very interested in how this tool can help us in an upcoming redesign. Thanks!

  8. patrick c walsh Says:

    Nathan,
    I have sent the spreadsheet to your e-mail address. Please let me know if it proved of any use during your re-design.

    Best of luck

  9. Pat Ratulangi Says:

    Love the methodical approach – would appreciate if you could share your spreadsheet as well.
    thanks!

  10. patrick c walsh Says:

    Pat,
    I’ve sent a copy to your e-mail address. Let me know if it works out OK

    Best of luck


  11. […] Content Value Analysis for Intranets Part 2 – A methodology When trying to tell an intranet manager or a content owner the hard truth (e.g. their content sucks) it can sometimes result in a ‘ yes it does – no it doesn’t ‘ sort of debate that usually goes nowhere. […]

  12. patrick c walsh Says:

    If anyone wants to access the CVA spreadsheet it is now available from the Information Architecture Institute website http://www.iainstitute.org/tools/


  13. […] For a more complete description of the methodology see Part 2 […]


  14. Patrick, this is excellent. Thanks for the detailed methodology and accompanying tools. I plan to try this out on a future project.

  15. patrick c walsh Says:

    J. Todd Bennett,

    Many thanks for the feedback. I’d be delighted if you could post the outcome

    Best of luck


  16. […] people randomly and ask permission first.  As discussed in Content Value Analysis ensuring the sample is random is crucial in ensuring that the results are robust. A sort of […]

  17. Alexwebmaster Says:

    Hello webmaster
    I would like to share with you a link to your site
    write me here preonrelt@mail.ru


  18. Hey, cool tips. I’ll buy a glass of beer to that man from that chat who told me to visit your blog🙂

  19. Thanh Sibell Says:

    Sincerely, your article goes to the nitty-gritty of the topic. Your pellucidity leaves me wanting to know more. I will instantly grab your feed to keep up to date with your website. Sounding Out thanks is simply my little way of saying bravo for a remarkable resource. Accept my warmest wishes for your incoming publication.


  20. […] traditional Content Inventory won’t achieve these goals. A technique such as Content Value Analysis will as it relies on sampling which means that, as an initial exercise, it can be completed in a […]


  21. Ewa,
    Thanks for your comment. I’ll upload the form once its finished
    tr..


  22. […] web…Sind die heutigen In… on Intranet 2.0: the need for …estetik Cerrahi on Content Value Analysis for Int…patrick c walsh on So what are the real […]


  23. […] Comments Elaine on Boom and bust – the intr…Designing intranet s… on Content Value Analysis for Int…Designing intranet s… on Boom and bust – the intr…Интерфейсы / […]

  24. heidisaz Says:

    Hi Patrick,
    I’m stumbling my way thru and teaching myself how to do content analysis on a higher-ed website. Your methodology confirms my gut instinct of how to attack the project. I would be grateful if you’d share form you’ve used to track analysis.
    Thanks!

  25. Minna Says:

    Admito que me tienes enganchada. Un fuerte abrazo

  26. Jay Says:

    I would love to see what you’ve done. Would you be able to send me a completed sample?

    • patrick c walsh Says:

      Jay,
      Sorry but I can’t. I’ve changed jobs since the post was done and it would be difficult to get any agreement to make these documents public.

      However if you want to have a go and you run into any problems let me know and I’ll see what I can do to help. Perhaps bets to start with a small section of the intranet (for me one I know has poor content) and do a small scale exercise first to see if it works for you

      Best of luck
      Patrick


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: