Last year, I wrote a post about what managers searched for in our products. Essentially, this was a summary of what we found when we carried out out our regular search term analysis which we conduct quarterly, but taken over the whole of 2010.
This year, I wanted to be a bit more thorough and we made some changes to the process. So I asked a member of our Editorial team, the wonderful Jess Adair, to summarise what we did. This will give you an idea of what was involved and might help if you want to carry out a similar activity in your own organisation as part of a learning needs analysis.
You’ll find the results and some analysis at the end of the post, but how we got there is just as important as what we found.
Over to Jess …
Our first stop was to use our analytics package to generate a list of all the individual terms our users searched for in 2010 and 2011 .
We knew, however, that this alone would only go some way to tell us which management topics were most popular with our users. Because the report returned every unique term our users searched for, the list generated was, unsurprisingly, extremely long and with a good deal of overlap between some terms (e.g. ‘change management’, ‘change’ and ‘Change Management’ were all listed separately ).
Faced with such a long list, we needed to decide on a sensible cut-off point for our analysis; we knew that looking at every single search term, including the thousands that had only been searched for on one occasion, would not be particularly helpful. We noticed that after a certain point in both the 2010 and 2011 lists, the numbers of ‘unique’ searches dropped off quite dramatically. With this in mind, and given that many subsequent search terms were actually synonyms of the more popular words and phrases, we decided to restrict our analysis to identify the 30 most searched for terms in 2010 and 2011.
Our next job was to develop a method for identifying all of the synonyms for each of the search terms we wanted to analyse. To do that, we took the following systematic approach:
1. Identify a key word for each of the search terms (for example, ‘time’ in ‘time management) and search for other iterations of this in each list. In 2010, ‘Time Management’, ‘time management model’ and ‘Time management’ were all popular terms (depending on what analytics package you use, the search terms might be case sensitive), along with ‘time management’, which appeared at the top of the original list.
2. Disregard any results that aren’t relevant to the management concept to which the original search term refers. For example, ‘time sheet’ appeared in the results for ‘time’, but was not included because it does not directly relate to ‘time management’.
3. Look for other words and phrases that relate to the original search term. For time management, we looked for phrases such as ‘workload’ and ‘deadlines’. As before, we disregarded any results that weren’t relevant, and included those that were.
4. Record the total number of synonyms and add this figure to the number of searches for the original search term to create an aggregated total for each of the first top search terms.
5. ‘Rinse and repeat’ to generate an aggregated total for each of the search terms displayed in the original analytics reports.
So what impact did all this have on our results? Quite a substantial one!
An initial review of the data would suggest that ‘time management’ was the most searched-for term in 2010. However, after taking all the synonyms into account, change management shot to the top of the list, with almost twice as many synonymous searches as there were for the term itself. The results took a similar twist in 2011, with leadership - originally the tenth most search for term – soaring to the top spot after we counted its synonyms.
Undertaking this exercise has given us a much clearer picture of the topics that our users were really searching for in 2010 and 2011 – you can see what this looks like in the two lists below (those who are more visual can feast your eyes on the Wordle images we produced at the end of the post ).
But what else do these results tell us about the day-to-day learning needs of leaders and managers?
|Popularity||2010 Results||2011 Results|
|4||time management||change management|
|12||personal development plan||vision (+mission, values, strategy)|
|24||project management||project management|
|25||report writing||learning styles|
|26||medication management||personal development plan|
|30||health and safety||excel|
Owen will carry on the story ...
It may seem a little boring, but the most striking thing about the search analysis exercises we conduct is how little the results change from quarter to quarter, year to year. An interesting story would be how the financial crisis affected what managers were looking for support with, but the truth is that we’ve seen pretty consistent results over the years.
Two key things have stayed the course:
1) Managers seek out content on the ‘basics’ more than any other kind of topic. Change, leadership, communication, coaching … they’re all the core components of what you’d expect to find in any management development programme. A question that arises from this is: if these are core subjects that are covered by management development programmes, why are managers looking for additional support with them so frequently?
2) The on-demand needs of managers are incredibly diverse. In both the use of our content, and the searches that managers make, we see an extreme example of ‘long tail’ demand. For example, in 2011 over a third of all searches across our products were terms that were completely unique i.e. they were only searched for once by one person. Over half of all searches were terms that were used three times or less. Even when you wrap up the synonyms, the content requirements are very broad.
To show you just how extreme this ‘long tail’ demand is, this is a to-scale graph of the search term frequency:
This information is available in your organisation. Every search term and every click on your intranet or LMS can be analysed to draw out hidden patterns and stories that can and should inform your strategy. All it takes is putting the request in to the right person and taking some time to do the analysis.
Although we see a similar picture across most organisations we work with, there is some variation. In a very few cases, this variation is significant. So, it would be wrong to look at our results and assume they automatically apply to your own organisation. However, they do provide a very strong indication that your organisation’s mangers are likely to have very similar needs to the ones highlighted in our results. There’s only one way to find out for sure ...
 If you’re not sure how to do this, your IT department will be able to help you. It’s a five minute job, don’t let them tell you different! For those that are interested, we use both Google Analytics and server side logging to track usage. However, for the purposes of this exercise, Google Analytics was easier to use.
 The analytics service is case sensitive when it comes to search terms.
 The wordle representations of the results, where the more frequent the search, the larger the word appears: