Survey Says! Measuring Intranet User Response

By Paul Chin

Originally published in Intranet Journal (18-Apr-2003)

back back to portfolio

If a developer builds an intranet in a company and no one is there to use it, does it really make a difference?

While it's not quite as profound as the tree in the forest and it probably won't lead to enlightenment or any higher state of consciousness, it's a valid question that should be asked more often.

There are still some holdouts in the bleeding-edge community who believe that an intranet's success can be measured by the technology used to build it. Regardless of how high-tech and sophisticated an intranet is, it doesn't really do any good unless employees actually use it. If they don't, it will serve no purpose other than the self-satisfaction of those who built it, and perhaps some scattered ooh's and ah's from fellow technophiles.

User response is the true measure of an intranet's success because an intranet is a product that's built by, and for, the user community. In this respect, users are ultimately the ones who will determine whether or not it meets their needs and whether they will accept the system as a valuable day-to-day business tool instead of a casual diversion to be stared at like the bearded lady at a travelling circus.

So, how do you gauge the user response of your intranet? How do you determine who is looking at what types of information? There are two main techniques:

Passive and active methods are not mutually exclusive—they are two halves of a whole—and should be used to complement one another.

Passive Methods

Passive methods allow you to gather hard facts—statistics about who is accessing what types of information and when—without needing to actively involve the end-user community. At the risk of sounding a little too Bogart-esque, I sometimes like to use the term "shadowing your users" because that's precisely what you're doing; you're gathering valuable data about your users' online activity on the intranet without their knowledge.

This is not a covert activity that needs to be done under the cover of darkness with your night-vision goggles and frogman suit. It's standard practice that should be performed on a regular basis by every intranet owner with a large user-base. After all, not all users have the time to send you their feedback.

Web server software such as Microsoft's IIS and Apache's HTTP Server maintain logs of who is accessing the various resources on the site. Although the format of these logs may vary, most log entries contain basic information about site access: the host name or IP address of the requester, the user name (if log-on credentials are required), the date and time of the request, the resource being requested, the return code and the size of the resource. Below is an example of the CLF (common log file) format: - - [31/Mar/2003:22:31:16 -0800]
"GET /Products/Software/index.html HTTP/1.0" 200 97

There's only so much you can do with thousands upon thousands of these log entries, however, because it's raw data without any added-value. You need to be able to quickly process this data into something readable to the average intranet owner. The good news is that you don't have to worry about trying to make sense out of all these log files. There are hundreds of Web site analysis tools that can do the job for you and they come in all shapes and sizes, from commercial and shareware to freeware and open source. A list of many of these tools can be found at Google's Web directory.

Web log analysis tools allow you to compile raw log files and generate detailed, value-added, and reader-friendly reports. Although it depends on the particular software's functionality, at the very least it should provide you with both numerical and graphical representations of site usage. Reports can be generated to show hourly, daily, weekly, monthly hits; statistics for individual users and which pages they have accessed in a given period; which pages have been accessed the most; and bandwidth usage—just to name a few.

WebTrends log analysis

WebTrends log analysis tool showing example of a graphical report

When analyzing your Web server log reports, the most important thing to take into consideration is the difference between casual visitors and actual users. Casual visitors jumping from page to page every few seconds are probably not sticking around long enough to read anything and may inflate the numbers in your reports. But this is the nature of passive methods. They do a great job of delivering hard and objective facts—who is accessing what—but they don't give you any indication of one key aspect of intranet response: user satisfaction. This is why passive methods are not enough to provide you with a complete picture of intranet usage. In order to do this, you have to use more active methods.

WebLog Expert log analysis tool

WebLog Expert log analysis tool showing example of a numeric report

Active Methods

Active methods involve a certain level of interaction with the end-user community—whether through open discussion or gathering data from a survey. By providing users with an outlet to voice their opinion, you'll get a much better indication of user satisfaction that you just can't get by reviewing Web logs alone.

There are many ways to obtain this feedback; the most often used are a simple e-mail address, an online feedback form, or a user survey (online or otherwise).

An e-mail address is the simplest way to allow your users to provide their feedback since it requires no more effort than supplying a link to an e-mail address on your site. However, it's also the least effective. The problem lies in the fact that e-mail is free-form feedback. Users will have the ability to write just about anything they feel is relevant, from their dissatisfaction with the site's navigation system to the way the background colors make their dogs bark and chase their tail for hours-on-end. It will be difficult to mine and graph this type of information with any accuracy, leaving you with a pile of e-mail letters and no way to quantify the results.

In order to make sense out of user feedback you need to provide them with a template. And the best way to accomplish this is by creating an online user survey that gathers responses on key areas of your intranet such as user-friendliness, quality of the site's content, ease in which information can be located, and site cosmetics. The responses to the survey questions can be selected from a drop-down list (such as Satisfied, Not satisfied, or No Opinion), expressed as a numeric rating, or a combination of the two. Many surveys include an area at the end to enable users to input their free-form feedback as well.

Creating a user survey allows you to collect and graph quantifiable results, but make sure that you don't overdo it. It's important to choose your questions carefully—covering the most important aspects of your site—and not to make the survey too long. A survey with 20 well-crafted questions will be more effective than a survey with 50 vague questions, not to mention the fact that users may be discouraged if they're confronted with a lengthy survey.

Unfortunately, as effective as surveys are in gathering information about user response, the failing of active methods is one of human nature. People are much more inclined to voice their opinion when they are dissatisfied with something than when they are happy. This is why stores have complaint departments and not satisfaction departments.

Intranets that meet or exceed user expectation will most likely be taken for granted. Sure, you may get the odd e-mail or phone call now-and-again from a user telling you how much they appreciate the system but more often than not, satisfied users will be a silent majority. On the other hand, if something doesn't quite measure up to par, you're likely to hear an uproar. This may create an imbalance of loud complaints and quiet satisfaction.

You need to even out the scale by getting as many user responses as possible—good and bad. You can place a prominent link to the survey on your intranet's home page to catch users' attention, place a visible announcement in high traffic areas around the company, or offer up some type of incentive for completing the survey (a popular incentive is to hold a drawing among survey participants to win several days vacation time).

Final Thoughts

There are a number of routine tasks that need to be done by any intranet owner such as archiving older information, updating the site with new and relevant information, and checking for broken links. Reviewing and measuring user response should be included on that list as well.

An intranet is built to support your end-users. And similarly, as their needs change, so should your intranet. Only by listening to what your users have to say and by studying their intranet activity can you get an accurate reading on how the system should progress. The success and evolution of your intranet will depend on this feedback.

Copyright © 2003 Paul Chin. All rights reserved.
Reproduction of this article in whole or part in any form without prior written permission of Paul Chin is prohibited.