Evaluation

The most obvious evaluation focuses on the text itself.  Once you have drafted a text, you can evaluate whether the text will succeed for the audience you assessed during planning.

There are 2 groups of text-based evaluation methods:

1. Tool based
2. Expert review

For many, the common tool-based method would be a checklist that sets out the elements for a clear communication, such as a logical structure, plenty of white space and a short average sentence length. Increasingly, there are software products that automatically assess a text (particularly the expression) to estimate its readability or the level of active voice.

All of these tools provide useful information, but you should never rely on them alone to predict whether a document will work. And make sure that they cover a range of elements rather than just one indicator (such as readability).

The next step up is to have an expert reviewer evaluate the text. This might be an editor, technical writer, or a usability specialist. They may use a standard or apply methods such as user and task analysis to help them. 

Text-based evaluation is usually the quickest and least expensive quality control. It can quickly highlight major opportunities that will improve a text.

The second type of evaluation moves from looking at the text itself to testing it with people who are representative of your readers.

User-focused evaluation methods fall into 2 groups:

1. Preference testing
2. Performance testing

Preference testing involves asking readers what they think about a text. You can do this one-on-one, in small focus groups, or even using large scale surveys.

The most common example would be asking a colleague to take a look at a document before you send it. In more complex tests, you might set up a protocol with a series of standard questions that you give to a large number of users.

Performance testing moves from asking readers their opinion about a text to evaluating how it actually performs. Often this can reveal that the real performance differs from what users expect.

The easiest performance tests are 'think-aloud' interviews, where a tester will sit with a user as they read a document and describe their experience when attempting to use it.

The third area of evaluation looks at the actual outcomes that a text achieved once you have published or distributed it.

This can then help you improve a text iteratively if it has an ongoing use (such as a web page) or it is going to be reissued to other readers (such as an insurance policy). But even reviewing the outcome of a simple email can help you improve future emails that you write.

There are 2 groups of outcome-focused evaluation methods, and they mirror the 2 user-focused methods:

1. Attitudes and behaviours
2. Transactions

The more qualitative methods assess whether a text secured a reader's agreement or changed something in their behaviour.

The simplest example is whether your reader approved a decision that you recommended, such as in a proposal or a submission or even a letter. More complex qualitative outcomes might be whether readers decided to change a behaviour, such as in response to a public health campaign.

In many cases, you may be able to measure the performance of a text if it has a quantitative outcome. For example, a sales letter or marketing email might elicit a measurable response rate that translates into new sales. Or the error rates in processing a form might fall, leading to reduced (and measurable) costs.

Outcome evaluation can be the most powerful evaluation method of all in that it provides evidence of the actual impact of a communication.  Yet because some of what it finds comes 'after the event' it is easy to neglect it and what it can teach us for future communication.

Selecting the right evaluation methods will depend on the context for a communication. Factors that will influence your choice include the:

• number of readers
• complexity of the content
• urgency of the communication
• potential impact
• cost of evaluation
• risks involved.

Generally, when you are writing a document with a high number of readers that will have a significant impact, it is worth investing in more extensive evaluation. And if you are selling a commercial product, even time-consuming testing will be justified if it improves the returns.

On the other hand, if there are few readers and there is little impact or risk, then you won't want to invest in full-scale testing. But you should still do some low-scale text evaluation and assess the outcomes.

The key is not to prefer one method over another, but to select the right mix for each context.

Also bear in mind that most methods can be scaled to size. You might do some performance testing by taking a few colleagues for a coffee rather than pay for a usability lab. But the insight this gives you can make all the difference to success.