Over the past year, one of the most pleasurable tasks I have undertaken within AMEC is to help create, alongside David Rockland from KetchumPleon, AMEC’s first international chapter in the USA, consisting of global agency research heads, and US-based AMEC members, including Cision, VMS, Dow Jones and Burrelles Luce. It was from this group that the original idea of the Measurement Principles was developed, culminating, after much hard work, with the presentation, vote and declaration of Measurement Principles at the AMEC European Summit on Measurement in Barcelona. These Principles were voted on, and agreed by nearly 200 delegates, and included 7 key points, namely:

1. Importance of Goal Setting and Measurement
2. Measuring the Effect on Outcomes is Preferred to Measuring Outputs
3. The Effect on Business Results Can and Should Be Measured Where Possible
4. Media Measurement Requires Quantity and Quality
5. AVEs are not the Value of Public Relations
6. Social Media Can and Should be Measured
7. Transparency and Replicability are Paramount to Sound Measurement

The full text can be found here

As the AVE debate has been such a touchpoint of debate in the sector over the past few years, it was perhaps inevitable that Principle 5 would be the principle trigger for follow up comment from clients, agencies and measurement suppliers. Generally speaking, whilst nearly all comments have been supportive of eliminating AVEs as a meaningful (or accurate) reflection of PR effort and impact, the immediate follow up question is nearly always: “so what should we use to replace AVE?”.
In the detailed background to the Principles, the following text was agreed by the Summit delegates: “Where a comparison has to be made between the cost of space from earned versus paid media, validated metrics should be used, stated for what they are…”
In considering next steps for the Principles, AMEC’s US Agencies Research Leaders chapter decided to expand the discussion around validated metrics, in order to identify how best to create and implement measures that could be both meaningful and actionable. Formalised into one of two AMEC Taskforces set up to take the Principles forward, the group’s focus is on creating a framework document that would help all sides (clients and suppliers) identify metrics that would be more business focused, and certainly more consistent and robust than AVE.
Although this is still work in progress, after much debate, the group agreed that whatever metrics are selected, they could only be considered validated where they are based on the very first Measurement Principle, which asserts that goal setting is paramount to measurement, and also on the third principle which asserts that measurement should be related to business outcomes wherever possible. Taking these two principles as our starting point gave a clear direction for a framework in which the metrics we would consider could be validated against a communications model that is commonly used to reflect business process – namely the marketing cycle.
Our current work in progress is therefore to identify how various broad communications activities – ranging from brand and product marketing, through crisis and issues management to employee engagement – can utilise specific metrics that would be valid for that activity, and pertinent to each stage of the purchase/behaviour cycle. The four stages of the process are: awareness → knowledge → interest → intent/support, which taken together would all, in an ideal world, lead to action.
A first draft has been issued for more consideration, but even in its raw form, I’m convinced that this framework, if generally accepted, will form the basis for a real shift away from AVE to more meaningful and robust, business-focused metrics.
From a supplier perspective, the thinking from the taskforce will provide us with a clear set of alternative metrics to present to clients seeking to escape the AVE maw. Importantly, these new metrics are not the product of theoretical or academic thinking, but are grounded in practical application. Having both suppliers and clients working on these metrics has ensured not only a unique consensus, but also a better understanding of what can be delivered in practical terms, against what clients would aspirationally like to achieve.
From the client perspective, my personal hope is that, because these ideas around validated metrics represent a broad shared view, there can now be a more meaningful debate between communications professionals and their management colleagues – a debate that will focus on real business goals and strategies, rather than endless toing and froing about how best to determine the value of PR efforts in simplified $ terms.
Someone much wiser than me said that all too often, metrics are based upon what it is easiest to measure, rather than what should be measured. I believe that this new framework, when it is released at the New Hampshire Summit in October, will represent a major breakthrough in moving towards meaningful and valued measures of ROI.

London, 27th August 2010.

But there’s an unstated assumption behind the technology promise: that it is necessary to analyse all or a very large percentage of these conversations in case we miss something. Given that the overwhelming majority of blogs and social media sites have an audience of two (the author and his mother), it’s hard to imagine there is much real influence being exerted.

Even if we did want to track every single conversation, your assertion that automated analysis can yield accurate and consistent measures of sentiment flies in the face of research we conducted recently among a global sample of developers, practitioners, academics and users of these tools. We found no system capable of delivering reasonable accuracy levels around sentiment – certainly nowhere near the levels needed for making business decisions.

We have found an enduring demand for human-based measurement programmes – humans can discriminate irony and sarcasm, they can interpret rules, not just follow them, and they are flexible in dealing with new topics and issues… certainly not computers’ strong points.

View the full text from Research Live magazine