Vendor Reports: Statistics, damn lies, incompetence and outright BS
As you can probably tell from the lead article in InSight #232, I enjoy reading a good report, research paper or ABS release. Using validated and relevant data can provide the basis of a useful and interesting article when that data is dissected, analysed and combined with intelligent commentary. For many years, smart recruitment agencies have been using the data at their disposal to create salary surveys or market reports, which they use as a credibility building exercise with their client and prospect base.
Some recruitment vendors have also gone this path, two of the most prominent examples being the careerone Attracting and Retaining Talent series of reports and the Seek Employment Index.
The reputation-enhancing value of these reports relies on the assertions or conclusions being carefully drawn from a statistically significant, and relevant, sample size. Making assertions based on inadequate data or incorrect calculations from this data, creates the strong likelihood that your attempt at credibility-building will backfire on you.
I would regard a recent recruitment vendor effort, The Bullhorn 2012 Staffing and Recruiting Trends, Asia Pacific as, unfortunately, falling into this latter category.
Just to pick up on a few things:
-
The survey had just 85 respondents, 55 from Australia and the balance from Singapore, Japan and New Zealand. This is a tiny sample size given that in Australia alone, we have somewhere between 2,200 and 3,400 recruitment firms (depending upon which research you believe).
There are enormous differences between agencies that are based in capital cities, recruiting white collar professionals compared to those agencies that are based in the regional areas or suburbs, recruiting technical or industrial staff. Bullhorn fails to provide any summary of the contributing firms’ profile by geography, recruitment specialisation or size. Generalising about such a diverse sector is, potentially, both a minefield and close-to useless in drawing meaningful conclusions about the data.-
The reports states ‘The survey covered recruiters across the region, including Australia, New Zealand, Japan, Singapore and Southeast Asia’. It doesn’t say whether there was only one survey completed per firm (I assume so) or who (their job title) within the agency responded to the survey (a critical piece of information as you will see shortly when the question about ‘your remuneration’ is asked).
The report states ‘average compensation across all levels was A$131,154 in 2011’. This figure of $131,154 is completely meaningless without context. What does ‘all levels’ actually mean ? Is it every employee of the recruitment company, including support staff? Are non-fee earning CEOs/owners included or just fee earners? Does it just apply to the individuals that responded to the survey (hence the need to know their position title)? Nowhere in the Bullhorn report are we told this information.
The response options to the question ‘What is your expected total compensation for 2012 (salary and bonus)?’ were $50k remuneration ranges (ie <$50k, between $50k and $100k, between $100k and $150k etc.) Statistically it’s BS to claim a specific average can be arrived at from a bunch of ranges as distinct from each respondent providing a single remuneration figure. This is an embarrassing error no Stats 101 student would make.By now you are probably thinking something along the lines of, ‘C’mon Ross, this is just a harmless piece of vendor puffery that nobody takes too seriously, why are you getting on your high horse about it?’Well, the problem was that the Bullhorn report became wider ‘news’ when on 16 May 2012 industry news service ShortList led it’s early afternoon market round-up with the (incorrect) headline Average Asia-Pac recruiter earned $131k in 2011.I am unsure what aspect of the report Bullhorn were hoping to have as their headline generator but ShortList, unfortunately (for both ShortList and Bullhorn), decided that the part of the report that was most newsworthy was the aforementioned statement (on page 9) of the report ‘average compensation across all levels was A$131,154 in 2011’.As I articulated earlier, the Bullhorn report does not say that this average figure was for recruiters only, it specifically stated ‘all levels’. Uncharacteristically the normally rock-solid ShortList received a double dose of #fail because they reported very rubbery data as news and then made errors of fact (if you can call the Bullhorn figures ‘facts’), the combination of which made both themselves and Bullhorn look less-than credible (putting it kindly).Worse was to come. After reading the ShortList story, prominent rec-to-rec blogger, Craig Watson picked up the story and wrote a funny and scathing blog post False Recruiting Economy… I blame the Tooth Fairy about the report, quoting the inaccurate ShortList story; a story that was drawn from statistically insignificant, ill-defined and incorrectly calculated data.Craig told me that this particular blog post was his second-most popular post this year, receiving almost double the number of views than his posts normally receive. He also mentioned this post generated a much higher ‘share’ rate on LinkedIn and ‘retweet’ rate on Twitter, compared to his previous blog posts.Bullhorn only have themselves to blame for this PR disaster. Rule number one of releasing reports and research is to check the other similar, publicly available research on the same topic to see how your research and conclusions stack up.The Recruitment Industry Benchmarking (RIB) Report and the Scott Recruitment Recruitment Salary both contain data about average recruiter salaries that would have alerted Bullhorn, had they bothered to investigate, that their claims on average salaries would, at the very least, be bound to raise a few eyebrows across the recruitment community.If you intend to write a market report or something similar, then I would recommend you do the following1) Survey a statistically valid sample size.2) Provide a breakdown of the respondents (and data) by relevant categories (eg size, location, job title etc).3) Ensure your calculations are consistent with the laws of statistics.Scroll To Top4) Ensure your conclusions or assertions are supported by the data.5) Have an external skilled analyst or report writer review your final draft to make sure you aren’t making embarrassing statistical errors or unsubstantiated claims that the data does not support.Bullhorn’s intentions were honourable and I applaud them as a company prepared to invest time and money in providing our industry with useful information.In future, they just need to be more careful and ensure they get it right.
Great Article Ross…. I've had a couple of candidates come through the last couple of weeks believing they were underpaid by… whatever their salary was short of $131k!!
Point taken Ross. We will amend our story to clarify some of the issues you've raised.
100% of New Zealanders surveyed vote this Ross Clennett's best blog post of the year!
OK, the sample size of one isn't really reflective of all of NZ, but who cares? The headline caught your eye, the click through was achieved, recorded and presented to online advertisers looking to place their advertising dollar on the most-viewed web pages.
Such is the way of modern journalism that the facts are irrelevant. Look at the traditional summer transfer news frenzy in the English Premier League where publications like the Sun will practically invent transfer rumours, sometimes based off reading a tweet from a fake twitter account, sometimes invented by player's agents to get their client more money at their current club, and so on. We as consumers of media are to blame for our voracious appetites for scandal, tragedy and intrigue, causing journalists to race to publish attention grabbing headlines without the time to research and corroborate the facts.
Your stance is admirable Ross, but sadly quality journalism has all but died off in the internet age.