MTurk online surveys: 3 peeves of mine

I’ve been working on Amazon MTurk a lot recently in the run up to Christmas - it’s a good way of saving for gifts, especially when I know that what I plan to buy is sold on Amazon.  But some of the problems with the Mechanical Turk online surveys have been annoying me. 

Don’t get me wrong, I enjoy doing surveys.  They seem like less hassle than some of the other tasks sometimes, the pay varies and is maybe going down gradually over time, but often still isn’t bad in MTurk terms.  The requesters are also generally pretty reliable, especially if the MTurk requester is a university academic or student, so I worry less about not getting paid, or there being some sort of scam going on.

But recently when I search for surveys, there have been more and more scams that are calling themselves surveys appearing (you know the sort, they always offer you an "easy money for 5 minutes work" and say they're going to pay you $20!).  I mean signing up for a credit report can in no way be considered a survey!  Neither can “test my website by entering all your personal details and pressing submit”!  I guess I had considered online surveys to be a haven from the worst scams, but now I’m realizing that I was probably being over-optimistic.

Another really annoying thing that happens to me sometimes with surveys is that I approve the hit, open another window then do and the survey – but when I go back to the original window it tells me that I’ve run out of time.  Basically, the requester has given an absurdly short amount of time to do the survey and I have not noticed.  It’s happened to me before and I think it’s actually the same requester who keeps doing it (I hope it’s not deliberate, but who knows?!)

Which brings me onto my third and final gripe, which never ceases to annoy me.  Why is it that MTurk requesters can block specific turkers, but the turker can’t block a requester that they don’t like?  I am maybe being cynical, but I think it tells you where the turker features in the Amazon scheme of things – down at the bottom of the pile!

Anyway, enough moaning.  I can’t say for sure if I will blog again before Christmas.  If I don’t, have a happy festive season, turkers, and a prosperous new year!

Writing an MTurk blog: A look at the best ones

Sustaining an MTurk blog isn’t always easy.  Like most turkers, I suspect, my enthusiasm for MTurk ebbs and flows.  There are periods where I do a lot of turking, periods where I do just a little each day, and periods where I do no turking at all.  The reasons for this are varied.  Sometimes there are HITs on MTurk that I enjoy and pay reasonably well, so I spend more time on MTurk to exploit them.  Sometimes I am forced to spend more time on MTurk purely because I need some extra money.  Other times, I have other things happening in my life and I haven’t the time, need, or motivation to be on MTurk.

I have been looking at other blogs recently, principally ones that are written by turkers and aimed at other MTurk workers - there aren’t actually that many active ones around, it seems.  The official Amazon MTurk blog, for instance, tends to focus more on the needs of requesters and programmers, than the requirements of turkers.

Judging by some of the other MTurk blogs out there that I have managed to find, many turkers start out with the notion that they can make enough regular money off MTurk to sustain themselves for a long period of time.  Then they gradually become disillusioned if this doesn’t happen and some get angry with being poorly treated, or at the quality of some of the HITs.  Some of the most entertaining and informative writing comes from disillusioned turkers, although their blogs are often short-lived.

“Mechanical Turk Blog” was clearly motivated to write an MTurk blog in order to highlight what he saw as the scams and injustices of MTurk, of which there are many.  I think it is a continual source of disappointment to all of us that Amazon can appear unwilling, or unable, to police the posted HITs better when they are clearly aimed at gathering workers’ contact details for spamming purposes, or worse still, are aimed at collecting personal info to be used for identity theft.  These sort of HITs appear regularly on MTurk, sometimes masquerading as a “survey” or a “website test”.  Other times, turkers are being asked to perform tasks that are of questionable ethics, and/or legality.

Having made his point, “Mechanical Turk Blog” now seems to have melted back into cyberspace.

Turk Lurker’s blog has some entertaining and informative reads in it!  He has a dry sense of humor, referring to Mechanical Turk as the “Slave Yard” and giving his blog the url: “MTurkey”.  Strange that his posts suddenly stop in 2007 and then suddenly reappear in 2011, but these things happen, I guess – maybe his wife had a baby or something?

Eric Cranston wrote an interesting turker blog, but unfortunately it petered out in 2007 after 2 years of posts.

There are many blogs by people on how to make money from home which refer to MTurk in passing, but it they generally just write a single article and they often don’t give you much more information or insight than a general introduction to MTurk for beginners.

Anybody know of any more interesting MTurk blogs?

Recovering from a bad HIT Approval Rate

As any experienced turker will tell you, recovering from a bad HIT approval rate can be a frustrating experience.  It seems to take forever to come back from the damage that just one or two MTurk HIT rejections can do and all the time your HIT approval rate is damaged, your ability to make money online is compromised.

If you are new to turking and aren’t aware, Amazon Mechanical Turk calculates your HIT approval rate by comparing how many of your MTurk HITS have been rejected by requesters for HITs submitted against how many have been accepted.  The resulting percentages are called your HIT rejection rate and your HIT approval rate.  (All these figures can be seen on the Dashboard part of your Mechanical Turk account.)

The problem is that many of the requesters insist on a HIT approval rate of 95% or better as a qualification requirement and if you don’t meet the qualification standard, they don’t allow you do their HITs.  These MTurk HITS are very often some of the best paid ones and so getting a bad HIT approval rate can really damage your ability to make money online.

As I have mentioned before, when I started out with Mechanical Turk, I didn’t appreciate the problems that having a bad HIT approval rate might bring, so I just tried out lots of different HITs without really caring too much whether some of them got rejected or not.  This came back to haunt me and I learnt my lesson accordingly.

More recently, however, I have seen my HIT approval rate drop again through no fault of my own.  This was apparently because, as I found out from turkers at TurkopticonToolbar, a requester who had been reliable in the past suddenly started using new (faulty) software which was wrongly automatically rejecting HITs that were actually completely fine.  I did complain about it, of course, but it was to no avail, as usually happens!  (See 10 reasons why I hate Mechanical Turk)

Anyway, I’ve spent my time recently working my socks off on getting my HIT approval rating back up to 95%.  It isn’t fun.  As well as my choice of HITs being reduced, each time I do a new HIT I am worried about getting another rejection and getting pushed back down again (once bitten, twice shy!). 

I’m not sure if I mentioned it in my comparison of MTurk and Microworkers: ­MTurk or Microworkers, which is best?,  but Microworkers are more generous with when it comes to recovering from a bad HIT approval rate.  With Microworkers your rate will recover naturally with time, even if you don’t do anything - so long as you don’t keep picking up rejections you will be okay.

In conclusion, my advice would be to not let your MTurk HIT approval rate fall below 95% if you can possibly help it.  If it does happen, I would focus on getting it back up by doing ‘safe’ HITs, that’s to say HITs being offered by MTurk requesters that you trust, as you can’t afford to get any more rejections.  Be prepared for a long haul, though - because the HIT approval rate is a proportional figure, you will need many new HIT approvals to make up for just a few rejections.