How to pass AI training assessments

On task marketplaces such as Outlier and DataAnnotation, the assessment does the work of an interview. You are given tasks much like the paid ones, and what you hand in decides whether paid work ever appears. This guide covers what these tests measure, how to prepare, how to work through them, and the mistakes that sink capable candidates.

What an assessment is testing#

Platforms do not publish their answer keys, but OpenAI's 2022 InstructGPT paper describes one screening in unusual detail. Candidate labelers flagged sensitive content and ranked model outputs, and both tasks were scored by agreement with the researchers' own labels. They also wrote responses to delicate prompts, which the researchers rated on a seven-point scale. The paper reports soft cutoffs of 75 percent agreement and an average score of 6 out of 7 on the written responses.

Platform assessments cover more ground, from code to chemistry, but expect the same logic: your judgments are compared with expert ones, and your writing is held to a rubric. The skills fall into six groups.

Perfect agreement is not the bar. In OpenAI's 2020 work on summarization, trained labelers agreed with the researchers on 77 percent of a subset of comparisons, while the researchers agreed with each other on 73 percent. Experts disagree on close calls; when you face one, a clear justification is your best evidence of sound judgment.

Before you begin#

1. Apply in your strongest track. If you can, take the test as a chemist, a lawyer or a Python developer rather than as a generalist; where to apply first matches profiles to platforms.
2. Clear more time than you need. Pick a window when you think clearly, assume you cannot pause, and plan for longer than any estimate the platform gives.
3. Set up a clean workspace. A computer rather than a phone, a stable connection and a quiet room; if the screening uses video, test your camera and microphone too. Keep one tab for the test, one for the guidelines and one for checking facts.
4. Read every guideline before the first question. Guidelines define what "better" means, and the definition changes with the purpose. For InstructGPT, OpenAI had labelers put helpfulness to the user first when labeling training data, but truthfulness and harmlessness first in its final evaluations. The same pair of responses can rank differently under each rule.
5. Get identity checks right. Use your legal name exactly as it appears on your ID, and verify from the country where you actually live, without a VPN. Outlier's community guidelines, for one, forbid misrepresenting who you are or where you are located.

During the assessment#

Read twice, then plan. On the first pass, find what the prompt asks; on the second, its constraints: audience, length, format, scope and anything the response must avoid. Write the constraints down and check each response against every item; a missed constraint is the easiest error to prove. For writing tasks, outline before you draft: the answer, the support, the order.

Verify what sounds certain. Confident prose hides errors well. A 2023 study titled "Human Feedback is not Gold Standard" found that an assertive tone skewed how many factual errors annotators noticed, and that preference scores under-represented factuality. Anthropic researchers found that people sometimes preferred a convincingly written response that agreed with the user over a correct one. Unless the instructions rule out outside research, check every claim your judgment depends on, starting with the confident ones and those that agree with the user.

Do not reward length for its own sake. A study led by researchers at the University of Texas at Austin found that, in the settings it examined, much of the reward gain from RLHF came from longer responses. A rater who prefers the longer answer by default teaches a model that habit. Judge length against what the prompt needed.

Justify with specifics. A reviewer should be able to check your reasoning in under a minute. Name the decisive issue, point to where it occurs, say why it matters under the guideline, and acknowledge the flaws of the response you preferred. Rubrics that hold up covers the same discipline from the grader's side.

Never paste AI-written text. The test measures your judgment, so a model's words prove nothing about you. Model output also leaves traces. In a 2023 study, EPFL researchers reran a summarization task on Amazon Mechanical Turk and, by combining keystroke data with a classifier for machine-written text, estimated that 33 to 46 percent of the workers had used a language model. AI training platforms sell human judgment to labs, so a contributor caught passing off machine text risks losing the account, not just the attempt.

A weak and a strong justification#

The prompt: "In two or three sentences, explain to a 10-year-old why the sky is blue."

Response A. Sunlight is a mix of many colors. As it passes through the air, tiny gas molecules scatter blue light much more than red light. Scientists call this Rayleigh scattering. So when you look up, blue light reaches your eyes from every direction.

Response B. The sky is blue because it reflects the ocean, which covers most of our planet. That is why the sky looks so blue on a clear day.

A weak justification:

A is better. It is more accurate and more detailed, and B is too short.

It states a verdict, not a reason. "More accurate" does not say what B gets wrong, the prompt did not ask for detail, and B is not too short; it fits the limit.

A strong justification:

A is much better. B's only explanation, that the sky reflects the ocean, is false: air molecules scatter the short blue wavelengths of sunlight far more strongly than the long red ones, which is why the sky is blue over deserts too. A states that mechanism correctly and in plain words. It has two minor flaws: at four sentences it runs one over the limit, and the sentence naming Rayleigh scattering adds jargon a 10-year-old does not need. Neither outweighs a false central claim.

The strong version names the decisive error and corrects it, holds the winner to the same standard, and explains why the trade-off falls as it does. Its verdict and its reasoning agree.

Coding and writing assessments#

Coding#

Coding assessments usually ask you to write a solution, review one, or both, and the code under review is often a model's. Coding for AI labs goes deeper.

• Run it, or trace it. Execute the code if the environment allows; otherwise walk a small input through by hand. Looking right is not evidence.
• Test the edges. Empty input, a single element, duplicates, negative numbers, very large values, unusual characters, and the first and last pass of every loop. This is where plausible-looking code tends to break.
• Check the tests, too. When OpenAI had professional developers re-screen the SWE-bench coding benchmark in 2024, the problems they flagged included task descriptions too vague to define a correct fix and unit tests so specific they could reject valid solutions. If a task ships with tests, ask whether they check what the task describes, no more and no less.
• Explain trade-offs. Say what your approach costs as well as what it buys: a hash map trades memory for speed, and a clever one-liner can trade away readability.
• Write comments someone could act on. "Fails on empty input" is a start. "max(scores) raises a ValueError when scores is empty, but the docstring promises None; add a guard and a test for []" is a finished comment.

Writing#

Writing tasks usually ask for an ideal response, or a rewrite of a weak one. Hold your draft to three standards.

• Structure. Lead with the answer, then support it. Use headings or lists only when the content has that shape, and follow any format the prompt requests exactly.
• Concision. Cut the preamble, the restated question and the closing summary that repeats the body. A word limit is an instruction, not a suggestion.
• Tone. Match the audience the prompt names. Be direct and neutral; skip flattery, moralizing and needless disclaimers. Hedge only where the uncertainty is real.

Proofread twice: once for errors, and once against your list of constraints.

After you submit#

Then you wait, and the wait may end in silence. Silence is ambiguous: your application may still be under review, you may not have passed, or you may have passed with no current project that needs your profile, since marketplaces release work as projects arrive. Whatever the reason, log the date, keep applying elsewhere through the gigs directory, and check the help center before writing to support.

Retake rules are set by each platform. Find them before you start; if you cannot, assume you have one attempt. Do not open a second account to try again. Platform rules commonly forbid multiple accounts, and a breach can cost you the account you already have.

If you pass, expect more tests: on marketplaces, projects often bring their own onboarding, guidelines and short assessments. While you wait for work, ASV's own paid projects are another place to use the same skills.

The mistakes that sink good candidates#

• Rating by taste. Preferring the response you would have written over the one the guideline favors.
• Vague justifications. "More helpful" and "better written" state a verdict without supporting it.
• Mistaking polish for quality. Rewarding length, confidence or agreement with the user without checking the substance.
• Missing a constraint. A word limit, a format, an audience or an exclusion in the prompt, left unchecked.
• Contradicting yourself. A rating of "minor issues" next to a justification that describes a false claim.
• Fixing commas, missing errors. Spending the justification on typos while a wrong figure goes unmentioned.

When you are ready, both platforms below put assessments at the front door. Check eligibility first: each accepts applicants only from certain countries, which rules out many readers in Africa. Current processes, pay ranges and country notes are on the Outlier and DataAnnotation profiles. Where we have a referral link, Apply buttons on ASV use it; our disclosure page explains how that works.

Sources#

• Training language models to follow instructions with human feedback (OpenAI, via arXiv)
• Learning to summarize from human feedback (OpenAI, via arXiv)
• Human Feedback is not Gold Standard (University of Edinburgh and Cohere, via arXiv)
• Towards Understanding Sycophancy in Language Models (Anthropic, via arXiv)
• A Long Way to Go: Investigating Length Correlations in RLHF (University of Texas at Austin, via arXiv)
• Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks (EPFL, via arXiv)
• Introducing SWE-bench Verified (OpenAI)
• Community Guidelines (Outlier)