1. Participant experience is part of data quality

I have conducted social research for years, and I participate whenever I have the opportunity. Seeing research from both sides keeps one fact visible: the instrument is also an experience, and the quality of that experience shapes what people provide.

What participation reveals

Participating in other people's research makes recurring problems difficult to miss. Instruments can be too long, repetitive, short on context, and weakened by careless screening and branching. I can understand a weak survey produced by someone doing so for the first time. What is harder to accept is the same experience from organisations fielding surveys at large scale. They may have strict requirements for what the instrument must collect, yet no equivalent requirement for the experience to be good for the participant.

One reason these surveys remain weak is that questions are crammed into one of many survey platforms without reconsidering the experience. We have largely replicated the paper questionnaire online rather than asking what an online-first experience should be. Online-first means device-first, and today that also means mobile-first.

The best questionnaires move naturally, make the context of each question clear, and prepare the participant for what comes next. They make it easy to provide the information the researcher actually needs. Their quality makes the prevailing standard harder to excuse.

What the data reveals

The participant experience directly affects the data. I have spent much of my time analysing large survey datasets, so I began to see where people checked out, where a task stopped working, and where an instrument began producing answers that were technically complete but difficult to believe. A finished report rarely shows how much unusable data was removed, or how much doubtful data remained because the study still had to produce a result.

I learned the same lesson earlier, while conducting computer-assisted telephone interviews. I was reprimanded for straying from the script when a question prompted someone to describe a difficult experience. The correct operational response was to acknowledge as little as possible and return them to option A or B. I was not especially good at that job because people do not naturally answer as instruments require. Older participants in particular often wanted to tell me what happened. The questionnaire demanded a fixed response.

An online form hides that encounter. If none of the answers fit, a participant who wants the incentive still has to give the system what it demands: an answer. Paper surveys occasionally reveal what the form excluded. While entering data from a regional public-health study, I found comments written in the margins: profane, profound, and sometimes more immediately relevant than the questions we had asked. People wanted their stories to be heard even when the instrument had nowhere to put them.

The interview and the handwritten margins revealed the same conflict. People were trying to tell us what had happened to them. The instrument was trying to turn that account into a permitted response. Good human research has to preserve enough structure to answer the research question without treating the part that does not fit as noise.

Design for interruption

Research is rarely the highest priority in a participant's life. People answer while travelling, working, caring for someone, waiting for an appointment, losing reception, or simply running out of attention. Interruption should be treated as expected behaviour, not participant failure.

A well-designed service saves each meaningful contribution as it is made, shows the participant what has been retained, and lets them resume from their last completed thought rather than merely reopening the last screen. Once somebody has given us an answer, our process should not make them give it again because we failed to hold or retrieve it. Re-asking can still be legitimate when a participant is revising an answer, reconfirming something that may have changed, resolving a contradiction, or contributing to an intentional time series. The service should say which of those it is doing and why.

The useful invitation is not “please start again”. It is “what you gave us has been retained; continue when you can”. That promise has to include correction, withdrawal, deletion, and recontact choices, with any limits explained before participation rather than discovered afterwards.

US telephone poll response rates have fallen for decades (Kennedy & Hartig, 2019). Respondents under cognitive load may satisfice rather than answer carefully (Krosnick, 1991). In a web-survey experiment, longer stated questionnaires reduced starts and completions, while questions placed later produced faster, shorter, and more uniform answers (Galesic & Bosnjak, 2009). Westwood showed that an autonomous AI respondent could pass standard attention checks while producing coherent, persona-consistent answers (Westwood, 2025). We cannot keep offering people weak incentives and poor experiences, then treat declining participation as something participants have done to us.

Spend human attention carefully

Where possible, do not push a raw question battery onto participants merely because the research team has not narrowed it. Synthetic methods can map a topic, compare structures, screen possibilities, and help sharpen hypotheses before human attention is spent. Client knowledge and existing evidence should do the same work.

A team that begins with a hundred possible concepts might use those signals to identify twelve worth investigating, then ask people to evaluate the twelve. The synthetic stage reduces burden; it does not become evidence of what people prefer. Questions about priorities, trade-offs, circumstances, and lived experience still need answers from people.

I develop that boundary in Synthetic research: claims must match evidence, including what synthetic work can reveal, how it should be tested, and what it cannot claim.

2. Match the method to the question, person, and device

A long web form is only one way to collect human research data. Some questions are suited to a rating or a short choice. Others need a person's own words, a photograph, a short recording, a diary entry, a receipt, or another artefact from the experience. Asking someone to compress a complicated experience into a number can discard the very information we hoped to understand.

Photo elicitation, for example, can broaden and deepen a qualitative interview (Gill, 2024). The principle extends to videos, screenshots, proof of purchase, and material captured at the moment something happens. If entry into a category matters, verified evidence of that entry can be more valuable than another unsupported screening answer. That higher standard of evidence should also earn the participant a higher reward.

Use the device

These experiences should be mobile-first, not merely desktop questionnaires squeezed onto a smaller screen. A randomised crossover experiment found that smartphone respondents could provide careful answers, but small sliders and date pickers created input errors; the task has to be easy to perform on a touchscreen (Antoun, Couper, & Conrad, 2017). We do not have to inherit standard form controls. We can use the whole surface of the device to make one task unmistakable, show exactly what was recorded, and make correction easy.

That does not mean every participant should be expected to swipe, drag, or manipulate the same control. Dexterity, sensation, vision, and familiarity with a device differ. Offer a brief practice step, clear feedback, and an alternative way to answer. The purpose is to help people respond accurately through an interaction they can use.

Interface quality is data quality

I have seen what happens when an instrument is not fit for purpose. In an academic health-data role, I worked with records entered by numerous nurses through an interface that made year of birth easy to enter incorrectly. Implausible dates appeared systematically. For studies of children's growth, date of birth was not a minor field; it was essential to the analysis. The problem was well known and remained unresolved. Interface quality had become data quality.

Making a task easier or more satisfying does not require gamification. In one experience-sampling experiment, virtual rewards increased responses among the people who used them most, but made their momentary reports slightly less reliable (Dejonckheere et al., 2024). Points and badges can become another demand placed on the participant. The better aim is a satisfying interaction: clear progress, appropriate feedback, useful variety, and, where possible, something of value returned to the person.

Two examples of designed participation

These examples put the method into visible form. The first is an interactive prototype; the second is a visual exploration. Neither is a finished product or validated instrument. They are design hypotheses about how a familiar research task might change when participation is designed for the device and the person using it.

The choice task demonstrates one question at a time, a response that does not force a false preference, and interaction traces retained separately for later testing. The voice example demonstrates richer expression, repeated participation, visible confirmation, and direct payment. Neither establishes that the measure is valid, accessible to every participant, or better than an existing instrument. Those questions require separate evaluation.

Recruitment, consent, screening, participation, support, payment, feedback, and recontact form one service from the participant's point of view. That becomes especially visible when researchers return to the same people. Optimising the instrument while neglecting the surrounding service cannot create a relationship worth maintaining.

3. Build panels with relationships

When change is the claim

A fresh sample at each wave can estimate how an aggregate measure differs from one point to the next. It cannot tell us how the same people's views or circumstances changed. If 200 people answer in January and a different 200 answer in April, a flat average can conceal individuals moving in opposite directions, while an apparent shift can partly reflect who happened to enter each sample. If the decision depends on within-person change, the design needs within-person data.

That makes longitudinal relationships especially valuable for commercial tracking. Instead of repeatedly drawing convenient samples from an opaque pool, researchers should be able to return to known participants and ask stable questions over time. Diary methods and experience sampling already show the value of repeated, in-the-moment evidence (Csikszentmihalyi & Larson, 1987; Stone & Shiffman, 1994; Bolger, Davis, & Rafaeli, 2003).

Academic researchers often work under severe constraints. An undergraduate convenience sample, a prize draw, or a small one-off study may be the only feasible way to see whether an early idea has any promise. Commercial research that informs real expenditure has a different opportunity. There is money in the system, and some of it should be used to improve the sample, the relationship, and the experience. Not every improvement costs more, but the parts that do should be treated as research infrastructure rather than avoidable overhead.

A relationship worth maintaining

I am interested in how curated, longitudinal commercial panels can be designed, developed, and maintained: known and segmented groups of people whom researchers can return to, who understand the relationship they are entering, and who are paid fairly and predictably. Their profiles can become richer with consent, their participation can span scheduled studies and rapid pulses, and the provenance of the sample can be described instead of hidden behind a provider's assurance.

An established relationship can also stop every study from beginning at zero. With permission, a panel can maintain verified attributes that are relevant across studies, such as age range, location, household circumstances, category participation, or previous research activity. A researcher can then request the participants who fit the study without asking each person to repeat the same screening and demographic questions. The study should receive only the attributes it needs, and participants should be able to see why those attributes are being used.

Build a panel that can be described honestly rather than promising perfect representation. Recruit people from relevant communities, establish their characteristics with a small set of screening questions, and report results by known groups rather than implying that every result generalises to everyone. The limits of non-probability samples still apply (Baker et al., 2013), but those limits can be described instead of concealed. Participants should also be paid directly for the time and evidence they provide.

Trust has to run in both directions. Researchers often design as though the participant is an adversary to be caught speeding, straightlining, or answering carelessly. Participants have their own reasons not to trust us: repetitive questions, unexplained purposes, weak screening, thoughtless wording, delayed payment, and no accessible way to correct a problem. Every invitation asks them to take a risk on whether this will be a good survey or a bad one. Onboarding has to earn trust rather than merely test compliance.

A durable panel can also make participation habitual without making it extractive. Ask people when they can genuinely focus, perhaps a regular half-hour on Friday morning, and return at the time they chose. A predictable rhythm respects the fact that research invitations otherwise arrive between work, care, and household tasks. Rapid pulses then become part of a relationship, not an unexpected demand.

Give participants control of the relationship

A richer profile creates obligations as well as efficiencies. Participants should be able to inspect and correct what the panel holds about them, understand which attributes will be reused, choose whether they can be recontacted, and withdraw from the relationship. Withdrawal, profile correction, and deletion should be usable parts of the service rather than rights buried in a policy. Consent to join a panel is not unlimited consent for every later use.

The operator also has to monitor what continuity changes. Repeated participation can condition answers, long-serving members may differ from people who leave, and the most burdened participants may be the first to disappear. Track participation frequency, refusals, attrition, profile completeness, and replenishment by relevant group. A longitudinal panel produces better evidence only when those changes remain visible.

What already exists

Pieces of this future already exist. Prolific's longitudinal projects support multi-wave studies that return to the same participants, show them the schedule and payment upfront, and track retention across waves. Dscout's diary tools support repeated mobile activities with photographs, videos, and screen recordings. Australia's probability-based Life in Australia panel demonstrates that a repeatedly contacted national panel can be deliberately recruited, maintained, and evaluated (Kaczmirek et al., 2019). Those are important capabilities. The larger opportunity is to combine continuity, good instrumentation, richer modes, transparent sample quality, and a relationship participants would choose to maintain.

Evidence about who participated can strengthen a panel without validating every answer. A receipt may support category membership, but it does not establish the truth of every response; repeated participation can condition participants; and detailed profiles create privacy obligations. Make these strengths, limits, and provenance visible, and charge only for the quality genuinely established.

4. Set a higher standard for human research

Participants owe researchers nothing. Every invitation asks for time, attention, information, and trust, often from someone who has no prior relationship with the organisation asking. A higher standard begins by treating that contribution as something to earn rather than an input to extract. At minimum, a better research service should include:

• Pointed surveys that test specific hypotheses rather than pushing broad, repetitive exploration onto participants.
• Enough context to explain why the task matters without scripting the answer: an invitation to help with a decision, not a demand to surrender data.
• Longitudinal measurement with the same people when the claim concerns individual change, alongside transparent repeated cross-sectional estimates where those are appropriate.
• Rapid pulses for high-quality answers to small questions between scheduled waves, delivered at times participants have said work for them.
• Interruption-safe continuity through progressive saving, visible checkpoints, exact resumption, and no repeated answer caused by a failure to retain or retrieve prior work.
• Mobile-first tasks using large touch targets, short modules, the full device surface, and interaction modes matched to the participant.
• Low cognitive burden through careful segmentation and prioritisation before fieldwork. If comparing twenty items is unpleasant for the research team, it should not outsource that burden to the participant.
• Richer expression through voice, text, photographs, videos, diaries, receipts, and other relevant artefacts.
• Unmistakable confirmation so people can see exactly what the system recorded or classified and amend it immediately.
• Formal, separate validation of identity, category participation, attentive completion, consistency, and ability to use the response mode accurately.
• Direct, predictable payment that reflects the time, sensitivity, and evidential value of what was requested.
• Something returned where appropriate: a useful reflection, personal result, or clear account of how the contribution mattered.

Operate the complete journey as one service

Someone has to own the participant journey from invitation to final payment and recontact. Recruitment, screening, consent, the research task, support, correction, payment, feedback, withdrawal, and closure should be tested together before fieldwork begins. Outsourcing recruitment or hosting the instrument on another platform does not outsource responsibility for what the participant experiences.

Maintaining a minimum standard requires operationalisation. “Respect participants” is a value, not yet a control. The service has to define what a participant must be able to observe, the threshold that must still hold when time or budgets are tight, the evidence that shows whether it happened, who owns the result, and how a failure is repaired. Progressive retention, visible confirmation, refusal without penalty, predictable payment, and correction that reaches every downstream use still under the operator's control are examples of standards that can actually be tested.

Measurement validity and service quality need separate checks. A well-worded scale can still sit inside a confusing journey, while a polished interface can still collect the wrong construct. Test whether the instrument measures what the claim requires, and separately observe whether people understand the task, can provide the answer they intend, can recover from mistakes, and receive what they were promised.

Completion is not enough evidence that the service worked. Record abandonment, forced or amended answers, repeated prompts, support requests, payment delays, complaints, refusals, and willingness to participate again. Review those signals by device, response mode, and relevant participant group after every wave. Participant corrections and complaints are not administrative noise; they show where the service and the resulting data may have failed together.

Incentives often increase participation, but their effects depend on the mode, amount, timing, and population. In a meta-analysis of mail surveys, rewards included with the initial mailing increased response rates, while rewards conditional on return did not show the same effect (Church, 1993). A broader review likewise found that the effects and costs of incentives vary across survey designs (Singer & Ye, 2013). For a durable panel, payment also carries a relational message: careful participation is work, and the organisation values it.

5. Human research is a service-design opportunity

Humans belong wherever the evidence depends on lived experience, meaning, priorities, circumstances, or change within a person over time. Synthetic methods can narrow the field and help researchers ask sharper questions. They cannot provide the human evidence.

Better human research data requires better-designed participation: sharper questions, richer ways to respond, and durable relationships with known participants.

Organisations can continue buying access to a new sample for each study, asking the same screening questions, and accepting little visibility into the relationship behind the dataset. A higher standard is to understand and describe who is participating, use consented information without demanding it again, pay people properly, return to the same people when continuity matters, and give them ways to express more than a checkbox permits.

Not every organisation can build the complete model immediately. It can still make the next study shorter, preserve an interrupted response, improve correction and payment, reuse an established attribute with permission, or return to the same participants for one important question. Each improvement can make participation more respectful and the resulting evidence more defensible.

1. Thinking machines are now research subjects

Some people have The Lord of the Rings. I have Dune. I remember staying awake until four in the morning to finish the original novel. I loved the world, the films, and the miserable beauty of Arrakis.

In Dune's backstory, humanity created “Thinking machines”, suffered the consequences, and outlawed them after the Butlerian Jihad. Mentats, human beings trained as computational specialists, were developed to perform knowledge work that could no longer be entrusted to machines.

Thinking machines are real now. There will be negative consequences. Nothing is gained by pretending AI will not influence almost everything. I am interested in the opportunity to make something positive with these tools without pretending the potential for harm is not there. My practical question is: how can we benefit from them, and in what contexts can we depend on them?

2. Synthetic systems are not synthetic respondents

I was thinking of how to best describe what I do in plain language: I survey thinking machines. It reframes the work for me. Begin with the system, ask it structured questions, vary the conditions and learn whether its patterns hold.

Synthetic systems research is a term I use because it distinguishes this work from the synthetic-respondent research now attracting attention. Synthetic systems research treats the model or system itself as the thing being studied. Its basic claim is modest: this system produced this pattern under these conditions. Synthetic-respondent research makes the stronger claim that generated answers can stand in for what people, customers or a population would say.

The distinction matters. A model can generate a plausible answer for a fictional Gen Z consumer in Melbourne. That answer may contain useful regularities from human language and behaviour. It does not contain that person's experience, obligations, private knowledge, or consequences. A generated backstory can become a persona, a synthetic respondent, an audience, and eventually a “digital twin” before anyone has established what is being twinned.

3. Testing stopped me from selling a convincing result

A model can answer thousands of questions in a day without recruitment, incentives or fieldwork. The headline result, supporting evidence and visualisations can exist before you have even engaged with a client. Human research is expensive, slow and burdensome to both the researcher and the person participating.

Recently, over three weeks, I built a synthetic-research prototype that could run structured collections, repeat tasks, compare models and turn the output into an accessible plain-language report. Building it changed how I understood the benefits and limits of synthetic systems research. I intended it to generate hypotheses, not declare what customers thought.

This prototype was intended to make claims such as: this ranking or pattern is stable enough to be worth checking. Several collection and reporting components worked as intended. The implied claims did not. Its output looked convincing, but the weakness was not apparent until I tested it. I could show that the system had produced an answer. I could not rely on what that answer implied strongly enough to support the full claimed proposition.

Finding this limit was the consequence of doing the work. My background is in academic research, measurement, and applied data work. The danger would have been never finding it because the report looked finished. Once the central implication failed, I stopped preparing this new prototype as a product rather than make the evidence say more than it did.

4. AI changes what market research must study

Part of the attraction of synthetic research is that the survey methods market researchers have relied on are under increasing strain. There are more barriers to high-quality survey work than there used to be: falling participation (Kennedy & Hartig, 2019), lengthy questionnaires that reduce both participation and later response quality (Galesic & Bosnjak, 2009), and incentives that do not always align with the researcher's need for careful, valid answers.

Commercial panel members weigh the incentive, time and ease of completion when deciding whether to participate (Brosnan et al., 2021). Payment does not inherently produce bad data: research in a probability-based panel found little reason to worry that moderate incentives or incentive-motivated participation reduced response quality (Schwarz et al., 2022). The design problem is that payment is usually conditional on completion, while the quality we need is thoughtful participation.

In one web-survey experiment, the fastest lower-education respondents were most vulnerable to response-order effects on unipolar rating scales. The same study warned that speed is only a proxy for satisficing among particular respondents and items, not a blunt exclusion rule (Malhotra, 2008).

Researchers have built attention checks, timing thresholds and consistency tests around these problems. The defences are useful, but they are not perfect. A new problem now looms: AI agents create an attack surface because one person can use them to complete instruments repeatedly and at scale. Westwood's autonomous synthetic respondent passed 99.8% of 6,000 attention-check trials while producing coherent, persona-consistent answers and simulating ordinary interaction traces (Westwood, 2025). Verifying that a panel member is a person does not establish that the answers were produced carefully by that person.

The quality of the experience matters beyond one dataset. A confusing, repetitive or disrespectful survey teaches participants that surveys are not worth their attention. Poor instruments consume the trust on which better instruments also depend. Synthetic respondents look like a solution to this burden, but substituting generated answers does not repair the human claim. It changes what was measured.

Preference is often built, not retrieved

The familiar commercial story is that people hold stable preferences which careful questioning can retrieve. Consumer-research evidence has challenged that story for decades. People often construct a preference in response to the task, the available alternatives and the information in front of them (Bettman, Luce, & Payne, 1998). A brand tracker, conjoint task or attitude scale can therefore produce a consistent answer without recovering a durable object that existed before the interview.

This matters most when the real decision is information-heavy. A buyer comparing a car, mortgage or business platform encounters explanations, alternatives, constraints and recommendations before choosing. The research task is not only to ask what is hidden inside the customer. It is also to understand the environment in which a preference takes shape.

AI is becoming part of that environment

AI is entering purchasing journeys, especially where decisions require more comparison and guidance. It is too early to call it a replacement for search: an analysis of 973 e-commerce sites found that ChatGPT referrals accounted for less than 0.2% of traffic and underperformed most established channels. The same study found stronger results in complex product categories, where people need more information, comparison and guidance (Kaiser & Schulze, 2026). These information environments are becoming research objects in their own right.

When a model is part of the decision context, its category account becomes observable. We can study what it makes prominent, what changes with the prompt, what survives repetition and where systems disagree. That is a real market-research object. It is evidence about the information environment, not a shortcut to declaring what customers think.

5. Synthetic results are promising but bounded

The cleanest empirical questions concern the system itself. What does this model produce when asked this question under these conditions? Which patterns survive a change in wording, order, model or time? Those questions are real and useful. They do not require us to pretend the generated speakers are people.

The model's fluency can obscure the distinction. The material looks credible because producing credible-looking material is the task. Unless we actively search for vulnerabilities, a confident answer can pass through the workflow without anyone seeing how much it depends on the context we supplied.

There are credible positive results. Argyle and colleagues found that a language model conditioned on real survey backstories could reproduce nuanced patterns across several US human samples (Argyle et al., 2023). Park and colleagues built agents for 1,052 people using extensive self-report data. Interview-only agents reached 83% of the participants' own two-week consistency benchmark. Agents combining interviews and surveys reached 86% (Park et al., 2026).

These were not strangers invented from a handful of demographic and behavioural descriptors. The stronger simulations began with extensive data from the actual individuals being modelled. Data from real people was the foundation of the method.

Models can also predict some experimental results. Cui and colleagues attempted 156 replications from psychology and management. Models recovered 73% to 81% of main effects, although interactions were less reliable and effect sizes were consistently larger than in human studies (Cui et al., 2025). Ashokkumar and colleagues reported strong correlations between model predictions and 469 effects from 70 preregistered survey experiments. Their predictions still systematically overestimated effect sizes and performed less strongly in a second archive (Ashokkumar et al., 2026).

This supports pilot testing, hypothesis generation and deciding which interventions are worth testing with people. It does not establish that an arbitrary synthetic sample represents a population.

Average resemblance is not enough

Bisbee and colleagues generated more than 3.6 million model responses matched to 7,530 participants in the American National Election Study. Some overall averages were close. The synthetic responses had less variation and frequently produced different relationships between variables. Among coefficients that differed significantly from their human counterparts, nearly a third changed sign (Bisbee et al., 2024).

For me, a third of the materially different relationships changing direction is not an acceptable discrepancy. A reader can decide what level of error they will tolerate. I would not use that result to describe a market structure.

A category average is rarely the whole commercial decision. We care about who differs, what predicts choice, where an effect reverses and how uncertain the estimate is. A plausible mean can conceal a false market structure. Claims about demographic or geographic representativeness remain hypotheses until they are tested against the people they claim to represent. If we do not look, we do not know.

The prompt helps make the person

The prompt does not merely describe the synthetic person. It helps create the answer. That makes persona construction part of the instrument, not an administrative preface. Wang, Morgenstern and Dickerson found that models used as human replacements could misportray identity groups, flatten differences within them and turn identity prompts into essentialised representations (Wang et al., 2025). Lutz and colleagues found that portrayals varied with role format, demographic priming and model (Lutz et al., 2025).

Wording, position, model, date and persona construction belong in the method, not behind the curtain. Pezeshkpour and Hruschka found large performance gaps when the same answer options were reordered (Pezeshkpour & Hruschka, 2024). Zhuo and colleagues found prompt sensitivity across models and tasks (Zhuo et al., 2024). Bisbee and colleagues found changes after small wording variations and across a three-month collection period (Bisbee et al., 2024). If these conditions are not tested and recorded, we do not know which part of the result belongs to the question and which part belongs to its presentation.

6. Claims must match methods

Synthetic research will become cheap enough for almost anyone to feed in a brief and produce a polished result. The important question is what quality control sits between the ingredients and the thing handed to a client. Familiar research disciplines already provide much of what this new instrument needs.

My own testing was exploratory: a fast attempt to find failure, not a formal validation study. It showed how much a polished final dataset can hide. Testing has to be designed into the work before the output starts looking authoritative. I advocate the following behaviours as a practical standard for synthetic research:

• Define the claim, or label the exploration. “The model ranks these brands” and “customers rank these brands” are different studies.
• Preserve the instrument and metadata. Keep prompts, examples, model identifiers, dates, parameters, transformations, exclusions and costs. Cost is not evidence, but it matters when deciding whether a method is usable.
• Test the influence of wording and position. Use semantically equivalent prompts and rotate answer choices, labels and evidence.
• Repeat across relevant conditions. One answer is not a measurement. Retest across fresh runs, model families, versions and time where stability matters, then report the degree of agreement. A more expensive model may provide a useful second opinion, but price is not a quality guarantee.
• Use controls and leakage checks. Include invented alternatives, blinded labels and comparisons where no difference should exist. Check whether the prompt already contains the conclusion.
• Track failure. Record failed responses, retries and the tasks on which they cluster. Retries can make a final table look complete while hiding how often the instrument failed along the way.
• Inspect the structure of response values, not only the mean. Compare variance, distributions, subgroup differences, interactions and sign. Tidy data can still support the wrong substantive ranking.
• Validate externally at the level of use. If the claim concerns people, purchases or markets, test it against independent human-research or behavioural data.

Synthetic methods can expose assumptions, pretest instruments, search for failure cases and help decide where scarce human research should be spent. They need extensive testing, particularly when a method is extended beyond the people, domain or conditions in which it was evaluated.

The minimum standards also have to be operationalised. Saying that a method is careful, respectful, transparent, or evidence-led does not maintain those qualities. The system has to turn each value into an observable behaviour, a threshold, an owner, evidence, a repair path, and a test at the point where the claim reaches a reader. For synthetic research, that means enforced claim boundaries, recorded provenance, visible caveats, refusal states, and checks that can stop an unsupported conclusion rather than merely warn about it in a methods note.

With human grounding and statistical correction, synthetic methods may also extend limited datasets. Krsteski and colleagues found that synthetic generation alone produced substantial bias, while allocating human observations to rectification reduced that bias below 5% in their settings. Performance varied substantially across questions and datasets: a method that helped on one item could perform poorly elsewhere. That reinforces the need to test the method in the domain where it will be used (Krsteski et al., 2026).

Use these systems for what they can demonstrate. Test the boundaries that matter. If the bridge fails, narrow the claim before the work reaches a decision.

A claim ladder for synthetic evidence

1. System claim: this model produced this pattern under these recorded conditions.
2. Robust system claim: the pattern survived declared wording, order, repeat, model and time contrasts.
3. Externally grounded claim: the hypothesis was tested against independent human-research or behavioural data from the population, construct and setting it claims to describe.
4. Decision claim: the validated relationship is current, precise and relevant enough to guide this decision.

Repeated generation can strengthen the first claim. Cross-model agreement can help with the second. Neither automatically reaches the third. A thousand samples from one underlying system do not become a thousand independently observed lives.

A blended approach requires an explicit boundary. Before collecting anything, decide which claims belong to the system, which concern people, what evidence each method can provide, and what result would force the claim to narrow. Otherwise, a polished synthesis can quietly give one source authority it never earned.

7. Hypotheses must leave the machine

Synthetic systems research can help us craft better hypotheses. But a hypothesis has to leave the machine. Putting it into another model may produce a better hypothesis, or simply hypotheses about hypotheses. If the claim concerns people, we need to ask real people real questions.

No single research signal contains the whole truth. Surveys see declared answers. Conversation exposes language, context and contradiction. Behaviour shows what happened under actual constraints. A model shows what that system tends to produce from the information and framing available to it.

These signals do not become interchangeable because they point in the same direction. In psychometrics, reliability concerns the consistency and precision of scores across relevant replications of a measurement procedure. Test-retest repetition is one form, not the whole concept (American Educational Research Association et al., 2014). Validity asks whether the evidence supports the interpretation and use we intend. A model can be highly reliable and still systematically wrong about the world.

Hypotheses generated by synthetic results need a plan for what comes next. Triangulation is one way to sharpen and test them by taking more than one bearing. Use people, behaviour or a genuinely different method. If the signals disagree, investigate the disagreement instead of averaging it away. The synthesis becomes the actionable result, not whichever source produced the cleanest chart.

Organisations will become accustomed to generating possibilities in hours. They will not want to wait weeks for a long conventional survey to validate every one. We need faster and more respectful ways to collect human research data: short quantitative decisions, better qualitative elicitation, voice and contextual methods, and hybrid designs that ask people only for the information the machine cannot credibly supply.

The next challenge is to make human research faster, more respectful, and worth the time people give it. That is the subject of the companion article.