Synthetic systems research

If the AI-mediated layer is where belief about a category now forms, it needs a methodology of its own. There is no underlying corpus to obtain. There is only the output produced in response to a prompt.

Survey the model

The move is to survey the model. Design prompt instruments with the same care as customer surveys, systematically vary context across demographic and segmentation frames, and run repeated tests at scale. The aim is to observe how a model positions a category across conditions. Each category is viewed through three lenses: the jobs customers hire it for, the capabilities it delivers, and the perceptions it generates. The result is a structured view of how the model represents the category, and therefore a view of the information environment shaping decisions made through it.

The discipline: consensus across many readings

A single prompt is one draw, not an answer. Ask again, or vary the framing slightly, and the result can change. The method therefore relies on repetition: the same question asked many times under systematically varied conditions, then reduced to what remains stable across runs. This separates signal from prompt-specific artefact. What survives is consensus: the part of the representation that holds across variation. It is a reliability and validity discipline, applied to a new instrument.

A name for it

I call this synthetic systems research, because the field needs a name and existing terms are doing other work. It refers to the systematic study of synthetic information systems such as large language models, generative search tools, and recommendation engines, understood as information environments that shape customer decisions.

It is distinct from synthetic respondent research, or digital twin approaches. Both use related technology and overlap in method, but they serve different purposes. Synthetic systems research studies the model as an object in the world that people consult when deciding. Synthetic respondent research uses the model as a proxy for the people making those decisions.

That distinction matters more now that synthetic respondents have become a funded category, with digital twins and synthetic audiences entering mainstream use. The recurring criticism is methodological: without rigorous validation, these systems risk laundering model bias into confident-looking outputs.

Synthetic systems research is the proactive approach, while digital twin work is defensive. It does not claim to simulate customers. It studies the systems customers are increasingly consulting before they decide, producing signal grounded in a real and observable object: the current state of an information layer that is already in use.

This is not yet a clearly defined commercial territory. It is emerging, and the window to define it is still open.