Knowing what we know
Thinking about technology brought me to an old an interesting question: what do we know when we know something? This is an entire philosophical field on its own but I’ll try and limit it here to the field of research and technology.
Qualitative research
I worked in qualitative research for more than a decade, starting my career as an ethnographer, moving on to more structured qualitative research (groups and one on ones) before moving into the now popular field of cultural insight. What struck me, especially from the point of view of a cultural researcher, is how we take people at their word when they say something.
I’m not implying that consumers lie. But I am aware that when consumers respond, you’re not really working with a ‘consumer responding’, instead you’re participating in a human conversation about a topic that you suggested. This means that the whole conversation happens in a social context with all the posturing and covering of a normal interaction. People are agreeable or disagreeable. Some people are proud, others are self reflective, or subservient - the list goes on.
Early in my career I often heard the phrase ‘but the consumer said so’ often using a soundbite, and taking the literal meaning to justify a decision. This of course treats people as access points into a universal database of the segment. Which they are not.
As the understanding of insight matured, the phrase ‘people can only tell you what the think they think’ emerged. And true as this is, it still ignores the context. A more accurate phrase would be something like ‘people can only tell you what they think they should be telling you given the context’.
What can we know from qualitative research?
The first and often most important thing we need to keep in mind is what people can only talk of things that they are aware of. This might sound simplistic but if someone tells you of a product, an event or a service we can know that this lives in their head somewhere. It might be in error, but more often than not, these very real facts are mostly true.
They can tell us how they want to feel or how they believe they feel about it - which once again is more often than not a relatively accurate reflection. Especially when upwards of 30 people tell you the same sort of thing.
When we start asking why they feel this way or why they behaved a certain way we have to be more careful. When properly interrogated, a conversation can feel like those deep cave exploration where a tricky section in a conversation suddenly opens up into some sort of magical underground cathedral. These are the sort of conversations where, after it concluded, respondents ask if we can do this again - it’s a sort of therapy, a journey of self discovery.
Another area that should be left for experienced researchers is understanding the flow of the conversation for insight. If a conversation of breakfast cereal constantly ends up in the space of the purchasing moment and packaging, if the consumer shows high price awarenes and if constant and subtle reference is made to luxury brands (referring to their car as ‘the BMW’ and for instance), you might find something interesting down the avenue of aspiration and what ‘providing for your family’ looks like. Someone who understands the ingredients on the box and who happens to have a set of bamboo toothbrushes on the edge of the counter might require a different route.
Cultural insight
This idea of ‘people can only tell you what they think they think’ became exceptionally clear when working in cultural insight. The field of cultural insight ranges from trend spotters all the way to theoretical thinkers with most practitioners spanning the entire spectrum.
Qualitative research can reveal someone’s favourite superhero and give you a rationale (‘I like superman because he is strong and humble’). Cultural insight can tell you why this makes sense in a modern world and why those same people might, in a few years, prefer Batman or even Dead Pool.
This shift and the prediction of change is based on a combination of current and projected shifts, how those shifts connect to the existing web of values and behaviours and how those in turn tie back into the framework of what a culture is and what it takes to remain stable.
In our everyday research we will therefore look at changes in fashion, shifts in consumer goods, changes in perception of politics and economics, legislative changes, polling results, the topics in public petitions and more.
What do we know when we’ve gathered our input?
It’s tempting to gather all of the social and cultural input you can find and say this is either good or bad. As a professional, you cannot really tell good from bad and it’s not your job to do that. You might want to repackage your findings in the context of your audience to reflect a stance, but at analysis stage, you’re just looking for what is.
Let’s imagine a subculture - a fictional version of Klunking (the original mountain biking culture). A nice old one can hopefully detach us from subjective feelings. The original mountain biking culture consisted of local cyclists who modified city bikes for mountain use. Being able to modify a bike and then being of such an inclination to take your own creation down a mountain was a barrier to entry and shaped the culture.
But as mountain bikes became a mass produced and spare parts easy to find this barrier to entry faded. You still needed to be daring, but you could now find a bike without having to make one. This introduced a broader range of people, manufacturing now made economic sense, bikes become robust and stopped ‘clunking’ down the hill and slowly the concept, the sport and later the culture of mountain biking emerged. A culture that is vastly different from the Klunkers of old.
If you simply looked at the mass production of the mountain bike in the context of Klunking you don’t really ‘know’ anything cultural. It’s only when you combine this with the cultural mechanic of group access that it starts making sense and we can start making predictions about social dynamics.
In the world of cultural insight, observable facts only become useful knowledge in the context of cultural theory, established patterns and stable mechanisms.
Computer generate data
There are many types of data we can think about, observational, desk research and quant data along with the endless combination of source data and analysis methodologies. I’m not going to go through all of these but they are all worth thinking about when we want to build something on findings we derive from it. I’d like to think about more about something new - AI generated findings.
It just calculates the next word
I’m not a fan of that rhetorical device where you add the word ‘just’ in front of something amazing in order to undermine it. GPT does not ‘just’ predict the next word. It is not ‘just’ math. It is an mathematical computation that generates seemingly intelligent output. There’s nothing ‘just’ about it. It’s a machine that delivers something that could pass for actual knowledge.
What do we really know when an LLM gives us something? GPT 1 and 2 were in some ways seen as a nice idea but not a serious thing. It managed to predict a next token on small models (117M parameters for GPT 1). Concerns about scaling, training data and its fundamental ability to generate anything meant that the tech and AI industry remained sceptical.
But when essentially the same logic started generating very impressive results around 2020, the mood changed. This is for very good reason, GPT 3 starting delivering really impressive outputs and it would seem that scale actually meant something.
What do we have to understand
In the context of knowing however, it is important to understand what we’re looking at when we look at an LLM output. A person can tell us what they think they think. This means we have to understand the limitations of reporting on our subjective experience of the world. Cultural research and give as the surface level indicators that plays out on a theoretical framework. This means we have to understand how surface level shifts tie to cultural reality. We can tie our findings back to the real world in a way that we understand.
When it comes to generative AI, do we know what we have to understand, how do we think about data that is generated by stacking one most plausible token after the next? When the model itself holds no actual data or facts, other than weights and vectors and the ability to generate plausible tokens.
When we try to understand a multimedia ethnographic report, we don’t have to understand how cameras work (beyond operating it). But we do have to understand people and we have to understand how editing is done. It helps us contextualise the output. We know which complex components we should focus on. But with generative AI content, it’s not that clear.
Process mimics intent
When GPT5 does a web search, it feels as if it knows that it does not know. You would be excused for thinking there is a sense of self awareness wrapped around the knowledge. But a simple as a word search can start to create this illusion (if the prompt contains ‘today’s weather’ it won’t be in the training data). This is almost charming in its simplicity.
What is interesting is that the LLM itself can be queried on its knowledge. And it can answer, not because it knows what it knows, but because when you prompt it properly, the next most plausible series of tokens will result in either a yes or a no. To make this a bit more intuitive think about it like this. If all the training data on an event in WW2 was done on people saying ‘I don’t know about this event’ or ‘I really need to do more reading on this event’, we end up with a model where the embeddings for that event would overlap heavily with embeddings for uncertainty language.
‘Knowledge’ from an LLM’s point of view is therefore overlapping embeddings. Strong associations. The reason why it ‘knows’ obscure facts is down to scale. With enough training data, it can generate patterns between tokens that when interpreted by a human becomes knowledge. A low temperature sharpens the probability distribution, which strongly biases toward the top choice (Paris is the capital of… France). But when the temperature goes up, the curve flattens and the model can select from lower probability options. Paris can then become the capital of fashion or art.
What is important to note here is that those claims are not wrong - the sequence of tokens did not generate inaccurate real world knowledge (like Paris is the capital of Germany) or random words (like Paris is the capital of marble ascent the puddle). In fact, in a certain context according to the right person, Paris can be the capital of fashion. It not only makes linguistic sense, it makes actual sense. And while the LLM seems to understand the nuance of being a capital both in the geopolitical sense and industry leadership sense, it has no ability to store or understand the actual nuance, it’s simply inherited from training data.
Math vs magic
AI feels human. This is arguably one of its strongest and most convincing characteristics. It speaks with the confidence of a true expert (or a con artist). It’s hard to tell the difference. And it does so at speed with a good command of most languages.
Being labeled a polymath is certainly a high achievement - someone with substantial knowledge or expertise in multiple, often unrelated, fields, and the ability to integrate and apply that knowledge across disciplines. These people are no doubt well informed but because their knowledge is hard to verify and because they come across as intelligent, we often defer to them or take what they say at face value. Especially in a social context where it’s safe to do so. It’s also worth noting that people who accumulate facts or knowledge tend to respect it and attempt to be truthful.
AI can easily simulate this phenomenon and as models improve, they don’t just sound like they know stuff, their output can actually be reliably interpreted as accurate knowledge.
A lot still has to be uncovered about AI and a lot will change not only in the technology but also how we use it. The story goes that when motion camera was first invented, the first movies were simply recordings of stage plays. The immediate benefit was recording and viewing at a different time and location. It took a while before they realised that the camera introduced a whole new way of story telling. This will hold true for many technologies and no doubt for AI.
It is important however that in our process, we do not become over reliant on it, especially in areas where it can simply not outperform a person. And for now, the best we can do is to understand the technology in as much as we’re able to, keep tracing our data back to the real world, and maintain a common sense scepticism about we can expect from it.
