Why is everyone obsessed with judgement and taste when talking about AI?

I've noticed there seems to be an obsession at the moment with talking about how critical judgement and taste are when discussing the transformational power of AI. This often seems to be in an almost conciliatory way, "well, we will be getting rid of 40% of our workforce, but those lucky people that remain will be really important to bring judgement and taste".

Jack Dorsey mentions it in this video:

if I had to say one thing that myself or anyone in the company has to do, I guess it’s this overused phrase of “judgment,” but it is judgment against what we intend to build in the world. And is it aligned with that judgment? Is it aligned with the values? Is it aligned with the taste that we have? And is it unique or is it not?

Andrej Karpathy also refers to taste and judgement (and aesthetics and oversight) in this video

But what does it mean, in this new age of AI for a human to be an overseer of taste and judgement? What even is good judgement?

Ok, so AI disrupts the status quo, and the role of the remaining humans in an organisation shift to providing this magical human-only element?

This feels a bit wrong, for the following reasons.

The mediocrity problem

Most people have really bad taste.

There are psychological drivers for this. The Mere Exposure effect, Cognitive Load, Social Acceptance and Normality, and Homeostasis. So, the likely conslusion of soon to be achieved future state that Karpathy, Dorsey and others bang on about is a scenario where AI does most of the grunt work, humans are the ultimate arbiters of "taste" and "judgement", and for the majority of the time that judgement is going to be poor.

But thats not what they mean. There is a really fundamental assumption underpinning the Silicon Valley statements around taste and judgement, and that is that they are able to pay a premium so that the humans they have in their company have the best taste and the best judgement that money can buy. But that's not the reality for the majority or organisations.

If the world's taste and judgement distribution sits on a bell curve, what hope do the majority of companies have that will be within 1 standard deviation? We get mediocrity. In truth maybe thats no different from today. It's unrealistic to expect everyone's taste and judgement to be exceptional, it's unrealistic to expect every company to be exceptional.

And maybe thats what we deserve? isn't the current era of AI just text-prediction fed on an internet-scale diet that will regress towards mean?

Also...

What even is good judgement?

AI told me 🙃 that the components of good judgement are:

Pattern Recognition: Using past knowledge and experience to recognise underlying dynamics in new situations

Active listening and openness: seeking out diverse perspectives and challenging your own assumptions before deciding

Bias awareness: Recognising personal blind spots, ego, or emotional triggers, and taking steps to neutralise them

Systems thinking: synthesizing multiple pieces of incomplete or weak evidence into a coherent, balanced view

Discernment: The ability to see things exactly as they are and prioritise what is truly important over superficial details

Here's the thing

What really is the gap between a human with mediocre judgement and an adequately fine tuned and context-aware AI, across the 5 areas listed above?

Does the future hold a new tiering of organisational effectiveness? where exceptional companies with exceptional budgets retain human overseers with exceptional taste and judgement? and the majority of companies, the centre of the bell curve get by with near-zero human judgement, because the gap between mediocrity and good enough AI-judgement is too close to call?