Aligning machines with human values requires an understanding of what those values are. It can, however, be argued that neither humanity nor human values can be known or modelled in a complete way.
Foucault believes that knowledge or knowledge of anything is always historically situated and shaped by unseen power structures. Every culture at any moment has its own “system of knowledge or understanding” (episteme). And so alignment of an AGI depends on the epistemic norms of those who design it. Apart from the usual power dynamics dictating the values of the “designer” as human values, such a machine will inevitably be incapable of stepping outside its own norms, and understand the plurality of human values.
The non-articulate, irreducible and tacit nature of human knowledge has been pointed out also by M. Polanyi (The tacit dimension). Indeed, it is argued that many aspects of human values are hidden in context, experience and intuition and that such tacit meanings cannot be codified without losing essential meaning.
But even if it were possible to formalize and centralize all human knowledge and values in one place, thereby constructing a perfectly aligned AGI, F. A. Hayek(The knowledge problem) warns of the dangerous consequences and risks of overconfidence in such systems.