Even if there were a set of universally agreed-upon human values, a perfectly aligned AGI poses a significant risk to the democratic process. In a democracy, authority is a matter of participation, contestation and revisability. If the authority given to a perfectly aligned machine falls outside the realm of this democratic process in that it may be considered as one “that knows best” i.e. is beyond doubt removes itself from a democratic process, turning governance into optimization, while treating dissent as error.
Habermas emphasizes deliberation as a fundamental part of a democratic process. He underscores that the legitimacy of political decision springs from inclusive and evidence based public reasoning. A perfectly aligned machine that delivers a decision bypassing the public process/debate is a technocratic agent that undermines the democratic processes, thus will be misaligned with the collective decision making process.
Similarly, the legitimacy of a perfectly aligned machine can be brought into question as, at least in John Rawls’ view. In his “Liberal principles of legitimacy”, Rawls argues that legitimacy of political power must be grounded in a constitution the essentials of which are endorsed by citizens based on their common human reason. In this sense, a perfectly aligned machine will lack legitimacy unless it is grounded in procedures and public justifications. This is a problem as the decision of an even perfectly aligned AGI will be a matter of optimization and algorithm. Algorithmic alignment may be well beyond the understanding of the public and so it will necessarily be outside a democratic process and as such lack legitimacy. It will have a coercive power based on technical specification and beyond public reason, rendering it illegitimate.