When is Automated Decision Making Legitimate?

A book chapter summary by Esha Srivastava


  1. Legitimacy vs Fairness (Distributive Justice):

We know that sometimes automated decisions may feel unjust and biased. To address such concerns, the legitimacy of an automated system must be questioned-whether it is fair and just. Legitimacy, in this context, means having a fair system. The importance and establishment of legitimacy is recognized by several institutions; hence, integrating a set of rules that align with social values is very important. For instance, a private firm might overwork employees if not regulated by rules aligned with work ethics. People trust systems because they believe decisions will be fair and just. Therefore, it is important to understand whether it is morally justifiable to use machines or automated methods in particular scenarios.

  1. Machine learning is not a replacement for human decision making

Of course, there is no doubt that the human ability to make decisions cannot be replaced easily, let alone giving machines the sole power to make decisions. Human decisions are often case-based and include moral and ethical reasoning. However, they may also reflect biases and stereotypes. According to the Strawman view: “decisions based on machine learning models might be difficult for people to understand; humans are black boxes too. And while there can be systematic bias in machine learning, they are often demonstrably less biased than humans.” Machine learning can help reduce such biases and unfair judgments that might be overlooked by humans.

  1. Bureaucracy and Arbitrary decision making 

While the term bureaucracy often carries a negative connotation, it is important to recognize that bureaucratic systems have historically played a crucial role in safeguarding against problematic forms of decision-making, particularly arbitrary decision-making. At times, bureaucracy has helped ensure that decisions remain fair, just, and transparent by adhering to formal procedures and structured processes. Arbitrariness is especially concerning because it undermines evidence-based and justifiable decision-making. Scholars Kathleen Creel and Deborah Hellman distinguish between two types of arbitrariness. The first, procedural arbitrariness, refers to decisions made inconsistently or on an ad hoc basis, emphasizing a lack of procedural regularity. The second, substantive arbitrariness, involves decisions that may be consistent in application but lack underlying reasoning or justification. This second type raises critical questions about the rationale and legitimacy of the decision-making framework itself.

Table 1: Types of Arbitrariness

  1. Three forms of Automation

Automation might undermine important procedural protections in bureaucratic decision-making. So, the question becomes: what exactly does machine learning help automate? There are three types of automation. The first (Automating Pre-Existing Rules) involves applying rules set by humans, through a traditional policy-making process, into software that automates their application in particular cases. Machine learning has no direct role here. The second (Automating Informal Human Judgments) type of automation uses machine learning to replicate the informal judgments of humans. For example, grading a student’s creative writing essay through automated assessment software that tries to emulate human subjectivity. The third (Learning New Decision Rules from Data) form of automation does not rely on bureaucratic rules or human judgment but instead learns decision-making rules from data—detecting patterns and formulating rules. Machine learning and statistical techniques are central to this type of automation. For instance, identifying areas for police patrol based on historical crime data.

Table 2: Three different forms of automation


  1. Mismatch between target and goal

When individuals make decisions, they typically align their targets with their goals. However, decision-makers, whether human or algorithmic, often lack clearly defined goals, or their goals are multifaceted and complex. In many cases, their goals and targets do not align. This misalignment makes it essential to measure the true outcomes of interest and to thoughtfully consider alternatives.

For instance, predicting the occurrence of future crimes is not the same as reducing crime. Accurate predictions may lead to more arrests, but not necessarily to crime prevention. If the actual goal is to reduce crime, then relying on crime rates as prediction targets might be ineffective. To manage complexity, decision-makers often resort to using proxy targets—such as arrest data as a stand-in for actual crime. However, proxies can diverge from the true intent of a decision and, in some cases, actively work against it. This makes aligning goals and targets critically important. Choosing a prediction target is not merely a technical decision, it is a normative and ethical one. If fairness is a concern, both the proxy used and the underlying goal must be examined critically.

  1. Failure to consider relevant information 

Another significant limitation in automated decision-making lies in the failure to consider all relevant information. Both bureaucratic systems and machine learning models frequently rely on coarse groupings derived from statistical generalizations based on historical data. While such generalizations may be accurate on average, they can be unfair or misleading in individual cases. This issue is exacerbated in automated systems, which often do not allow individuals, especially those from underrepresented groups with limited data representation, to contribute additional, contextually relevant information. This lack of nuance raises serious concerns about fairness and accountability, particularly when efficiency is prioritized over individual circumstances and realities.

  1. Limits of Induction 

One major issue in machine learning is overfitting—a concept familiar to anyone working with data. Overfitting represents a form of arbitrary decision-making, where predictive validity becomes an illusion, often emerging from coincidental or spurious patterns. For example, consider a coach who selects runners based on the color of their sneakers. While this might show success within a particular dataset, the pattern is clearly arbitrary and lacks real-world justification.

To mitigate overfitting, practitioners often turn to simpler models or use techniques like train-test splits. However, these approaches do not address distributional shifts—situations where the real-world context differs significantly from the training data. Moreover, many machine learning models rely on correlation rather than causation, which introduces significant ethical concerns. As Frank Pasquale warns, models that lack causal grounding risk producing unfair or irrational outcomes. In such cases, accuracy alone is not sufficient. What matters is the reasoning behind the predictions—whether they are justifiable, fair, and grounded in meaningful logic.

  1. Right to accurate predictions?

Even if we assume a best-case scenario—no overfitting, sufficient data, and minimal distributional shifts—there remains a crucial question: How accurate is “accurate enough” when high-stakes decisions are at play? Low accuracy can compromise the multifaceted objectives that decision-makers pursue. In this light, legitimacy is not solely about accuracy; it also concerns whether the model aligns with moral and ethical standards. We must ask: Is this level of accuracy enough to justify serious outcomes? Do these models reflect the complexity of real-world goals, or are they merely optimizing for simplified metrics?

This is why legitimacy in machine learning extends beyond performance metrics. It requires attention to fairness, ethical considerations, and alignment with broader purposes—ensuring that automated decisions are not only efficient but also just and meaningful.

  1. Agency , recourse and culpability

Finally, it is crucial to consider whether the decision is ethically and morally right and beneficial to people and society. For example, advising someone to move neighborhoods to qualify for a loan may not be feasible. At the heart of this issue is moral responsibility—people shouldn’t be penalized for outcomes beyond their control, such as the death of a family member. Conversely, if decision criteria are too easily manipulated, they risk being gamed—like teaching to the test. Even sincere attempts at improvement can lead people to focus on superficial indicators, rather than real merit. A fair system must not just optimize predictions—it must uphold values of agency, recourse, and ethical responsibility, judging people based on criteria they can meaningfully influence.


  1. Conclusion 

To conclude, it is important to understand the gravity of legitimacy in decision-making using automated systems or machine learning. These systems may carry biases and stereotypes that humans hold. It is essential to provide individuals the right to question decisions—this is recourse—so they receive justified and rational answers. Authorities and decision-makers must make decisions that are morally just and aligned with ethical values, not penalizing individuals for circumstances beyond their control. In my opinion, every dataset represents real human lives, and any decisions derived from this data can have a profound impact on individuals. That's why it is essential to minimize biases and stereotypes by using machine learning models that are specifically designed to detect and address these issues wherever possible.


Reference: 

Barocas, S., Hardt, M., & Narayanan, A. (2023). When is automated decision making legitimate? In Fairness and machine learning: Limitations and opportunities (pp. 23–47). MIT Press. https://fairmlbook.org


Comments

Post a Comment

Popular posts from this blog

Modelling genomic language using NLP and LLMs

Image
Author : Navya Tyagi  Genomic data consists of DNA, RNA, and protein sequences that can be represented as strings of unstructured text. These sequences can be very large in size. For instance, human DNA is made of 3 billion A,G,C, and T letters. There are hidden patterns that can be considered equivalent to "words" in a natural language. But all of these words are not known and more importantly the grammar that genomic lanaguege follow is not well understood. These biological words with critical functions are of interest to study disease and development processes. Sometime a mutation in these "words" may result in a disease condition. Determining these "words" with biological function is a computational challenge. Figure 1: a) Code of DNA can be written using letters A,C,G,T. A pairs wiht T and G pairs with C making a double stnraded (helical) structure out of two DNA strands. b) message from DNA is transcribed to RNA (also known as messenger RNA). The ...
Read more

Machine Learning: Interpretability vs Explainability

Image
 Authors: Navya Tyagi & Esha Srivastava, Tyagi Lab, RMIT University, Melbourne, Australia   Introduction   We are aware that machine learning models are built upon vast amounts of pre-existing context and are constantly learning. The advent of machine learning is meant to assist humans in making complex decisions. However, there are still many misconceptions and uncertainties about how much trust one should place in machine learning when it comes to decision-making—especially when these decisions could impact an individual’s life. But what if there was a way to at least understand or backtrack how machine learning makes decisions? Hence, in this blog, we are going to discuss interpretability and explainability. We often come across these terms and may use them interchangeably. However, it is important to understand the distinction between them. Interpretability refers to converting the implicit information within a neural network into human-interpretable insights. Th...
Read more