Highlighted Selections from:

The Scored Society: Due Process for Automated Predictions


Citron, Danielle Keats and Pasquale, Frank A., The Scored Society: Due Process for Automated Predictions (2014). Washington Law Review, Vol. 89, 2014, p. 101-; U of Maryland Legal Studies Research Paper No. 2014-8. Available at SSRN: http://ssrn.com/abstract=2376209

p.2: Big Data is increasingly mined to rank and rate individuals. Predictive algorithms assess individuals as good credit risks, desirable employees, reliable tenants, and valuable customers. People’s crucial life opportunities are on the line, including their ability to obtain loans, work, housing, and insurance. As pervasive and consenquential as automated scoring is, so is its secrecy and lack of oversight. Procedural regularity is essential for those stigmatized by “artificially intelligent” systems. Lessons from our due process tradition can provide basic safeguards for our scoring society. Regulators should be able to test scoring systems to ensure their fairness and accuracy. Individuals should be granted meaningful opportunities to challenge adverse decisions based on scores miscategorizing them. -- Highlighted apr 20, 2014

p.4: Consider these examples. Job candidates are ranked by what their online activities say about their creativity and leadership. Software engineers are assessed for their contributions to open source projects, with points awarded when others use their code. Individuals are assessed as “likely” to vote for a candidate based on their cable usage patterns. Recently released prisoners are scored for their likelihood of recidivism. -- Highlighted apr 20, 2014

p.5: Sometimes, individuals can score the scorers, so to speak. Landlords can report bad tenants to data brokers while tenants can check abusive landlords on sites like ApartmentRatings.com. On sites like Rate My Professor, students can score professors who can respond to critiques via video. In many online communities, commenters can in turn rank the interplay between the rated, the raters, and the raters of the rated, in an effort to make sense of it all (or at least award the most convincing or popular with points or “karma”). -- Highlighted apr 20, 2014

p.5: Although mutual scoring opportunities among formally equal subjects exist in some communities, the realm of management and business more often features powerful entities who turn individuals into ranked and rated objects, often to their detriment. -- Highlighted apr 20, 2014

p.6: Threat assessments result in arrests or the inability to fly even though they are based on erroneous information. Political activists are designated as “likely” to commit crimes. -- Highlighted apr 20, 2014

p.6: Advocates applaud the removal of human beings and their flaws from the assessment process. Automated systems are claimed to rate individuals all in the same way, thus averting discrimination. But this account is misleading. Human beings program predictive algorithms. Their biases and values are embedded into the software’s instructions, known as the source code, and predictive algorithms. Scoring systems also mine datasets containing inaccurate and biased information. There is nothing unbiased about scoring systems. -- Highlighted apr 20, 2014

p.6: Does everything work out in a wash because information is seen in its totality? We can’t rigorously test this theory because scoring systems are shrouded in secrecy. Although some scores like credit are available to the public, the scorers refuse to reveal the method and logic of their predictive systems. -- Highlighted apr 20, 2014

p.7: Just as concerns about scoring systems are heightened, their human element is diminishing. Although software engineers initially identify the correlations and inferences programmed into algorithms, Big Data promises to eliminate the human “middleman” at some point in the process. -- Highlighted apr 20, 2014

p.7: If scoring systems are to fulfill engineering goals and retain human values of fairness, we need to create backstops for human review.-- Highlighted apr 20, 2014

p.8: Algorithmic scoring should not proceed without expert oversight. This debate is already developing in the field of “killer robots,” where military theorists have described the following distinctions in terms of potentially autonomous, artificial-intelligence-driven weapons:

  • Human-in-the-Loop Weapons: Robots that can select targets and deliver force only with a human command;
  • Human-on-the-Loop Weapons: Robots that can select targets and deliver force under the oversight of a human operator who can override the robots’ actions; and
  • Human-out-of-the-Loop Weapons: Robots that are capable of selecting targets and delivering force without any human input or interaction.

-- Highlighted apr 20, 2014

p.8: Human rights advocates and computer scientists contend that “Human-out-of-the-Loop Weapons” systems violate international law because AI systems simply cannot adequately incorporate the rules of distinction (“which requires armed forces to distinguish between combatants and noncombatants”) and proportionality. They create a “responsibility gap” between commanders and killing machines. Such decisions arguably are the unique responsibility of persons using holistic, non-algorithmic judgment to oversee complex and difficult situations. -- Highlighted apr 20, 2014

p.8: Just as automated killing machines violate basic legal norms, stigmatizing scoring systems should be similarly suspect. We should not simply accept their predictions. Scoring systems are often assessed from an engineering perspective, as a calculative risk management technology making tough but ultimately technical rankings of populations as a whole. -- Highlighted apr 20, 2014

p.8: Although algorithmic predictions harm individuals’ life opportunities often in arbitrary and discriminatory ways, they remain secret. Human oversight is needed to police these problems. -- Highlighted apr 20, 2014

p.9: This Essay uses credit scoring as a case study to take a hard look at our scoring society more generally. Part II describes the development of credit scoring and explores its problems. Evidence suggests that what is supposed to be an objective aggregation and assessment of data—the credit score—is arbitrary and has a disparate impact on women and minorities. Critiques of credit scoring systems come back to the same problem: the secrecy of their workings and growing influence as a reputational metric. Scoring systems cannot be meaningfully checked because their technical building blocks are trade secrets. Part III argues that transparency of scoring systems is essential. It borrows from our due process tradition to introduce human values and oversight back into the picture. Scoring systems and the arbitrary and inaccurate outcomes they produce must be subject to expert review. -- Highlighted apr 20, 2014

p.11: Long before the financial crisis, critics have questioned the fairness of credit scoring systems. According to experts, the scores’ “black box” assessments were “inevitably subjective and value-laden,” yet seemingly “incontestable by the apparent simplicity of [a] single figure.” There are three basic problems with credit scoring systems: their opacity, arbitrary results, and disparate impact on women and minorities. -- Highlighted apr 20, 2014

p.12: An ambitious consumer could try to reverse-engineer credit scores, but such efforts would be expensive and unreliable; worse, they might be barred by contract. -- Highlighted apr 20, 2014

p.12: Responding to the confusion, books, articles, and websites offer advice on scoring systems. Amazon offers dozens of self-help books on the topic, each capitalizing on credit scoring’s simultaneously mystifying and meritocratic reputation. Hucksters abound in the cottage industry of do-it-yourself credit repair. -- Highlighted apr 20, 2014

p.13: In the Kafkaesque world of credit scoring, merely trying to figure out possible effects on one’s score can reduce it. -- Highlighted apr 20, 2014

p.14: The notion is that the more objective data at a lender’s disposal, the less likely a decision will be based on protected characteristics like race or gender. But far from eliminating existing discriminatory practices, credit-scoring algorithms are instead granting them an imprimatur, systematizing them in hidden ways. -- Highlighted apr 20, 2014

p.14: Credit scores are only as free from bias as the software and data behind them. -- Highlighted apr 20, 2014

p.15: Beyond biases embedded into code, some automated correlations and inferences may appear objective, but may in reality reflect bias. Algorithms may place a low score on occupations like migratory work or low paying service jobs. This correlation may have no discriminatory intent, but if a majority of those workers are racial minorities, such variables can unfairly impact consumers’ loan application decisions. -- Highlighted apr 20, 2014

p.15: Nonetheless, evidence suggests that credit scoring has a negative, disparate impact on traditionally disadvantaged groups. -- Highlighted apr 20, 2014

p.15: The National Fair Housing Alliance has criticized credit scores for disadvantaging women and minorities. -- Highlighted apr 20, 2014

p.19: Predictive scoring may be an established feature of the Information Age, but it should not continue without check. Meaningful accountability is essential for predictive systems that sort people into “wheat” and “chaff,” “employable” and “unemployable,” “poor candidates” and “hire away,” and “prime” and “subprime” borrowers. -- Highlighted apr 20, 2014

p.19: Procedural regularity is essential given the importance of predictive algorithms to people’s life opportunities—to borrow money, work, travel, obtain housing, get into college, and far more. Scores can become self- fulfilling prophecies, creating the financial distress they claim merely to indicate. -- Highlighted apr 20, 2014

p.19: When scoring systems have the potential to take a life of their own, contributing to or creating the situation they claim merely to predict, it becomes a normative matter, requiring moral justification and rationale. -- Highlighted apr 20, 2014

p.20: How should we accomplish accountability? Protections could draw insights from what one of us has called “technological due process”— procedures ensuring that predictive algorithms live up to some standard of review and revision to ensure their fairness and accuracy. -- Highlighted apr 20, 2014

p.20: This is not to suggest that due process guarantees are required. Due process only applies to state actors, which FICO and credit bureaus are not. Nonetheless, the underlying values of due process—transparency, accuracy, accountability, participation, and fairness—should animate the oversight of scoring systems given their profound impact on people’s lives. Scholars have built on the “technological due process” model to address private and public decision-making about individuals based on mining of Big Data. -- Highlighted apr 20, 2014

p.21: Individuals should have the right to inspect, correct, and dispute inaccurate data, and to know the sources (furnishers) of the data. Ironically, some data brokers now refuse to give out their data sources because of “confidentiality agreements” with the sources. That hubris (hiding behind privacy values in order to violate consumer privacy) would not stand for consumer reporting agencies covered by FCRA. And it should not stand for data brokers. -- Highlighted apr 20, 2014

p.21: Second, at the calculation of data stage, ideally such calculations would be public, and all processes (whether driven by AI or other computing) would be inspectable. In some cases, the trade secrets may merit protection, and only a dedicated, closed review should be available. But in general, we need to switch the presumption in situations like this: away from an assumption of secrecy, and toward a default expectation that people deserve to know how they are rated and ranked. -- Highlighted apr 20, 2014

p.21: We believe that, given the sensitivity of scoring, and the possibility of racially-inflected data to enter scores, scoring systems should be subject to licensing and to audit requirements when they enter critical settings like employment, insurance, and health care. Such licensing could be completed by private entities that are themselves licensed by the EEOC, OSHA, or the Department of Labor. This “licensing at one remove” has proven useful in the context of health information technology. The idea here is that with a technology as sensitive as scoring, fair, accurate, and replicable use of data is critical. We cannot rely on companies themselves to “self-regulate” toward this end—they are obligated merely to find the most efficient mode of processing, and not to vindicate other social values. Licensing can serve as a way of assuring that public values inform this technology. -- Highlighted apr 20, 2014

p.22: The FTC’s concerns about predictive algorithms have escalated with their increasing use. In March 2014, the FTC is hosting a panel of experts to discuss the private sector’s use of algorithmic scores to make decisions about individuals, including their credit risk with certain transactions, likelihood to take medication, and influence over others based on networked activities. The FTC has identified the following topics for discussion:

  • How are companies utilizing these predictive scores?
  • How accurate are these scores and the underlying data used to create them?
  • How can consumers benefit from the availability and use of these scores?
  • What are the privacy concerns surrounding the use of predictive scoring?
  • What consumer protections should be provided; for example, should consumers have access to these scores and the underlying data used to create them?
  • Should some of these scores be considered eligibility determinations that should be scrutinized under the Fair Credit Reporting Act?

-- Highlighted apr 20, 2014

p.23: FTC Chairwoman Edith Ramirez has voiced her concerns about algorithms that judge individuals “not because of what they’ve done, or what they will do in the future, but because inferences or correlations drawn by algorithms suggest they may behave in ways that make them poor credit or insurance risks, unsuitable candidates for employment or admission to schools or other institutions, or unlikely to carry out certain functions.” In her view, predictive correlations amount to “arbitrariness-by-algorithm” for mischaracterized consumers. -- Highlighted apr 20, 2014

p.23: Indeed. As Chairwoman Ramirez powerfully argues, decisions-by-algorithm require “transparency, meaningful oversight and procedures to remediate decisions that adversely affect individuals who have been wrongly categorized by correlation.” Companies must “ensure that by using big data algorithms they are not accidently classifying people based on categories that society has decided — by law or ethics — not to use, such as race, ethnic background, gender, and sexual orientation.” -- Highlighted apr 20, 2014

p.23: The FTC’s expert technologists could test scoring systems for bias, arbitrariness, and unfair mischaracterizations. To do so, they would need to view not only the datasets mined by scoring systems but also the source code and programmers’ notes describing the variables, correlations, and inferences embedded in the scoring systems’ algorithms. -- Highlighted apr 20, 2014

p.24: For the review to be meaningful in an era of great technological change, the FTC’s technical experts must be able to meaningfully assess systems whose predictions change pursuant to AI logic. They should detect patterns and correlations tied to classificaitons that are already suspect under American law, such as race, nationality, sexual orientation, and gender. Scoring systems should be run through testing suites that run expected and unexpected hypothetical scenarios designed by policy experts. Testing reflects the norm of proper software development, and would help detect both programmers’ bias and bias emerging from the AI system’s evolution. -- Highlighted apr 20, 2014

p.25: Ideally, the logics of predictive credit scoring systems should be open to public inspection as well. There is little evidence that the inability to keep such systems secret would diminish innovation. The lenders who rely on such systems want to avoid default---that in itself is enough to incentivize the maintenance and improvement of such systems. There is also not adequate evidence to give credence to “gaming” concerns—i.e., the fear that once the system is public, individuals will find ways to game it. While gaming is a real concern in online contexts, where, say, a search engine optimizer could concoct link farms to game Google or other ranking algorithms if they became public, the signals used in credit evaluation are far costlier to fabricate. -- Highlighted apr 20, 2014

p.26: “Technological due process” insists that automated systems include immutable audit trails to ensure that individuals receive notice of the basis of decisions against them. -- Highlighted apr 20, 2014

p.27: Just as the authors of the children’s Choose Your Own Adventure series helped pave the way to the cornucopia of interactive entertainment now offered today, so, too, might creative customer relations demystify credit scoring. Interactive modeling, known as “feedback and control,” has been successfully deployed in other technical contexts by a “values in design” movement. It has promoted automated systems that give individuals more of a sense of how future decisions will affect their evaluation. -- Highlighted apr 20, 2014

p.29: We concede that incidental indicators of good credit can become much less powerful predictors if everyone learns about them. If it were to become widely known that, say, the optimal number of credit accounts is four, those desperate for a loan may be most likely to alter their financial status to conform with this norm. However, we should also ask ourselves, as a society, whether this method of judging and categorizing people—via a secretive, panoptic sort—is appropriate. -- Highlighted apr 20, 2014