Who Rules Big Data? Law, Knowledge, and Power

Who rules Big Data? It’s a topic as vast as the billions of pieces of information collected each day on U.S. citizens. And so the Law School’s signature Mitchell Lecture, in tackling this issue, called on not one but three thoughtful and provocative lecturers for this year’s presentation.

The Mitchell Lecture, established in 1950, has always brought a critical eye to important contemporary issues, and no issue is more pressing than the use, and potential misuse, of data in today’s society. Whether it is Facebook posts, police license plate scanners, credit ratings or YouTube videos, marketers, law enforcement officials and bureaucrats have more information at their disposal than ever before, and ever more sophisticated tools to analyze it. With that information comes power – and the question of how the law can help manage and circumscribe that power.

Introduced by Professor Martha T. McCluskey, chair of this year’s Mitchell Lecture Committee, the speakers at the March 27 event found cause for concern in myriad ways.

University of Maryland Professor Frank Pasquale is the author of a new book, The Black Box Society: The Secret Algorithms That Control Money and Information. He drew on material from that book in talking about the rankings and ratings that data analysis makes possible. More than 8,000 consumer rankings are compiled, he said, but the process by which they’re arrived at is not transparent – hence the “black box” in which these algorithms operate.

“As a society, we always want the boiled-down version,” Pasquale said. “‘Just give me the number.’ But the more we study ranking and rating in general, the more critical we become and the more we have to question that move from the consumer’s history to the score, or from the narrative to instantaneous evaluation.”

And the information being collected is more and more detailed. Soon, he said, all cars will come with “black boxes” that record such data as speed and acceleration; “insurers will be able to tell if you went 2 mph over the speed limit.” Similarly, he said, cell phones can be programmed to monitor a walker’s gait, information that could have implications for health insurers or law enforcement. “These kinds of black boxes are increasingly monitoring every aspect of our lives,” Pasquale said.

The problem, he said, is that as people in power increasingly use these “quantitative social indicators” to guide their decisions, they end up distorting behavior in unhelpful ways. An oft-used example is student testing, which is intended to monitor educational progress but can warp the curriculum instead.

The use of data collection and analysis in police work was the focus of the second speaker, University of California, Davis, law professor Elizabeth Joh.

Many police departments are turning to the tools of big data. For example, some departments are experimenting with “predictive policing”: the use of vast collections of data on past criminal activity to predict places where crime is likely to occur. In other cases, big data tools are being used to identify persons at high risk of being involved in future crimes. Chicago police, for example, analyze social networks to maintain a “heat list” of people who are connected socially to homicide victims. Those people face a very high risk of being involved in a future homicide, either as a victim or perpetrator.

An important legal question concerns how Fourth Amendment protections apply to a technique like automated license plate readers, which generate detailed portraits of individual behavior. With the collection of license plate scans, the police have access to vast amounts of data, and they have the ability to repurpose that data in different ways.

The final speaker, Virginia Eubanks, is an associate professor of women’s studies at SUNY Albany; her interest is in how automated decision-making affects those seeking public services, such as food stamps or Medicaid. Eubanks focused on a recent situation in Indiana, which implemented IBM technology to automate the application process for such public services. The result, she said, was a disaster for those seeking benefits. Rampant technical failures were only part of the problem; the system, which was dealing with a surge of applicants following the 2007 recession and Midwest flooding, routinely denied applications on the basis of minor anomalies. There were egregious examples: a deaf woman whose application was rejected because she couldn’t submit to a telephone interview, for example, or a woman whose Medicaid coverage was canceled because she was in the hospital and couldn’t complete her application. It was estimated that from 150,000 to 700,000 people “lost access to their life-sustaining benefits,” Eubanks said.

One problem, she said, was that the automated system worked too well, by applying “every rule precisely to the letter, every time. If you follow the letter of the law, the system basically freezes. The Indiana system is a zero-tolerance technology; it can’t adjust to variation or change.”

The notions of privacy and informed consent, Eubanks said, such a flashpoint for many, “are not meaningful in the context of public services. People in need of these programs trade their information and rights away in order to receive benefits. It’s not that they don’t want privacy, but they don’t expect the government to not be in their business. You can’t afford to refuse to provide any of this information, because if you do, you’re thrown out of the system for failure to cooperate.” 

Virginia Eubanks, an associate professor of women’s studies at SUNY Albany and a Ford Academic Fellow at the New America Foundation, will talk about technology and surveillance in the administration of social welfare programs. She is the author of Digital Dead End: Fighting for Social Justice in the Information Age, and is working on a book about digital surveillance in poor and working-class U.S. neighborhoods.

Elizabeth Joh, a professor of law at the University of California, Davis, will talk about data analysis and surveillance in the criminal justice system. Joh’s research focuses on the regulation of the police, with special emphasis on new surveillance technologies.

Frank Pasquale, a professor of law at the University of Maryland, will discuss his new book, The Black Box Society: The Secret Algorithms That Control Money and Information. Pasquale’s research addresses the challenges posed to information law by rapidly changing technology, particularly in the health care, finance and Internet industries.