Cities confront the challenges of algorithm transparency

There is a growing movement within cities to demonstrate transparency in the use of data and algorithms. However, frontrunners are finding that there are significant obstacles to delivering on these commitments.

Amsterdam has adopted the Tada manifesto with six principles for the responsible use of data. It also launched a mandatory online register for government and private sector sensors (excluding those such as police cameras) and along with Helsinki was one of the first cities to create an algorithm register detailing how algorithms are used in municipal services. Cities including Toronto and Barcelona are planning to follow the example.

To date, Amsterdam’s register has seven listings including licence plate recognition for parking management, personalised approaches to reducing serious crime, early detection of poverty risk, and prioritising investigations into illegal holiday rentals. Helsinki’s AI register also has seven entries mainly related to chatbots and library services.

Speaking on a recent webinar organised by the London Office of Technology and Innovation (LOTI), Douwe Schmidt, Data Ethicist, City of Amsterdam, said the implementation of the algorithm register has proven “pretty difficult” so far.

“We’re not happy yet with where we are,” he commented. “There are a lot of hidden algorithms in the city, where things are codified in some kind of codes and decisions are being taken.

“And it also poses a real challenge for the organisation to unearth all the parts where we use algorithms and then properly explain them.”

‘Ethics theatre’?

Researchers from the University of Oxford, the Alan Turing Institute and Cardiff University said in a paper last year that algorithm registers can work but that: “The AI registers, in their current incarnation, are unbearably light” and only provide a limited overview of the algorithmic systems used and developed by the cities.

Such efforts could exclude more sensitive applications of AI and amount to what the authors call ‘ethics theatre’.

Schmidt told Cities Today: “The register was purposefully released in beta stage when it was not yet clear how it should exactly work and what it should do. This way Amsterdam hoped that academics, NGOs and others could comment on what was missing.

“Giving the project beta stage, there would be enough room for change and improvement. Also, as we can see now, filling it with algorithms is the real hard part. We wanted to make sure that the public-facing part of the register was in accordance with people’s expectations.”

Schmidt believes there is a problem between government and researchers that he calls the “mutual stranglehold” where public institutions fear criticism from researchers and avoid sharing anything they aren’t 100 percent confident about, and in turn academics and the public get frustrated with governments for only sharing work that is complete but has no space for alteration or criticism.

“In that light, the algorithm register can be seen as a welcome breath of fresh air that tries to cut through this stranglehold,” he said.

He added: “To make the register in the first place proved an important step to convince more and more people that we should document our algorithms in a universal way to be able to be transparent about this. But that work is now the biggest hurdle to overcome and won’t happen overnight.”

Efforts to assess and document new algorithms and find and document existing algorithms are well underway but require “a significant investment of people’s time to complete,” Schmidt said.

Mikko Rusama, Chief Digital Officer for the City of Helsinki, said Helsinki’s register focuses specifically on artificial intelligence (AI). It is a “work in progress” and intended as a “window into the artificial intelligence systems used by the City of Helsinki” where residents can find quick overviews or more detailed information and give feedback.

He told Cities Today: “Currently, there are not many AI use cases. EU privacy law [GDPR] sets strict limitations on data use for ‘sensitive use cases’. We aim at full transparency on data and AI use.”

He added that the AI Register is only one component of the city’s work around ethical data use. Other aspects include a metadata model for algorithms to standardise descriptions, procurement standards for ethical AI, city-wide ethical principles for data and AI, and a governance model with a compliance and ethics board.

Helsinki is also working on implementing MyData principles – a digital permission management system through which users can grant or revoke consent for use of their data.

Definitions

Martha Norrick, Chief Analytics Officer and Director of the Office of Data Analytics in the City of New York, also highlighted some of the challenges of algorithm transparency.

“I think that’s a universal challenge across maybe all of our experiences,” she said during the webinar. “Even just what is an algorithm and what counts as an automated decision versus a non-automated decision.

“Definitions are challenging in this space and thinking about what rises to the level of impact, where is the line and how do we separate analysis and algorithms and all of the different ways that data is being used.”

“We don’t want to create a system where anytime somebody sorts a spreadsheet using a column, that counts as an algorithm and needs to be disclosed,” Norrick said. “In the end, sorting is an algorithm so it’s a complicated set of considerations to weigh.”

New York’s first summary of “high priority” algorithmic tools used in city agencies was published in 2020. High priority means tools that were developed with AI or machine learning techniques and those that collect or analyse personally identifying information.

The reporting process was initiated by New York’s first-ever Algorithms Management and Policy Officer (AMPO). The position was created by former Mayor Bill de Blasio in 2019 but was folded back into the Office of Data Analytics when Mayor Adams took office and consolidated various technology-related agencies under the Office of Technology and Innovation.

New York’s Chief Technology Officer Matthew Fraser told Cities Today in a recent interview: “The AMPO was consolidated, and the Chief Analytics Officer picked up the responsibilities associated with that because it’s one of those things where someone created an office – it was an office of one, two people. Two people aren’t capable of managing the entire workload that’s associated with that.”

Trade secrets

There is growing pressure for cities to be transparent about algorithms. A recent report from the Electronic Privacy and Information Center (EPIC) looked at automated decision-making (ADM) in Washington, D.C. and found at least 29 ADM systems from more than 20 agencies. According to the report, these include systems to predict which students are likely to miss school or to not graduate, as well as systems to screen housing applicants, identify potential food benefits fraud, predict recidivism, and inform juvenile sentencing decisions.

The researchers said it was not possible to identify all ADM systems in D.C. because many agencies were unwilling to share information due to companies’ claims of trade secrets or other commercial protections. In addition, they said the public does not have sufficient access to these systems to understand whether they are producing high-quality, accurate and fair decisions.

The report states: “Understanding automated decision-making is important because they make decisions that alter the paths of our lives. People deserve to know whether these systems are accurate, fair and unbiased, both on their own merits and compared to possible alternatives.”

Representatives from Washington, D.C. government did not respond to multiple requests for comment on the report.

Data ownership

During the webinar, Amsterdam, New York and London highlighted their ongoing efforts to embed data ethics.

Schmidt said Amsterdam’s Tada manifesto was a success and that awareness-raising throughout the whole city organisation was key. This included harnessing a network of people who are already knowledgeable and passionate about data ethics to support their colleagues. Training is continuing on the ethical implications of data and an ‘ethics leaflet’ stays with projects as they develop and move through departments, covering purpose, risks and mitigation. Schmidt likened this to the side effects information received with medicines.

In addition, Schmidt said that contracts with technology suppliers now state that all data generated through projects will be owned completely by the City of Amsterdam.

He commented: “This was quite a big shock for a lot of suppliers of technology because they were used to owning the data. It provides a steady contract and money flow because there’s a vendor lock-in problem. To break out of this was a big thing.”

Norrick sees New York’s Office of Data Analytics as a “central hub for information” on data ethics and using data responsibly. The city has over 3,000 datasets and almost 300,000 municipal employees and operates under a “patchwork of regulatory regimes” of local, state and federal laws, having no overarching legislation like GDPR.

Norrick’s team provides training and resources for city employees to help “systematise” data use and implement a checklist process.

She added that it’s also crucial to help employees consider critical questions about whether the data that they’re using to answer a particular question is appropriate and representative. Data from the 311 information and reporting service is a potentially rich resource, for example, but it must be remembered that not everyone uses 311, Norrick noted.

Public training hosted by volunteer ‘open data ambassadors’ such as librarians has also been successful and aims to help people use New York’s open data platform.

LOTI (a membership organisation of over 20 boroughs and the Greater London Authority) recently appointed its first data ethicist, Sam Nutt, and launched a data ethics service and toolkit to help local authorities with resident engagement, governance and skills related to data ethics.

Jay Saggar, Head of Data and Smart Cities at LOTI, said: “In our view, data projects will fail if there isn’t trust and if there isn’t a reflection and then a mitigation of potential bias and disproportionate impact in those data projects.

“It’s a reputation thing but it’s also about doing the right thing for the citizens. For us, there really shouldn’t be any data projects without data ethics.“

[This article was updated on December 15, 2022 to incorporate comments from the City of Helsinki].