The only way to contain coronavirus cases in NZ is to go to Alert Level 4 for a brief period.
New Zealand currently has an unknown number of coronavirus cases wandering around our community. A number of observations support this, first is the experience of many other countries, second is the sudden growth in confirmed cases, third is that we have a cohort of international arrivals (and returning New Zealanders) who came to NZ between March 5-15th. This means they arrived before the mandatory 14-day quarantine, but still within the virus incubation period. Virtually all our new cases are from this cohort.
We now have a 4-level alert system. This is an excellent tool to get everyone on the same page and clearly to communicate both risk and required actions.
We should be at Level 4 right now.
This would allow us to reduce to Level 3 and then Level 2 as quickly as possible and avoid protracted disruption. I will explain.
The target we are chasing is invisible and our confirmed data is always behind the brute facts of the situation. In a situation like this the ONLY strategy that can succeed is to draw the circle of containment as wide as possible and then move inwards. This means setting Alert Level 4 right now, stay at home.
Over the next 12 or so days, ALL the coronavirus cases would then reveal themselves with symptoms developing (or if it is a mild case the infection passing almost unnoticed) without infecting anyone else. Once we have identified ALL the cases, then they go into isolation and we have stopped the spread.
After two weeks, towns, cities or regions with no cases revealed can be set back to Level 3 and even Level 2 and business can carry on. Towns or regions with cases might hold Level 4 for another week. But everyone can very quickly get back to Level 2.
We can then release the strict controls and just deal with one case at a time as they crop up, all of which (theoretically by that point) will be cases of returning New Zealanders who will reveal their symptoms while in 14-day self-quarantine and this will not be problematic.
We then implement widespread temperature checks, hand washing, stigmatization of coughing etc. But business can carry on.
What this implies
Whether Winston Peters’ encouraging of New Zealanders overseas to return is helping or facilitating the problem. Places like Wuhan now look relatively safe. As long as people don’t transit through airports where they could catch the virus, there are many places that might be safer than attempting to return, and the likelihood of bringing disease back into a quarantine zone.
In the future when we get warning of a novel outbreak we ought to immediately and dramatically close down the world for 2-3 weeks for exactly the reasons that I’ve outlined above. This should have happened around Jan 20th. Previous times that we might have taken this approach would have been with SARS, Ebola, MERS, and now Coronavirus. This is about 4 times per 20 years, or about 20 times per century. It is far preferable to suffer 2 weeks of GDP losses twenty times than 18 months once. This is a no brainer, but the mechanisms need to be coordinated ahead of time. A 20:1 false positive rate needs to be seen as acceptable.
Hopefully, New Zealand decision makers see the logic in this and raise us to Level 4 today, for a period of 2 weeks to minimise our losses.
New Zealand has just shut its borders to foreign visitors for the first time in history in an attempt to mitigate the impact of COVID-19. In what follows, I outline the case for border closure, the conditions under which it is rational, and then note that this is merely one measure against one catastrophe. Moving forward, we should get used to analysing and preparing responses to such large-scale risks.
Closing the border in a pandemic
Pandemics impose costs on society. These costs include up front hospital costs, ICU costs, lost productivity due to illness, and death, and downstream costs including human and emotional toll and long-term productivity loss.
Closing the border also imposes costs, these include disruption, visitor revenue forgone and more than likely reductions in trade and business downturn. But importantly, closing the border can also fail, if a significant amount of disease gets into the country anyway (eg from returning kiwis), and if there is a substantial outbreak despite border closure then the costs of closure and the costs of the pandemic are incurred.
Keep the border open in a pandemic or keep the border closed, it’s a lose-lose situation. However, previous research has demonstrated that provided a pandemic is serious enough, meaning that a sufficiently large proportion of the population are likely to get sick (say, 40%) and the disease has a sufficiently high case fatality (say, very much worse than Spanish flu of 1918) then the costs of border closure pale in comparison to the costs of the pandemic. Border closure is then a no brainer.
Border ‘filtering’ and border ‘closure’
But there is border closure and then there is border closure. The current New Zealand policy, let’s call it ‘filtering at the border’, allows for the free passage of returning New Zealanders, provided they agree to pass through the ‘airlock’ of 14 day quarantine, and also allows for New Zealanders to leave and then return again (although this is advised against). Both processes could bring disease into the country. A policy of ‘complete’ as opposed to ‘filtered’ border closure would prohibit the movement of kiwis too.
Would ‘complete’ border closure ever be justified. Possibly yes. Imagine if the case fatality rate of COVID-19 was 100% instead of 1%. There could really be no justification for risking anyone entering the country at all until the pandemic was over. If such a case ever arose, then immediately sealing the border and becoming a ‘refuge’ is a completely rational approach.
But back to ‘filtered’ closure. The hope is that by filtering out foreign arrivals, then the impact of COVID-19 will be much less. This could still prove fruitless, either if there is established community transmission from the cases already here, or if returning kiwis bring the virus home and spread it. In our previous research we modelled a scenario where border closure is attempted but fails, resulting in 90% of the unmitigated case-load. For a pandemic of more than 10 times the severity of Spanish flu, this was still a cost-effective measure (minimizing losses) under some assumptions. However, for swine flu (2009) it was never cost-effective to close the border.
How bad is COVID-19?
COVID-19 is an intermediate case. Under the ‘base case’ model of 40% infected, half being asymptomatic and 1% case fatality for the rest, it could cause 1 million symptomatic cases, and 10,000 or more deaths in New Zealand. This is a huge human, social and economic burden with ramifications for our economy for many years. But these costs (the economic ones) only barely balance out the lost visitor revenue from say 6 months of border closure. So, at first glance the case for closure is equivocal.
If trade continues, though curtailed by a policy of border closure, then the numbers become important. If closing the border (successfully) means the pandemic costs you $11 billion less than it would have otherwise (which is the result of modelling 72% infected and 2.3% dying, hospital costs, productivity loss, death, etc), then it is worth closing the border even if there is going to be an $11 billion drop in trade over the period of border closure (about 20% of trade for a six month period).
What is hard to predict, but is becoming clearer, is the global impact of the pandemic on travel and trade. If tourism drops to zero despite our actions, and trade plummets too, then the economic benefits of closure compared to no closure, suddenly appear much greater. You lose nothing by closing the border if there is no tourism and no trade anyway.
These are the numbers that need to be evaluated for a range of scenarios. And on the basis of these numbers COVID-19 looks to be right in the middle, between swine flu and disaster.
This means the decision may come down to factors other than strict economic ones and three factors play a key role: equity, access to care, and the value of a ‘quality adjusted life year’ (basically one year of good health).
Equity, access and value of life
COVID-19 differentially impacts the elderly. They suffer more severe illness on average and bear the brunt of deaths. This may also be true of other minority groups in New Zealand, we don’t know this yet. Our health system places a lot of weight on equitable treatment and prioritising ventilators is not equity. The best strategy to make sure everyone has a fair chance might be to simply close the border and keep it out.
COVID-19 places an immense burden on critical care facilities, and our modelling suggests that with an infinite supply of ICU beds COVID patients could occupy enough of them to cost the health system $1.9 billion or more. However, we only have enough beds that $270 million worth could ever be continuously in use over a six-month period. So, the analysis suggests costs well in excess of those that would be realised (ie the outbreak seems to get cheaper). But this results in unmet health need. And likely would result in preventable deaths. Which have a long-term cost themselves. Again, these are the factors on the scales that need to be weighed.
The modelling research we did used Treasury figures for the value of a year of good health, which are derived from Pharmac funding decisions. Basically, how much does Pharmac spend on medicines that lead to a gain in one year of good healthy life?
It may be that we New Zealanders decide to put much more value on healthy life than this (double? triple? it can’t be infinite or there would never be such a thing as a health budget, it would be a blank cheque). But this new valuation, if imputed in the models, can change the outcome of these economic analyses.
So, closing the border to protect vulnerable populations, to ensure fair access to care, or to save lives can be a very reasonable decision, even when it is not strictly an economically rational one.
Border closure can still fail
Bear in mind that border closure could still fail and the probability of failure should be built into economic modelling. We could still end up with just as many cases as in the no closure scenario. Many other variables in this equation (such as the exact epidemiological parameters of COVID-19 in certain contexts) also remain unknown.
Given the high degree of uncertainty, it may be rational to close the border to assess the situation. This is because, although borders can be re-opened at a later date (even re-opened almost immediately), we may lose the window of opportunity to close them if we don’t act now. This decision and the precise timing are very difficult, and although detrimental economically, it may be wise to move fast, as the government has in fact done.
But now we risk a very long waiting game. The virus is likely to be controlled in some regions of the world and not in others, resulting in a patchwork of ‘hot’ and ‘cold’ zones and severe travel disruption until a vaccine is available. There is almost a paradox in being too successful. If we keep it out then we must sustain our vigilance for the long haul or risk an outbreak every bit as bad as it can be, just down the track.
Global Catastrophic Risks
COVID-19 is a very harmful event, but it is not a global catastrophic threat. There are worse scenarios waiting in the shadows, and we must be prepared for them. Our experience with COVID-19 and the substantial fall-out will hopefully now prime decision makers to be receptive to the idea of building resilience against catastrophic risk. There are far more devastating biological threats that could arise, whether naturally or through biological manipulation, there are threats from other technologies, such as artificial intelligence (AI), geo-engineering, nuclear technologies, as well as threats from climate change, supervolcanoes, near earth objects, and others. COVID-19 has shown us how fragile our just-in-time systems are, how pursuit of a point of extra return can leave us undercapitalised, without cash flow, without inventory, at the mercy of global forces.
The government ought to take a risk portfolio approach to global catastrophic threats, and invest in assessing the probability of various risks, their magnitude and how to build resilience against them.
There is no reason why we can’t have an EQC-style fund to protect against risks that hit only once per generation, but hit hard. There is no reason why we couldn’t have walked through scenarios like COVID-19 with all sectors collaborating to identify the bottlenecks such as ventilator availability, testing locations, health workforce, data systems to track those in quarantine and so on.
The current event involves a biological virus, what if the next event involves a digital threat, would we close our internet borders? What is the case for doing so or for not doing so? This all needs to be scenario-ized.
The first significant step is to institutionalise the learnings from COVID-19, the phrase ‘global catastrophic risk’, and a commitment to undertake research, analysis, planning and systems testing much more often and more robustly.
I’ve just spent the last three days reading Stuart Russell’s new book on AI safety, ‘Human Compatible’. To be fair I didn’t read continuously for three days, this is because the book rewards thoughtful pauses to walk or drink coffee, because it nurtures reflection about what really matters.
You see, Russell has written a book about AI for social scientists, that is also a book about social science for AI engineers, while at the same time providing the conceptual framework to bring us all ‘provably beneficial AI’.
‘Human Compatible’ is necessarily a whistle-stop tour of very diverse but interdependent thinking across computer science, philosophy and the social sciences and I am recommending that all AI practitioners, technology policymakers, and social scientists read it.
The key elements of the book are as follows:
No matter how defensive some AI practitioners get, we need to all agree there are risks inherent in the development of systems that will outperform us
Chief among these risks is the concern that AI systems will achieve exactly the goals that we set them, even if in some cases we’d prefer if they hadn’t
Human preferences are complex, contextual, and change over time
Given the foregoing, we must avoid putting goals ‘in the machine’, but rather build systems that consult us appropriately about our preferences.
Russell argues the case for all these points. The argument is informed by an impressive and important array of findings from philosophy, psychology, behavioural economics, and game theory, among other disciplines.
A key problem as Russell sees it, is that most present day technology optimizes a ‘fixed externally supplied objective’, but this raises issues of safety if the objective is not fully specified (which it can never be), and if the system is not easily reset (which is plausible for a range of AI systems).
Russell’s solution is that ‘provably beneficial AI’ will be engineered according to three guidelines:
The machine’s only objective is to maximize the realization of human preferences
The machine is initially uncertain about what those preferences are
The ultimate source of information about human preferences is human behaviour
There are some mechanics that can be deployed to achieve such design. These include game theory, utilitarian ethics, and an understanding of human psychology. Machines must defer to humans regularly, ask permission, and their programming will explicitly allow for the machines to be wrong and therefore be open to being switched off.
Agree with Russell or disagree, he has provided a framework to which disparate parties can now refer, a common language and usable concepts accessible to those from all disciplines to progress the AI safety dialogue.
If you think that goals should be hard-coded, then you must point out why Russell’s warnings about fixed goals are mistaken. If you think that human preferences can always be predicted, then you must explain why centuries of social science research is flawed. And be aware that Russell preempts many of the inadequate slogan-like responses to these concerns.
I found an interesting passage late in the book where the argument is briefly extended from machines to political systems. We vote every few years on a government (expressing our preferences). Yet the government then acts unilaterally (according to its goals) until the next election. Russell is disparaging of this process whereby ‘one byte of information’ is contributed by each person every few years. One can infer that he may also disapprove of the algorithms of large corporate entities with perhaps 2 billion users acting autonomously on the basis of ‘one byte’ of agreement with blanket terms and conditions.
Truly ‘human compatible’ AI will ask us regularly what we want, and then provide that to us, checking to make sure it has it right. It will not dish up solutions to satisfy a ‘goal in the machine’ which may not align with current human interests.
What do we want to want?
The book makes me think that we need to be aware that machines will be capable of changing our preferences (we already experience this with advertising) and indeed machines may do so in order to more easily satisfy the ‘goals in the machine’ (think of online engagement and recommendation engines). It seems that we (thanks to machines) are now capable of shaping our environment (digital or otherwise) in such a way that we can shape the preferences of people. Ought this be allowed?
We must be aware of this risk. If you prefer A to B, and are made to prefer B, then how is this permitted? As Russell notes, would it ever make sense for someone to choose to switch from preferring A to preferring B, given that they currently prefer A?
This point actually runs very deep and a lot more philosophical thought needs to be deployed here. If we can build machines that can get us what we want, but we can also build machines that can change what we want, then we need to figure out an answer to the following deeply thought-provoking question, posed by Yuval Noah Harari at the end of his book ‘Sapiens’: ‘What do we want to want?’ There is no dismissive slogan answer to this problem.
What ought intelligence be for?
In the present context we are using ‘intelligence’ to refer to the operation of machines, but in a mid-2018 blog I posed the question what ought intelligence be used for? The point being that we are now debating how we ought to deploy AI, but what uses of other kinds of intelligence are permissible?
The process of developing and confronting an intelligence other than our own is cause for some self-reflexive thought. If there are certain features and uses of an artificial intelligence that we wouldn’t permit, then how are we justified in permitting similar goals and methods of humans? If Russell’s claims that we should want altruistic AI have any force, then why do we permit non-altruistic human behaviour?
Are humans ‘human compatible’?
I put down this book agreeing that we need to control AI (and indeed we can, according to Russell, with good engineering). But if intelligence is intelligence is intelligence then must we necessarily turn to humans, and constrain them in the same way so that humans don’t pursue ‘goals inside the human’ that are significantly at odds with ‘our’ preferences?
The key here is defining ‘our’. Whose preferences matter? There is a deep and complex history of moral and political philosophy addressing this question, and AI developers would do well to familiarise themselves with key aspects of it. As would corporations, as would policymakers. Intelligence has for too long been used poorly.
Russell notes that many AI practitioners strongly resist regulation and may feel threatened when non-technical influences encroach on ‘their’ domain. But the deep questions above, coupled with the risks inherent due to ‘goals in the machine’, require an informed and collaborative approach to beneficial AI development. Russell is an accomplished AI practitioner speaking on behalf of philosophers to AI scientists, but hopefully this book will speak to everyone.
The Global Health Security Index which considers pandemic threats has just been published. Unfortunately NZ scores approximately half marks (54/100), coming in 35th in the world rankings – far behind Australia. This poor result suggests that the NZ Government needs to act promptly to upgrade the country’s defences against pandemic threats…
The AI Forum of New Zealand has just published a report on AI and Health in the New Zealand context. The report trumpets some of the potential cost savings and efficiencies that AI will no doubt bring to the sector over the next few years. However, there are other interesting findings in the research report worth highlighting.
Enhanced teamwork and patient safety
AI that employs a combination of voice recognition, and natural language processing could help monitor and interpret healthcare multidisciplinary team interactions and help to ensure that all information relevant to a discussion has been raised or acknowledged.
This is important because we know that there are many failures in healthcare teamwork and communication, which often have their root cause in failures of information sharing and barriers to speaking up in a hierarchical team setting. This can and does impact patient safety. Including an AI assistant in future health teams could help overcome barriers to speaking up and sharing information.
Overcoming fallible human psychology
We also know that a range of psychological biases are at work when healthcare staff make decisions. These biases include a tendency to be influenced by recent experience (rather than statistical likelihood) and the tendency to attend to information that confirms the beliefs already held. Furthermore, doctors and other clinicians do not often get feedback on their diagnoses. This can lead to a tendency to increase confidence with experience without a parallel increase in accuracy.
One key promise of medical AI is that it can fill in the gaps in clinical thinking by providing a list of potential diagnoses or management plans with a statistical likelihood that each is correct. This kind of clinical decision support system could overcome one key failure in diagnosis, which is that the correct diagnosis is often not even considered.
However, in order to embrace these tools clinicians will also need to understand their own fallibility, the psychology of decision making and the very human cognitive processes that underpin these shortcomings. Intelligent digital systems undoubtedly have their own shortcomings, and their intelligence is best suited to particular kinds of problem. Human psychology suffers from complementary blind spots and it is the combination of artificial and biological intelligence that will advance healthcare.
Ensuring safe data
Another issue discussed in the AI Forum’s report is the need to make health data available in a form that AI can consume without worrying breaches of privacy. There is a significant challenge facing developers to find ways to de-identify free text (and other) data and present it in a form that is both machine readable and also unable to be re-identified whether intentionally or accidentally.
There is a risk that identifiable data could be used, for example, to prejudice insurance premiums on the basis of factors that people do not have control over. The process of de-identifying is proving to be very difficult. For example, even with names and addresses and such identifying features removed from clinical records (itself a challenging task given the many ways that such information can be recorded) there is still the possibility that merging datasets such as mobile phone location data held by, say Google, with clinical records that record the day and time of appointments, could intentionally or inadvertently identify individuals. Issues such as this need to be solved as we move forward with AI for healthcare.
We now need cost-utility analyses
The next step is to catalogue the AI tools that we presently have available and begin assessing the potential impact of these systems. Funders and provider institutions need to conduct cost-effectiveness analyses on these new tools and prioritise those that both increase effectiveness and clinical safety while also reducing time and saving costs. These investments might well take priority over investments in expensive new pharmaceuticals that marginally improve outcomes at great additional expense.
There is likely to be a lot of low hanging fruit in routine, repetitive teamwork and diagnostic tasks that AI is suited to assist with and where the public health dollar will go a long way, benefitting many patients not just a few.
AI offers many promising applications in the health setting and all those involved in the sector would be advised to read reports such as the AI Forum of NZ’s report on AI and Health in NZ, and think creatively about how AI might help solve the grand challenges in healthcare in the 21st Century.
I recently published an academic paper about island refuges in extreme pandemics with my co-author Nick Wilson. The paper has become the focus of media attention, including articles by Newsweek, Fox News and IFLScience.
Unfortunately, the key argument of the paper has been misconstrued in several reports. The headlines included click bait such as, ‘If A Pandemic Hits, This Is Where Humanity Should Go To Survive’ and ‘Scientists rank safest spots to flee to if apocalyptic outbreak begins to wipe out humanity’.
The conclusion of our argument was almost the opposite. We argued that preparations could be made ahead of the fact, so that the designated island refuge(s) could be closed off almost immediately when signals indicate a catastrophic pandemic is emerging. There would be no fleeing to the ‘lifeboat’ allowed.
In fact, the metaphor of a ‘lifeboat’ is misleading, because people scramble for a lifeboat when the disaster strikes. Our argument (which I explain below for those who have not read our paper) is that the islands most likely to be able to reboot a flourishing technological human culture after the event, should be identified ahead of time, and plans enacted to prevent people arriving when the catastrophe strikes (i.e. through border closure measures).
In the literature on catastrophic and existential risk mitigation, the concept of an ‘information hazard’ is the idea that some information when spread actually increases the risk of a catastrophic outcome. The ‘run to the island refuge’ approach is an information hazard. If people actually behaved like this, it would undermine the effectiveness of the refuge, and increase the probability of (in this case) human extinction.
Our argument is about preserving technological culture, ensuring the continuation of the ‘human project’ and the flourishing of future generations. It is not about saving people around the world at the time of the catastrophe.
The fact that any particular people might survive the event is incidental to the bigger picture that some people could survive and ensure a future full of human beings and full of technological culture.
We identified three promising options for such an island refuge to preserve humanity. And one might argue that it is not fair on the diverse cultures of the world if just Iceland or just Australia, for example, survive the catastrophe.
However, there is nothing preventing plans for a designated refuge to include representation from all the worlds people, perhaps a system of rotating work visas, so that the refuge can host, at any given time, a sample of the members of each of the world’s jurisdictions. Whoever happens to be there at the time of the catastrophe would then effectively get ‘locked in’ and would represent their culture as the world is re-booted.
Providing such visas and hosting these diverse people could be an undertaking that the designated refuge nation(s) take on, perhaps in lieu of providing development assistance to the rest of the world, so that the refuge nation is not unfairly burdened should the refuge never be needed (which is also a likely outcome).
Our paper is only the first step in a conversation about how the world might mitigate the risk from extreme pandemics, and we encourage other ideas so that we can little by little implement strategies that reduce the probability such unlikely but devastating events can destroy humanity.
The TLDR version of our paper (570 words):
The risk of human extinction is probably rising. This is because, although the background risk from such natural events as supervolcanic eruption, or asteroid strikes has remained consistent, the risk from technological catastrophes rises with our technological prowess (think nuclear, nano, bio, and AI technologies) which is by definition unprecedented in human history.
There are a number of reasons to work to preserve the future existence of humanity. These include the value of the vast number of future human lives, the worth of continuing key cultural, scientific and artistic projects that span long periods of time, and the cost-effectiveness of mitigating disaster.
We assume that the continuation of large and advanced civilizations is more valuable than the persistence of only a handful of subsistence societies.
A pandemic or perhaps more probably multiple pandemics occurring together poses a risk to the existence of humanity. The WHO hypothesizes about ‘Disease X’, a disease about which we yet know nothing, and which could emerge from increasingly accessible biotechnological manipulations. Such an event(s) would fall under Nick Bostrom’s concept of ‘moderately easy bio doom’, where a small team of researchers in a small laboratory, working for a year or so, might engineer a catastrophic biological threat. This would only need to occur once to threaten the entire human species.
The probability of such an event is unknown but is likely to be non-zero. A survey of experts in 2008 concluded on the basis of expert opinion that an existential threat level pandemic has a 2% chance of occurring by the year 2100. Two percent times a 10 billion population, means we are looking at an expected human life loss of 200 million people. Reducing the probability of such an event even by half a percent would be hugely worthwhile, particularly if it can be done with existing resources, and within existing governance structures.
One way to mitigate the risk of a pandemic killing all humans, is to designate some location as a refuge (just as with nuclear fall-out shelters in the cold war), which is quarantined (i.e. no one goes in or out) as soon as the triggering event is observed. Land borders are easily crossed by those infected with a disease, but islands have escaped pandemics in the past.
Key features required by such an island refuge are: a sufficient population, skills, and resources to preserve technological culture and rebuild society, and self-sufficiency and resilience to ride out the crisis.
We set about trying to identify the most likely locations (at the level of island nations) that satisfy these criteria. To do this we formalized a 9-point scoring tool covering: population size, number of visitors, distance from nearest landmass, risk of natural hazards, infrastructure and resources (using GDP per capita as a proxy), energy and food self-sufficiency, political stability and social capital.
When scored on our 0-1 metric, Australia, then New Zealand, then Iceland appear to be the most promising island refuges.
A lot of work yet remains to be done. Perhaps there are better interventions to protect the world from an existential level pandemic? Perhaps some other factor(s) is/are critically important to the success of an island refuge that we have not yet included (strong military, the presence of universities or server farms for the world’s knowledge, or some other factor)? How long could the optimal islands hold out? Is there some threshold of population that is essential to ensure that key skills such as nuclear physics or neurosurgery are preserved? We now encourage discussion and debate around these (and many other) issues.
In this post I aim to describe some of the ethical issues around the use of algorithms to make or assist decisions in recruitment and for managing gig employment.
When discussing ethics we are trying to deduce the right thing to do (not the permissible thing, or the profitable thing, or the legal thing).
AI in recruitment
Consider Usha, who is a software engineer specialising in machine learning. Let’s imagine for the purposes of example, that she is in fact the most qualified and experienced person in the applicant pool for an advertised position, and would in fact perform the best in the role out of the entire applicant pool. In her application:
She uses detectably ‘female’ language in resume
She notes she didn’t start coding until the age of 18
She was the founding organiser of LGBTQ on campus
She also has a non-Western name and her dark skin tone made it difficult for an AI system to register her affect during a recorded video interview with a chat bot.
Faced with this data, an AI recruitment algorithm screened her out. She doesn’t get the job. She didn’t even get an face to face interview. Given the circumstances, many of us might think this was wrong.
Perhaps it is wrong because some principles such as fairness, or like treatment of like, or equality of opportunity have been transgressed. Overall, an injustice seems to have occurred.
In his book Future Politics, Jaime Susskind lays out the various was in which an algorithm could lead to unjust outcomes.
Data-based injustice: where problematic, biased or incomplete data leads the algorithm to decide unfairly
Overt: the algorithm contains explicit rules discriminating against some people, e.g. discriminating against people on the basis of sexual orientation.
Implicit: the algorithm discriminates systematically against some kinds of people due to correlations in the data, e.g. discriminating against those who didn’t start learning to code until after the age of 18 might discriminate against women due to social and cultural norms.
The neutrality fallacy: equal treatment for all people can propagate entrenched bias and injustice in our institutions and society.
Susskind notes that most algorithmic injustice can be traced back to actions or omissions of people.
Another way of formulating algorithmic ethics is in terms of human rights. In this case, rather than look to the outcome of a process to decide whether it was just or not, we can look to the process itself, and ask whether the applicant’s human rights have been respected.
But it might be that case that ongoing ethical discourse could lead us to new rights in the age of AI, perhaps:
The right to not be measured or manipulated?
The right to human interaction?
The foundations for reasoning about rights transgressions or whether outcomes are just or unjust are found in ethical systems. Such systems have been constructed and debated by philosophers for centuries.
The Institute of Electrical and Electronic Engineers (IEEE) recognises this and in their (300 page!) report ‘Ethically Aligned Design’, they identify and describe a number of Western and non-Western ethical systems that might underpin considerations of algorithmic ethics. The IEEE notes that ethical systems lead us to general principles and general principles define imperatives. The IEEE list and explain eight general principles for ethically aligned design.
One example of an ethical system is what might broadly be considered the ‘consequentialist’ system, which determines right and wrong according to consequences. A popular version of this approach is utilitarianism, the ethical approach that seeks to maximise happiness. As an example, under utilitarianism, affirmative action can be good for society as a whole, enriching the experience of college students and enhancing representation in public institutions and making everyone happier in the end. This approach tends to ensure that we act ‘for the greatest good’.
Another example of an ethical system is deontology or ‘rules based’ ethics. Kantianism is a version of deontology, which argues that ethical imperatives come from within us as human beings, and the right thing to do boils down to ensuring that we treat all people with dignity, ensuring that they are never a mere means to an end but an end in themselves. This approach tends to lead to the formulations of right and duties. For example, it would be wrong to force someone to work without pay (slavery) because this fails to respect their freedom, autonomy, humanity and ultimately dignity, irrespective of the outcomes.
In their report, where the IEEE deduces their general principles of ethically aligned design from a foundation of these ethical systems, the authors note that, “the uncritical use of AI in the workplace, and its impact on employee-employer relations, is of utmost concern due to the high chance of error and biased outcome.”
The IEEE approach is not the only published declaration of ethical principles relevant to algorithmic decision making. The Berkman Klein Institute has catalogued and compared a number of these from public and private institutions.
The Gig Economy
Let’s turn now to gig work. Think of Rita, she is a gig worker working for a hypothetical home cleaning business that operates much like Uber. Rita’s work is monitored by GPS ensuring she takes the most direct route to each job, she’s not sure whether the company tracks her when she’s not working. Only time spent cleaning each house is paid, and the algorithm keeps very tight tabs on her activities. Rita gets warning notifications if she deviates from the prescribed route, such as when she needs to pick her son up from school and drop him to the babysitter. She gets ratings from clients, but one woman, an historical ethnic rival, always rates her low even when she does a good job, the algorithm warns her that it’s her last chance to do better. Rita stresses about the algorithm, feels constantly anxious and enters a depression. She misses work, has no sick pay to draw upon, and spirals downward.
We may conceive of such algorithms as ‘mental whips’ and feel concerned that when acting punitively they may be taking data out of context. Furthermore, the ethically appropriate response from the algorithm to, say, an older worker who falls ill might well be different from that to a wayward youth who slacks off. Justice may not be served by equal treatment.
Phoebe Moore has noted “[such] human resource tool[s] could expose workers to heightened structural, physical and psychosocial risks and stress.” – this is worse if workers feel disempowered.
Surveillance and the Panopticon
Many of the issues around gig management algorithms boil down to issues of surveillance.
Historic surveillance had limitations (e.g. a private detective could only investigate one employee at a time). However, with technological advance we can consider surveillance taken to its purest extreme. This is the situation Jeremy Bentham imagined with his panopticon. A perfect surveillance arrangement where one guard could observe all prisoners in a prison (or workers in a factory for that matter) at all times, without the guard being seen themselves. As soon as workers know this is the situation their behaviour changes. When a machine is surveilling people, people serve the machine, rather than machines serving people.
The panopticon is problematic for a number of reasons. Firstly, there is an unfounded assumption of innate shirking. There may be no right to disconnect (especially if the employer performs 24/7 surveillance of social media).
As with Rita there are risks that surveillance data can be taken out of context. We also know that the greater the surveillance, the greater the human demands for sanctions on apparent transgressions.
Finally, the system lacks a counter measure of ‘equiveillance’, which would allow the working individual to construct their own case from evidence they gather themselves, rather than merely having access to surveillance data that could possibly incriminate them.
Ethically we must ask, who is this situation benefitting? Employment should be a reciprocal arrangement of benefit. But with panopticon-like management of workers, it seems that some interests are held above those of others. Dignity may not be respected and workers can become unhappy. It could be argued that Rita is not being treated as an end in herself, but only as a mere means.
It’s true that Rita chose to work for the platform, and by choosing surveillance, has willingly forgone privacy. But perhaps she shouldn’t be allowed to. This is because privacy has group level benefits. A lack of privacy suppresses critical thought and critical thought is necessary to form alliances and hold those that exploit workers to account.
As a society we are presently making a big deal about consumer privacy, but what about employee privacy and protections? Ethics demands that we examine these discrepancies.
We might want to ensure that humans don’t become a resource for machines, where the power relationship is reversed, where human behavior (like Rita’s) is triggered by machine activity rather than the other way around. The risk is not that robots will take our jobs, we will become the robots living ultra-efficient but dehumanized lives
Ghost Work author Mary Gray says, “[one] problem is that the [algorithmic gig] work conditions don’t recognize how important the person is to that process. It diminishes their work and really creates work conditions that are unsustainable.” This argument contains both consequentialist and deontological points against overzealous algorithmic management of people.
Is there a duty to use algorithms?
I’ve called into question some of the possible uses of algorithms in recruitment and managing the gig economy. Potential injustice seems to lay in wait everywhere in bad data, implicitly unjust rules, and even neutral rules. But when are algorithms justified? What if customer satisfaction really is ‘up 13%’? Is this an argument for preserving the greatest happiness at the expense of a few workers? Or perhaps techniques for ‘ethically aligned design’ could lead to systems that overcome the ‘discriminatory intent’ in people and also enhance justice (dignity) in the process.
“We can tweak data and algorithms until we can remove the bias. We can’t do that with a human being,” – Frida Polli, CEO Pymetrics.
However, the duty to respect human dignity may require some limitations on the functions and capability of AI in recruitment and the management of gig work. We need to examine what limitations.
Australia’s Chief Scientist, Dr Alan Finkel, has proposed the ‘Turing Certificate’, a recognised mark for consumer technologies that would indicate whether the technology adheres to certain ethical standards. This discussion should be ongoing.
Finally, the irony that we implement oversight and regulatory force to combat the use of surveillance and algorithmic force is not lost on me…