by Klay Brooks
Human beings and our institutions are collectively interacting with artificially intelligent systems every day. We can all sense that this is changing the world as we know it. It happened so quickly that it’s been hard to keep up, but I sat in on a vendor demo today that finally made this feel very real to me in a new way.
Following that meeting, I just pondered for hours. My brain was running wild with realizations about this initiative that the boss has me spearheading. The anxiety wasn’t only coming from how critical this is and how much responsibility is resting on my shoulders. There were enormous ethical considerations that kept piling up as well. The CEO is understandably concerned about the future of the consulting industry as a whole, given the unstoppable rise of more intelligent computing, but I found other things to keep me up at night. I had a ceremonial sip of whiskey and finished what work I could. Now I am drained and probably need some water and a nap, but my head is spinning.
–
For an intelligent system to be fair, it must embrace a diverse and inclusive worldview. That sounds trite though—what does that mean? Our own individual views are often built primarily on self-preservation and advancement, sometimes directly opposed to what we believe should be done for others. If we are the ones teaching these machines how to think and make decisions, how unbiased can any AI really be?
A hypothetical scholarship is awarded based on a new, AI-driven selection process that claims to be the most unbiased system on the market. The recipient views it as a well-deserved recognition of their hard work and potential. Although the top two candidates for the award have identical qualifications, the winner’s application includes words more similar to the ones in the description of the best candidate in the selection system. What nobody knows is that an intern typed those words on the website a few years ago, and an implementation technician hastily copy-pasted them into their new AI-driven selection system in the early hours of the morning on the day of go-live.
An insurance claim is denied. The customer service representative says that the decision is based on their system’s analysis. The patient demands to know more details. The rep lets slip that she doesn’t know the reason—it’s all decided by an AI. The patient pays out of pocket, and that rep’s days at the company are numbered. Their team lead calls a meeting to reiterate that under no circumstances should they mention the AI.
This is already happening. I recently showed my friend an applicant tracking system (ATS) compatible resume template and helped her to rephrase her skills and experience based on the job description that was posted online. Based on the data we provided to the ATS, she was highly qualified. She received an email almost instantly after she applied, and the system let her schedule her own interview for the next day.
The inner machinations of these enterprise automations are unimaginable to most. Many people simply do not understand how computers interpret what we put into them. Artificial intelligence is built to interpret the data we give it. Input false or unrecognizable information—”bad data”—and then the whole ethical conversation snowballs.
To win a battle with a computer, you have to know how to cheat. This ultimately defeats fairness, no matter how much diversity and inclusivity we painstakingly build into the code. Was there ever really fairness in this regard? No, not at all. I am not arguing that there is any less equity. Nowadays, getting what you want is simply a matter of how well you can game the system rather than who knows your dad or what the critics say. In the end, what is the right outcome? It is subjective. It depends. Unfortunately, it is currently not possible to get an AI to understand “it depends.” You need to specifically define what it depends on.
The personal experiences of the real people who set up these systems will inevitably shape the values used to make decisions with little to no human intervention. The ways these artificially intelligent solutions are set up will have even greater impact as the technology continues to progress and evolve. Where is the equity and inclusion in this process? Whose voice is being amplified here? AI has the potential to be a false god, an arbiter beholden to an algorithm and dressed up to look like natural selection. It will inevitably favor someone or something, and no amount of transparency can make this world of rapidly increasing complexity any less unfair.
My spiral truly began when I slowly started to realize that I would be helping clients understand how to set up their new intelligently automated systems to make decisions in the ways they see fit. That setup is going to mimic their worldview. The end result is still subjective, for better or for worse, so is anything really changing besides how fast it gets done? Yes, actually, and this change could be exponential depending on how many times that one slightly flawed interpretation gets recycled from one client to the next in the name of fast cash and keeping up with the times.
My task is to act as a guide, providing maps and tools to explorers of these new frontiers. It is my duty to explain the risks and benefits of certain trails they can take and the tools they can take with them. Nobody fully understands what might happen when they embark on this expedition. It cannot always be measured or deemed completely safe. I must ensure that the people I serve understand the opportunities and are at the same time well aware of the dangers.
This is just an introduction to the complexity of implementing artificially intelligent systems—planning the journey, communicating inherent risks and benefits, and navigating the ethical challenges of technological progress. My eyes are on the horizon as we all collectively move toward this strange future, armed with tools that might seem primitive to us in just a couple years.
In the government technology space, the vendors who build AI solutions often tout better citizen engagement, easier access to information, and more effective communication between constituents and their elected officials. We should look at this very closely before buying in. How do their concerns get filtered through the chatbot, categorized, and coded into the sentiment analysis report that is scheduled to appear in their elected official’s inbox every Monday?
Who builds the report? AI.
Who categorizes the responses? AI.
Who teaches the model how to do both?
Our team, together with the administrative staff of the client organization, will build these models and release them to the world. We will feed the machines our clients’ needs, and we will test that model as rigorously as we possibly can. Add into the mix a few developers with tight deadlines and a limited understanding of the impact of what they are building, and thus the future is pioneered, pixel by pixel.
These tools, as complex and subjective as any individual, are already being propagated in our towns, our states, and our nations. We can make it work for us and bring home the bacon, but it’s a unique burden to know how the sausage will be made.
a false god, an arbiter beholden to an algorithm and dressed up to look like natural selection
Klay Brooks is a Senior Consultant at Avero Advisors, specializing in digital transformation and public sector solutions. With a background in cultural anthropology, Klay brings a unique human-centered approach to business analysis and process improvement.