Sven Peterson (sven.peterson@premera.com) is Vice President of Compliance, Ethics and Regulatory Services | Corporate Compliance and Ethics Officer at Premera. Rose Riojas (rosa.riojas@premera.com) is Ethics Program Manager at Premera. The views expressed herein are theirs alone and are not intended to represent those of Premera.
Marie’s phone buzzed as she was getting ready for work. It was a text message from a number she didn’t recognize. Figuring it was a work colleague, she opened the text. To her surprise, it was a message inviting her to join group therapy sessions at a new studio located only a few blocks away. The message explained that based upon her recent healthcare experiences and purchases, Marie would likely benefit from addressing unresolved mental health issues before they became pressing. Furthermore, the new studio was covered by Marie’s health insurance.
Marie was stunned. She wondered who was behind the outreach to her and how they had obtained her information. What did they know about her medical history? Wasn’t that supposed to be protected information? Did they know her purchasing habits? The whole situation was creepy. She decided to reach out to her state’s attorney general’s office. She also posted on social media and reached out to her local television station.
While this example is fictional, it is inspired by true events, and it raises numerous ethical, reputational, and legal/regulatory issues. As a starting point, Marie is of course correct that there are many protections in place regarding personal data and privacy, including the Health Insurance Portability and Accountability Act (HIPAA) regarding certain healthcare data, the Gramm-Leach-Bliley Act regarding certain financial data, and other federal and state laws and case law.
But the law in this area is changing and is often unclear, with policy-makers racing to catch up with technological developments. In the meantime, companies need to consider what the right thing to do is from an ethical perspective, and how their actions will be considered by stakeholders such as customers, citizens, the press, legislators, regulators, and their own employees—all of whom may have differing and sometimes conflicting views. The absence of clear legal standards and widely accepted social norms makes it much more difficult for companies to navigate this terrain.
Ethical issues to consider
In our story, Marie’s first reaction centered around the perceived invasion of privacy. She wondered who was really contacting her and how they had obtained her data. Legally, there are different rules governing use and sharing of data by a healthcare provider or a health plan, or by a vendor who may be seeking to enter the market. However, even if those rules are followed to the letter of the current law, they may leave consumers and the public unsatisfied and concerned, leading to reputational harm for companies as well as possible investigations and legal action, which will be expensive and harmful to reputation even if ultimately successfully defended. Finally, these types of narratives often drive policymaking that sets future standards.
In the current case, artificial intelligence (AI) and machine learning (ML) were likely used to combine various data sets in order to determine that Marie would likely benefit from the program and, perhaps, had the means to pay. These data sets could have involved commercially available information regarding Marie’s purchasing habits, in addition to some healthcare data (though how healthcare data could be used is legally restricted depending upon the user and source). In order to better understand the issues at stake, let’s clarify how we define AI and ML. We will follow Microsoft in defining them as follows: “Artificial intelligence is the capability of a computer system to mimic human cognitive functions such as learning and problem-solving . . . Machine learning is an application of AI. It’s the process of using mathematical models of data to help a computer learn without direct instruction. This enables a computer system to continue learning and improving on its own, based on experience.”[1]
While Marie first focused on whether the offer was an invasion of her privacy, it could also be seen as an unfair distribution of services benefitting her. To see why, consider who all has been offered the service and why—and who it was not offered to. Are all similarly situated people being reached out to? While Marie may not want the service, it is certainly possible that others would want it but have not been targeted for outreach. This could be because the underlying data sources were themselves biased, which often occurs as the result of systemic bias in our society. While we frequently think of AI systems as neutral or without bias, when ML is applied to data, the underlying bias is replicated and reflected in the results. Consider Optum, which developed an algorithm to assist in identifying high-risk patients who may require additional medical care. Optum did not intend for the algorithm it used to be biased, and in fact, it explicitly excluded race and used a seemingly neutral variable: predicted future healthcare costs.[2] Unfortunately, this variable was a proxy for race due to racial differences in access to and usage of healthcare for similar health conditions. As a result, more care was directed to white patients than to Black patients with similar conditions. Optum modified the algorithm once the bias was discovered by changing it to predict future patient health conditions rather than costs, which significantly reduced biases.[3]
Even technology firms have inadvertently implemented biased algorithms. For example, in 2015, Amazon realized that a system it had developed to review résumés preferred male over female candidates.[4] In this case, the data set of applications over the last ten years were largely male, and therefore the application of the data set led to a preference for male applicants. Amazon eventually ended the project.
In addition to concerns regarding privacy, bias, and fairness in provision of services (possible discrimination), another issue to consider is accuracy. In Marie’s story, we don’t know if Marie really could benefit from the services in question, whether the underlying data is accurate, or if the way that ML may have combined data has resulted in accurate conclusions. If it hasn’t, this could create additional reputational or even legal risks for the company in question, particularly in a sensitive space.
As these examples and many others show, the use of AI and ML on potentially biased data sets can result in discriminatory outcomes. Often this occurs because a seemingly neutral variable (e.g., health spending for a particular condition) acts as a proxy for a protected variable such as race. Unfortunately, removing all such proxies can be difficult, particularly in an unequal society. This may result in outcomes that are illegal or unethical, for example, if they result in disparities in rates charged for members of different races for the same services. In a society that is structurally unfair, there can also be a trade-off between accuracy and fairness.
