Abstract
Objective To understand the perspectives of primary care clinicians and health system leaders on the use of artificial intelligence (AI) to derive information about patients’ social determinants of health.
Design Qualitative study.
Setting Ontario, Canada.
Methods Semistructured, 30-minute virtual interviews were conducted with eligible participants across Ontario wherein they were asked about their perceptions of using AI to derive social data for patients. A descriptive content analysis was used to elicit themes from the data.
Main findings A total of 12 interviews were conducted with 7 family physicians, 3 clinical team members of various health professions, and 2 health system leaders. Five main themes described the current state of social determinants of health information, perceived benefits of and concerns with using AI to derive social data, how participants would want to see and use AI-derived social data, and suggestions for ethical principles that should underpin the development of this AI tool.
Conclusion Most participants were enthusiastic about the possibility of using AI to derive social data for patients in primary care but noted concerns that should be addressed first. These findings can guide the development of AI-based tools for use in primary care settings.
Artificial intelligence (AI) has become a ubiquitous part of our society and use within health care settings has been increasing. One emerging application of AI is exploring ways to identify social determinants of health (SDoH) for patients in the health care system.1-3 Social determinants of health are the socioeconomic positions that can shape one’s health status and include income, race or ethnicity, education, housing, occupation, gender, and other material or social factors.4 While the influence of social attributes and economic situations on health outcomes is well-established,4-7 this information is not routinely collected in clinical settings. Previous attempts to capture SDoH directly from patients have been relatively successful8; however, the COVID-19 pandemic disrupted this type of data collection as it is often gathered during in-person appointments. Artificial intelligence (specifically machine learning and natural language processing) could augment primary SDoH collection by developing models using large clinical and narrative data found in electronic medical records (EMRs) to create labels or phenotypes of patients on relevant social categories,1,9,10 rather than collecting this information directly from patients.
An important prerequisite in the design, deployment, and use of AI systems in health care is ensuring that such tools are co-designed with a multidisciplinary team that includes end users.11 This can provide end users with a better understanding of how the algorithms work, which in turn fosters greater trust in AI and its outcomes.12 There are several factors that influence the extent to which clinicians will trust AI: the complexity of the algorithm, sensitivity of the data, personal cognitive biases, lack of subject (or AI) knowledge, and the role of AI in a particular task or organization.12 Clinician trust in AI and the likelihood of AI adoption can be enhanced if there is transparency, fairness, and robustness in the AI development process and in the interpretation of outcomes.12,13
This study builds on our team’s ongoing work to derive SDoH information from primary care EMR data using machine learning and natural language processing. Building and deploying AI tools such as this is still relatively exploratory in Canadian health care settings, and our knowledge of the potential uptake and use of AI-based tools by clinicians is limited. This study is part of a participatory co-design process intended to understand the perspectives and preferences of family physicians, primary care teams, and health organization leaders around the use and presentation of AI-derived social data.
METHODS
Study design and setting
We followed the United Kingdom’s Design Council framework for innovation, which outlines 4 phases in the co-design process14: first, discover, where a broad spectrum of perspectives about a specific problem or stakeholder need is captured; second, define the issue by distilling and synthesizing information and ideas; third, develop, where the scope of work is expanded to generate and refine multiple possible solutions; and fourth, deliver the final product or design that best addresses the stated need.
This research used a narrative approach to undertake the first 2 phases, discover and define, which was achieved through exploratory, semistructured interviews with various end users and stakeholders. The study took place in Ontario, the most populous province in Canada, and was led by authors affiliated with the Department of Family and Community Medicine within St Michael’s Hospital, a major academic hospital located in downtown Toronto. The Unity Health Toronto Research Ethics Office approved this project (REB 22-036).
Participants and recruitment
Eligible participants included professionals who care for patients within a primary care setting or who are familiar with primary care EMR systems or data. This could include family physicians, members of the care team (eg, nurses, allied health professionals), and health system or organization managers or leaders. Participants were required to speak and understand English or French, be 18 years or older, and be able to provide informed consent.
A combination of convenience and snowball sampling was used to recruit participants through existing networks of the study team, such as the Department of Family and Community Medicine at Unity Health Toronto, Ontario Health Teams, the provincial health ministry, and primary care practice-based research networks across Ontario. The interviews took place between June 2022 and January 2023.
Data collection
Thirty-minute interviews were conducted virtually via Zoom videoconferencing software using a semistandardized interview guide, followed by a brief electronic questionnaire asking about participant sociodemographic, clinical, and professional characteristics, and about general knowledge of AI (supplementary material, available from CFPlus*). Participants were emailed an informed consent document beforehand and were asked to provide oral consent before starting the virtual interview. The interview guide was developed by the study team based on previous research work and collective expertise with AI tools for SDoH collection and use in primary care settings. Participants were provided with a short introduction outlining the purpose of the study, including our team’s preliminary development of machine learning models combined with natural language processing applied to structured and unstructured EMR data to derive information on 5 social determinants (ie, immigrant status, housing instability, income insecurity, gender identity, and sexual orientation [with other determinants to be considered in the future]). The interview topics consisted of understanding participants’ knowledge of AI, ease of technology use, and current access to social data; how they perceive AI-derived SDoH data in general (not specific to our previous work); and potential uses and presentation of AI-derived SDoH data if such a tool were to exist.
This study team consisted of several family physicians (K.W., N.R., and A.P.), a family medicine resident (S.L.), and quantitative and qualitative researchers with expertise in areas related to AI, data science, primary care, quality improvement, and SDoH (S.G., S.D., N.R., M.A., and A.P.). Two study team members (S.G. and S.L.) who were familiar with primary care practice and research conducted the virtual interviews, which were audiorecorded using Zoom and transcribed using Otter.ai software. Transcriptions were reviewed for accuracy by the interviewer. Interviews and analysis were overseen by an experienced qualitative researcher (S.D.) who also provided initial training to the interviewers.
Analysis
Preliminary data coding was conducted alongside the interviews. Once thematic saturation was reached, final coding and descriptive thematic content analysis were conducted. Transcripts were coded independently by 2 study team members (S.G. and S.L.) using deductive and partially inductive methods. Codes were then reviewed together to begin grouping similar codes and propose thematic categories and subcategories (if necessary). Any discrepancies in the coding or category assignment were resolved by a qualitative expert (S.D.). Preliminary themes were reviewed by all members of the study team.
Coding and analysis of the transcribed text was conducted using Nvivo 12 software version 12.6.0.959. Qualtrics software was used for the electronic post-interview survey.
RESULTS
In total, 12 interviews were conducted. Most participants were female (66.7%) and were diverse in age and location (Table 1). Seven were family physicians, 3 were clinical team members of various health professions, and 2 held leadership roles in the health system or in clinical practice. In terms of overall AI knowledge, 7 reported they had minimal knowledge and 5 described themselves as moderately or very knowledgeable.
Five main themes were identified related to the use of AI to derive patient SDoH information: current state of SDoH information in health systems and clinical settings, perceived benefits, concerns, presentation and use of AI-derived SDoH data, and ethical principles that should underpin the development of this AI tool.
Current state of SDoH information in health systems and clinical settings
Participants were asked what kind of SDoH information they currently had access to and how they were able to access it. Clinician respondents described that while some SDoH data are available in EMRs (eg, within patient history, profile, or free-text notes), they were usually limited to sex (often derived from sex reported on provincial health care cards and not necessarily indicating one’s gender identity), with some additional information on housing, occupation, and extended health benefits. Most SDoH were simply not available to clinicians or health system decision makers, such as ethnicity, race, income, gender identity, and sexual orientation.
Some respondents described additional mechanisms for collecting SDoH information from patients, such as surveys or questionnaires that were distributed within their practices, or through linkage to other data sources, as was the case for one decision maker. One clinician from a rural setting noted that they knew about their patient’s social circumstances, but this information was not documented explicitly in the chart:
I’m in a small community, I tend to know my patients quite well. So I would say it’s probably more, I don’t know if [infer] is the right word, but what I know of them just from looking after them for the time that I have, and just being familiar with the community in which I live and work. (Family physician)
Perceived benefits of using AI to derive SDoH information
Participants recognized various potential benefits; primarily, most anticipated that AI could help reduce the time and effort required to capture SDoH data in busy clinical practices in addition to producing more complete SDoH data that could be organized in a useful, efficient way for clinicians. Some also noted that the use of AI could alleviate uncomfortable questions around more sensitive social determinants and possibly eliminate conversations that would be distressing for patients. One family physician remarked, “People often feel distressed about … having financial difficulties.… [It] reinforces that distress by asking about it again.”
All clinical respondents anticipated that access to AI-derived SDoH data could result in potential improvements to patient care and assist in clinical decision making. Some stated that this would also provide a way to act more readily on social injustices or health inequities by identifying SDoH earlier and by offering appropriate resources for their situation. Additionally, it was suggested that AI could potentially alleviate clinician bias by providing a more objective assessment of a patient’s social determinants rather than based on assumptions.
Finally, participants saw ways for these data to be used for population health or other secondary uses (eg, quality improvement). A health system leader remarked, “We know that certain groups are unable to access health care. And if they don’t access health care, they don’t show up in the databases, and then they can’t get reflected in funding formulas.” The ability to scale AI-derived SDoH data to other regions with greater barriers in accessing health care may improve funding for those who are already marginalized.
Concerns about using AI to derive SDoH information
Many concerns were identified regarding the use of AI to derive SDoH information, which could be summarized in 3 domains: data or AI-related concerns, concerns about the use of the AI tool or AI-derived data, and advocacy regarding patient perspectives and involvement.
Data or AI-related concerns. Almost all participants expressed concerns about potential biases that might exist in the underlying EMR data or algorithms. Additionally, many questioned whether the accuracy of the AI-derived social data would be sufficient. For instance, some SDoH may be temporal and change over time, such as income and education, which may cause the accuracy of the algorithm to suffer. One participant also noted that some SDoH may be difficult to quantify, such as “ethnicities and mixed races.” Respondents noted EMR data may not be applicable or relevant when generating SDoH information using AI. Finally, respondents were concerned that the available technology within their practice sites may pose limitations.
Concerns about the use of the AI tool or AI-derived data. A number of participants cautioned that the use of AI-derived social data may further stigmatize patients who already face discrimination or that an AI tool could propagate health inequities:
A lot of my patients, especially my very socially vulnerable patients that are coming from countries that they escaped, … they’re already stigmatized, because they have HIV or drug addictions, and they want to keep as low of a profile as possible. (Clinician)
Artificial intelligence–derived SDoH data may have the unwanted consequences of revealing personal information that patients may not want to disclose or have not yet made public (eg, gender identity or sexual orientation).
Some participants mentioned that they were unclear whether the AI model would be transparent enough to understand or to explain to patients, which in turn may lead to mistrust. Participants also worried about potential misuse of the AI tool or derived social data that could potentially lead to inappropriate care or unintended patient harm:
So the [AI] data says this patient [has a] same sex partner, for example, and [if] this is incorrect—then it goes on the cumulative patient profile, it’s visible, and [this] gets faxed to a specialist. And the specialist asks [about it] … so loss of trust and faith if the data … [are] incorrect and driving incorrect decisions. (Clinician)
Finally, there was some apprehension that the technology would be commercialized and used for purposes other than for the provision of high-quality patient care.
Advocacy regarding patient perspectives and involvement. Many participants expressed the need for patient engagement, including patient involvement in the design of AI-based tools, patient consent, and acceptability from patients. More than half of the participants indicated uneasiness around whether AI tools would ensure patient confidentiality and privacy:
There’s a whole patient citizen dynamic that I think you’d want to understand really, really well. Because I could see lots of groups being potentially upset that … a faceless algorithm in the back was classifying them into certain kinds of groups.… [Y]our core identity is how you see yourself and self-identify. (Health system leader)
Presentation and use of AI-derived SDoH data
When asked how this information should be presented, most clinical participants preferred having data directly integrated into their current EMRs; however, participants were divided on whether the data should be accessible through a dashboard. Regardless of the presentation format, participants emphasized that the social data should be easily accessible and not distracting when viewed in patients’ charts.
Potential applications of AI-derived social data suggested by respondents were categorized into patient-level and population-level (or aggregate) uses.
Patient-level applications. This included specific uses, such as automated program suggestions or referrals (ie, social intervention resources, automated patient reminders) and confirming the AI prompt by asking whether patients had a specific social need. In general, participants indicated they wanted the AI tool to assist with prioritization of patient care to be able to inform or change clinical care, decisions, or processes, and to standardize or curate important patient information. When discussing health insurance coverage and social assistance, one clinician stressed that AI should do the following:
At least help coordinate [or] suggest these things so the nurse and doctor can … get social work involved [and] start pushing this through. As opposed to getting a notification that says, “Oh this person is vulnerable.” No, I want to go further than that, give me solutions. (Clinician)
Population-level applications. There were a number of uses described by participants, many of which included tasks that already occur but would benefit from the addition of SDoH or social needs information. For instance, most wanted simply to better understand their patient population or needs at a practice level. Respondents also indicated this would be helpful in identifying health care utilization patterns and measuring patient access and outcomes of care. Other suggestions included using AI-derived social data to inform program design and evaluation, and to assist with risk adjustment.
Ethical principles desired when building and implementing an AI tool to derive SDoH data
Participants suggested many different ethical considerations when designing and implementing AI tools to derive patient social data, and these were summarized into 7 principles: non-maleficence—the AI and outcomes must do no harm to the patient; beneficent—the AI tool needs to demonstrate that it can provide a potential benefit to the patient; equity and fairness—the AI tool should be fair and equitable (eg, it needs to perform similarly between subgroups of patients, as well as promote equitable resource allocation); patient engagement or governance—this was a recurring sentiment across all participants as patient knowledge and participation were seen as critical to appropriate use of these AI tools in a clinical setting; privacy and security—patient data being kept secure and confidential was an important factor, particularly given the sensitive nature of SDoH, and the outcomes of the AI tool (identifying categories of SDoH) needed to be carefully managed to ensure only appropriate clinicians or staff members could access this at an individual level; transparency and clarity—this would help both patients and clinicians understand more about AI outcomes, particularly if a clinician needed to explain why the EMR might be suggesting a specific social or economic need; and trustworthy—this was critical for embedding AI into clinical settings and to encourage clinicians and clinical staff to use the tool.
DISCUSSION
Patient SDoH data are important for both providers and health organization leaders—they are necessary for the provision of tailored, comprehensive patient care; to understand and act on inequities in care, access, or outcomes; and to better inform resource allocation and system planning. This study provides an understanding of preferences and concerns from potential end users of AI-derived social data to facilitate future work to design and present these data to clinicians and health system leaders in a useful and actionable way.
A recent pan-Canadian deliberative dialogue with primary care providers, patients, and decision makers identified several priority areas where AI is most needed: patient documentation, practice operations, and triage tasks.15 Our findings corroborated this in the context of a desire to improve access to patient social data and use it for equitable care and resource allocation. While participants expressed enthusiasm about the potential benefits of AI-derived social data for their patients, there were also important concerns raised that focused on ensuring patient safety, confidentiality, and compassionate care. It is well known that issues around bias and fairness exist in machine learning models and when used in health care settings this can perpetuate health inequities, discrimination, and patient harm, especially for women and those in racial and ethnic minorities.16 This stems largely from biases embedded in the clinical data used to train these models, which could derive from differential access to health care (and subsequently, which patients are included or excluded from the data), lack of diversity in data sets, and societal biases that may be reflected in the documentation patterns or clinical care processes within EMRs.16-19 Given this, an AI tool for deriving social data may be better suited to understanding panel- or population-level social determinants at an aggregate level instead of using it to inform direct patient care. This would also depend on the validity of the final model, including the minimum accuracy threshold users are willing to accept. The interviewers did pose this question to participants, but many reported that it would depend on the specific SDoH attribute and expressed uncertainty around what would be an appropriate level of accuracy (4 tentatively stated they preferred an accuracy between 80% and 90% and 2 said they believed accuracy over 90% was needed).
Concerns described by participants are not trivial and reflect the complexities of caring for patients facing social or economic hardships. As such, the AI development process needs to fit well within clinical workflow and include strategies for addressing social needs if any are identified. If clinics do not have enough staff, have limited time with patients, or face inadequate social resources in the community, then this proposed AI tool will not be an effective (and arguably, ethical) solution and could contribute to distress among patients and health care providers.20
A few strategies are required to deploy AI appropriately and effectively in primary care. Continued improvements in data quality are important for ensuring accurate algorithms and reducing bias,21 as well as for designing robust evaluations of AI-based models or tools before, during, and after deployment into clinical settings to understand impacts on clinical care or practice management. Additionally, approaches such as co-design and “human-in-the-loop” (ie, models requiring human interaction) used throughout AI development and deployment processes will help alleviate concerns around misuse and improve transparency and use of the tools.21,22 This is particularly relevant for ensuring patient and community needs and interests are being honoured. Finally, there is a clear need for clinician education about what AI is and what it does, and what it means to evaluate algorithms; this was also identified by providers as being an important enabler of AI use in primary care.15
Limitations
This study was limited by its modest sample size and restriction to 1 province in Canada, although thematic saturation can often be reached in 9 to 17 interviews.23 Further, the small numbers of family physicians and primary care organization leaders in each subgroup were likely not representative of their broader populations, and the views expressed here may not reflect all relevant perspectives, ideas, and concerns. More importantly, the patient perspective was not included in this study, although work is ongoing by our team and colleagues to understand patient views on AI-derived social data.15,24
Conclusion
While AI holds great promise for improving primary care efficiencies and enhancing patient care, there are many justifiable concerns expressed by our sample of primary care clinicians and health system leaders in Ontario. Engaging in thoughtful, multidisciplinary collaborations to design and implement AI tools, particularly around social data, is an important first step before addressing the specific concerns raised and doing so in accordance with the ethical principles articulated by participants.
Footnotes
↵* Supplementary material is available from https://www.cfp.ca. Go to the full text of the article online and click on the CFPlus tab.
Contributors
Drs Stephanie Garies, Andrew Pinto, and Karen Weyman conceptualized and designed the study. Drs Noor Ramji and Mo Alhaj contributed to conceptualizing the study. Drs Garies and Simon Liang conducted the interviews and analyzed the transcripts. Dr Garies drafted the initial manuscript. Dr Steve Durant advised on the methods and provided qualitative methods training and expertise. All authors critically reviewed and revised drafts of the manuscript and approved the final manuscript.
Competing interests
Dr Stephanie Garies received financial support for this work through the Canadian Institutes of Health Research Health System Impact Fellowship Program in partnership with Unity Health Toronto. The Canadian Institutes of Health Research played no role in the study design, data collection, analysis and interpretation of data, or writing of the manuscript.
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.
- Copyright © 2024 the College of Family Physicians of Canada