Call for Theoretically Driven Quantitative Measures of Short-Term Medical Mission Effectiveness

It is common practice among randomized controlled trials and intervention studies to contrast risks and benefits before implementing on a larger scale (Melnyk & Morrison-Beedy, 2012; Sidani, 2014). Risk/benefit analysis is not a common practice, however, among short-term medical missions (STMM). STMMs are often comprised of a team of clinical personnel sent from a developed country to low and middle income countries (LMICs) to help with a specific medical need. Duration of a STMM can range between one day and a few years (Martiniuk, Manouchehrian, Negin, & Zwi, 2012). Many medical schools in developed countries encourage summer work experience in low and middle income countries (LMICs). Approximately 40% of medical students in the UK take a STMM trip over the summer (Martiniuk et al., 2012).  Additionally, many physicians take time off work to provide their skills in resource poor settings (Paul H Caldron, Impens, Pavlova, & Groot, 2017). The long term impacts of these STMMs are largely unknown and difficult to quantify. In a positive sense, STMMs can help in disaster relief contexts, where the burden of need exceeds the capacity of the local infrastructure. STMMs can also help mitigate burn out in physicians and staff working in resource poor settings (Paul H. Caldron, Impens, Pavlova, & Groot, 2016). Unfortunately, not all STMMs are well organized, and many end up producing long term damage in the communities being served (Rozier, Lasker, & Compton, 2017). A better understanding of STMMs is paramount, as some research indicates that these programs have increased in recent years, with the United States sending out the largest numbers of STMMs (Paul H. Caldron et al., 2016; Sykes, 2014). In fact, incidence of new physicians participating in STMMs has remained at approximately 10% since the early 2000’s resulting in a steady increase in prevalence (Paul H. Caldron et al., 2016). Despite growth in STMMs, very few programs collect outcomes data, and even fewer collect long term follow-up information (approximately 25%) (Sykes, 2014). It will be argued that a quantitative and holistic measure of STMM effectiveness must be created to evaluate the risks and benefits of individual STMMs.

The myriad of issues resulting from STMMs include, but are not limited to: lack of adequate follow-up, physicians exceeding their typical expertise, and disregard for informed consent (Paul H. Caldron et al., 2016; Martiniuk et al., 2012). A meta-analysis of STMMs by Martiniuk et al (2012), evaluated 230 peer-reviewed publications describing STMM programs. Among the reviewed publications, many of the STMM programs admitted that they were not sustainable long-term, were potentially minimally effective and not cost-effective (Martiniuk et al., 2012).  One publication poignantly noted the discrepancy in cost effectiveness: “what business did our team of 10 members (have in doing this, given the 10 members) have spent approximately $30,000 toward travel and hotel costs…when the entire cost of building a new 30-bed wing for the hospital in Ghana was $60,000?” (Abdullah, 2008). Another concern mentioned by Martiniuk et al. (2012), is ‘surgical tourism’, where surgeons hope to encounter rare cases that they may never see in a developed country: “It is a veritable feast of interesting cases” (Boyd, 1998). When the goal of the physician is personal, it becomes difficult to translate the STMM into a long-term benefit for the community. Physicians who often perform STMMs often pull business away from local doctors, offering services for free, thus discouraging growth of local medical practices (Bartelme, 2015).

Another issue of STMMs is a lack of involvement of the local community in the planning and implementation of the STMM. Rozier et. al. (2017) studied the perceptions of STMM organizers as well as the community staff on site. While the organizers of the STMM were mostly interested in providing direct patient care, the community staff repeatedly emphasized the need for continued educational support and capacity building (Rozier et al., 2017). In direct contrast to the STMM goals, 70% of the community staff sampled preferred receiving medical education rather than patient care aid. Community staff also preferred a longer length for the STMM, believing that only one week is essentially a “recipe for trouble” (Rozier et al., 2017). In fact, another meta-analysis of STMMs revealed that only 48% included an educational component (Sykes, 2014). Community workers noted that patients not arriving, medical equipment malfunctioning or other organizational issues may render a one-week trip useless. Community staff also complained about a minimal volunteer selectivity and a lack of acculturation of volunteers. Surprisingly, Rozier et. al (2017) found that only 18% of STMMs sampled had a regular community partner, with the large majority switching between community-based organizations or not having a community partner at all. It should be noted that the STMMs analyzed in Rozier et al (2017) were collected from the peer-reviewed literature, and therefore are more likely to be fairly well organized and effective compared to STMMs that are not published. Thus, Rozier’s estimates of community participation may be an overestimation. These concerns are not new to STMMs, however, there has been no method of accountability or measure of effectiveness to date.

Despite the damaging effects of a poorly conducted STMM, some STMMs provide strengths where non-governmental organizations (NGOs) and local hospitals are lacking. First, economic costs of STMMs are usually lower than government-based aid programs, mostly because physicians will fund their own trip and do not require payment for services (Paul H. Caldron et al., 2016). Caldron et al (2016) argues that policy makers might be able to incentivize and utilize STMMs as a way to offset the costs of U.S. foreign aid. Furthermore, it is difficult to ignore the need for skilled healthcare workers in LMICs, especially surgeons (Farmer & Kim, 2008). Patients in rural areas often have to travel a long distance to find a doctor who is trained, and this journey can be dangerous. Maternal deaths due to post-partum hemorrhage are easily preventable in a well-trained hospital facility, yet 216 out of 100,000 live births result in maternal death globally (WHO, 2016).

Given the potential benefits and harms of a STMM, it is critical to develop a quantitative method for conducting a risk/benefit analysis of an STMM. This topic has already gathered some attention, but more work is needed. A recent systematic review of recommended practices for STMMs reveals little to no consensus in terms of patient care, duration of stay, medical education, program organization or financing (Roche, Ketheeswaran, & Wirtz, 2017). Roche et al. (2017) concluded that a global standard must be established for STMMs.

Interestingly, a Health Impact Assessment Tool (HIAT) for STMMs had already been published by Maki and colleagues in 2008 (Maki, Qualls, White, Kleefield, & Crone, 2008).  Five surveys were created from qualitative, in-depth interviews of six missions program directors, staff and patients concerning STMM quality. The interviews revealed six major themes: cost, impact, education, efficiency, sustainability and preparedness. These themes were incorporated into a 120-item, Likert scale, self-report survey, which was administered to five separate STMM directors, staff and patients. Maki’s results indicated that most STMMs performed well in impact and cost, but not as well in education, preparedness, efficiency and sustainability. It should be noted, however, the lowest mean score was 64% (education), indicating STMMs tended to rate themselves highly. Maki acknowledges the possibility of response bias, where directors and staff may not respond objectively, rating their STMM in more positive light. Maki also did not perform reliability or validity analyses on the survey itself. Therefore, precision and accuracy of the survey is unknown. Furthermore, internal construct validity could have been evaluated to understand how well the survey measures the six constructs. Unfortunately, Maki’s tool has been unutilized and has not undergone further development. Skyes’s (2014) review of STMMs revealed only one out of 21 recent publications even mentioned Maki’s tool.  Sykes theorizes that perhaps many organizations are unaware such a measurement exists.

Regardless, a measurement tool like the one developed by Maki et. al. should be created to evaluate the true impact of the STMM. The measure should incorporate basic information like morbidity, education, and costs. Based on a large sampling of similar data from STMMs, a comparison score could be created for each domain of interest. A good place to start are Maki’s six themes: cost, impact, education, efficiency, sustainability and preparedness.

First, STMMs should record the overall expenses of their trip, including flights, food and lodging. Expenses could be balanced against medical equipment being donated and the level of training that the volunteers bring to the site. Second, Maki related impact to the quality of the STMM as perceived by the patients and local staff. The local community should evaluate the STMM on levels of cultural competence, professionalism, productivity and level of need. Organizers of the STMM should listen to feedback provided by the local staff and adapt accordingly. Third, successful education of the local medical staff should be measured using exams or simulation training exercises. Long term impact of the STMM is diminished if the local hospitals are unable to adequately monitor patients for complications. If staff are professionally trained, then it may be possible to pass on newest surgical techniques, diagnostic tools and other medical knowledge. Because education was of the highest priority for local staff (Rozier et al., 2017), outcomes measurement for education should be incorporated. Fourth, Maki defined efficiency as measurable patient outcomes including complications, morbidity and mortality. Sykes found that only between 60-80% of STMMs recorded morbidity and mortality data (Sykes, 2014). A continued partner organization in the local community should be able to track long-term patient outcomes. If the STMM changes local partners and regions, the STMM should at least maintain contact with the host organization to track patient progress. Fifth, sustainability is the level of long-term impact the STMM has on capacity-building and eventual independence of the local site. This would be a more difficult construct to measure, and the sustainability level should reflect the relative need of the region. Communication with the community on what types of training, equipment and resources are needed before starting the trip is essential for long-term sustainability. The goal of the STMM should not be mutual dependency, but rather guided steps towards regional independence. Lastly, a measure of preparedness could be evaluated by both the STMM and the host staff. Preparedness can include measures of team functionality, cultural competence and communication.

Obviously, there are limitations to a universal measure of STMM effectiveness. Not all STMMs work with a host organization. There are cases where, (for example, disaster relief efforts or remote villages with no health clinic), it may not be possible to maintain contact with patients or a local clinic. In these situations, a STMM measure like what has been described above, may not be appropriate. However, there should be some method for quality assessment implemented by the director of the STMM before engaging in such relief efforts. Obviously, informed consent should be practiced and a medical code of ethics should be strictly adhered to.

There is a great need for measurement and evaluation of STMMs. With the potential for harm, STMMs should be held accountable for their work. With the potential for benefit, STMMs should be able to share the effectiveness of the program with other STMMs, non-governmental and governmental organizations. With such a measurement available, local hosting hospitals and organizations can easily evaluate if the STMM program would be beneficial to them, rather than hoping for the best. This tool would be beneficial for directors of STMMs to identify serious gaps in their program, and make changes as needed. With the increasing numbers of STMMs, a method to evaluate and analyze their effectiveness is becoming paramount.