Pragmatic Free Trial Meta
Pragmatic Free Trail Meta is an open data platform that allows research into pragmatic trials. It gathers and distributes clean trial data, ratings and evaluations using PRECIS-2. This allows for a variety of meta-epidemiological studies to compare treatment effect estimates across trials of different levels of pragmatism.
Background
Pragmatic trials provide real-world evidence that can be used to make clinical decisions. However, the use of the term "pragmatic" is not uniform and its definition as well as assessment requires further clarification. Pragmatic trials should be designed to guide clinical practice and policy decisions, rather than confirm an hypothesis that is based on a clinical or physiological basis. A pragmatic trial should also strive to be as close to real-world clinical practice as is possible, including its participation of participants, setting and design as well as the execution of the intervention, as well as the determination and analysis of the outcomes, and primary analysis. This is a major distinction from explanation trials (as described by Schwartz and Lellouch1) which are designed to provide more thorough proof of the hypothesis.
Truly pragmatic trials should not blind participants or clinicians. This can result in an overestimation of the effect of treatment. Practical trials also involve patients from various healthcare settings to ensure that their results can be applied to the real world.
Finally, pragmatic trials must focus on outcomes that matter to patients, like the quality of life and functional recovery. This is especially important for trials that involve invasive procedures or have potentially dangerous adverse consequences. The CRASH trial29, for instance, focused on functional outcomes to compare a two-page report with an electronic system for monitoring of patients admitted to hospitals with chronic heart failure, and the catheter trial28 utilized urinary tract infections that are symptomatic of catheters as the primary outcome.
In addition to these features pragmatic trials should reduce the procedures for conducting trials and data collection requirements in order to reduce costs. Furthermore, pragmatic trials should seek to make their findings as relevant to actual clinical practice as is possible by making sure that their primary analysis is the intention-to-treat approach (as described in CONSORT extensions for pragmatic trials).
Despite these guidelines, a number of RCTs with features that defy pragmatism have been incorrectly self-labeled pragmatic and published in journals of all kinds. This can lead to misleading claims about pragmatism, and the term's use should be standardised. The development of the PRECIS-2 tool, which provides an objective standard for assessing practical features is a good initial step.
Methods
In a pragmatic study it is the intention to inform clinical or policy decisions by showing how an intervention could be implemented into routine care. This is different from explanatory trials that test hypotheses regarding the cause-effect relationship in idealised settings. In this way, pragmatic trials can have a lower internal validity than explanation studies and are more susceptible to biases in their design analysis, conduct, and design. Despite their limitations, pragmatic studies can be a valuable source of information to make decisions in the context of healthcare.
The PRECIS-2 tool measures the level of pragmatism that is present in an RCT by assessing it across 9 domains that range from 1 (very explicative) to 5 (very pragmatic). In this study the domains of recruitment, organisation and flexibility in delivery, flexibility in adherence, and follow-up scored high. However, the main outcome and method of missing data were scored below the practical limit. This indicates that a trial can be designed with effective pragmatic features, without damaging the quality.
It is difficult to determine the level of pragmatism within a specific trial because pragmatism does not have a binary characteristic. Some aspects of a study can be more pragmatic than others. The pragmatism of a trial can be affected by changes to the protocol or logistics during the trial. Additionally 36% of 89 pragmatic trials discovered by Koppenaal et al were placebo-controlled or conducted before licensing and most were single-center. They aren't in line with the norm, and can only be called pragmatic if their sponsors agree that the trials are not blinded.
A common aspect of pragmatic studies is that researchers try to make their findings more meaningful by analyzing subgroups within the trial sample. However, this can lead to unbalanced comparisons with a lower statistical power, increasing the risk of either not detecting or incorrectly detecting differences in the primary outcome. In the instance of the pragmatic trials that were included in this meta-analysis this was a significant problem since the secondary outcomes weren't adjusted for differences in the baseline covariates.
In addition the pragmatic trials may have challenges with respect to the collection and interpretation of safety data. This is because adverse events are typically reported by participants themselves and are prone to reporting delays, inaccuracies or coding errors. It is essential to improve the accuracy and quality of outcomes in these trials.
Results
While the definition of pragmatism may not require that all trials be 100 percent pragmatic, there are benefits to including pragmatic components in clinical trials. These include:
Increasing sensitivity to real-world issues as well as reducing study size and cost as well as allowing trial results to be faster implemented into clinical practice (by including patients from routine care). But pragmatic trials can have disadvantages. The right kind of heterogeneity, for example could help a study generalise its findings to many different settings or patients. However the wrong kind of heterogeneity can decrease the sensitivity of the test, and therefore reduce a trial's power to detect even minor effects of treatment.
A variety of studies have attempted to classify pragmatic trials using different definitions and scoring methods. Schwartz and Lellouch1 developed a framework to distinguish between explanation-based trials that support a physiological or clinical hypothesis as well as pragmatic trials that aid in the selection of appropriate therapies in clinical practice. The framework was comprised of nine domains, each scored on a scale ranging from 1 to 5, with 1 indicating more explanatory and 5 indicating more pragmatic. The domains included recruitment of intervention, setting up, delivery of intervention, flexible adhering to the program and primary analysis.
Pragmatic KR tool3 was an adapted version of the PRECIS tool3 that was based on the same scale and domains. Koppenaal et al10 devised an adaptation to this assessment called the Pragmascope that was simpler to use in systematic reviews. They discovered that pragmatic systematic reviews had higher average score in most domains, but lower scores in the primary analysis domain.
The difference in the primary analysis domain could be explained by the fact that the majority of pragmatic trials process their data in an intention to treat method while some explanation trials do not. The overall score was lower for pragmatic systematic reviews when the domains on organisation, flexible delivery and follow-up were merged.
It is crucial to keep in mind that a pragmatic study should not mean a low-quality trial. In fact, there is an increasing number of clinical trials that employ the word 'pragmatic,' either in their title or abstract (as defined by MEDLINE however it is not precise nor sensitive). These terms may indicate that there is a greater appreciation of pragmatism in abstracts and titles, but it's not clear whether this is reflected in the content.
Conclusions
As appreciation for the value of real-world evidence grows commonplace and pragmatic trials have gained momentum in research. They are clinical trials randomized which compare real-world treatment options instead of experimental treatments in development. They involve patients that are more similar to the patients who receive routine medical care, they utilize comparators which exist in routine practice (e.g. existing medications) and depend on the self-reporting of participants about outcomes. This approach can overcome the limitations of observational research for example, the biases associated with the reliance on volunteers, and the limited availability and codes that vary in national registers.
Other advantages of pragmatic trials include the ability to use existing data sources, and a higher chance of detecting meaningful changes than traditional trials. However, they may still have limitations which undermine their effectiveness and generalizability. For instance the rates of participation in some trials could be lower than expected due to the healthy-volunteer effect and incentives to pay or compete for participants from other research studies (e.g. industry trials). Practical trials are often restricted by the need to recruit participants on time. In addition some pragmatic trials lack controls to ensure that the observed differences are not due to biases in trial conduct.
The authors of the Pragmatic Free Trial Meta identified 48 RCTs that self-labeled themselves as pragmatist and published up to 2022. The PRECIS-2 tool was used to assess the degree of pragmatism. It includes areas such as eligibility criteria and flexibility in recruitment and adherence to intervention and follow-up. They found that 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or more) in at least one of these domains.
Trials that have a high pragmatism score tend to have more expansive eligibility criteria than traditional RCTs, which include very specific criteria that are unlikely to be present in the clinical setting, and include populations from a wide variety of hospitals. According to the authors, could make pragmatic trials more useful and useful in everyday practice. However, they don't guarantee that a trial will be free of bias. Furthermore, the pragmatism of trials is not a fixed attribute A pragmatic trial that doesn't contain all the characteristics of a explanatory trial can yield valid and useful results.
