Objective: To compare the effectiveness and efficiency of a cloud-bases system (CBS) versus traditional MS Excel® spreadsheets (MES) for data management in a Mexican private tertiary care hospital's Intensive Care Unit (ICU).

Methods: A retrospective, before-after study was conducted in an 11-bed polyvalent ICU. Data was collected for two 7-month periods: MES (June 1, 2023 - December 31, 2023), using Excel, and CBS (January 1, 2024 - July 31, 2024), using the Epimed system ®. The study evaluated data completeness, SAPS III calibration (using Hosmer-Lemeshow, Brier score, and ROC analysis), data entry time, and physicians' perception of simplicity (via a 5-point Likert scale survey). Statistical analyses included t-tests, Mann-Whitney U tests, Chi-square tests, Fisher's exact tests, and DeLong's test.

Results: A total of 409 patients were included (221 MES, 188 CBS). Missing data was prevalent in the MES period but absent in the CBS period. The Sanders-modified Brier score indicated a potential improvement in SAPS III calibration in the CBS period (0.0907 vs. 0.1514), although Hosmer-Lemeshow and DeLong tests showed no significant differences. First-year residents spent significantly less time on data entry with CBS (4.5 minutes vs. 8.9 minutes, p=0.02). Physicians perceived CBS as easier to use, with a clearer interface and better organization (p<0.05 for these three aspects).

Conclusion: The implementation of a CBS was associated with significant improvements in data completeness, potential improvement in SAPS III calibration, reduced data entry time for first-year residents, and positive perceptions of simplicity. These findings highlight the potential benefits of electronic data management systems in enhancing data quality and efficiency in critical care settings. Further research is needed to assess the long-term impact on clinical outcomes and cost-effectiveness.

Keywords: Intensive Care Unit (ICU), Electronic Health Records (EHR), Data Management, SAPS III, Epimed Monitor ICU®

The effective management of data within Intensive Care Units (ICUs) is paramount for ensuring the delivery of high-quality and safe patient care. Accurate and efficient data recording is not only fundamental for informed clinical decision-making at the bedside but also plays a crucial role in evaluating the ICU performance, driving continuous quality improvement initiatives, and supporting robust clinical research.¹ The increasing complexity of critical care, coupled with the growing volume of data generated per patient, has underscored the limitations of traditional, manual data management methods, such as those relying on Microsoft Excel® spreadsheets (MES). These manual systems are often prone to human error, can be exceedingly time-consuming, pose challenges to data security, and hinder the efficient analysis of large datasets, ultimately limiting their effectiveness in guiding quality improvement and research endeavors.^2,3 In the Latin American context, where the adoption of health information technologies may face specific challenges related to resource availability and variability in digital infrastructure, the transition to efficient electronic systems becomes even more relevant.⁴

In response to these challenges, the field has witnessed a significant shift towards the adoption of advanced electronic data management systems. One such system is the Epimed Monitor ICU®, a cloud-based system (CBS) specifically designed to streamline data collection, analysis, and reporting within the ICU environment. This platform offers among other things real-time data visualization, automated calculations of severity scores, and benchmarking capabilities.⁵

Despite the growing body of evidence supporting the use of advanced data management systems in critical care, there is still a need for reports evaluating their impact within specific healthcare contexts. This study aims to address this gap by focusing on a private tertiary care hospital in Mexico, where the implementation of electronic systems for ICU data management may face unique challenges and opportunities. The primary objective of this study is to compare the effectiveness and efficiency of a CBS for ICU performance versus the traditional use of MES for data management within the hospital's polyvalent ICU. Specifically, the study will evaluate the impact of the CBS on: (1) the completeness of data collected (missing data), (2) the calibration of the Simplified Acute Physiology Score III (SAPS-III) score for predicting hospital mortality, (3) the time spent by physicians on data entry, and (4) physicians' perception of the system's simplicity and ease of use.

Study design and setting

This study employed a retrospective, before-after design to evaluate the impact of implementing a CBS on data management within the ICU of a private academic tertiary care hospital in Mexico. The study was conducted in the 11-bed polyvalent ICU of Hospital H+ Queretaro in Mexico, which provides care for a wide range of critically ill patients.

Study periods

Data was collected and analyzed for two distinct periods: MES Period from June 1st, 2023, to December 31st, 2023. During this period, data management relied on the traditional method of manual entry into Microsoft Excel® spreadsheets. CBS Period from January 1st, 2024, to July 31st, 2024. This period followed the implementation of the Epimed Monitor ICU® system for data management within the ICU. Both periods were chosen to be of similar duration (7 months each) to ensure comparability of data and to minimize potential seasonal variations in patient population and ICU performance.

Study population

All patients admitted to the ICU during the MES and CBS periods were included in the study. Patients transferred to a different Hospital and those that died within 24 hours from admission were excluded.

Data collection

Critical Care residents (first and second year) are in charge for the data collection of every patient admitted to the ICU. All residents received a two-week course of 2-3 hours a day on how the databases were filled and the importance of the data in every period. The ultimate responsibility for veracity and completeness relies on the on-call Critical Care Staff Physician, which in all cases is a Certified Board Critical Care Physician.         

Data for the MES period were retrospectively collected from the existing ICU database, which consisted of spreadsheets maintained by the critical care residents. Information was extracted and entered a separate, anonymized database for analysis. Data for the CBS period were obtained directly from the system. The system allows for the generation of customized reports, which were created homologous to Excel spreadsheet reports into the same anonymized study database.

Patient demographic and clinical (baseline) characteristics:

Some of the baseline variables of the patients in the ICU presented for this study were patient's age in years at the time of ICU admission, sex, primary reason for ICU admission, categorized according to major diagnostic categories, intrahospital location before ICU admission, presence of pre-existing chronic medical conditions, including but not limited to hypertension, diabetes mellitus, chronic obstructive pulmonary disease (COPD), chronic kidney disease (CKD), and heart failure, SAPS III Score at admission, length of Stay in ICU and finally, ICU and Hospital Mortality.

Evaluation of effectiveness and efficiency

Missing data

The number and percentage of incomplete data fields were meticulously recorded for each patient in both the MES and CBS periods. Specific fields considered essential for this analysis included: Patient´s ID, origin, date of admission, age, sex, diagnosis, comorbidities, SAPS-III, days of stay and destination.

Calibration of SAPS-III

The predicted probability of hospital mortality was obtained directly from the SAPS III score for each patient in both periods. It is important to mention that for the MES period the Score was calculated for each patient externally in different platforms and then transcribed into the spreadsheet by each use. For the CBS, one would capture the laboratory, and the cloud-based system would automatically calculate the Score and mortality probability.

SAPS-III calibration was tested by a few methods: 1) A multiple logistic regression model including the covariates predicted mortality and electronic database system 2) Hosmer-Lemeshow Goodness-of-Fit (GOF) Test to evaluate the calibration of the SAPS-III model in each period. 3) A Sanders-modified Brier score as a measure of disagreement between the observed outcome and forecast (prediction) for each period. 4) De-Long test of AUC equality of Receiver Operating Characteristic (ROC) Curve and Area under the Curve (AUC) were calculated for each period.

Time spent on data entry

The average time (in minutes) spent by physicians on data entry per patient was recorded for both systems in a sample of 20 patients for each type of system. Time the data entry process, starting from the moment the physician began entering data for a new patient and ending when all required fields were self-reported as completed.

Perception of simplicity

A brief survey was designed to assess physicians' perceptions regarding the ease of use, interface clarity, and overall efficiency of each data management system (MES and CBS). The survey utilized a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree) for each question. The survey was carried out to the critical care residents at the end of each study period. The survey may be examined in the Supplementary material. Prior to distribution, the survey instrument underwent content validation by a panel of senior critical care physicians to ensure clarity and relevance of the questions.

Statistical analysis

Continuous variables of baseline characteristic were presented as median and interquartile range (IQR). Student's t-test, while the Mann-Whitney U test was used to compare variables between the two periods. Categorical variables were presented as frequencies and percentages. The Chi-square test or Fisher's exact test (when appropriate) was used to compare proportions between the two periods, including the percentage of missing data. For analyses involving variables with missing data, such as the SAPS-III calibration comparisons between periods, a complete case analysis approach was employed, excluding patients with missing values for the specific variables required for that analysis.

SAPS-III calibration was tested by: 1) A marginal effects plot based on predictions of a multiple logistic regression model at fixed increments of 0.1 (10%) in predicted probability of death. The model included the predicted probability of death (calculated by SAPS III) and the database system (MES vs. CBS) as covariates. The GOF test was conducted separately for each period, stratified by deciles of estimated probability of death. 2) Hosmer-Lemeshow Goodness-of-Fit (GOF) Test to evaluate the calibration of the SAPS-III model in each period. 3) The Brier Score is a loss function that assesses the performance of a prediction model by summarizing its discrimination and calibration. A lower Brier score indicates better performance, with 0 being the best possible score and 0.25 representing a non-informative model with a 50% event incidence. A Sanders-modified Brier score as a measure of disagreement between the observed outcome and forecast (prediction) for each period, is a variation of the standard Brier score used to evaluate the accuracy of binary predictions; it essentially removes the variance of the outcome from the calculation, making it more sensitive to model comparisons when dealing with different prevalence rates. 4) Receiver Operating Characteristic (ROC) Curve and Area under the Curve (AUC) were calculated for each period to assess the discriminatory ability of the SAPS-III score in predicting hospital mortality and a De-Long test of AUC equality was executed.

Time Spent on critical values entry and Simplicity Survey through Likert scale scores were compared between the MES and CBS periods using the Mann-Whitney U test.

A p-value of less than 0.05 was considered statistically significant for all analyses. Statistical analysis was performed using Stata 16.1 (StataCorp LLC, Texas, USA).

Ethical considerations

This study was approved by the Institutional Review Board (Ethics and Research Committee) of Universidad Autonoma del Estado de Queretaro, affiliated university for Critical Care program.  The study was deemed minimal risk, as it involved the retrospective analysis of anonymized, routinely collected data. Patient data confidentiality was strictly maintained throughout the study. All data were anonymized before analysis, and no individual patient identifiers were included in the study database or any reports. Informed consent was obtained from the physicians participating in the simplicity survey.

Results

Patient flow and baseline characteristics

A total of 409 patients were admitted to the ICU between June 1, 2023, and July 31, 2024. Of these, 221 were admitted during the MES period and 188 during the CBS period. No patients were excluded from the analysis. The median age of the patients was 66 years (IQR 47-78), and 235 (57.74%) were male. Patients in the CBS period were more likely to have reported comorbidities (140 [74.4%] vs. 114 [51.6%]; p<0.01), and admission from the Cardiac Catheterization Laboratory (CCL) was more frequent in this group (40 [21.2%] vs. 26 [11.8%]; p<0.01). Conversely, patients in the MES period were more often admitted from the general hospital ward (55 [24.9%] vs. 29 [15.4%]; p<0.01), and spine surgery was a more common admission diagnosis (19 [8.6%] vs. 5 [2.7%]; p=0.01). A comprehensive description of patient baseline characteristics and a comparison between groups are presented in Table 1.

Evaluation of effectiveness and efficiency

Missing data

Missing data was a prevalent issue during the MES period. All variables in the baseline characteristics dataset had at least two missing values. In contrast, no values were missing for any variable in the CBS period. The patterns of missing data, along with the number of patients with missing values for each variable, are illustrated in Figure 1.

Figure 1 Plot of patterns of missing values in MES period.

Calibration of SAPS III

The calibration of the SAPS III score was used as an indirect measure of the accuracy and efficiency of data entry in each database system. A Hosmer-Lemeshow GOF test was performed following a multivariate logistic regression model. For the MES period, which included 204 patients (17 were excluded due to missing values), a χ2 of 6.64 (p=0.57) was obtained. For the CBS period, a χ2 of 7.99 (p=0.43) was calculated. Margins were calculated from the predictions at fixed increments of 0.1 (10%) in predicted probability of death. A plot of these margins and their 95% confidence intervals (CIs) are shown in Figure 2.

Figure 2 Predictive margins with 95% Confidence interval or death prediction by predicted death probability.

The Sanders-modified Brier score, which measures the difference between a grouped forecast and the actual event and is particularly useful for comparing models across periods with potentially different event prevalences, was 0.1514 for the MES period and 0.0907 for the CBS period. Finally, ROC curves were generated for mortality prediction in each period, and a DeLong test was performed to compare the AUCs. The AUC for the MES period was 0.8933 (again with 17 exclusions), and 0.8902 for the CBS period. Although the AUC for the MES period was slightly larger, the DeLong test yielded a p-value of 0.9483, indicating no statistically significant difference between the two AUCs. A graphical representation of both ROC curves is presented in Figure 3.

Figure 3 ROC curves for death prediction for both periods.

Time spent on data entry

The median time spent filling in critical data values was not statistically different between the two systems: 6 minutes (IQR 4-7.5) for MES vs. 5 minutes (IQR 3-7) for CBS (p=0.43).  However, when the analysis was stratified by the residents’ year of training, a statistically significant difference was observed among first-year residents. For this group, the median time spent using MES was 8.9 minutes (IQR 6-11), compared to 4.5 minutes (IQR 4-7) using CBS (p=0.02).  No statistically significant difference was found among second-year residents, with a median time of 4 minutes (IQR 3-4) for MES and 5 minutes (IQR 3-7) for CBS (p=0.21). A multiple linear regression analysis, which included both the database system and the year of residency as independent variables, did not find the database system to be independently associated with the time spent on data entry. However, the year of residency was independently associated with the time spent on data entry (data not shown). A box plot illustrating these results is presented in Figure 4.

Figure 4 Box plot of time spent in minutes in filling critical data values.

Perception of simplicity

The CBS period was associated with significantly higher scores on three specific questions related to the perception of simplicity. First, for the statement "Overall, the system was easy to use," the median score in the CBS period was 5 (IQR 4.5-5) compared to 3.5 (IQR 2.5-4) in the MES period (p=0.03). Second, the statement "The system's interface was clear and intuitive," the median score was 5 (IQR 4.5-5) in the CBS period versus 2 (IQR 2-3) in the MES period (p=0.02). Finally, for the statement "The system's organization was clear and logical," the median score was 4 (IQR 3.5-4) in the CBS period compared to 2.5 (IQR 1.5-3) in the MES period (p=0.03). The remaining two survey questions, including the overall assessment, "Overall, I was satisfied with the system," did not show statistically significant differences between the two periods. A box plot illustrating these results is presented in Figure 5.

Figure 5 Box plot of each question for simplicity perception.

Interpretations of results

Main findings

This study aimed to compare the effectiveness and efficiency of a CBS against the traditional use of MES for data management within the ICU of a private academic tertiary care hospital in Mexico. The findings demonstrate that the implementation of a CBS was associated with significant improvements in data completeness, a potential improvement in the calibration of the SAPS III score due to better compliance in filling critical values, a reduction in data entry time for first-year residents and a perceived increase in simplicity of use, interface clarity and system organization, highlighting the potential benefits of adopting advanced electronic data management systems in critical care settings.

Comparison with previous studies

One of the most striking findings was the complete absence of missing data in the CBS period, a stark contrast to the MES period where some variables had at least two missing values. This substantial improvement in data completeness can be directly attributed to the inherent features of the CBS, which likely includes data validation rules, mandatory fields, and automated prompts that minimize the possibility of incomplete entries. These findings are consistent with previous studies that have reported similar reductions in missing data following the implementation of electronic data management systems in ICUs.^5,6 Automated methods like the Critical care Database for Advanced Research (CEDAR) have been developed to support ICUs by automating data extraction from EHR systems, which can reduce the need for manual data collection and improve data quality.⁷ Electronic information systems have been shown to encourage more detailed documentation compared to handwritten records, capturing specific data elements that might otherwise be omitted, thus enhancing data completeness.⁸ Complete and accurate data are crucial for reliable performance evaluations, accurate severity-of-illness scoring, and valid benchmarking comparisons. However, it is important to acknowledge that while the CBS significantly improved data quality in terms of completeness, this study did not directly evaluate the impact of this improvement on patient clinical outcomes, such as mortality or length of stay. By minimizing missing data, the CBS has the potential to enhance the quality of care and patient safety.

Regarding the calibration of the SAPS III score, although the Hosmer-Lemeshow test and the DeLong test for comparing AUCs did not show statistically significant differences between the two periods, the lower Sanders-modified Brier score in the CBS period suggests a potential improvement in the overall accuracy of mortality predictions. This finding is particularly noteworthy because accurate risk prediction is essential for appropriate resource allocation, informed clinical decision-making, and valid comparisons of ICU performance. The improved Brier score in the CBS period could be attributed to the more complete and accurate data captured by the system, as well as to the system's automated calculation of the SAPS III score, which eliminates the potential for human error during manual calculation. These results align with previous research demonstrating that electronic systems can improve the accuracy of severity-of-illness scoring. One study developed the Philips Critical Care Outcome Prediction Model (CCOPM), which uses automated data capture to improve ICU mortality risk prediction. This model outperformed traditional models like APACHE IVa/IVb in terms of discrimination and calibration, suggesting that automated systems can enhance the accuracy of severity scoring by mitigating biases introduced through manual documentation.[9] Another study compared the Rothman Index (RI), an EHR-derived risk score, with the Sequential Organ Failure Assessment (SOFA) score in a cardiac ICU setting. While the SOFA score had superior discrimination, the RI demonstrated better calibration, indicating that EHR-derived scores can provide valuable prognostic information and improve resource allocation.¹⁰ High-frequency data collection through electronic systems has also been shown to improve mortality models in the ICU. By incorporating high-frequency physiological data, models demonstrated better discrimination and accuracy compared to traditional scoring systems, highlighting the potential of electronic systems to enhance severity-of-illness scoring.¹¹

In terms of data entry time, the study found that while there was no overall difference between the systems, first-year residents spent significantly less time entering data using the CBS. This difference may be attributed to the system's more intuitive interface, automated calculations, and built-in data validation features, which may be particularly beneficial for less experienced users. This finding suggests that the CBS could potentially reduce the workload and improve the efficiency of junior physicians, allowing them to dedicate more time to direct patient care. While second-year residents did not show a difference between the systems, this could be because they were already proficient in using MES due to their greater experience, and had less familiarity with the CBS, since it was a recently implemented system.

Furthermore, the survey results indicated that physicians perceived the CBS as easier to use, having a clearer interface, and better organization compared to MES. It is plausible that the structured training sessions provided prior to the CBS period contributed positively to these perceptions of simplicity, particularly among less experienced residents. This enhanced usability could contribute to increased user satisfaction, reduced errors, and improved data quality. However, it is worth noting that the overall satisfaction with the system did not differ significantly between the two periods. This could reflect a common challenge in clinical settings: initial user resistance or the adaptation period required when implementing new technologies, even those perceived as more intuitive. The inherent learning curve associated with any new system might temper overall satisfaction initially. Studies comparing manual and electronic data collection from integrated critical care EMR systems have found excellent concordance between the two, with electronic systems providing reliable and efficient data capture for quality improvement and research purposes.¹² This could be due to the inherent resistance to change that often accompanies the adoption of new technologies, the learning curve associated with the CBS, or other factors not captured by the survey questions.

Limitations and strengths

This study has several limitations that should be considered when interpreting the findings. First, the before-after study design is susceptible to temporal trends and confounding factors that may have influenced the results. Although efforts were made to minimize potential biases by selecting similar study periods, it is possible that other changes in ICU practice or patient population occurred during the study period that could have affected the outcomes. Second, the study was conducted in a single center, which may limit the generalizability of the findings to other ICUs with different patient populations, staffing models, or organizational structures. Third, the relatively short study period may not have been sufficient to fully capture the long-term impact of the CBS, particularly regarding user adaptation and learning curve effects. Fourth, the reliance on self-reported data for the time spent on data entry and the perception of simplicity could introduce recall bias or social desirability bias. Lastly, the baseline characteristics showed differences between both periods, with a higher proportion of patients with comorbidities and patients from cardiac catheterization laboratory in the CBS period, potentially acting as a confounding variable.

Future research

Future research should aim to address these limitations by conducting prospective, randomized controlled trials in multiple centers with longer follow-up periods. Additionally, future studies could explore the cost-effectiveness of the CBS compared to traditional methods, considering not only the direct costs of implementation and maintenance but also the potential cost savings associated with improved data quality, reduced errors, and enhanced efficiency. Further investigation into the specific features of the CBS that contribute to improved data completeness, accuracy, and user satisfaction would also be beneficial. Benchmarking, the process of comparing one's performance against that of similar units or institutions, the usefulness as a tool for improving quality in critical care should be explored.^13,14 Finally, assessing the impact of the system on clinical outcomes, such as length of stay, mortality, and adverse events, would provide a more comprehensive evaluation of its overall value.

In conclusion, this study provides evidence that the implementation of a CBS in a private academic tertiary care hospital in Mexico was primarily associated with significant operational improvements: enhanced data completeness, reduced data entry time for first-year residents, and a perceived increase in system simplicity, usability, and organization. Although a potential improvement in the calibration of the SAPS III score was noted, the direct benefits observed in this study relate fundamentally to data management quality and efficiency. These findings underscore the potential of advanced electronic data management systems to address critical challenges in ICU data handling. While this study did not directly measure changes in patient clinical outcomes, the observed enhancements in data accuracy, completeness, and workflow efficiency are foundational steps that can pave the way for future improvements in the quality and safety of care delivered to critically ill patients. Further research is needed to confirm these findings, evaluate the long-term impact of the CBS on specific clinical outcomes and cost-effectiveness, and explore its utility in benchmarking efforts. Nevertheless, this study contributes valuable insights into the real-world application of electronic data management systems in a Mexican ICU context, suggesting the CBS holds considerable promise as a tool to support better critical care practices.

We would like to thank the Critical Care Medicine residents in Hospital H+ Querétaro for their determination and help.

Dr. Rodríguez-Guillén is an indirect consultant of Epimed Solutions ®; Dr. Zaragoza is a member of the Directive Board of Critical Care College of Mexico, which has an agreement with Epimed Solutions, he has not received any fees. All other authors declare no potential conflicts of interest exist.

1. Vincent J-L, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14(2):207.
2. Johnson AE, Stone DJ, Celi LA, et al. The MIMIC Code Repository: enabling reproducibility in critical care research. J Am Med Inform Assoc. 2018;25:32-39.
3. Breslow MJ, Rosenfeld BA, Doerfler M, et al. Effect of a multiple-site intensive care unit telemedicine program on clinical and economic outcomes: an alternative paradigm for intensivist staffing. Crit Care Med. 2004;32:31-38.
4. Estenssoro E, Alegría L, Murias G, et al. Organizational issues, structure, and processes of care in 257 ICUs in Latin America: a study from the Latin America Intensive Care Network. Crit Care Med. 2017;45:1325-1336.
5. Zampieri FG, Soares M, Borges LP, et al. The Epimed Monitor ICU Database®: a cloud-based national registry for adult intensive care unit patients in Brazil. Rev Bras Ter Intensiva. 2017;29:418-426.
6. Soares M, Borges LP, Bastos LDSL, et al. Update on the Epimed Monitor Adult ICU Database: 15 years of its use in national registries, quality improvement initiatives and clinical research. Crit Care Sci. 2024;36:e20240150en.
7. Schenck EJ, Hoffman KL, Cusick M, et al. Critical care Database for Advanced Research (CEDAR): an automated method to support intensive care units with electronic health record data. J Biomed Inform. 2021;118:103789.
8. Apkon M, Singhaviranon P. Impact of an electronic information system on physician workflow and data collection in the intensive care unit. Intensive Care Med. 2001;27:122-130.
9. Liu X, Armaignac DL, Becker C, et al. Improving ICU risk predictive models through automation designed for resiliency against documentation bias. Crit Care Med. 2023;51:376-387.
10. Kunitomo Y, Thomas A, Chouairi F, et al. Electronic health record risk score provides earlier prognostication of clinical outcomes in patients admitted to the cardiac intensive care unit. Am Heart J. 2021;238:85-88.
11. Todd J, Gepp A, Richards B, et al. Improving mortality models in the ICU with high-frequency data. Int J Med Inform. 2019;129:318-323.
12. Brundin-Mather R, Soo A, Zuege DJ, et al. Secondary EMR data for quality improvement and research: a comparison of manual and electronic data collection from an integrated critical care electronic medical record system. J Crit Care. 2018;47:295-301.
13. Woodhouse D, Berg M, van der Putten J, et al. Will benchmarking ICUs improve outcome? Curr Opin Crit Care. 2009;15(5):450-455.
14. Salluh JIF, Chiche JD, Reis CE, et al. New perspectives to improve critical care benchmarking. Ann Intensive Care. 2018;8(1):17.