Last week I wrote about the first phase of system evaluation:  project.  This week I will write about the second phase:  strategic.  As an analyst, we are busy building the computerized medical system that I don’t even think about  if the users are happy with what I delivered to them.  I created an intake and output flowsheet for the Intensive Care Unit. My questions are is the electronic version of the intake and output flowsheet easier for the nurses than the paper version? How long does it take them to document the patient’s I’s and O’s?

Outcomes are evaluated to determine:  financial investment, resource investment, determine if the system did was it was intended to, and meaningful use reporting.  Meaningful use reporting is required now.  Providers must report their performance on two types of measures:  functional and interopertability, which relates to the useage of electronic health functionality and clinical quality measures, which relates to the quality of care provided to patients.

There are eight evaluation categories:

• clinical outcomes
• financial outcomes
• research outcomes
• adoption
• user satisfaction
• workflow impact measures
• productivity
• patient safety and quality

Dr. Sendstack shared her experience and research study from the National Institute of Health (NIH).  Her team consisted of three people:  nursing informatics specialist, senior clinical analyst, and herself, the Deputy CIO.  The three of them conducted a brain storming session and utilized resources such as Agency for Healthcare Research and Quality (AHRQ) Health Information Technology Evaluation Toolkit (2009) and STARE-HI (2008) - Statement on reporting of evaluation studies in Health Informatics.  The first item on the agenda that Dr. Sengstack accomplished was compile a list of questions such as “Did the medication reconciliation module result in less omissions of patients’ home medications?” or “Did the alert to stop inaccurate entry of weights work?”  Determining the question is the beginning of the evaluation process.

The second step in this research process is prioritizing outcome evaluations:

• organizational mission
• input from key stakeholders
• extent of potential benefit
• ease of access to data
• complexity of design

Dr. Sengstack recommended keeping the research study design:

• simple
• retrospective using these methods:  chart review, data query, survey, and focus group
• prospective:  time-motion/observational and pre-post study

The next step in the research study is data collection and analysis.  In this step, think about what data will be collected, who will collect it, how will it be collected, what date range will be used, and what will you do once the data is collected.

Additionally, documentation is key in a research study.  The majority of research studies use the following example to document the findings.

• Abstract
• Introduction and background
• Methods
• Results
• Discussion
• Conclusion

 The following example is the one used in the webinar by Dr. Sengstack.  The question was Does the current ICU flowsheet provide accurate calculations of intake and output?

The research study had 30 ICU patients, paper flowsheets were reviewed and electronically re-calculated.  The findings were discrepancies in calculations were found 63% of the time.

If you decide to evaluate your current system and have collected data, your next question may be so what do I do with the data?  You may want to follow up with key stakeholders in your institution, distribute or post the completed document within your institution and lastly, determine how to improve the practice in your institution.

The lessons learned from Dr. Sengstack and her staff were

1.)  Identifying the questions to ask the clinicians were challenging

2.)  Involvement of key stakeholders

3.)  Determining how to define the problem so that it can be measured was also challenging

4.)  Stakeholders did not know what to do with the data

5.)  Strategic evaluation of a clinical system can be done with only three people

6.)  You do not need a grant to evaluate the outcomes of a clinical system

After reviewing this webinar, it had me think about my own institution and how we have had a clinical system for the past 35 years. My questions are has anyone evaluated the effectiveness of the clinical system in the nursing department.  Many clincians come to us to fix the ease of use for them, but is it really working? Are all of the alerts that they ask for helping them and if it is helping them in what way?