Question

categorize the problems faced in evaluation of health information systems into three areas: the complexity of the object of evaluation, the complexity of the evaluation project, and the motivation to perform evaluation and a framework to help address these challenges

Answer 1

First Problem Area: Complexity of the evaluation object

When understanding IT as part of the IS of an organization, it is clear that evaluation requires not only an understanding of computer technology, but also of the social and behavioral processes that affect and are affected by the technology. This complexity of the evaluation objects has some important consequences. First, the introduction of IT takes time. It is not enough to implement the technology and then to immediately measure the effects. Users and workflow need a lot of time to get used to new tools and to completely exploit the new possibilities (Palvia, Sharma, & Conrath, 2001). Thus, evaluation results can develop and change during this first period of use. Then, even after an introduction period, the evaluation object may steadily change (Moehr, 2002; moving evaluation target). For example, the use of IT may be affected by changes in work organization, or in staff. It is nearly impossible to reach a stable situation in a flexible healthcare environment which makes evaluation results dependant of the point in time where the evaluation took place. In addition, each IS in our definition is quite unique. While the IT may be similar in various departments, workflow, users and used functionality may be different. In addition, the organization of its introduction as well as the overall user motivation may differ. Thus, even when the same IT is introduced, its effects may be varying (Kaplan & Shaw, 2002). The influence of such factors on the results of an evaluation study is often hard to disentangle (Wyatt, 1994), posing the problem of external validity (Moehr, 2002): Many evaluation studies may be valid only for the particular institutions with their specific IS.

The complexity of the evaluation object is an inherent attribute in healthcare IT evaluation and cannot be reduced. However, there are some ways to handle this problem in evaluation studies. To address the problem of external validity, the IT and its environment that is going to be evaluated should be defined in detail before the beginning of the study. Not only the software and hardware used should be described, but also the number of users and their experience and motivation, the way IT is introduced and used, the general technical infrastructure (e.g., networks) and any further aspects that may influence the usage of IT and its effects. The functionality and the way it is really used should also be of importance. Only this information may allow interpretation of the study results and comparison of different locations. Then, to address the problem of the moving evaluation target, all changes in the IT and its interaction with the users should be carefully documented during the study. For example, change s in workflow, in staffing, or in hardware or software should be documented with reference to the ongoing evaluation. This permits the explanation of changes and differences in effects measured during the study period. Another approach to address the problem of the moving evaluation target may be to define smaller evaluation modules. This would allow the evaluation design or evaluation questions to be adapted to changes in the environment. Each module answered a question related to a defined phase of the introduction of the IT. In addition, an evaluation must be planned in a long-term perspective in order to allow the users and the environment to integrate the new IT. Hence enough resources for long-term evaluation (e.g., over several months or even years) should be available.

second problem Area: complexity of the Evaluation Project

Evaluation of IT is performed in the real and complex healthcare environment, with its different professional groups, and its high dependency on external influences such as legislation, economic constraints, or patient clientele. This poses problems to the evaluation projects, meaning the planning, executing and analyzing of an IT evaluation study. For example, the different stakeholders often have different conceptions and views of successful IT (Palvia et al., 2001). The different stakeholder requirements can serve as a frame of reference for evaluation during the early phases of the IT life cycle, but also guide evaluations during later phases. In each case, multiple-stakeholder views may lead to a multitude of (possibly conflicting) evaluation questions (Heathfield et al., 1999).

Depending on the point of view adopted, the evaluation will require different study designs and evaluation methods. The evaluation researcher must decide, for example, on the evaluation approach, on the adequate evaluation methods (e.g., quantitative vs. qualitative), and on the study design (e.g., RCT vs. observational study). Each has its own advantages and drawbacks ( Frech-tling, 1997; Moehr, 2002), making their selection a rather challenging endeavor. This multitude of possible evaluation questions and available evaluation methods makes the planning of an evaluation study quite complex.

The complexity of the evaluation project has several consequences. First, the overall success of IT is elusive to define (Palvia et al., 2001), and it is therefore often difficult to establish clear-cut evaluation criteria to be addressed in a study (Wyatt, 1994). Each stakeholder group may have individual questions, and a universal evaluation in terms of absolute or relative benefits is usually not feasible (or, from a more subjectivistic point of view, not even possible). It is also unrealistic to expect that the IT itself will have a direct and easy to measure effect on the outcome quality of patient care, like in a drug trial (Wyatt, 1994). Thus, indirect measures are often used such as user satisfaction or changes of clinical processes, which, however, do not give a really complete picture of the benefits of IT. Often, changes in the evaluation questions may occur during the study (e.g., based on intermediate evaluation results, new insights, changes in stakeholders’ opinions, or changes of the IT [scope creep]; Dewan & Lorenzi, 2000). Changes in study questions, however, may be difficult to balance with study resources. Finally, the selection of adequate evaluation designs and evaluation methods is often regarded as a problem during evaluation studies. Evaluators may not be sufficiently aware of the broadness of available approaches, or be too deeply embedded in either the qualitative or the quantitative paradigm, neglecting the possible contributions of the complementary approach. Thus, inadequate methods or study designs may be chosen which may not be able to answer the original study questions.

The following suggestions may be useful in order to deal with the complexity of the evaluation project. First, it is recommended that the general intention of the evaluation and the starting point should be agreed early on. In principle, evaluation should start before the new IT is implemented, in order to allow for early gathering of comparative data, and then continue during all phases of its life cycle (VATAM, 2000). Then, the areas of evaluation should be restricted to aspects which are of most importance to the involved stakeholders, and which can be measured with the available resources. A complete evaluation of all aspects of a system (such as economics, effectiveness, and acceptance) is usually not feasible. A balance between the resources of a study and the inclusion of the most relevant aspects has to be found. In addition, sufficient time should be invested into the definition of relevant study questions. All involved stakeholder groups should discuss and agree on the goals of evaluation (VATAM, 2000). The selected study questions should be relevant for decision-making with regard to introduction, operation or justification of IT (Ammenwerth et al., 2004). Conflicting goals should be discussed and solved, as they are not only problematic for an evaluation, but for the overall management of new IT. Fourth, when new evaluation questions emerge during the study, they should only be included in the study design when it is possible without creating problems. Otherwise, they should be tackled in consecutive studies. Each shift in evaluation questions must thoroughly be documented. For each study question, adequate methods must be chosen. A triangulation of methods may be useful to best answer the study questions (Heathfield, Pitty, & Hanka, 1998). For example, to address the effects of a nursing documentation system, both quantitative methods (time measurement, user acceptance scales, documentation quality measurement) as well as qualitative methods (focus group interviews) were used. We will discuss this example later on in more detail.

Third Problem Area: Motivation for Evaluation

An evaluation study can normally only be conducted when there is sufficient funding, and a sufficient number of participants (e.g., staff members, wards). Both these variables depend on the motivation of stakeholders (e.g., hospital management) to perform an evaluation. Sometimes, this motivation is not very high, because, for example, of fear for negative outcome, or of fear for revealing deficiencies of already implemented technology (Rigby, Forsstrom, Roberts, & Wyatt, 2001). In addition, the introduction of IT in an organization is a deep intervention that may have large consequences. It is thus often very difficult to organize IT evaluation in the form of an experiment, and to easily remove the system again at the end of the study in case the evaluation was too negative.

Even with a motivated management, it may be difficult to find suitable participants. Participating in a study usually requires some effort from the involved staff. In addition, while the users have to make large efforts to learn and use a new, innovative system, the benefit of joining a pilot study is usually not obvious (the study is conducted in order to investigate possible effects), but participation may even include some risks for the involved staff such as disturbances in workflow. In summary, due to the given reasons, the hospital management as well as involved staff members is often reluctant to participate in IT evaluation studies.

The described problem has consequences for the study. Without the support and motivation of the stakeholders to conduct an evaluation study, it will be difficult to get sufficient resources for an evaluation and sufficient participants willing to participate. Second, due to the given problems, the study organizer tends to recruit any participant who volunteers to participate. However, those participants may be more motivated to participate than the “normal” user. This leads to the well-known volunteer effect, where results are better when participants are motivated. In addition, evaluation results are not only important for the involved units, but also for the overall organization or for similar units in other organizations. To allow transfer of results, the pilot wards or pilot users must be sufficiently representative for other wards or users. But, as each IT within its environment is quite unique (see Problem Area 1); it is difficult to find comparable or representative participants.

To increase the number of participants, two approaches should be combined. First, the responsible management should be informed and motivated to support the study. The result of an evaluation study may be important to decide on new IT, and to support its continuous improvement. Then, the possible participants could be directly addressed. It should be made clear that the study provide s the opportunity to influence not only the future development of IT in healthcare but also the own working environment. User feedback of study results may act as an important driving force for users to participate in the study. Offering financial compensation or additional staff for the study period may help to gain support from participants and from management. As in clinicaltrials, multicentric studies should be considered (Wyatt & Spiegelhalter, 1992). This would largely increase the number of available participants. This means however, that study management requires much more effort. A multicentric study design is difficult when the environment is completely different. In addition, the variation between study participants will be bigger in multicentric trials than in single-center ones. This may render interpretation and comparison of results even more difficult (cp. discussion in Problem Area 1)