Question

For questions 1 - 4, please consider the following [fictitious] scenario.

Lehigh Valley Health Network, a not-for-profit health system with 8 hospitals, is considering switching to a project scoring methodology to evaluate project potential for capital allocation purposes. The CEO of the health system, Dr. Brian Nester, believes that such a methodology would help the leadership team get a more complete picture of the implications of different projects, beyond purely financial measures. The goal for any decision support methodology such as this one is to provide a useful tool for decision makers that leads to good decisions consistently. Dr. Nester has hired an analyst (you) to help them out with developing this methodology.

Q1 [10 points].

Propose a set of criteria that the LVHN team could use to evaluate projects for capital allocation. Organize your criteria in appropriate categories or buckets (e.g. you may consider a subset of operational impact criteria, a subset of financial criteria, community impact etc.). Finally, for each criterion in each bucket, provide a 1-3 sentence rationale behind including it in your selection of criteria to be used for evaluation.

Some (hopefully helpful) hints:

• To identify relevant criteria, you may wish to consider the mission and values of the organization, the different stakeholders that may be part of the LVHN community, the various aspects of the organization that may be impacted etc.
• Too many criteria creates a situation where any meaningful comparison becomes difficult; on the other hand, too few criteria may not provide sufficient “resolution” to be able to differentiate between alternative projects. Unfortunately, there is no magic number - hence the need to provide a rationale and think carefully about what criteria should be included. In general, you want to aim for the fewest that you believe capture all relevant dimensions of analysis.
• Finally, for a given criterion, consider the “ability to measure” how well a given project can satisfy it - in the end, a criterion for which we cannot determine how well a project satisfies it is not that useful. Note that it is possible for a specific criterion not to be itself measurable, e.g. “the project increases the level of confidence that the community has in LVHN services” - how would someone measure that? It is, however, possible to see some measurable effects of this, for instance an increase in “second opinion requests” addressed to LVHN physicians (which we could assume to mean that people trust LVHN more and therefore come to LVHN for second opinions).

[Bonus] 0.5 points for each specific metric that you can use to evaluate how a criterion is impacted by a given project - e.g. “increase in # second opinion requests can be used to measure the improvement in community trust in LVHN physicians”.

Answer 1

Different Criteria we should take into account while evaluating a project before putting it into action.

Normally, Projects are analysed on the basis of six key criteria (relevance, coherence, effectiveness, efficiency, overarching developmental impact and sustainability.

1. The criterion of relevance is used to assess whether the project fulfils an important function from a development perspective ("priority"), and whether its design was fundamentally suited to achieving the goals associated with the project ("validity of the results chain").

2.The criterion of coherence refers to the compatibility of the development policy measure taken with other interventions and policy objectives in the partner hospital and/or sector. This means that the effects of DC interventions are considered in a more systemic and context-coherent manner, distinguishing between internal and external coherence. While internal coherence focuses more on the consistency of the formulated intervention with the policy objectives of the same institution/government, external coherence addresses the complementarity of the formulated intervention with interventions of other actors. External coherence should take greater account of the coordination between different actors and thus emphasize the questions of synergies and duplication of interventions.

3. The criterion of effectiveness is used to assess whether a development project achieved its goals. In order to allow a meaningful comparison of the targeted and actual outcomes, the project's goals should be expressed in the form of quantifiable levels of production or consumption. Any unintended positive or negative effects that can be observed are also included in the evaluation.

4. The criterion of efficiency is used to assess a project's cost-effectiveness. The central issue here is the economical use of resources. The evaluation is carried out on two different levels. On the one hand, an assessment is made of whether the effort required to provide goods or services was appropriate ("efficiency of production"). However, even more important is what we call "allocation efficiency" – achieving an adequate ratio between the funds used and the effects achieved. This involves looking into what other methods were available that could have achieved similar results.

5. In addition to a project's direct goals, there is also the overarching developmental impact, the big objectives that are the reason why the decision was made to promote the project in the first place – for example the impact on health in the case of improving the water supply. It is always not possible to measure overarching impacts. In such cases it is necessary to check their plausibility and estimate them using circumstantial evidence.

6. Our aim is to achieve not only short-term improvements but also sustainable results. For this reason we investigate whether any improvements are likely to endure. We consider the criterion of sustainability to have been met if the project-executing agency or target group is in a position to successfully continue the promoted project once the external financial or technical support has been withdrawn.

7.

The key criteria are initially assessed on the basis of a six-point rating scale. Scores of 1 to 3 indicate a "successful" project, while 4 to 6 are "unsuccessful". For the criterion of sustainability we only use a four-point scale, which mainly reflects the anticipated future trend (albeit with a certain degree of uncertainty). A score of 4 indicates "insufficient sustainability".

The partial scores for the six key criteria are combined in accordance with a specific weighting that is defined for each project in order to give an overall score. This overall rating indicates at a glance whether a project has been successful, and to what extent.

It should be noted that a project can generally be considered developmentally “successful” only if the achievement of the project objective (“effectiveness”), the impact on the overall objective (“overarching developmental impact”) and the sustainability are rated at least “satisfactory” (level 3).

The project evaluation and selection process involves the use of quantitative and qualitative analyses to identify and address crtitical aspects of the design basis, scope of work and other contract deliverables.

Quantitative analyses deal with project economics and support judgment-based selection decisions when the data is quantifiable and measurable. Quantitative methods enable bid ranking and subsequent financial number crunching.

Criteria to be taken into account in the case of  LVHN team are as follows-

Health, Environmental and Safety (HES) Risk Decision Matrix

Although not conventionally included among project selection techniques, occupational health and safety and environmental protection have risen to a position of prominence in project concept selection. This is because a catastrophic incident during the project’s lifecycle could significantly impact economic viability and the organization’s reputation and market capitalization. A variation of the decision matrix described above, the HES risk matrix involves the use of a multidisciplinary team in identifying major accident scenarios and assessing the likelihood and severity of their consequences. The project alternative is evaluated with respect to each accident scenario and the residual risk is estimated after considering the prevention and mitigation controls in place. The sum total of the risk ratings enables the ranking of alternatives in terms of HES risk.

Cost Effectiveness

Also referred to as economic efficiency, the Cost Effectiveness method of project evaluation is used where the NPV or lifecycle cost cannot be justified as overriding project criteria, as in the case of safety, project performance, availability and reliability. Remer and Nieto (1995) suggested a cost effectiveness methodology that involves evaluating the alternatives against those criteria, using a fixed-cost or fixed-effectiveness approach. A sensitivity analysis is then conducted by manipulating the cost and effectiveness criteria to determine the most optimal cost/effectiveness ratio for project ranking and selection.

Hoskold’s Rate of Return (HRR)

Hoskold’s method is a quantitative technique that uses annual worth (AW), annual deposit to a sinking fund (dp) and replacement value (RV) at the end of the project lifecycle in ‘n’ number of years. Mathematically, HRR = (AW – dp)/RV; where dp = IA (A/F, is, n), I being the initial investment and is, the interest received on the sinking fund deposit (assuming it equals the Minimum Attractive Rate of Return). By calculating the HRR for the peroject alternatives, it is possible to rank them in order of preference.

Organization Based Information Architecture (OBIA)

OBIA was proposed by Messner & Sanvido (2001) as a framework for collecting and organizing the information needed for project evaluation. OBIA is divided into five categories.

1. Organization: the evaluating organization, owners, potential competitors, other organizations related to the project and joint venture partners.
2. Commitment: contractual agreements including scope, compensation, time, legal issues and selection criteria.
3. Process: management processes and facility processes, including strategic planning, acquisitions and managing commitments.
4. Environment: physical, legal, cultural, economic and resource environment regaring the process, facility or organization.
5. Facility: the infrastructure and buildings, including land, utilities, site equipment and technical, architectural and structural systems.

Each project alternative is evaluated using a checklist to verify conformity to the desirable specifications for the project and the percentage conformity is evaluated and ranked for comparison.

Project success is assured when the project evaluation criteria is measurable and the evaluation process is structured and meticulously implemented. Project selection decisions, on the other hand, should be compatible with the organization’s mission and business strategy. Sound selection decisions provide assurance that budgetary, time and technical requirements will be met, ensure project economics and profitability are achievable and foster success in the operating environment.