Question

An short essay "why it is important to evaluate the adequacy of development activities by assessing: (a) the adequacy of project request and approval procedures; (b) the adequacy of feasibility studies; (c) the adequacy of, and adherence to, standards and procedures relating to the: (i) design phase; (ii) development phase; (iii) testing phase; and (iv) implementation phase; (d) the adequacy of project change controls; (e) the appropriate inclusion of organizational personnel throughout the project's life cycle; (f) the effectiveness of project communication and reporting procedures; and (g) the accuracy, effectiveness, and control of project management tools.

The resource:

• State the audit objective of the procedure, which can found in the FFIEC development and acquisition booklet at the end of chapter in Appendix A. Locate the page number(s) in the FFIEC development and acquisition booklet where the underlying topics for the procedures are covered and record the page number(s). This should be the support for the claims you make in your essay. You should show me that you have read the booklet by your response. Please refrain from expressing your opinion unless you have significant experience in the field.

• State and explain why this procedure is being performed, including a discussion of the risks/issues that the procedure is designed to address. If the procedures addresses such risks/issues as appropriate policies or internal controls established by an auditee, explain the significance of lack of compliance. Remember to review the pages in the FFIEC development and acquisition booklet before answering this question.

Answer 1

Project Planning Standards

Organizations should establish appropriate project planning standards. The standards should require management to develop project plans that are detailed in proportion to a project's characteristics and risks. Management should develop well-defined plans for all projects.

Project plans should describe existing system benefits and weaknesses, explain project goals, and identify user, information, system, and network requirements. Such explanations and descriptions enhance team members' abilities to understand project objectives and develop systems that meet organizational needs. Project plans should identify quality assurance procedures; risk management procedures, including security features which will be needed; testing procedures; and documentation procedures. Additionally, project plans should detail cost, staffing, resource, and training requirements.

Configuration Management Standards

Organizations should establish configuration management standards. Configuration management involves controlling project component changes in order to minimize project disruptions and ensure original objectives are addressed. Configuration management standards should require the identification of baseline configurations (original versions) of hardware, software, services, documentation, and project management plans. Standards should also be in place to ensure all changes to baseline versions are evaluated, approved, documented, and disseminated. Refer to the "Maintenance" section for additional configuration management details.

Quality Assurance Standards

Quality assurance is a critical part of well-managed development and acquisition projects. Comprehensive quality assurance, risk management, and testing standards provide the best means to manage project risks and ensure software includes expected functionality, security, and operability. Quality assurance procedures should be applied to internally and externally developed programs.

Management should establish quality assurance standards that address:

Commitment - Successful projects require commitment from all involved parties. Senior managers should adequately support and promote projects throughout an organization to enhance organizational acceptance of a project. End users should assist in defining and testing functional requirements and project teams should efficiently complete tasks. All parties should clearly define their expectations and effectively communicate throughout a project. Management's failure to implement or support quality assurance programs decreases an organization's ability to detect project weaknesses and programming errors quickly. The later weaknesses and errors are detected, the more difficult and costly they are to correct.

Completeness - Each phase within a project life cycle includes procedures to follow and items to deliver. Therefore, quality assurance programs should parallel all phases of the life cycle. For example, the project initiation phase includes the presentation of a business case, the request for desired functional requirements, and the identification of interconnected system components. Quality assurance personnel should verify the justification for a project, the necessity of requested functional features, and the accuracy of system connections before projects move into the project planning phase. Audit and compliance employees should assist quality assurance personnel verify compliance with internal and external project requirements.

Scalability - Projects vary in size and complexity. Quality assurance standards should match project characteristics and risks.

Measurability - Organizations cannot evaluate a project's success accurately unless they assess results against defined expectations. Therefore, quality assurance personnel should assess the quality of products and processes against measurable standards, metrics, and expectations.

Tracking - Project personnel should properly record, report, and monitor problems to ensure effective problem resolution.

Independence - Audit and quality assurance personnel should be independent of the project they are reviewing

Risk Management Standards

Organizations should establish risk management standards and procedures for all complex or mission-critical projects. Risk management activities, which are sometimes included within quality assurance programs, should include procedures for identifying and managing internal and external project risks.

The procedures should be designed to ensure internal and external project risks are identified and assessed, reported and monitored, and appropriately managed. Managing identified risks requires organizations to develop strategies regarding risk acceptance, mitigation, transfer, or a combination of these. For example, a mitigation strategy to address risks involved with initiating projects having a high number of functional requirements might be to review the project scope and eliminate any unnecessary functional requirements. Transferring an identified risk, such as natural disasters, might be accomplished by acquiring insurance policies.

Testing Standards

Management should establish testing standards that require the use of predefined, comprehensive test plans, end-user involvement, and documented test results. Additionally, testing standards should prohibit testing in production environments or with live data. If copies of live (customer) data are used during tests management should ensure that appropriate standards exist to protect the confidentiality of that data. Management can use test data generators, which are software applications that generate representative testing data based upon predefined parameters, to develop appropriate testing data. Numerous automated applications are also available that test program logic, functional operability, and network interoperability.

Documentation Standards

Organizations should establish appropriate documentation standards. Documentation consists of detailed descriptions and explanations of technology applications, systems, and procedures. Documentation enhances a user's ability to use, review, or modify the applications, systems, and procedures. Management should maintain documentation for all technology resources, including nontechnical policy and procedural guidance, and technical information such as hardware and software configurations, and system and application source codes. The quality and quantity of the documentation should be commensurate with the characteristics and risks of the associated resource. For example, high risk applications should be more formally documented than applications that are considered low risk by the organization.

Examination procedures:

Examiners should not expect organizations to employ formal project management techniques in all situations. Reviews should be risk focused and center on ensuring project management standards, controls, and procedures are present and commensurate with the characteristics and risks of the projects under review.

OBJECTIVES AND PROCEDURES

Objective 1: Determine the scope of the Development and Acquisition review.

1. Identify strengths and weaknesses relating to development, acquisition, and maintenance activities, through a review of:

Prior reports of examination;

Internal and external audits;

Regulatory, audit, and security reports from key service providers;

Organizational charts;

Network topology maps; and

Résumés of technology managers.

2. Review management's response to report and audit findings to determine

The adequacy and timing of corrective actions;

The resolution of root causes rather than just specific issues; and

The existence of outstanding issues.

3. Review applicable documentation and interview technology managers to identify:

The type and frequency of development, acquisition, and maintenance projects;

The formality and characteristics of project management techniques;

The material changes that impact development, acquisition, and maintenance activities, such as:
- Proposed or enacted changes in hardware, software, or vendors;
- Proposed or enacted changes in business objectives or organizational structures; and
- Proposed or enacted changes in key personnel positions

Objective 2: Assess the level of oversight and support provided by the board and management relating to development, acquisition, and maintenance activities.

1. Assess the level of oversight and support by evaluating:

The alignment of business and technology objectives;

The frequency and quality of technology-related board reporting;

The commitment of the board and senior management to promote new products;

The level and quality of board-approved project standards and procedures;

The qualifications of technology managers; and

The sufficiency of technology budgets.

Objective 3: Assess the organizational structure in relation to the appropriateness of assigned responsibilities concerning technology systems and initiatives.

1. Evaluate organizational responsibilities to ensure the board and management:

Clearly define and appropriately assign responsibilities;

Appropriately assign security, audit, and quality assurance personnel to technology-related projects;

Establish appropriate segregation-of-duty or compensating controls; and

Establish appropriate project, technology committee, and board reporting requirements.

Objective 4: Assess the level and characteristics of risks associated with development, acquisition, and maintenance activities that could materially impact the organization.

1. Assess the risks identified in other objectives and evaluate the adequacy of risk management programs regarding:

Risk identification and assessment procedures;

Risk reporting and monitoring procedures; and

Risk acceptance, mitigation, and transfer strategies.

Objective 5: Assess the adequacy of development project management standards, methodologies, and practices.

1. Evaluate the adequacy of development activities by assessing:

The adequacy of, and adherence to, development standards and controls;

The applicability and effectiveness of project management methodologies;

The experience of project managers;

The adequacy of project plans, particularly with regard to the inclusion of clearly defined:
- Phase expectations;
- Phase acceptance criteria;
- Security and control requirements;
- Testing requirements; and
- Documentation requirements;

The formality and effectiveness of quality assurance programs;

The effectiveness of risk management programs;

The adequacy of project request and approval procedures;

The adequacy of feasibility studies;

The adequacy of, and adherence to, standards and procedures relating to the:
- Design phase;
- Development phase;
- Testing phase; and
- Implementation phase;

The adequacy of project change controls;

The appropriate inclusion of organizational personnel throughout the project's life cycle;

The effectiveness of project communication and reporting procedures; and

The accuracy, effectiveness, and control of project management tools.

Objective 6: Assess the adequacy of acquisition project management standards, methodologies, and practices.

1. Assess the adequacy of acquisition activities by evaluating:

The adequacy of, and adherence to, acquisition standards and controls;

The applicability and effectiveness of project management methodologies;

The experience of project managers;

The adequacy of project plans, particularly with regard to the inclusion of clearly defined:
- Phase expectations;
- Phase acceptance criteria;
- Security and control requirements; and
- Testing, training, and implementation requirements;

The formality and effectiveness of quality assurance programs;

The effectiveness of risk management programs;

The adequacy of project request and approval procedures;

The adequacy of feasibility studies;

The adequacy of, and adherence to, standards that require request-for-proposals and invitations-to-tender to include:
- Well-detailed security, reliability, and functionality specifications;
- Well-defined performance and compatibility specifications; and
- Well-defined design and development documentation requirements;

The adequacy of, and adherence to, standards that require:
- Thorough reviews of vendors' financial condition and commitment to service; and
- Thorough reviews of contracts and licensing agreements prior to signing;

The adequacy of contract and licensing provisions that address:
- Performance assurances;
- Software and data security provisions; and
- Source-code accessibility/escrow assertions;

The adequacy of project change controls;

The appropriate inclusion of organizational personnel throughout the project's life cycle;

The effectiveness of project communication and reporting procedures; and

The accuracy, effectiveness, and control of project management tools.

Objective 7: Assess the adequacy of maintenance project management standards, methodologies, and practices.

1. Evaluate the sufficiency of, and adherence to, maintenance standards and controls relating to:

Change request and approval procedures;

Change testing procedures;

Change implementation procedures;

Change review procedures;

Change documentation procedures;

Change notification procedures

Library controls; and

Utility program controls.

Objective 8: Assess the effectiveness of conversion projects.

1. Evaluate the effectiveness of conversion projects by:

Comparing initial budgets and projected time lines against actual results;

Reviewing project management and technology committee reports;

Reviewing testing documentation and after-action reports;

Reviewing conversion after-action reports;

Interviewing technology and user personnel; and

Reviewing suspense accounts for outstanding items.

Objective 9: Assess the adequacy of quality assurance programs.

1. Assess the adequacy of quality assurance programs by evaluating:

The board's willingness to provide appropriate resources to quality assurance programs;

The completeness of quality assurance procedures (Are the deliverables of each project, and project phase, including the validation of initial project assumptions and approvals, appropriately assured?);

The scalability of quality assurance procedures (Are the procedures appropriately tailored to match the characteristics of the project?);

The measurability of quality assurance standards (Are deliverables assessed against predefined standards and expectations?);

The adherence to problem-tracking standards that require:
- Appropriate problem recordation;
- Appropriate problem reporting;
- Appropriate problem monitoring; and
- Appropriate problem correction;

The sufficiency of, and adherence to, testing standards that require:
- The use of predefined, comprehensive test plans;
- The involvement of end users;
- The documentation of test results;
- The prohibition against testing in production environments; and
- The prohibition against testing with live data;

The sufficiency and effectiveness of testing programs regarding:
- The accuracy of programmed code;
- The inclusion of expected functionality; and
- The interoperability of applications and network components; and

The independence of quality assurance personnel.

Objective 10: Assess the adequacy of program change controls.

1. Evaluate the sufficiency of, and adherence to:

Routine and emergency program-change standards that require appropriate:
- Request and approval procedures;
- Testing procedures;
- Implementation procedures;
- Backup and backout procedures;
- Documentation procedures; and
Notification procedures;

Controls that restrict the unauthorized movement of programs or program modules/objects between development, testing, and production environments;

Controls that restrict the unauthorized use of utility programs, such as:
- Policy prohibitions;
- Monitoring of use; and
- Logical access controls;

Library controls that restrict unauthorized access to programs outside an individual's assigned responsibilities such as:
- Logical access controls on all libraries or objects within libraries; and
- Automated library controls that restrict library access and produce reports that identify who accessed a library, what was accessed, and what changes were made; and

Version controls that facilitate the appropriate retention of programs, and program modules/objects, revisions, and documentation.

Objective 11: Assess the adequacy of patch-management standards and controls.

1. Evaluate the sufficiency of, and adherence to, patch-management standards and controls that require:

Detailed hardware and software inventories;

Patch identification procedures;

Patch evaluation procedures;

Patch request and approval procedures;

Patch testing procedures;

Backup and backout procedures;

Patch implementation procedures; and

Patch documentation.

Objective 12: Assess the quality of application, system, and project documentation, and the adequacy of documentation controls.

1. Assess the adequacy of documentation controls by evaluating the sufficiency of, and adherence to, documentation standards that require:

The assignment of documentation-custodian responsibilities;

The assignment of document authoring and approval responsibilities;

The establishment of standardized document formats; and

The establishment of appropriate documentation library and version controls.

2. Assess the quality of application documentation by evaluating the adequacy of internal and external assessments of:

Application design and coding standards;

Application descriptions;

Application design documents;

Application source-code listings (or in the case of object-oriented programming: object listings);

Application routine naming conventions (or in the case of object-oriented programming: object naming conventions); and

Application operator instructions and user manuals.

3. Assess the quality of open source-code system documentation by evaluating the adequacy of internal and external assessments of:

System design and coding standards;

System descriptions;

System design documents;

Source-code listings (or in the case of object-oriented programming: object listings);

Source-code routine naming conventions (or in the case of object-oriented programming: object naming conventions); and

System operation instructions.

4. Assess the quality of project documentation by evaluating the adequacy of documentation relating to the:

Project request;

Feasibility study;

Initiation phase;

Planning phase;

Design phase;

Development phase;

Testing phase;

Implementation phase; and

Post-implementation reviews.

Note: If examiners employ sampling techniques, they should include planning and testing phase documentation in the sample.

Objective 13: Assess the security and integrity of system and application software.

1. Assess the quality of open source-code system documentation by evaluating the adequacy of internal and external assessments of:

Evaluate the security and integrity of system and application software by reviewing:

The adequacy of quality assurance and testing programs;

The adequacy of security and internal-control design standards;

The adequacy of program change controls;

The adequacy of involvement by audit and security personnel in software development and acquisition projects; and

The adequacy of internal and external security and control audits.

Objective 14: Assess the ability of information technology solutions to meet the needs of the end users.

1. Interview end users to determine their assessment of technology solutions.

Objective 15: Assess the extent of end-user involvement in the system development and acquisition process.

1. Interview end users and review development and acquisition project documentation to determine the extent of end-user involvement.