Question

Explain the use of audit sampling methods that work for internal auditor within an organization.

Answer 1

There are explaination of sampling internal audit -

Auditors know it may not be practical to examine all available evidence due to its volume and dispersal. In those cases, a sample is selected to evaluate against the audit criteria and help develop the audit conclusion.

The risk is that the sample may not be representative of the total set of people, documents, practices, and records being assessed. As a result, the audit conclusion may be different than if the auditor had examined the whole population.

So, we may over-audit by taking samples that are too large and waste time and resources. Or, we may under-audit by taking samples that are too small and end up not detecting numerous nonconformities.

It is important to apply the appropriate use of sampling, since it is closely related to the confidence that people will place on the audit conclusion.

Sample Definition
A sample is a small part of anything, intended as representative of the whole. It may not be practical to examine all available data. For example, records may be too numerous or dispersed, or may be too time consuming or costly.

So, sampling selects less than 100% of the items to obtain and evaluate evidence to form an audit conclusion. The Goal is confidence that the audit objectives can be met. The Risk is that the samples may not be representative of the entire population. The Conclusion may be different than if you were able to examine the whole population.

Audit Principle
ISO 19011:2011, Guidelines for Auditing Management Systems, describes an “Evidence-based Approach” that is based on sampling.

This audit principle (in clause 4.f) uses a rational method for reaching reliable and reproducible audit conclusions. It states that audits are conducted during a finite period of time and with finite resources, so audit evidence should be verifiable, and based on samples of available information. The principle concludes by saying that the appropriate use of “sampling” is closely related to the “confidence” that can be placed in the audit conclusions.

Audit Guidance
The ISO 19011:2011 auditing guidance standard also states that audit procedures should address the use of appropriate sampling methods (5.3.5). Does your audit procedure cover sampling? Most audit procedures I review do not.

Lead auditors should be aware of sampling techniques when preparing the audit plan (6.3.2.1), and the audit plan should cover the extent of the sampling needed to obtain sufficient audit evidence (6.3.2.2). Work documents may include a specific sampling plan (6.3.4)

More Audit Guidance
But, ISO 19011:2011 is not through on the sampling subject. It also states in 6.4.2 that the opening meeting should clarify that the audit evidence will be based on a sample of information. Later, it states that information should be collected by means of appropriate audit sampling (6.4.6).

The closing meeting should advise participants that the audit evidence was collected based on information samples (6.4.9). Then in the “Auditor Competence” section it states that auditors should understand the appropriateness and consequences of sampling techniques (7.2.3.2). The ISO 19011:2011 edition even added an Annex B.3 with two pages devoted to audit sampling.

The 2002 edition of ISO 19011 only mentioned samples and sampling 9 times. The current 2011 edition mentions those terms 68 times! Think audit sampling was considered important?

Sampling Steps
Audit sampling typically involves these six steps:

1. First, establish the objectives of the sampling plan, e.g., you may want to reduce the audit disruption, yet have a representative sample that provides confidence in the audit conclusions.
2. Next, define the extent and composition of the population. What is the audit scope?
3. Then, select a sampling method. For quality audits, it will likely be judgmental sampling. However, you can still use a statistical method to help identify items within your sample.
4. At this point, determine the sample size to be taken. If you want a statistically valid sample, you will calculate the sample size for the desired confidence level.
5. Now you are ready to conduct the sampling activity, followed by the sixth step.
6. Evaluating, reporting, and documenting the results.

Sampling Methods
When we know our sampling objective, and the extent and composition of the population to be assessed, we will select our sampling method before determining the sample size. You may decide to use “judgmental” (non-statistical) sampling and rely on the auditor knowledge, skills, and experience. But, if you need a statistical estimate of the effect of uncertainty on the audit findings and audit conclusion, then a “statistical” sampling method will be selected.

Statistical
Before choosing a statistical sampling method, you should consider if the outcomes to be examined are attribute-based or variable-based. More on that later. There are four primary methods for statistical sampling:

1. Systematic – Picking every nth item. This would be appropriate for looking over a period of time.
2. Random – Selecting a random sample, which could involve a random number generator.
3. Stratified – Divides the population into homogeneous subgroups that need to be represented.
4. Cluster – Divides the population into heterogeneous clusters that match the population.

We will discuss each of these four methods later in the article.

Non-statistical
As mentioned earlier, we may want to use “judgmental” sampling based on auditor knowledge, skills, and experience. For example, the auditor may know which items have had problems in the past, or may be a higher risk to the organization.

“Convenience” sampling, sometimes referred to as “haphazard” sampling, uses samples that are readily available. I will focus on judgmental sampling later in the article, but first, let’s look at statistical sampling.

Statistical Sampling
Statistical sampling uses a sample selection process based on probability theory.

Attribute-based sampling is used when there are only two possible outcomes for each sample, for example, conforming or nonconforming when assessing completed forms to the procedural requirements.

Variable-based sampling is used when the sample outcomes occur in a continuous range, for example, the number of security breaches over time.

Sampling Plan
Key elements that will affect your audit sampling plan are the:

• Size of the organization (since it affects the population size)
• Number of competent auditors (available to share the sampling load)
• Frequency of audits during the year (sample wide or sample deep)
• Time of the individual audit (duration available for samples to be taken)
• Any externally required confidence level (forcing statistical sampling)

Of course, the use of statistical methods does not eliminate the need to still exercise auditor judgment. You won’t be on auditor auto-pilot.

Confidence Level
A sampling risk of 5% (which equates to a 95% confidence level) accepts the risk that 5 in 100 samples will not reflect the values that would be seen if the entire population was examined. Auditors should document the sampling work and include the:

• Description of the population to be sampled
• Sampling criteria to be used for evaluation (what is considered an acceptable sample)
• Statistical parameters and methods that were used
• Number of samples evaluated and the results obtained.

Statistical tools can calculate sample sizes for different populations, occurrence rates, and confidence levels. For example, for a population of 1000, and a 90% confidence level that no more than 5% of the items are nonconforming, you would sample 45 items.

Sample Sizes If we increase the confidence level to 95%, and changing the nonconformity rate to 1% or less, it would expand the sample for a population of 1000 to 259, far more than 45.

Financial audits typically use statistical sampling methods. However, due to time constraints and cost factors, quality audits often use non-statistical sampling methods.

Systematic Sampling
Systematic sampling selects every Nth item in the population as the sample. For example, to sample 30 items out of a population of 360, the sampling interval would be N=12.

You would select a random starting point within the first interval, e.g., between 1 and 12, you could randomly pick 7. Then, you would extract every 12th item, i.e., 7, 19, on to 343, 355.

However, you have to ensure the systematic sampling interval does not introduce bias. For example, if a warehouse has even locations on one side of the aisle and odd locations on the other side, and you selected every 50th item, and started at 20, you would never pick an odd location.

Random sampling would be more suitable for this scenario. Remember, auditor judgment is still needed for statistical sampling. Systematic sampling is most appropriate for looking over a period of time, since it ensures an even spread of selected items in a sample of the timeframe under review.

Random Sampling
Random sampling gives each item in the sampled population an equal chance of being selected. That means picking one item, has no impact on the probability of any other item being selected.

One random sampling method is to generate a random number for each item to be sampled, sort these random numbers, and then take the top or bottom X, where X is the sample size. Note that you can use the “=RAND()” function in Excel to generate the random numbers.

Stratified Sampling
When items from each subgroup within the population need to be represented, you can use “stratified” sampling. To do that, divide the population into subgroups, or strata. Then select random or systematic samples from within each subgroup.

The sampling fraction for each subgroup may be taken in the same proportion that the subgroup has in the population. For example, you would randomly select customers of each type in proportion to the number of customers of that type in the population.

Suppose that 70% of your customers are commercial and 30% are government. You could divide the population into those two groups and take 70% of your samples from the commercial group and 30% of your samples from the government group.

Cluster Sampling
In “stratified” sampling, the subgroups are homogeneous. In “cluster” sampling, the cluster is as heterogeneous as possible to match the population.

A random sample is taken from within one or more selected clusters. For example, if there are 20 small projects in the scope, you might use cluster sampling to randomly select 4 projects as representative for the audit.

Unless clusters are selected randomly, and many are sampled, you cannot always generalize about the entire population. For example, random sampling from all parts produced last week, or for a specific product, may cause sampling bias.

Judgment Sampling
Judgment-based sampling relies on the knowledge, skills, and experience of the audit team. The sampling should consider:

• Previous audit experience within the audit scope (may spend more time if little experience)
• Complexity and interaction of the processes (sample more if a complex process)
• Changes in technology and the management system (sample more if higher risk)
• Previously identified risk areas and improvement areas (consider weak areas)
• Output from monitoring of the management system (listen to feedback on issues)

The sample should provide coverage of all types of items within the population. A drawback of judgment-based sampling is there can be no statistical estimate of the effect of uncertainty on your audit findings and conclusions.

Convenience Sampling
Another type of non-statistical sampling is “convenience” sampling. With this method, you select a nearby and readily available sample. Sounds easy. However, in most cases, you cannot draw conclusions about the total population, because the sample is not likely to be representative.

For example, when auditing purchase orders, you may be tempted to assess the ones on the buyer’s desk. However,

• The buyer may have placed “correct” ones on the desk to be used for the audit
• Or, the POs may be recently, carefully created knowing the audit was imminent
• Or, the POs on the desk may be from the few suppliers handled by that buyer

Sampling Concerns
If you are not careful, your sample may be invalid, too large, or too small.

Invalid Sample
If a sample is not representative of the population, it is not a valid sample. Since auditors rely on samples to form their audit opinion, an invalid sample could lead to an invalid conclusion.

Over-Auditing
If the sample taken is too large, it was unnecessary and has wasted valuable time. You want to select the smallest sample leading to valid audit results.

Under-Auditing
If the sample is too small, it may not be representative of the population, which means there is a risk that nonconformities will not be detected.

Sampling Risk
Speaking of risk, if you are sampling two out of ten items, and there is one nonconforming item, what is the risk of missing that nonconformity with a random sample of only two items?

There are 45 different combinations when taking two items at a time from a population of ten. That means there would be nine combinations with the bad item, and 36 combinations without it. So, while the percent of bad items is 10%, the risk of selecting a sample without the bad item is 36 of 45, or an 80% risk. That highlights the impact of sample size on the detection risk.

Sample Selection
Audit samples are taken to evaluate if practices are producing results that meet stated requirements. Our audit samples include which procedures to review, people to interview, activities to observe, and records to examine.

However, there is not enough time available to look at everything. In fact, you may have been given an audit assignment with a specific audit duration, which ends up being a limiting factor for your sampling.

The audit sample should be relevant and representative, which means it should be selected by the auditor, not the possibly “loaded dice” offered by the auditee. How thoughtful of them.

Your audit checklist, if you use one, may define the planned sample through the selected areas to assess, documents to review, and records to examine.

There will be a difference between your “planned” sample and the “actual” sample, as you adjust to the responses, practices, and evidence during the audit. Your auditor notes will record the sample actually taken, which will be helpful in reporting results and guiding future audits of the area.

Evidence Types
There are four types of evidence: Documents, Observations, Records, and Statements, which forms the acronym D-O-R-S.

This audit evidence is compared to the four types of requirements to judge conformity: Legal, Organization, Customer, and Standard, which forms the acronym L-O-C-S. If the objective evidence indicates a requirement is not being met, then a nonconformity has been identified.

Remember, when you interview someone, you are listening to their understanding of the “current” process. When you review documents, you are hopefully looking at the “current” versions. When you observe work activities, you are viewing the “current” practices.

However, to assess conformity in the “past”, since the last audit, you must examine records generated during that time interval, not just the records generated today.

Evidence Samples
How many people, documents, activities, and records should be sampled during a quality audit? The sample sizes should be based on the population size and business risk. And, larger samples will provide more confidence in the audit conclusions.

However, the samples will be limited due to the audit’s cost and operational disruption, as well as, by the time allocated for the audit, and the types of evidence being examined.

For example, with monthly management reviews, and semi-annual audits, you could look at only six records and have a 100% sample since the last audit.

If 1000 purchase orders have been issued since the last audit, picking 20 representative orders would be a 2% sample over the past six months. However, for a statistically valid audit sample for a 95% confidence level, and a nonconformity rate of 1% or less, you would have to sample 259 orders, or a 26% sample. When was the last time you examined 259 POs in an audit?

Record Sampling
Consider the time period to be assessed, when you select records be included in your sample. Remember that old records were created by old processes. If you find a problem in the past, similar nonconformities may not exist with the current process.

So, focus on records related to the current processes. Consider the records that have been generated since the last audit. However, you should still select some older records for assessing adherence to record retention policies.

Sampling Disclaimer
Audit results reflect the situation you found at the time of the audit. You relied upon a limited sample taken during a brief period of time. Therefore, consider including a disclaimer like this one in your audit report:

This audit was based on random samples and every aspect of the system was not necessarily covered. Therefore, nonconformities may exist that have not been identified in this report.

Explain to the audited organization that if no nonconformities were found in sample, that does not mean there are no nonconformities in their system.

Summary
To recap, audit results will be based on sampled information. The selected samples should be representative of the population. It is important to understand the different audit sampling techniques, and to be aware of the uncertainty introduced by sampling.

You want to determine an acceptable sample size for confidence of all parties: the audit client, the auditee, and the audit team. Take the steps necessary to improve your judgment-based sampling, since that will likely be the primary sampling method for your audits. Avoid under-auditing (selecting too few samples).

Sample all types of evidence: people, documents, activities, and records. Report the sampling level, and include a disclaimer in your audit report.