Audit of the Compliance Verification System (CVS): Meat

February 2011

---

Table of Contents

• Executive Summary
  • 1.1 The Compliance Verification System
  • 1.2 Scope And Objectives
  • 1.3 Findings And Recommendations
  • 1.4 Statement Of Assurance
  • 1.5 Audit Opinion
• 2.0 About the Audit
  • 2.1 The CFIA's Compliance Verification System
  • 2.2 Scope And Objective
  • 2.3 Methodology And Approach
• 3.0 Findings and Recommendations
  • 3.1 Introduction
  • 3.2 Control
  • 3.3 Governance
  • 3.4 Risk Management
• Appendix A – CVS Details
• Appendix B – Detailed Audit Criteria
• Appendix C – Audit Sampling Details
• Appendix D – Description of Quality Verification
• Appendix E: Management Response and Action Plan

---

Executive Summary

1.1 The Compliance Verification System

The responsibility of the CFIA is to verify the effective implementation of industry systems through systematic audits of a manufacturer's controls, practices and records along with on-site verification of the process in operations.

The Compliance Verification System (CVS) is a task-based inspection tool used to verify regulatory compliance. It provides clear and consistent direction to inspectors and can be applied to a broad range of inspection activities.

1.2 Scope and objectives

This audit focused on meat inspection activities as identified in "Chapter 18 - Compliance Verification System" of the Meat Hygiene Manual of Procedures (MOP). The MOP is an administrative and technical manual that describes how to achieve compliance with the Meat Inspection Act and Meat Inspection Regulations. Chapter 18

• defines the management framework governing CVS,
• provides guidance on the verification task process, and
• provides guidance on the reporting tools used when conducting CVS activities.

The audit objective was to provide assurance that CFIA's meat inspection activities are in compliance with the Meat Hygiene Manual of Procedures (MOP).

The scope of the audit was limited to

• the reported compliance with the MOP,
• the effectiveness of training and quality management activities, and
• the effectiveness of tracking and reporting verification tasks at a national level.

Specific inspection tasks and activities with regard to how they were carried out were not assessed as part of this audit.

1.3 Findings and recommendations

Finding 1.0: A new CVS curriculum and continual training plan has been developed

The CFIA has recently developed, and is in the process of implementing, a national training plan for meat processing inspectors. The plan includes a new course curriculum that addresses required skills and provides continual training for new and existing inspectors.

Finding 2.0: Performance information may not be complete

Performance information may not be complete and does not accurately reflect current CVS requirements and achievements.

Recommendation 1.0:

The National Inspection Division should ensure that performance information is complete and accurately reflects current CVS requirements and achievements.

Finding 3.0: Quality verifications are not delivered as planned

Quality verifications to determine how well inspection processes are working are not being delivered as required. Analysis of verification data is not undertaken below national levels.

Recommendation 2.0:

The Operations Strategy and Delivery Directorate should ensure that quality verifications are undertaken and reported as planned.

1.4 Statement of assurance

In my professional judgment as Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria. The opinion is applicable only to the entities examined and within the scope described herein.

1.5 Audit opinion

In my opinion, the Compliance Verification System for meat has risks related to the risk management and governance processes relative to the CVS - Meat. These require management attention.

Peter Everson
Chief Audit Executive, CFIA

2.0 About the audit

2.1 The CFIA's Compliance Verification System

The Canadian Food Inspection Agency (CFIA) Meat Programs function with the collaboration of a wide variety of stakeholders. Within this environment of shared responsibilities, the CFIA's top priority is to ensure consumer health by enforcing national and international food safety and inspection standards.

The Food Safety Enhancement Program (FSEP) dictates the requirements food processors must meet, including standard operating procedures, to ensure that

• their establishments comply with all regulatory and CFIA program requirements, and
• their food safety management system meets all requirements of the FSEP manual.

Hazard Analysis Critical Control Point (HACCP) is an internationally recognized food safety management system. It is focused on prevention. Under HACCP, the food manufacturer is responsible for

• identifying all hazards associated with its operation,
• identifying the critical control points along the process, and
• implementing effective measures to control and monitor the critical points to ensure the safety of the final product.

The CFIA is responsible for recognizing and verifying the effectiveness of HACCP systems through

• systematic reviews of a manufacturer's controls, practices and records, and
• on-site verification of the process in operation.

The Compliance Verification System (CVS) is a task-based inspection tool used to verify regulatory compliance. It provides clear and consistent direction to inspectors across a broad range of inspection activities. In April 2008 it was introduced in all federally registered meat establishments. Further details on CVS (including inspector roles) are included in Appendix A.

Major CVS activities include:

• conducting on-site verification and employee interviews,
• doing product sampling, and
• reviewing a company's operational records.

While performing these activities, if an inspector identifies a deviation from the establishment's HACCP plan, the nature of the issue identified will dictate the course of corrective action. For example, slight deviations are recorded as acceptable with comments, whereas deviations to the applicable regulatory requirements or HACCP system are unacceptable and will result in a corrective action request (CAR). A CAR requires the plant to submit a written action plan and correct the problem by a specified date. Food safety issues are dealt with immediately.

In December 2008, the CFIA conducted an audit of the implementation of CVS. That audit found that the design of CVS was generally well-developed. The audit also recommended a future audit with a scope that would include compliance with Chapter 18 of the Meat Hygiene Manual of Procedures. This audit is a result of that recommendation.

2.2 Scope and objective

This audit focused on meat inspection activities as identified in "Chapter 18 - Compliance Verification System" of the Meat Hygiene Manual of Procedures (MOP). The MOP is an administrative and technical manual describing how to achieve compliance with the Meat Inspection Act and Meat Inspection Regulations. Chapter 18

• defines the management framework governing CVS,
• provides guidance on the verification task process, and
• provides guidance on reporting tools when conducting CVS activities.

The audit objective was to provide assurance that CFIA's meat inspection activities are in compliance with the MOP.

The scope of the audit was limited to:

• the reported compliance with the MOP,
• the effectiveness of training and quality management activities, and
• the effectiveness of tracking and reporting verification tasks at a national level.

Specific inspection tasks and activities (with regard to how they were carried out) were not assessed as part of this audit.

The work included meat inspection activities from April 1, 2009, to March 31, 2010, in the Operations Branch. The audit was initiated in October 2009 and fieldwork was completed in May 2010.

2.3 Methodology and approach

For the purposes of this audit, compliance was tested against the policies and procedures of CFIA's Meat Hygiene Manual of Procedures (Chapter 18).

Criteria

Audit criteria and detailed sub-criteria were developed to serve as standards against which our assessment could be made, clarify the audit objectives and form a basis for the work plan and the conduct of the audit. These criteria can be found in Appendix B. For each of the criteria, the relationship to governance, control, and risk management—or a combination of the three—was identified. For this audit, we determined the audit criteria to be as follows.

Control

• The Agency provides employees with the necessary training, tools, and information to support the discharge of their responsibilities.

Governance

• Management receives sufficient, complete, timely and accurate information.
• Management has identified and monitors appropriate performance measures linked to planned results.

Risk management

• Management identifies and assesses the existing controls to manage its risks and assesses and responds to identified risks.

Approach

An audit program detailed the audit procedures and tests to be carried out to conclude on the audit objective. The audit program included

• identifying and collecting documents and data;
• analysing data;
• evaluating and interpreting documents and data;
• recording observations and variances; and
• drawing conclusions on observations with regard to the stated criterion.

The audit team completed the following activities:

• reviewed the CFIA Meat Hygiene Manual of Procedures (Chapter 18),
• reviewed documentation related to the implementation and history of the Compliance Verification System;
• conducted planning activities, including describing the population of establishments across the country and outlining the proposed audit approach, audit criteria, sampling plan and audit program;
• conducted interviews and inquiries with various CFIA inspectors, supervisors and managers across the country;
• analyzed worksheets, along with the quality management system and data quality of the CVS reports, across the Agency; and
• undertook detailed worksheet testing of CVS task delivery.

Interviews and inspection worksheet testing were conducted from March 2010 to May 2010 at 54 of the 767 registered establishments in Canada (as of April 8, 2010). This provided a cross-section of establishments by type and region in CFIA's Quebec, Ontario and Western Areas.

The reported delivery of planned CVS tasks as recorded by inspection staff was assessed. For our testing exercise, we selected a non-statistical random sample of 324 CVS tasks representing 4424 separate inspection activities. The detail of the sample taken for this purpose is provided in Appendix C.

3.0 Findings and recommendations

3.1 Introduction

This section presents detailed findings from the Audit of the Compliance Verification System (CVS): Meat at the CFIA. Findings are based on the evidence and analysis, gathered throughout the planning and conduct phases of the audit.

3.2 Control

Finding 1.0: A new CVS curriculum and continual training plan has been developed

The CFIA has recently developed, and is in the process of implementing, an enhanced national training plan for meat processing inspectors. The plan includes a new course curriculum that addresses required skills and provides continual training to new and existing inspectors.

The new meat hygiene training modules comprise the core of the national training program. In addition to the modules, the trainer also has various tools at their disposal, such as coaching and job shadowing. Management indicated they intend to deliver this new training to all meat inspection staff on an on-going basis.

The delivery of former CVS training program was assessed as part of this audit, the recent changes to the overall CVS training program render those findings no longer relevant going forward.

3.3 Governance

Finding 2.0: Performance information may not be complete

We expect that management receives sufficient, complete, timely and accurate information for decision making. We also expect that the reported completion of required CVS inspection activities would reflect those planned for and articulated in the Establishment Profiles and Task Tracking Tables.

The "Meat Program Delivery Rate National Summary Report" is a key senior management document. It is used for monitoring and decision making regarding the delivery of the meat program. It is provided quarterly to senior management.

While there is a strong performance reporting protocol in place, we found that current CVS performance reporting practices result in management receiving either incomplete information or information that may not reflect the achievement of CVS delivery rates. As depicted in the chart below, CFIA inspectors report on CVS tasks and activities based on two things: first, the approved Establishment Profile; and second, adjustments made by inspectors to reflect on-going changes in establishment business and processes.

As depicted in the chart below, CFIA inspectors report on CVS tasks and activities based on two things: first, the approved Establishment Profile; and second, adjustments made by inspectors to reflect on-going changes in establishment business and processes.

CVS Control Points / Documents

Performance reporting

While senior management is provided with the overall CVS task completion rate on a timely basis (i.e. quarterly), we found that there were opportunities to enhance performance reporting of CVS:

• The report does not provide a focus on inspection activities that may be of more importance from a food safety perspective, such as direct food safety tasks (Section 1) and sampling tasks (Section 7).
• CVS coordinators and inspection managers are not receiving sufficient delivery information at the establishment, sub-district or regional levels.
• The report does not provide for the weighting of frequency rates (for example daily, monthly, quarterly). Missing one quarterly task reduces the overall delivery rate by 25 percent, while missing a daily task, relating to food safety, only reduces the overall delivery rate by less than 1 percent.
• Section 7 tasks are not included in the "Meat Program Delivery Rate National Summary Report". However, a separate report for Section 7 tasks relating to Listeria is produced on a quarterly basis. Our analysis of a non-representative sample of the records related to 19 Section 7 tasks found that their recorded delivery rate was less than expected.
• The National Inspection Division (NID) Quarter 4 "Meat Program Delivery Rate National Summary Report" noted that some weekly Worksheets were not submitted during fiscal year 2009-10. The result is that completed inspection activities are not being captured and reported on.

Reported compliance

Our analysis of the completion data reported by inspectors and recorded in the CVS database confirmed the same rate of reported non-compliance as provided to senior management in the "Meat Program Delivery Rate National Summary Report."

While the data being provided to management is accurate, we noted that there are practices that may lead to over- or under-reporting. Eight Establishment Profiles were validated against CVS data and we found that there were seven cases in which the tasks recorded as undertaken by inspectors on their Task Tracking Tables did not match the expectations contained in the Establishment Profiles.

The data recording and reporting convention does not take into account on-going changes in establishment businesses and processes. Differences between Establishment Profiles and Task Tracking Tables lead to completion rates being misstated for the following reasons:

• inspectors are conducting tasks that are not identified as planned in the database and therefore are not reported on, and
• tasks that are no longer relevant to a particular establishment are being recorded as not being delivered.

There is a risk that ineffective data analysis does not provide senior management sufficient information needed for monitoring and decision making.

Recommendation 1.0:

The National Inspection Division should ensure that performance information is complete and accurately reflects current CVS requirements and achievements.

3.4 Risk management

Finding 3.0: Quality verifications are not delivered as planned

In 2004-2005, the Operations Branch initiated a concerted effort to improve the consistency and overall quality of the activities it conducts (including inspection, audit, and certification). The branch identified implementing a quality management system (QMS) that embodied modern quality management and improvement practices as the best method to achieve the improvements. This was supported by the corporate strategy, in order to implement quality management throughout the CFIA. The QMS quality verification and improvement cycle for delivery of inspections, audits, certificates, etc. involves the following:

1. Preparing quality verification plans: The supervisor or regional veterinary officer prepares an annual quality verification plan using the QMS. Plans are approved by the inspection managers and regional directors.
2. Conducting quality verifications: Quality verifications, led by supervisors and regional program officers with inspectors' participation, are conducted to determine how well the inspection, audit, or certification process is working.
3. Analyzing data: Verification data is analyzed on an ongoing basis (at least every quarter) to identify current quality levels and opportunities for improvement.
4. Developing improvement plans: Improvement plans are developed and implemented to address root causes for quality loss.
5. Preparing quarterly reports: all supervisors, inspection managers, regional directors and Area executive directors prepare quarterly reports so that the QMS continues to operate as designed.

The Quality Management System (QMS) Manual defines what quality verifications (QVs) should be conducted and their expected frequency (see Appendix B). We expected that QVs were fully delivered in keeping with the quality verification plans; and that analysis and reporting identifies current quality levels and opportunities for improvement.

There are nine (9) core activities for meat inspection (Appendix D). We examined a sample of 56 quality verifications across the nine core activities conducted in fiscal year 2009-10. We found that 76 percent of the QVs expected in the quality verification plans were delivered across the four CFIA Areas. The delivery rate for QVs varied across CFIA Areas; from 93 percent in Ontario to 62 percent in Quebec.

Click on image for a larger view

Results of inspection activities sample testing

Two factors may contribute to completion rates less than those expected in the quality verification plans and to the variance in delivery between CFIA Areas:

• QVs are designed to be completed as a team, yet teams are not always available.
• QVs are not being done on a routine basis, as front line staff stated that they can be administratively burdensome and they do not have the time to dedicate to the QVs. For meat processing supervisors, the time commitment increases with the number of inspectors reporting to each supervisor. The QV rate for the core activities conducted in slaughter establishments does not depend on the number of inspectors.

QV data is analyzed and reported to identify systemic issues that prevent the full delivery of CVS tasks. This analysis is currently undertaken nationally and has not been rolled out to Area or regional levels as intended.

Continued limited and incomplete levels of quality verifications to determine how well the inspection process is working increases the risk that management will not be aware of current quality levels and opportunities for improvement. Without full implementation and analysis at all levels, a QMS is not able to achieve continual improvements to the consistency and overall quality of the activities delivered by Agency staff, including regulatory, service, and incident management activities.

Recommendation 2.0:

The Operations Strategy and Delivery Directorate should ensure that quality verifications are undertaken and reported as planned.

Appendix A: CVS details

The CFIA's Compliance Verification System (CVS) is a tool used by CFIA inspection staff to assess operators' compliance with the regulated requirements for: food safety; non-food safety (for example, identification, labelling and animal welfare); export; and systems.

The CVS is designed to

• integrate the Meat Programs' inspection activities (tasks) into one system,
• highlight the accountability of regulated parties,
• focus CFIA activities on compliance verification,
• provide CFIA inspection staff with clearly defined tasks to enhance uniform delivery,
• verify that the operator's written programs are implemented and that they are effective to meet regulatory requirements, and
• support the strategic direction of the CFIA business plan to move regulated parties and government towards a more science-based risk management system.

The CVS consolidates related inspection activities (including inspections, sampling, and export certifications) and the Food Safety Enhancement Program (FSEP) into one integrated system based on Hazard Analysis Critical Control Point (HACCP). CVS includes the inspection activities required under the following:

• Multi-Commodity Activities Program (MCAP): regular core inspection activity using 1427 inspection report.
• Food Safety Enhancement Program (FSEP): quarterly HACCP regulatory system audits (RSA) using the audit reports and corrective action request (CAR) inspection reports.
• Export to the USA: monthly inspection activities targeting HACCP systems using the Basic Compliance Checklist (BCC).

Click on image for a larger view

The Compliance Verification System

Click on image for a larger view

Compliance Verification System - Inspector Role

Appendix B: detailed audit criteria

Audit objective:

The audit objective was to provide assurance that the CFIA's meat inspection activities are in compliance with the Meat Hygiene Manual of Procedures, and that appropriate processes exist for adequate governance, risk management and control over meat inspection activities.

Audit criteria:

Governance

• Management receives sufficient, complete, timely and accurate information.
  • Information for decision making is accurate and submitted to management.
  • Information input into the CVS database is accurate and timely.
  • Information presented to management is accurate and has been subjected to quality assurance.
• Management has identified and monitors appropriate performance measures linked to planned results
• The establishment profiles are accurate and updated according to procedures.
• The task tracking table mirrors the establishment profile and is used to monitor areas of non-delivery and assesses task completion results.
• Non-compliance in the delivery of mandatory tasks is identified and resolved.

Risk management

• Management identifies and assesses the existing controls that are in place to manage its risks and identified risks are assessed and responded to.
  • The quality management system (QMS) and the quality verification (QV) assessments are conducted to ensure control weaknesses and risks to achieving compliance to CVS task delivery are identified.
  • Improvement plans are captured and monitored.
  • There have been observable improvements to activities that were identified as being medium- to high-risk.

Control

• The Agency provides employees with the necessary training, tools, and information to support the discharge of their responsibilities.
  • CVS training has been delivered to inspection staff and is being tracked.
  • CVS training for new inspectors is delivered in a timely fashion.

Appendix C: audit sampling details

Sample details by CVS task and the number of activities represented is provided below.

CVS task categories	number of tasks	number of activities
Section 1.1 - Food Safety: Critical Control Points	48	400
Section 1.2 - Food Safety: Prerequisite Programs	74	496
Section 1.3 - Food Safety: Slaughter	7	1802
Section 2.1 - Non Food Safety: Labelling Practices	46	138
Section 2.2 - Non Food Safety: CFIA Stations and Facilities	46	46
Section 3.1 - Export: Export USA	43	369
Section 3.2 - Export: Export Other than the USA	31	119
Section 3.3 - Export: Foreign Country Follow-up Requirements	3	3
Section 5.1 - Animal Welfare and Animal Health	7	565
Section 7.1 - Sampling: Operator Sampling	12	452
Section 7.2 - Sampling: CFIA Sampling	7	34
Totals	324	4424

Appendix D: description of quality verification

Extract from the Quality Management System Manual:

Quality Verifications gather data defining the performance of the core activity. The data is obtained by assessing verification criteria which are linked to the key macro steps in the inspection process. There are two types of Quality Verifications:

• File Reviews. This is a quality verification done using only documents generated by the inspection or other documents related to the conduct or support of the inspection such as training certification lists, standard reference lists,. ID Cards, etc.
• On site verifications. A quality verification which uses all the same inspection documents as in a file review, but also requires the person conducting the quality verification to directly observe the conduct of the inspection or audit activity. In cases where the inspection or audit activity occurs over several days the person conducting the verification may not be with the person or team conducting the inspection for the complete time period.

For each core activity a standard list of verification criteria has been established. Each verification criteria is linked to a specific macro step of the inspection or audit process.

The same set of verification criteria is used for both file review verifications and on-site verifications. For file review verifications normally not all criteria can be assessed. The QMS database identifies if a criteria can only be assessed during an On Site Verification

There are nine (9) applicable core activities for meat inspection.

• CA27 Establishment - Registration and renewal
• CA41 Complaints and investigations
• CA57 Import product inspection
• CA58 Export product certification
• CA63 Slaughter - Domestic product inspection - Poultry/Rabbits
• CA65 CVS Sections 1, 2, 3, 5 & 7 tasks
• CA66 CVS Section 4 tasks (HACCP system design verification)
• CA70 Red meat slaughter domestic product inspection
• CA72 Humane transportation CVS Pilot

Example of the requirement for quality verifications (extract from Annex C of the QMS Manual)

CA 65: CVS - Meat Verification Sections 1, 2, 3 & 7 Tasks

Slaughter Establishments and traditional Slaughter Establishments:

• Core Activity 65 (CA65) is divided into two parts:
  1. Observe and evaluate the delivery of 3 CVS tasks taken from Section 1, 2, or 3 documentation and follow up associated on them. The tasks chosen should be some of the more difficult to conduct or critical to food safety. Over time, different inspectors should be observed.
  2. Conduct a file review for one week of records. The week chosen should contain a CAR or an Acceptable with comments (if possible).

Processed Meat Establishments

• On-site: 3 on site verifications with each inspector
  • At the processing facility the supervisor has to observe the delivery of one task and when all tasks for the week are completed review all documentation for the week including task reports, weekly reports, as well as any CARs or Corrective Action Plan Assessments.
  • Of the three on sites, one will be where the inspector is conducting a follow up to a previous CAR
• File Reviews: 10% of inspections up to a maximum of 5 file reviews.
• The file reviews must be on inspections where a CAR was issued; if there are not 5 inspections with CARs, choose a file without a CAR.
  • The area CVS coordinator will schedule a NetMeeting session once per quarter. During those meetings, two files (weeks where CAR were issued) will be reviewed.