ARCHIVED - Audit of the Quality Management System (QMS) - Report

This page has been archived

Information identified as archived is provided for reference, research or record-keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

November 2013

Table of Contents

1.0 Executive Summary

Introduction

The Canadian Food Inspection Agency (CFIA, the Agency) plays a key role in maintaining Canada's food safety system. The method in which food is produced and distributed has undergone fundamental changes in recent years. The CFIA is reacting to these changes by implementing an aggressive change agenda.

In response to pressures from increased globalization in the food industry and advances in science and technology, the Agency is modernizing its approach to food inspection to maintain a robust approach to food safety and consumer protection. The Government of Canada's 2011 Budget committed $100 million over five years for the CFIA to modernize its food inspection system.

As part of this modernization initiative, the Agency has committed to develop an improved food inspection model. The model includes an approach to system performance that supports a philosophy of continuous improvement, measuring outcomes and delivering findings that can be used to adapt, improve, and be more effective at managing risks to food safety.

In 2004-2005 the Operations Branch introduced a Quality Management System (QMS) as part of an effort to improve the consistency and overall quality of the activities it conducts. The goal of QMS is continuous improvements to the consistency and overall quality of the regulatory, service, and incident management activities delivered by Operations staff.

In addition to the goal of improving consistency and the overall quality of inspection activities, the QMS and its outputs are being used by the Agency to demonstrate meeting international requirements, such as:

  • the United States Department of Agriculture (USDA) requirement for daily presence in meat establishments; and
  • the European Union (EU), the QMS has been proposed as a method to respond to foreign country audit recommendations.

Audit Objective and Scope

The objective of the audit was to provide assurance on the adequacy of the management framework, practices and controls in place for the Quality Management System (QMS).

The audit included both the design and the operation of the QMS. This included all QMS activities in all branches in National Headquarters and the four Areas. Site visits were carried out during the planning phase to the Ontario Area and during the conduct phase, the Atlantic Area, Quebec Area and the Western Area were visited. The QMS IT system was reviewed to determine the extent to which it supports and adapts to program needs; however, the IT system controls were not audited.

The audit covered QMS activities from April 2011 to end of the conduct phase, May 31, 2013.

Key Findings and Conclusion

Key Findings

The QMS program was developed by the Operations Branch and is supported by a QMS Manual that is approved and maintained by the Operations Branch at the Director level. The program is further supported by several procedural guidance documents but at the time of the audit many were in draft, not approved, in the early stages of development or not posted on the Agency's intranet. The audit also observed that certain aspects of the manual were not current and/or not being followed.

While the QMS Manual covers roles and responsibilities indicating a requirement for inter-Branch consultation for the resolution of systemic issues, a formal mechanism to achieve the necessary engagement of other branches was only recently instituted. Minimal senior Agency-level oversight has contributed to the QMS not attaining the necessary traction to affect change at all levels in the organization. Also, the absence of a robust QMS framework limits Operations Branch senior management's ability to establish, demonstrate and communicate its commitment to a positive quality culture.

The audit noted that quality verifications are being conducted as planned. As such, the QMS provides a tool to demonstrate supervisory presence in the Meat Hygiene Program, an important ongoing quality control. Quality verifications resulted in the identification and documentation of quality loss issues (QLI); however, there exists gaps in the Supervisor and Area Reports whereby not all systemic QLIs are being tracked and noted on the National QLI Log.

The audit also noted that identified QLIs were not adequately analyzed and resolved as outlined in the QMS Manual. While someQLIs can be resolved easily and immediately at the local level, available documentation indicated a high occurrence of unresolved issues including both local and national level issues.

While the audit did find gaps in the overall framework, the QMS National Team has drafted a three-year enhancement plan which aims at addressing the weaknesses in the QMS program. The goal is to strengthen the infrastructure supporting the QMS which is aimed to strengthen the overall quality of inspection activities.

The CFIA is in the early stages of its Transformation Agenda (TA) which aims to make broad, significant enhancements to how the Agency delivers its programs and activities. As the Agency modernizes its regulatory and program frameworks it will be necessary for the QMS to be functioning effectively to ensure continuous improvement to the overall quality of the regulatory, service and incident management functions.

The TA will have an impact on the Branch's quality program, and the activities which it now covers. There is a risk that quality issues will not be addressed unless the role of QMS and its participants are redefined in a strengthened quality framework for the Agency.

Conclusion

In conclusion, the QMS framework needs to be strengthened through an update and finalization of documented procedures – most importantly, the assignment of responsibility and accountability for the management and resolution of systemic national quality loss issues.

The QMS is not fully meeting the expectations of the Operations Branch as stated in the QMS Manual. The current system to evaluate how well inspection activities are being delivered needs to be strengthened. The QMS is not fully assessing whether programs are delivered as designed, in a consistent, efficient and relevant manner.

The QMS needs to be applied with rigour and discipline in order to realize its objectives of improving consistency and quality of the regulatory activities being delivered. This includes regular comprehensive analysis to identify the causes and resolutions of specific and systemic issues.

2.0 Introduction

2.1 Background

The Canadian Food Inspection Agency (the Agency or CFIA) directly supports the Government of Canada's priorities for bolstering economic prosperity, strengthening security at the border and of the safety of the food supply, protecting the environment and contributing to the health of Canadians. To effectively fulfill its responsibilities in safeguarding Canada's food and sustaining its animal and plant resource base, the CFIA aims to achieve one strategic outcome which is a safe and accessible food supply and plant and animal resource base.

The CFIA works with its partners to implement food safety measures, to manage food/ animal/ plant risks and emergencies, as well as to promote the development of food safety and disease control systems. The objective is to maintain the safety of Canada's high-quality agriculture, aquaculture/fisheries and agri-food products.

In 2004-2005 the Operations Branch introduced a Quality Management System (QMS) as part of an effort to improve the consistency and overall quality of the activities it conducts. The goal of QMS is continuous improvements to the consistency and overall quality of the regulatory, service, and incident management activities delivered by Agency staff.

The administrative structure supporting QMS falls within the Business Management and Planning Office (BMPO) in the Operations Branch. A National QMS Coordinator (Director) within the Operations Branch is responsible for its overall operation and oversight. Each Area and most Regions have their own QMS Coordinator(s) responsible for providing training, ensuring activities are delivered as expected, coordinating quality loss validations and resolution, and generatingQMS reports. These staff report to Area management and not to the QMS National team.

As set out in the QMS Manual, quality management in the Operations Branch follows a six (6) step cycle:

  • A work plan is developed for every sub-district;
  • Verifications are conducted by the Supervisor, Regional Program Specialist, Veterinarians In Charge, Regional Veterinarian Officers or Area Program Specialist and are entered into a database;
  • Data is reviewed;
  • Quality loss issues (QLIs) are identified;
  • Data analysis is conducted to determine potential root causes; and,
  • Improvement plans are developed and implemented.

QLIs are identified as either local or national issues. Local issues (or specific issues) are to be remedied at the field level. These issues are often simple and may be resolved immediately between the Supervisor and Inspector. National issues (or systemic issues) typically require resolution at a higher level. They are escalated through quarterly reports and included on a National QLI Log for tracking purposes. The National QMS Coordinator maintains this log and communicates QLIs to appropriate staff for resolution.

The Agency does not formally track QMS resource expenditures; however, an estimate of $7.14M was made by the Operations Branch in 2009-10. In addition to the QMS National coordination staff, who is fully dedicated to QMS, many layers of operational staff are involved in its activities. Inspection Supervisors, for example, devote approximately 10-15% of their time to QMS activities.

A high level flow chart of the QMS process is attached at Appendix B.

2.2 Objective

The objective of the audit was to provide assurance on the adequacy of the management framework, practices and controls in place for the Quality Management System (QMS).

2.3 Scope

The audit included both the design and the operation of the QMS. This included all QMS activities in all branches in NHQ and the four Areas. A site visit was carried out during the planning phase to the Ontario Area and during the conduct phase the Atlantic, Quebec and Western Areas were visited.

The QMS IT system was reviewed to determine the extent to which it supports and adapts to program needs; however, the IT system controls were not audited.

The audit covered QMS activities from April 2011 to end of the conduct phase, May 31, 2013.

2.4 Methodology and Approach

A control and risk assessment was completed during the planning phase of the audit. The assessment considered Treasury Board Secretariat's Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors Core Management Controls, the CFIA Quality Management System Manual, and the International Organization of Standardization (ISO 9000). Based on this assessment, and background research, the audit criteria were derived (see Appendix A).

The audit team visited the four CFIA Areas during the audit to conduct process walkthroughs and mapping, interviews and to gather documentation. Interviews were conducted with 72 individuals, both in the Areas and at Headquarters, in the Operations Branch, Policy and Programs Branch and Corporate Management Branch.

A sample of 120 Quality Verifications (QVs) delivered between April 1, 2011 and September 30, 2012 was randomly selected from the QMS database. The cut-off date of September 30, 2012 allowed for all subsequent steps in the QMS process to have been completed. The 120 QVs were used by the audit team to test the effectiveness of controls, determine the rate of occurrence of the controls deviations and provide an objective basis for evaluating the testing results. This represents a 1.2% (120/9951) coverage of the QVs for the testing period of the audit.

The breakdown of the audit sample is as follows:

QVs by Business Line
QVs by Business Line Sample Population
Animal 23 1839
Food 85 6721
Plant 12 1391
Total 120 9951
QVs by Area
QVs by Area Sample Population
Atlantic 28 1636
Quebec 29 1679
Ontario 20 3145
Western 43 3431
Unknown 0 60
Total 120 9951

2.5 Statement of Conformance

The audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of CFIA's Internal Audit quality assurance and improvement program. Sufficient and appropriate auditing procedures were performed and evidence gathered in accordance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing to provide a high level of assurance over the findings and conclusion in this report. The findings and conclusions expressed in this report are based on conditions as they existed at the time of the audit, and apply only to the entity examined.

3.0 Findings and Recommendations

3.1 Quality Management Framework

The expectations, outcomes, scope and roles and responsibilities of the QMS have not been clearly defined and communicated which contributes to gaps in the quality framework.

We expected that an operational quality framework was in place, for establishing and reviewing quality objectives. We expected the framework to contain the following elements:

  • An Operations Branch Quality Policy, defining roles and responsibilities and a governance structure that would spread positive culture from top down and provide for oversight;
  • Fully documented processes for the QMS system;
  • An up-to-date quality manual; and
  • Built-in flexibility to adapt to changing inspection activities

We found that there is an Operations Branch QMS Manual that has a documented and defined objective, which is to continuously improve consistency and the overall quality of regulatory, service and incident management activities delivery by Agency staff. The manual is approved and maintained by the Operations Branch at the Director level.

The manual is written with an Operations Branch perspective as an Operations tool. Performance expectations and outcomes of QMS have not been defined, nor does it clearly articulate the roles and responsibilities of all contributors at all levels to the QMS process.

We found that in addition to the absence of a Quality Policy, many elements of the existing Operations Branch QMS framework are in draft format, not approved, in the early stages of development or not posted on the Merlin Intranet Site. Further, there are no documented controls defined to review, update and approve documents. There is no quality policy document to set the overall tone and formal direction of what is expected.

The QMS program was developed by the Operations Branch in 2004 with a bottom-up approach, in reaction to industry complaints concerning operational inconsistency. As such, the QMS program has had limited success in securing engagement and commitment outside of the Branch. Interviews indicated that Agency staff generally view the QMS as an Operations Branch initiative.

Further, the role of the QMS has evolved since its inception in 2004. It is now also used to demonstrate international commitments such as the United States Department of Agriculture (USDA) requirement for supervisor oversight in registered meat establishments. The QMS is also put forward as a solution to covering off recommendations made in foreign audits, internal audits, and program evaluations.

The CFIA's Change Agenda will have an impact on the Branch's existing QMS program and the activities which are within its current scope. There is a risk that quality issues will not be addressed unless the role of QMS and its participants are redefined in a strengthened quality framework for the Agency.

Recommendation 1.0:

The Vice-President of the Operations Branch should ensure that the framework governing the Operations Branch's QMS program be reviewed and assessed to confirm it meets operational requirements, as well as outside stakeholder expectations, and make the necessary adjustments so the QMS can effectively measure system performance.

3.2 Planning and Commitment

There is an opportunity to strengthen senior management engagement to ensure the QMS is functioning effectively at all levels in the organization.

We expected that the QMS program and related quality activities would be on the Operation Branch's Senior Management agenda and afforded the benefit of governance oversight activities to ensure a coordinated effort at all levels in the organization.

We found that QMS activities and results are not being routinely tabled at Agency governance committees. Staff interviewed during the audit indicated a perceived lack of involvement by Agency senior management, which is evidenced by a large number of unresolved issues requiring inter-Branch solutions and the general absence of communication.

It was noted during interviews that Operations Branch's Senior Management oversight of the QMS program is perceived to be minimal and as a result, front-line staff are not fully engaged. Staff is seeing benefit at the local level but the lack of resolution of QLIs at the national level has caused complacency, frustration and a lack of full commitment. It is estimated that inspection staff dedicate between 10-15% of their time to QMS activities. Management commitment is recognized at the Area, Regional and Sub-District level, nevertheless front-line staff feels that commitment by senior management is not evident.

As QMS observations are not being routinely analyzed and reported at the national level, there is currently no mechanism to ensure the required commitment from other Branches to resolve the identified national issues. Data analysis and performance reporting are essential QMS activities that are required to ensure effective engagement at all levels in the Operations Branch and a strong commitment by other Branches for the resolution of national issues. The lack of inter-Branch collaboration has contributed to quality loss issues not being adequately managed as detailed in Section 3.3 of this report.

Therefore, over the last year the National QMS team has made a number of advancements that are aimed at improving the overall quality management program. QMS results are now being considered in the Agency's business planning cycle and there are plans to incorporate QMS priorities with business line priorities. We understand that Area and Regional Program Specialists in the Atlantic Area have always participated in QVs, and now the other three Areas are following the same structure and incorporating the Specialists into their QVs. A three-year (2012-2013 to 2014-2015) QMS Enhancement Plan has been developed to implement identified improvements to the Operations Branch QMS program. The Agency's Long Term Strategic Plan also identifies adopting a systematic quality management approach across the Agency by 2014.

A QMS Priority Report was provided to the Vice President, Policy and Programs Branch (PPB) by the Operations Branch in December 2011. This report identified several top findings across four broad themes:

  • Lack of training prerequisites defined in the program policy for all Business Lines;
  • Access to reference documentation and material in all Business Lines;
  • File maintenance requirements not defined in program policies; and
  • Manuals and guidance to Operations Branch staff are incomplete, out of date, or in draft form.

A management action plan was developed by PPB to address this Priority Report and the Operations Branch has reported that two concrete corrective actions have been implemented to date. According to the QMS National Team, the completion of the remaining items is currently underway and on-track.

Two committees were recently created in June 2012, in part, to establish better communication and collaboration between PPB and the Operations Branch:

  • A Director level Steering Committee, with representatives from both the Area Operations Branch and the Policy and Programs Branch (PPB), to provide advice and to champion QMS activities; and,
  • The QMS Quality Network comprised of National Program experts, to help resolve quality losses at the working level.

At the time of this audit, the two committees were just created and there had only been a couple of meetings for the Steering Committee, and the QMS Quality Network was scheduled to meet at least once per year.

Recommendation 2.0:

The Vice-President of the Operations Branch should ensure that the QMS program is afforded appropriate governance and oversight to demonstrate the Branch's commitment to quality and to ensure engagement and collaboration amongst Branches for planning QMS activities and resolving issues.

3.3 Monitoring and Continuous Improvement

There is evidence that the planning and quality loss identification processes are working as intended; however, analysis and the resolution of quality loss issues is not being carried out as intended.

We expected that the Quality Verifications (QV) be carried out as planned and that identified Quality Loss Issues (QLIs) would be appropriately analyzed and resolved.

We found that quality verification work plans are being developed respecting National guidance and are generally being delivered. The conduct of QVs also provides a tool to demonstrate supervisory presence, an import ongoing control. Based on our sampling results, delivery of the work plans are resulting in the identification of quality loss issues. However, the QLIs, particularly systemic issues are not being addressed.

The National QMS team produces volumetric reports to monitor the QV delivery rates. These reports are produced quarterly and are distributed to Area Executive Directors, Regional Directors and the Vice President of Operations. Our review of the reports indicated that the QMS QV delivery-rate for FY 2012-13 has increased from the previous year (2011-12) with most Areas showing delivery improvements: West has increased from 84 to 87%, Quebec from 59 to 65%, Ontario from 88 to 100%, and Atlantic remained unchanged at 84%.

Based on our sample, identified quality loss issues were not adequately analyzed and resolved. In a sample of 120 quality verifications, 121 specific (local) issues were identified and 235 systematic (national) issues were identified. The audit showed that only 53% of the identified specific issues and 5% of systemic issues contained documented resolutions. It was noted, based on interviews that some specific issues may be resolved immediately at the local level without being documented.

Some of the contributing factors for the low rate of resolution identified by the audit include the lack of expertise in data analysis, lack of root-cause analysis, limited inter-branch engagement and the breakdown of reporting of QLIs into the roll-up reports (Area and National). In terms of reporting, there were identified gaps in the carry forward of identified QLIs from the Quality Verifications, to the Supervisor's Quarterly Report, the Semi-Annual Report, and the National QLI log.

Recommendation 3.0:

The Vice-President of the Operations Branch should implement efficient and sustainable analysis and performance reporting processes to ensure accurate QLI identification and effective corrective action planning and resolution.

Appendix A: Audit Criteria

LOE #1 - Planning and Commitment: CFIA management is committed to quality at all steps and at all level to aspire to continuous improvement

  • 1.1 Senior Management is committed to QMS and continuously improving its effectiveness.
  • 1.2 Oversight bodies are established and receive key information to allow for monitoring of objectives, strategies and results.
  • 1.3 The Agency assigns resources that are sufficient to implement and maintain the QMS to ensure the goals of continuous improvement are met.

LOE #2 - Focus on Results: The QMS meets the Agency's requirements and expectations

  • 2.1 A quality framework exists, including a quality policy, for establishing and reviewing quality objectives.
  • 2.2 The Agency has identified and documented the processes needed for the quality management system and their application throughout the organization.
  • 2.3 A Quality Manual is controlled and maintained and includes the scope of QMS and documented procedures.
  • 2.4 Management identifies, assesses and responds to risks to QMS.
  • 2.5 The QMS is flexible enough to adapt to changing inspection activities.

LOE #3 - Monitoring and Continuous Improvement: Effectiveness of controls is assessed by management and the results are communicated to those whom are accountable

  • 3.1 Methods for the identification and control of quality losses are in place.
  • 3.2 The collection and analysis of data is being carried out to evaluate areas of improvement.
  • 3.3 Corrective action is taken to eliminate the cause of nonconformities in order to prevent recurrence.
  • 3.4 Quality Verifications are carried out in accordance with annual work plans.
  • 3.5 Management monitors actual performance against planned results and adjusts course as needed.
  • 3.6 The Agency plans and implements the monitoring, measurement, analysis and improvement processes for QMS.

Appendix B: Quality Verification and Improvement Cycle

Flowchart - Quality Verification and Improvement Cycle. Description follows.
Description for Quality Verification and Improvement Cycle

Step 1: Develop Annual Quality Verification Plan
Step 2: Conduct Verification
Step 3: Review of Data
Step 4: Identify Quality Loss Issues
Step 5: Complete Root Cause analysis
Step 6: Develop and implement improvement actions
Follow-up
Evaluate the impact of the plan
Review of Data, Analysis and Implementation of improvement actions
Quarterly Reporting

Source: CFIA QMS Manual – Quality Verification and Improvement Cycle

Date modified: