Language selection

Search

Archived - Food Safety Action Plan Evaluation

This page has been archived

Information identified as archived is provided for reference, research or record-keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Contents

Appendices available from Audit and Evaluation Branch on request

Abbreviations

AEB
Audit and Evaluation Branch
AIRS
Automated Import Reference System
CAP
Canadian Agricultural Products
CBSA
Canada Border Services Agency
CFIA
Canadian Food Inspection Agency
CIHR
Canadian Institutes of Health Research
CGII
Canada Gazette Part II
CMB
Corporate Management Branch
DEHP
Di-ethyl hexyl phthalate
DPR
Departmental Performance Report
EBP
Enabled Business Projects
EAC
Evaluation Advisory Committee
EPA
Effective Project Approval
FCSAP
Food and Consumer Safety Action Plan
FFVP
Fresh Fruit and Vegetables Program
F/P/T
Federal/provincial/territorial
FSAP
Food Safety Action Plan
FTE
Full-Time Equivalent
GDP
Gross Domestic Product
GFSI
Global Food Safety Initiative
GoC
Government of Canada
HC
Health Canada
HS
Harmonized System
IFS
Imported Food Sector
IM
Information management
IST
Import Surveillance Team
IT
Information technology
MAF
Management Accountability Framework
MC
Management Committee
MOU
Memorandum of Understanding
NFRS
Non-federally registered sector
OGD
Other Government Departments
O&M
Operations and maintenance
PHAC
Public Health Agency of Canada
PMO
Project Management Office
PWGSC
Public Works and Government Services Canada
P/Ts
Provinces and Territories
RASCI
Responsibility, Accountability, Support, Consulted and Informed
RMAF
Results Based Management Accountability Framework
SC
Steering Committee
SPAC
Senior Project Advisory Committee
USFDA
United States Food and Drug Administration

Executive Summary

The Food and Consumer Safety Action Plan (FCSAP) was announced in December 2007 by the Government of Canada. It encompasses a series of initiatives to modernize and strengthen Canada's safety system for food, health and consumer products, and to better support the collective responsibilities that government, industry and consumers have for product safety.Footnote 1 The Food Safety Action Plan (FSAP) is a five-year (2008-2013)Footnote 2 project for the Canadian Food Inspection Agency (CFIA or "Agency") and is a major component of the Government of Canada's broader FCSAP.

The announcement of FCSAP was made following a number of significant, high-profile international safety incidents associated with food, health and consumer products. These incidents included the discoveries of lead in children's toys and melamine in infant formula, as well as the global withdrawal of some prescription medications. The announcement of FCSAP was the Government of Canada's (GoC) response to these events.

The CFIA's food portion of FCSAP is referred to as FSAP. As noted above, the CFIA has the primary responsibility for implementing FSAP, with support from Health Canada's Health Products and Food Branch and the Public Health Agency of Canada (PHAC). Food products are anything that people eat or drink, and includes chewing gum, bottled water and ingredients to be used in food.

The implementation of FSAP was a significant project for the CFIA. The Agency faced numerous challenges throughout the implementation of FSAP, particularly in the first two years. This period entailed less 'doing' and more 'planning' than had initially been intended. The implementation delays affected progress, ultimately resulting in a number of outputs that cannot be achieved within the current five-year period and limiting the progress towards achievement of outcomes. The Project Management Office (PMO) was established in 2009 to manage the implementation of FSAP. The director of the PMO reports to the Vice President, Corporate Management Branch.

Good performance measurement data are available for tracking the achievement of outputs and reporting in the Departmental Performance Report and the Report on Plans and Priorities. However, there is little data available to support the achievement of outcomes. This is partly attributed to the fact that a formal performance measurement strategy was not fully implemented and that performance targets were not clearly identified at the outset of FSAP.

Many individuals interviewed throughout this evaluation indicated that inspection modernization is currently a significant undertaking, not unlike FSAP. Due to the experiences with FSAP, the Agency is better equipped to manage the implementation of inspection modernization. In fact, many interviewees noted that the lessons learned from FSAP implementation with regard to project management and governance have directly informed inspection modernization.

In the second year of the project, the Agency identified the need to re-assess the deliverables as originally envisioned. Corrective actions were taken to ensure the success of FSAP. The PMO assured the delivery of FSAP within the five-year time frame (2008-09 to 2012-13), more specifically that the activities would be achieved on time and within budget (except for those that are dependent on legislative and regulatory changes). The passage of the Safe Food for Canadians Act in November 2012 strengthens the FSAP work of the CFIA.

This evaluation identified several positive findings. However, the need to improve outcome level performance measurement is critical to determining whether FSAP has made progress towards accomplishing the incremental impacts that it was intended to have. The FSAP 'steady state' period would be an appropriate time to begin monitoring the impacts of all the work completed during the five-year FSAP implementation period.

The CFIA consults regularly with industry regarding FSAP. Although appreciative of the level of consultation, industry requires more information on how FSAP fits with all the food safety initiatives being implemented by the Agency. The resulting holistic picture will allow industry to provide the most up-to-date information to its members.

The recommendations provided in this evaluation report are meant to inform future similar initiatives as well as FSAP through the transition period. The lessons learned were self-identified during the implementation of FSAP and have already informed new CFIA initiatives.

The details associated with findings, conclusions, and recommendations and lessons learned are located in Sections 3, 4 and 5 respectively. The following is a summary of these findings, conclusions, recommendations and lessons learned.

Finding: There is little documented evidence to support the achievement of expected outcomes by March 31, 2013. However, there are examples of specific successes at the activity level. There was little evidence suggesting that outcome data were collected or that there was a mechanism to collect such data on a systematic basis. Based on the data collected through the evaluation, there are isolated examples of short-term outcomes. Based on this finding and conclusion, it is recommended that the CFIA develop and formally implement an outcomes based performance measurement strategy for FSAP activities (Recommendation 1.1).

Finding: There are very little data regarding the cost-effectiveness of FSAP delivery. Limited data suggest that the costs of conducting certain activities were underestimated. The cost-effectiveness of FSAP activities could not be determined due to the lack of useful data. Alternate ways of undertaking the activities were not identified. As recommended previously, the CFIA should develop and formally implement an outcomes based performance measurement strategy for FSAP activities. (Recommendation 1.1 and Lessons Learned 1.2)

Recommendation 1.1: Develop and formally implement an outcomes based performance measurement strategy for FSAP activities.

Finding: Industry struggles to understand how the various food safety initiatives are related. Industry appreciates that it has been consulted on food safety issues, but has difficulty in understanding how various food safety initiatives are related. The Agency needs to provide industry with a mechanism to ensure businesses understand holistically what these initiatives mean to them. (Recommendation 1.2)

Recommendation 1.2: Develop an approach to better communicate with industry how the range of food safety initiatives fit together.

Finding: The majority of funding was used for its intended purposes; however, not all allocations were aligned to the original budget plans. Limited planning leading up to and in the early years of FSAP implementation resulted in challenges with spending budget allocations on a timely basis. Once created, the PMO managed budget issues. However, funding continued to be lapsed throughout the five years. This finding and conclusion leads to the lesson learned that major initiatives are best managed by a project management office from initiation, as good project management can better support spending within budget allocations. (Lesson Learned 1.1)

Finding: FSAP was a lesson in horizontal project management for the Canadian Food Inspection Agency. The implementation of FSAP was a challenge for the Agency due to its lack of experience in project management. This would partially explain the challenges faced during the first two years. The introduction of the PMO helped to address most of the issues (Lesson Learned 1.1).

Finding: The FSAP governance structure and decision-making process has improved since undergoing restructuring in 2011. The initial governance structure, which was in place until February 2011, was ineffective in comparison with the current design. The governance structure comprised too many committees and the members with decision-making authority often were not present at meetings. Since governance restructuring, including the expansion of the PMO in February 2011, decision making has improved, and roles and responsibilities are generally clearer. An important lesson learned was that major initiatives are best managed by a project management office from the time of initiation (Lesson learned 1.1).

Lesson Learned 1.1: Major initiatives are best managed by a project management office from initiation.

Finding: A clearly identified need was established prior to the announcement of the Food Safety Action Plan. Although the initiative was announced primarily in response to major international food safety issues, there was evidence of due diligence to provide support for the need for the activities comprising the FSAP. This due diligence consisted chiefly of demonstrating the complexity of a growing global food trade and illustrating Canada's volume in that trade. An important lesson learned in arriving at this conclusion was that an analysis of Agency and market skills and availability, as well as of the costs of implementation, better supports the implementation of initiatives (Lesson Learned 1.2).

Lesson Learned 1.2: An analysis of Agency and market skills and availability, as well as costs of implementation ensures better support for the implementation of initiatives.

1 Introduction and Context

1.1 History of the Food Safety Action Plan

The Food and Consumer Safety Action Plan (FCSAP) announced in December 2007 by the Government of Canada. It encompasses a series of initiatives to modernize and strengthen Canada's safety system for food, health and consumer products, and to better support the collective responsibilities that government, industry and consumers have for product safety.Footnote 3 The Food Safety Action Plan (FSAP) is a five-year (2008-2013Footnote 4) project for the Canadian Food Inspection Agency (CFIA or the "Agency") and is a major component of the Government of Canada's (GoC) broader FCSAP.

The announcement of FCSAP was made following a number of significant, high-profile international safety incidents associated with food, health and consumer products. These incidents included the discoveries of lead in children's toys and melamine in infant formula, as well as the global withdrawal of some prescription medications. The announcement of FCSAP was the GoC's response to these events.

The implementation of FCSAP has been carried out under the responsibilities of the Minister of Health and Minister of Agriculture and Agri-Food Canada. As such, the following four federal departments and agencies have been responsible for FCSAP activities:

  1. Health Canada (HC);
  2. the Canadian Food Inspection Agency (CIFA);
  3. the Public Health Agency of Canada (PHAC); and
  4. the Canadian Institutes of Health Research (CIHR).

Activities related to consumer products and some health products are the responsibility of HC, PHAC, and CIHR. Consumer products are products that adults and children commonly use for personal, family, household or garden purposes or in recreation or sports. Health products include human and veterinary drugs, vaccines, blood and blood products, natural health products, biologics and genetic therapies, medical devices and other therapeutic products.

The CFIA's food portion of FCSAP is referred to as the FSAP and, as noted above, CFIA has the primary responsibility for implementing FSAP, with support from HC's Health Products and Food Branch and from PHAC. Food products refer to anything that people eat or drink, and includes chewing gum, bottled water and ingredients to be used in food.

The primary focus of FSAP was initially on imported ingredients, imported produce, mycotoxins in cereals, undeclared allergens, and imported and domestic plant or animal-derived food products in the non-federally registered sector (NFRS). However, the focus of FSAP activities was eventually expanded to include the imported food sector which encompasses imported agricultural products intended for use as food or an ingredient in food.

The Agency received a total of $223.4 million in incremental funding for FSAP over a five-year period (2008-09 through 2012-13) and $54.9 million per year after that (see Table 1). A spending gap of $38.8 million is anticipated. This is discussed further in Section 3.1.

Table 1: CFIA Funding for FSAP
Year 1
2008-09
Year 2
2009-10
Year 3
2010-11
Year 4
2011-12
Year 5
2012-13
Total FSAP funding On-going support
Plan Actual Plan Actual Plan Actual Plan Actual Plan Actual Plan Actual Plan
$
million
22.9 14.2 39.3 33.2 52.4 42.0 52.4 40.3 56.4 55.0 223.4 184.6 54.9

Source: FSAP Deliverables Status Report: Financial Table (forecast as of September 30, 2012; CFIA internal document)

1.1.1 Project Management Office

The PMO was established in 2009 to manage the implementation of FSAP. The director of the PMO reports to the Vice President, Corporate Management Branch. Additionally, the Business Sponsor, who is not directly part of the PMO, but who has a role in FSAP delivery reports to the Vice President, Policy and Programs Branch. There are also a number of additional managers and staff with roles and responsibilities.

A unique feature of the PMO is the Business Team. The Business Team is comprised of Business Leads, who are the liaisons between the PMO and the Food Business Line.

Transition of the components and activities of the FSAP Initiative into regular Agency operations is currently underway. A nine-month extension to FSAP was approved by the Senior Project Advisory Committee (SPAC) in December 2012. It is anticipated that on December 31, 2013 the transition will be nearing completion.

The logic model, developed in 2011, (Figure 1) illustrates the linkages between FSAP's 12 activity areas and the expected outputs and outcomes.

Figure 1: Food Safety Action Plan Logic Model

Click on image for larger view
Food Safety Action Plan Logic Model. Description follows.

Description for image - Food Safety Action Plan Logic Model

The Food Safety Action Plan Logic Model has six levels:

  1. Pillars
  2. Activities
  3. Outputs
  4. Immediate Outcomes
  5. Intermediate Outcomes
  6. Ultimate Outcomes

There are three pillars at the first level:

  1. Active Prevention
  2. Targeted Oversight
  3. Rapid Response

The pillar active prevention points to four activities (level 2)

  1. Better understanding of food safety risk: risk mapping and baseline surveillance, which points to the outputs (level 3):
    • Risk profiling database, data for risk mapping, MOUs (access info), targeted surveys, improved and validated lab methods, re-engineered risk prioritization, profiling and mapping processes, baselines, which points to the immediate outcome (level 4):
    • Improved ability to identify, assess, and prioritize potential food safety risks.
  2. Working with industry to implement effective risk mitigation: development and implementation of food safety control systems by industry, implementation of importer licensing, enhanced offshore efforts/collaboration, which points to the outputs (level 3):
    • Industry consultations, evaluation and verification approaches, guidance to industry, regulatory consultations, identified importers, importer licensing policy, business model, industry awareness, trading partner collaboration and MOUs, risk mitigation strategies, which points to the immediate outcome (level 4):
    • Increased capacity to understand the importer industry and to work with industry on the design and implementation of effective risk mitigations strategies.
  3. Strengthening standard setting and appropriate regulatory backstops: strengthened regulatory backstops, which points to the outputs (level 3):
    • Gap analysis, Canadian Agricultural Products Act, strategy/plan, new Canadian Agricultural Products Regulations, which points to the immediate outcome (level 4):
    • Enhanced legislative and regulatory base related to food safety.
  4. Engaging Canadians in labelling decision-making: public consultation and engagement tools, "Product of Canada" labelling, which points to the outputs (level 3):
    • Consultation with Canadians, new "Product of Canada/Made in Canada" labelling policy, food safety labelling requirements, which points to the immediate outcome (level 4):
    • Improving understanding of labelling by the public.

The pillar targeted oversight points to two activities (level 2):

  1. Enhancing inspection of high-risk sectors: enhanced inspection of high risk sectors, which points to the outputs (level 3):
    • Inspections and verification of industry food safety control systems, core food safety control system criteria, new hires and delivery of training, new inspection methods, which points to the immediate outcome (level 4):
    • Increased oversight of high risk sectors.
  2. Targeting import control measures: border blitzes, enhanced tracking of imports, which points to the outputs (level 3):
    • Port of entry or border blitzes, new hires and delivery of training, comprehensive import control program and tracking system, which points to the immediate outcome (level 4):
    • Increased ability to prevent unsafe food from entering Canada.

The pillar rapid response points to two activities (level 2):

  1. Enhancing recall capacity, which points to the outputs (level 3):
    • Investigations and recalls, new hires and delivery of training, validated recall and investigation methodology, which points to the immediate outcome (level 4):
    • Strengthened capacity to respond to food safety investigations and food recalls.
  2. Targeting consumer risk communication, which points to the outputs (level 3):
    • Assessment of results of targeted surveys, disseminated food recalls and allergy alerts, targeted food safety messaging, national safe food handling program, which points to the immediate outcome (level 4):
    • Improved consumer awareness of food safety issues and recalls.

Each immediate outcome described above points to the same four intermediate outcomes (level 5):

  1. Enhanced capacity to identify and mitigate risks.
  2. Better prioritized inspection, improved compliance and enforcement.
  3. Improved consumer engagement and ability to make informed decisions.
  4. Increased industry and stakeholder involvement in the safety of food.

The four Intermediate outcomes point to one ultimate outcome (level 6):

  1. Public Health risks associated with the food supply and transmission of animal diseases to humans are minimised and managed; Modernized and strengthened food safety system and increased collaboration and information sharing among government partners, industry and consumers.

1.2 Performance Measurement of the Food Safety Action Plan

A Results-based Management Accountability Framework (RMAF) for the FCSAP was prepared in 2008.Footnote 5 In 2010, the PMO developed a performance measurement system specific to FSAP. This performance measurement system was not finalized, owing in part to the lack of targets defined by program areas at the outset of the initiative, based on documentation and comments received during the evaluation process.Footnote 6

Despite the lack of a formal or final FSAP performance measurement system, it should be noted that the PMO has been effective in collecting performance data, even though it is working with draft performance measures. The PMO is responsible for collecting performance information as it relates to the FSAP activities associated with each of the CFIA's branches for government reporting purposes. The PMO has developed templates that are used to collect consistent performance data about FSAP deliverables on a quarterly basis from the CFIA's branches. At the end of each year, the PMO uses these performance data as input to for the HC and the CFIA DPRs.

Although useful for measuring the performance of activities and outputs, this approach has not been effective in documenting progress towards achieving outcomes. There are several factors affecting the lack of outcome data. For example, the current stage of activities within FSAP is trailing originally planned timelines, making it difficult to measure even short term impacts or results. Another reason for the lack of outcome data is that performance reporting was not designed to collect data at the outcome level. As further described later in this report, the initial two years of FSAP represented a period of change for the Agency. The initial project management framework was found to be inadequate for managing FSAP and was modified in 2010. As a result, progress was slow early on. While some short-term successes have been demonstrated through the data collected as part of this evaluation, many longer-term successes will not be realized before March 31, 2013, which is the five-year mark for FSAP.

The lack of outcome performance data is a concern given the Agency's objective of moving towards outcome-based programming and regulation. In this model the Agency will evolve from prescribing requirements (to industry) to specifying expected performance. This means that in the future it will be important for programs to show, through documented evidence, that their activities have made an impact toward end-state objectives, rather than tracking whether individual tasks have been completed.

1.3 Objectives of the Food Safety Action Plan

The objectives of the FSAP are as follows:

Twelve activities were carried out across eight priority areas, as follows:

  1. Better Understanding of Food Safety Risk
    1. Risk Mapping and Baseline Surveillance
  2. Working with Industry to Implement Effective Risk Mitigation
    1. Development and Implementation of Food Safety Systems by Industry
    2. Implementation of Importer Licensing
    3. Enhanced Offshore Efforts
  3. Strengthening Standard Setting and Appropriate Regulatory Backstops
    1. Appropriate Regulatory Backstops
  4. Engaging Canadians in Labelling Decision-Making
    1. Public Consultation and Engagement Tools
    2. Product of Canada Labeling
  5. Enhancing Inspection of High-Risk Sectors
    1. Enhanced Inspection of High Risk Sectors
  6. Targeting Import Control Measures
    1. Border Blitzes
    2. Enhanced Tracking of Imports
  7. Enhanced Recall Capacity
    1. Enhanced Recall Capacity
  8. Targeted Consumer Risk Communications
    1. Targeted Consumer Risk Communications

The eight priority areas are grouped under the three FCSAP themes:

  1. Active Prevention;
  2. Targeted Oversight; and
  3. Rapid Response.

The initial scope of FSAP as defined in 2008 was redefined in April 2010 via the FSAP Project Scope Document.Footnote 8 This document is the cornerstone that guides all current FSAP activities and also provides foundational information as it was understood in 2008-2009. Intended to be used as a reference document for persons governing, working in or reviewing the progress of the FSAP project. The Scope document is one of a suite of project management documents designed to facilitate the management of the project. The specific objective of this document is to prevent scope creep by providing the baseline against which to request changes to the scope. The document provides descriptions and guidance as to the deliverables, activities, required resources (financial and human), and broad timelines of all 12 areas of activity of the FSAP project.

Activities that are considered to be outside the scope of FSAP under each of the activity areas are described in Table 2. It should be noted that work may be considered beyond the scope of FSAP, but not necessarily outside the Agency's long-term vision of food safety.

Table 2: Summary of Work beyond the Scope of FSAP's Five-Year Period (2008-09 to 2012-13)
Area of Activity Beyond Scope
1. Risk Mapping and Baseline Surveillance
  • Risk mapping of commodities outside the NFRS, other than the four initial priority areas (produce, mycotoxins, undeclared allergens and imported ingredients) and those identified through the risk prioritization and profiling processes
  • Targeted surveys on commodities outside the four priority areas and those identified through the risk prioritization and profiling  processes
  • Non-plant or animal-derived commodities in the NFRS (e.g. salt, bottled water, etc.), based on interview data
2. Development and Implementation of Food Safety Systems by Industry
  • Registration of domestic establishments
  • Education and awareness campaigns for both domestic manufacturers and importers
  • Guidance documents for specific commodities beyond the four priority areas and those identified as priority through risk mapping processes
3. Implementation of Importer Licensing
  • Food importer education campaigns
  • Identification of foreign suppliers
4. Enhanced Offshore Efforts
  • Memorandum of Understanding (MOUs) with all emerging source countries
  • Verification of food safety requirements at source
  • MOUs as the only means of agreement, based on interview data
5. Appropriate Regulatory Backstops
  • No elements considered beyond scope
6. Public Consultation and Engagement Tools
  • Planning and delivery of consultations on policy beyond the scope of the FSAP areas of activity
7. Product of Canada Labeling
  • Long-term policy amendment activities
8. Enhanced Inspection of High-Risk Sectors
  • Training programs outside those necessary to train inspectors to enhance delivery of programs for the high priority areas identified above
  • Inspection and verification of industry food safety control systems outside the scope of the areas of activity of the FSAP project
  • Inspection of high-risk sectors in foreign countries
9. Border Blitzes
  • Border activities related to commodities with traditionally robust import programs (e.g. meat and fish)
10. Enhanced Tracking of Imports
  • Commodities with traditionally robust import programs (e.g. meat and fish)
11. Enhanced Recall Capacity
  • Enhanced resources to respond to all recalls
12. Targeted Consumer Risk Communications
  • Additional resources for internal dissemination of information on FSAP

In order to implement some of the FSAP activities, such as importer licensing and tracking, the Agency has undertaken a series of IM/IT enabled business projects (EBP) to build a licensing application and re-platform an import control application. The EBPs are outside of the scope of this evaluation. The systems are expected to begin operation in the latter part of fiscal year 2013-14 and are largely dependent on the approval of the IFS regulations.

1.4 The Food Safety Action Plan in Relation to Other Food Safety Initiatives

The FSAP Program Design Document (December 2011) which outlines the interdependencies among and between the FSAP areas of activity and is used to facilitate the management of FSAP states that:

"FSAP measures have been designed to support the better identification of risks in the food supply, to build in preventative risk mitigation approaches from the start, and to use a targeted oversight approach to verify that industry's preventative approaches are effective and there is a rapid response when problems do occur. As a result, FSAP [is]… providing a foundation for future investments related to enhanced surveillance and inspection modernization and legislative and regulatory renewal." Footnote 9

According to the FSAP Program Design Document (December 2011), other food safety initiatives are potentially affected by FSAP. This is because FSAP is not a program in-and-of itself but rather an umbrella of activities affecting the work of the entire Agency with regard to food, either directly or indirectly. The other initiatives potentially affected include the following:

1.5 Legislation and Regulations Related to FSAP

Two major components of FSAP were related to the development and revision of legislation and regulations. The first was to establish new legislation. The primary purpose was to "…make food as safe as possible for Canadians, in part by targeting unsafe practices, increasing penalties and requiring licensing and registration."Footnote 11 The new Safe Food for Canadians Act amalgamates four statutesFootnote 12 and modernizes food safety. The Act provides the legislative authority required for new IFS regulations, which is the second component, and received royal assent on November 22, 2012.

The IFS regulatory activities consisted of using the regulation-making authority under the Canadian Agricultural Products Act (CAPA) to provide additional oversight to imported food in the NFRS, by adding new regulations and effectively requiring importers to register for a licence. The IFS regulations are a key item under FSAP, because many other components are dependent on their approval.

During the five-year period of FSAP, the drafting of these regulations experienced several delays that have delayed overall FSAP progress. An initial draft of the regulations went through the approval process in 2009-10; however, several issues affected final approval. When the draft regulations neared final approval, they were identified as 'high impact',Footnote 13 which triggered the need for rigorous additional research evidence. This included a full literature review and cost-benefit analysis. This additional research affected the originally anticipated approval date of the regulations.

The other issue that arose at the same time is that as part of the Small Business Lens initiative,Footnote 14 the GoC's requirements for regulatory packages are now more stringent. Once the Agency's cost-benefit analysis work was approved, it had to be validated with stakeholders. This process was completed in November 2012.

Figure 2 illustrates the timeline for the royal assent of the Safe Food for Canadians Act and the implementation of IFS regulations. It should be noted that the dates shown beyond 2012 are anticipated or estimated only. As FSAP has been extended until December 31, 2013, it is expected that this work will be complete by that date and full implementation will begin on January 1, 2014.

Figure 2: Timeline for Implementation of the Safe Food for Canadians Act and the IFS Regulations
Timeline for Implementation of the Safe Food for Canadians Act and the Imported Food Sector Regulations. Description Follows.
Description for image - Timeline for Implementation of the Safe Food for Canadians Act and the IFS Regulations

In October 2012:

  • The Safe Food for Canadian Act in Parliament.
  • Imported food sector regulations published in Canada Gazette II.

In November 2012:

  • Royal Assent of the Safe Food for Canadians Act.

In Winter/Spring 2013:

  • Ministerial approval of final imported food sector regulations.

In October 2013:

  • Imported food sector regulations were coming into force.
  • Importer licensing was implemented.

In January 2014:

  • Border restrictions were implemented.

1.6 Food Safety Action Plan Governance

Changes in the governance structure were made between 2008 and 2011. The initial governance structure (2008) consisted of a vice-president level steering committee; a director/executive director level management committee; and six horizontal working groups representing the areas of work. In February 2011, a new governance structure was implemented, consisting of the FSAP Senior Project Advisory Committee (SPAC); the FSAP Steering Committee (SC); and the FSAP Management Committee (MC). Table 3 provides a brief overview of the roles and responsibilities of each of these committees, while Figure 3 illustrates the FSAP governance structure and how it links back to the larger CFIA governance structure through the Senior Management Committee of the Agency.

Table 3: Summary of Roles and Responsibilities of Current FSAP Governance Bodies Footnote 15
Committee Description of Roles and Responsibilities
Senior Project Advisory Committee (SPAC)
  • Responsible for project oversight, gating, monitoring and reporting, and improving project management capability.
  • Major decisions come before the SPAC for approval. The SPAC provides recommendations to the CFIA Human Resources and Corporate Management Committee (HRCMC).
  • The Committee (HRCMC) is chaired by the Project's Executive Sponsor, the VP of the Policy & Programs Branch. The SPAC's secretariat services are provided by the FSAP PMO.
  • Meetings are monthly and can be called as required.
Steering Committee (SC)
  • Responsible for the FSAP budget (though SPAC provides approvals).
  • Responsible for enabling project success through appropriate resource allocation, communication and project oversight.
  • Provides oversight, coordination and guidance on implementation of the FSAP project plan.
  • Meetings are scheduled biweekly and can be called or cancelled depending on need.
  • Makes recommendations to the SPAC and is chaired by the Project's Business Sponsor, the Associate VP of Strategic Projects and Integration Sector. The SC's secretariat services are provided by the PMO.
Management Committee (MC)
  • Preliminary level of collaborative, consensus-based decision making, providing a challenge function between branches.
  • Responsible for delivering FSAP and supporting CFIA functions.
  • Members have the delegated authority to conduct the necessary collaborative activities, represent their organization's needs and contribute to the collaborative planning and delivery required to meet objectives.
  • Makes recommendations to the SC.
  • Chaired by the Project leader, the Senior Project Director, FSAP PMO. The MC's secretariat services are provided by the PMO.
Figure 3: Current FSAP Governance Structure
Current Food Safety Action Plan Governance Structure. Description follows.
Description for image - Current FSAP Governance Structure

Figure three outlines the current governance structure for FSAP.

The FSAP Management Committee reports to the FSAP Steering Committee and the FSAP Steering Committee informs the FSAP Senior Project Advisory Committee. These three FSAP committees report to the FSAP Project Management Office, which in turn informs the Treasury Board Secretariat Chief Information Officer Branch. In addition, the FSAP Project Management Office reports to Enterprise Project Management, which informs the Human Resources and Corporate Management Committee and the CFIA Senior Management Committee.

The chair of the FSAP Senior Project Advisory Committee informs the Food and Consumer Safety Action Plan Steering Committee. The chair of the FSAP Steering Committee informs the Food Task Force and the Food and Consumer Safety Action Plan Coordinating Committee.

Source: Canadian Food Inspection Agency. Food Safety Action Plan: Program Design. Version 1. December 20, 2011

The decision-making process follows the hierarchy among the committees, beginning with the MC and ending with the SPAC. The MC makes decisions on how work will proceed and presents such options for consideration by the SC. The SC makes consensus-based decisions on which options will be brought forward. It also decides when options or paths may be altered or discarded. When consensus decisions cannot be made, the issues are presented to the SPAC by the Chair of the SC. The SPAC normally decides on the issue but on occasion may require the CFIA Human Resources and Corporate Management Committee to rule on issues. Such occasions would typically relate to budget estimates that change as a result of a better understanding of the size of the regulated community and the data, reporting, and maintenance impacts on the IM/IT needs, as well as after the licensing pilot is completed.Footnote 16 Terms of reference exist for each of the committees.Footnote 17

1.7 Other Audits and Evaluations Related to the Food Safety Action Plan

Reports of the Auditor General in December 2000 and March 2004 noted that Canada's regulatory system was no longer adequate to deal with the rapidly expanding and complex global marketplace. The December 2000 audit report stated that "…the Agency needs a strategic approach to manage imported commodities."Footnote 18 The report also highlighted the need to focus on managing the NFRS, and to develop a more comprehensive approach to assessing risks. The March 2004 report made similar observations related to health and safety against risks.

More recently, an internal audit of the FSAP was completed by the CFIA in 2011-12. The scope of the audit was initially to focus on the first 2.5 years of FSAP implementation. On guidance by the CFIA Audit Committee, the period was extended by 10 months, spanning a timeframe of April 2008 to October 2011. The reason for the increased scope was to provide the opportunity to review more recently implemented practices and processes associated with the management of FSAP, which were considered significant. Further, the extension made it possible to review the FSAP Transition Plan, which is the plan for transitioning from the five-year implementation period to a point at which some activities would become part of regular CFIA activities (steady state) beginning on April 1, 2013.Footnote 19 This Plan currently exists in draft form and will not be finalized until fall of 2013, given that the SPAC has extended FSAP to December 31, 2013.

The audit noted that:

"the FSAP initiative is currently in its fourth year. While there has been much progress to enhance the overall project management framework being employed, we found that some components could not be fully implemented as intended and need more time to mature. Commencing April 2010, there was a shifting emphasis within the Agency towards a more focused approach on how projects are to be managed. In particular, there was greater engagement by Senior Management on the status, success and progress of large Agency initiatives, such as FSAP. This led to the formal roll-out of an Agency Enterprise Project Management Office (ePMO), which provides tools and templates as well as training on various aspects of Project Management to Agency personnel." Footnote 20

The evaluation has confirmed this observation.

1.8 Evaluation Objective and Scope

Evaluation Objective

The evaluation of the FSAP was identified as a priority in the CFIA's Evaluation Plan (March 2009; 2010). As required by the Treasury Board Secretariat (TBS) Evaluation Policy (2009) and in accordance with the supporting directive and standard, the primary objective of this evaluation was to assess the relevance (continued need, alignment with government priorities, and alignment with federal government roles and responsibilities) and performance (achievement of expected outcomes) as well as demonstration of efficiency and economy of FSAP and to provide recommendations to improve effectiveness and efficiency, as necessary. This evaluation will inform HC's evaluation of FCSAP.

Evaluation Scope

This evaluation covers the period from April 2008 to November 2012 and encompasses all of the activities for which the CFIA received funding, excluding the IM/IT components of FSAP. In order to implement some of the FSAP activities, such as importer licensing and tracking, the CFIA has undertaken a series of IM/IT enabled business projects (EBP) to build a licensing application and re-platform an import control application. The EBPs were outside of the scope of this evaluation.

2 Methodology

The evaluation approach and methodology were developed in consultation with an evaluation advisory committee (EAC). The EAC was co-chaired by the CFIA's Head of Evaluation and the Executive Director of Food Safety Strategies Directorate. The EAC was composed of representatives from the CFIA's branches (Policy and Programs, Operations, Science, Public Affairs, Human Resources, Corporate Management, and Audit and Evaluation), as well as an external committee member with evaluation expertise.

The evaluation was undertaken in three phases:

  1. In the first phase (September-October 2011), the evaluation team undertook a scoping exercise and developed a detailed evaluation framework, including specific evaluation questions, indicators, and corresponding data sources and data collection methods.
  2. The second phase (November 2011-November 2012) consisted of fieldwork to gather the necessary information and data.
  3. In the third phase (November-December 2012), the evaluation team analyzed and integrated the collected information and data and then synthesized and reported the evaluation findings. EAC committee members provided input and feedback on preliminary findings and on the draft evaluation report. Any clarifications and revisions necessary in light of the comments received were made accordingly.

The evaluation methodology incorporated the collection and review of data and information obtained from multiple sources and through multiple methods in order to facilitate triangulation and cross-examination of findings from multiple lines of evidence for each evaluation question. The following sections present an overview of the evaluation methods and their limitations. An evaluation matrix, presenting the eight evaluation questions, associated indicators, proposed methodologies and data sources is provided in Appendix A.

2.1 Methods

Each method was selected for a specific purpose, as described in Table 4.

Table 4: Methods and Purpose
Methods Purpose
Documentation Review
  • Appropriate for responding to evaluation questions that require the same or similar data as those required for the draft performance measurement strategy.
  • Useful for gathering an initial understanding of changes (that were documented) over the course of the FSAP's history and for supplementing other data sources.
Key Informant Interviews
  • Appropriate for understanding the key achievements, issues, challenges and other nuances that are likely not identifiable in documentation.
  • ppropriate for gathering the perspectives of internal stakeholders in FSAP.
Focus Group
  • Appropriate for gaining in-depth information on perceptions, insights and specific issues from a group of individuals.
Industry Interviews
  • Appropriate for collecting data related to some of the positive and negative effects of FSAP, with a focus on outcomes.
  • Appropriate for gathering the perspectives of external stakeholders of the FSAP.
Case Studies
  • Appropriate for illustrating FSAP impacts and lessons learned.
  • In part, selected to offset a lack of outcome level data, a potential risk identified during evaluation design.

2.1.1 Documentation Review

In order to complete the documentation review, the PMO was contacted at the outset of the evaluation. The PMO possessed many of the documents including performance data templates. As such, the annual reports and DPR were critical documents that were reviewed.

The documentation most relevant to this evaluation belonged essentially to one of the following two types:

  1. Planning/project management documents; or
  2. Results or performance reports.

Although these were the two key categories of documents, a review of media reports and other government documents was also useful, as they were data sources external to the CFIA. In total, 42 documents were reviewed as part of this methodology. The complete list of documents reviewed is located in Appendix C.

2.1.2 Key Informant Interviews

A total of 31 interviews were conducted with key informants identified by the CFIA, representing all levels of management across the following:Footnote 21

An interview guide was sent to the interviewees in advance. It should be noted that some questions were expanded depending on the interviewee's knowledge, background and role in FSAP, to ensure the most fruitful information as possible. A copy of the guide can be found in Appendix B; additional information about the key informant interviews can be found in Appendix D.

2.1.3 Focus Group

A focus group was conducted with the FSAP business leads (part of the PMO), individuals who liaise between the PMO and the CFIA's branches. These individuals, in addition to members of the EAC, were valuable in identifying other potential sources of data, including additional interviewees from across the Agency.

2.1.4 Industry Interviews

Interviews with four industry associations were conducted to gauge the extent to which some immediate outcomes have been achieved.Footnote 22 The interviewees were selected based on their previous engagement with the CFIA about FSAP, and on the size of the industry segments that they represent. Representatives of the largest industry associations with knowledge of FSAP were asked to participate.

The following industry associations were targeted:

  1. The Canadian Association of Importers and Exporters Inc (representing large and medium-sized importers and exporters across Canada and globally);
  2. The Canadian Supply Chain Food Safety Coalition (representing national organizations and associations, provincial and territorial groups and associations, and other allied groups);
  3. Food and Consumer Products of Canada (representing the food, beverage, food-service and consumer products industry); and
  4. The Retail Council of Canada (representing 43,000 storefronts of all retail formats across Canada, including grocery).

An interview guide and an introduction letter were sent to the interviewees before they were contacted by telephone to schedule interviews. A copy of the industry interview guide is provided in Appendix B.

2.1.5 Case Studies

Three case studies were conducted to illustrate how FSAP has had an impact on food safety in Canada. These cases were selected by asking interviewees to indicate potential incidents, situations or outputs that would demonstrate lessons learned about FSAP and illustrate its impact. A list of 10 potential case studies was developed. Based on a combination of the frequency with which the cases were mentioned and an estimation of the availability of data sources, the list was reduced to three.

Brief reports aimed at illustrating the challenges and benefits of implementing components of the FSAP were prepared, and are attached as Appendix E. Each case study consisted of a review of relevant documentation and interviews with key informants.

The three case studies, and their respective abstracts, are as follows:

Tsunami Response – On March 11, 2011, Japan experienced an earthquake measuring 9.0 on the Richter scale. A powerful tsunami subsequently took place in the Pacific coast, particularly at an area in the northern part of the country. These natural disasters combined to cause significant destruction and loss of life. In addition, the tsunami precipitated a meltdown at the Fukushima Daiichi nuclear power plant. The meltdown continued for weeks, with continual radiation leakage into the environment.

Crops and other food products on a number of farms in several prefectures in Japan, including the Fukushima prefecture, were at risk of contamination. This included vegetable products as well as raw milk. As a result, Canada implemented a policy to require additional export documentation from importers, certifying that products were safe.Footnote 23

Enhanced import controls were a key component of the CFIA response to mitigate the risk that Japanese products contaminated with radioactive substances would enter the Canadian market. These controls included a forum for importers to attest to the origin of the imported products; targeted testing of all imports arriving without an attestation form or without a certificate of analysis; and a random sampling plan for all other food and feed products imported from any prefecture in Japan.Footnote 24

In addition, the updating of Harmonized Service codes (conducted as part of FSAP) ensured that affected food products could be tracked to a finer level of detail. This allowed the CFIA to identify products for inspection by the Canada Border Services Agency (CBSA).

Plasticizer (DEHP) in Imported Foods – In June 2011, Taiwan's Department of Health notified the CFIA of the presence of di ethyl hexyl phthalate (DEHP) in some food products processed in Taiwan. The products in which DEHP was determined to be present included instant teas, fruit jellies, fruit juices, food powders and flavoured water. DEHP is a colourless, oily liquid plasticizer often used in flexible plastics such as those intended for the packaging of foods or food containers, but also in products such as flooring, imitation leather, rainwear, footwear, upholstery, wires and cables, tablecloths, shower curtains, toys and medical tubing.Footnote 25

Through targeted surveillance (funded by FSAP) and strong collaboration with international counterparts, the CFIA was able to efficiently and effectively manage this incident. The sharing of information between the CFIA, Taiwan and the U.S. permitted the development and adaptation of testing methods. Close cooperation with the United States Food and Drug Administration allowed sampling and testing of products in the targeted survey plan to be split between laboratories, ensuring efficient validation of methods and a timely response. The evidence indicated that the management of the incident was effective and completed at a cost less than what it potentially could have been in the absence of FSAP.

Targeted Surveys and Risk Communication – Targeted surveys consist of the sampling and testing of targeted food and hazard combinations to identify the presence of harmful allergens, chemicals or bacteria. Upon the completion of targeted surveys, reports on their results are produced by the Food Safety Division. This information is shared not only within the CFIA but with other departments and agencies as well as with the public. Public communications on the results of targeted surveys is done via several communications outlets, through the Public Affairs Branch of the CFIA. This communication is an important step in raising consumer awareness about food safety risks and issues, which is an outcome of FSAP.

2.2 Analysis and Integration of Data

Although the interviews were not designed or used to collect quantitative data, the interpretation of findings should take into account the extent to which certain perceptions or views were expressed or shared by the interviewees. Table 5 defines terms used in this report to quantify the proportion of interviewees who expressed similar experiences, views and opinions.

Table 5: Definitions of Terms Used to Quantify Qualitative Data
Term Proportion of Interviewees
Majority Findings reflect the experiences, views and opinions of more than 75% of those interviewed
Most Findings reflect the experiences, views and opinions of more than 50%, but no more than 75% of those interviewed
Many Findings reflect the experiences, views and opinions of more than 25%, but no more than 50% of those interviewed
Some Findings reflect the experiences, views and opinions of more than 10%, but no more than 25% of those interviewed
Few Findings reflect the experiences, views and opinions of less than 10% of those interviewed

Multiple methodologies were applied to collect data from multiple sources. Data were validated within and across methods. Where different methods or sources produced similar findings, those findings were assumed to have greater validity. Conversely, findings generated by a single method or source were treated with caution. This approach increases the validity of the evaluation findings by triangulating multiple lines of evidence.

2.3 Challenges, Limitations and Mitigation Strategies

The evaluation challenges and limitations and the corresponding mitigation strategies are described in Table 6.

Table 6: Evaluation Challenges, Limitations and Risk Mitigation Strategies
Challenges and Limitations Mitigation Strategies
The most significant limitation to this study is that some of the planned outputs have not yet been produced, and others have only recently been produced. Consequently, there has not been sufficient time since the production of outputs for many of the intended outcomes of FSAP to have occurred. In other cases, outputs were achieved by the end of Year 2 but the outcome data are not sufficient to draw meaningful conclusions. All data sources that could potentially yield information regarding outcomes were investigated. However, since many intended outcomes have not yet occurred, the reporting of information on outcomes is necessarily weak.
Although every attempt was made to collect and review all relevant documentation associated with FSAP and its performance, there is no guarantee that all such documents were reviewed. Several requests were made for documents believed to be of interest; however, there is a small risk that some number of important documents may not have been provided. Although there is no guarantee that 100% of the critical documentation was reviewed, the evaluators strongly believe that those documents with the most vital information were received and reviewed. Requests for documentation were made during interviews; further adding to our confidence that nothing critical was missed.
Regarding the industry interviews, while there was a preference to directly survey a sample of specific importers from across Canada. It was not feasible to take this approach, given that the CFIA does not yet know who all of the importers are. Instead, interviews were conducted with large industry associations. Although these organizations are assumed to reflect their members, there is no guarantee that the interviewees were able to gather opinions directly from their constituent members. Interviewees representing industry associations were given adequate time to seek input from constituents, and information about the interview was provided in advance as well. It is believed that most interviewees did indeed seek the input of their members directly; however, these organizations reflect the opinions of their members, and it is thus thought that the information that they provided reflects a large proportion of food importers in Canada.

Since the CFIA provided the names of the internal stakeholders interviewed for this evaluation, a selection or sampling bias is possible.

External stakeholders from other government departments were excluded.

The risk of selection bias is considered low overall. The names provided for interviews by the CFIA reflect the interviewees who represent mix of individuals who collectively provide a high level of breadth and depth of history and evolution of the FSAP. Interviewees from the very senior levels of the CFIA down to the manager level were identified and interviewed. Additionally, the interview guide was created such that the interviewees could speak to the questions to which they were best able to respond.

An interdepartmental evaluation of FCSAP is expected to include all government stakeholders (e.g. HC, PHAC and CIHR).

Outcome data were not systematically collected, meaning that there was no pre-existing data source that could be mined for information to assess performance in terms of achievement of outcomes. Primary data were collected through the interview process as well as through case studies. These methodologies provided a limited amount of data to support an assessment of some FSAP outcomes. Although, the interview data in this regard were largely anecdotal, the case studies yielded more concrete evidence.

3 Key Findings and Conclusions

3.1 Introduction and Context

Currently, FSAP is in the final year of a five-year implementation period (2008-09 to 2012-13). The evaluation has identified numerous areas of progress in producing deliverables; however, not all of the intended outputs will be achieved in the first five-year period as expected. The same is true of most of the intended outcomes.

In many respects, FSAP suffered from a lack of project management in the initial two years of implementation. The Agency was cognizant of those shortfalls and implemented several practices to improve project the management of projects such as FSAP. Improved or added practices included an increase in the level of engagement by senior CFIA management with respect to the status, success and progress of large-scale project implementation. Ultimately, the PMO was enhanced to allow it to manage all aspects of FSAP implementation. It also led to changes in governance over such projects.

The findings are presented in accordance with the Treasury Board Policy on Evaluation (2009) for the two core issues – relevance and performance.

3.2 Relevance

Finding 1.1: The Food Safety Action Plan is aligned with the Canadian Food Inspection Agency and the Government of Canada priorities.

The following are the intermediate and ultimate outcomes of FSAP, based on the logic model. The FSAP areas of activity are listed under each intermediate outcome below. All FSAP areas of activity are aligned with the ultimate outcome.

These outcomes are aligned with the mandate of the CFIA. According to the CFIA's 2012-13 Report on Plans and Priorities, the Agency's strategic outcome is "…a safe and accessible food supply and plant and animal resource base." Furthermore, the achievement of the stated FSAP outcomes, as well as the outcomes of the Food Safety Program,Footnote 26 supports the Agency's Food Safety and Nutrition Risks program activity,Footnote 27 which directly contributes to "Healthy Canadians," one of the Government of Canada's outcomes.Footnote 28 In addition, all the interviewees who were asked about the alignment between the FSAP objectives and the mandate of the CFIA confirmed that FSAP directly supports the achievement of the CFIA's aforementioned strategic outcome.

Finding 1.2: A clearly identified need was established prior to the announcement of the Food Safety Action Plan.

The need for increased oversight in the NFRS was identified as a result of a number of high profile international incidents that caused, or had the potential to cause serious harm to people worldwide. Two of the more notable incidents were the discovery of melamine in infant formula (2008), and the discovery of children's toys containing lead (2007), all manufactured in China. Although Canadians were unharmed, many people, particularly in China, died or suffered harmful long-term effects as a result of these incidents. The GoC made a commitment to reduce the risks to Canadians in this area with the announcement of FCSAP.

The majority of the interviewees were consistent in describing the needs identified to be addressed through FSAP:

Documentation supporting the funding for the FSAP indicated that the food industry in Canada employed more than 200,000 Canadians, representing the largest industry in the manufacturing sector, and accounting for $24 billion in the country's gross domestic product (GDP) in 2005.Footnote 29 In 2009, the food manufacturing industry employed more than 200,000 CanadiansFootnote 30 and accounted for GDP in the magnitude of $19 billion, a figure that has increased each year since 2004.Footnote 31

Furthermore, according to Statistics Canada figures,Footnote 32 the total value of food imported into Canada grew from $14.2 billion in 1997 to $23.7 billion in 2007.

Canada's key trading partners have taken steps to modernize their food safety systems. These partners are increasingly demanding that Canada demonstrate that its food safety system is effective, as a prerequisite to accessing their markets.Footnote 33

Finally, the 2000 and 2004 Auditor General reports discussed some of the inadequacies of Canada's regulatory system, particularly given the rapidly expanding and complex global marketplace, as previously noted in Section 1.7.

The output data indicate that FSAP has enabled the Agency to add oversight to the expanding imported NFRS. Although already in existence as part of national monitoring programs, surveillance activities were enhanced by the addition of targeted surveys and border blitzes funded under FSAP. The result was increased efforts in a particularly vulnerable area, namely the NFRS, using a risk-based approach.Footnote 34

Finding 1.3: FSAP related activities will need to continue beyond 2012-13.

The majority of the interviewees indicated that it is critical for activities to continue for two reasons in particular. First, FSAP comprised in part a series of outputs intended to contribute to a series of key outcomes. According to the majority of the interviewees, much of the preliminary work leading to the outputs is still being completed. The fact that many outputs are only being achieved in the final year of FSAP does not appear to be an issue of efficiency, according to the interviews but rather seems to be a result of the nature of many of the activities. For example, establishing risk profiles for food products required extensive research, including numerous targeted surveys to determine risk levels. Some interviewees indicated that there was very little existing research, meaning that much of the activity in this area was primary research conducted by the CFIA. Many interviewees expressed concern that the initial timelines for implementing FSAP and completing tasks were unrealistic given the size of the project and the Agency's limited experience in implementing such projects.

Additionally, a series of delays associated with the drafting of the IFS regulations occurred during implementation. This delay affected not only the regulations but other key components of FSAP such as the importer licensing system and the core food safety control systems for industry. This effect is discussed in further detail in Section 3.4.

The PMO has begun implementing a transition plan to reduce its involvement in FSAP, given that it will move to steady state on January 1, 2014. Table 7 shows the status of activities to date.

Table 7: Anticipated Transition Status by Area of Activity
Area of Activity Objective Transition Status
Risk Mapping and Baseline Surveillance To better identify, assess, and prioritize potential food safety hazards through risk mapping, information gathering and sampling and testing of foods on the Canadian marketplace Will transition by April 1, 2013
Food Safety Systems To work with internal and external stakeholders on the design and implementation of effective risk mitigation and preventive control strategies Will transition after April 1, 2013
Importer Licensing System To work with internal and external stakeholders on the design and implementation of an importer licensing regime Will transition after April 1, 2013
Enhanced Offshore Efforts To develop and/or advance relationships at the international level to further food safety mitigation approaches Transitioned
Strengthening Regulatory Backstops To develop and implement regulations to enhance food safety Will transition after April 1, 2013
Public Consultation Engagement Tools To consult and communicate with Canadians on the development of a modernized "Product of Canada/Made in Canada" labelling policy that supports informed consumer choice and reflects the needs of modern consumers Transitioned
Product of Canada Labelling To develop and implement a revised "Product of Canada/Made in Canada" labelling policy Transitioned
Enhanced Inspections of High Risk Sectors To provide increased oversight of high-risk sectors (identified through targeted surveys and risk mapping) through the verification of domestic and imported industry food safety control systems Transitioned
Border Blitzes To undertake additional port-of-entry or border blitzes in order to better prevent unsafe food from entering Canada Will transition by April 1, 2013
Enhanced Tracking of Imports To implement a more comprehensive import control program and tracking system in order to better prevent unsafe food from entering Canada Transitioned
Enhanced Recall Capacity To strengthen capacity to respond to an expected increase in the number of food safety investigations and food recalls due to enhanced surveillance testing Transitioned
Targeted Consumer Risk Communications To develop and deliver a series of communications initiatives targeting consumers to improve awareness of food safety issues and recalls and to help consumers better protect their health Transitioned

The PMO will maintain its role in providing services to program areas to meet deliverables and to assist in transitioning activities into regular Agency operations. This transition is intended to be complete by December 31, 2013.

An amount of $54.9 million per year in ongoing funding for FSAP activities beyond 2012-13 has been projected. Table 8 shows the yearly funding required beyond 2012-13 by project activity area.Footnote 35

Table 8: Ongoing Funding Beyond 2012-13 by Project Activity Area
Project Activity Area On-going Funding
($M)
Risk Mapping and Baseline Surveillance 11.25
Food Safety Systems 1.59
Implementation of Importer Licensing (includes IM/IT Direct Support) 7.06
Enhanced Offshore Efforts 1.79
Appropriate Regulatory Backstops 0.60
Public Consultaion and Engagement Tools 0.00
Product of Canada Labelling 0.44
Enhanced Inspection of High-Risk Sectors 12.00
Border Blitzes 0.82
Enhanced Tracking of Imports 0.92
Enhanced Recall Capacity 5.68
Targeted Risk Communication 1.16
FSAP Totals 43.3

Source: FSAP Program Design - Costing Appendix D

It should be noted that there is a difference of $11.6 million between the initial estimates ($54.9M) and the actual planned funding ($43.3M). This difference has arisen because $43.3 million is the direct funding that the CFIA's branches will be allocated in their 2013-14 budgets, and the remaining $11.6 million is to cover overhead, refresh of previous capital purchases and a program reserve, as broken down in Table 9.

Table 9: Description of FSAP Related Overhead Costs
Description of Overhead Amount
($M)
Corporate overhead, which includes various services (i.e. finance, legal, audit and evaluation, human resources – administration of compensation and benefits and IM/IT) 5.3
Space fit-up (refresh of cubicles, leasehold improvements, cabling and other real property charges incurred by the CFIA) 0.7
PWGSC Accommodation (rent paid to PWGSC for space occupied by individuals covered under the direct and corporate overhead spending) 2.9
Capital refresh strategies (fleet, scientific equipment and hardware/software) 1.7
Amount being held in surplus by the Senior Project Advisory Committee (SPAC) given that not all branches have completely identified ongoing deliverables. 1.0
Total 11.6

Source: FSAP - Program Design Costing Substantive Figures

According to the FSAP Program Design Document, the scheduled funding for FSAP is based on a clearly defined scope and scale, and required to complete the activities shown in Table 7. A change in scope or scale or the program may inhibit the ability of the CFIA to complete some activities.

The FSAP will require an ongoing number of 296 FTEs for activities continuing into steady state beyond 2012-13.Footnote 36 The majority of these positions are in the science and professional job categories (i.e., Biological Services (BI), Chemistry (CH), Engineering and Scientific Support (EG), Economics (ES), Computer Systems (CS), with others in administration (i.e., Administrative Services (AS), Clerical and Regulatory (CR), Information Services (IS), Programme Administration (PM)) and in corporate overhead.Footnote 37 Many of these positions have been filled to date. Table 10 shows the yearly resources required beyond 2012-13 by project activity area.

Table 10: Ongoing FTEs beyond 2012-13 by Project Activity Area
Project Activity Area On-going FTEs
Risk Mapping and Baseline Surveillance 22
Food Safety Systems 13
Implementation of Importer Licensing (includes IM/IT Direct Support) 51
Enhanced Offshore Efforts 14
Appropriate Regulatory Backstops 4
Public Consultaion and Engagement Tools 0
Product of Canada Labelling 4
Enhanced Inspection of High-Risk Sectors 113
Border Blitzes 9
Enhanced Tracking of Imports 9
Enhanced Recall Capacity 49
Targeted Risk Communication 8
FSAP Totals 296

Source: FSAP Program Design - Costing Appendix D

3.3 Performance - Effectiveness

Finding 2.1: The majority of the planned outputs have been produced, or appear likely based on the rate of progress to be produced before March 31, 2013. Outputs are being utilized as intended.

Outputs associated with FSAP activities were identified through performance reporting as early as 2008, as evidenced by data found in the documentation. In an effort to illustrate the progress made in producing key FSAP outputs, a rating system was designed. The rating takes into account the level of progress in achieving the output, as well as the likelihood that the output will be accomplished prior to the end of the FSAP (March 31, 2013).

Figure 4 summarizes the extent to which the key outputs of FSAP have been completed. These ratings are based on data collected during the evaluation using all data collection methodologies. In a limited number of cases, quantitative data were not available. As such there was a reliance on qualitative data provided through the interviews. The outputs that have been achieved are inherently directly attributable to FSAP given that the majority of the activities have been targeted at imported foods in the NFRS – the primary focus area of FSAP.

The section following Figure 4 provides evidence of progress around the achievement of outputs. Note that the description of each output is by both its original title according to the logic model and current wording as per the FSAP Program Design Document.

Figure 4: Summary of Progress in Achieving Outputs

Click on image for larger view
Summary of Progress in Achieving Outputs. Description follows.

Description for image - Summary of Progress in Achieving Outputs
Figure 4: Summary of Progress Achieving Outputs
Output (according to logic model) Output (alternative wording - FSAP Program Design Document) Achievement Rating Status
Targeted Surveys Enhanced Inspection of High-Risk Sectors/Risk Mapping and Baseline Surveillance 4 On-going Activity
Risk Profiling Database Risk Mapping and Baseline Surveillance 4 On-going Activity
Industry Consultations Development and Implementation of Food Safety Control Systems by Industry 4 Complete
Imported Food Sector Regulatory Consultations Strengthened Regulatory Backstops (Regulations) 2 On-going Activity
MOUs and Trading Partner Collaborations Enhanced Offshore Efforts 4 On-going Activity
Canadian Agricultural Products (CAP) Act Strategy Strengthened Regulatory Backstops (Food Bill) 3 Awaiting Approval
Imported Food Sector Regulations Strengthened Regulatory Backstops (Regulations) 2 Awaiting Approval (Target Oct 2013)
Consultations with Canadians Public Consultation and Engagement Tools / Targeted Consumer Risk Communications 4 On-going Activity
New Labelling Policies Product of Canada Labelling 4 Complete
Inspection and Verification Control Systems Implementation of Importer Licensing 2 Implementation Awaiting Approval of Regulations
Core Food Safety Control System Criteria Development and Implementation of Food Safety Control Systems by Industry 2 Awaiting Approval of Regulations
Pot of Entry / Border Blitzes Border Blitzes 4 On-going Activity
Comprehensive Import Control and Tracking Enhanced Tracking of Imports 3 Phase 10 of 11 currently
Investigations and Recalls Enhanced Recall Capacity 4 On-going Activity

Legend:

  • 1 - Limited progress to date and will not be achieved by March 31, 2013
  • 2 - Some progress and likely not to be achieved by March 31, 2013
  • 3 - Significant progress and will be achieved by March 31, 2013
  • 4 - Achieved

Targeted Surveys (Enhanced Inspection of High-Risk Sectors / Risk Mapping and Baseline Surveillance) Achieved

Much effort has been expended on undertaking targeted surveys and preparing and validating reports to present results. Targeted surveys have been conducted in both the areas of chemistry and microbiology.

In total, 55 targeted survey reports were completed in chemistry, 33 in microbiology, and 14 on allergens between 2008-09 and 2011-12. In addition, an additional 47 targeted survey reports in chemistry, 29 in microbiology, and 14 on allergens that are currently underway or planned through 2012-13. In microbiology multiple surveys are combined into one survey report.

Figure 5 illustrates the number of targeted survey reports completed by year.

Figure 5: Completed, Underway, and Planned Targeted Survey Reports by Year (2008-09 through 2013-14)
Targeted Survey Reports by Year (2008-09 through 2013-14). Description. Follows.
Description for image - Completed, Underway, and Planned Targeted Survey Reports by Year (2008-09 through 2013-14)

Figure 5: Illustrates the number of targeted survey reports completed, underway or planned by year

In 2008-09 the following targeted survey reports were completed: 5 chemical; 4 microbial.

In 2009-10 the following targeted survey reports were completed: 9 chemical; 5 microbial; 2 allergen.

In 2010-11 the following targeted survey reports were completed: 20 chemical; 10 microbial; 6 allergen.

In 2011-12 the following targeted survey reports were completed: 21 chemical; 14 microbial; 6 allergen.

In 2012-13 the following targeted survey reports were underway or planned: 23 chemical; 12 microbial; 7 allergen.

In 2013-14 the following targeted survey reports were underway or planned: 24 chemical; 17 microbial; 7 allergen.

Source: FSAP Target Survey Reports (2008 through 2013-14) Allergen, Chemical and Microbial

Risk Profiling Database (Risk Mapping and Baseline Surveillance) Achieved

There have been several accomplishments with respect to risk mapping. One key output in this regard was the purchase and populating of the FDA iRisk tool. The FDA iRisk is a tool in which food hazard and commodity combinations are ranked based on their human health risk level. Although input is an ongoing process, the purchase of the tool and the entry of new data and new food hazard combinations into iRisk was an important output in FSAP. According to interview data, there have been 70 rankings of food hazard combinations, of which 16 have been peer-reviewed to date. The data for these food hazard combinations are documented in the FDA iRisk repository.

Further, FDA iRisk has facilitated a risk-based approach to targeted surveys. Data from the targeted surveys can be linked to FDA iRisk through specific modules to allow for evaluation of data from food establishments (production volumes, type of processing and compliance history) and characterization of the level of risk and potential benefit of inspection and surveys. If targeted surveys on a particular food hazard combination find no or low risk, then targeted surveys for specific food hazard combination may cease, and efforts will be directed to other areas with potentially greater risks.

Industry Consultations Development and Implementation of Food Safety Control Systems (by Industry) Achieved and Imported Food Sector Regulatory Consultations (Strengthened Regulatory Backstops) Some Progress

Multiple types of consultations with industry and other stakeholders have occurred throughout FSAP implementation. Some consultations were general and covered multiple issues and stakeholders, and other consultations specifically targeted industry and consisted of various topics, including food safety compliance, risk identification and product labelling. Regulatory consultations were very specific to the IFS regulations and occurred prior to and during the drafting of these regulations. The regulatory consultations targeted more stakeholder groups beyond industry. For example, those consultations often included other levels of government as well as other stakeholders within the CFIA and the GoC.

In total, 100 regulatory consultations took place between 2010-11 and November 1, 2012, as shown in Figure 6.

Figure 6: Regulatory Consultations Held or Planned by Year (2010-11 through 2012-13)
Regulatory Consultations Held or Planned by Year (2010-11 through 2012-13). Description follows.
Description for image - Regulatory Consultations Held or Planned by Year (2010-11 through 2012-13)

Figure 6: Regulatory Consultations Held or Planned by Year (2010-11 through 2012-13)

In total, 100 regulatory consultations took place between 2010-11 and November 1, 2012.

In 2010-11, 10 regulatory consultations took place.

In 2011-12, 62 regulatory consultations took place.

In 2012-13, 28 regulatory consultations took place.

Source: Communication and Engagement Activities to Date

The approaches to these consultations have varied, with a significant proportion of the consultations occurring through in-person meetings with stakeholders, as shown in Figure 7. In addition to these documented consultations, there are an additional series of regulatory consultations planned in 17 jurisdictions over the coming year, as the new CAP regulations are approved and implemented.

Figure 7: Regulatory Consultations Held or Planned by Type (2010-11 through 2012-13)
Regulatory Consultations Held or Planned by Type (2010-11 through 2012-13). Description follows.
Description for image - Regulatory Consultations Held or Planned by Type (2010-11 through 2012-13)

The following list outlines the consultations held or planned by type from 2010-11 through 2012-13:

  • 1% of regulatory consultations were held/ planned online.
  • 16% of regulatory consultations were held/planned via teleconference.
  • 24% of regulatory consultations were held/planned via a presentation.
  • 53% of regulatory consultations were held/ planned via in-person meeting.
  • 1% of regulatory consultations were held/ planned via videoconference.
  • 5% of regulatory consultations were held/ planned via working group.

Source: Communication and Engagement Activities to Date

The majority of the industry interviewees indicated that the CFIA has a significant volume of engagement with industry, not only for FSAP related issues but for other Agency initiatives as well. According to most industry interviewees, this engagement has been beneficial with regard to industry's ability to provide input. However, each engagement tends to focus on particular issues. This limits industry's understanding of how particular issues are aligned with initiatives. For example, when the CFIA and industry meet, industry does not always understand how its comments for one set of issues may be affecting another set of issues as part of other initiatives.

Furthermore, some industry interviewees reported that they are not always provided with the level of detail they need about certain issues to provide the most useful feedback. In many cases, very high level discussions occur, sometimes limiting industry's ability to make an informed contribution to the conversation.

Finally, while industry indicated that it appreciates the flow of imported food safety information from the CFIA, industry uses many other international sources of information to ensure that it is monitoring the safety of imports. For example, many companies are already meeting requirements under the Global Food Safety Initiative (GFSI), as explained under Finding 1.3.

MOUs and Trading Partner Collaboration (Enhanced Offshore Efforts) Achieved

Formal discussions have been held with a number of countries based on anticipated and current trade volumes. Interactions have taken place with China, Guatemala, Mexico, the U.S. and India. Establishing MOUs or other formal agreements is a critical component for ensuring food entering Canada is safe for consumption. Such MOUs have been established with China and Guatemala.

It should be noted that test establishing MOUs is not always the most appropriate method for reaching agreement on food safety matters with trading partners, according to a few interviewees. These interviewees explained that depending on the nature of the issue and the country involved, other written agreements can be developed, which can be equally as important as a formal MOU.

CAP Act Strategy (Strengthened Regulatory Backstops (Food Bill) Significant Progress and Imported Food Sector Regulations (Strengthened Regulatory Backstops (Regulations) Some Progress

Progress has been made on creating the Safe Food for Canadians Act and the IFS regulations. Although the former proceeded with minimal challenge, the latter experienced a number of delays, as noted previously. Regarding the Safe Food for Canadians Act, the legislation received royal assent on November 22, 2012. The IFS regulations are still under development. Numerous process delays, noted previously, have led to a lengthier duration to complete a draft.

Despite the issues and negative effects on timelines, it is anticipated by most interviewees that the regulations will be approved; although the new target date is October 2013. This is significant because there is also an intention to require importers to register for a licence under these regulations, and such information would be captured electronically in an IT system developed under FSAP. Although this IM/IT component of FSAP was not in the scope of this evaluation, it is understood that even though the component is complete, it effectively will not be implemented by its original April 2013 target date, until the regulations are completed.

Consultation with Canadians (Public Consultation and Engagement Tools/Targeted Consumer Risk Communication) Achieved

New Labelling Policies (Product of Canada Labelling) Achieved

This output was one of the first to be fully achieved (2008). The activity in this area consisted of a revision to the CFIA's Product of Canada policy, and subsequent compliance monitoring in relation to these revisions through verification activities, including label reviews and inspections. A series of stakeholder consultations were conducted following the revisions in order to gauge impact and reactions. On July 15, 2008, the revised policy was announced by the GoC.Footnote 38

Monitoring is now ongoing and effectively in a steady state, according to documents and most interviewees. The data show that as of July 30, 2012 the number of Product of Canada issues has declined since 2008-09 (see Figure 9).Footnote 39

Figure 8: Number of Product of Canada Issues Raised by Year (2008-09 to July 30, 2012)
Number of Product of Canada Issues Raised by Year (2008-09 to July 30, 2012). Description follows.
Description for image - Number of "Product of Canada" Issues Raised by Year (2008-09 to July 30, 2012)

The following lists the number of "Product of Canada" issues raised by year from 2008-09 to July 30, 2012.

  • In 2008-09, there were 18 "Product of Canada" issues;
  • In 2009-10, there were 33 "Product of Canada" issues;
  • In 2010-11, there were 30 "Product of Canada" issues;
  • In 2011-12, there were 23 "Product of Canada" issues;
  • In 2012-13, there were 4 "Product of Canada" issues.

Source: Summary Table of Operations FSAP Data from FY 2008-2013, provided by Imported and Manufactured Food Division (IMFD) on September 12, 2012

Although the reasons for this trend are not clear, it is not unreasonable to assume that as food importers have become more familiar with the Product of Canada policy, issues have been corrected. Some industry interviewees explained that their members complied with requirements; however, the evaluation did not seek to confirm such compliance.

Additionally, some industry interviewees expressed concern about the inability of their members to use 'Product of Canada' labelling because their products miss the 98% threshold by a small percentage. This is mainly the result of the use of sugar or spices that do not grow in Canada because of the climate but are used in particular products. These interviewees explained that their members incurred additional costs (e.g. changing labels and experimenting with alternative ingredients) to comply.

Inspection and Verification Control Systems (Implementation of Importer Licensing) Some Progress

Work has been completed or initiated with respect to how the importer licensing system and fee system will work. However, owing primarily to the delays encountered in drafting IFS regulations and obtaining approval for them, importer licensing and preventive food safety control systems are not likely to be achieved within the five-year period ending March 31, 2013, according to most interviewees. A few interviewees indicated that the revised target timeframe for the importer licensing system is October 2013.

Core Food Safety Control Systems Criteria (Development and Implementation of Food Safety Control Systems by Industry) Some Progress

There has been some limited progress in developing food safety control criteria for food manufacturers to date. In June 2011, the CFIA published the General Principles of Food Hygiene, Composition and Labelling on its website. This document represents an assessment tool to assist food manufacturers in establishing manufacturing practices that maintain food safety and meet regulatory requirements.Footnote 40

Additionally, the CFIA's Guide to Food Safety was posted online in September 2010. The Guide to Food Safety is a voluntary tool that provides the Canadian food industry with generic guidance on how to design, develop and implement effective preventive food safety control systems.Footnote 41

Although these examples demonstrate progress in developing tools for assisting industry in developing and implementing preventive food safety control systems, this output cannot be fully achieved or measured fort imported foods until the importer licensing system is in place. Importers will have to attest that they have developed and implemented such food safety control systems in order to be granted a licence for importing foods.Footnote 42 As previously noted, the importer licensing system cannot be implemented until the IFS regulations are finalized; therefore, this output will not be made operational by March 31, 2013. As with other outputs, however, most interviewees are confident that it is simply a timing issue and that the output and related outcomes will be achieved as the various interdependent activities are completed.

Port of Entry / Border Blitzes Achieved

The number of border blitzes that have been conducted over the course of FSAP implementation have increased each year, as planned. A total of 261 border blitzes had been carried out as of September 12, 2012. Figure 9 provides a breakdown of border blitzes conducted by year and by area.

Figure 9: Number of Port of Entry / Border Blitzes Executed by Year and Area (2008-09 to July 30, 2012)
Number of Port of Entry / Border Blitzes Executed by Year and Area (2008-09 to July 30, 2012). Description follows.
Description for image - Number of Port of Entry / Border Blitzes Executed by Year and Area (2008-09 to July 30, 2012)

The following list outlines the number of port of entry/border blitzes executed by year and area from 2008-09 to July 30, 2012.

  • In 2008-09: 4 occurred in the West, 16 in Ontario, 5 in Quebec, and 2 in the Atlantic.
  • In 2009-10: 10 occurred in the West, 38 in Ontario, 12 in Quebec, and 4 in the Atlantic.
  • In 2010-11: 10 occurred in the West, 48 in Ontario, 11 in Quebec, and 4 in the Atlantic.
  • In 2011-12: 12 occurred in the West, 46 in Ontario, 12 in Quebec, and 4 in the Atlantic.
  • From April 1, 2012 to July 30, 2012: 5 occurred in the West, 10 occurred in Ontario, 4 in Quebec, and 4 in the Atlantic.

Source: Summary Table of Operations FSAP Data from FY 2008-2013, provided by Imported and Manufactured Food Division (IMFD) on September 12, 2012

One of the lessons learned under FSAP was that border blitzes were not as useful when actually conducted at the border. A new approach was taken in the Quebec Area where in order to effectively address the issue these targeted inspections were shifted from points of entry to points of retail. The focus shifted from border inspection to import surveillance, which was determined to be more effective in keeping potentially hazardous food products from reaching consumers. An import surveillance team was established in Montreal, and visits retail locations to gather intelligence about certain products. For example, the team may identify juices with non-permitted additives or information, such as allergens missing from labels. This intelligence is used to identify importers that are routinely importing potentially hazardous or non-compliant food products and thus enables the CFIA to contact the importer to rectify the issue(s). Although only the Quebec Area has implemented such a retail-based approach, a few interviewees indicated that others are interested in doing the same, and it is anticipated that this will occur during the FSAP steady-state period.

Comprehensive Import Control and Tracking (Enhanced Tracking of Imports) Significant Progress

The main activities with respect to import control and tracking were implemented to ensure more control over imported food in the NFRS. The objective was to identify importers and their products, understand where the products are originating from and determine when these products arrive in Canada. In order to accomplish this, the mechanism used to code products was aligned with international practices, codes and standards (e.g. the use of HS codes was key as each commodity is assigned a unique numerical code for tracking).Footnote 43

Foods in the NFRS are coded based on the World Customs Organization's Harmonized Commodity Description and Coding System's (HS) six digits, and further refined by applying the CFIA's Other Government Department (OGD) coding system (an additional six digits). Prior to FSAP, the CFIA could not obtain the level of detail on products that is required for analyses on imported food in the NFRS, because the HS codes were limiting. As part of FSAP, a 12-phase process is underway to recode the food commodities by adding digits to the existing HS codes. Most interviewees indicated that as of October 29, 2012, the process was in Phase 10. All phases are expected to be fully completed by March 31, 2013, with the exception of the alcoholic beverages category. It was explained that additional consultation is required between the CFIA and alcoholic beverage importers before HS codes can be added for this category.

Investigations and Recalls (Enhanced Recall Capacity) Achieved

A majority of the interviewees with knowledge of investigations and recalls indicated significant gains in this area due to FSAP. The data show that the number of recalls and incidents has consistently increased since 2008-09, an increase that also correlates to the increasing number of activities under the Enhanced Inspection of High Risk Sectors that have occurred each year (see Figure 10). These data suggest that as import surveillance has increased, the number of recalls and incidents has increased because more issues are being identified. In total, there have been 1,250 activities, and 2,025 recalls or recall incidents within IMFD and the Fresh Fruit and Vegetables Program (FFVP) between 2008-09 and 2011-12.

Case study evidence gathered during the evaluation suggests that from the perspective of laboratory methods and testing, there have been improved capabilities for identifying chemicals, microbes and allergens in food. For instance, in the case of the DEHP case study (described in further detail in Finding 2.2), the CFIA was able to quickly identify the hazard in partnership with its U.S. counterparts, resulting in a swift recall.

Figure 10: Import Surveillance Activities and Recalls or Recall Incidents by Year (2008-09 through 2011-12)
Import Surveillance Activities and Recalls or Recall Incidents by Year (2008-09 through 2011-12). Description follows.
Description of image - Import Surveillance Activities and Recalls or Recall Incidents by Year (2008-09 through 2011-12)

The following lists the number of enhanced inspection of high risk sectors activities from 2008-09 to 2011-12:

  • 263 in 2008-09;
  • 230 in 2009-10;
  • 336 in 2010-11;
  • 421 in 2011-12.

The following lists the number of recalls or recall incidents from 2008-09 to 2011-12:

  • 433 in 2008-09;
  • 442 in 2009-10;
  • 510 in 2010-11;
  • 640 in 2011-12.

Source: Summary Table of Operations FSAP Data from FY 2008-2013, provided by Imported and Manufactured Food Division (IMFD) on September 12, 2012

Finding 2.2: There is little documented evidence to support the achievement of expected outcomes to March 31, 2013; however, there are examples of specific successes at the activity level

It was expected that by the end of the five-year FSAP implementation period on March 31, 2013, a number of outcomes as described in the logic model (Figure 1), at least at the intermediate level would be observed. The data available do not provide evidence that outcomes at any level have been achieved, or will be achieved by that date. In fact, in viewing performance from the start of the initiative to March 31, 2013, the data available indicate that it has not been successful in having its anticipated impact. However, improved project management since 2010-11 has, in part, led to progress on outputs, as shown in the previous section. In addition, there are specific examples of successes resulting from FSAP activities from across the Agency.

One of the key issues with regard to evaluating the achievement of outcomes is that the data are not documented in any formal way. Although a draft performance measurement strategy for FSAP exists, performance data, at the outcome level are not systematically collected, as previously explained in Section 1.2. In addition, it is too early for many outcomes to have occurred, given the challenges around staffing and project management in the first two years of implementation, as well as the delays associated with the IFS regulations.

Through the course of the evaluation, a limited amount of data in support of outcomes measurement were collected, primarily through interviews (representing largely anecdotal data), and through three case studies that were completed. It should be noted that there is some evidence to support achievement of some outcomes, such as the following:

A few interviewees expressed reservations about the perceived successes of other activities, particularly in relation to targeted surveys and border blitzes. These interviewees stated that although a number of targeted surveys and border blitzes had been undertaken, and had identified a limited number of potential dangers, this does not necessarily demonstrate that the Agency has enhanced its abilities to identify food safety risks on an ongoing basis. This section of the report contains a summary of performance associated only with outcomes for which data could be collected.

Improved ability to identify, assess and prioritize potential food safety risks (Active Prevention)

The case studies conducted as part of this evaluation provide some evidence to support progress towards the improved ability to identify, assess, and prioritize food safety risks. For example, the response to the DEHP incident demonstrated such an improvement. Through the targeted survey plan developed for that incident, the CFIA is better equipped to identify, assess and prioritize food safety risks associated with phthalates. This incident afforded the CFIA the opportunity to work closely with the USFDA to adapt the laboratory methodology received from Taiwan to identify DEHP in food products. This methodology is now being used to continue monitoring for the presence of that chemical in imported NFRS foods. Additionally, this incident provides some evidence that the CFIA has developed a collaborative working relationship with other countries.

Increased ability to prevent unsafe food from entering Canada (Targeted Oversight)

The tsunami response provided evidence to support progress in achieving an increased ability to prevent unsafe food from entering Canada. The updating of HS codes in the Automated Import Reference System (AIRS) was a critical part of managing the tsunami response involving potentially dangerous food products originating in Japan. The CFIA was able to flag certain products in AIRS using the updated codes and descriptions. As products were arriving at ports, the CBSA was able to find out if the product required attestation, by selecting the product in AIRS and determining whether the CFIA had flagged it for attestation or other requirements. Prior to FSAP, this was not a possibility for foods in the NFRS.

Improved consumer awareness of food safety issues and recalls (Rapid Response)

There is very little evidence about consumer awareness, and the data that do exist are circumstantial. A report prepared for the CFIA titled "Food Safety: Canadians' Awareness, Attitudes and Behaviours (2011-12)",Footnote 44 found that in fall of 2011, 67% of survey respondents indicated confidence in Canada's food safety system. Of those survey respondents who reported weak confidence in Canada's food safety system, 25% of them based their response on there being too many recalls and food scares in the news. However, the findings in this report are not specific to FSAP, nor can they be directly linked to FSAP activities.

The lack of information, anecdotal or otherwise, about progress in achieving outcomes is reflective of a lack of systematic outcome data collection, as well as the fact that some outputs have not yet been produced, or have been produced only recently. Although there are still outputs that remain to be produced and are expected to result in further progress towards outcome achievements, program managers and evaluators expected that some of the intended immediate outcomes associated with activities such as targeted surveys, border blitzes and updates to the HS coding system would be supported by the data. However, this evaluation found only isolated examples identified through case studies. It is possible that formal and systematic performance measurement at the outcome level may yield a collection of incidents, which would further suggest that outcomes are being achieved.

Finding 2.3: Industry struggles to understand how the various food safety initiatives are related.

Industry has been consulted and continues to be involved in the implementation of FSAP activities and other food safety related initiatives led by the CFIA. Although most industry interviewees appreciate being engaged throughout the process of implementing these initiatives, they also expressed concern that they do not understand how the individual initiatives are related. Most interviewees said that they do not understand how their input for activities associated with one initiative may impact activities associated with another initiative. These interviewees believe that with a better understanding of what the initiatives are individually and collectively trying to accomplish, they may be able to offer thoughtful input when engaged by the CFIA.

3.4 Performance - Efficiency and Economy

Finding 3.1: The majority of funding was used for its intended purposes; however, not all allocations were aligned to the original budget plans.

Not all funds were used for the purposes originally intended. In terms of actual spending against planned spending, expenditures have been less than originally intended with the FSAP budget. Figure 11 provides a breakdown of total FSAP planned versus actual or forecasted spending by year.

Figure 11: FSAP Planned Versus Actual/Forecasted Spending by Year
Food Safety Action Plan Planned Versus Actual/Forecasted Spending by Year. Description follows.
Description for image - FSAP Planned Versus Actual/Forecasted Spending by Year

The following lists the planned and actual/forecasted spending from 2008-09 to 2012-13:

  • In 2008-09, the planned spending was $22.9 million and the actual/forecasted spending was $14.2 million.
  • In 2009-10, the planned spending was $39.3 million and the actual/forecasted spending was $33.2 million.
  • In 2010-11, the planned spending was $52.4 million and the actual/forecasted spending was $42.0 million.
  • In 2011-12, the planned spending was $52.4 and the actual /forecasted spending was $40.3 million.
  • In 2012-13, the planned spending was $56.4 and the actual/forecasted spending was $55.0 million.

Source: FSAP Deliverables Status Report: Financial Table – forecast as of September 30, 2012

Yearly spending of less than the allocated funding has resulted in a total spending gap in the amount of $38.8 millionFootnote 45. Figure 12 illustrates the planned versus actual FSAP spending by area of activity from 2008-09 to 2012-13.

Figure 12: Planned Versus Actual/Forecasted Spending by Area of Activity (2008-09 through 2012-13)
Planned Versus Actual/Forecasted Spending by Area of Activity (2008-09 through 2012-13). Description follows.
Description for image - Planned Versus Actual/Forecasted Spending by Area of Activity (2008-09 through 2012-13)

The following lists the planned and actual/forecasted spending by activity for the period 2008-09 to 2012-13.

  • Targeted consumer risk communication: $7.2 million was planned and the actual/forecasted spending was $5.9 million.
  • Enhanced recall capacity: $25.0 million was planned and the actual/forecasted spending was $27.9 million.
  • Enhanced tracking of imports: $17.7 million was planned and the actual/forecasted spending was $12.1 million.
  • Border blitzes: $5.1 million was planned and the actual/forecasted spending was $6.4 million.
  • Enhanced inspection of high-risk sectors: $54.2 million was planned and the actual/forecasted spending was $32.0 million.
  • “Product of Canada” labelling: $8.9 million was planned and the actual/forecasted spending was $3.6 million.
  • Public consultation and engagement tools: $1.0 million was planned and the actual/forecasted spending was $0.9 million.
  • Strengthened regulatory backstops: $5.5 million was planned and the actual/forecasted spending was $2.7 million.
  • Enhanced offshore efforts: $9.2 million was planned and the actual/forecasted spending was $3.6 million.
  • Implementation of importer licensing: $35.8 million was planned and the actual/forecasted spending was $25.7 million.
  • Food safety systems: $5.0 million was planned and the actual/forecasted spending was $10.3 million.
  • Risk mapping and baseline surveillance: $48.8 million was planned and the actual/forecasted spending was $53.5 million.

Source: FSAP Deliverables Status Report: Financial Table – forecast as of September 30, 2012

Annual allocations for FSAP were not always spent in each fiscal year for the following reasons:

According to a few interviewees, the delays in finalizing the IFS regulations were a key reason behind the inability to spend full allocations within the intended timeframes. There were numerous activities associated with the IFS regulations that had to be continuously postponed as a result of not having a draft set of regulations with which to work.

There were also delays in other specific areas such as Targeted Risk Communication and Risk Mapping and Baseline Surveillance, which included activities regarding the communication of targeted survey results. Reports associated with completed targeted surveys were approximately 18 to 24 months behind scheduled completion – meaning that the targeted surveys had been completed but reports had not been generated or published. The reasons include many of those listed above. First, a high volume of turnover was experienced over a short time within the Food Safety Division (the division responsible for these reports). The inability to replace these individuals in a timelier manner was exacerbated when the CFIA underwent reorganization and the Food Safety Division was brought under the Science Branch. At the same time, food safety components of Canada's Economic Action Plan were being implemented, further reducing staff availability. The challenges affected the ability of the CFIA to complete reports in a timely fashion.

Lastly, a new, more comprehensive review process including HC and CFIA Senior Management sign off was implemented in 2011. The reason for this was the experience associated with one of the first targeted surveys to be completed (ochratoxin). The publication of the findings from this targeted survey drew a high level of concern and questioning from a specific industry group, given the fact that HC was in the process of establishing standards. The process is reportedly working well at present; however, this increased oversight means that the process for posting targeted survey reports for Canadians is now lengthier.

Of the $37.5 million that was unspent on FSAP from 2008-09 to 2011-12:

At the time of the Evaluation, an additional surplus of 1.4 million was forecasted for fiscal year 2012-13, bringing the total spending gap to $38.8Footnote 46 million, noted earlier in the report.

Finding 3.2: There is very little data regarding the cost-effectiveness of FSAP delivery.

A few interviewees noted that the level of cost analysis, particularly around skills and staffing needs for FSAP implementation fell short of where it should have been. According to these interviewees, neither the costs nor the level of expertise required to undertake certain FSAP activities was well understood prior to implementation. Limited data suggest that there was underestimation in the costs of conducting certain activities.

For example, there was a concern on the part of a few interviewees that the formulas used to estimate laboratory activity costs were not well constructed. The funding to deliver laboratory activities for targeted surveys and method development has been adjusted to reflect the current level of funding necessary to deliver the services in a laboratory setting. The laboratory service delivery model has integrated a mixed approach in which activities are conducted by third party laboratories and by CFIA network laboratories. Implementing this model and adjusting the funding level permitted to deliver the laboratory services as per the work plans.

Finding 3.3: FSAP was a lesson in horizontal project management for the CFIA.

Interviewees were asked to summarize key points, and lessons learned about FSAP and its implementation. Most interviewees described a steep learning curve associated with implementing FSAP. These interviewees explained that when FSAP was announced, it represented the largest initiative that the CFIA had been faced with implementing in its history. This came with challenges.

The Agency's project management rating in the Management Accountability Framework (MAF) further supports the presence of issues with managing projects. From 2009-10 to 2010-11 the CFIA received a rating of 'Opportunity for Improvement' in the 'Investment Planning and Management of Projects' Area of Management MAF Assessment.

According to many interviewees, the initial two years of FSAP implementation were inefficient. The lack of progress in the first two years of FSAP implementation was largely due to the overall management approach. The PMO was established during the latter part of Year Two, resulting in many improvements, according to most interviewees. For instance, the PMO, in conjunction with the CFIA's branches, developed work plans, established budgets and also monitored and maintained those work plans and budgets. This structure was critical to facilitating progress towards outputs, and improving efficiency in FSAP implementation.

This experience has informed other large CFIA initiatives such as inspection modernization. The project management lessons learned from FSAP have reportedly facilitated more effective implementation of such initiatives. This was not only an important finding in this evaluation, but was also found to be the case in the 2011 internal audit of FSAP.

Finding 3.4: The Food Safety Action Plan governance structure and decision-making process has improved since undergoing restructuring in 2011.

The Audit of the Project Management of the Food Safety Action Plan (AEB 2011,) found that under the initial governance structure, the Steering Committee did not meet regularly, and that manager delegates were often the only contributors in meetings of the Management Committee.Footnote 47 Many interviewees confirmed that the Management Committee meetings were often comprised of delegates who were not in a position to make decisions on issues that required attention. In addition, the Management Committee in particular was perceived to be simply a forum for each CFIA branch to discuss the work that it was doing, but not to effectively resolve issues. The original governance structure was described by many of these interviewees as needing improvement.

Most interviewees indicated that once the governance structure was altered, its overall effectiveness improved. Many of these interviewees attribute the improved governance to the influence of the PMO. The PMO has played a key role in ensuring key issues are brought to the attention of various committee members and are discussed so that appropriate resolutions are reached. The PMO achieved this in part by creating a forum in which some issues were discussed and managed outside of committee meetings, because they were not necessarily the types of issues requiring committee involvement. This ensured the most important issues were resolved in committee.

Finding 3.5: Working relationships between the CFIA and its counterparts were critical to the successes of the Food Safety Action Plan.

The importance of successful working relationships with other federal government departments and agencies, other levels of government and with international counterparts was identified as a success factor in undertaking FSAP activities. For example, in the case of the CFIA's response to the tsunami incident in Japan, the relationship with the CBSA was particularly important. The two federal agencies were required to work together closely to ensure that imported food products coming from Japan were pulled aside and checked for safety.

To do so, a process involving the CBSA and the CFIA was put in place whereby CBSA identified food and feed imports from Japan and flagged them for the CFIA. The CFIA followed a process to ensure that appropriate documentation was attached to the shipment, and when that was the case, the CFIA recommended to the CBSA that the shipment be released. Some of the shipments were flagged for random sampling and testing and then released to market when determined safe. According to most interviewees, this process worked very well, and the relationship between the two agencies was strengthened.

In other instances, some interviewees noted that not all working relationships are formally structured. For instance, some interviewees explained they have regular bilateral discussions with colleagues at HC to exchange information particularly on standards and policy setting. The CFIA must be kept informed of potential changes in this regard to ensure that its laboratory methodologies are adequate to meet such standards and policies. Laboratory methodology development and improvement is a component of FSAP.

Lastly, some interviewees explained that information sharing with international counterparts, particularly within the USFDA, is important as a source of knowledge and as a cost-saving mechanism. For instance, within the human resources function, CFIA management has been able to use existing USFDA materials with minor modifications rather than recreating something that already exists. This information is useful for training because it relates to some of the activities being carried out under FSAP, which the U.S. has been doing for many years.

4 Conclusions

The implementation of FSAP was a significant project for the CFIA. The CFIA faced numerous challenges throughout its implementation, particularly in the first two years. This period entailed less 'doing' and more 'planning' than had initially been intended. Most FSAP activities will continue beyond 2012-13 either because they will become ongoing steady state activities within the CFIA or because delays prevented them from being completed within the five-year period.

The following four outputs will not be achieved in full within the five year FSAP implementation period:

  1. Imported Food Sector Regulatory Consultations;
  2. Imported Food Sector Regulations;
  3. Core Food Safety Control Systems for Industry; and
  4. Importer Licensing System.

The PMO was created in year two (2009-2010) to better manage FSAP and to provide structure and guidance to the program areas that were undertaking FSAP activities. The PMO played a critical role in organizing activities, providing necessary guidance on a variety of issues and establishing ongoing performance measurement, at least on the level of outputs. Most of the individuals interviewed as part of this evaluation indicated that they had observed improved facilitation of FSAP activities with the introduction of the PMO.

The creation of the PMO had the following effects:

The working relationships with numerous other government departments and agencies as well as with counterparts in other countries have ensured that many activities have been completed in a timely manner with positive results.

Industry appreciates that it has been consulted on food safety issues but it struggles to understand how the various food safety initiatives are related. The Agency needs to provide industry with a mechanism to ensure that businesses understand holistically what these initiatives mean to them.

The cost-effectiveness of FSAP activities could not be determined owing to the lack of useful data. Alternate ways of undertaking the activities were not identified. The existing data show that cost estimates for activities in some areas may have been underestimated when compared to actual costs. In these cases, the issue was rectified.

One of several major food safety initiatives being implemented by the CFIA, FSAP has a specific role among all initiatives and is aligned with the health and food safety priorities of the GoC. The other initiatives that have taken place, are underway or are anticipated are as follows:

Many interviewees indicated that inspection modernization is currently a significant undertaking, not unlike FSAP. Due to the experiences with FSAP, the Agency is better equipped to manage the implementation of inspection modernization. In fact, many interviewees noted that the lessons learned from FSAP implementation, with regard to project management and governance have directly informed inspection modernization.

5 Recommendations and Lessons Learned

Many of the issues that arose during the implementation of FSAP were identified for the first time prior to this evaluation either through the 2011 audit, or through observations made by the Agency when the PMO was created in 2010. As such, some issues were corrected before the conclusion of the evaluation.

This evaluation identified several positive findings. However, the need to improve outcome level performance measurement is critical to determining whether FSAP has made progress toward improved food safety. The FSAP steady state period would be an appropriate time to begin monitoring the impacts of all the work completed during the five-year FSAP implementation period. Improvements in the use of performance measures that support the measurement of outcomes will become increasingly important for the CFIA as it moves to outcomes-based programming in its modernization efforts, particularly in the regulatory sphere.

In addition, industry needs to be better informed not only about its responsibilities under the various outputs of FSAP but under other initiatives as well, recognizing that once FSAP has met its deliverables, it will no longer exist as a stand-alone initiative. This begins by ensuring industry understands how all the food safety initiatives being implemented by the Agency fit together, which would then allow industry to better understand what the initiatives collectively mean to businesses.

The recommendations provided here are meant to inform future similar initiatives, as well as FSAP through the transition period. It is important to note that in some cases the lessons learned that were self-identified during FSAP implementation have already informed new CFIA initiatives.

Table 11 details the links between the findings, conclusions and recommendations.

Recommendation 1.1: Develop and formally implement an outcomes based performance measurement strategy for FSAP activities.

Although a draft PMS existed specifically for FSAP, the strategy was clearly neither finalized nor formally implemented. Performance measurement of activities that carry over into steady state (e.g. importer licencing and core food safety control systems) as well as of the ongoing activities that will occur in steady state (e.g. targeted surveys, border blitzes and enhanced inspection of high-risk sectors ), is important for monitoring progress towards achieving outcomes and for making potential adjustments.

Given the Agency's move towards outcome-based programming (discussed in Section 1.2) an outcomes-based PMS, including specific and measurable targets, should be developed and implemented for the FSAP. This PMS could be the foundation for the CFIA's food safety activities.

Recommendation 1.2: Develop an approach to better communicate with industry how the range of food safety initiatives fit together.

Easier access to information about various food safety initiatives, including FSAP and inspection modernization, needs to be provided to industry. Although consulted on these initiatives, industry indicated a low level of awareness about the linkages between the different initiatives on which it has provided feedback.

A holistic approach is needed to ensure that the work being undertaken as part of inspection modernization and other initiatives builds on the work completed under FSAP. Such an approach, incorporated with the perspectives of key stakeholders, will better ensure improved food safety with minimal disruption to food trade and related issues, by allowing industry to better understand what these various initiatives will mean to industry in the future.

Lesson Learned 1.1: Major initiatives are best managed by a project management office from initiation.

The challenges with implementing FSAP during the first two years are well documented throughout this report. Significant enhancements were made across the initiative with the creation and subsequent expansion of the PMO. The existence of the PMO allowed CFIA's branches to focus primarily on the activities for which they were responsible, with the project horizontally managed from a central office. Furthermore, the PMO was responsible for establishing good performance reporting at the activity and output levels, which was an important and necessary task for monitoring short term progress.

Lastly, the financial data combined with other qualitative data collected through the evaluation suggest that effective project governance and an effective PMO reduce schedule delays resulting in under spending.

Lesson Learned 1.2: An analysis of Agency and market skills and availability, as well as costs of implementation ensures better support for the implementation of initiatives.

Prior to the implementation of any such ambitious initiative in the future, the CFIA should undertake an analysis to understand the impact on the workload of its staff and management, as well as to determine whether the skills exist within the federal public service or can be quickly found in the job marketplace. Ensuring that time is taken to understand issues associated with the skill levels of staff and management, the availability of that staff and management, gaps between what will be required for implementation and what skills and availability exist within the Agency, and the extent to which those gaps can be filled with additional hires, will be important for improving efficiency in the early years of new initiatives.

Furthermore, the costs of activities should be better anticipated through rigorous cost analysis that should take into account not only human resource costs but also the costs of materials that may be required to undertake specific activities. The cost analysis should be based on existing data from within the Agency, and from elsewhere when such information is determined relevant and comparable.

Table 11 provides the conclusions and recommendations/lessons learned by finding.

Table 11: List of Findings, Conclusions and Recommendations/Lessons Learned
Findings Conclusions Recommendations / Lessons Learned

Relevance

Finding 1.1: The Food Safety Action Plan is aligned with the Canadian Food Inspection Agency and the Government of Canada priorities.
One of several major food safety initiatives being implemented by the CFIA, the Food Safety Action Plan has a specific role among all initiatives and is aligned with the health and food safety priorities of the Government of Canada. N/A
Finding 1.2: A clearly identified need was established prior to the announcement of the Food Safety Action Plan. Although the initiative was announced primarily in response to major international food safety issues, there was evidence of due diligence to provide support for the need for the activities comprising the Food Safety Action Plan. This due diligence consisted chiefly of demonstrating the complexity of a growing global food trade and illustrating Canada's volume in that trade. Lesson Learned 1.2: An analysis of Agency and market skills and availability, as well as costs of implementation, better supports the implementation of initiatives.
Finding 1.3: FSAP related activities will need to continue beyond the end of the program in 2012-13. The Food Safety Action Plan was intended to continue beyond its initial five-year implementation period, and ongoing funding was provided for from the onset of the project. Most activities will continue beyond 2012-13 either because they will become ongoing operations within the Agency or because delays prevented them from being completed within the five-year period. N/A

Performance

Finding 2.1: The majority of the planned outputs have been produced or appear likely based on the rate of progress, appear likely to be produced before March 31, 2013. Outputs are being utilized as intended.
The Project Management Office has successfully monitored performance at the activity and output levels. The majority of the intended outputs were produced and are being measured. It is anticipated that all outputs will now be produced by December 31, 2013, with their utilization continuing as part of Agency operations. N/A
Finding 2.2: There is little documented evidence to support the achievement of expected outcomes by March 31, 2013; however, there are examples of specific successes at the activity level. There is little evidence suggesting that outcome data were collected or that there was a mechanism to collect outcome data on a systematic basis. Based on the data collected through the evaluation, there are isolated examples of short-term outcomes. Recommendation 1.1: Develop and formally implement an outcomes based performance measurement strategy for FSAP activities.
Finding 2.3: Industry struggles to understand how the various food safety initiatives are related. The Agency needs to provide industry with a mechanism to ensure that businesses understand holistically what these initiatives mean to them. Recommendation 1.2: Develop an approach to better communicate with industry how various food safety initiatives fit together.
Finding 3.1: The majority of funding was used for its intended purposes; however, not all allocations were aligned to the original budget plans. Limited planning leading up to and in the early years of Food Safety Action Plan implementation resulted in challenges with spending budget allocations on a timely basis. The PMO, once created, managed the budget issues. However, funding continued to be lapsed throughout the five years. Lesson Learned 1.1: Major initiatives are best managed by a project management office from initiation.
Finding 3.2: There are very little data regarding the cost-effectiveness of Food Safety Action Plan delivery. Limited data suggest that there was underestimation in the costs of conducting certain activities. The cost-effectiveness of Food Safety Action Plan activities could not be determined, owing to the lack of useful data. Alternate ways of undertaking the activities were not identified. The available data show that the costs for activities in some areas may have been underestimated when compared with actual costs. In these cases, the issue was rectified.

Recommendation 1.1: Develop and formally implement an outcomes based performance measurement strategy for FSAP activities.

Lesson Learned 1.2: An analysis of Agency and market skills and availability, as well as costs of implementation ensures better support for the implementation of initiatives.
Finding 3.3: Food Safety Action Plan was a lesson in horizontal project management for the Canadian Food Inspection Agency. The implementation of Food Safety Action Plan was a challenge for the Agency due to its lack of experience in project management. This would partially explain the challenges faced during the first two years. The introduction of the Project Management Office helped to address most of the issues. Lesson Learned 1.1: Major initiatives are best managed by a project management office from initiation.
Finding 3.4: The Food Safety Action Plan governance structure and decision-making process has improved since undergoing restructuring in 2011. The initial governance structure, which was in place until February 2011 was ineffective in comparison with its current design. It was comprised of too many committees, and those with decision-making authority often were not present at meetings. Since restructuring of governance, including the expansion of the PMO in February 2011 decision making has improved, and roles/responsibilities are generally clearer. Lesson Learned 1.1: Major initiatives are best managed by a project management office from initiation.
Finding 3.5: Working relationships between the CFIA and its counterparts were critical to the successes of the Food Safety Action Plan. The working relationships between numerous other government departments and agencies, as well as with counterparts in other countries have ensured many activities have been completed in a timely manner with positive results. N/A
Date modified: