Evaluation of the CFIA's Food Safety Program Modernization - Part 1
Final Report

List of Abbreviations

AC

Project Advisory Committee

AEB

Audit and Evaluation Branch

AHP

Animal Health and Zoonotics Program

CAHSN

Canadian Animal Health Surveillance Network

CFIA

Canadian Food Inspection Agency

CFSIN

Canadian Food Safety Information Network

CNPHI

Canadian Network for Public Health Intelligence

EC

Evaluation Committee

ESDP

Electronic Services Delivery Platform Project

FIMS

Food Inspection Modernization System

FSMI

Food Safety Modernization Initiative

FSP

Food Safety Program

GTA

Greater Toronto Area

HR

Human Resources Branch

IFIM

Improved Food Inspection Model

IM/IT

Information Management/Information Technology

ISO

International Organization for Standardization

MEL

Modernizing Equipment & Laboratories

MRAP

Management Response and Action Plan

OPS

Operations Branch

PAA

Program Alignment Architecture

PMC

Policy Management Committee

PPB

Policy and Programs Branch

PREP

Pre-Requisite Employment Program

PRP

Plant Resources Program

RPP

Report on Plans and Priorities

SCC

Standards Council of Canada

SFCA

Safe Food for Canadians Act

SFCAP

Safe Food for Canadians Action Plan

SMC

Senior Management Committee

TB

Treasury Board of Canada

TBS

Treasury Board Secretariat of Canada

WG

Working Group

Executive Summary

The Canadian Food Inspection Agency (CFIA or Agency) is a science-based regulatory agency guided by the following strategic outcome: "A safe and accessible food supply and plant and animal resource base" (CFIA, 2014a, p. 6).

The Food Safety Program (FSP) is an established, long-standing program at the CFIA.­ The objectives of the program are to "mitigate risks to public health associated with diseases and other health hazards in the food supply system and to manage food safety emergencies and incidents" (CFIA, 2014a, p. 21). The FSP uses a considerable portion of the Agency's annual resources. For example, from fiscal year 2011-12 through fiscal year 2014-15, the FSP accounted for between 45 and 50 per cent of the CFIA's overall expenditures.

Following the release of the Report of the Independent Investigator into the 2008 Listeriosis Outbreak (the Weatherill Report) in 2009, the Government of Canada committed funds in Budget 2011 to modernize Canada's food safety system (Government of Canada, 2011). As a result, the CFIA developed the Food Safety Modernization Initiative (FSMI) - a suite of eight projects designed to improve the FSP. CFIA recognizes the need for change within the Agency to address risks affecting its operations. These risks include the effectiveness and ability of programs such as FSP to support the overall strategic outcome.

This evaluation, carried out between January 2015 and March 2016, examines the $87.4 million initially allocated for the FSMI between fiscal years 2011-12 and 2014-15^Footnote 1 . It was atypical of Government of Canada evaluations, as it did not attempt to directly assess an established and ongoing program.

Key Findings and Recommendations

Key Findings

• There is a continued need for FSP and a demonstrated need for the FSMI.
• The design of individual FSMI projects is aligned with overall project objectives.
• The FSMI supports government-wide and CFIA priorities. It will enhance how the CFIA carries out its activities.
• Despite some delays, project activities are producing their respective outputs, but there is a lack of performance measurement to track the initiative's effects on the FSP.
• Project delays are largely a reflection of associated FSMI dependencies.
• Communication and stakeholder buy-in are common challenges across all FSMI projects.
• Financial data supports project delays, but it is projected most funding will be spent within the five-year timeframe ending in fiscal year 2015-16. Remaining funding has been extended to fiscal year 2017-18.
• There is minimal evidence to support the efficiency of FSMI projects, in part due to delays; however the fact that all projects were either implemented or are scheduled to be completed without significant overages provides reasonable evidence of efficiency.
• Generally, most FSMI initiatives are at too early a stage to report on outcomes. However, there is no evidence of a plan to track the initiative's effects on Agency programming.

Relevance: Need, Alignment with Government Priorities, and Alignment with Federal Roles and Responsibilities

Overall, the evaluation found the FSMI to be relevant and necessary for modernizing the FSP. Furthermore, FSMI projects are in line with the recommendations from the Weatherhill report, federal priorities, and the Government of Canada's Blueprint 2020 vision. The evaluation also found all FSMI projects were well designed to meet established program needs and objectives.

These projects represent the beginning of long-term change activities at the Agency; therefore, ongoing efforts will be required to fully realize their intended benefits on CFIA programs. Without this, there is a risk the effectiveness of the initiative's investments will be undermined. Ensuring appropriate levels of effort rests in part on maintaining CFIA staff and external stakeholder buy-in for change.

The following recommendation is meant to establish a culture of change and support Agency program improvements.

Recommendation 1: The Agency should establish and monitor an internal and external communication process to share ongoing information about the FSMI projects and their benefits.

Performance: Achievement of Outputs, Outcomes, and Demonstration of Efficiency and Economy

Despite some delays, all FSMI project activities are progressing along their intended plans and are producing outputs. Measureable outputs produced under the FSMI include:

• Completion of a new food inspection model
• Completion of a system plan for an Electronic Service Delivery Platform (ESDP)
• Completion of a plan for the Food Safety Information Network
• Implementation of core and refresher training for inspectors
• Implementation of the first wave of an improved food inspection model
  

Of particular concern, the evaluation demonstrates there is a lack of an established and effective means of measuring the influence of FSMI projects and their impacts on the FSP, as well as broader Agency programming. Without an established and effective means for measurement, there is a challenge in establishing the effectiveness of FSMI investments and, therefore, justifying future investments in Agency change initiatives.

The following recommendation is meant to establish the basis for measuring the effectiveness and efficiency of change initiatives affecting the FSP.

Recommendation 2: The Agency should develop and implement a performance measurement strategy to track how FSMI projects are affecting the Food Safety Program. The strategy should include:

• Indicators directly linked to overall Food Safety Program outcomes
• Indicators to measure the effects of FSMI investments on program efficiency

1.0 Introduction

The Canadian Food Inspection Agency's (CFIA) 2014 Departmental Evaluation Plan identified a two-part evaluation on the Food Safety Program (FSP), an established and long standing program at the CFIA.^Footnote 2 This report presents the findings of the FSP Modernization Evaluation - Part 1, with a focus on the impact of the Food Safety Modernization Initiative (FSMI). This initiative is a suite of eight projects designed to improve the FSP.

The evaluation was conducted in accordance with the Treasury Board (TB) Policy on Evaluation (2009) and its supporting Directive and Standard. The evaluation focused on the relationship between FSMI and FSP, and therefore examined issues of relevance and performance of both.

The evaluation was carried out between April 2015 and March 2016, and examined fiscal years 2011-12 through 2014-15.

1.1 Report outline

The remainder of this report is laid out in the following four sections:

• Section 2.0 describes the FSP and associated FSMI projects
• Section 3.0 details the evaluation approach, including limitations and mitigation strategies
• Section 4.0 outlines the evaluation findings
• Section 5.0 provides the concluding remarks and recommendations

2.0 The Food Safety Program

The CFIA is a science-based regulatory agency guided by the following strategic outcome: "A safe and accessible food supply and plant and animal resource base" (CFIA, 2014a, p. 6). The FSP is one of three main programs contributing to this strategic outcome. The other two programs are the Animal Health and Zoonotics Program and the Plant Resources Program.

These three main programs are supported by two others found under the Agency's Program Alignment Architecture (PAA):

• International collaborations and technical agreements
• Internal services

Within the FSP, there are a number of commodity-based sub-programs. These include:

• meat and poultry
• eggs
• dairy
• fish and seafood
• fresh fruits and vegetables
• processed products
• imported and manufactured food products (CFIA, 2014a, p. 6)

Figure 1 presents the Agency's PAA.

Figure 1: CFIA Program Alignment Architecture

The CFIA Program Alignment Architecture is a series of boxes that display the CFIA's strategic outcome, programs and sub-programs. The figure is organized vertically with the CFIA strategic outcome at the top, programs in the middle and sub programs at the bottom.

Strategic Outcome

The strategic outcome box is a pale blue rectangle with the following text inside of it: "Strategic Outcome A safe and accessible food supply and plant and animal resource base"

Program

The program section of the diagram consists of five grey rectangles with the following text written in them from left to right:

• P 1 - Food Safety Program
• P 2 - Animal Health and Zoonotics Program
• P 3 - Plant Resources Program
• P 4 - International Collaboration and Technical Agreements
• P 5 - Internal Services

Sub Program

Underneath the "P 1 - Food Safety Program" rectangle are seven blue rectangles, organized vertically, they have the following text written in them from top to bottom:

• SP 1.1 - Meat and Poultry
• SP 1.2 - Egg
• SP 1.3 - Dairy
• SP 1.4 - Fish and Seafood
• SP 1.5 - Fresh Fruit and Vegetables
• SP 1.6 - Processed Products
• SP 1.7 - Imported and Manufactured Food Products

Underneath the "P 2 - Animal Health and Zoonotics Program" rectangle are three blue rectangles, organized vertically, they have the following text written in them from top to bottom:

• SP 2.1 - Terrestrial Animal Health
• SP 2.2 - National Aquatic Animal Health Program
• SP 2.3 - Feed

Underneath the "P 1.3 - Plant Resources Program" rectangle are four blue rectangles, organized vertically, they have the following text written in them from top to bottom:

• SP 3.1 - Plant Protection
• SP 3.2 - Seed
• SP 3.3 - Fertilizer
• SP 3.4 - Intellectual Property Rights

The "P 4 - International Collaboration and Technical Agreements" rectangle does not have anything underneath it.

Underneath the "P 5 - Internal Services" rectangle are four blue rectangles, organized vertically, they have the following text written in them from top to bottom:

• SP 5.1 - Governance and Management Support
• SP 5.2 - Resource Management
• SP 5.3 - Asset Management Services

2.1 Objectives

The FSP is an established, long-standing program at the CFIA. Its objectives are to "mitigate risks to public health associated with diseases and other health hazards in the food supply system and to manage food safety emergencies and incidents" (CFIA, 2014a, p. 21).

FSP aims to achieve these objectives by:

• ensuring awareness of, and verifying industry compliance with, the relevant regulations and standards;
• responding to food safety emergency situations;
• undertaking food-safety related public awareness and engagement activities;
• ensuring that consumers have access to safe food and nutritional information;
• preventing instances of unfair food market practices; and
• supporting Agency participation in international organizations and collaborations involving food safety (CFIA, 2014a, p. 21).

Figure 2 presents the FSP logic model^Footnote 3. The model demonstrates how groups of activities under the FSP are meant to influence the Agency's strategic outcome.

Figure 2: FSP Logic Model

This is a diagram that describes the activities and outcomes of the Food Safety Program. On the left side of the diagram is a legend. The legend has the following text written in it from top to bottom:

• Strategic Outcome
• Intermediate Outcomes
• Immediate Outcomes
• Activities

The logic model is organized the same as the legend with "Strategic Outcome" at the top and "Activities" at the bottom.

Activities

The Activities section is surrounded by a rectangle with a blue dashed edge. Inside of this rectangle are eight blue rectangles organized horizontally in two rows of four boxes. The bottom four blue rectangles have the following titles from left to right:

• Regulatory and policy analysis as well as development
• Program design, advice, and training
• Science Advice and Laboratory Services
• Internal management

The top four blue rectangles have the following titles from left to right:

• Communication and stakeholder engagement
• Inspection, surveillance, and certification
• Contingency and preparedness
• International engagement and standard setting

There is an arrow going from the "Activities" section to the "Immediate Outcomes" section.

Immediate Outcomes

The Immediate Outcomes section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle are five green rectangles organized in horizontally. The five green rectangles have the following titles from left to right:

• Awareness of health related risks among the Canadian population
• Awareness and policies, regulations, and legislation among the Canadian public and other stakeholders
• Compliance with program, policies, requirements, and regulations among stakeholders
• Preparedness to prevent, address, and manage food related emergencies
• Contributions to international standards and agreements for hum-related risks

There is an arrow going from the "Immediate Outcomes" section to the "Intermediate Outcomes" section.

Intermediate Outcomes

The Intermediate Outcomes section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle are two green rectangles organized horizontally on top of six green rectangles organized horizontally. The two top green rectangles have the following titles from left to right:

• International foods are accessible to Canadians
• International markets are accessible to Canadian food

The six bottom green boxes have the following title from left to right:

• General public engages in behaviour to maintain food safety
• Domestic products are compliant with Canadian requirements
• Import products are complaint with Canadian requirements
• Canadian standards are recognized internationally
• Risks to the Canadian food supply are mitigated, minimized, or managed
• Canadian interests are reflected in science-based international rules, standards, and arrangements

There is an arrow going from the "Import products are complaint with Canadian requirements" rectangle to the "International foods are accessible to Canadians" rectangle.

There is an arrow going from the "Risks to the Canadian food supply are mitigated, minimized, or managed" rectangle to the "International markets are accessible to Canadian food" rectangle.

There is an arrow going from the "Canadian standards are recognized internationally" rectangle to the "International foods are accessible to Canadians" rectangle and the "International markets are accessible to Canadian food" rectangle.

There is an arrow going from the "Intermediate Outcomes" section to the "Strategic Outcome" section.

Strategic Outcome

The Strategic Outcome section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle is a pale blue rectangle with the following text inside of it "A safe and accessible food supply and plant resource base."

2.2 Risks

Like the other CFIA programs, the FSP faces a number of risks to its effectiveness and its ability to support the Agency's strategic outcome. This includes risks related to:

• Information management and IM/IT infrastructure - the program's ability to make risk-based decisions due to a lack of timely, accurate, and useful data and information.
• Inspection effectiveness - the program's ability to expeditiously prevent, detect, and respond to food safety threats.
• Scientific capability - the program's ability to use scientific capability to adapt and respond in a timely manner.
• Legislative, regulatory and program framework - the current legislative, regulatory and program framework's ability to support the effective delivery of the Agency's mandate.
• Managing change - the program's ability to effectively manage change on an ongoing basis. Change including modernization, staffing and etc.
• Transparency and leveraging relationships - the Agency's ability to capitalize on the opportunity to increase transparency and accountability to stakeholders.
• Emergency management - the program's ability to respond to multiple simultaneous or large-scale emergencies (CFIA, 2014a, pp. 9-12).

2.3 Stakeholders

The FSP targets a number of stakeholders, including:

• consumers
• producers
• industry
• other federal government departments and agencies
• provincial and territorial governments
• international organizations
• other countries (CFIA, 2014b, pp. 28-29)

The CFIA regularly interacts with these groups, ensuring their perspectives are considered in the development of FSP policies and strategies (CFIA, 2014b, pp. 28-29).

2.4 Resources

The FSP uses a considerable portion of the Agency's annual resources. As Table 1 indicates, between fiscal years 2011-12 and 2014-15, the FSP accounted for between 45 and 50 per cent of the Agency's overall expenditures.

Source: (Government of Canada, 2014, 2016)

As Table 2 demonstrates, between fiscal years 2011-12 and 2014-15, approximately half of all Full-Time Equivalents (FTEs) at the CFIA were dedicated to the FSP. During this time period, there was an increase in FSP staffing relative to other programming.

Source: (Government of Canada, 2013a, 2013b, 2014)

2.5 Supporting change at the CFIA

For more than a decade, the CFIA has recognized the need for change. Changes to Agency operations have been, and continue to be, meant to modernize its servicesfood safety, and are driven by:

• outdated food safety legislation and regulations;
• an increased need for the Agency to oversee a larger number of sectors;
• new varieties of food products demanded in Canada;
• new technologies in food production;
• increased consumer expectations for food safety information; and
• new approaches to food safety in jurisdictions outside of Canada (Crawford, 2015, p. 6).

The need for change is also emphasized in the CFIA's Long Term Strategic Plan. The plan notes the Agency must incorporate the following across all programming:

• "[an] increased focus on prevention;
• [a] strengthen[ed] citizen-centred service delivery culture;
• optimize[d] performance; [and]
• diverse talent supported by modern tools" (CFIA, 2013, p. 6).

The CFIA is also focussed on ensuring change activities align with the Government of Canada's Blueprint 2020 vision, which attempts to develop:

• "an open and networked environment that engages citizens and partners for the public good;
• a whole-of-government approach that enhances service delivery and value for money;
• a modern workplace that makes smart use of new technologies to improve networking; access to data and customer service; and
• a capable, confident and high-performing workforce" (CFIA, 2013, p. 10).

The Royal Assent of the Safe Food for Canadians Act (SFCA) on November 20, 2012, and the regulations to be made thereunder (the Safe Food for Canadians Regulations,) provide the basis for significant change in the food safety environment (CFIA & SFCA, 2012).

The SFCA directly addresses the need for updated legislation and regulations and will, upon fully coming into force, repeal and replace four existing Acts: the Fish Inspection Act, the Meat Inspection Act, the Canadian Agriculture Products Act, and the food related provisions of Consumer Packaging and Labelling Act.

The Act and forthcoming regulations will provide better management of food safety risks, more consistent inspection across all food commodities, greater clarification on established industry requirements, and improved consumer protection. They will also allow the Agency to focus more on prevention (CFIA & SFCA, 2012; Crawford, 2015).

2.6 The Food Safety Modernization Initiative (FSMI)

Following the release of Report of the Independent Investigator into the 2008 Listeriosis Outbreak (the Weatherill Report) in 2009, the Government of Canada committed funds in Budget 2011 to modernize Canada's food safety system (Government of Canada, 2011). As a result, the CFIA developed the FSMI.

The FSMI can be broken down into three main elements:

• inspection system modernization
• enhanced science capacity
• improved information management/information technology (IM/IT)

These three elements can be further broken down into eight distinct projects (sub-initiatives), which collectively support FSP modernization.

Table 3: FSMI Project Descriptions

Figure 3 presents the FSMI logic model. Developed for the FSP Evaluation - Part 1, the logic model shows project linkages to the collective outcomes of the FSMI. The FSMI logic model is colour-coded to identify:

• the elements of the FSMI being reviewed during the current evaluation (green)
• those that are related to these elements but not examined directly during the evaluation (blue)
• the intended outcomes of the projects, both individually and collectively (yellow)

To the extent possible, the logic model is organized according to the elements noted in Table 3 above. The largest exception involves improved IM/IT. The outcome under this element supports the achievement of outcomes for all other FSMI elements and their associated projects.

Figure 3: FSMI Logic Model

This is a diagram that describes the activities, outputs and outcomes of the Food Safety Modernization Initiative. On the left side of the diagram is a legend. The legend has the following text written in four vertical grey boxes from top to bottom:

• Projects/Sub-initiatives
• Activities
• Outputs
• Outcomes

On the top of the diagram is a horizontal legend made up of three grey boxes. These boxes represent the three elements of the FSMI. The three horizontal boxes have the following text written in them from left to right:

• Enhanced Science Capacity
• Inspection System Modernization
• Improved IM/IT

Projects/Sub-initiatives

The projects/sub-initiative section is made up of eight boxes and they are organized horizontally. Seven of the eight boxes are green and one box is blue.

From left to right, the first three boxes are below the "Enhanced Science Capacity" vertical legend box. The following text is written within the boxes from left to right:

• Developing a Laboratory Network Strategy, in a green box
• Modernizing Equipment and Laboratories, in a green box
• Enhanced Laboratory Response Capacity, in a green box

Continuing horizontally from left to right, the next four boxes are under the "Inspection System Modernization" vertical legend box. The following text is written within the boxes from left to right:

• Verifying Industry Compliance with Health Canada's revised listeria policy, in a blue box
• Recruitment and training of inspections, in a green box
• Improved food inspection model, in a green box
• Electronic service delivery platform (ESDP) , in a green box

Continuing from left to right, the last and eighth box is under the "Improved IM/IT" vertical legend box. The following text is written within the box:

• Increased efficiency through improved IM/IT, in a green box

Activities

The activities section is made up of fourteen boxes and they are organized horizontally.

From left to right, the first four boxes are below the "Enhanced Science Capacity" vertical legend box. All fourteen boxes are green. The following text is written within the boxes from left to right:

• Developing a plan for a Laboratory Network Strategy
• Modernizing laboratory installations and equipment
• Hire new laboratory staff
• Develop research papers on rapid testing methods

Continuing horizontally from left to right, the following six boxes are under the "Inspection System Modernization" vertical legend box. The following text is written within the boxes from left to right:

• Develop "refresher" training program
• Develop core (PREP) training program
• Develop national recruitment strategy
• Develop new food inspection delivery model
• Identify IM/IT solutions for the new food inspection delivery model
• Develop ESDP system

Continuing horizontally from left to right, the last four boxes are under the "Improved IM/IT" vertical legend box. The following text is written within the boxes from left to right:

• Pilot remote inspector IM/IT tools
• Pilot new and standardized IM/IT tools
• Develop data standards for information integration
• Acquire additional data storage and foundational infrastructure

There is an arrow pointing downwards from the "Developing a Laboratory Network Strategy" projects/sub-initiatives box to the "Developing a plan for a Laboratory Network Strategy" activities box.

There is an arrow pointing downwards from the "Modernizing Equipment and Laboratories" project/sub-initiatives box to the "Modernizing laboratory installations and equipment" activities box.

There is an arrow pointing downwards from the "Enhanced Laboratory Response Capacity" to the "Hire new laboratory staff" and the "Develop research papers on rapid testing methods" activities box.

There is an arrow pointing downwards from the "Recruitment and training of inspections" projects/sub-initiatives box to the "Develop "refresher" training program," "Develop core (PREP) training program" and the "Develop national recruitment strategy" boxes.

There is an arrow pointing downwards from the "Improved food inspection model" projects/sub-initiatives box to the "Develop new food inspection delivery model," "Identify IM/IT solutions for the new food inspection delivery model" and "Develop ESDP system" boxes.

There is an arrow pointing downwards from the "Electronic service delivery platform (ESDP)" projects/sub-initiatives box to the "Develop ESDP system" activities box.

There is an arrow pointing downwards from the "Increased efficiency through improved IM/IT" projects/sub-initiatives box to the "Pilot remote inspector IM/IT tools," "Pilot new and standardized IM/IT tools," "Develop data standards for information integration" and "Acquire additional data storage and foundational infrastructure" activities box.

Outputs

The outputs section is made up of twelve boxes and they are organized horizontally in two vertical rows, the top row has ten boxes, and the bottom row has two boxes. Four boxes are blue and eight boxes are green.

From left to right, the first three boxes are below the "Enhanced Science Capacity" vertical legend box. Two boxes are on the top row and one box on the bottom row. The following text is written within the boxes from left to right on the top row:

• Implementing the laboratory network strategy, in a blue box
• Validate and embed rapid testing methods into laboratory operations, in a green box

The following text is written within the box on the bottom row:

• Laboratories share scientific information, in a blue box

Continuing horizontally from left to right, the following seven boxes are under the "Inspection System Modernization" vertical legend box. Six boxes are on the top row, and one box is on the bottom row. One box is blue and six are green. The following text is written within the boxes from left to right on the top row:

• Collection of listeria samples, in a blue box
• Implement refresher training program, in a green box
• Implement core (PREP) training program, in a green box
• Implement national recruitment strategy, in a green box
• Implement new food inspection delivery model, in a green box
• Implement ESDP system, in a green box

The following text is written within the box on the bottom row:

• Exchange of electronic information between CFIA, industry, and foreign authorities, in a blue box

Continuing horizontally from left to right, the following two boxes are under the "Improved IM/IT" vertical legend box. The following text is written within the boxes from left to right on the top raw:

• Implement remote inspector IM/IT tools
• Implement new and standardized IM/IT tools

There is an arrow pointing downwards from the "Develop a plan for a Laboratory Network Strategy" activities box to the "Implementing the laboratory network strategy" outputs box and then an arrow downwards from this outputs box to the "Laboratories share scientific information" outputs box.

There is an arrow pointing downwards from the "Develop research papers on rapid testing methods" to the "Validate and embed rapid testing methods into laboratory operations" outputs box.

There is an arrow pointing left from the "Validate and embed rapid testing methods into laboratory operations" outputs box to the "collection of listeria samples" outputs box.

There is an arrow pointing downwards from the "Verifying industry compliance with Health Canada's revised listeria policy" projects/sub-initiatives box to the "Collection of listeria samples" outputs box.

There is an arrow pointing downwards from the "Develop refresher training program" activities box to the "Implement refresher training program" outputs box.

There is an arrow pointing downwards from the "Develop core (PREP) training program" activities box to the "Implement core (PREP) training program outputs box.

There is an arrow pointing downwards from the "Develop national recruitment strategy" activities box to the "Implement national recruitment strategy" outputs box.

There is an arrow pointing downwards from the "Develop new food inspection delivery model" and "Identify IM/IT solutions for the new food inspection delivery model" activities boxes to the "Implement new food inspection delivery model" outputs box.

There is an arrow pointing downwards from the "Develop ESDP system" activities box to the "Implement ESDP system" outputs box.

There is an arrow pointing from the "Implement new food inspection delivery model" outputs box to the "Implement ESDP system" outputs box and an arrow pointing in the other direction.

There is an arrow pointing downwards from the "Pilot remote inspector IM/IT tools" activities box to the "Implement remote inspector IM/IT tools" outputs box.

There is an arrow pointing downwards from the "Pilot new and standardized IM/IT tools" activities box to the "Implement new and standardized IM/IT tools" outputs box.

Outcomes

The outcomes section is made up of seventeen boxes and they are organized horizontally and in three vertical rows. The top row has eleven boxes organized horizontally, two are blue and nine are yellow. The middle row has five boxes organized horizontally, two are blue and three are yellow. The bottom row has one horizontal box that stretches across the width of the entire diagram, it is yellow.

From left to right, the first four boxes on the top row, and the first box on the middle row are below the "Enhanced Science Capacity" vertical legend box. The following text is written within the boxes from left to right on the top row:

• More efficient collection and use of scientific information, in a blue box
• More efficient laboratory instillations and equipment, in a yellow box
• Greater laboratory capacity, in a yellow box
• More efficient testing procedures, in a yellow box

The following text is written within the box on the middle row:

• Improved detection and response, in a yellow box

Continuing horizontally from left to right, the next seven boxes are under the "Inspection System Modernization" vertical legend box. There are four boxes on the top row, one is blue and three are yellow. There are three boxes on the middle row, two are blue and one is yellow.

The following text is written within the boxes from left to right on the top row:

• Earlier identification of contamination in high-risk RTE non-meat, in a blue box
• Existing inspectors have consistent inspection skills and knowledge, in a yellow box
• Newly hired inspectors have consistent inspection skills and knowledge, in a yellow box
• Consistent and efficient approach for all commodity types, in a yellow box

The following text is written within the boxes from left to right on the middle row:

• Consistent and more efficient food inspection approach applied across commodity types, in a yellow box
• Integrated and improved industry access to CFIA services, in a blue box
• Facilitated pre-clearance decisions and improved market access, in a blue box

Continuing horizontally from left to right, the next four boxes are under the "Improved IM/IT" vertical legend box. There are three boxes on the top row and one box on the middle row. All boxes are yellow.

The following text is written within the boxes from left to right on the top row:

• Consistent and efficient inspection activities, in a yellow box
• Tools in place to ensure CFIA program delivery consistency, in a yellow box
• Improved Agency data collection and data management systems, in a yellow box

The following text is written within the box on the middle row:

• Improved Agency wide data collection and management, in a yellow box

The box on the bottom outcome row is yellow and is underneath all three vertical legend boxes, the "Enhanced Science Capacity" vertical legend box, the "Inspection System Modernization" box and the "Improved IM/IT" vertical legend box. The text within this box is as follows:

• Strengthened federal food safety system

There is an arrow pointing downwards from the "Laboratories share scientific information" outputs box to the "More efficient collection and use of scientific information" outcomes box.

There is an arrow pointing downwards from the "modernize laboratory instillations and equipment" activities box to the "More efficient laboratory instillations and equipment" outcomes box.

There is an arrow pointing downwards from the "Hire new laboratory staff" activities box to the "Greater laboratory capacity" Outcomes box.

There is an arrow pointing downwards from the "Validate and embed rapid testing methods into laboratory operations" activities box to the "More efficient testing procedures" outcomes box.

There is an arrow pointing downwards from the "Collection of listeria samples" outputs box to the "Earlier identification of contamination in high-risk RTE non-meat" outcomes box.

There is an arrow pointing downwards from the "More efficient collection and use of scientific information," the "More efficient laboratory instillations and equipment," "Greater laboratory capacity," the "More efficient testing procedures" and the "Earlier identification of contamination in high-risk RTE non-meat" outcomes boxes to the "Improved detection and response" outcomes box.

There is an arrow pointing downwards from the "Implement refresher training program" outputs box to the "Existing inspectors have consistent inspection skills and knowledge" outcomes box.

There is an arrow pointing downwards from the "Implement core (PREP) inspector training program" and the "Implement national recruitment strategy" outputs boxes to the "Newly hired inspectors have consistent inspection skills and knowledge" outcomes box.

There is an arrow pointing downwards from the "Implement new food inspection delivery model" outputs box to the "Consistent and efficient approach to inspection for all commodity" outcomes box.

There is an arrow pointing downwards from the "Exchange of electronic information between CFIA, industry, and foreign authorities" outputs box to the "Integrated and improved industry access to CFIA services" and the "Facilitated preclearance decisions and improved market access" outcomes boxes.

There is an arrow pointing downwards from the "Implement remote inspector IM/IT tools" outputs box to the "Consistent and efficient inspection activities" outcomes box.

There is an arrow pointing downwards from the "Existing inspectors have consistent inspection skills and knowledge," the "Newly hired inspectors have consistent inspection skills and knowledge," the "Consistent and efficient approach to inspection for all commodity," and the "Consistent and efficient inspection activities" outcomes boxes to the "Consistent and more efficient food inspection approach applied across commodity types" Outcomes box.

There is an arrow pointing downwards from the "Implement new and standardized IM/IT tools" outputs box to the "Tools in place to ensure CFIA program delivery consistency" Outcomes box.

There is an arrow pointing downwards from the "Develop data standards for information integration" and the "Acquire additional data storage and foundational infrastructure" activities boxes to the "Improved Agency data collection and data management systems" Outcomes box.

There is an arrow pointing downwards from the "Improved detection and response," the "Consistent and more efficient food inspection approach applied across commodity types," the "Integrated and improved industry access to CFIA services," the "Facilitated preclearance decisions and improved market access," and the "Improved Agency data collection and data management systems" outcome boxes to the "Strengthened federal food safety system" outcomes box.

2.7 FSMI resources

A total of $139.8 million was allocated to the FSMI for fiscal year 2011-12 to fiscal year 2015-16. This includes $40 million that was reallocated from existing CFIA resources. Details on the CFIA internal reallocation are provided in Appendix C.

From the total of $139.8 million, $22.6 million was allocated to verifying industry compliance with Health Canada's Revised Listeria Policy. The Listeria project is not considered in the evaluation. In addition, $3 million was allocated to Health Canada as part of the initiative.

In summary, minus these allocations, the CFIA's portion of FSMI funding was $114.2 million.

Table 4 shows how the $114.2 million was distributed across the seven remaining projects in each of the five fiscal years from 2011-12 to 2015-16. The evaluation focussed on the funding and associated activities undertaken during fiscal years 2011-12 to 2014-15.

3.0 Evaluation Objectives

In accordance with the Treasury Board Policy on Evaluation (2009) and its associated Directive and Standard, the evaluation assessed the following core issues:

• continued need for the program (relevance)
• alignment with government priorities (relevance)
• alignment with federal government roles and responsibilities (relevance)
• achievement of expected outcomes (performance)
• demonstration of efficiency and economy (performance)

This evaluation was atypical of Government of Canada evaluations, as it did not attempt to directly assess an established and ongoing program. Rather, it examined a suite of time-limited projects and investments meant to improve the way in which the FSP - and more broadly, the Agency - operates.

Given the nature of the FSMI and its relationship to both the FSP and broader change at the CFIA, the evaluation also needed to:

• assess whether the FSMI projects represent an appropriate direction for Agency change; and
• assess whether the FSMI projects' collective impacts will effectively contribute to the Agency's operations beyond the completion of FSMI funding.

Therefore, this evaluation was designed to meet both TB requirements and the information needs of the Agency.

3.1 Evaluation team and support

The CFIA's Evaluation Directorate managed the evaluation and conducted it with the assistance of PRA Inc. The evaluation was guided by an Advisory Committee and a Working Group, which reviewed and provided feedback on the evaluation plan, evaluation matrix, logic model, findings, report and Management Response and Action Plan.

The evaluation team developed and conducted the following:

• the evaluation Matrix
• a document and literature review
• a data review and cost analysis
• interviews
• analysis and reporting

3.2 The evaluation matrix and FSMI logic model

To guide its work, the evaluation team developed an evaluation matrix, which included a list of the evaluation questions, organized according to the core TB issues (Appendix B). Each question was aligned with indicators and data collection methods. Aligning evaluation questions with the evaluation methods ensured the matrix maintain the evaluation scope. It also helped ensure that the evaluation methods were designed to address all of the identified evaluation questions.

Given the absence of an integrated performance measurement strategy and an approved logic model for FSMI, the evaluation team worked closely with members of the evaluation Working Group to finalize the logic model (Figure 3). The team modified a draft version of an FSMI logic model that was developed at the time of the initiative's inception with information gathered during preliminary discussions with members of the Working Group. This model outlined not only the immediate goals of the FSMI, but also highlighted their intended downstream impacts on Agency operations and ultimately strengthening the food safety system. This model was subsequently used to guide the evaluation, as it presented an overall view of the FSMI's expected impacts.

3.3 Data collection activities

The following data collection activities were conducted as part of the evaluation:

3.3.1 Document and literature review

The document and literature review leveraged existing documentation from within the Agency, along with academic and professional literature related to food safety.

A standardized template was used for the systematic review of this documentation and literature, and facilitated an overall understanding of these works. Information gathered through this review was valuable in addressing nearly all of the evaluation questions. It was particularly important in identifying FSMI project details and impacts, as well as changes to projects over time.

3.3.2 Data review and cost analysis

The data review and cost analysis generated quantitative financial information and, where possible, output and outcome information on FSMI projects. This information provided insight into the efficiency and economy of FSMI expenditures, specifically:

• whether FSMI funds were used as planned
• whether expenditures were reasonable
• whether projects were achieving their intended outcomes

3.3.3 Interviews

Thirty-seven interviews were conducted with representatives from Policy and Programs Branch, Operations Branch, Science Branch, Human Resources Branch, and IM/IT Branch. Interviewees were chosen based on their knowledge about the FSMI and related projects. The interviews were conducted after the document and literature review, in order to supplement existing information with contextual insight and interpretation on the part of program delivery staff and stakeholders. Further, it allowed the evaluation team to identify gaps and inconsistencies in information.

Eight interview guides were developed to facilitate semi-structured interviews - one for individuals associated with each of the seven FSMI projects, and a final guide for individuals involved with the initiative as a whole. These are provided in Appendix D.

3.4 Analysis and reporting

Information from the various data collection activities was analyzed in order to make conclusions regarding each of the questions included in the evaluation matrix. This triangulation process allowed the evaluation to use the strengths of each of the data collection activities to their best possible advantage. It also allowed the team to compare or confirm findings across data collection activities, as well as to supplement and contextualize the findings from one data collection activity to another.

The triangulation completed during the evaluation analysis also provides an opportunity to identify key information gaps. Not all questions identified in the evaluation matrix had an equivalent amount of data or information to support associated conclusions.

3.5 Limitations and challenges

The evaluation limitations and challenges, and the corresponding mitigation strategies, are described in Table 5 below.

4.0 Findings

The evaluation identified a number of project-specific observations, as well as observations that were similar across projects.

This section uses consolidated findings tables to present findings from the evaluation (Table 6 and Table 7). The intent is to allow readers to quickly identify the similarities and differences among the projects, and to facilitate general statements about the FSMI as a whole. A summary of observations precedes each table.

Observations included in the consolidated findings tables were compiled from information collected through all of the evaluation's data collection activities. A list of the documentation used to support the development of the tables is included in Appendix A.

4.1.1 Need and Objective: There is a continued need for FSP and a demonstrated need for the FSMI.

4.1.2 Project Design: The design of individual FSMI projects is aligned with overall project objectives.

4.1.3 Priorities, roles, and responsibilities: The FSMI supports government-wide and CFIA priorities. It will enhance how the CFIA carries out its activities.

4.2.1 Implementation and Outputs: Despite some delays, project activities are producing their respective outputs, but there is a lack of performance measurement to track the initiative's effects on the FSP.

4.2.1.1 Dependencies: Project delays are largely a reflection of associated FSMI dependencies.

4.2.1.2 Challenges: Communication and stakeholder buy-in are common challenges across all FSMI projects.

4.2.2 Economy: Financial data is in line with project delays, but it is projected most funding will be spent within the five-year timeframe ending in fiscal year 2015-16. Remaining funding has been extended to fiscal year 2017-18.

4.2.2.2 Efficiency: There is minimal evidence to support the efficiency of FSMI projects, in part due to delays; however the fact that all projects were either implemented or are scheduled to be completed without significant overages provides reasonable evidence of efficiency.

4.2.3 Outcomes: Generally, most FSMI initiatives are at too early a stage to report on outcomes. However, there is no evidence of a plan to track to track the initiative's effects on Agency programming.

4.1 Relevance

4.1.1 Need and objectives

The activities of the FSP strive to ensure Canadians have access to safe food (including imported food), and that the food we export is safe. The continued need for FSP is supported by evidence of the need for a stronger focus on prevention, a stronger focus on compliance, increased efficiency, and improved consistency for FSP. These needs point to broad outcomes, such as modernizing in order to strengthen the FSP, which the FSMI is striving to achieve.

This is supported by the Government of Canada's commitment to strengthen food safety in Canada by allocating funds directly for the purpose of modernizing the food safety system through inspection and science (Government of Canada, 2011). The CFIA's Long Term Strategic Plan and Corporate Risk Profile further support the need for the FSP, and place significant importance on modernizing the program.

While the FSP plays an important role in ensuring safe food in Canada, a number of factors continue to impact the effectiveness of its activities. These include:

• changes in the global supply chain and the volume of trade
• changes in consumer behaviour and expectations
• advances in science and technology
• changes to international regulations
• the changes in the structure of the industry (CFIA, 2012b; CFIA, 2013j)

The CFIA's need to address these risks supports the relevance of the FSMI, which focusses on inspection effectiveness, scientific capabilities, IM/IT infrastructure, and transparency and leveraging relationships with consumers and industry. Under this initiative, the CFIA is modernizing the FSP through, a stronger focus on prevention and compliance, a citizen-centered service culture, optimizing performance, building capacity, and modernizing tools (CFIA, 2013j).

In many instances it is possible to understand the needs targeted by FSMI projects as the intended individual project outcomes. These projects share similar collective outcomes that support the overarching needs for FSP. For example, the need for a consistent approach to conducting inspections is being addressed by the design and implementation of a single inspection approach and a consistent approach to recruitment and training across all food commodities. Together, these activities address the need for consistency in the approach of the FSP. The relationship between the need for FSP and the need for FSMI, as observed during the evaluation process, underpins the relationship between the implementation of the FSMI projects and their eventual influence on the FSP and further Agency programming.

Table 6 demonstrates that the needs and objectives of each of the FSMI projects are closely aligned. For example, in the case of the Improved Food Inspection Model (IFIM) project, the need for consistency of inspection through a systems-based approach aligns well with the project's objective of shifting to an audit-based inspection ap proach that is based on the implementation of hazard control plans.

4.1.2 Project Design

In some cases, as with ESDP and IFIM, the Agency has looked to other jurisdictions, such as the United States, the Netherlands, Australia, and New Zealand for evidence of project similarity and success. These reviews examined not only the applicability of other IM/IT solutions and approaches, but also the success of alternative approaches to inspection in other jurisdictions. In the case of IFIM, proven industry practices have also been incorporated into project design. This includes moving toward using established health and safety plan approaches, as well as International Organization for Standardization (ISO) standards for control procedures.

Others, like the FSIN project, have leveraged concepts from similar projects that have been implemented domestically. The FSIN project is a clear example of how theory provides a strong justification for the project. Specifically, the project's approach rests on the idea that building a laboratory network will allow better information sharing, will lead to better and more readily-available food safety data. This will support a more preventative approach to food safety. In certain cases, such as with the Modernizing Equipment and Laboratories (MEL) project, activities represent an extension of the ongoing work of the Agency, allowing the project to leverage this existing experience. It is clear that a strong theoretical justification for the work remains.

Table 6 indicates that individual FSMI projects have all leveraged existing evidence, theories, or experience bases in their design.

4.1.3 Priorities, roles, and responsibilities

Overall, FSMI projects align with federal priorities. For instance, all projects support the federal priority of a healthy Canadian population. Many FSMI projects also support priorities outlined in the Government's Blueprint 2020 vision, which calls for a modern and service-oriented federal government. For example, the investments in IM/IT and ESDP represent a significant step toward better use of technology and improving service delivery. Individually and collectively, FSMI projects directly support the Agency's ability to meet its strategic outcome.

The FSMI projects will not substantially change the Agency's roles and responsibilities with respect to food safety; however, some initiatives may result in subtle changes in the way the CFIA interacts with industry, and scientific and international partners. For example, inspection modernization is intended to provide greater clarification around the role of industry vs. the role of the CFIA with respect to food safety. Industry has the principal role in food safety, while the CFIA is responsible for overseeing industry compliance with regulations.

Table 6 presents observations regarding the need, federal government and Agency priorities, project objectives and project design that support the FSP and its modernization through FSMI. This information supports the analysis presented in Section 4.1.

4.2 Performance

4.2.1 Implementation and outputs

There were a number of activities planned under each of the FSMI projects. For example, in the case of the IFIM project, planned activities included:

• identifying common food safety objectives, strategies, and processes;
• developing a draft inspection model; and
• eventually developing and implementing a new inspection model.

Progress was made in all FSMI projects, and measurable outputs were produced. For example:

• the CFSIN planning stage was completed
• the MEL project completed its planned equipment purchases and is scheduled to complete its laboratory renovation work by March 2016
• recruitment and training activities were implemented well within the five-year FSMI project timeline, with the delivery of core^Footnote 8 and refresher training to inspectors activities such as the PREP program and refresher training having largely met their delivery targets.

Certain aspects of other projects have yet to be completed. For example, the design and implementation of ESDP has been delayed by about two years. As well, IFIM - now being expanded to include commodities beyond the FSP - was only in the early stages of implementation^Footnote 9 at the time of the evaluation. The standardized inspection approach had been validated in 125 facilities across the fish, feed, dairy, and greenhouse sectors, and the Agency was in the process of implementing it nationally for those commodities. Subsequent sectors are expected to follow a similar pattern, until all commodity areas have been transitioned over to the common inspection approach.

4.2.1.1 Dependencies

Delays in the implementation of the IFIM [which includes its IT component Food Inspection Modernization System (FIMS)] and the ESDP are good examples of projects being interdependent.

The FIMS and the ESDP are closely related, and progress in one is often dependent on the other. This dependency led to the decision to merge these two projects under the FSMI. However, both also had associated dependencies outside the scope of the FSMI. For example, effective implementation of the IFIM is supported by the regulatory change underway at the Agency.

Other dependencies, such as buy-in and participation with respect to enhancing science capacity, had fewer consequences, as projects are on track to be completed within the original five-year timeframe. For example, the MEL project was dependent on approval by Health Canada for extra space, the capital equipment procurement process, and PWGSC project management.

4.2.1.2 Challenges

Interviewees were asked to speak to the challenges faced by each of the FSMI projects. Many of the challenges identified in the consolidated findings table are common across other ongoing change activities within the CFIA. For example, common challenges include:

• obtaining staff, industry, and international stakeholder buy-in for changes in the Agency operations
• maintaining the momentum of long-term projects, particularly in the face of staff turnover among those responsible for their implementation
• understanding and accommodating the dependencies between change activities when attempting to improve programming at the CFIA

These challenges, along with project delays and observations made by interviewees during the evaluation, point to the need for:

• ongoing communication with stakeholders prior to, during, and after change activities;
• fostering a culture of change at the Agency; and
• undertaking a more integrated approach to individual change-related projects.

Building momentum appears to be an important aspect of successful change activities, and the need to do so appears even more acute given implementation delays in some of the FSMI projects noted above.

Table 7 presents observations regarding the implementation, production of outputs, achievement of outcomes, and efficiency of FSMI projects. This information supports the analysis presented in Section 4.2.

4.2.2 Efficiency and economy

4.2.2.1 Economy

Table 8 (below) provides information on budgets and expenditures, by project, for the four initially planned years of the FSMI and forecasts subsequent years. The figures include:

• Initial funding allocation - the initial amount of funding allocated to a project.
• Actual budget - total funding allocated to each project, which takes into account the transfers of responsibilities and funding (reductions) to Shared Services Canada (SSC), Treasury Board approved funding profile changes (re-profiles), and the allocation (addition) of prior year funding lapses carried forward through an established Department of Finance and TB mechanism.
• Expenditures - total actual resources utilized by the projects.
• Less authorities carried forward to the next fiscal year - includes budget reallocations as a result of CFIA decisions to carry forward funding from a previous year's variance into the current year, through an established Department of Finance and TB mechanism.
• Variance - unspent resources for a project in a given fiscal year. The difference between the amount planned and the amount actually spent for a given project of program in a given year. Calculated by subtracting expenditures for a given year from the actual budget in the same year. (Actual Budget less Expenditures)

The reinvestment of specific lapsed (the amount the variance captures) initiative funding back into the initiative is an Agency decision, not a TB requirement. These funds are pooled with other Agency variances and carried forward into subsequent years based on annual CFIA priorities and senior management decision making.

It is important to note FSMI allocations provided by TB were not considered fenced funding. The Agency independently decided to fence all funds associated with the FSMI, in order to track expenditures over time and ensure those funds remained within the initiative. Decisions to reallocate funding are made through the CFIA's governance structure. Since the Agency elected to track all FSMI funding, the overall level of funding for the initiative remained largely unchanged, despite changes to individual project allocations. The ESDP project, however, was tracked slightly differently, as it fit the federal government's requirements for tracking under the ePMF^Footnote 12 structure.

Table 8 demonstrates funding allocations for the FSMI. This information supports the analysis presented in Section 4.2.2.

Table 8: FSMI Funding Allocations - Overal Snapshot

Source: Corporate Management Branch, CFIA

Table 8 shows most FSMI projects are expected to spend their adjusted funding during the five-year period; however, some notable exceptions exist.

Recruitment and Training, for example, had a total spending variance of 52.8 per cent during fiscal years 2011-12 through 2014-15. This is a result of an early end to the planned refresher programming, along with an interruption to the planned recruitment and training activities due to delays in the implementation of IFIM. That project is currently implementing Wave 1 of a multi-wave roll out, and training needs to be aligned with the final operational model. At the moment, it does not appear that much of the variance in funding for Recruitment and Training will be reallocated to future years of the project; however, it will remain within the FSMI.

Another project with significant variances (38.5 per cent) over the initial four years was ESDP (FIMS). The merger of ESDP and FIMS in 2013 may have contributed to this. Delays in ESDP implementation have also resulted in a significant portion of its budget being re-profiled into later fiscal years through 2017-18. These re-profiled amounts are currently forecasted to be spent within that timeframe.

The IM/IT variance of 19.5 per cent is forecasted to be spent in the fifth year of the FSMI. In addition, reallocations from the IM/IT project took place as certain IM/IT functions were consolidated under Shared Services Canada (SSC), as a change in government structured responsibilities occurred during the FSMI period.

The development of a laboratory network strategy - although initially planned to be completed in three years - had an overall variance of 23.1 per cent for the four-year period. The funds have been reinvested in the project for fiscal year 2015-16.

Finally, there was about $3 million in variances (52.5 per cent) associated with the modernizing laboratories and equipment project during the initial four years of FSMI. However, much of this funding is forecasted to be reallocated to the project for the 2015-16 fiscal year.

In summary, changes in the funding allocations for the FSMI, from 2011-12 to 2014-15, came through either re-profiling or carry forward of funding based on senior management decisions. While the overall variance shown in Table 8, Column VI, indicates 27.1 per cent of FSMI funds were not spent, it does not take into account this funding that was authorized to be carried forward to the next fiscal year. When you consider the authorized carried forward amount of $16,647,499 (Column IV), the overall variance is reduced to 5.5 per cent. If the funds forecasted fiscal year 2015-16 are spent as planned, close to 95 per cent of the budgeted FSMI funds will have been spent.

Considering re-profiled amounts that were pushed out to fiscal years 2016-17 and 2017-18, along funding transferred to SSC, it is possible to forecast the $114.2 million initially allocated for the five-year FSMI project will be spent.

4.2.2.2 Efficiency

It is difficult to establish efficiency of project expenditures under FSMI due to lack of a similar, fully-costed and implemented alternative to each of the projects. Even projects that examined very close alternatives at their early stages - such the laboratory modernization and equipment renovation projects - involved estimates for planning purposes, rather than implemented alternatives. Other projects, like the IM/IT and ESDP projects, looked at established best practices, similar past work and alternative strategies prior to implementation.

4.2.3 Outcomes

When examining the FSMI logic model, immediate outcomes relate more directly to individual projects. However, the ultimate intent of all FSMI projects is to enhance the FSP and ultimately to strengthen the food safety system.

With the exception of the Recruitment and Training project, the evaluation was unable to find evidence of measured outcomes for the projects examined. In the case of recruitment and training, concerted efforts were taken to demonstrate core and refresher training among inspectors, along with the acquisition of skills, resulted in successful inspection work. Evidence also suggests that more such tracking is planned for the future.

The lack of a performance measurement strategy is particularly problematic, given many of the Agency's current FSP performance measures relate to stakeholder compliance with program requirements. Without additional information, these will be insufficient to assess the impact of the FSMI on the FSP and broader Agency programming.

For example, compliance measures used to track program requirements under the previous inspection model are different from compliance measures tracking program requirements under the new inspection model. Since these two forms of measuring compliance are different, it will not be possible to compare them in order to determine which form of compliance measurement is more effective or efficient. The reality is, compliance measurement is being modernized to align with the new audit-based inspection approach that is risk-based and designed to enhance the Agency's inspection system.

The Agency's plan to use a risk-based approach to determine inspection activities will further complicate the use of compliance measures as a tool for understanding improvements in food-related risks to the Canadian public. Under this model, appropriate targeting of resources to high-risk producers and industries will likely be associated with identifying higher levels of non-compliance. This does not necessarily reflect a systematic increase in food safety risk.

Both of these complications suggest more direct measures of food safety risks and other aspects of CFIA program performance will be necessary to determine FSMI effectiveness and efficiency.

The need for better performance measurement becomes even more acute when examining the need for each FSMI project outside the context of the initiative itself. As Table 7 notes, there are clear plans to build on all of the FSMI projects to support change beyond the FSP. In all likelihood, this will require not only concerted effort on the part of the Agency, but substantial additional funding. Without the means to establish the effectiveness of past change initiatives such as the FSMI, accessing future funding could become difficult.

5.0 Conclusions

This evaluation found individual projects under the FSMI were well designed to meet established program needs and objectives. Despite some delays, the performance of the FSMI is upheld by the evidence that individual projects are on track to be completed with only minor setbacks.

These projects represent the beginning of long-term change activities at the Agency; therefore, ongoing efforts will be required to fully realize their intended benefits on CFIA programs, including the FSP. Without this, the effectiveness of these investments could be undermined.

Perhaps more importantly, there is a lack of a performance measurement approach for FSMI projects and their impacts on the FSP, as well as broader Agency programming. This could pose challenges in establishing the effectiveness of FSMI investments and, therefore, justifying future investments in Agency change initiatives.

5.1 Recommendations

Relevance: Need, Alignment with Government Priorities, and Alignment with Federal Roles and Responsibilities

Overall, the evaluation found the FSMI to be relevant and necessary for modernizing the FSP. Furthermore, FSMI projects are in line with the recommendations from the Weatherhill report, federal priorities, and the Government of Canada's Blueprint 2020 vision. The evaluation found all FSMI projects were well designed to meet established program needs and objectives.

These projects represent the beginning of long-term change activities at the Agency; therefore, ongoing efforts will be required to fully realize their intended benefits on CFIA programs. Without this, there is a risk the effectiveness of the initiative's investments will be undermined. Ensuring appropriate levels of effort rests in part on maintaining CFIA staff and external stakeholder buy-in for change.

The following recommendation is meant to establish a culture of change and support Agency program improvements.

Recommendation 1:

The Agency should establish and monitor an internal and external communication process to share ongoing information about the FSMI projects and their benefits to those involved.

Performance: Achievement of Outcomes, and Demonstration of Efficiency and Economy

Despite some implementation delays, FSMI performance is supported by the fact individual projects are on track to be completed with minor setbacks.

Of particular concern, the evaluation demonstrates there is a lack of an established and effective means of measuring the influence of FSMI projects and their impacts on the FSP, as well as broader Agency programming. Without an established and effective means for measurement, there is a challenge in establishing the effectiveness of FSMI investments and, therefore, justifying future investments in Agency change initiatives.

The following recommendation is meant to establish the basis for measuring the effectiveness and efficiency of change initiatives affecting the FSP.

Recommendation 2:

The Agency should develop and implement a performance measurement strategy to track how FSMI projects are affecting the Food Safety Program. The strategy should include:

• Indicators directly linked to overall Food Safety Program outcomes
• Indicators to measure the effects of FSMI investments on program efficiency

Appendix A – References

References (cited in final report)

CFIA. (2013). Blueprint 2020: Building tomorrow's CFIA together: Getting started - getting your ideas.

CFIA. (2014a). Canadian Food Inspection Agency 2014-15 Report on Plans and Priorities.

CFIA. (2014b). Canadian Food Inspection Agency Food Program Framework.

CFIA, & SFCA. (2012). Safe Food for Canadians Act. Retrieved February 12, 2016, from https://www.inspection.gc.ca/about-the-cfia/acts-and-regulations/regulatory-initiatives/sfca/eng/1338796071420/1338796152395

Crawford, J. (2015). Strengthening Canada's Food Safety System - CASA 99th Annual Educational & Training Seminar April 20-13, 2015.

Government of Canada. (2011). The Next Phase of Canada's Economic Action Plan: A Low-Tax Plan for Jobs and Growth. Retrieved November 17, 2015, from http://www.budget.gc.ca/2011/plan/Budget2011-eng.pdf

Government of Canada. (2013a). 2012-2013 Departmental Performance Report. Retrieved August 10, 2015, from https://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2012-2013-dpr/eng/1377176926809/1377177134114?chap=3;%202013-14%20DPR%20:%20https://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2013-2014-dpr/eng/1409769354767/1409769355486?chap=4#s7c4

Government of Canada. (2013b). Section II - 2011-2012 Departmental Performance Report. Retrieved August 10, 2015, from https://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2011-2012-dpr/eng/1348777953917/1348778053447?chap=4

Government of Canada. (2014). 2013-2014 Departmental Performance Report. Retrieved August 10, 2015, from https://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2013-2014-dpr/eng/1409769354767/1409769355486?chap=3#s5c3

Government of Canada. (2016). 2014-2015 Departmental Performance Report. Retrieved February 3, 2016, from https://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2014-2015-dpr/eng/1442253072937/1442253073921?chap=2

References (to inform findings)

Available upon request

Appendix B – Evaluation Matrix

Evaluation of the CFIA's FSP-PART 1 Evaluation Matrix

Issue #1: Continued Need for the Foods Safety Program (FSP) and Food Safety Modernization Initiative (FSMI)

Issue #2: Alignment with Government Priorities/data

Issue #3: Alignment with Federal Roles and Responsibilities

Issue #4: Delivery of FSMI Expected Outputs

Issue #5: Achievement of FSMI Expected Outcomes

Issue #6: Demonstration of Efficiency and Economy

Appendix C – CFIA Internal Reallocation

Table 9 presents the distribution of the internal reallocations within the FSMI in each of the fiscal years 2011-2012 through 2015-2016.

• In the 2008-2009 fiscal year, the Agency identified annual efficiencies in order to repay a $20 million loan from the management reserve. This loan was initially provided to pay for retroactive payments associated with a reclassification of inspector positions. Following the repayment of this loan, the efficiencies were maintained and the associated resources for the 2012-2013 through 2015-2016 fiscal years were relocated to the FSMI.
• In the 2009-2010 fiscal year, the Agency faced $5.3 million in additional rent requirements from Public Works and Government Services Canada (PWGSC). Again, the Agency identified internal efficiencies in order to repay this rental requirement over the subsequent years. The reduction in rental costs that produces these efficiencies was maintained, allowing for reallocations towards FSMI in fiscal years 2013-2014 through 2015-2016.
• In the 2010-2011 fiscal year the CFIA was required to identify efficiencies from 1.6million annual payment on a $8.1 million loan for the fix-up of its Ottawa headquarters location. The final payment on this loan was to be paid in the 2014-2015 fiscal year, allowing for reallocation towards FSMI in the 2015-2016 fiscal year.
• An internal administrative audit identified a variety of efficiencies within the Agency, resulting in $13.1 million of resource reallocation towards FSMI. These reallocations took place in fiscal years 2013-2014 through 2015-2016.

Appendix D – Interview Guides

Food Safety Modernization Initiative - Overall

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved directly or indirectly with one or more of these projects / sub-initiatives, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the projects/sub-initiatives will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

These issues relate to FSMI and FSP generally, and we would like you to reflect on them from a higher level or Agency strategic perspective when answering the questions on the subsequent pages of this interview guide.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA?

1. Please briefly describe your involvement with FSMI and FSP.

Need for Programming

2. From your perspective, what needs does/will FSMI address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically, does FSMI address? Could you please explain how it is intended to do so?

Alignment with Government Priorities

4. From your perspective, will the implementation of FSMI affect the way that FSP supports the Agency's strategic outcome (A safe and accessible food supply)?

5. Did/will the implementation of FSMI affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

6. Has/will the implementation of FSMI affect the roles and responsibilities of the CFIA in any way? If so, what changes do you anticipate?

7. Has/will FSMI change(d) the scope of the CFIA's work or its involvement with other federal departments?

8. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

9. Thinking about FSMI, have there been challenges to implementation and if so, what were they? What has been the impact (positive or negative)?

10. Are there key dependencies that have/had to take place for FSMI to be a success? Please elaborate.

11. Were there any unexpected benefits from the implementation of FSMI?

12. Considering the expected outcomes of FSMI, have any been realized? Do you anticipate others to be realized in the future?

13. Please describe the performance monitoring and reporting approach for FSMI. How is it used to support the implementation of the initiative? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

14. What have been the implications of any deviation(s) from planned spending that took place during the implementation of FSMI?

15. Thinking about the FSP, in what way(s) has/will FSMI affect(ed) the efficiency of the program?

Conclusion

16. Are there any other points regarding FSMI that you would like to discuss and that would be relevant to the evaluation work currently underway?

17. In addition, are there documents that could help the evaluation team better understand the points we discussed today?

Thank you.

Improved Food Inspection Model

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with the Improved Food Inspection Model, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on the Improved Food Inspection Model specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean the Improved Food Inspection Model.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way? If so, how?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/ will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any the following?

• Completion of the main activities identified for your project
• implementation of a new inspection model;
• improved inspection efficiency; and
• a consistent application of inspections throughout the Agency's business lines;

16. Please describe the performance monitoring and reporting approach for your project. How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team better understand the points we discussed today?

Thank you.

Electronic Service Delivery Platform

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with the ESDP, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on the ESDP specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean the ESDP.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address.

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of the following:

• Completion of the main activities identified for your project
• improved information exchange between the CFIA and industry, and foreign authorities;
• improved industry use of CFIA services; and
• increased pre-clearance activities.

16. Please describe the performance monitoring approach to your project. How is it used to support the continued development and implementation of your project over time? Would you continue to use this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviations(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, how has or is your project likely to affect the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there are other documents about your project that could help the evaluation team understand the points we discussed today?

Thank you.

Recruitment and Training of Inspectors

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency Through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with the Recruitment and Training of Inspectors, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on the Recruitment and Training of Inspectors specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean the Recruitment and Training of Inspectors.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically, does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place in order for your project to be a success? Please elaborate

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

• Completion of the main activities identified for your project:
• consistency in the inspector recruitment and training;
• an appropriate level of skills and knowledge among inspectors;
• an appropriate number of inspectors for the agency; and
• inspectors that are trained to implement the new food inspection model?

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Laboratory Network Strategy

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with Developing a Laboratory Network Strategy, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Developing a Laboratory Network Strategy specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Developing a Laboratory Network Strategy.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does or will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically, does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

• Completion of the main activities identified for your project

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Modernization of Equipment and Laboratories

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with Modernizing Equipment and Laboratories, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Modernizing Equipment and Laboratories specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Modernizing Equipment and Laboratories.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that have/had to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

• Completion of the main activities identified for your project:
• improved laboratory efficiency; and
• increased laboratory capacity; and
• improved laboratory response to emergencies.

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending, if any, that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Enhancing Laboratory Response Capacity

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with Enhancing Laboratory Response Capacity, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Enhancing Laboratory Response Capacity specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Enhancing Laboratory Response Capacity.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way? If so, how?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of the following:

• Completion of the main activities identified for your project;
• increased laboratory capacity; and
• improved laboratory response to emergencies.

16. Please describe the performance monitoring approach to your project. How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviations(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program ?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents about your project that could help the evaluation team understand the points we discussed today?

Thank you.

Improved IM/IT

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

• The Improved Food Inspection Model;
• The Electronic Services Delivery Platform (ESDP);
• The Recruitment and Training of Inspectors;
• Developing a Laboratory Network Strategy;
• Modernizing Equipment and Laboratories;
• Enhancing Laboratory Response Capacity; and
• Increased Efficiency through Improved Information Management/Information Technology (IM/IT).

Since you have been involved with Increased Efficiency through IM/IT, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

• the continued need for FSP and FSMI;
• their alignment with government priorities;
• their alignment with federal roles and responsibilities;
• the delivery of FSMI related outputs;
• their achievement of objectives/outcomes; and
• the demonstration of efficiency and economy.

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Increased Efficiency through IM/IT specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Increased Efficiency through IM/IT.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place in order for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

• Completion of the main activities identified for your project:
• allowing for remote access to CFIA networks; and
• use of IM/IT tools by inspectors.

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.