Language selection

Search

Archived - Evaluation of the Canadian Food Inspection Agency's (CFIA's) Federal Assistance Program
Final Report

This page has been archived

Information identified as archived is provided for reference, research or record-keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

PDF (1.15 mb)

Table of Contents

Executive Summary

Under the Financial Administration Act, an evaluation of the Federal Assistance Program (FAP) is required every five years. In addition, Treasury Board's (TB) Evaluation Policy requires that all direct program spending is evaluated every five years. This report presents the results of the evaluation undertaken in 2015.

The FAP is currently the CFIA's sole contribution program and is used to fund projects and initiatives that advance the Agency's strategic outcome; namely a safe and accessible food supply, plant and animal resource base.

The FAP is unique in that it is not a standard federally-funded contribution program; individual branches are required to use their operating budgets (i.e., A-base funding) as the source of funds for each contribution (i.e. there is no separate/distinct federal budgetary "vote"). As well, contribution recipients are not solicited in a public venue, but are identified by the CFIA officials in their individual dealings with stakeholder groups. Recipients can be universities, non-governmental organizations, international bodies and any other organization or individual except for other Canadian federal government departments or agencies.

The FAP has annual maximum allowable expenditures of $4.5 million, though it averages about $1.5 million. Over the four-year period of the evaluation's scope (2011-15), the FAP contributed $5.9 million over 23 agreements, with the smallest agreement at $5,500 and the largest at $2 million.

Key Findings and Recommendations

Overall, the program was found to be effectively managed, providing the Agency with opportunities to support third parties in activities that contribute to the CFIA's strategic outcome of a safe and accessible food supply and plant and animal resource base.

The findings and recommendations of the evaluation are provided below by the five core evaluation issues of the TB Policy on Evaluation (2009).

Relevance: Continued Need for FAP

The evaluation found evidence of a strong need for a broad contribution program, i.e., one that supports the Agency's strategic outcome. The FAP provides the flexibility to access expertise outside the Agency, build research partnerships, national and international, and support research of mutual interest, build capacity, and support other initiatives of value to the CFIA.

CFIA managers who used FAP were strongly supportive of it, and, as noted below, the Program has been successfully addressing its intended outcomes. However, the evaluation found a general lack of awareness of the Program in the Agency, which may have contributed to its under-usage, spending only one third of its $4.5 million annual funding limit.

Recommendation 1: The CFIA should increase awareness of the FAP across the Agency.

Relevance: Alignment with Government Priorities

FAP is aligned with federal government and CFIA priorities. All FAP funded project objectives were found to be consistent with the CFIA's strategic outcome.

Relevance: Alignment with Federal Government Roles and Responsibilities

Given that all FAP projects are consistent with the Agency's strategic outcome of "a safe and accessible food supply, plant and animal resource base," it is appropriate for the federal government as represented by the CFIA to be managing and delivering the FAP.

Performance: Achievement of Expected Outcomes

Most of the expected immediate outcomes have been achieved: supporting international engagement; veterinary and science capacity building; organizational development; and knowledge enhancement, all linked to the CFIA's strategic outcome.

Although good progress has been made since 2013, the use of indicators to report on performance is uneven across projects, and there is no systematic critical review of recipient reports, which could enhance objective monitoring and allow for a more cohesive review of all FAP projects.

Recommendation 2: The CFIA should develop and implement more standardized and detailed project and program performance monitoring and reporting.

Performance: Demonstration of Efficiency and Economy

The value of FAP is in its flexibility as a funding vehicle, its ability to leverage funds and its use as a tool to build partnerships, meaningful outreach/exposure and capacity beyond the CFIA. The Program appears to be efficiently and effectively managed. There is, however, room for improvement in terms of the administrative burden placed on recipients.

There is great variation in the type of projects, from one-day workshops to research networks involving multiple Canadian and international partners. Small and straight forward projects such as a workshop could involve a less extensive risk assessment, less frequent reporting by the recipient, and/or delegated approval (e.g., by Branch Head instead of President).

Recommendation 3: The CFIA should scale project administrative requirements based on type and size.

There was an expression of concern surrounding multi-year agreements and repeat agreements. While all FAP projects are subject to strict and robust governance, there is no mechanism in place that provides guidance around repeat and multi-year funding.

Recommendation 4: The CFIA should develop guidelines for the funding of repeat FAP projects over multiple years.

1. Introduction

The Canadian Food Inspection Agency's (CFIA's) 2015 Evaluation Plan included an evaluation of the Federal Assistance Program (FAP), to be completed in accordance with the Treasury Board Policy on Evaluation (2009) and the supporting Directive and Standard. The evaluation addressed the five core evaluation issues:

2. Program Profile

2.1 Background

In 1997, the CFIA established the Federal Inspection System Assistance Program, which was later renamed the Federal Assistance Program (FAP). The FAP is currently the CFIA's sole contribution program and is used to fund projects and initiatives that advance the Agency's strategic outcome; namely a safe and accessible food supply, plant and animal resource base. The FAP is funded annually from individual branch budgets or via other special initiatives (e.g., Food Safety Action Plan or Growing Forward). As such, FAP is a tool CFIA program managers utilize to broaden their reach by supporting third-party projects that are intended to contribute toFootnote 1:

The FAP has evolved significantly since its introduction. The following chart outlines the program's evolution.

Figure 2-1: Evolution of FAP Activities

1997
CFIA introduces "Federal Inspection System Assistance Program; later renamed to the Federal Assistance Program

2006
First version of Terms and Conditions for the FAP approved (revised and approved by Minister Ritz in 2012)

2006-2010
Ad hoc FAP agreements are coordinated by Costing, Financial Policy and Internal Control, CMB

2011
Following internal audit and evaluation, FAP is assigned to Business Integration Services, PPB

2012
FAP is moved to Program, Regulatory and Trade Policy Division, PPB, as a result of PPB restructuring

2013-2015
FAP program design and all key documents, forms and tools are revised or developed; existing FAP agreements are approved, coordinated and managed via evolving program materials and administrative practices

Source: CFIA (February 6, 2014), FAP Deck; Presented to CFIA Policy Integration Committee, with minor edits

2.2 Governance, Roles and Responsibilities

The FAP is unique in that it is not a standard federally-funded contribution program; individual branches are required to use their operating budgets (e.g., A-base funding) as the source of funds for each contribution (i.e. there is no separate/distinct federal budgetary "vote"). As well, contribution recipients are not solicited in a public venue, but are identified by the CFIA officials in their individual dealings with stakeholder groups with the aim of broadening the CFIA's reach by supporting collaborative and partner initiatives.

There are a number of key individuals responsible for the delivery of the program as follows:

  1. FAP Contribution Coordinator (CC) – Located within the Program, Regulatory and Trade Policy Directorate, Policy and Programs Branch, the CC is the main contact for the Program. The FAP CC advises Responsibility Centre Managers on the FAP process, maintains data on the progress of all FAP applications and ensures that relevant performance indicators are included in each FAP contribution agreement. In addition, the CC develops the recipient audit plan and coordinates the three-year forecast planning.Footnote 2, Footnote 3
  2. Responsibility Centre Managers (RCM) ensure the projects they recommend for funding under FAP contribute to the Agency's strategic outcome and meet the required Terms and Conditions. They are responsible for conducting a risk assessment for every project and obtaining Branch Head approval.Footnote 4 RCMs liaise with the applicant/recipient, and are expected to develop three-to-five relevant performance indicators (in collaboration with the recipient) for each contribution agreement they manage, and monitor the status of the project, including payments and progress.Footnote 5, Footnote 6
  3. Branch Head is required to sign and sponsor all FAP funding requests, risk assessments and Contribution Agreements that originate within their branch.Footnote 7
  4. Recipients are responsible for preparing and completing a FAP Request for the Funding Application packageFootnote 8 in collaboration with the RCM and for submitting progress reports and/or a final project report according to the schedule in the Contribution Agreement.Footnote 9

2.3 Recipients and Projects

The list of potential recipients is almost all encompassing, with the only stated exclusion being other federal departments, agencies, and Crown corporations.Footnote 10

According to the FAP Guide, projects selected under the FAP generally support:

Over the four-year period from 2011-12 to 2014-15, there were 23 agreements with 15 different recipient organizations to sponsor 18 individual projects (several projects had multiple agreements). The recipient organizations and projects supported by FAP from 2011-12 to 2014-15 are shown in Table 2-1.

Table 2-1: List of Recipient Organizations and Projects supported by FAP from 2011-12 to 2014-15
Recipient Project Number of Contribution Agreements Contribution Amount
Canadian Cattle Identification Agency (CCIA) (2012-13) Traceability National Information Portal – CCIA Requirements 1 $33,414
Canadian Veterinary Medical Association (2012-15) Canadian Veterinary Reserve (CVR) 1 $689,068
Conference Board of Canada (2014-15) Third Report of the Food Safety Performance World Ranking Study 1 $60,000
Conseil québecois des espèces exotiques envahissantes (2013-14) Emerald Ash Borer Management Workshop – Quebec 1 $5,500
Genome Alberta (2014-16) The 2014 Program on Research and Innovation Leading to Rapid Genomics Response to the Porcine Epidemic Diarrhea virus (PEDv) 1 $100,000*
International Plant Protection Convention (IPPC) (2013-14) Special International Plant Protection Convention Trust Fund 1 $376,000
Safe Food Canada – The Learning Partnership (2014-15) Preparing to implement Safe Food Canada (SFC) – The Learning Partnership 1 $350,000
University of Guelph (2014-15) Research support to the Multi-Criteria Risk Prioritization Framework for Food Safety Hazards in Canada 1 $38,000
University of Montreal (2011-14) Ateliers d'Initiation au Leadership Vétérinaire (ILV)/Veterinarian Leadership Workshops 3 $30,000
University of Montreal (2013-15) (GREZOSP)/Epidemiology and Public Health Research group 1 $31,666
UPEI – Atlantic Veterinary College (2011-15) Canadian Regulatory Veterinary Epidemiology Network (CRVE-NET) 3 $800,000
Veterinarians without Borders – Canada (2014-15) Strengthening Leadership and Capacity for Securing Animal and Public Health in Canada and the International Community 1 $250,000
Veterinarians without Borders – Canada (2011-14) Veterinarians without Borders – Canada 2 $1,200,000
Williams and Associates Forestry Consultants Ltd. (2013-14) Emerald Ash Borer Management Workshop – Ontario 1 $20,000
World Bank (2013-14) Global Food Safety Partnership 1 $300,000
World Health Organization – CODEX (2013-14) World Health Organization Enhanced Participation in Codex Alimentarius Commission 1 $200,000
World Organisation for Animal Health (2011-114) Extraordinary Contribution to the World Organisation for Animal Health 1 $1,500,000
World Organisation for Animal Health (2014-15) OIE Global Conference on Aquatic Animal Health: 'Riding the wave to the future' 1 $20,000
TOTAL 23 $6,003,648

Source: FAP Contribution Agreements. *Genomics Alberta is only to receive the contribution amount in fiscal year 2015/16.

2.4 Resources

As noted above, the FAP is funded annually from individual branch budgets. Alternative forms of funding may be identified via other special initiatives. The total value of all contributions under the FAP in any one year cannot exceed $4.5 million. The maximum amount payable to any one recipient in any one year cannot exceed $2 million. The Terms and Conditions for the FAP indicate that contributions are provided at the minimum value to attain the project's expected results.

Table 2-2 below outlines annual funding for FAP projects from 2011-12 to 2014-15. On average, actual total contributions have amounted to $1.5 million, well below the $4.5 million limit per annum. Over the four-year period, the FAP contributed $5.9 million over 23 agreements, for an average award per contribution agreement of $260,000; the smallest agreement was $5,500 and the largest $2 million.

Table 2-2: Annual FAP Funding (2011-12 to 2014-15)
2011-12 2012-13 2013-14 2014-15 Total
Total Annual FAP Funding $1,215,000 $1,477,914 $2,121,666 $1,089,068 $5,903,648

Source: Recipients and Annual Values of Contribution Agreements

2.5 FAP Logic Model

The Logic Model, included in the FAP Performance Measurement Strategy was revised by the evaluation team and validated with program management and through the evaluation process. The Logic Model is provided below in Figure 2-2. The evaluation identified four outcomes, slightly different from those listed by the Program, as noted in section 2.3 above and discussed below. Indicators and performance measures have been drafted and will be used and validated in the future.

Figure 2-2: FAP

Click on image for larger view
Federal Assistance Program (FAP) Logic Model. Description follows.

Description for flowchart – Federal Assistance Program (FAP) Logic Model

The diagram describes the activities and outcomes of the Federal Assistance Program (FAP) logic model, starting from Input (stage 1) and ending with the CFIA Strategic Outcome of "A safe and accessible food supply, plant and animal resource base" (stage 7).

Each of the stages leads to the stage after. For example, Input (stage 1) leads to Activities (stage 2).

Inputs
(Stage 1 of 7)
  • a1) Project/Recipient Identification and Request for Funding Application (leading to a2)
  • a2) Approved Request for Funding Application Package (result of a1 and leads to b1)
  • a3) Approved Claim and Supporting Documentation (leads to b2 and b3)
Activities
(Stage 2 of 7)
  • b1) Processing Agreements Requests (result of a2, leading to c1)
  • b2) Processing claims/Advances/Payments (result of a3, leading to c2)
  • b3) Monitoring Project Progress(result of a3, leading to c3)
  • b4) Planning and Reporting (leading to c6, c7 and c8)
Outputs
(Stage 3 of 7)

The Outputs section is made up of eight boxes. The eight boxes from left to right are:

  • c1) Signed Agreements (result of b1, leading to d1, d2, d3 and d4)
  • c2) Approved claims /Advances/Payments (result of b2, leading to d1, d2, d3 and d4)
  • c4) Recipient Audit Plan (result of c3, leading to e1, e2, e3, e4 and e5)
  • c3) Risk Management Strategy (result of b3, leading to c4, c5 and e1, e2, e3, e4 and e5)
  • c5) Progress/Performance Report (result of c3, leading to e1, e2, e3, e4 and e5)
  • c6) Quarterly disclosure of agreements >$25K (result of b4, leading to e1, e2, e3, e4 and e5)
  • c7) Three-Year Forecast (result of b4, leading to e1, e2, e3, e4 and e5)
  • c8) FAP Annual Report (result of b4, leading to e1, e2, e3, e4 and e5)
Project-level Immediate Outcomes
(Stage 4 of 7)

The Project-level Immediate Outcomes section is surrounded by a rectangle with a purple dashed edge. Inside of this rectangle are four boxes organized horizontally. The four boxes have the following titles from left to right:

  • d1) Scientific/Technical Knowledge Advanced/Enhanced (result of c1 and c2, leading to e1, e2, e3, e4 and e5)
  • d2) Individual Knowledge and Skills Developed/Improved (result of c1 and c2, leading to e1, e2, e3, e4 and e5)
  • d3) International Collaborations Expanded/Strengthened (result of c1 and c2, leading to e1, e2, e3, e4 and e5)
  • d4) Organizations or Initiatives Established/Sustained (result of c1 and c2, leading to e1, e2, e3, e4 and e5)
Program-level Immediate Outcomes
(Stage 5 of 7)

The Program-level Immediate Outcomes section is surrounded by a rectangle with a purple dashed edge. Inside of this rectangle are five boxes organized horizontally. The five boxes have the following titles from left to right:

  • e1) Promotion and awareness of policies, legislation and science-based regulations (result of d1, d2, d3, d4 and c3, c4, c5, c6, c7, c8, leading to f1, f2, and f3)
  • e2) Collaborations contribute to international standards for human, animal and plant related risks (result of d1, d2, d3, d4 and c3, c4, c5, c6, c7, c8, leading to f1, f2, and f3)
  • e3) Awareness of risks related to food supply, plant and animal resource base among stakeholders (result of d1, d2, d3, d4 and c3, c4, c5, c6, c7, c8, leading to f1, f2, and f3)
  • e4) Preparedness to prevent, address, and manage food, plant, and animal-related emergencies (result of d1, d2, d3, d4 and c3, c4, c5, c6, c7, c8, leading to f1, f2, and f3)
  • e5) Effective management of contributions (result of d1, d2, d3, d4 and c3, c4, c5, c6, c7, c8, leading to f1, f2, and f3)
CFIA Intermediate Outcomes
(Stage 6 of 7)

The CFIA Intermediate Outcomes section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle are three boxes organized horizontally. The three boxes have the following titles from left to right:

  • f1) Risks to Canadian and International food supply, plant and animal resource base are managed, minimized and mitigated Sustained (result of e1, e2, e3, e4 and e5, leading to g1)
  • f2) Canadian standards are recognized internationally (result of e1, e2, e3, e4 and e5, leading to g1)
  • f3) Compliance with program policies, requirements and regulations among stakeholders (result of e1, e2, e3, e4 and e5, leading to g1)
CFIA Strategic Outcome
(Stage 7 of 7)

The CFIA Strategic Outcome section is surrounded by a rectangle with a green dashed edge. The following text is written inside of this rectangle box:

  • g1) "A safe and accessible food supply and plant resource base." result of f1, f2 and f3)

The FAP logic model consists of:

There are two types of immediate outcomes that are expected to result from the inputs, activities and outputs noted above: those relating to individual FAP projects; and those relating to the FAP program overall.

CFIA Intermediate Outcomes: When completing the Request for Funding Application Package, it is the responsibility of the RCM to articulate how the proposed FAP initiative or project will contribute to the advancement of the expected results (intermediate outcomes) of their program areas. Currently these outcomes are listed in the old logic modelFootnote 11 as: i) risks to the Canadian public associated with the food supply system are mitigated; ii) risks to Canadians from the transmission of animal diseases to humans are minimized; and, iii) risks to the Canadian animal and plant resource base are minimized. These were modified by the evaluation team to align with a more recent Agency logic model (see Figure 2.2), as follows: (a) risks to Canadian and International food supply, plant and animal resource base are managed, minimized and mitigated, (b) Canadian standards are recognized internationally, and/or (c) compliance with program policies, requirements and regulations among stakeholders.

CFIA Strategic Outcome: A safe and accessible food supply, plant and animal resource base. The collective inputs, activities, outputs and outcomes of the FAP provide evidence to support the advancement of the strategic outcome.

2.6 Past Evaluation and Audit

An evaluation of the FAP was conducted in fiscal year 2010-11 covering the period from June 2006 to October 2010. An audit was also conducted during this same time frame covering the period from June 2006 to August 2010. The audit and evaluation resulted in one joint recommendation:

The President should designate a senior CFIA executive accountable and responsible for the Federal Assistance Program to establish a program entity and to ensure that the program meets all applicable governance, risk management and control expectations for a federally-funded contribution program.Footnote 12,Footnote 13

Significant measures and controls have been put in place to address this recommendation, and most are fully operationalized. These include moving the FAP from the Corporate Management Branch to the Policy and Program's Branch, the designation of a senior CFIA executive accountable and responsible for the FAP management, and the creation of numerous risk management and governance documents, including a Performance Measurement Strategy, Recipient Risk Assessment and Management Framework, and Program Guide. The only measure not fully operationalized at the time of the evaluation was the Performance Measurement Strategy. As such, it was not examined in detail. Its implementation will be examined during the next evaluation, within five years.

3. Methodology

The CFIA's Evaluation Directorate managed the evaluation and conducted it with the assistance of Hickling Arthurs Low Corporation. The evaluation was guided by an Advisory Committee which reviewed and provided feedback on the evaluation plan, findings, report and Management Response and Acton Plan. The evaluation was conducted over the period from May 2015 to January 2016.

3.1 Evaluation Scope

The evaluation covers the period from April 2011 to March 2015. All FAP activities were analyzed including all relevant areas of the Agency providing significant support to the Program.

3.2 Evaluation Methods

The following methods were applied to collect data from multiple sources:

3.2.1 Document and Data Review

The document and data review covered:

Details are provided in Appendix A.

3.2.2 Interviews

Thirty-two interviews were completed with senior CFIA management, FAP management and support (e.g., CFIA Finance and Public Affairs), responsibility centre managers (RCMs), and FAP recipients. The distribution of planned and completed interviews is shown in Table 3-1. The interviews followed an interview guide to ensure consistency of information collected and were semi-structured to allow for effective probing of the issues; different interview guides were used for each interview type and are provided in Appendix B. Interviews were conducted in the interviewees' official language of choice, in-person or by telephone.

Table 3-1: Number of Planned and Completed Interviews by Interview Type
Interview Type Planned Completed
Senior CFIA Management 6-7 8
FAP Management and Support 3-5 6
Responsibility Centre Managers (RCMs) 7-8 8
FAP Project Recipients 7-9 10
Total 23-29 32

The value of the 10 projects covered by the evaluation interviews accounts for 89 per cent of the $5.9 million disbursed by FAP over the 2011-15 study period.

3.2.3 Comparative Review

A comparative review of four similar programs was undertaken to investigate how other government departments and agencies (OGDs) manage contribution programs with no separate/distinct vote, i.e., those under $5 million per year. The review addressed evaluation questions related to relevance and performance, and provided insight into best practices for FAP management.

All contribution programs examined, including FAP, have an annual spending maximum of $4.5 million. By contrast, contribution programs with an annual minimum funding of $5 million and over require a separate federal budgetary vote. The four programs reviewed are:

The comparative review was based on findings from a review of program documentation and interviews conducted with CNSC and DHCP. The findings are included throughout section four of this report, and summarized in Appendix C.

3.3 Analysis and Integration of Data

Although the interviews were not designed or used to collect quantitative data, the interpretation of findings takes into account the extent to which certain perceptions or views were expressed or shared by the interviewees. Table 3-2 defines terms used in this report to quantify the proportion of interviews who expressed similar experiences, views and opinions.

Table 3-2: Definitions of Terms Used to Quantify Qualitative Data
Term Proportion of interviews
Findings reflect the experiences, views and opinions of:
Majority more than 75% of those interviewed
Most more than 50% but no more than 75% of those interviewed
Many more than 25% but no more than 50% of those interviewed
Some more than 10% but no more than 25% of those interviewed
Few less than 10% of those interviewed, when more than 2 people

3.4 Limitations

The evaluation challenges and limitations and the corresponding mitigation strategies are described in Table 3-3 below.

Table 3-3: Overview of evaluation challenges, limitations and mitigation strategies
Challenge and limitation Mitigation strategy
In 2014, a requirement was imposed on project managers to identify performance indicators linked to the Agency's outcomes, and to measure results. Many of the projects In the period being evaluated (2011-15), particularly those prior to 2014, do not have substantive performance data. This limitation was mitigated by the evaluation's efforts to corroborate findings from different sources; sometimes referred to as triangulation. As well, the evaluation endeavoured to identify performance indicators for individual project outcomes that would be applicable to future projects and for the Program. Draft performance indicators for FAP are attached as Appendix D.
Most interviewees have a vested interest in the program. This limitation was mitigated by requiring interviewees to explain their perspectives and provide examples where appropriate. The findings from the interviews were triangulated with findings from other data sources (document review, and where possible, program performance data).
As LAC's Documentary Heritage Community Program (DHCP) was only launched in September 2015, the profile of DHCP is only based on document review. The documentation on LAC's DHCP is extensive and detailed, allowing the evaluation to develop a fairly complete profile. This does not represent a challenge to the comparative review since the three other profiles each have several years' delivery experience and have provided best practices and lessons learned relevant for FAP.

3.5 Evaluation Issues, Questions and Methods

A matrix showing the evaluation issues, questions and methods used is shown in table 3-4.

Table 3-4: Matrix of Evaluation Issues, Questions and Methods
Evaluation Issues and Questions Methods
Doc/Data Review Interviews Comparative
Relevance: Continued Need for FAP and Contribution Payments
1. What is the rationale for funding FAP projects? Check Check Check
2. Is there a continued need for FAP? Check Check
Relevance: Alignment with government priorities
3. Are FAP project objectives consistent with government-wide priorities and the CFIA's mandate? Check Check
Relevance: Alignment with federal government roles and responsibilities
4. Are the roles and responsibilities of the CFIA clear and understood by internal and external stakeholders in regards to the management and delivery of FAP? Check Check
Performance: Achievement of expected outcomes
5. To what extent have FAP projects produced the intended outcomes? Check Check Check
6. Have there been any unintended outcomes resulting from FAP projects? Check Check Check
Performance: Demonstration of efficiency and economy
7. Are FAP projects cost-effective as currently delivered? Alternatives? Check Check Check
8. Is there duplication or gaps with other federal or provincial government programs and/or activities? Check Check Check
9. Is the funding of FAP projects an efficient use of the Agency's resources? If not, how could these resources be better utilized? Check Check Check
10. What performance monitoring has been done and are the tools and systems adequate to provide meaningful feedback? Check Check Check

4. Findings

The evaluation findings are presented by evaluation question, as presented in table 3-4 above.

4.1 Relevance: Continued need for the program

4.1.1 What is the rationale for funding FAP projects?

The rationale for funding FAP projects is to access expertise outside the CFIA, build capacity, support international collaborations and establish organizations or initiatives.

As part of the process to update the FAP logic model, the evaluation examined: (a) the linkage between the rationale for funding FAP projects and the expected results as indicated by the contribution agreements and key informant interviews (addressed below), and (b) whether the actual results achieved contributed to the Agency's strategic and intermediate outcomes (addressed in section 4.4).

The rationale and the expected results of the 18 FAP projects are shown in Table 4-1. The evidence for the rationale was obtained through a review of "objectives" and "expected results" sections of the contribution agreements. This was corroborated through interviews with RCMs and project recipients, and by a review of project reports.

Table 4-1: Rationale for Funding and Expected Results of FAP Projects from 2011-12 to 2014-15
Rationale for Funding FAP Projects Expected Results Number (%) of
Projects
Access expertise outside the CFIA, build research partnerships with these experts and conduct research on areas of mutual interest Scientific and technical knowledge is advanced and/or enhanced 9 (50%)
Build capacity related to the Agency's mandate, train and enhance skills of stakeholders outside CFIA Individual knowledge and skills are developed and/or improved 6 (33%)
Support international collaborations and agreements International collaborations are expanded and/or strengthened 6 (33%)
Establish and/or sustain organizations or initiatives Organizations or initiatives are established or sustained 4 (22%)

Note: Some projects had more than one reason for funding; e.g., support was provided to the World Organisation for Animal Health to access outside expertise and to support international collaborations.

Interviewees were asked to explain the process by which program managers identify FAP projects. CFIA senior managers and RCMs said that 17 of the 18 projects were identified by CFIA officials.Footnote 14

The four OGDs reviewed have similar rationales for funding projects. As noted in Table 4-2, building capacity is a rationale for funding by all four programs in the comparative review, accessing outside expertise and supporting international collaborations by two programs, and establishing and/or sustaining organizations or initiatives by one program.

Table 4-2: Funding Rationale of Four Contribution Programs without a Separate Vote in other Departments or Agencies
Department/Agency Program
(without a separate vote)
Access expertise, build research partnerships Build capacity, train and enhance skills Support international collaborations Establish and/or sustain organizations or initiatives
Canadian Nuclear Safety Commission Research Support Program (RSP) Check Check Check
Canadian Nuclear Safety Commission Participant Funding Program (PFP) Check
Library and Archives Canada Documentary Heritage Community Program (DHCP) Check Check
Parks Canada General Class Contribution Program (GCCP) Check Check Check

Source: Appendix C

4.1.2 Is there a continued need for FAP?

There is a consistent need for FAP as it is a mechanism that allows the CFIA to extend its reach, address risks and leverage knowledge and resources not otherwise obtainable.

There is a consistent need to leverage expertise outside the CFIA according to most senior CFIA managers and RCMs interviewed for the evaluation. These interviewees also noted that FAP, as the CFIA's sole contribution program, is sometimes the only vehicle to provide funding to address these needs. The section of the contribution agreements entitled "how the results will contribute to the CFIA's mandate" supports interviewee responses as this is where links are made between project goals and the CFIA's strategic outcome.

Some FAP projects are aligned with risks identified in the CFIA's Corporate Risk Profile (CRP). For example, Emergency Management and Inspection Effectiveness are both identified as risks under the CRP, which are addressed by the FAP projects Canadian Veterinary Reserve (CVR) and Safe Food Canada (SFC).

A majority of project recipients interviewed (8) indicated that without FAP funding, the scope of their project would have been smaller and implementation would most likely have been delayed. Two project recipients indicated that their project would not have proceeded in the absence of FAP funding.

There appears to be limited awareness of FAP within the Agency. Although a question on awareness of FAP was not asked, many CFIA interviewees (senior managers, RCMs, FAP management) cited a lack of awareness as a weakness of FAP. RCMs noted that FAP is not well known within the Agency and suggested that more could be done to promote the Program. Supporting this point is the lack of full utilization of the Program. Actual expenditures over the last four years have only averaged around $1.5 million, well below the $4.5 million limit per annum.

4.2 Relevance: Alignment with federal government priorities

4.2.1 Are FAP project objectives consistent with government-wide priorities and the CFIA's mandate?

All FAP funded project objectives were found to be consistent with CFIA's mandate and government-wide priorities.

All FAP contribution agreements are required to outline the project's objectives and how the intended results will contribute to CFIA's strategic outcome. As noted in Table 4-2, the intended results can be grouped into four areas: (1) advancing / enhancing scientific and technical knowledge; (2) developing / improving individual knowledge and skills; (3) expanding / strengthening international collaborations; and (4) establishing / sustaining organizations or initiatives. These results are clearly aligned with the Agency's strategic outcome of "a safe and accessible food supply, plant and animal resource base", which supports the CFIA's and federal government's priority for food safety, as identified in the following:

While the FAP program is designed to cover any and all projects, a few interviewees wondered if FAP could be utilized for a strategic issue or priority if need be. This utilization could work as long as it is not the only focus for funding in any particular year. This would allow FAP to continue to be used for any and all projects and not just those priority issues identified.

4.3 Relevance: Alignment with federal government roles and responsibilities

4.3.1 Alignment with Federal Government Guidelines for Contribution Agreements

The Treasury Board of Canada Secretariat's (TBS's) Policy on Transfer Payments indicates that a procurement contract is used to obtain goods or services, whereas a transfer arrangement (e.g., contribution agreement) is used to transfer monies or make in-kind contributions from the federal government to individuals, organizations or other levels of government to further government policy and the department's objectives. A key aspect of transfer payments is that they do "not result in the acquisition by the Government of Canada of any goods, services or assets."Footnote 19

The evaluation found, based on applying the TBS "checklist tool"Footnote 20 for all 18 FAP projects, the transfer payment arrangement to be more appropriate than procurement contracts. For example, the CFIA did not directly acquire a good or service from any of the 18 FAP projects, and all projects advanced the Agency's strategic outcome (as shown below in Section 4.4).

The TBS identifies five forms of transfer payments: contributions, grants, other transfer payments (OTP), alternative funding arrangements (AFAs), and flexible transfer payments (FTPs). Of the five forms, contributions are most appropriate for programs like FAP. A contribution is defined as a conditional transfer whereby specific terms and conditions must be met or carried out by a recipient before costs are reimbursed, whereas a grant is an unconditional transfer payment where eligibility criteria and applications received in advance of payment sufficiently assure that the payment objectives will be met. OTPs are "based on legislation or an arrangement that normally includes a formula or schedule as one element used to determine the expenditure amount." Examples of OTPs are transfers to other levels of government such as Established Program Financing and transfers to the territorial governments. AFAs and FTPs are transfers specific to Aboriginal Affairs and Northern Development Canada.Footnote 21

An advantage of a contribution (which requires funding recipients to report on results) over a grant (which does not) is the higher level of accountability. The evaluation found all 18 FAP projects compliant with the Terms and Conditions for contributions as outlined in TBS's Directive on Transfer Payments.Footnote 22

4.3.2 Are the roles and responsibilities of the CFIA clear and understood by internal and external stakeholders in regards to the management and delivery of FAP?

A clear understanding was evident by both internal and external stakeholders on the roles and responsibilities of the CFIA in regards to the management and delivery of FAP. This was, for example, indicated by all senior CFIA management and RCM interviewees.

FAP management has developed many templates and guidelines to help guide the management and delivery of FAP, as listed in Appendix A.

FAP management also provides hands on support to the RCMs throughout the entire project lifecycle from project development, approval, monitoring, and reporting. The majority of senior CFIA managers and RCMs indicated that FAP templates, guidelines and hands on support are useful for managing and delivering FAP projects. Most internal interviewees also identified the existence of a dedicated FAP CC as a strength of the program.

A few internal interviewees, however, felt guidelines on when to use a contract or a contribution agreement would be helpful, as the difference between the two is not easily understood. Closely related to this issue is the definition of direct vs. indirect benefits of project activities to the Agency vs. the greater good; as a contract is used for direct benefits and a contribution for indirect benefits.

External interviewees have a similar perspective with respect to the management and delivery of FAP projects. The majority of FAP recipients indicated that support from the RCMs and FAP management was good.

4.4 Performance: Achievement of expected outcomes

4.4.1 To what extent have FAP projects produced the intended outcomes?

Of the 13 completed FAP projects, over the four-year period from 2011-12 to 2014-15, all have produced some if not all of their intended immediate outcomes.

Results achieved by projects are in line with those defined in their respective agreements. All 13 completed FAP projects submitted a final report. Although there was a significant variance in recipient report evidence and detail, they all clearly showed achievement of intended outputs as well as evidence of achievement of intended immediate outcomes. In addition, the majority of interviewees, both RCMs and project recipients, spoke to the successful achievement of intended outcomes. The outcomes from each project is shown below in Table 4-3 listed according to the four key areas, as per the FAP logic model.

Five FAP projects are still ongoing (GREZOSP, CRVE-NET, U of Guelph, Safe Food Canada, and Genome Alberta) and for these, final outcomes are not yet determined but project activities are progressing well.

Table 4-3: Immediate Outcomes of FAP Projects from 2011-12 to 2014-15
Recipient Project Status Scientific / Technical Knowledge Advanced / Enhanced Individual Knowledge and Skills Developed / Improved International Collaborations Expanded / Strengthened Organizations or Initiatives Established / Sustained
Canadian Cattle Identification Agency (CCIA) Traceability National Information Portal – CCIA Requirements Completed project Check
Canadian Veterinary Medical Association Canadian Veterinary Reserve (CVR) Completed project Check Check
Conference Board of Canada Third Report of the Food Safety Performance World Ranking Study Completed project Check
Conseil québecois des espèces exotiques envahissantes Emeral Ash Borer Management Workshop – Quebec Completed project Check
Genome Alberta The 2014 Program on Research and Innovation Leading to Rapid Genomics Response to the Porcine Epidemic Diarrhea virus (PEDv) Ongoing project Check
International Plant Protection Convention (IPPC) Special International Plant Protection Convention Trust Fund Completed project Check
Safe Food Canada – The Learning Partnership Preparing to implement Safe Food Canada (SFC) – The Learning Partnership Ongoing project Check Check
University of Guelph Research support to the Multi-Criteria Risk Prioritization Framework for Food Safety Hazards in Canada Ongoing project Check
University of Montreal Ateliers d'Initiation au Leadership Vétérinaire (ILV)/Veterinarian Leadership Workshops Completed project Check
University of Montreal Groupe de recherche en épidémiologie des zones et santé publique (GREZOSP)/Epidemiology and Public Health Research group Ongoing project Check
UPEI – Atlantic Veterinary College Canadian Regulatory Veterinary Epidemiology Network (CRVE-NET) Ongoing project Check Check
Veterinarians without Borders – Canada Strengthening Leadership and Capacity for Securing Animal and Public Health in Canada and the International Community Completed project Check Check
Veterinarians without Borders – Canada Veterinarians without Borders – Canada Completed project Check
Williams and Associates Forestry Consultants Ltd. Emeral Ash Borer Management Workshop – Ontario Completed project Check
World Bank Global Food Safety Partnership Completed project Check Check
World Health Organization – CODEX World Health Organization (WHO) Project and Fund for Enhanced Participation in Codex Alimentarius Commission Completed project Check
World Organisation for Animal Health Extraordinary Contribution to the World Organisation for Animal Health Completed project Check Check
World Organisation for Animal Health OIE Global Conference on Aquatic Animal Health: 'Riding the wave to the future' Completed project Check Check

Some examples of project outcomes produced in the four key areas are as follows:

  • Organizations and Initiatives established/sustained: FAP support instrumental in founding and launching the Global Food Safety Partnership (World Bank); Veterinarians without Borders Canada, Canadian Veterinarian Reserve (CVR); and most recently Safe Food Canada.

Safe Food Canada – The Learning Partnership ($850,000 over 2 years 2014-16) is developing consistent, standardized training in food safety, that applies to both industry and regulators, and is expected to be a key support for CFIA's future training needs for all inspectors.

4.4.2 Have there been any unintended outcomes resulting from FAP projects?

Most recipients and RCMs noted that FAP projects resulted in other benefits and broader outcomes for both the CFIA and the recipient community, as stated in final reports submitted by recipients. In all cases, these unintended outcomes were positive, although varied and broad reaching, with many having longer term impacts. They ranged from closer, more appreciative relationships between CFIA veterinarians and those in private practice, thus affecting future collaborative efforts, to improvements in Canada's visibility and reputation in food safety at an international level. Listed below are further examples of these unintended positive outcomes, according to RCMs and recipients.

For CFIA these include:

For recipients these included:

For both CFIA and recipients these included:

4.5 Performance: Demonstration of efficiency and economy

4.5.1 Are FAP projects cost-effective as currently delivered? Alternatives?

The majority of projects appear to be cost effective as currently delivered. The program is delivered with minimal staff resources; the FAP CC, who resides in the Policy and Programs Branch (PPB), is the only full-time resource and spends a self-estimated 90 per cent of her time administering FAP. A few junior staff within PPB, along with CFIA Public Affairs and Finance, support the FAP CC on an as needed basis.

Significant effort has been exerted in the last two years to frame program requirements and put guidance and controls in place. Standardized templates and clauses (e.g., official languages clause, template for further distribution of funds, proactive disclosure procedure, summary information template, etc.) are now in place with frequent improvements introduced. Most recently, a standardized recipient reporting requirements template was developed to accompany all contribution agreements.

Individual branches also dedicate time to manage their own FAP projects and liaise with recipients, via the designated RCM. RCM time invested ranges from an estimated two per cent to 10 per centFootnote 23 over the course of a FAP project. RCMs typically invest most of their time at the front-end of a project; however, depending on the schedule/nature of a project, RCM allotted time to a given project can fluctuate. One notable exception is Safe Food Canada which required upwards of two to three days/week RCM time over 18 months before it was launched. Several other RCMs noted significant upfront investment due to: (i) complexity of international organization requirements (e.g., World Health Organization, World Bank); and, (ii) change in funding mechanism and subsequent change in nature of recipient/CFIA relationship (Canadian Veterinary Reserve).

Most recipients had no particular issue with the application and agreement process, while some noted it to be slow and complained of the paperwork. There was similar feedback on FAP reporting requirements which many project recipients found to be burdensome and time consuming, thus adding cost to managing the project. However, most recipients also noted they understood the need for transparency and appropriate oversight. Some recipients felt this still could be maintained with reduced reporting requirements.

Suggestions for improving cost-effectiveness made by some RCMs and senior management include: (i) consolidating FAP program information and up-to-date guidance documents on-line (this has since been initiated by the FAP CC); (ii) reducing the administrative burden/control for smaller and more simple projects; and, (iii) continuing to leverage additional/matching fundsFootnote 24 to extend CFIA's reach and create synergies.

Alternatives to the use of contribution agreements could include contracts, however, these involve a competitive process and may entail more administrative burden than the current contributions.Footnote 25 Collaborative research agreements could be considered for research projects where both partners self-fund. Collaborative research agreements can enhance scientific capabilities through the exchange of knowledge, expertise and experience as well as provide access to scientific facilities and other sources of funding. This is particularly useful where stakeholders have similar research interests and common publication profiles. However, issues of shared intellectual property may arise and affect outcomes.

4.5.2 Is there duplication or gaps with other federal or provincial government programs and/or activities?

FAP is viewed as unique, understood to be the Agency's only contribution program, and was not seen to overlap or duplicate with other funding mechanisms/programs or activities. There were no specific incidences of duplication or overlap found, although given the broad range of projects that can be funded, the potential does exist for this to occur. There has been some co-funding of projects, which has in some cases used MoUs.

The majority of recipients also noted FAP does not duplicate any other program (federal or provincial) that they were aware of. However, in general, there is an inherent potential bias of funding recipients to avoid identifying overlap or duplication, which could then restrict future program usage. However, there was no specific evidence of such bias identified by the evaluation. Some recipients indicated that FAP was a good complement to other funding they had received (e.g., Ontario Ministry of Agriculture Food and Rural Affairs, AAFC Growing Forward) and thus allowed them to further leverage project activities.

4.5.3 Is the funding of FAP projects an efficient use of the Agency's resources? If not, how could these resources be better utilized?

For most projects, and the Program in general, funding expended through the FAP was found to be an efficient use of Agency resources.

Overall, the value of FAP is in its flexibility as a funding vehicle, its ability to leverage funds (co-funding and in-kind contribution come from industry, academia and provinces) and its use as a tool to build collaboration partnership, meaningful outreach/exposure and capacity beyond CFIA. The outcomes affect more than CFIA and demonstrate efficiency due to the synergies that are created. The majority of both recipients and RCMs would participate in another FAP project.

Two issues were, however, raised:

Veterinarians without Borders (VWB) had repeat, multi-year funding agreements with FAP since 2006. Initially, this helped establish the Canadian chapter of the organization, which was then, over time, able to obtain other sources of funding, reducing its FAP support from $500,000 in 2010 to $250,000 in 2015.

Other repeat, multi-year agreements include those with the University of Prince Edward Island Atlantic Veterinary College for the Canadian Regulatory Veterinary Epidemiology Network, which received funding from 2011 to 2015, starting at $250,000 and decreasing to $150,000.

Finally, the World Organization for Animal Health (OIE), received $500,000 for a consecutive four years from 2010 to 2015.

Table 4-4 displays the annual FAP contributions amounts for the above examples.

There are currently no particular rules, policies or guidelines for repeat and multi-year funding. As with all program intended outcomes, indicators for the establishment or sustaining of organizations have been included for program performance monitoring. See Appendix D indicator #11.

Table 4-4: Multi-Year projects from 2010-11 to 2014-15
Organization Project 2010-10 2011-11 2012-13 2013-14 2014-15 Total
Veterinarians without Borders – Canada (VWB) Support to nurture and expand programs; to increase Canadian expertise in dealing with global health challenges; to leverage resources and partnerships. $ 500,000 $ 450,000 $ 400,000 $ 1,350,000
Support to nurture and expand programs; to increase Canadian expertise in dealing with global health challenges; to leverage resources and partnerships. $ 350,000 $ 350,000
Strengthening Leadership and Capacity for Securing Animal and Public Health in Canada and the International Community. $ 250,000 $ 250,000
Sub-total $ 1,950,000
World Organisation for Animal Health (OIE) To support OIE in determining international science based guidelines that govern safe and competitive trade and to provide Canadian expertise to support OIE's Headquarters and regional activities. $ 500,000 $ 500,000 $ 500,000 $ 500,000 $ 2,000,000
Sub-total $ 2,000,000
University of Prince Edward Island (UPEI) Atlantic Veterinary College Canadian Regulatory Veterinary Epidemiology (CREV-Net) $ 250,000 $ 250,000
Canadian Regulatory Veterinary Epidemiology (CREV-Net) $ 250,000 $ 250,000
Canadian Regulatory Veterinary Epidemiology (CREV-Net) $ 150,000 $ 150,000 $ 300,000
Sub-total $ 800,000

Some CFIA interviewees noted that more clearly defined project activities and improved performance indicators would be helpful in demonstrating the efficient use of Agency resources (i.e., value for money) when funding recipients report on their progress.

4.5.4 What performance monitoring has been done and are the tools and systems adequate to provide meaningful feedback?

FAP performance monitoring is guided by the FAP Performance Measurement Strategy which sets out the information, tools and targets required for ongoing monitoring and reporting. Current project requirements include both recipient progress and final reports. These reports are managed and reviewed by individual RCMs. There was no standardized template for reporting for the study period, though this was being addressed at the time of this report's writing. Reports are therefore inconsistent in level of detail, content and format.Footnote 27 Even though standardized reporting is lacking, most reports include meaningful feedback related to activities completed and results achieved. In addition, there is no systematic critical review of these reports.

Another key oversight tool is program recipient audits. Audits are completed by an external auditor and selected based on an assessment of risk and materiality; this activity is coordinated and managed by the FAP Coordinator. For the period of this evaluation, two recipient audits were completed, for Veterinarians without Borders and Canadian Veterinarian Medical Association. No issues of concern were identified.

The FAP also has a three-year Program funding forecast (currently available for the period 2015 to 2018), a quarterly disclosure process where agreements over $25,000 are posted on the CFIA's external website, and risk assessmentsFootnote 28 conducted by RCMs for contributions agreements over $25,000.

The use of performance indicators has been initiated, as of 2014, with 10 of the most recent contribution agreements containing them. Prior to this, performance was measured against expected results and project objectives as stated in the contribution agreement. A review of these performance indicators found them to be reasonable for measuring the success of the overall project; however, many still require refinement to be measureable.Footnote 29 It was suggested by some RCMs and FAP management that example performance indicators (i.e., a menu/list of options), that RCMs and recipients could draw on and consider when preparing contribution agreements, would be helpful in creating a more standardized and rigorous approach to performance measurement (as noted above in Section 2.5 – indicators and performance measures have been drafted by the evaluation team for future program validation).

Senior management noted they would benefit from a more global overview of FAP project results and how they contribute to the CFIA's strategic outcome. There is value in producing an annual report to address this interest, to capture a broader understanding of both achievements and impacts as a result of the FAP.

5. Conclusions and Recommendations

The 2010 evaluation and audit of the Federal Assistance Program concluded that the Program required better governance, risk management and controls. It was clearly evident in this evaluation that these management tools and structures have now been effectively put in place and are operating as planned, with minor exceptions. For example, there are components, such as performance measurement, that have only been developed in the past year and are expected to be implemented shortly.

The TB evaluation Directive outlines issues related to relevance and performance that evaluations are to assess. The relevance of FAP is supported by the clearly stipulated requirement that its projects support the Agency's strategic outcome. The evaluation found the projects to be so aligned. The need for the Program was also clearly demonstrated, partly because it is the only contribution program the Agency has access to, but also evident from the support for the program and interest in future usage by both CFIA management and funding recipients. Increasing the opportunity to take advantage of the Program is the intention of the first recommendation:

The performance of the Program is evident in the success of its projects, though performance measures have been inconsistent and no roll up of outcome reporting has been attempted. This led to the second recommendation:

Governance, reporting and administrative requirements were found to be burdensome for smaller and less complex projects. The third recommendation is made to support projects more efficiently:

Concern was expressed over long term funding through multi-year and repeat agreements, for which the CFIA currently has no guidelines.

In conclusion, the Program is relevant and effective, with only minor administrative adjustments recommended.

Appendices

A. References

A.1 Core FAP Management Documents

  1. Canadian Food Inspection Agency (no date). Federal Assistance Program Performance Measurement Strategy.
  2. Canadian Food Inspection Agency (no date), Federal Assistance Program Recipient Risk Assessment and Management Framework.
  3. Canadian Food Inspection Agency, (no date) Federal Assistance Program Guide.
  4. Canadian Food Inspection Agency (Amended October 1, 2014). Federal Assistance Program Terms and Conditions.
  5. Canadian Food Inspection Agency (November 27, 2014). CFIA Federal Assistance Program – an Overview, Vivienne Atkinson, Program, Regulatory and Trade Policy Directorate (slide deck 1 of 3).
  6. Canadian Food Inspection Agency (no date). FAP Proactive Disclosure Procedure; Summary Information Template.
  7. Canadian Food Inspection Agency (no date). FAP Transmittal Slip Templates
  8. Canadian Food Inspection Agency (no date). FAP Domestic Contribution Agreement template (see page 10, Section 37 for official languages clause)
  9. Canadian Food Inspection Agency (no date). FAP Contribution Agreement template for Further Distribution of Funds (detailed information regarding Ultimate Recipients, see page 21, Appendix C: Selection Framework and Contribution Agreement Requirements)
  10. Canadian Food Inspection Agency (no date). FAP Recipient Reporting Requirements; Appendix C or D of Contribution Agreement.

A.2 Project Databases

  1. FAP Project Database 2011-12, 2012-13, 2013-14, 2014-15
  2. FAP Project Database Table of Recipients and Values of contributions from 2010-11 to 2014-15
  3. FAP Project Database FAP Three-year Forecast 2015-16 to 2017-18
  4. Project contribution agreements, claims, progress and final reports, and audit reports

B. Interview Guides

B.1 Background (common to all guides)

Thank you for agreeing to be interviewed as part of the Evaluation of the Canadian Food Inspection Agency's Federal Assistance Program (FAP). CFIA's Evaluation Directorate has engaged a team of outside evaluation specialists at Hickling Arthurs Low Corporation to assist it with the conduct of the evaluation.

The FAP is currently the CFIA's sole contribution program and is used to support projects and initiatives that advance the Agency's strategic outcome; namely a safe and accessible food supply, plant and animal resource base. The FAP is funded annually from individual branch budgets. As such, FAP is a tool CFIA program managers utilize to broaden their reach by supporting third-party projects that are intended to contribute to:

The objectives of the evaluation are to assess the relevance and performance of the program in accordance with the Treasury Board Policy on Evaluation.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Once approved, the final evaluation report will be made public by the CFIA in accordance with Treasury Board policy.

Your interview is expected to take less than 45 minutes. If you have any questions about this study, please do not hesitate to contact the Evaluation Manager.

B.2 Interview Guide: Senior CFIA Management

Questions

Relevance

Performance

B.3 Interview Guide: FAP Managment

Questions

Relevance

Performance

B.4 Interview Guide: Responsibility Centre Managers

Questions

Relevance

Performance

B.5 Interview Guide: Project Recipients

Questions

B.6 Interview Guide: Other Departments and Agencies

Questions

B.7 Interview Guide: Legal Services

Questions

B.8 Interview Guide: Public Affairs

Questions

C. Summary of OGD Programs without a Separate Vote

C.1 Overview

The four other government department and agency (OGD) programs without a separate vote included in the comparative review are:

It is noted that the contribution programs examined, including FAP, have an annual spending maximum of $4.5M. By contrast, contribution programs with an annual minimum funding of $5M and over require a separate federal budgetary vote.

A summary overview of the findings from this comparison is provided below in Figure C-1, followed by summary profiles of each program. As noted in the figure, FAP is designed very much like OGD programs without a separate vote.

Figure C-1: Summary Overview of Four OGD Programs without a Separate Vote and FAP
Department / Agency Program Contribution Limits Program Budget Avg. Annual Expenditures Multi-Year? Limit on Number of Agreements Awarded to One Recipient? Scaling on Project Administrative Requirements? Sign-off Level Horizontal Senior Oversight? Aspects of Mandate Addressed? Advertised or Competitive? Assess Value for Money? Performance Measurement Reporting Third Party Delivery?
Canadian Nuclear Safety Commission Research Support Program (RSP) $550K per annum $4.5M per annum $1.5M actual per annum Yes No No President Yes, DG-level Research Advisory Committee All Non-competitive: Identified by CNSC staff similar to FAP Every 5 years, qualitative assessment only Each major recipient is required to have a LM and PM Strategy Progress and final reports on project outcomes required by contribution recipients No reporting required for grant recipients Annual Program Performance Report No
Canadian Nuclear Safety Commission Participant Funding Program (PFP) $250K per annum $925K for contributions (plus $175K for administration for a total of $1.1M per annum) $300K actual pa for contributions Yes No No VP Regulatory Affairs Outside Funding Review Committee Some Advertised Every 5 years, qualitative assessment only Standard set of indicators used on project outcomes and efficiency Recipients report on outcomes for their project Progress and final reports required N/A
Library and Archives Canada Documentary Heritage Community Program (DHCP) Small: <$15K per annum; Large: $15K to $100K per annum $1.5M budget pa N/A Yes for large projects $15K-$100K; No for small projects <$15K Doesn't appear to be Interim assessment required for multi-year projects Librarian and Archivist of Canada Internal Subject Matter Experts and External Advisory Committee Most Advertised and Competitive N/A, new program Indicators based on, and linked to, project outcomes and program objectives Developed standardized format for reporting – "Final Assessment and Financial Report" template Final assessement and financial report required No, for small projects; Yes, for large projects
Parks Canada General Class Contribution Program (GCCP) $1M per annum $4.5M per annum $4.3M actual pa Yes No No PC CEO No All Non-competitive: identified by PC staff (mostly field, some national) similar to FAP Every 5 years, qualitative assessment only Indicators based on project objectives and budget. Variety of receipients and activities make it difficult to use standard set of indicators Do not have standard reporting template as projects vary significantly, but do link project results to department's strategic outcomes Interim and final reports required. Frequency of reporting based on departmental risk assessment Yes, with supplementary clauses in contribution agreement
Canadian Food Inspection Agency Federal Assistance Program (FAP) $2M per annum $4.5M per annum $1.5 M actual pa Yes No No President No All Non-competitive: identified by CFIA staff Every 5 years, qualitative assessment only Use of performance indicators initiated in 2013 Standardized reporting planned for Progress and final reports on project outcomes required by contribution recipients Yes, with supplementary clauses in contribution agreement

C.2 CNSC Research Support Program (RSP)

The Canadian Nuclear Safety Commission's (CNSC) Class Grants and Contribution Program was implemented in 1984 to provide funding to third parties to conduct research and development (R&D) and to provide regulatory support. The CNSC's Class Grants and Contribution Program consist of two parts:

The purpose of the CNSC's Class Grants and Contribution Program is to provide access to advice, expertise, experience, and information that would otherwise not be available through agreements with private sector and other agencies and organizations, individuals and not-for-profit organizations in Canada and elsewhere.

C.2.1 Objectives/Purpose

The purpose of the Research Support Program (RSP) is to provide the CNSC with scientific, technical and other advice and information it needs to support the Commission's role and responsibility as outlined in the Nuclear Safety and Control Act (NSCA)Footnote 31.

The objectives of the RSP are to enable research, development and the management of activities in order to reduce uncertainties regarding health, safety, security and environmental issues.Footnote 32

The specific objectives of the RSP are to:

C.2.2 Comparison to FAP

Similarities to FAP

CFIA's FAP and CNSC's RSP are very similar. Both programs:

Differences from FAP

Program aspects the RSP has that are different from the FAP:

C.2.3 Summary

The RSP and the FAP are very similar. Both programs support regulatory research, support activities to minimize risk, involve domestic and international partners, identify projects through staff networks, have average actual expenditures of $1.5 million per annum, allow multi-year funding, allow repeat agreements to the same recipient, and have a dedicated program coordinator.

C.3 CNSC Participant Funding Program (PFP)

C.3.1 Objectives/Purpose

The CNSC's established the Participant Funding Program (PFP) under the Terms and Conditions of its Class Grants and Contribution Program to improve the regulatory review process by enhancing Aboriginal, public, and stakeholder participation in environmental assessments and public Commission proceedings, in order to help bring valuable information to the Commission.

The purpose of the PFP is toFootnote 34:

PFP's objectivesFootnote 35 are to:

C.3.2 Comparison to FAP

Similarities to FAP

Both FAP and PFP:

Differences from FAP

Program aspects the PFP has that are different from the FAP:

C.3.3 Summary

The main similarity is that both programs support activities that promote effective regulatory frameworks which in turn help to maintain public confidence.

The main difference is that the PFP's budget is cost recovered from CNSC's licensees, which includes an annual operating budget for the CNSC to coordinate and administer the program. Senior level oversight on final funding decisions for applications is made by CNSC senior management and is based on recommendations provided by a Funding Review Committee external and independent from the CNSC. PFP opportunities are also advertised and recipients are selected through a competitive review process based on eligibility, relevance and the potential value to the Commission's decision making process.

C.4 LAC Documentary Heritage Communities Program (DHCP)

Library and Archives Canada (LAC) launched the Documentary Heritage Communities Program (DHCP) in 2015/16 and will invest $1.5 million per year over five (5) years to implement the initiative for a total $7.5 million investment.

C.4.1 Objectives/Purpose

The purpose of the DHCP is to ensure that Canada's continuing memory is documented and accessible to current and future generations by adopting a more collaborative approach with local documentary heritage communities. The program will be delivered in the form of contributions that will support the development of Canada's local archival and library communities by increasing their capacity to preserve, provide access to and promote local documentary heritage. Additionally, the Program will provide opportunities for local documentary heritage communities to evolve and remain sustainable and strategic.

The DHCP provides financial assistance to the Canadian documentary heritage community for activities that fulfill the following objectives and related activities.

Objective 1: Increase access to, and awareness of Canada's local documentary institutions and their holdings. Eligible activities include:

Objective 2: Increase the capacity of local documentary heritage institutions to better sustain and preserve Canada's documentary heritage. Eligible activities include:

C.4.2 Comparison to FAP

Similarities to FAP

Similarities between FAP and DHCP:

Differences from FAP

The differences between the two programs include:

C.4.3 Summary

Many key aspects of the DHCP management are similar to FAP, however, the main difference is that DHCP is externally focused with a competitive process in place for project selection and a defined program budget and duration ($7.5 million over 5 years).

C.5 Parks Canada General Class Contribution Program (GCCP)

The Parks Canada Agency's (PCA) General Class Contributions Program (GCCP) is one of three contribution programs within the Agency. The GCCP is not a "program" in the traditional sense but rather functions as a general funding authority available to Parks Canada managers to support a wide range of recipient projects. It was originally launched in 2006-07 and since that time has sponsored a wide variety of both domestic and international projects aligned with the Agency's mandate.

C.5.1 Objectives/Purpose

The objectives of the GCCP are to assist recipients in conducting activities and delivering projects that will support Parks Canada in fulfilling its mandate to:

Eligible projects and activities can include:

As such, the expected purpose and outcomes of the funding are:

C.5.2 Comparison to FAP

Similarities to FAP

Both FAP and GCCP:

Differences from FAP

Program aspects the GCCP has that are different from the FAP:

C.5.3 Summary

GCCP and FAP are similar in numerous ways. Both programs do not advertise externally and have the flexibility to select and define projects to fund the Agency's most urgent priorities. Further, both FAP and GCCP are viewed more as funding mechanism as opposed to a traditional government program with centralized budgets and program offices. As with FAP management, Parks Canada also struggles with the definition of direct vs indirect benefit of project activities to the Agency vs. the greater good and subsequently, the applicability of a contract vs. a contribution agreement. Another challenge that faces both FAP and GCCP is the somewhat cumbersome and lengthy application process, particularly for contributions with smaller dollar value and lower risk.

D. Draft Performance Measurement Indicators

The following is a menu/list of options of performance indicators and is designed for RCMs and recipients to consider and use when monitoring and reporting on project progress/success. This list is by no means exhaustive. It should be understood that particular projects may require additional indicator development on the part of recipients and RCMs.

The indicators are linked to the outputs and outcomes as shown in the FAP logic model.

The intermediate and strategic outcome indicators are primarily for the FAP management team to use as a tool when monitoring and reporting on the FAP. However, they should still be considered by recipients and RCMs.

Where applicable, include planned versus actual.

D.1 Outputs

D.2 Project-level Immediate Outcomes

Outcome Specific Indicators General Indicators
  • 8) Scientific / Technical Knowledge Advanced / Enhanced
    • a) List/number of publications, presentations, and if possible, whether peer reviewed
    • b) List/number of citations, awards (long term)
    • c) Guide/manual developed for applying/utilizing the new knowledge
    • d) Risk profiles/frameworks developed/enhanced
  • a) Reports on development /knowledge completed/received
  • b) Description of knowledge acquired/disseminated
  • c) Workshop delivered as planned
  • d) Recommendations developed from/ratified at workshops
  • e) New procedures/plans/frameworks/technology ratified/finalized
  • f) Participant satisfactory with knowledge gained
  • g) Recruitment and engagement obtained planned/achieved
  • 9) Individual Knowledge and Skills Developed / Improved
    • a) Development of new curriculum/enhancements to existing curriculum
    • b) Number of individuals trained/workshops, planned vs. target
  • 10) International Collaborations Expanded / Strengthened
    • a) Number of collaborations continued / established / broadened
    • b) Number of person-days by Canadian and CFIA stakeholders involved in the international collaboration activity(s)
    • c) List/number of Canadian, CFIA stakeholders, and international bodies participating in the collaborations
    • d) Number of meetings/consultations, workshops, committees, coalitions etc.
    • e) Sharing of methods/procedure/ideas regarding science-based regulations and regulatory frameworks
    • f) Number of documents, methods/procedures regarding science-based regulations and regulatory frameworks shared with international fora/organizations

D.3 Program-level Immediate Outcomes

D.4 CFIA-level Intermediate Outcomes

Date modified: