Archived - Evaluation of the Canadian Food Inspection Agency's (CFIA's) Federal Assistance Program
Final Report

Table of Contents

• Executive Summary
• 1. Introduction
• 2. Program Profile
  • 2.1 Background
  • 2.2 Governance, Roles and Responsibilities
  • 2.3 Recipients and Projects
  • 2.4 Resources
  • 2.5 FAP Logic Model
  • 2.6 Past Evaluation and Audit
• 3. Methodology
  • 3.1 Evaluation Scope
  • 3.2 Evaluation Methods
  • 3.3 Analysis and Integration of Data
  • 3.4 Limitations
  • 3.5 Evaluation Issues, Questions and Methods
• 4. Findings
  • 4.1 Relevance: Continued need for the program
  • 4.2 Relevance: Alignment with federal government priorities
  • 4.3 Relevance: Alignment with federal government roles and responsibilities
  • 4.4 Performance: Achievement of expected outcomes
  • 4.5 Performance: Demonstration of efficiency and economy
• 5. Conclusions and Recommendations
• Appendices
• A. References
  • A.1 Core FAP Management Documents
  • A.2 Project Databases
• B. Interview Guides
  • B.1 Background (common to all guides)
  • B.2 Interview Guide: Senior CFIA Management
  • B.3 Interview Guide: FAP Management
  • B.4 Interview Guide: Responsibility Centre Managers
  • B.5 Interview Guide: Project Recipients
  • B.6 Interview Guide: Other Departments and Agencies
  • B.7 Interview Guide: Legal Services
  • B.8 Interview Guide: Public Affairs
• C. Summary of OGD Programs without a Separate Vote
  • C.1 Overview
  • C.2 CNSC Research Support Program (RSP)
  • C.3 CNSC Participant Funding Program (PFP)
  • C.4 LAC Documentary Heritage Communities Program (DHCP)
  • C.5 Parks Canada General Class Contribution Program (GCCP)
• D. Draft Performance Measurement Indicators
  • D.1 Outputs
  • D.2 Project-level Immediate Outcomes
  • D.3 Program-level Immediate Outcomes
  • D.4 CFIA-level Intermediate Outcomes

Executive Summary

Under the Financial Administration Act, an evaluation of the Federal Assistance Program (FAP) is required every five years. In addition, Treasury Board's (TB) Evaluation Policy requires that all direct program spending is evaluated every five years. This report presents the results of the evaluation undertaken in 2015.

The FAP is currently the CFIA's sole contribution program and is used to fund projects and initiatives that advance the Agency's strategic outcome; namely a safe and accessible food supply, plant and animal resource base.

The FAP is unique in that it is not a standard federally-funded contribution program; individual branches are required to use their operating budgets (i.e., A-base funding) as the source of funds for each contribution (i.e. there is no separate/distinct federal budgetary "vote"). As well, contribution recipients are not solicited in a public venue, but are identified by the CFIA officials in their individual dealings with stakeholder groups. Recipients can be universities, non-governmental organizations, international bodies and any other organization or individual except for other Canadian federal government departments or agencies.

The FAP has annual maximum allowable expenditures of $4.5 million, though it averages about $1.5 million. Over the four-year period of the evaluation's scope (2011-15), the FAP contributed $5.9 million over 23 agreements, with the smallest agreement at $5,500 and the largest at $2 million.

Key Findings and Recommendations

Overall, the program was found to be effectively managed, providing the Agency with opportunities to support third parties in activities that contribute to the CFIA's strategic outcome of a safe and accessible food supply and plant and animal resource base.

The findings and recommendations of the evaluation are provided below by the five core evaluation issues of the TB Policy on Evaluation (2009).

Relevance: Continued Need for FAP

The evaluation found evidence of a strong need for a broad contribution program, i.e., one that supports the Agency's strategic outcome. The FAP provides the flexibility to access expertise outside the Agency, build research partnerships, national and international, and support research of mutual interest, build capacity, and support other initiatives of value to the CFIA.

CFIA managers who used FAP were strongly supportive of it, and, as noted below, the Program has been successfully addressing its intended outcomes. However, the evaluation found a general lack of awareness of the Program in the Agency, which may have contributed to its under-usage, spending only one third of its $4.5 million annual funding limit.

Recommendation 1: The CFIA should increase awareness of the FAP across the Agency.

Relevance: Alignment with Government Priorities

FAP is aligned with federal government and CFIA priorities. All FAP funded project objectives were found to be consistent with the CFIA's strategic outcome.

Relevance: Alignment with Federal Government Roles and Responsibilities

Given that all FAP projects are consistent with the Agency's strategic outcome of "a safe and accessible food supply, plant and animal resource base," it is appropriate for the federal government as represented by the CFIA to be managing and delivering the FAP.

Performance: Achievement of Expected Outcomes

Most of the expected immediate outcomes have been achieved: supporting international engagement; veterinary and science capacity building; organizational development; and knowledge enhancement, all linked to the CFIA's strategic outcome.

Although good progress has been made since 2013, the use of indicators to report on performance is uneven across projects, and there is no systematic critical review of recipient reports, which could enhance objective monitoring and allow for a more cohesive review of all FAP projects.

Recommendation 2: The CFIA should develop and implement more standardized and detailed project and program performance monitoring and reporting.

Performance: Demonstration of Efficiency and Economy

The value of FAP is in its flexibility as a funding vehicle, its ability to leverage funds and its use as a tool to build partnerships, meaningful outreach/exposure and capacity beyond the CFIA. The Program appears to be efficiently and effectively managed. There is, however, room for improvement in terms of the administrative burden placed on recipients.

There is great variation in the type of projects, from one-day workshops to research networks involving multiple Canadian and international partners. Small and straight forward projects such as a workshop could involve a less extensive risk assessment, less frequent reporting by the recipient, and/or delegated approval (e.g., by Branch Head instead of President).

Recommendation 3: The CFIA should scale project administrative requirements based on type and size.

There was an expression of concern surrounding multi-year agreements and repeat agreements. While all FAP projects are subject to strict and robust governance, there is no mechanism in place that provides guidance around repeat and multi-year funding.

Recommendation 4: The CFIA should develop guidelines for the funding of repeat FAP projects over multiple years.

1. Introduction

The Canadian Food Inspection Agency's (CFIA's) 2015 Evaluation Plan included an evaluation of the Federal Assistance Program (FAP), to be completed in accordance with the Treasury Board Policy on Evaluation (2009) and the supporting Directive and Standard. The evaluation addressed the five core evaluation issues:

• Continued need for the program (relevance);
• Alignment with government priorities (relevance);
• Alignment with federal government roles and responsibilities (relevance);
• Achievement of expected outcomes (performance); and
• Demonstration of efficiency and economy (performance).

2. Program Profile

2.1 Background

In 1997, the CFIA established the Federal Inspection System Assistance Program, which was later renamed the Federal Assistance Program (FAP). The FAP is currently the CFIA's sole contribution program and is used to fund projects and initiatives that advance the Agency's strategic outcome; namely a safe and accessible food supply, plant and animal resource base. The FAP is funded annually from individual branch budgets or via other special initiatives (e.g., Food Safety Action Plan or Growing Forward). As such, FAP is a tool CFIA program managers utilize to broaden their reach by supporting third-party projects that are intended to contribute to^Footnote 1:

• Protecting Canadians from preventable health risks;
• Delivering a fair and effective regulatory regime for Canadians;
• Sustaining the plant and animal resource base; and
• Promoting the security of Canada's food supply and agriculture resource base for Canadians.

The FAP has evolved significantly since its introduction. The following chart outlines the program's evolution.

Figure 2-1: Evolution of FAP Activities

Source: CFIA (February 6, 2014), FAP Deck; Presented to CFIA Policy Integration Committee, with minor edits

2.2 Governance, Roles and Responsibilities

The FAP is unique in that it is not a standard federally-funded contribution program; individual branches are required to use their operating budgets (e.g., A-base funding) as the source of funds for each contribution (i.e. there is no separate/distinct federal budgetary "vote"). As well, contribution recipients are not solicited in a public venue, but are identified by the CFIA officials in their individual dealings with stakeholder groups with the aim of broadening the CFIA's reach by supporting collaborative and partner initiatives.

There are a number of key individuals responsible for the delivery of the program as follows:

1. FAP Contribution Coordinator (CC) – Located within the Program, Regulatory and Trade Policy Directorate, Policy and Programs Branch, the CC is the main contact for the Program. The FAP CC advises Responsibility Centre Managers on the FAP process, maintains data on the progress of all FAP applications and ensures that relevant performance indicators are included in each FAP contribution agreement. In addition, the CC develops the recipient audit plan and coordinates the three-year forecast planning.^Footnote 2, ^Footnote 3
2. Responsibility Centre Managers (RCM) ensure the projects they recommend for funding under FAP contribute to the Agency's strategic outcome and meet the required Terms and Conditions. They are responsible for conducting a risk assessment for every project and obtaining Branch Head approval.^Footnote 4 RCMs liaise with the applicant/recipient, and are expected to develop three-to-five relevant performance indicators (in collaboration with the recipient) for each contribution agreement they manage, and monitor the status of the project, including payments and progress.^Footnote 5, ^Footnote 6
3. Branch Head is required to sign and sponsor all FAP funding requests, risk assessments and Contribution Agreements that originate within their branch.^Footnote 7
4. Recipients are responsible for preparing and completing a FAP Request for the Funding Application package^Footnote 8 in collaboration with the RCM and for submitting progress reports and/or a final project report according to the schedule in the Contribution Agreement.^Footnote 9

2.3 Recipients and Projects

The list of potential recipients is almost all encompassing, with the only stated exclusion being other federal departments, agencies, and Crown corporations.^Footnote 10

According to the FAP Guide, projects selected under the FAP generally support:

• National or international organizations that promote and contribute to food safety, animal health and plant protection;
• International standard-setting and regulatory bodies that establish worldwide regulatory frameworks and early warning systems;
• Third-party scientific survey and research work that is relevant to the CFIA's mandate; and
• Emergency projects by other levels of government, industry, and related associations that respond to disease outbreaks.

Over the four-year period from 2011-12 to 2014-15, there were 23 agreements with 15 different recipient organizations to sponsor 18 individual projects (several projects had multiple agreements). The recipient organizations and projects supported by FAP from 2011-12 to 2014-15 are shown in Table 2-1.

Source: FAP Contribution Agreements. *Genomics Alberta is only to receive the contribution amount in fiscal year 2015/16.

2.4 Resources

As noted above, the FAP is funded annually from individual branch budgets. Alternative forms of funding may be identified via other special initiatives. The total value of all contributions under the FAP in any one year cannot exceed $4.5 million. The maximum amount payable to any one recipient in any one year cannot exceed $2 million. The Terms and Conditions for the FAP indicate that contributions are provided at the minimum value to attain the project's expected results.

Table 2-2 below outlines annual funding for FAP projects from 2011-12 to 2014-15. On average, actual total contributions have amounted to $1.5 million, well below the $4.5 million limit per annum. Over the four-year period, the FAP contributed $5.9 million over 23 agreements, for an average award per contribution agreement of $260,000; the smallest agreement was $5,500 and the largest $2 million.

Source: Recipients and Annual Values of Contribution Agreements

2.5 FAP Logic Model

The Logic Model, included in the FAP Performance Measurement Strategy was revised by the evaluation team and validated with program management and through the evaluation process. The Logic Model is provided below in Figure 2-2. The evaluation identified four outcomes, slightly different from those listed by the Program, as noted in section 2.3 above and discussed below. Indicators and performance measures have been drafted and will be used and validated in the future.

Figure 2-2: FAP

The FAP logic model consists of:

• Inputs: When an RCM has identified a particular initiative or project that may be eligible for a FAP contribution and for which funds are available, the RCM works with the applicant to prepare the Request for Funding Application Package. When an approved FAP project or initiative is being or has been executed, the recipient provides approved expenses claims and supporting documentation to the CFIA for reimbursement.
• Activities: From the initiation of the FAP application process and throughout the lifecycle of an approved FAP project or initiative, the RCM and the CC conduct various tasks required to put forward applications to gain approval and funding for the project, and to process and keep track of requests for advance payments (if applicable) and received expense claims. The RCM must also monitor the progress of the project through liaison with the recipient and by reviewing submitted progress reports for acceptability. The CC coordinates the annual three-year forecast (planning) process, develops the recipient audit plan, and prepares the Annual FAP Report. On a quarterly basis, the CC arranges for web publication of a summary of each FAP Contribution Agreement awarded for $25,000 and above.
• Outputs: The items (outputs) created as a result of the inputs and activities described above include: approved and signed FAP Contribution Agreements; payments for approved advance requests; reimbursement payments for authorized expenses; recipient audit plan; recipient risk assessment and risk management strategy; recipient progress and performance reports; annual FAP Report; quarterly web publication (disclosure) of FAP Contribution Agreements amounting to more than $25,000; and a three-year forecast of proposed FAP projects and funding amounts.

There are two types of immediate outcomes that are expected to result from the inputs, activities and outputs noted above: those relating to individual FAP projects; and those relating to the FAP program overall.

• Project-level Immediate Outcomes: The anticipated immediate outcomes of individual FAP projects are (a) scientific and technical knowledge is advanced and/or enhanced, (b) individual knowledge and skills are developed and/or improved, (c) international collaborations are expanded and/or strengthened, and (d) organizations or initiatives are established or sustained.
• Program-level Immediate Outcomes: The anticipated immediate outcomes for the FAP program overall include (a) promotion and awareness of policies, legislation and science-based regulations, (b) collaborations contribute to international standards for human, animal and plant related risks, (c) awareness of risks related to food supply, plant and animal resource base among stakeholders, (d) preparedness to prevent, address, and manage food, plant, and animal-related emergencies, and (e) effective management of contributions.

CFIA Intermediate Outcomes: When completing the Request for Funding Application Package, it is the responsibility of the RCM to articulate how the proposed FAP initiative or project will contribute to the advancement of the expected results (intermediate outcomes) of their program areas. Currently these outcomes are listed in the old logic model^Footnote 11 as: i) risks to the Canadian public associated with the food supply system are mitigated; ii) risks to Canadians from the transmission of animal diseases to humans are minimized; and, iii) risks to the Canadian animal and plant resource base are minimized. These were modified by the evaluation team to align with a more recent Agency logic model (see Figure 2.2), as follows: (a) risks to Canadian and International food supply, plant and animal resource base are managed, minimized and mitigated, (b) Canadian standards are recognized internationally, and/or (c) compliance with program policies, requirements and regulations among stakeholders.

CFIA Strategic Outcome: A safe and accessible food supply, plant and animal resource base. The collective inputs, activities, outputs and outcomes of the FAP provide evidence to support the advancement of the strategic outcome.

2.6 Past Evaluation and Audit

An evaluation of the FAP was conducted in fiscal year 2010-11 covering the period from June 2006 to October 2010. An audit was also conducted during this same time frame covering the period from June 2006 to August 2010. The audit and evaluation resulted in one joint recommendation:

The President should designate a senior CFIA executive accountable and responsible for the Federal Assistance Program to establish a program entity and to ensure that the program meets all applicable governance, risk management and control expectations for a federally-funded contribution program.^Footnote 12,^Footnote 13

Significant measures and controls have been put in place to address this recommendation, and most are fully operationalized. These include moving the FAP from the Corporate Management Branch to the Policy and Program's Branch, the designation of a senior CFIA executive accountable and responsible for the FAP management, and the creation of numerous risk management and governance documents, including a Performance Measurement Strategy, Recipient Risk Assessment and Management Framework, and Program Guide. The only measure not fully operationalized at the time of the evaluation was the Performance Measurement Strategy. As such, it was not examined in detail. Its implementation will be examined during the next evaluation, within five years.

3. Methodology

The CFIA's Evaluation Directorate managed the evaluation and conducted it with the assistance of Hickling Arthurs Low Corporation. The evaluation was guided by an Advisory Committee which reviewed and provided feedback on the evaluation plan, findings, report and Management Response and Acton Plan. The evaluation was conducted over the period from May 2015 to January 2016.

3.1 Evaluation Scope

The evaluation covers the period from April 2011 to March 2015. All FAP activities were analyzed including all relevant areas of the Agency providing significant support to the Program.

3.2 Evaluation Methods

The following methods were applied to collect data from multiple sources:

3.2.1 Document and Data Review

The document and data review covered:

• Contribution Agreements;
• Recipient funding and expenditure data;
• Recipient reporting documents (progress reports and final reports);
• FAP documents (e.g., Terms and Conditions, Guide, Recipient Risk Assessment and Management Framework, Performance Measurement Strategy)
• Relevant Acts and regulations;
• Previous evaluation and audit (2011); and
• Other relevant documents (e.g., Treasury Board guide on grants and contributions).

Details are provided in Appendix A.

3.2.2 Interviews

Thirty-two interviews were completed with senior CFIA management, FAP management and support (e.g., CFIA Finance and Public Affairs), responsibility centre managers (RCMs), and FAP recipients. The distribution of planned and completed interviews is shown in Table 3-1. The interviews followed an interview guide to ensure consistency of information collected and were semi-structured to allow for effective probing of the issues; different interview guides were used for each interview type and are provided in Appendix B. Interviews were conducted in the interviewees' official language of choice, in-person or by telephone.

The value of the 10 projects covered by the evaluation interviews accounts for 89 per cent of the $5.9 million disbursed by FAP over the 2011-15 study period.

3.2.3 Comparative Review

A comparative review of four similar programs was undertaken to investigate how other government departments and agencies (OGDs) manage contribution programs with no separate/distinct vote, i.e., those under $5 million per year. The review addressed evaluation questions related to relevance and performance, and provided insight into best practices for FAP management.

All contribution programs examined, including FAP, have an annual spending maximum of $4.5 million. By contrast, contribution programs with an annual minimum funding of $5 million and over require a separate federal budgetary vote. The four programs reviewed are:

• 1) Canadian Nuclear Safety Commission (CNSC):
  • Research Support Program (RSP).
• 2) Canadian Nuclear Safety Commission (CNSC):
  • Participant Funding Program (PFP).
• 3) Library and Archives Canada (LAC):
  • Documentary Heritage Community Program (DHCP).
• 4) Parks Canada (PC):
  • General Class Contribution Program (GCCP).

The comparative review was based on findings from a review of program documentation and interviews conducted with CNSC and DHCP. The findings are included throughout section four of this report, and summarized in Appendix C.

3.3 Analysis and Integration of Data

Although the interviews were not designed or used to collect quantitative data, the interpretation of findings takes into account the extent to which certain perceptions or views were expressed or shared by the interviewees. Table 3-2 defines terms used in this report to quantify the proportion of interviews who expressed similar experiences, views and opinions.

3.4 Limitations

The evaluation challenges and limitations and the corresponding mitigation strategies are described in Table 3-3 below.

3.5 Evaluation Issues, Questions and Methods

A matrix showing the evaluation issues, questions and methods used is shown in table 3-4.

4. Findings

The evaluation findings are presented by evaluation question, as presented in table 3-4 above.

4.1 Relevance: Continued need for the program

4.1.1 What is the rationale for funding FAP projects?

The rationale for funding FAP projects is to access expertise outside the CFIA, build capacity, support international collaborations and establish organizations or initiatives.

As part of the process to update the FAP logic model, the evaluation examined: (a) the linkage between the rationale for funding FAP projects and the expected results as indicated by the contribution agreements and key informant interviews (addressed below), and (b) whether the actual results achieved contributed to the Agency's strategic and intermediate outcomes (addressed in section 4.4).

The rationale and the expected results of the 18 FAP projects are shown in Table 4-1. The evidence for the rationale was obtained through a review of "objectives" and "expected results" sections of the contribution agreements. This was corroborated through interviews with RCMs and project recipients, and by a review of project reports.

Note: Some projects had more than one reason for funding; e.g., support was provided to the World Organisation for Animal Health to access outside expertise and to support international collaborations.

Interviewees were asked to explain the process by which program managers identify FAP projects. CFIA senior managers and RCMs said that 17 of the 18 projects were identified by CFIA officials.^Footnote 14

The four OGDs reviewed have similar rationales for funding projects. As noted in Table 4-2, building capacity is a rationale for funding by all four programs in the comparative review, accessing outside expertise and supporting international collaborations by two programs, and establishing and/or sustaining organizations or initiatives by one program.

Source: Appendix C

4.1.2 Is there a continued need for FAP?

There is a consistent need for FAP as it is a mechanism that allows the CFIA to extend its reach, address risks and leverage knowledge and resources not otherwise obtainable.

There is a consistent need to leverage expertise outside the CFIA according to most senior CFIA managers and RCMs interviewed for the evaluation. These interviewees also noted that FAP, as the CFIA's sole contribution program, is sometimes the only vehicle to provide funding to address these needs. The section of the contribution agreements entitled "how the results will contribute to the CFIA's mandate" supports interviewee responses as this is where links are made between project goals and the CFIA's strategic outcome.

Some FAP projects are aligned with risks identified in the CFIA's Corporate Risk Profile (CRP). For example, Emergency Management and Inspection Effectiveness are both identified as risks under the CRP, which are addressed by the FAP projects Canadian Veterinary Reserve (CVR) and Safe Food Canada (SFC).

A majority of project recipients interviewed (8) indicated that without FAP funding, the scope of their project would have been smaller and implementation would most likely have been delayed. Two project recipients indicated that their project would not have proceeded in the absence of FAP funding.

There appears to be limited awareness of FAP within the Agency. Although a question on awareness of FAP was not asked, many CFIA interviewees (senior managers, RCMs, FAP management) cited a lack of awareness as a weakness of FAP. RCMs noted that FAP is not well known within the Agency and suggested that more could be done to promote the Program. Supporting this point is the lack of full utilization of the Program. Actual expenditures over the last four years have only averaged around $1.5 million, well below the $4.5 million limit per annum.

4.2 Relevance: Alignment with federal government priorities

4.2.1 Are FAP project objectives consistent with government-wide priorities and the CFIA's mandate?

All FAP funded project objectives were found to be consistent with CFIA's mandate and government-wide priorities.

All FAP contribution agreements are required to outline the project's objectives and how the intended results will contribute to CFIA's strategic outcome. As noted in Table 4-2, the intended results can be grouped into four areas: (1) advancing / enhancing scientific and technical knowledge; (2) developing / improving individual knowledge and skills; (3) expanding / strengthening international collaborations; and (4) establishing / sustaining organizations or initiatives. These results are clearly aligned with the Agency's strategic outcome of "a safe and accessible food supply, plant and animal resource base", which supports the CFIA's and federal government's priority for food safety, as identified in the following:

• Speech from the Throne 2013^Footnote 15 notes the Safe Food for Canadians Act was a significant milestone in strengthening Canada's food safety system.
• Budget 2014 (Strengthening Canada's Food Safety System)^Footnote 16 outlines the Government of Canada's commitment "to ensuring that Canadian families have confidence in the food they buy and eat." This includes "increasing scientific capacity" and "investments to enhance the Government's ability to respond proactively to food safety issues and improved market access for Canadian agriculture and agri-food products."
• Budget 2015 (Fostering Trade – Securing Agriculture Market Access)^Footnote 17 notes that "Canada is the fifth largest exporter of agricultural and agri-food products globally". Measures to promote exports include: (1) "a more active role in setting international science-based standards" which "will support the agriculture sector in continuing to expand and diversify into new markets and in continuing to capitalize on opportunities created"; and (2) indicating to trading partners that "agricultural and agri-food products produced in Canada are among the safest and highest quality in the world."
• 2015-16 Report on Plans and Priorities^Footnote 18 indicates that "mitigating risks to food safety is the CFIA's highest priority. Safeguarding the health and well-being of Canada's people, environment, and economy is the driving force behind the design and development of the CFIA's programs."

While the FAP program is designed to cover any and all projects, a few interviewees wondered if FAP could be utilized for a strategic issue or priority if need be. This utilization could work as long as it is not the only focus for funding in any particular year. This would allow FAP to continue to be used for any and all projects and not just those priority issues identified.

4.3 Relevance: Alignment with federal government roles and responsibilities

4.3.1 Alignment with Federal Government Guidelines for Contribution Agreements

The Treasury Board of Canada Secretariat's (TBS's) Policy on Transfer Payments indicates that a procurement contract is used to obtain goods or services, whereas a transfer arrangement (e.g., contribution agreement) is used to transfer monies or make in-kind contributions from the federal government to individuals, organizations or other levels of government to further government policy and the department's objectives. A key aspect of transfer payments is that they do "not result in the acquisition by the Government of Canada of any goods, services or assets."^Footnote 19

The evaluation found, based on applying the TBS "checklist tool"^Footnote 20 for all 18 FAP projects, the transfer payment arrangement to be more appropriate than procurement contracts. For example, the CFIA did not directly acquire a good or service from any of the 18 FAP projects, and all projects advanced the Agency's strategic outcome (as shown below in Section 4.4).

The TBS identifies five forms of transfer payments: contributions, grants, other transfer payments (OTP), alternative funding arrangements (AFAs), and flexible transfer payments (FTPs). Of the five forms, contributions are most appropriate for programs like FAP. A contribution is defined as a conditional transfer whereby specific terms and conditions must be met or carried out by a recipient before costs are reimbursed, whereas a grant is an unconditional transfer payment where eligibility criteria and applications received in advance of payment sufficiently assure that the payment objectives will be met. OTPs are "based on legislation or an arrangement that normally includes a formula or schedule as one element used to determine the expenditure amount." Examples of OTPs are transfers to other levels of government such as Established Program Financing and transfers to the territorial governments. AFAs and FTPs are transfers specific to Aboriginal Affairs and Northern Development Canada.^Footnote 21

An advantage of a contribution (which requires funding recipients to report on results) over a grant (which does not) is the higher level of accountability. The evaluation found all 18 FAP projects compliant with the Terms and Conditions for contributions as outlined in TBS's Directive on Transfer Payments.^Footnote 22

4.3.2 Are the roles and responsibilities of the CFIA clear and understood by internal and external stakeholders in regards to the management and delivery of FAP?

A clear understanding was evident by both internal and external stakeholders on the roles and responsibilities of the CFIA in regards to the management and delivery of FAP. This was, for example, indicated by all senior CFIA management and RCM interviewees.

FAP management has developed many templates and guidelines to help guide the management and delivery of FAP, as listed in Appendix A.

FAP management also provides hands on support to the RCMs throughout the entire project lifecycle from project development, approval, monitoring, and reporting. The majority of senior CFIA managers and RCMs indicated that FAP templates, guidelines and hands on support are useful for managing and delivering FAP projects. Most internal interviewees also identified the existence of a dedicated FAP CC as a strength of the program.

A few internal interviewees, however, felt guidelines on when to use a contract or a contribution agreement would be helpful, as the difference between the two is not easily understood. Closely related to this issue is the definition of direct vs. indirect benefits of project activities to the Agency vs. the greater good; as a contract is used for direct benefits and a contribution for indirect benefits.

External interviewees have a similar perspective with respect to the management and delivery of FAP projects. The majority of FAP recipients indicated that support from the RCMs and FAP management was good.

4.4 Performance: Achievement of expected outcomes

4.4.1 To what extent have FAP projects produced the intended outcomes?

Of the 13 completed FAP projects, over the four-year period from 2011-12 to 2014-15, all have produced some if not all of their intended immediate outcomes.

Results achieved by projects are in line with those defined in their respective agreements. All 13 completed FAP projects submitted a final report. Although there was a significant variance in recipient report evidence and detail, they all clearly showed achievement of intended outputs as well as evidence of achievement of intended immediate outcomes. In addition, the majority of interviewees, both RCMs and project recipients, spoke to the successful achievement of intended outcomes. The outcomes from each project is shown below in Table 4-3 listed according to the four key areas, as per the FAP logic model.

Five FAP projects are still ongoing (GREZOSP, CRVE-NET, U of Guelph, Safe Food Canada, and Genome Alberta) and for these, final outcomes are not yet determined but project activities are progressing well.

Some examples of project outcomes produced in the four key areas are as follows:

• Scientific/technical knowledge advanced/enhanced: Chair in Regulatory Epidemiology established at UPEI; enhancements made to Canada's Traceability National Information Portal for monitoring cattle; Decision/Risk Analysis Framework established in conjunction with University of Guelph for determining food safety priorities (ongoing).
• Individual knowledge and skills transferred/improved: Emerald Ash Borer workshops enhanced attendee's expertise and ability to manage the impact of disease outbreak; Workshops delivered by University of Montreal improved future veterinarians' leadership and communication skills.
• International collaboration expanded/strengthened: Assisted developing countries to participate in creating science based food safety/quality standards (World Health Organization); Assisted International Plant Protection Convention member countries to participate in the development of global phytosanitary standards; Support for global conference on international standards for aquatic animal health (World Organisation for Animal Health).

4.4.2 Have there been any unintended outcomes resulting from FAP projects?

Most recipients and RCMs noted that FAP projects resulted in other benefits and broader outcomes for both the CFIA and the recipient community, as stated in final reports submitted by recipients. In all cases, these unintended outcomes were positive, although varied and broad reaching, with many having longer term impacts. They ranged from closer, more appreciative relationships between CFIA veterinarians and those in private practice, thus affecting future collaborative efforts, to improvements in Canada's visibility and reputation in food safety at an international level. Listed below are further examples of these unintended positive outcomes, according to RCMs and recipients.

For CFIA these include:

• Positive impact on trade, due to an enhanced Canadian food security reputation;
• Access to international support in the event of a disease outbreak due to stronger country to country relations; and
• Improved recognition/visibility/profile for Canada through leading edge research networks and involvement with international agencies.

For recipients these included:

• Enhanced linkages with other organizations to leverage additional funding; and
• Better alignment of Canadian university curriculums, in the area of food safety and security, to meet industry needs (longer term benefit).

For both CFIA and recipients these included:

• Improved integration and understanding between industry and regulators; and
• Closer/improved relationship with CFIA, improved understanding of what each stakeholder brings to the table.

4.5 Performance: Demonstration of efficiency and economy

4.5.1 Are FAP projects cost-effective as currently delivered? Alternatives?

The majority of projects appear to be cost effective as currently delivered. The program is delivered with minimal staff resources; the FAP CC, who resides in the Policy and Programs Branch (PPB), is the only full-time resource and spends a self-estimated 90 per cent of her time administering FAP. A few junior staff within PPB, along with CFIA Public Affairs and Finance, support the FAP CC on an as needed basis.

Significant effort has been exerted in the last two years to frame program requirements and put guidance and controls in place. Standardized templates and clauses (e.g., official languages clause, template for further distribution of funds, proactive disclosure procedure, summary information template, etc.) are now in place with frequent improvements introduced. Most recently, a standardized recipient reporting requirements template was developed to accompany all contribution agreements.

Individual branches also dedicate time to manage their own FAP projects and liaise with recipients, via the designated RCM. RCM time invested ranges from an estimated two per cent to 10 per cent^Footnote 23 over the course of a FAP project. RCMs typically invest most of their time at the front-end of a project; however, depending on the schedule/nature of a project, RCM allotted time to a given project can fluctuate. One notable exception is Safe Food Canada which required upwards of two to three days/week RCM time over 18 months before it was launched. Several other RCMs noted significant upfront investment due to: (i) complexity of international organization requirements (e.g., World Health Organization, World Bank); and, (ii) change in funding mechanism and subsequent change in nature of recipient/CFIA relationship (Canadian Veterinary Reserve).

Most recipients had no particular issue with the application and agreement process, while some noted it to be slow and complained of the paperwork. There was similar feedback on FAP reporting requirements which many project recipients found to be burdensome and time consuming, thus adding cost to managing the project. However, most recipients also noted they understood the need for transparency and appropriate oversight. Some recipients felt this still could be maintained with reduced reporting requirements.

Suggestions for improving cost-effectiveness made by some RCMs and senior management include: (i) consolidating FAP program information and up-to-date guidance documents on-line (this has since been initiated by the FAP CC); (ii) reducing the administrative burden/control for smaller and more simple projects; and, (iii) continuing to leverage additional/matching funds^Footnote 24 to extend CFIA's reach and create synergies.

Alternatives to the use of contribution agreements could include contracts, however, these involve a competitive process and may entail more administrative burden than the current contributions.^Footnote 25 Collaborative research agreements could be considered for research projects where both partners self-fund. Collaborative research agreements can enhance scientific capabilities through the exchange of knowledge, expertise and experience as well as provide access to scientific facilities and other sources of funding. This is particularly useful where stakeholders have similar research interests and common publication profiles. However, issues of shared intellectual property may arise and affect outcomes.

4.5.2 Is there duplication or gaps with other federal or provincial government programs and/or activities?

FAP is viewed as unique, understood to be the Agency's only contribution program, and was not seen to overlap or duplicate with other funding mechanisms/programs or activities. There were no specific incidences of duplication or overlap found, although given the broad range of projects that can be funded, the potential does exist for this to occur. There has been some co-funding of projects, which has in some cases used MoUs.

The majority of recipients also noted FAP does not duplicate any other program (federal or provincial) that they were aware of. However, in general, there is an inherent potential bias of funding recipients to avoid identifying overlap or duplication, which could then restrict future program usage. However, there was no specific evidence of such bias identified by the evaluation. Some recipients indicated that FAP was a good complement to other funding they had received (e.g., Ontario Ministry of Agriculture Food and Rural Affairs, AAFC Growing Forward) and thus allowed them to further leverage project activities.

4.5.3 Is the funding of FAP projects an efficient use of the Agency's resources? If not, how could these resources be better utilized?

For most projects, and the Program in general, funding expended through the FAP was found to be an efficient use of Agency resources.

Overall, the value of FAP is in its flexibility as a funding vehicle, its ability to leverage funds (co-funding and in-kind contribution come from industry, academia and provinces) and its use as a tool to build collaboration partnership, meaningful outreach/exposure and capacity beyond CFIA. The outcomes affect more than CFIA and demonstrate efficiency due to the synergies that are created. The majority of both recipients and RCMs would participate in another FAP project.

Two issues were, however, raised:

• A concern raised by some senior managers was that sometimes little justification was provided for multi-year agreements and "repeat" recipients, particularly with "establishing organizations" type of projects, though OGD programs reviewed did allow for multi-year agreements. Concern was expressed in this regard given that some organizations may become dependent on long-term CFIA support to continue their existence; and
• Concern with delays in getting some projects approved due to a lengthy approval procedure (currently involves up to eight signatures)^Footnote 26 was raised by some recipients and RCMs.

Veterinarians without Borders (VWB) had repeat, multi-year funding agreements with FAP since 2006. Initially, this helped establish the Canadian chapter of the organization, which was then, over time, able to obtain other sources of funding, reducing its FAP support from $500,000 in 2010 to $250,000 in 2015.

Other repeat, multi-year agreements include those with the University of Prince Edward Island Atlantic Veterinary College for the Canadian Regulatory Veterinary Epidemiology Network, which received funding from 2011 to 2015, starting at $250,000 and decreasing to $150,000.

Finally, the World Organization for Animal Health (OIE), received $500,000 for a consecutive four years from 2010 to 2015.

Table 4-4 displays the annual FAP contributions amounts for the above examples.

There are currently no particular rules, policies or guidelines for repeat and multi-year funding. As with all program intended outcomes, indicators for the establishment or sustaining of organizations have been included for program performance monitoring. See Appendix D indicator #11.

Some CFIA interviewees noted that more clearly defined project activities and improved performance indicators would be helpful in demonstrating the efficient use of Agency resources (i.e., value for money) when funding recipients report on their progress.

4.5.4 What performance monitoring has been done and are the tools and systems adequate to provide meaningful feedback?

FAP performance monitoring is guided by the FAP Performance Measurement Strategy which sets out the information, tools and targets required for ongoing monitoring and reporting. Current project requirements include both recipient progress and final reports. These reports are managed and reviewed by individual RCMs. There was no standardized template for reporting for the study period, though this was being addressed at the time of this report's writing. Reports are therefore inconsistent in level of detail, content and format.^Footnote 27 Even though standardized reporting is lacking, most reports include meaningful feedback related to activities completed and results achieved. In addition, there is no systematic critical review of these reports.

Another key oversight tool is program recipient audits. Audits are completed by an external auditor and selected based on an assessment of risk and materiality; this activity is coordinated and managed by the FAP Coordinator. For the period of this evaluation, two recipient audits were completed, for Veterinarians without Borders and Canadian Veterinarian Medical Association. No issues of concern were identified.

The FAP also has a three-year Program funding forecast (currently available for the period 2015 to 2018), a quarterly disclosure process where agreements over $25,000 are posted on the CFIA's external website, and risk assessments^Footnote 28 conducted by RCMs for contributions agreements over $25,000.

The use of performance indicators has been initiated, as of 2014, with 10 of the most recent contribution agreements containing them. Prior to this, performance was measured against expected results and project objectives as stated in the contribution agreement. A review of these performance indicators found them to be reasonable for measuring the success of the overall project; however, many still require refinement to be measureable.^Footnote 29 It was suggested by some RCMs and FAP management that example performance indicators (i.e., a menu/list of options), that RCMs and recipients could draw on and consider when preparing contribution agreements, would be helpful in creating a more standardized and rigorous approach to performance measurement (as noted above in Section 2.5 – indicators and performance measures have been drafted by the evaluation team for future program validation).

Senior management noted they would benefit from a more global overview of FAP project results and how they contribute to the CFIA's strategic outcome. There is value in producing an annual report to address this interest, to capture a broader understanding of both achievements and impacts as a result of the FAP.

5. Conclusions and Recommendations

The 2010 evaluation and audit of the Federal Assistance Program concluded that the Program required better governance, risk management and controls. It was clearly evident in this evaluation that these management tools and structures have now been effectively put in place and are operating as planned, with minor exceptions. For example, there are components, such as performance measurement, that have only been developed in the past year and are expected to be implemented shortly.

The TB evaluation Directive outlines issues related to relevance and performance that evaluations are to assess. The relevance of FAP is supported by the clearly stipulated requirement that its projects support the Agency's strategic outcome. The evaluation found the projects to be so aligned. The need for the Program was also clearly demonstrated, partly because it is the only contribution program the Agency has access to, but also evident from the support for the program and interest in future usage by both CFIA management and funding recipients. Increasing the opportunity to take advantage of the Program is the intention of the first recommendation:

• Recommendation 1: The CFIA should increase awareness of the FAP across the Agency.

The performance of the Program is evident in the success of its projects, though performance measures have been inconsistent and no roll up of outcome reporting has been attempted. This led to the second recommendation:

• Recommendation 2: The CFIA should develop and implement more standardized and detailed project and program performance monitoring and reporting.

Governance, reporting and administrative requirements were found to be burdensome for smaller and less complex projects. The third recommendation is made to support projects more efficiently:

• Recommendation 3: The CFIA should scale project administrative requirements based on type and size.

Concern was expressed over long term funding through multi-year and repeat agreements, for which the CFIA currently has no guidelines.

• Recommendation 4: The CFIA should develop guidelines for the funding of repeat FAP projects over multiple years

In conclusion, the Program is relevant and effective, with only minor administrative adjustments recommended.

Appendices

A. References

A.1 Core FAP Management Documents

1. Canadian Food Inspection Agency (no date). Federal Assistance Program Performance Measurement Strategy.
2. Canadian Food Inspection Agency (no date), Federal Assistance Program Recipient Risk Assessment and Management Framework.
3. Canadian Food Inspection Agency, (no date) Federal Assistance Program Guide.
4. Canadian Food Inspection Agency (Amended October 1, 2014). Federal Assistance Program Terms and Conditions.
5. Canadian Food Inspection Agency (November 27, 2014). CFIA Federal Assistance Program – an Overview, Vivienne Atkinson, Program, Regulatory and Trade Policy Directorate (slide deck 1 of 3).
6. Canadian Food Inspection Agency (no date). FAP Proactive Disclosure Procedure; Summary Information Template.
7. Canadian Food Inspection Agency (no date). FAP Transmittal Slip Templates
8. Canadian Food Inspection Agency (no date). FAP Domestic Contribution Agreement template (see page 10, Section 37 for official languages clause)
9. Canadian Food Inspection Agency (no date). FAP Contribution Agreement template for Further Distribution of Funds (detailed information regarding Ultimate Recipients, see page 21, Appendix C: Selection Framework and Contribution Agreement Requirements)
10. Canadian Food Inspection Agency (no date). FAP Recipient Reporting Requirements; Appendix C or D of Contribution Agreement.

A.2 Project Databases

1. FAP Project Database 2011-12, 2012-13, 2013-14, 2014-15
2. FAP Project Database Table of Recipients and Values of contributions from 2010-11 to 2014-15
3. FAP Project Database FAP Three-year Forecast 2015-16 to 2017-18
4. Project contribution agreements, claims, progress and final reports, and audit reports

B. Interview Guides

B.1 Background (common to all guides)

Thank you for agreeing to be interviewed as part of the Evaluation of the Canadian Food Inspection Agency's Federal Assistance Program (FAP). CFIA's Evaluation Directorate has engaged a team of outside evaluation specialists at Hickling Arthurs Low Corporation to assist it with the conduct of the evaluation.

The FAP is currently the CFIA's sole contribution program and is used to support projects and initiatives that advance the Agency's strategic outcome; namely a safe and accessible food supply, plant and animal resource base. The FAP is funded annually from individual branch budgets. As such, FAP is a tool CFIA program managers utilize to broaden their reach by supporting third-party projects that are intended to contribute to:

• Protecting Canadians from preventable health risks;
• Delivering a fair and effective regulator regime for Canadians;
• Sustaining the plant and animal resource base; and
• Promoting the security of Canada's food supply and agriculture resource base for Canadians.

The objectives of the evaluation are to assess the relevance and performance of the program in accordance with the Treasury Board Policy on Evaluation.

Your views will be kept strictly confidential by the evaluation team, and only aggregated results will be included in the evaluation report. Once approved, the final evaluation report will be made public by the CFIA in accordance with Treasury Board policy.

Your interview is expected to take less than 45 minutes. If you have any questions about this study, please do not hesitate to contact the Evaluation Manager.

B.2 Interview Guide: Senior CFIA Management

Questions

• 1. Please describe the nature of your involvement with the FAP.

Relevance

• 2. In your view, is there a continued need for the FAP as a tool for CFIA program managers to broaden their reach and support third parties? If yes, what gap is it addressing, e.g., capacity-building, research, training?
• 3. How are FAP projects and recipients identified? What priorities/criteria are considered to determine FAP project eligibility?
• 4. Are there any changes that could be made to better align FAP with the Agency's mandate, objectives and priorities?
• 5. Are the roles and responsibilities of those managing the FAP clearly documented and understood?
• 6. Based on your involvement with FAP, can you identify overall strengths and/or weaknesses in relation to the management and delivery of FAP?
• 7. What processes and tools are in place to guide the FAP? In your view, are they clearly understood and utilized to improve program management and delivery?

Performance

• 8. Based on your experience, are there any ways in which the FAP is designed or operated that could be changed to improve its cost-effectiveness?
• 9. In your opinion, are there alternative delivery approaches to the FAP that would achieve the FAP's expected outcomes more efficiently (e.g., contract, "traditional" grants and contributions program, in-house, etc.)? If yes, please explain.
• 10. In your experience, does the FAP complement, duplicate, overlap or work at cross purpose with other government programs (federal, provincial)? If so, in what manner and to what extent?
• 11. Are you aware of how performance measurement is conducted for the FAP? Would you have any suggestions for improving this process?
• 12. Do you think the FAP projects as currently delivered are providing value for money for the Agency? If not, how could these resources be better utilized?
• 13. What mechanisms and processes could be put in place to better ensure that FAP projects are providing value for money for the Agency?
• 14. Are there any other additional comments you would like to add?

B.3 Interview Guide: FAP Management

Questions

• 1. Please describe your current position and role in relation to the FAP.

Relevance

• 2. How are FAP projects and recipients identified? What priorities/criteria are considered to determine FAP project eligibility?
• 3. In your view, is there a continued need for the FAP as a tool for CFIA program managers to broaden their reach and support third parties? If yes, what gap is it addressing, e.g., capacity-building, research, training?
• 4. Are there any changes that could be made to better align FAP with the Agency's mandate, objectives and priorities?
• 5. Are the roles and responsibilities of those managing the FAP clearly documented and understood?
• 6. Based on your involvement with FAP, can you identify overall strengths and/or weaknesses in relation to the management and delivery of FAP?
• 7. What processes and tools are in place to guide the FAP? In your view, are they clearly understood and utilized to improve program management and delivery?

Performance

• 8. Approximately how much of your time and that of your staff is spent on FAP-related activities? Are there other CFIA expenses required to manage FAP, e.g. financial?
• 9. Based on your experience, are there any ways in which the FAP is designed or operated that could be changed to improve its cost-effectiveness?
• 10. In your opinion, are there alternative delivery approaches to the FAP that would achieve the FAP's expected outcomes more efficiently (e.g., contract, "traditional" grants and contributions program, in-house, etc.)? If yes, please explain.
• 11. Do you have a policy to determine the eligibility and valuation of in-kind contributions? Are recipient organizations required to provide matching funds to support project activities?
• 12. In your experience, does the FAP complement, duplicate, overlap or work at cross purpose with other government programs (federal, provincial)? If so, in what manner and to what extent?
• 13. Do you think the FAP projects as currently delivered are providing value for money for the Agency? If not, how could these resources be better utilized?
• 14. What mechanisms and processes could be put in place to better ensure that FAP projects are providing value for money for the Agency?
• 15. Are there any other additional comments you would like to add?

B.4 Interview Guide: Responsibility Centre Managers

Questions

• 1. Please confirm the FAP projects you were responsible for.

Relevance

• 2. How did you become aware of the FAP? Was the application and agreement process, as required by the FAP Guide and the program's terms and conditions, clear and effectively managed?
• 3. How were the FAP projects and recipients identified? What priorities/criteria did you consider to determine FAP project eligibility?
• 4. In your view, is there a continued need for the FAP as a tool for CFIA program managers to broaden their reach and support third parties? If yes, what gap is it addressing, e.g., capacity-building, research, training?
• 5. Are there any changes that could be made to better align the FAP with the Agency's mandate, objectives and priorities?
• 6. Are the roles and responsibilities of those managing the FAP clearly documented and understood?
• 7. Would you initiate and lead another FAP project? Why/why not?
• 8. Based on your involvement with the FAP, can you identify overall strengths and/or weaknesses in relation to the management and delivery of FAP?
• 9. What processes and tools are in place to guide the FAP? In your view, are they clearly understood and utilized to improve program management and delivery?

Performance

• 10. Can you describe the direct and indirect benefits of the FAP project(s) you have managed to your directorate/branch/group, as well as external stakeholders? Did the results and achievements of the FAP project unfold as expected? Are there any downstream or future benefits expected from the project?
• 11. As a result of the project, could you indicate whether your directorate/branch/group have, or expect to, achieve any of the following impacts:
  • Protect Canadians from preventable health risks;
  • Support the Canadian food safety regulatory regime (e.g., international standard setting),
  • Sustain the plant and animal resource base
  • Promote the security of Canada's food supply and agricultural resource base
• 12. Can you describe any other benefits and broader outcomes, both intended and unintended, that resulted from the FAP project(s) you have managed?
• 13. Approximately how much of your time and that of your staff is spent on FAP-related activities? Are there other CFIA expenses required to manage FAP, e.g., financial?
• 14. Based on your experience, are there any ways in which the FAP is designed or operated that could be changed to improve its cost-effectiveness?
• 15. In your opinion, are there alternative delivery approaches to the FAP that would achieve the FAP's expected outcomes more efficiently (e.g., contract, "traditional" grants and contributions program, in-house, etc.)? If yes, please explain.
• 16. In your experience, does the FAP complement, duplicate, overlap or work at cross purpose with other government programs (federal, provincial)? If so, in what manner and to what extent?
• 17. Do you think the FAP projects as currently delivered are providing value for money for the Agency? If not, how could these resources be better utilized?
• 18. What reporting requirements were put in place to monitor the progress/performance of the FAP project's you manage?
• 19. In your view, were these requirements reasonable (manageable) and an effective way to capture the results of the FAP project(s) you manage? In your view, did they provide meaningful feedback to CFIA? Why/why not?
• 20. Were project performance indicators put in place? If yes, was this part of the agreement process? How were the results measured?
• 21. In your view, were these performance indicators adequate in capturing project performance? Do you have any suggestions for improving the performance indicators and the process of measuring results against them?
• 22. What mechanisms and processes could be put in place to better ensure that FAP projects are providing value for money for the Agency?
• 23. Are there any other additional comments you would like to add?

B.5 Interview Guide: Project Recipients

Questions

• 1. Please provide a brief overview of your FAP project (objectives, timing and activities to date).
• 2. How did you become aware of the FAP? Was the application and agreement process clear and effectively managed?
• 3. If funds to support your project had not been approved by the FAP, how would this have changed the viability, scale and/or timing of your project? Would you have proceeded with your project/initiative without the FAP funding?
• 4. How closely was the CFIA project manager involved in defining and managing your FAP project? Did the CFIA project manager contribute in any way to the success of your project?
• 5. Would you participate in another FAP project? Why/why not?
• 6. Based on your involvement with the FAP, can you identify overall strengths and/or weaknesses in relation to the management and delivery of FAP?
• 7. Can you describe the direct and indirect benefits of your FAP project(s); to your organization and other stakeholders? Did the results and achievements of your FAP project(s) unfold as expected? Are there any downstream or future benefits expected from the project(s)?
• 8. As a result of the FAP project(s), could you indicate whether your organization has, or expect to, achieve(d) any of the following impacts:
  • Protect Canadians from preventable health risks;
  • Support the Canadian food safety regulatory regime (e.g., international standard setting);
  • Sustain the plant and animal resource base;
  • Promote the security of Canada's food supply and agricultural resource base.
• 9. Can you describe any other benefits and broader outcomes, both intended and unintended, that resulted from your FAP project(s)?
• 10. Do any requirements of the FAP add costs to the project that were found to be inappropriate?
• 11. In your experience, does the FAP complement, duplicate, overlap or work at cross purpose with other government programs (federal, provincial)? If so, in what manner and to what extent? Have you ever received funding from these programs?
• 12. What reporting requirements were put in place to monitor the progress/performance of the (your) FAP project?
• 13. Did you (or your team) create performance indicators for your project(s)? If yes, were these indicators helpful in assisting you to measure the results/outcomes of your project? Were these indicators a program requirement?
• 14. Are there any other additional comments you would like to add?

B.6 Interview Guide: Other Departments and Agencies

Questions

• 1. Can you please describe the grant/contribution program your department/agency delivers that is similar to FAP, i.e. does not have a separate and distinct vote. Is there a particular reason/logic as to why it is delivered in this way? Please describe the objectives of the program, eligibility of projects and recipients, how are roles and responsibilities defined etc. Is there any documentation on the program you can share with us?
• 2. Can you please describe the senior-level oversight mechanisms in place for this grant/contribution program?
• 3. Do you have a policy to determine the eligibility and valuation of in-kind contributions? Are recipient organizations required to provide matching funds to support project activities?
• 4. Do you set a limit as to how often a grant/contribution can be awarded to any one recipient (for on-going or one-off projects)?
• 5. Is value for money assessed re: outcomes of funded projects? (i.e., how are results/outcomes of contributions measured and reported, by recipients and your department/agency?)
• 6. Do you have any other comments, in regards to lessons learned or best practices related to the delivery of this program, that you might like to add?

B.7 Interview Guide: Legal Services

Questions

• 1. Can you explain/describe the legal guidance you have been providing the FAP? What are the legal risks faced by this type of program?
• 2. What additional legal definitions/mechanisms /tools could be provided to the FAP management in order for them to more effectively delivery the program e.g., interpretation of terms, international templates, etc.?
• 3. Do you have a policy to determine the eligibility and valuation of in-kind contributions?
• 4. Do you have any additional comments you would like to provide in regards to the delivery and management of the FAP?

B.8 Interview Guide: Public Affairs

Questions

• 1. What type of outreach and communication/ liaison do you undertake on behalf of the FAP (externally/internally)? What are the processes in place for this communication and how have they changed over the past 5 years?
• 2. In your opinion, what is the current level of awareness and interest of the FAP within the Agency and with external stakeholders? What are its communication challenges?
• 3. Do you have any additional comments you would like to provide in regards to the delivery and management of the FAP, in the context of public affairs and communications?

C. Summary of OGD Programs without a Separate Vote

C.1 Overview

The four other government department and agency (OGD) programs without a separate vote included in the comparative review are:

• 1) Canadian Nuclear Safety Commission (CNSC):
  • Research Support Program (RSP)
• 2) Canadian Nuclear Safety Commission (CNSC):
  • Participant Funding Program (PFP).
• 3) Library and Archives Canada (LAC):
  • Documentary Heritage Community Program (DHCP).
• 4) Parks Canada (PC):
  • General Class Contribution Program (GCCP).

It is noted that the contribution programs examined, including FAP, have an annual spending maximum of $4.5M. By contrast, contribution programs with an annual minimum funding of $5M and over require a separate federal budgetary vote.

A summary overview of the findings from this comparison is provided below in Figure C-1, followed by summary profiles of each program. As noted in the figure, FAP is designed very much like OGD programs without a separate vote.

C.2 CNSC Research Support Program (RSP)

The Canadian Nuclear Safety Commission's (CNSC) Class Grants and Contribution Program was implemented in 1984 to provide funding to third parties to conduct research and development (R&D) and to provide regulatory support. The CNSC's Class Grants and Contribution Program consist of two parts:

• 1) Research Support Program which includes grants and contributions for external R&D and other related scientific activities; and
• 2) Participant Funding Program (introduced in 2011) which includes contributions to augment avenues available for public input into the Commission's regulatory processes.^Footnote 30

The purpose of the CNSC's Class Grants and Contribution Program is to provide access to advice, expertise, experience, and information that would otherwise not be available through agreements with private sector and other agencies and organizations, individuals and not-for-profit organizations in Canada and elsewhere.

C.2.1 Objectives/Purpose

The purpose of the Research Support Program (RSP) is to provide the CNSC with scientific, technical and other advice and information it needs to support the Commission's role and responsibility as outlined in the Nuclear Safety and Control Act (NSCA)^Footnote 31.

The objectives of the RSP are to enable research, development and the management of activities in order to reduce uncertainties regarding health, safety, security and environmental issues.^Footnote 32

The specific objectives of the RSP are to:

• acquire independent expertise, advice and information needed to support timely regulatory judgement and decisions;
• assist in the identification and assessment of operational problems which may give rise to health, safety, security or environmental hazards;
• assist in the development of capability and tools to be able to address health, safety, security or environmental issues;
• facilitate the assessment for the technical or scientific basis of licensing decisions and encourage licenses to address these issues; and
• aid in the development of nuclear safety standards.^Footnote 33

C.2.2 Comparison to FAP

Similarities to FAP

CFIA's FAP and CNSC's RSP are very similar. Both programs:

• support regulatory research;
• support for and strengthen standards;
• collaborate with Canadian and international stakeholders;
• support activities that protect the health of Canadians by minimizing risks;
• support activities that help to maintain public confidence;
• identify projects through staff networks and are therefore, not advertised nor is program information provided on the organization's website;
• are supported and coordinated by a secretariat (CNSC's Regulatory Research and Evaluation Division for RSP and CFIA's Program, Regulatory & Trade Policy for FAP);
• involve program officers (CNSC Technical Authority / CFIA responsibility centre manager) in the development and ongoing monitoring of projects;
• have a three-year plan (RSP) / program forecast (FAP);
• have average actual expenditures of $1.5 million per annum;
• allow multi-year funding;
• allow repeat agreements to the same recipient; and
• require a sign-off by the President.

Differences from FAP

Program aspects the RSP has that are different from the FAP:

• In addition to contributions, the RSP also provides grants. Over the five-year period from 2008-09 to 2012-13, RSP disbursed $5 million of which grants accounted for 7% and contributions 93%. For each individual contribution recipient, the annual maximum payable is $550K; and for each individual grant recipient, the annual maximum is $35K;
• RSP has two major recipients; the Canadian Standards Association Nuclear Standards Program (CSA/NSP) which accounted for 45% and the Organization of Economic Co-operation and Development's Nuclear Energy Agency (OECD/NEA) for 14% of RSP's $5 million expenditures from 2008-09 to 2012-13. It is noted that RSP only has contribution agreements with CSA/NSP and OECD/NEA, there are no grants;
  • Grants are only provided to organizations other than the CSA/NSP and OECD/NEA;
• Logic models and performance measurement strategies are developed for each of RSP's three recipient types: (1) CSA/NSP, (2) OECD/NEA and (3) other organizations; and

C.2.3 Summary

The RSP and the FAP are very similar. Both programs support regulatory research, support activities to minimize risk, involve domestic and international partners, identify projects through staff networks, have average actual expenditures of $1.5 million per annum, allow multi-year funding, allow repeat agreements to the same recipient, and have a dedicated program coordinator.

C.3 CNSC Participant Funding Program (PFP)

C.3.1 Objectives/Purpose

The CNSC's established the Participant Funding Program (PFP) under the Terms and Conditions of its Class Grants and Contribution Program to improve the regulatory review process by enhancing Aboriginal, public, and stakeholder participation in environmental assessments and public Commission proceedings, in order to help bring valuable information to the Commission.

The purpose of the PFP is to^Footnote 34:

• Enable the application (specifically the Environmental Assessment of Designated Projects section) of the Canadian Environmental Assessment Act, 2012 (CEAA 2012) on the part of the CNSC;
• Ensure more timely processes in project reviews and reduce the risk of time-consuming and costly delays because of challenges due to adequacy of process; and
• Encourage effective public participation ensuring that public concerns and values are taken into consideration during CNSC hearing processes, in order to promote an open and balanced review process, and strengthen the quality and credibility of reviews;
• Help fulfil the CNSC's constitutional and other obligations for consultation with Aboriginal groups on projects potentially affecting their rights and interests.

PFP's objectives^Footnote 35 are to:

• Enhance Aboriginal, public, and stakeholder participation in environmental assessments and public Commission hearings or proceedings at CNSC; and
• Help stakeholders better understand technical information related to environmental assessments (EAs) and licensing in order to bring valuable information to CNSC through informed and topic-specific interventions and assist the Commission Tribunal in making fully informed decisions.

C.3.2 Comparison to FAP

Similarities to FAP

Both FAP and PFP:

• support activities that promote effective regulatory frameworks;
• support activities that help to maintain public confidence;
• are supported and coordinated by a secretariat (CNSC's Policy, Aboriginal and International Relations Division for PFP and CFIA's Program, Regulatory & Trade Policy for FAP);
• allow multi-year funding; and
• allow repeat agreements to the same recipient.

Differences from FAP

Program aspects the PFP has that are different from the FAP:

• funding for PFP is provided through cost recovery from CNSC's licensees;
• senior-level oversight on application funding; an independent Funding Review Committee provides funding recommendations to CNSC senior management who make final funding decisions;
• the PFP administration operates on a cost-recovery basis, and has an annual operating budget of $175K for administration and salary costs;
• PFP opportunities are advertised, eligible applicant are required to submit a funding applications if interested; and
• recipients are not required to submit receipts, but are strongly advised to keep receipts for 3 years in case of an audit.

C.3.3 Summary

The main similarity is that both programs support activities that promote effective regulatory frameworks which in turn help to maintain public confidence.

The main difference is that the PFP's budget is cost recovered from CNSC's licensees, which includes an annual operating budget for the CNSC to coordinate and administer the program. Senior level oversight on final funding decisions for applications is made by CNSC senior management and is based on recommendations provided by a Funding Review Committee external and independent from the CNSC. PFP opportunities are also advertised and recipients are selected through a competitive review process based on eligibility, relevance and the potential value to the Commission's decision making process.

C.4 LAC Documentary Heritage Communities Program (DHCP)

Library and Archives Canada (LAC) launched the Documentary Heritage Communities Program (DHCP) in 2015/16 and will invest $1.5 million per year over five (5) years to implement the initiative for a total $7.5 million investment.

C.4.1 Objectives/Purpose

The purpose of the DHCP is to ensure that Canada's continuing memory is documented and accessible to current and future generations by adopting a more collaborative approach with local documentary heritage communities. The program will be delivered in the form of contributions that will support the development of Canada's local archival and library communities by increasing their capacity to preserve, provide access to and promote local documentary heritage. Additionally, the Program will provide opportunities for local documentary heritage communities to evolve and remain sustainable and strategic.

The DHCP provides financial assistance to the Canadian documentary heritage community for activities that fulfill the following objectives and related activities.

Objective 1: Increase access to, and awareness of Canada's local documentary institutions and their holdings. Eligible activities include:

• conversion and digitization for access purposes;
• development (research, design and production) of virtual and physical exhibitions, including travelling exhibits;
• collection, cataloguing and access based management; and
• commemorative projects.

Objective 2: Increase the capacity of local documentary heritage institutions to better sustain and preserve Canada's documentary heritage. Eligible activities include:

• conversion and digitization for preservation purposes;
• conservation and preservation treatment;
• increased digital preservation capacity (excluding digital infrastructure related to day-to-day activities);
• training and workshops that improve competencies and build capacity; and
• development of standards, performance and other measurement activities.

C.4.2 Comparison to FAP

Similarities to FAP

Similarities between FAP and DHCP:

• DHCP encourage program proponents to work closely with departmental staff to define and submit their request for funding; similar to FAP where project proponents work closely with RCMs to define and prepare funding requests;
• have detailed program Guidelines as well as a number of supporting documents: (i) Application Form, (ii) Project Budget From (iii) Application Checklist and (iv) Final Assessment and Financial Reporting; similar to FAP's Program Guide, Request for Funding form, Contribution Agreement Task Checklist and Recipient Reporting Requirements;
• have service standards in place (for acknowledgement, decision and payment);
• DHCP allow for multi-year funding, similar to FAP;
• allow for repeat funding to same recipient for second or third phases of same project, however DHCP requires proponent to re-apply along with the Final Assessment and Financial Report of original project;
• have no requirement for matching funds and recipients may redistribute funding to one or more eligible third parties;
• have no defined policy on in-kind contributions to value non-financial contributions from either project proponent and/or other project partners; and
• have no formal horizontal senior level oversight;
• final decision/sign-off rests with the Deputy Minister. For DHCP the responsibility rest with the Librarian and Archivists of Canada which is equivalent to CFIA President.

Differences from FAP

The differences between the two programs include:

• DHCP is advertised externally, funding is awarded based on a competitive process with defined selection criteria. As noted above, funding recommendations are made by an External Advisory Committee,
• DCHP has two funding thresholds, whereas FAP only has one, (up to $15K and $15 to $100K) with reporting requirements and payment terms defined accordingly, i.e. larger multi-year projects must provide interim assessment reports while smaller projects are only required to submit one final report, smaller projects receive 100% of their funds upfront while larger projects are subject to a 15% holdback;
• under DHCP, organizations that are administered by, or receive regular annual operational funding from any level of government, are ineligible, including colleges and universities;
• each DHCP project must present an evaluation strategy which includes the identification of pertinent performance measures (qualitative and/or quantitative) based on clearly articulated outcomes based the project type (and linked to DHCP's objectives) for which funding is requested. Although FAP contribution agreements do include detail on expected results and the project's alignment with CFIA's mandate, specific performance indicators have only recently been included in agreements (since 2014) and the project's ability to assess performance is not formally evaluated prior to funding;
• DHCP has a defined template for the project Final Assessment and Financial Report. The results are compiled and released annually, and will be made publicly available through Library and Archives Canada's Departmental Performance Report. It should be noted that FAP recently defined standardized Recipient Reporting requirements (not yet in use) and plans to compile an Annual Report, but has formalized this process;
• DHCP's service standards are posted externally;
• DHCP funding recommendations are provided by an External Advisory Committee comprised of archivists, librarians and academics from universities, provincial government and museums; and
• DHCP only awards multi-year funding to organizations that have had a previous relationship with LAC and have demonstrated financial stability. DHCP's maximum level of support is $100,000 per project per government fiscal year (April 1 to March 31) while FAP has set a maximum limit of $2 million to any one recipient per government fiscal year.

C.4.3 Summary

Many key aspects of the DHCP management are similar to FAP, however, the main difference is that DHCP is externally focused with a competitive process in place for project selection and a defined program budget and duration ($7.5 million over 5 years).

C.5 Parks Canada General Class Contribution Program (GCCP)

The Parks Canada Agency's (PCA) General Class Contributions Program (GCCP) is one of three contribution programs within the Agency. The GCCP is not a "program" in the traditional sense but rather functions as a general funding authority available to Parks Canada managers to support a wide range of recipient projects. It was originally launched in 2006-07 and since that time has sponsored a wide variety of both domestic and international projects aligned with the Agency's mandate.

C.5.1 Objectives/Purpose

The objectives of the GCCP are to assist recipients in conducting activities and delivering projects that will support Parks Canada in fulfilling its mandate to:

• preserve and protect nationally significant examples of Canada's natural and cultural heritage; and
• present and foster public understanding, appreciation and enjoyment in ways that ensure the ecological and commemorative integrity of these places for present and future generations.

Eligible projects and activities can include:

• provision of seed and interim funding to co-operative associations providing enhanced (value-added) services for visitors (services outside PCA's direct delivery mandate);
• project specific funding for research on and in support of commemorative and ecological integrity issues;
• project specific funding to promote education, outreach;
• provision of operational support for national historic sites – for sites not owned or administered by the Agency ;
• project specific funding to Aboriginal groups to obtain expertise related to land negotiations, park/site management planning and relationship building;
• project specific funding to Aboriginal groups for capacity development for economic opportunities and preservation of culture;
• operational support to management boards;
• community relationships/project specific support to municipalities within national parks to meet environmental or other standards set by the Agency (e.g. sewage, water treatment plants);
• project specific funding for ecological protection to nature conservation groups re: land assembly and easements bordering national parks. The lands will not be owned by PCA but are adjacent to national parks and enhance their ecological integrity;
• operational support to not-for-profit organizations to help establish areas of national historic or ecological significance;
• provision of funding to municipalities, for-profit businesses and non-government and non-profit groups to enable collaboration in connecting Canadians to nature;
• provision of funding to international organizations to support specific events or for knowledge and capacity building in fields of mutual interest;
• provision of funding to specific international organizations to pay assessed contributions on behalf of Canada; and
• provision of funding to foreign governments for projects related to shared ecosystems.

As such, the expected purpose and outcomes of the funding are:

• Canadians recognize, appreciate and are engaged in the values of natural and cultural conservation;
• stakeholders are engaged in terms of interest and involvement of common objectives towards ecological or cultural integrity;
• Parks Canada managers and stakeholders have access to a better knowledge base for informed decision making and dialogue on commercial, ecological or aboriginal issues of mutual interest;
• visitors are provided opportunities to purchase souvenirs or experience traditional meals (e.g. Fortress of Louisburg);
• heritage assets are protected, secured and researched; and
• targeted audiences are educated in such areas as ecology, safety and other issues.

C.5.2 Comparison to FAP

Similarities to FAP

Both FAP and GCCP:

• do not advertise externally and there is no defined program budget or central source of funding. Funds are re-allocated from individual business unit's O&M budgets. During Parks Canada's annual planning process each business unit identifies potential projects for the upcoming year and the opportunity cost of funding priority areas;
• encourage project proponents to work closely with Agency staff to define and submit their request for funding. Under FAP project proponents work closely with RCM's to define and prepare funding requests. Potential project recipients are identified by local managers or result from unsolicited proposals;
• Parks Canada has detailed Terms and Conditions for the program as well as a number of supporting documents: (i) Contribution Agreement Evaluation Form and (ii) Contribution Agreement Template. All similar to FAP's Program Guide, Request for Funding form and Contribution Agreement Task Checklist;
• have senior level sign off, by Parks Canada's CEO (equivalent to CFIA's President) who has final approval authority for all GCCP projects;
• require submission of both interim (for projects ≥ $25K) and final reports and do not have a standardized reporting format. It should be noted that FAP recently defined standardized Recipient reporting requirements but they have not yet been implemented by RCMs;
• no standardized performance indicators are in place for monitoring project performance. This was deemed to be particular difficult to implement given the significant variety of both recipients and project activities. However, contribution agreements contain sections on (i) Performance Measurement (describe how performance of the project and its key results will be monitored and evaluated) and (ii) Reporting (describe reporting requirements of the Recipient, in terms of the dates for submission of narrative, financial and other reports.);
• allow for repeat funding^Footnote 36 to same recipient; for Parks Canada a particular priority on stakeholders that are in close proximity to a National Park or historical site;
• have no requirement for matching funds and recipients may redistribute funding to one or more eligible third parties. Many of Parks Canada's projects involve Aboriginal groups and Parks Canada is usually the sole funding agent;
• have no defined policy on in-kind contributions to value non-financial contributions from either project proponent and/or other project partners; and
• have no formal horizontal senior-level oversight. Managers are expected to monitor projects following a risk-based plan developed during the negotiation phase of, and included in, each agreement. Similar to RCMs, local managers are to maintain files with copies of reports, evidence of results achieved, and details of project expenses and payments.

Differences from FAP

Program aspects the GCCP has that are different from the FAP:

• GCCP allows for multi-year funding^Footnote 37 in circumstances such as land claim negotiations. In such cases, Parks Canada's CEO pre-authorization is required before multi-year funding is discussed with the potential recipient. This measure is intended to limit the Agency's liabilities and provide for future budget flexibility.
• GCCP applications must include a communication plans, approved by Parks Canada's Director of Corporate Communications. These plans are utilized to identify project activities, objectives and results to both internal and external audiences.
• GCCP projects under $25K may not require submission of an interim report, only one final report.

C.5.3 Summary

GCCP and FAP are similar in numerous ways. Both programs do not advertise externally and have the flexibility to select and define projects to fund the Agency's most urgent priorities. Further, both FAP and GCCP are viewed more as funding mechanism as opposed to a traditional government program with centralized budgets and program offices. As with FAP management, Parks Canada also struggles with the definition of direct vs indirect benefit of project activities to the Agency vs. the greater good and subsequently, the applicability of a contract vs. a contribution agreement. Another challenge that faces both FAP and GCCP is the somewhat cumbersome and lengthy application process, particularly for contributions with smaller dollar value and lower risk.

D. Draft Performance Measurement Indicators

The following is a menu/list of options of performance indicators and is designed for RCMs and recipients to consider and use when monitoring and reporting on project progress/success. This list is by no means exhaustive. It should be understood that particular projects may require additional indicator development on the part of recipients and RCMs.

The indicators are linked to the outputs and outcomes as shown in the FAP logic model.

The intermediate and strategic outcome indicators are primarily for the FAP management team to use as a tool when monitoring and reporting on the FAP. However, they should still be considered by recipients and RCMs.

Where applicable, include planned versus actual.

D.1 Outputs

• 1) Service Standards met for:
  • a) Percentage of Agreements signed and
  • b) Percentage of Claims/Advances/Payments processed
• 2) Recipient Audit Plan
  • a) Audit conducted at least once every two years
• 3) Risk Management Strategy
  • a) Risk assessments conducted by Responsibility Center Managers (RCMs) for Agreements >$25K as scheduled
  • b) Risk identification abide by the Risk Management Strategy
  • c) Risks are identified with mitigation strategies, proportional to the levels of a proponent's assessed risk
• 4) Progress Performance Reports
  • a) Percentage or number of reports received by RCMs
  • b) Percentage or number of reports received by RCMs in a timely manner
  • c) Reports have relevant performance indicators to report on progress
  • d) Frequency of project reporting is based on the project's assessed level of risk
  • e) Reports demonstrate results, intended and otherwise
• 5) Quarterly disclosure of agreements >$25K
  • a) Posted on CFIA external website
• 6) 3-Year Forecast
  • a) Prepared by end of fiscal year
• 7) FAP Annual Report
  • a) Annual report complete; includes project results
  • b) Annual report completed on time

D.2 Project-level Immediate Outcomes

• 11) Organizations or Initiatives Established / Sustained
  • a) Number of organizations/initiatives established and fulfilling their mandate
  • b) Number of meetings/consultations, workshops, committees, coalitions
  • c) Ratio of CFIA contributions to total funding, per project, per year
  • d) Long term plan developed, by association or organization
  • e) Strategy for future planning, e.g. to extend network

D.3 Program-level Immediate Outcomes

• 12) Awareness of policies, legislation, and science-based regulations
  • a) Number of events for dissemination of knowledge
  • b) Number (and name) of countries participating, including the identification of those exporting to Canada
  • c) Target audiences reached
• 13) Collaborations contribute to international standards for human, animal, and plant-related risks
  • a) Number of formal agreements: MoUs, LOUs etc.
  • d) Number (and name) of countries participating, including the identification of those exporting to Canada
  • b) Number of standards affected
  • c) Risk framework(s) development/enhanced and disseminated
• 14) Awareness of risks related to food supply, the plant and animal resource base among stakeholders
  • a) Number of events for dissemination of knowledge
  • b) Number of countries participating, including the identification of those exporting to Canada
  • c) Target audiences reached
• 15) Preparedness to prevent, address, and manage food, plant, and animal-related emergencies
  • a) Evidence of adoption/application of new knowledge/ skills in their work
  • b) Number of adopted new methods/plans for preparedness
  • c) Number of training events and those who participated, number and type
• 16) Effective management of contributions
  • a) Number of progress reports approved by RCM/total # of progress reports
  • b) Number of final reports approved by RCM/ total # of final reports
  • c) Service Standards met
  • d) Project proponent satisfaction
  • e) RCM satisfaction
  • f) Senior Management satisfaction

D.4 CFIA-level Intermediate Outcomes

• 17) Risks to the Canadian and International food supply, plant and animal base are mitigated, minimized, or managed
  • a) New/additional risk frameworks/controls adopted by stakeholders
• 18) Canadian standards are recognized internationally
  • a) Canadian standards are (considered in international body decision-making) reflected in science-based international regulations and standards
• 19) Compliance with program policies, requirements and regulations among stakeholders
  • a) Domestic and imported products are compliant with Canadian requirements
  • b) Canadian exported products are compliant with recipient countries