Archived - Fertilizer Program Evaluation

Table of Contents

• Abbreviations and Glossary
• Executive Summary
• 1.0 Introduction
• 2.0 Evaluation Methodology and Limitations
• 3.0 Findings Issue 1: Continued Need for Program
• 4.0 Findings Issue 2: Alignment with Government Priorities
• 5.0 Findings Issue 3: Alignment with Federal Roles and Responsibilities
• 6.0 Findings Issue 4: Achievement of Expected Outcomes
• 7.0 Findings Issue 5: Demonstration of Efficiency and Economy
• 8.0 Conclusions and Recommendations
• Annex A: Fertilizer Program Results Chains – Annotated Observations
• Annex B: Evaluation Matrix
• Annex C: Bibliography
• Annex D: Interview Questions
• Annex E: Case Study on Fertilizer Risks

Abbreviations and Glossary

Abbreviations

AAFC – Agriculture and Agri Food Canada
CFIA – Canadian Food Inspection Agency
CFPF – Canadian Fertilizer Products Forum
ESOC – Emerging Substance of Concern
g – gram
IMIT – Information Management / Information Technology
IT – Information Technology
L – Litre
ng – nanogram
NPK – Nitrogen, Potassium, Phosphate
PMRA – Pest Management Regulatory Agency
SA – Sub Activity
TS – Total Solids

Glossary

Fertilizer: Any substance or mixture of substances, containing nitrogen, phosphorus, potassium or other plant food, manufactured, sold or represented for use as a plant nutrient

Dioxins and Furans: Common names for toxic chemicals that are found in very small amounts in the environment, including air, water and soil. As a result of their presence in the environment, they are also present in some foods

Inoculant: A micro-organism which is introduced into the soil to improve growth of crops

Micronutrients: The nutrients needed in small amounts by plants: iron, manganese, zinc, copper, molybdenum, boron and chlorine

Soil amendment: Any material added to a soil to improve its physical properties, such as water retention, permeability, water infiltration, drainage, aeration and structure

Supplement: Any substance or mixture of substances, other than a fertilizer, that is manufactured, sold or represented for use in the improvement of the physical condition of soils or to aid plant growth or crop yields

Executive Summary

The evaluation of the Fertilizer Program was identified as a priority in the 2012 Evaluation Plan. Performing this evaluation during the 2012-13 fiscal year allowed the Evaluation Directorate to provide observations in support of the Canadian Food Inspection Agency's (CFIA's) review of the Program design and Fertilizers Regulations, as part of its regulatory modernization initiative.

The study focused on the issues of relevance and performance with regard to the safety and labelling accuracy goals of the Fertilizer Program.

The Fertilizer Program continues to be relevant to the needs of both Canadians and the marketplace in terms of promoting safety and labelling accuracy. The Program aligns with both Government of Canada priorities and the priorities of the CFIA. For the most part, CFIA roles are well aligned with other government departments. There have been some challenges coordinating with other groups on dual property products (e.g. supplement and pesticide), and with provinces in terms of the marketplace status of waste-derived products (e.g. products sold or distributed using other means). Some challenges also exist with regard to internal CFIA coordination among policy, program, science, monitoring, and enforcement functions, which in turn causes a gap in integrated risk management.

Given its context and conditions, the Fertilizer Program has been very successful in engaging stakeholders and recruiting them to assist the Program in its transition away from an efficacy focus and prescriptive requirements towards outcome-based approaches and tiered safety-focused orientation (risk-based model).

The strong engagement of sector stakeholders has both enabled a speedy policy change and helped to reduce registration backlogs – in spite of structural irritants such as information management and information technology limitations, and increasing product complexity.

Gaps exist with regard to inspection, analytical testing, and enforcement (i.e. sanctions such as detention or prosecution), in terms of both executing desired sampling and frequency of verifications and product coverage, and processes or mechanisms to allow the Fertilizer Program to hone in on higher risk areas.

Recommendation

1) The Fertilizer Program should implement a mechanism to align its pre-market assessment and marketplace monitoring activities (inspection and analytical testing) with areas that pose the greatest risk to Canada's food, plants and animals and the environment.

• Emerging risks should be identified proactively to allow adequate risk mitigation while enabling flexibility in the regulatory framework and reducing administrative burden whenever possible and appropriate.

1.0 Introduction

1.1 Background

The Fertilizer Program is located under program activity 3, Plant Resources Program, of the CFIA's Program Alignment Architecture which links to the overall strategic objective of having a safe and accessible food supply and plant and animal resource base.

The objective of the CFIA's Fertilizer Program is to help ensure that fertilizer and supplement products imported or sold in Canada are safe (with respect to human, plant, animal health and the environment), efficacious for the intended purpose, and properly labelled according to the Fertilizer Act and Regulations. The Act covers products sold for home and garden applications, and for commercial and agricultural uses. The CFIA regulates all fertilizer products and supplements that are imported or sold in Canada. Provincial and municipal authorities regulate the manufacture, transport, use and disposal of fertilizer and supplements products. The CFIA works in conjunction with provincial and municipal authorities to help ensure that fertilizer products and supplements meet the highest standards.

The activities of the Program have been categorized into four key areas: Communications, Assessment and Registration, Verification and Compliance, and Interdepartmental and International Cooperation and Standard Setting. These are displayed in a set of results chains in Annex A, introduced by the Program's logic model.

Registration and pre-market assessment are required for some fertilizer and supplement products before they are imported into or sold in Canada. Some products are exempt from registration under the Fertilizers Act; however, once in the marketplace they must still meet prescribed standards. The CFIA also receives requests from companies that manufacture or import fertilizers and supplements that are exempt from registration to perform a pre-market assessment to ensure their products meet regulatory requirements. These assessments are voluntary and conducted by the CFIA free of charge.

During 2011-2012, the Fertilizer Program processed 827 product submissions, 417 of these submissions related to product registrations, field trial research authorizations, label reviews and data reviews. The other 410 submissions processed included responses to stakeholder complaints, inspector requests and inquiries from regulated parties as to the regulatory and import requirements for fertilizers and supplements.^Footnote 1

The larger numbers between 2008 and 2010 reflect a clearing of backlogged files. During this period, the Crop Inputs Division (Fertilizer Section and Fertilizer Safety Office) was fully staffed and the priority was to eliminate the backlog of submissions that had accumulated over a number of years. The increase in volume during this period can also be partially attributed to the implementation of formalized tracking for non-registration submission types, such as stakeholder inquiries and importer requests. In addition, a regulatory amendment that changed the registration period from a maximum of two years to a full three years was made in April 2009, which further explains the subsequent decrease in submissions (that is, re-registrations) following the 2008 to 2010 period. Service delivery standards were also established at this time for completing registrations.

A future increase in the number of submissions is possible, as the burden of substantiating product efficacy is no longer required. This requirement presented a deterrent to some because of the time and cost involved, which likely reduced the variety of products in the market (See Section 6.5).

As part of the changes the Agency is implementing through its regulatory modernization agenda, a number of additional product types are proposed for exemption from registration, thus reducing regulatory burden and the need for pre-market assessment of well established and safe products.

To help ensure its proficiency in delivering its mandate, the Fertilizer Program engages in program design, policy development and standard setting activities. By updating program design and developing new policies and standards, the Fertilizer Program aims to ensure that its actions reflect current science and are appropriately calibrated to risk. Furthermore, the Fertilizer Program participates in workshops and collaborates with industry, multiple levels of government including other federal government departments, and international groups.

Modernization of the Fertilizer program was initiated as early as 2006 with a more recent comprehensive review of the Regulations as part of the Agency transformational agenda and regulatory modernization. The initial phases of this process resulted in the formation of a permanent consultative stakeholder body – the Canadian Fertilizer Products Forum. The current review focuses on a shift towards more risk-based approaches and outcome-based regulations whenever appropriate or applicable. It aims to lessen regulatory burden on products that are deemed safe and have a well-established history of use and to remove prescriptive provisions from the regulations, thus providing for greater flexibility and less red tape on the regulated sector.

Some of the work completed so far under the modernization agenda for the Fertilizer Program includes:

• A review and amendment (in consultation with the CFPF) of Schedule II in the Fertilizer Regulations, related to names, definitions and compositional criteria;
• Implementation of service delivery standards for file review
• Streamlining policies and administrative requirements
• Implementation of a provisional registration model, to reduce efficacy data requirements in preparation for its removal from the regulations, and;
• Collaborative work with the United States on potential rule and criteria harmonization.

Significant additional changes to program administration are being planned as part of regulatory renewal and modernization. These include:

• Amend definitions for product types
• Clarify general exemptions
• Revise exemptions from registration
• Streamline labelling requirements
• Simplify submission format
• Extend the registration period from 3 to 5 years and re-define major and minor amendments
• Implement tiered registration model

1.2 Distribution of Fertilizer Program activities among CFIA Branches, Authorities and Resources

Key activities are distributed across three CFIA branches as follows:

Policy and Programs Branch

• Product assessment: Processing of industry submissions and conducting evaluations related to safety, efficacy and labelling
• Product registration: Certain products require registration with the CFIA before they can be sold in or imported into Canada
• Program design: The efforts of the Fertilizer Program are calibrated based on performance history and areas of risk
• Policy development: Policies, manuals, and procedures are reviewed and updated based on the best available science
• Regulatory development: Regulatory frameworks are reviewed and amendments are made
• Planning and coordination: This includes planning activities related to human resources, finance and staffing, as well as responding to corporate requests for information
• Standard setting: Standards are reviewed and updated based on the best available science and trade information
• Advice and support: Advice and training are provided to CFIA employees as well as external stakeholders. The Program also responds to requests from senior management
• Stakeholder consultations: National stakeholder organizations are consulted and engaged in policy development and review, program re-design, and modernization
• Functional direction and quota setting for marketplace monitoring
• Liaison with other government departments, federal, provincial, territorial and international bodies

Operations Branch

• Inspections and compliance: Conducting product inspections, taking samples for analysis, conducting surveys, registration verification, label reviews and research trial inspections
• Import control: This includes inspection of facilities and shipments plus the review of documents
• Enforcement: Use of a variety of enforcement tools such as notification of non-compliance, warning letters, detention or seizure of product, formal meetings with contravening parties, cancellation of registration, and, in cases of severe or repeated non-compliance, prosecution
• Client consultation: Technical advice is provided to stakeholders and input is provided when inspection issues arise
• Outreach and education: Attending industry and consumer events and conducting client consultation
• Sampling: Gathering samples and completing the required documentation

Science Branch

• Regulatory research and test development: Performing research to support the development of standards and policies based on best science and the development of new methods including collaborative partnerships
• Diagnostic services: Laboratory tests for certain nutrients, ingredients and contaminants

The Fertilizer Program is part of the Field Crops and Inputs Division of the CFIA which reports to the Plant Health and Biosecurity Directorate. These groups within the CFIA report to the Vice President, Policy and Programs. As outlined above, both Operations and Science branches also play key roles.

The legislative and regulatory authority for the CFIA's Fertilizer Program comes from the Fertilizers Act and Regulations. At the time of the evaluation study, the Fertilizers Regulations were under review as part of the Regulation Modernization Initiative.

The tables below show the amount of actual spending and staff ("Full Time Equivalents" or FTEs) for the CFIA's fertilizer activities, without any indirect costs.^Footnote 2

1.3 Key Considerations^Footnote 3

1.3.1 Broad Context: A changing risk environment for fertilizer use

The literature and document review, interviews, and consultations indicated important factors which constitute the broad risk environment in terms of influencing both the inherent risks of fertilizer use itself, and the behaviours of all marketplace actors and governors. (See Annex E for an overview of risks identified by the evaluation). The contextual factors outlined here are critical to understanding the findings, as presented starting at Section 3. As noted in the Treasury Board's guide on Theory-Based Approaches to Evaluation: "These factors are often essential in making causal inferences and need to be part of the evaluation design."^Footnote 4 The most important of these factors are listed below:

• Innovation creates complexity in products (e.g., microbial inoculants, nanomaterials, etc.)
  Complexity in turn creates uncertainty^Footnote 5, a lack of predictability and difficulties in predicting risks as well as creating an increase in risk potential. The factor affects all stakeholders. This includes the fact that the Science Branch has some limitations in its testing capacity, particularly for new and innovative products, including nanomaterials.
• Global food supply demand and pressures on agricultural sectors to increase productivity and yields
  The narrowing of the productivity advantage which has historically enabled Canadian farmers to remain competitive, and the disadvantage in terms of labour costs that Canadian farmers face when competing against many countries, has increased pressure to invest in innovative and relatively potent fertilizers and supplements.
• Use of non-conventional products like supplements to maximize yields
  In the same vein as the competitive pressures and innovation potential noted above, non-conventional products such as supplements further complicate the fertilizer use marketplace and risk profile.
• Multiple players in the marketplace, and highly diversified sector and distribution chains
  The fertilizer marketplace consists of large and small, sophisticated and unsophisticated actors (suppliers range from large multinational organizations to single family organic producers), and a diverse set of users ranging from agricultural production enterprises (such as farms), to recreational (such as golf courses) and home applications. Furthermore, products are typically distributed in myriad ways, including formally organized supply chains and direct sales. An example is products imported for the importer's own use, a practice which is currently exempt from registration.
• Community norms for the use of fertilizer (for example, attitudes toward environmental sustainability in urban versus rural areas)
  Attitudes and community norms with regard to the use of fertilizers and their regulation vary extensively, as do producers and user groups. This creates different pressures on governing bodies to regulate for safety. In the case of fertilizers, this may explain the heavy emphasis of Program attention on home and garden fertilizers which represent a small portion of the volume of fertilizer use, but are a strongly represented segment within bodies such as the working groups of the Canadian Fertilizer Products Forum (CFPF). The Forum is a stakeholder-led initiative launched in 2006 to improve the regulatory system for fertilizers and supplements by building a national consensus across all stakeholders.
• Pressure to use waste-derived products represents a major risk area (for example, reducing landfills and value in biosolids, etc.)
  Municipalities and provinces across Canada have taken steps to divert waste materials from landfills and promote beneficial uses such as application to land as sources of recyclable nutrients and organic matter. This in turn creates pressure (cost reduction opportunities) to use waste products as fertilizers and supplements. Unfortunately, experts suggest that this creates a heightened risk of contamination to water supplies and to food – often due to the interaction of waste materials with other materials and processes (see emerging contaminants below and Annex E).
• Emerging contaminants of concern (pathogens, persistent organic pollutants, dioxins and furans, pharmaceutical residues, etc.)
  As noted, competitive pressures, innovation, pressures to find alternative uses for waste (both industrial and municipal) and the fact that waste products may increasingly contain pathogens, pharmaceutical residues, etc., combine to create both an increased risk and a more uncertain risk profile for fertilizer use. There is also a lack of scientific research on some of the potential contaminants to assist regulators in determining safety requirements:
  "…there is a notable lack of a database regarding the presence in Canadian biosolids of a certain class of wastewater constituents termed emerging substances of concern (ESOCs). These ESOCs include an array of pharmaceuticals, personal care products, brominated flame retardants and industrial contaminants (such as plasticizers and surfactants). At the present time, the ability to detect ESOCs in the biosolids at the ng/L or ng/g TS (dry wt) exceeds the understanding of the scientific and public communities of any potential risk associated with their detected presence. . …While there is documentation in the literature on ESOCs in biosolids, especially in America and Europe, no focused study has been completed yet on a complete and recent inventory of ESOCs in Canadian biosolids."^Footnote 6
• Jurisdictional issues (Interagency, F/P/T, and International)
  Commercially sold fertilizer products fall clearly within the mandate of the CFIA. As well, federal and provincial/territorial environment and health ministries may have interest in and exercise regulatory authority over some fertilizer use. It is the provincial or territorial governments, rather than the CFIA, that exercise regulatory authority over fertilizers that are not sold (i.e. when they are used directly by the producer or given away). In addition, while fertilizer-pesticide combinations are regulated by both Health Canada's Pest Management Regulatory Agency (PMRA) and the CFIA, both requiring registrations, there is no mechanism for a single joint registration, although discussions between the two agencies have occurred on and off over the years. International issues may arise regarding restrictions and balance of regulatory burden, as well as harmonization of standards. (Such harmonization also applies to provinces.) All of these jurisdictional elements affect the risk profile for fertilizer use.

1.3.2 Agency Context: Key authorities, management processes and relationships

The review of literature and of Fertilizer Program documents, and previous regulatory evaluation work^Footnote 7, interviews and consultations, suggested that there are important factors which constitute the conditions at the CFIA affecting the Fertilizer Program's ability to achieve its goals. The most important of these are listed below:

• Authorities and the legal and regulatory framework
  • Legislative and regulatory authorities and capacity can expand or limit the bounds within which the program operates. For example, the lack of accreditation or licensing authorities limits alternative service delivery, and thus the opportunity to charge fees and recoup costs for services. Also, limited enforcement powers (such as lack of authority to issue Administrative Monetary Penalties^Footnote 8) may affect deterrence.
  • Prescriptive as opposed to outcome based regulations: On the one hand, prescriptive regulatory requirements in some areas may add burden on the regulated parties. On the other hand, performance- or outcome-based requirements can create ambiguity in interpretation, in turn creating expensive tests or justifications that may go beyond the capacity of smaller companies. For example, the term "misleading" for labels reflects a core CFIA objective for labels not to mislead the consumer, but there is ambiguity and uncertainty as to what constitutes "misleading". (Section 19 (2) states that "A label …shall not have printed thereon (a) any incorrect or misleading information or mark; or (b) a brand or name that would tend to deceive or mislead a purchaser with respect to the composition or utility of the product to which the label relates." Determining what might be misleading about the wording on a label, let alone a mark, is inherently subjective.^Footnote 9)
  • The nature of regulatory instruments: Stakeholders have noted that instruments have not changed or been modernized for the Fertilizer Program, and are based on antiquated regulations that now require a lengthy regulatory amendment process.
• Management functions and processes^Footnote 10
  • Human resource management systems need to recruit, educate, retain and generally support human resources to enable the program to work. Recent cuts have been identified as potentially problematic for the continued capacity of the Program to meet service delivery standards, particularly if there is an increase in requests for product registration, as has been projected by many industry representatives (See Section 6.5: Assessment and Registration). The seasonal nature of the fertilizer industry, with most products sold in the Spring, poses operational restrictions in that samples and inspections are concentrated during that period as opposed to distributed throughout the year, hence making it challenging to collect sufficient samples. Furthermore, fertilizer inspectors are cross-utilized among different programs and commodities, further compounding the operational resource allocation issue.
  • Financial budgeting, allocation and management systems must support the appropriate, timely and sufficient allocation of financial resources to be able to fund all key program functions. Under sampling due to resource constraints has limited the value of compliance performance data because of the unrepresentativeness of small samples (see Section 5.3).
  • Information management and information technology support systems need to enable program managers and officers to clearly, completely and swiftly generate, store, manipulate and communicate information to perform key program functions. The lack of e-registration tools creates an administrative burden both on the regulated parties and the Agency (see Section 6.3)
  • Planning, measurement, reporting and management functions should support the consistent and focused alignment of program processes and mechanisms to key objectives (i.e. enabling a safe and accurately informed marketplace).
• Regulatory processes and relationships
  • There is a need for capacity and mechanisms to effectively:
  • engage stakeholder communities in the regulatory process;
  • communicate processes concerning legitimacy of the regulations;
  • integrate the various regulatory requirements;
  • register products within service delivery standards;
  • screen processes to identify risks and potential offenders;
  • inspect (monitor and verify) products in the marketplace and strengthen mechanisms to deter non-compliance;
  • conduct analytical testing to help ensure safety of fertilizers and supplements (method development and validation, regulatory research, science capacity and expertise), and;
  • enforce (detain products and prosecute) to deter non-compliance.

The following sections address core evaluation issues with a view to examining Fertilizer Program relevance and performance in light of these key contextual factors. See Annex B for the Evaluation Matrix, included in the Evaluation Framework which was the planning document reviewed by the Evaluation Advisory Committee at the study's outset.

2.0 Evaluation Methodology and Limitations

2.1 Terms of Reference

The evaluation covered the period of 2007-12. For the purposes of this evaluation, the "Program" under evaluation consisted of the activities undertaken by the key Fertilizer Program areas including Policy and Programs, Operations and Science branches. As outlined in the evaluation framework (See Annex B), Treasury Board's five core evaluation questions on relevance and performance were used to frame the inquiry, following the Directive on the Evaluation Function, as well as the Standard on Evaluation.

In addition to answering the core evaluation questions, the evaluation was intended to:

• identify key information and service delivery gaps and issues prior to the modernization and as part of the Fertilizer Program core business;
• document the extent to which performance measurement is systematic, and identify structures across the Agency with a stake in the Fertilizer Program; and
• document the outcomes (with whom and in what conditions) that have been achieved as a result of the Fertilizer Program activities. (Note that as a result of Budget 2012 adjustments, the focus of the evaluation was on registration, label accuracy and safety, and not on efficacy or quality verification activities.)

The evaluation of the Fertilizer Program was identified as a priority in the 2012 Evaluation Plan to be completed by March 2013. Performing this evaluation during the 2012-13 fiscal year allowed the Evaluation Directorate to provide observations which may help inform changes to the program design and the Fertilizers Regulations.

The evaluation framework document was developed in the following context:

• Short timeframe for study – designed to limit scope to certain key areas.
• Build on existing good information showing practices and sector relations.
• Focused on the need to understand: How (and to what extent) the Fertilizer Program works for whom and in what conditions (see Section 2.2).
• Retrospective – but limited to 'go forward' areas of service (i.e. efficacy not directly reviewed as it was eliminated under Budget 2012).
• Designed around the five core evaluation issues.
  • Continued need for the program (Relevance)
  • Alignment with government priorities (Relevance)
  • Alignment with federal roles and responsibilities (Relevance)
  • Achievement of expected outcomes (Performance)
  • Demonstration of efficiency and economy (Performance)

An Advisory Committee was established at the outset, with the Head of Evaluation and the Executive Director of the Fertilizer Program area as co-chairs, with senior representatives from Science, Operations, Corporate Management and Public Affairs branches. In addition, one external member was included from the Evaluation unit at the Canadian Nuclear Safety Commission to increase the objectivity of the project. The Committee was responsible for providing strategic and technical advice to the Head of Evaluation and to identify important evaluation issues. This included advising on the scoping, methodologies and data sources, and reviewing, assessing and providing comments on deliverables and presenting to senior management for discussion (e.g., evaluation framework, findings and report).

The preliminary research and development of the evaluation framework and results chains and logic model were completed at the end of summer 2012, with presentation to the Advisory Committee in October. The core of the data collection occurred in October and November, with the report written in December 2012.

2.2 Evaluation Approach

The approach to this evaluation involved a combination of "realistic" evaluation and "contribution analysis". This approach is suggested for cases involving complicated and complex interventions (such as a regulatory intervention) where conventional designs are not feasible. For example, it is virtually impossible to consider an experimental design approach in a regulatory context. This approach is supported by a recently published Treasury Board guide^Footnote 11. The approach involved working closely with program management to establish key theories of change for its core functions, then testing them with key stakeholders through consultations and documentary and interview evidence (see Annex A). More than 25 representatives of five industry associations, seven fertilizer companies, CFIA Program, Operations and Science staff plus Health Canada and one provincial ministry were consulted. Due to the 'generative' nature of the approach, many stakeholders were consulted multiple times. In addition, and key to the data collection was a specialized (first of its kind) group consultation with over 80 government and industry sector stakeholders, held October 17, 2012, to validate the Fertilizer Program service 'value propositions' and to generate observations regarding important contextual factors. This consultation was used to guide follow-up consultations, interviews and document reviews.

The approach used evidence to test the hypotheses about what outcomes were produced by what mechanisms in what contexts. Theories of change involved the development of results logic and key results chains (see Annex A) for the intervention showing how the specific intervention was intended to work and the assumptions behind the theory. The theories of change (as represented by the results chains shown in Annex A) were developed through consultations on the basis of a range of stakeholders' views and information sources.

2.3 Data Collection

The evaluation team worked with the Fertilizer Program to identify the initial set of interviewees, and then added to that list with intelligence from the interviews, document and literature reviews. Multiple lines of evidence were incorporated into the evaluation analysis whenever possible.

The selection of industry (regulated party) interviewees, or "sampling", was based on participation in the Canadian Fertilizer Products Forum, as this is the most representative fertilizer industry body in Canada. The CFPF has five working groups addressing issues from safety to labelling. Members from each were interviewed, including four of the chairs. The evaluation team ensured that key industry representatives were interviewed to help mitigate the relatively low number (27) of interviewees overall (industry 12; CFIA 13; PMRA 1; Ontario provincial government 1). However, the key mitigation and strength of the methodology lies in the validation of the results chains with over 80 participants at the October 17 workshop, including over 60 industry participants.

There are, reportedly, a number of producers and sellers of fertilizer products, understood to be marginal players that remain outside the registration system, that is, are intentionally non-compliant. This remains a known unknown. It would require a very large study, much larger than this evaluation, to attempt to identify the number and type of these marginal players. It is, of course, highly unlikely that those avoiding compliance would prove forthcoming for interview purposes.

A key data source was the on-line industry discussions and documents in the Collaborase system which is used by all working groups for regulatory modernization. Collaborase is a non-CFIA online network that enables discussion and the distribution of documents and surveys.^Footnote 12

Many documents and interviews identified risks of fertilizer and supplement product use. A case study was written that brought all of this information together in one location in the report, namely Annex E.

Following a theory-based approach, this report sets out considerations for both the fertilizer use context and the Agency context and then organizes preliminary findings by the issues laid out in the September 2012 evaluation framework.

2.4 Evaluation Strengths and Challenges

The strength of realistic contribution analysis was the systematic, structured and disciplined approach that led to a greater understanding of results chains related to the Fertilizer Program and different contexts that affect its delivery.

The evaluation approach accounted for the following challenges:

• The complex nature of the Fertilizer Program regulatory and other activities (timelines, attribution issues, multiple actors, dynamic product changes, etc.). As outlined in Section 1.3.1 above, the context is a major focus of theory-based approaches;
• The links between the various Agency program areas (science, inspection, policy etc.);
• The somewhat complicated delivery mechanisms (e.g. multiple agencies, multiple groups, multiple jurisdictions, intermediaries and regulated parties). Again, the evaluations focus on context addresses these in section 1.3, and;
• The study has been focused on the last five years; however, the scope has not included efficacy assurance support, since this service has been discontinued for the future as a result of recent budget decisions. This means that an assessment of the relative costs, the relationships and perceived benefits of the Fertilizer Program has taken place within a certain context (i.e. the CFIA has played a strong marketplace support role) that will be discontinued in the future. Hence, speculations about how certain functions and relationships will continue going forward needs to be recognized as subject to the significantly changed context of the CFIA Fertilizer Program now limited to playing a role of safety verification and monitoring of accuracy in labelling.

There were two further key challenges or limitations found as part of the conduct of this study. First, the time lines were challenging because of the regulatory modernization agenda which required findings by December, although the Evaluation Advisory Committee did not approve the Framework, and thus data collection methods, until October. Second, gaps in monitoring coverage were found to exist such that true compliance levels are essentially impossible to estimate (see Section 6.5). These limitations created a significant barrier to the study's ability to conclude on program impacts – and therefore influenced the approach taken to data collection and analysis.

The realistic contribution analysis approach being adopted here was designed to maximize knowledge from previous learning (for example, what factors have been key in shifting regulatory programs to focus more on performance-based or outcomes-based standards rather than prescriptive ones?) and from other work before imposing original collection burden (e.g., see Section 6.2. Furthermore, the rigorous structuring of results logic allowed the team to hone in on relevant parts of the implementation change process, matching indicators of progress against predicted patterns, thus avoiding the need for a large survey of (mostly) partially informed participants.

Note that the approach allows for a constant honing of results expectations, findings and observations. This process enabled the building of generative learning throughout the project, and was not dependent on the weakest/slowest link to present significant findings. That is, given the short timeframes, the approach generated findings early enough to get a head start on final reporting before data collection was fully completed. This made the study useful to senior policy makers, planners and managers in helping the program go through its modernization process. The process of validating findings iteratively, through the interview and follow-up consultation process, led to the sharing of findings with the Program as the evaluation progressed, because the program's management was repeatedly consulted more than any other interviewee.

3.0 Findings Issue 1: Continued Need for Program

3.1 Why were Fertilizer regulations and the Fertilizer Program created? (What needs have the Fertilizer Program addressed?) Why was the Fertilizer Program initiated at CFIA?

As per the broad contextual factors noted in Section 1.3, literature, documents and interviews suggested that the need for the Fertilizer Program is clear, as it directly addresses the Agency's mandate of safe foods, plants, and animals. There is historical connection to Agriculture and Agri-Food Canada (AAFC), where the Program existed for decades prior to the creation of the CFIA in 1997. The expertise was maintained in the move to CFIA from AAFC, as it continued to be aligned with the Agency's mandate.

3.2 Who is served by the Fertilizer Program?

The Fertilizer Program stakeholders fall into two broad groups:

1. General public is served by the Program's protection of the safety and integrity of foods, plants and animals.
2. Agriculture sector producers, farmers, distributors, retailers and specialty market users of fertilizers and supplements (including horticultural producers, nursery owners, commercial greenhouse operators, golf courses, landscaping firms, and home and garden users) are served with improved marketplace certainty and integrity, and reduced product risks while maintaining innovation and 'balanced' regulatory burden (for example, minimal administrative requirements for registration of products, etc.).

3.3 What has been the reaction to date from regulated parties? (Note different segments as well as groups.)

A review of correspondence (for example, the online network Collaborase), interviews, and the evaluation team's observations from a 2012 two-day conference on fertilizer regulatory changes, indicates a generally positive reaction to direct Fertilizer Program interactions.

Interviews and participants of the Canadian Fertilizer Products Forum (CFPF) conference indicated that there has been exceptional engagement between the regulated parties and the CFIA (a model for others) concerning regulatory consultations and development. Respondents noted that there has been significant improvement over the past eight years in the interactions with the regulated parties. It was noted that there has been an effective and positive institutionalization of dialogue with industry through the formation of the CFPF as a permanent consultative body. Interviewees suggested that there has been a fundamental shift in the relationship with the stakeholder community following elimination of the file submission backlogs, creation of the CFPF and implementation of service delivery standards.

Some concerns were expressed by interviewees regarding specific registration requirements, marketplace monitoring and compliance verification practices, and scope and product coverage (see Performance issues in Section 6.0).

4.0 Findings Issue 2: Alignment with Government Priorities

4.1 Are there differences in the understanding of the theory of how the Fertilizer Program activities are supposed to work?

Good consistency was found in understanding across groups with regard to the Fertilizer Program alignment with food safety goals. This may be attributed, in part, to good relationships between the Program and sector as noted above.

4.2 Under what assumptions and circumstances do the Fertilizer Program activities align with a) Federal government priorities (for example. the Cabinet Directive on Regulatory Management), and b) the strategic outcomes of the Agency (that is, Safe Foods, Plants, and Animals).

Document reviews, interviews, and conference observations indicate a strong alignment with the Cabinet Directive on Regulatory Management. For example, the Fertilizer Program has clearly adopted a strong consultative approach to regulatory management. Efforts to streamline processes have been evident in both program actions and in results (such as the reduced backlog), and documents show a strong alignment with food safety priorities, as well as protection of the plant resource base and health of animals.

5.0 Findings Issue 3: Alignment with Federal Roles and Responsibilities

5.1 What is the understanding of roles of key actors in application of the Fertilizer Program?

Interviews showed a general acceptance of the Fertilizer Program's roles by key actors. Some concern was expressed at the inspection level regarding relative program priority vis-à-vis other regulatory activities and emergency responses. Each year there are urgent and unplanned requests for inspectors to take samples to address such things as a food recall. These unplanned samples reduce the number of samples inspectors can take in other areas, but the choice of which program area will have a reduction in its samples is variable each year. Many interviewees noted that the Fertilizer Program is seen as a low priority in the Agency relative to other programs and thus suffers more reductions in sampling due to unplanned and urgent requirements. (See also Section 5.3) Interviews suggested that there have also been some gaps at the inspection level in terms of awareness of the requirements and requirement interpretations (See Section 6.5: Verification and Compliance).

5.2 Are there differences in stakeholder understanding and interpretations of those roles? Are we talking about differences within certain stakeholder groups or between types of stakeholders?

No major discrepancies were discovered in terms of the interpretation of the Fertilizer Program roles in interviews or correspondence reviewed. This consistency may in part be attributable to the Program's strong engagement processes (see Section 6.5).

Interviews suggested that there have been some challenges in coordinating roles and responsibilities between PMRA at Health Canada and CFIA (for example, developing a single registration process for dual use products).

5.3 How well aligned are roles and responsibilities (within CFIA and among stakeholders)?

The roles alignment between the CFIA and among stakeholders appears appropriate. Among CFIA stakeholders, some differences in perspective were noted particularly among CFIA employees within the Program delivery chain. There is concern that the planning process for sampling (setting a sample quota) will result in a decreasing amount of sampling results to the Fertilizer Program caused by basing the sampling quotas on the level of sampling completed in the previous two years i.e. if delivery is low this will cause quotas to be decreased. The determination of the quota, or number of samples and inspection activities to be taken, has, in recent years, been based on the previous year's numbers, in consideration of the inspection resources available, something that has been referred to as "planning to capacities".

There has been some under-delivery of inspection quota when measured against the target of carrying out 90% of the planned fertilizer and supplement product inspections as outlined in the CFIA's 2011-12 Performance Measurement Framework. Furthermore, in some cases, inspectors have focused on "easy access" products at the retail level, as opposed to higher risk areas. This may be part of the reason that between 2009 and 2012 the number of samples taken by inspectors for the testing of nitrogen, phosphorous and potassium (NPK which is the most common and relatively low risk fertilizer blend) has been above 100% for the last few years. There is easy access to many of these products at home and garden retail outlets and at manufacturers and blenders. However, the sampling of metal content was 81% of its quota for the same time period^Footnote 13, as shown in Table 1 below. This is significant because metal content includes heavy metals, which is recognized as one of the biggest risks in fertilizers (see Annex E). As noted in Section 1.3.2, the under-representativeness of the sampling is further compounded by the challenge to operational resources due to the seasonal nature of the sector, with the majority of products available in the marketplace in the Spring.

Table 1

Some samples collected are not tested, usually because delays in shipment to the laboratory which undermines the integrity of the sample (potential contamination). This rejection of samples has amounted to an average of 1.5% of samples over the period 2008-12.

6.0 Findings Issue 4: Achievement of Expected Outcomes

6.1 Does the planned implementation of the Fertilizer Program initiatives tend to bend in actual usage? How and why?

Document reviews, interviews and group consultations suggested that processes generally appear to be much improved over the past several years. Some processes are exemplary; however, some concerns remain with regard to others. Findings are summarized by the Agency contextual factors below (described in Section 1.3.2).

6.2 How do the Fertilizer Program initiatives and regulatory activities work in different contexts and circumstances?

A literature review, document review, the knowledge gained from previous CFIA studies (for example, Evaluation of Administrative Monetary Penalties^Footnote 16) and interviews suggested that the Fertilizer Program regulatory processes work better when:

• There is clarity in the legislative and regulatory requirements, (for example, challenges associated with the interpretation of the "misleading" information clause – see Section 1.3.2).
• Conditions for effective inspections are facilitated (easy access to products, proper guidance, training, and capacity).
• Effective analytical testing is available for monitoring and verification purposes (including method development and validation).
• There are fewer stakeholders (for example, easier to regulate one producer selling directly to customers vs. complex production chains with varied provincial and municipal rules). This also applies to internal stakeholders – the fewer the groups that need to be involved, the better the process seems to work.
• The regulated community is engaged and willing to be part of a stable marketplace delivering medium to higher quality products. The problem players, prone to recurring violation, are often the so-called 'marginal actors' (for example, some waste product producers have proven problematic, as noted below).
• Emerging risk areas and future dangers are well understood by all stakeholders (foresight, environmental scanning) and properly managed (for example, use of waste derived materials carrying damaging contaminants such as serious food-borne pathogens, which led to a 2006 California spinach recall that left three dead).

The current regulatory exemption for the importation of fertilizers that are not for sale in Canada and for the importer's own use presents a potential risk to human, plant and animal health and the environment because there is no oversight concerning the types of products, the quantity applied or the location. This is expected to be addressed during the current regulatory modernization process.

Federal versus provincial oversight creates gaps when product proponents vary their answers as to whether they are selling the product to avoid either jurisdiction (federal vs. provincial). This is especially prevalent with waste derived materials. According to program leaders and other key stakeholders, this may be escaping surveillance. While there is clarity in the roles (provinces purview if fertilizers are given away, but federal if the same product is sold), that is, no apparent ambiguity, there appear to be some gaps in coordination to monitor those who may be attempting to avoid compliance with the regulations. (See also Section 6.5: Interagency Cooperation and Standards Setting.)

6.3 Do the Fertilizer Program initiatives achieve expected results with certain individuals, interpersonal relations, sectors and situations?

Interviews and consultations (such as the October 17, 2012, conference session) indicated the following:

• Some regulated parties seeking information on product registration found the CFIA Program website unhelpful. The organization of information was said to be non-intuitive. This limits the degree to which the desired outcome of "Increased awareness of fertilizer policies, regulations and legislation among stakeholders" could be met. See Annex A, including each of the results chains, where awareness is integral to the first chain on Communications and a foundation for each of the others.
• The lack of e-certification tools creates administrative burden on regulated parties. PMRA was often cited by regulated parties as an example of similar government function with e-certification. The Agency's plan for improving the Fertilizer Program's IT tools has not yet been articulated. E-certification tools are expected to allow for speedier processing, and would allow for reduced service standard times. The Agency's 2011-12 Performance Measurement Framework identifies the meeting of service standards 90% of the time as one of the two outcome measures and targets (the other being compliance, as discussed below). There are a variety of service standards for the Fertilizer Program, ranging from 10 days for a first review of a submission for a minor amendment to a registered fertilizer to 200 days for the first review of a major amendment. All of these are provided on the Agency's website (Trade Memoranda T-4-122), where the intention to cut these times by more than half in some cases is planned as a result of the changes planned during the current modernization agenda. The overall rate of compliance to these service standards is 96% for the fiscal years 2009-10 and 2010-11.
• Registration requirements are unclear. No forms filled out to register products were processed without a requirement for additional information, correction or clarification. Nobody got it right the first time.
• The planned inspections of fertilizer products (inspection quota) and the actual sampling that results from it is not representative of the market share, or areas of highest risk. The lack of representativeness, as noted above in Sections 1.3.2 and 5.3, and below in Section 6.5: Verification and Compliance, is a major obstacle to the Agency's ability to understand what true compliance rates are. As noted previously, the two outcome indicators identified in the Agency's Performance Measurement Framework are the meeting of service standards and compliance of samples. When sampling is not representative of the market and risks then the degree of compliance is similarly unrepresentative. The rate of compliance was targeted in the 2011-12 PMF at 90% and the Program's results are roughly at this level, with an average of 89% compliant samples in 2009-10, 87% in 2010-11 and 90% in 2011-12, as detailed in Table 2 below.

Table 2

As noted in Section 5.3 and discussed in Annex E, heavy metals carry a significant risk, and they have also contributed to poor compliance figures, as seen in inspection program 6, though this includes tests for nutrient guarantees.

Examples of products found to be non-compliant include adulterated products, as noted in Annex E, such as imports of fertilizers augmented with melamine to increase nitrogen readings, and gravel imported as urea. Domestic non-compliant products typically have ingredient quantities that exceed safe limits.

Regarding the question of types of Program results for certain individuals, interpersonal relations, sectors and situations: No significant issues related to certain individuals or interpersonal relations were identified. However, as discussed in Section 6.5, there are significant differences among sectors and situations.

6.4 What are some key principles to guide the Fertilizer Program elements application?

Interviewees and a content analysis of correspondence suggested that the following key factors (principles) affect the Fertilizer Program success.

• Clarity of clauses and interpretations: Regulatory clauses should be stated more clearly (for example. "misleading labelling" should be either extensively clarified or eliminated). Explicit examples should be used for registrations.
• Communications to the operational (inspection) level via policies and interpretations: Guidelines for interpreting the "misleading" label clause is one example.
• Communications to regulated parties: The move to performance-based (also known as outcome-based) requirements will be a challenge to some of the smaller industry stakeholders for which CFIA has agreed to provide "non-binding models" for guidance (formerly called "safe harbours") to help them adjust to the removal of prescriptive requirements. The CFIA defines "non-binding models" as "a non-binding code of practice or model system that provides assurance to a regulated party that the outcome is being met." Larger companies usually have the capacity to determine the required process to reach required outcomes, whereas smaller ones may require more guidance to replace the prescriptions currently in the regulations.
• Risk-based orientation needs to guide all activities and the integration of functions. The removal of efficacy leaves safety as the only Fertilizer Program focus, thus suggesting a change to a completely risk-based orientation.

6.5 Integrity of the implementation chain – are the Fertilizer Program elements applied consistently and cumulatively? Why or why not? How can or should this be addressed?

The findings from a review of functional results chains conducted as part of interviews and the October 17, 2012 workshop with participants, as well as related document reviews are summarized below. (See Annex A for a graphic depiction of findings.)

• Communications, Engagement and Education
  • The consultation process for regulatory change has been exemplary. The process has been widely praised by both the regulated parties and the CFIA.
  • The link to CFPF has been a strong asset in quickly and effectively taking steps to modernize Fertilizer Regulations. The relationship has also appeared to help reduce the registration backlog. The open communication has allowed for greater understanding of requirements on both sides. In addition, anecdotal evidence suggests that this may in turn have encouraged the registration of innovative products in the Canadian marketplace which would not have been registered otherwise (See below under Assessment and Registration).
  • While communication with the key industry associations has been exemplary, there is strong demand for guidance in response to the Budget 2012 removal of efficacy. CFPF working groups were asked how CFIA could assist them to adapt to the regulatory changes and seven of the ten suggestions from the group discussing efficacy were for the provision of guidance by the CFIA on various means or examples of replacing the quality standard that the efficacy testing provided.^Footnote 17
  • High satisfaction with consultation processes.
• Assessment and Registration
  • Concerns were expressed about the adequacy of CFIA resources available to handle work associated with an expected increase in registrations caused by the elimination of efficacy testing requirements. Some industry representatives stated that more products will now be entering the market because efficacy requirements no longer deter some companies, which is said to have limited the number of products available in Canada. Members of each of the CFPF working groups identified the need to increase import controls because of a concern that non-efficacious and potentially dangerous products are both potentially damaging to the environment, plants, food and animals, and to the industry's reputation. Because of the belief that more products will now be entering the market now that they do not have to prove their efficacy, the concern over imports has increased.
  • Assessment and registration times have greatly improved in the last eight years. Some concerns remain regarding whether the balance of focus is right. For example, areas like NPK fertilizers which have a well-established history of safe use may need less focus, while higher risk areas like waste-derived products will need more.
  • Given the perceived challenges with preparing complete registration packages, the pre-submission consultation was suggested as a process that could be utilized more frequently to assist applicants. This simply involves a meeting between the applicant and a CFIA Program representative to ensure the applicant is familiar with all the requirements, processes and time lines for completing a registration.
  • Adherence and improvements to service delivery standards for file reviews are very important to the stakeholder community.
  • Larger commercial producers seem committed to complying with the regulations while some smaller specialized players reportedly avoid compliance where possible. The limited number of inspections in high risk, marginal areas of the market (for example, sewage processors and other non-conventional products), may encourage such marginal players to take their chances and not register. (Note that confirmation of the behaviours beyond those tracked in the Program was beyond the scope of this study.)
  • Push to divert waste from landfills by using it in land application as fertilizers and soil amendments^Footnote 18 may present safety risks.
• Verification and Compliance
  • Program communications to inspectors is fairly good, but there are some gaps in terms of inspectors not consistently knowing the latest regulatory interpretations. Incidents were noted by various regulated parties where inspectors were not clear on the most current interpretations of Fertilizer Regulations. This may also be addressed with improved inspector training.
  • Concerns were expressed regarding the capacity and quota sampling reductions as well as 'convenience' sampling. It can be an inconvenience to obtain one sample of sewage compared with the ease of taking samples from numerous retail garden products at one store.
  • A suggestion was made during interviews that the system under-represents (riskier) hard to reach stakeholder groups. This refers but is not limited to the sellers of waste-derived products.
  • Label reviews appear problematic. Stakeholders reported difficulty in interpreting the "misleading" clause (See Section 1.3.2). In some cases, label reviews were questioned concerning priority and effect such as whether this the best use of limited sampling capacity and the impact on safety. Note that most product detentions over the past three years resulted from label violations (see Section 6.1).
  • Concerns that emerging risks (e.g., microbial products, nanotechnology, waste-derived products etc.) may not be adequately covered in the bigger system.
• Interagency Cooperation and Standards Setting
  • Provincial policies vary in terms of perceived risks.
    • In 2010, the federal Minister of the Environment noted that "…The actual application of biosolids is, for the most part, regulated through provincial and territorial legislation. The roles and responsibilities of provincial/territorial government bodies, as well as the scope of their legislation, vary widely."^Footnote 19
    • For example, in some provinces, land application for untreated septage is permitted in certain circumstances (AB, MB and ON as at 2010), while in most it is not.^Footnote 20 Some provinces and municipalities have restricted fertilizer use. For example, Manitoba has banned fertilizers with more than 1% phosphorous on residential lawns, urban parks and golf course, with some exceptions, and the municipality of Magog, Quebec has prohibited some fertilizer use for lawns. The vast majority, however, have no such restrictions.
  • In addition, and as previously noted, there appears to be some risk that groups may "give different stories" to different regulators to avoid inspection (for example. telling CFIA inspectors that they are not selling a product, and telling provincial inspectors that they are selling it).

7.0 Findings Issue 5: Demonstration of Efficiency and Economy

7.1 How do the Fertilizer Program initiatives compare, in terms of what they deliver, to alternatives?

Alternatives were not identified, beyond the move to outcome-based regulations being undertaken as part of the regulatory modernization process, and the focus on safety versus efficacy already being implemented. The development of a quality standard, such as that used by New Zealand (Fertmark) was discussed during the October 2012 Workshop, but this was only exploratory with no clear consensus.

Interviews, process observations and content analysis of correspondence suggested that the Fertilizer Program is better than most at the policy (regulation) development level. One fertilizer industry association referred to the "warp speed" for regulatory change that the Program has been managing. Substantive regulatory amendments require a minimum of 18 months to two years^Footnote 21, and the Fertilizer Program is on track to overhaul the entire regulation package in a similar time frame, in collaboration with key regulated party representatives. As previously noted, PMRA was suggested as a model by some interviewees for its IMIT tools such as its e-registration platforms and processes. The CFIA currently has no e-registration capabilities for the Fertilizer Program.

7.2 Are there more cost effective ways of achieving the same results (for example. in combination with other instruments or groups)? Are there good practices in comparable contexts which could be adopted or adapted for use by the Fertilizer Program?

Note that a number of suggestions for specific efficiencies were made by consulted parties in sections 6.3, 6.4 and 6.5. In addition to these points, the following suggestions were made:

• As noted above, PMRA was identified as an example for product registration tools and processes.
• Interviewees suggested that there is an opportunity for service fee increases if service standards are maintained and improved. (Note that for registrations and updates the CFIA charges a maximum of $1,000 for the various related services, while PMRA's maximum is $100,000^Footnote 22). Industry representatives were consistent in their perception that an increase could be acceptable, depending on the quality of the service provided by the Program.
• As noted previously, products that contain both fertilizers and pesticides, or single ingredient products that function as both, currently require two separate registrations. There is an opportunity to develop a single registration process.

7.3 What would happen without the Fertilizer Program initiatives and activities?

Interviews suggested strong consensus that the Fertilizer Program is needed. There was a strong concern by industry that without the Program (recognizing that the regulations would need to be altered regarding registration by the CFIA), compliance would decrease in the sector due to marginal players undermining consumer confidence in the product and leading to increased safety and marketplace risks. The perception by the regulated parties interviewed was that the CFIA's monitoring role diminishes the ability of marginal players to put ineffective and unsafe products into the marketplace.

7.4 Regarding the Fertilizer Program registration backlog, what have been contributing factors and how have these been resolved?

Documentary evidence and interviews showed that the registration backlog has been resolved but that upcoming regulatory changes associated with the Budget 2012 decision (i.e. the elimination of efficacy requirements) are likely to increase registration demands as efficacy standards will no longer be a barrier to bringing products into the Canadian marketplace (as noted in Section 6).

8.0 Conclusions and Recommendation

The Fertilizer Program continues to be relevant to the needs of both Canadians and the marketplace in terms of promoting safety and labelling accuracy. The program aligns with both Government of Canada priorities, and the priorities of the CFIA. For the most part, CFIA roles are well aligned with others, though there have been some challenges in coordinating with other groups on dual property products (pesticide and supplement) and with provinces and territories in terms of the marketplace status of waste-derived materials (for example, whether products are sold or distributed in other ways). Some challenges exist with regard to internal CFIA roles as linkages between and among policy, program, science, monitoring and enforcement functions can cause issues in terms of a lack of integrated risk management.

Given its context and conditions, the Fertilizer Program has been highly successful in engaging stakeholders, and recruiting them to assist the program in its transition away from an efficacy focus and prescriptive requirements towards a performance-based approaches and 'tiered' safety focused orientation (risk-based model).

The strong engagement of sector stakeholders has both enabled a speedy policy change and has helped to reduce registration backlogs – in spite of structural irritants such as information management and information technology limitations, and increasing product complexity.

Gaps exist with regard to inspection, analytical testing and enforcement (that is, sanctions or penalties such as detention or prosecution. The gaps in inspection and analytical testing relate to insufficient sampling and verification which in turn create a gap in knowledge with regard to emerging risks, as well as weakening deterrence when combined with limited enforcement action. This means that the Fertilizer Program is left without adequate means to gather on-going information on and then to hone in on areas of relatively higher risk.

Recommendation

The Fertilizer Program should implement a mechanism to align its pre-market assessment and marketplace monitoring activities (inspection and analytical testing) with areas that pose the greatest risk to Canada's food, plants and animals and the environment.

• Emerging risks should be identified proactively to allow adequate risk mitigation while enabling flexibility in the regulatory framework and reducing administrative burden whenever possible and appropriate.

Annex A: Fertilizer Program Results Chains – Annotated Observations

The following results charts were constructed in consultation with Fertilizer Program management and then used to focus interviews and a major workshop of stakeholders who both validated the logic and commented and provided evidence on the key assumptions and factors affecting performance. They show summary observations for the overall fertilizer Program Logic Model and for the results chains for the four key functions of:

1. Communications, Stakeholder Engagement and Education
2. Assessment and Registration
3. Verifications and Compliances (Product Monitoring / Regulation)
4. Interdepartmental and International Cooperation and Standard Setting

The observations are shown in yellow comment boxes.
The first figure is the logic model for the Fertilizer Program, which is aligned with the Agency's logic model, presented after the fourth and final results chain.

Annex B: Evaluation Matrix

Evaluation Matrix

The following evaluation matrix was included in the planning document for the evaluation, or evaluation framework, approved by the evaluation's advisory committee in October 2012.

Annex C: Bibliography

Bibliography

Ayres, Ian, and Braithwaite, John, Responsive Regulation: Transcending the Deregulation Debate, New York: Oxford University Press, 1992.

Binkley, Alex, Fertilizer standards in the works, September 4, 2012. ( "http://www.infomedia.gc.ca/inspection/articles/restricted/2012/09/nau201295928639854.htm"
http://www.infomedia.gc.ca/inspection/articles/restricted/2012/09/nau201295928639854.htm)
Accessed December, 2012.

Canadian Council of Ministers of the Environment, A Review of the Current Canadian Legislative Framework for Wastewater Biosolids, 2010.

Canadian Council of Ministers of the Environment, Guidance Document for the Beneficial Use of Municipal Biosolids, Municipal Sludge and Treated Septage, 2012

Canadian Fertilizer Products Forum discussions about fertilizer regulation modernization on the online network Collaborase

Canadian Fertilizer Products Forum, Working Group Report on Regulatory Efficiency,( "http://www.cfpf-fcpf.ca/_documents/pdf/WorkingGroupRegulatoryEnglishFINAL.pdf"
http://www.cfpf-fcpf.ca/_documents/pdf/WorkingGroupRegulatoryEnglishFINAL.pdf) Accessed August 2012.

Canadian Food Inspection Agency, 2011-2012 Performance Management Framework

…, Fertilizer Inspection Manual

…, Inspection Memorandums: Fertilizer Inspection Programs, for years 2011-12, 2010-2011, 2009-10

…, Manure Import Consultation document, 2012

Funnell, Sue and Rogers, Patricia, Purposeful Program Theory: Effective Use of Theories of Change and Logic Models, San Francisco: Jossey-Bass, 2011.

Health Canada, Dioxins and Furans, 2005. ("http://www.hc-sc.gc.ca/hl-vs/iyh-vsv/environ/dioxin-eng.php"
http://www.hc-sc.gc.ca/hl-vs/iyh-vsv/environ/dioxin-eng.php) Accessed January, 2013.

Hydromantis, Inc.; University of Waterloo; Trent University. Emerging Substances of Concern in Biosolids: Concentrations and Effects of Treatment Processes. Canadian Council of Ministers of the Environment, 2010. piv, (http://www.ccme.ca/assets/pdf/pn_1440_contam_invt_rvw.pdf) Accessed November, 2012.

Lamb, John; Sheaffer, Craig; Moncada, Kristine, Chapter 4 "Soil Fertility" in Moncada and Sheaffer (Ed.s) Risk Management Guide for Organic Producers, 2010, Regents of the University of Minnesota.
New Zealand Food Safety Authority, ACVM Standard for the Fertiliser Group of Agricultural Compounds, ("http://www.foodsafety.govt.nz/elibrary/industry/Acvm_Standard-Requirements_That.pdf"
http://www.foodsafety.govt.nz/elibrary/industry/Acvm_Standard-Requirements_That.pdf) Accessed September, 2012.

New Zealand Food Safety Authority, Fertiliser 'Slice of Life' Review, August 2009. ( "http://www.foodsafety.govt.nz/elibrary/industry/Slice_life-Chemicals_Medicines.htm"
http://www.foodsafety.govt.nz/elibrary/industry/Slice_life-Chemicals_Medicines.htm) Accessed September, 2012.

Office of the Auditor General, The potential health and environmental impact of using sewage sludge on agricultural land, 2010. Environment Canada's Response to Environmental Petition No. 296, OAG Petitions Catalogue, ("http://www.oag-bvg.gc.ca/internet/English/pet_296_e_34144.html"
http://www.oag-bvg.gc.ca/internet/English/pet_296_e_34144.html) Accessed November, 2012.

Treasury Board of Canada Secretariat, Cabinet Directive on Regulatory Management. 2012. ("http://www.tbs-sct.gc.ca/rtrap-parfa/cdrm-dcgr/cdrm-dcgrtb-eng.asp" http://www.tbs-sct.gc.ca/rtrap-parfa/cdrm-dcgr/cdrm-dcgrtb-eng.asp) Accessed August, 2012.

Treasury Board of Canada Secretariat, Theory-Based Approaches to Evaluation: Concepts and Practices, 2012. ("http://www.tbs-sct.gc.ca/cee/tbae-aeat/tbae-aeattb-eng.asp"
http://www.tbs-sct.gc.ca/cee/tbae-aeat/tbae-aeattb-eng.asp) Accessed December, 2012.

United States Environmental Protection Agency, Office of Compliance, Office of Enforcement and Compliance Assurance,Profile of the Agriculture Chemical, Pesticide, and Fertilizer Industry, EPA Office of Compliance Sector Notebook Project, September 2000. ("http://www.epa.gov/compliance/resources/publications/assistance/sectors/notebooks/agchem.pdf"
http://www.epa.gov/compliance/resources/publications/assistance/sectors/notebooks/agchem.pdf) Accessed November, 2012.

United States Environmental Protection Agency, Office of Solid Waste,; Centre for Environmental Analysis, Research Triangle Institute; Estimating Risk from Contaminants Contained in Agricultural Fertilizers Draft Report, August 1999. ("http://www.epa.gov/osw/hazard/recycling/fertiliz/risk/report.pdf"
http://www.epa.gov/osw/hazard/recycling/fertiliz/risk/report.pdf) Accessed November, 2012.

USA Gardener, The Complete Gardening Glossary, ("http://usagardener.com/glossary.php" http://usagardener.com/glossary.php) Accessed December, 2012.

Annex D: Interview Questions

Fertilizer Evaluation Interview Questions

Introduction

Please provide a brief description of your company/organization's involvement with the Fertilizer Program.

1. Who are the key beneficiaries of CFIA's Fertilizer Program?
2. Are CFIA's Fertilizer Program activities appropriate for its mandate? (MANDATE: Administering the Fertilizer Act and Regulations which are intended to ensure that all fertilizer and supplement products are safe for humans, plants, animals and the environment.)
   (Review of results chains)
3. What are the key CFIA Fertilizer program responsibilities?
4. Does the Fertilizer Program do what it says it will do, such as with its sampling, testing and registration, or is the work on the ground different from what is outlined in manuals and other documents?
5. Do the activities of the Program change under certain conditions or with certain parties? Are the differences in Program activities appropriate for the changed conditions?
6. Are Program results different for different parties? Are such differences appropriate?
7. What is the current role of performance- or outcome-based approaches and what are the options?
8. Are the activities, such as assessment, registration and monitoring applied consistently and cumulatively (e.g., taking into account past results)?
9. How do the Fertilizer Program elements compare, in terms of what they deliver, to alternatives?
10. Are there more cost effective ways of achieving the same results? (e.g. in combination with other instruments and / or groups?) Are there good practices in comparable contexts which could be adopted or adapted for use by the Fertilizer Program?
11. What would happen without the Fertilizer Program activities?
12. Regarding the Fertilizer Program registration backlog, what have been contributing factors and how have these been resolved?

Annex E: Case Study on Fertilizer Risks

Fertilizer Risks

Case Study for the Evaluation of the CFIA's Fertilizer Program, 2012

Objective

A case study was undertaken to provide the evaluation team with an understanding of the key risk areas for food, plants and animals resulting from fertilizers and supplement product imports and sales in Canada.

This case study is not in any way meant to be scientific, comprehensive or authoritative. It is intended to provide in one spot (evaluation report annex) a list of the major risks the evaluation uncovered.

Scope

The case study focused on the Canadian context, but included a review of American, European and Australasian documents and literature.

Methodology

Literature and document reviews, and interviews.

Specifically, an additional question was asked of the interviewees for the evaluation. All interviewees were provided with a set of questions in advance, but the question of key safety risks related to fertilizers and supplement products was not included. This was added for most of those interviewed, with 16 responding. In addition, two CFIA experts were interviewed specifically for safety issues.

Key literature included publications from the Canadian Council of Ministers of the Environment and the United States Environmental Protection Agency.

A bibliography is included in the evaluation report.

Fertilizer and Supplement Products

According to Canada's Fertilizers Act, "'fertilizer' means any substance or mixture of substances, containing nitrogen, phosphorus, potassium or other plant food, manufactured, sold or represented for use as a plant nutrient." Similarly, the International Fertilizer Industry Association defines fertilizers as "Any natural or manufactured material that contains at least 5% of one or more of the three primary nutrients - nitrogen (N), phosphorous (P), or potassium (K)". NPK fertilizers are generally considered low risk, although there is a potential danger from nutrient leaching and watershed contamination if excessive amounts are used of any one of these three common constituents of fertilizer.

Supplements: Any substance or mixture of substances, other than a fertilizer, that is manufactured, sold or represented for use in the improvement of the physical condition of soils or to aid plant growth or crop yields:

• Micronutrients: The nutrients needed in small amounts by plants: iron, manganese, zinc, copper, molybdenum, boron and chlorine;
• Inoculants: Micro-organisms which are introduced into the soil to improve growth of crops;
• Soil Amendments: Any material added to a soil to improve its physical properties, such as water retention, permeability, water infiltration, drainage, aeration and structure, and;

Application Rates: Too Much of a Good Thing

Safety depends largely on usage pattern. The Fertilizer Regulations uses the term "safe when used as directed". The primary concern is over-loading. Maximum thresholds are usually based on cumulative amounts on a 45 year time frame, a standard that is said to have, according to CFIA experts, originated as the average working lifetime of a farmer.

There is an underlying risk of greater contamination than scientifically projected because the determination of application rates is based on the assumption that soils have not previously been contaminated with anything else, i.e. the medium is pure to begin with. The CFIA does not test soil samples.

Emerging Substances of Concern

Competitive pressures, innovation, pressures to find alternative uses for waste (both industrial and municipal) and the fact that waste products may increasingly contain pathogens, pharmaceutical residues, etc., combine to create both an increased and more uncertain risk profile for fertilizer use. There is also a lack of scientific research on some of the potential contaminants to assist regulators in determining safety requirements:
"…there is a notable lack of a database regarding the presence in Canadian biosolids of a certain class of wastewater constituents termed emerging substances of concern (ESOCs). These ESOCs include an array of pharmaceuticals, personal care products, brominated flame retardants and industrial contaminants (such as plasticizers and surfactants). At the present time, the ability to detect ESOCs in the biosolids at the ng/L or ng/g TS (dry wt) exceeds the understanding of the scientific and public communities of any potential risk associated with their detected presence. …While there is documentation in the literature on ESOCs in biosolids, especially in America and Europe, no focused study has been completed yet on a complete and recent inventory of ESOCs in Canadian biosolids^Footnote 23."

Trace elements/heavy metals are of particular concern from waste-derived products, particularly sewage. It is currently unknown whether sewage intended for agricultural field application is sufficiently treated. Sewage also has the potential of creating organisms with antibiotic resistance. The presence of antibiotics and other antimicrobial compounds create a selection pressure on native soil organisms that contain antibiotic resistance genes leading to the spread and proliferation of resistance. Organisms carry antibiotic resistance and then multiply and transfer the resistance. This could cause environmental damage and food contamination. Furthermore, there is a high potential for plants to take up contaminants. The contaminants can then end up in the edible portion of the plant.

Another problem with heavy metals is "[t]hey are not degradable and can accumulate in animal and plant tissues and in the soil…" and therefore "it is important to monitor their concentration in the soil before and/or after application of municipal biosolids and treated septage^Footnote 24."

In addition to heavy metals, there are other inorganic contaminants including radioactive compounds if the primary material is derived/mined from contaminated areas or generated through recycling of industrial wastes.

Finally, some waste-derived products may be difficult or costly to dispose of. Selling them or giving them away as fertilizers has been an option for some. Commercially sold fertilizer products are regulated by the CFIA. When fertilizers are not sold (i.e. when they are used directly by the producer or given away) the province or territory typically regulates their transport manufacture and disposal including application to land. This has led to some challenges in coordination. Some waste product owners have told provincial inspectors that they are selling their products as fertilizer and then told the CFIA inspector that they are giving it away.

Innovation Creates Complexity in Products (e.g., microbial inoculants, nanomaterials, etc.)

Complexity in turn creates uncertainty, difficulties in predicting risks create an increase in risk potential. The factor affects all stakeholders. It is further compounded by CFIA's limitation in science and analytical capacity, particularly for new and innovative products, including nanomaterials and emerging contaminants of concern.

Manure

Increased usage of manure has followed increased organic farming because of the restrictions on non-natural fertilizers: "Compost manure, animal manures, and green manures are examples of commonly used organic fertilizers for short and long term fertility management^Footnote 25."

According to Agriculture and Agri-Food Canada (AAFC):

"In Canada, there has been a small but increasing organic agriculture sector since the early 1980s. In recent years, this sector has seen dramatic growth with organic food consumption developing at a faster rate than production^Footnote 26."

The number of certified organic farms increased from 1,172 in 1992 to 3,917 in 2009.

For food and plant health, the primary risk with the use of manure as a fertilizer lies in insufficient treatment or inappropriate source. Manure should not be applied before sufficient composting. The 2006 spinach recall in California was linked to a farm that was transitioning to organic and had been applying manure to its spinach fields. The resulting e-coli contamination put more than 100 in the hospital and killed 3.

Heavy metals can also be a problem, particularly from poultry operations^Footnote 27. More dangerous still is the run off contamination to waterways. Combined with mismanagement of water treatment, this has led to human fatalities, as occurred in Walkerton Ontario in 2000 (seven deaths). Currently, more than 10% of land used for agriculture in Canada has manure applied to it.

Manure imports are another danger area. Bat guano and manure from species other than ruminants, porcine and birds are not regulated according to the definition of "manure" in the Health of Animals Regulations. This is currently being addressed with an amendment that will require all imported manure products to have a certificate of treatment or some other guarantee of safety. Note that some bat guano, such as that from Indonesia, has contained ebola in the past (see last page for table of potential hazards).

Wetting Agents

Wetting Agents are considered soil amendments, or supplements. Some have dangerous properties, such as those using Polyacrylamide, which can contain neurotoxins.

Imports

Dubious sources: Chinese exports of a variety of products have led to contamination, such as with the use of melamine to increase nitrogen readings (intentional adulteration).

Import Exemption for Own Use

Currently, fertilizer and supplement products "imported for direct application to the soil by the importer and not for sale in Canada." (Fertilizers Regulations section 3(1)(f)). This presents a potential risk to human, plant and animal health and the environment because there is no oversight concerning the types of products, the quantities applied or their location, all for products that are not permitted for sale in Canada. This loophole is expected to be eliminated as part of the regulatory modernization initiative.

Microbial Inoculants and Consortia

There is growing interest in using viable microorganisms as supplements that increase plant growth and crop yield. Historically, this was largely focused on rhizobia that aid in nitrogen fixation and absorption by legumes. This trend has now expanded to a large array of microorganisms that stimulate plant growth, facilitate nutrient uptake and utilization, fight pathogenic microorganisms in soil or even induce resistance to abiotic stresses and a wide range of diseases in crop plants. This is a very beneficial utilization of microorganisms; however, these environmental isolates are often opportunistic pathogens (cause infection in immunocompromised individuals) and as such may pose human health and safety risks. In addition, use of complex microbial mixtures obtained from the environment poses contamination concerns as it is difficult to determine the exact composition of the mixture (hundreds of different species and strains). In this context, accurate identification of the components is necessary and analytical methodologies are often lacking.

Conclusion

Available research, marketplace trends and the observations and insights of knowledgeable stakeholders interviewed for this study, suggest that there are both growing known risks and increasing uncertainty (i.e. growing unknown risks) related to the use of fertilizer and supplement products in Canada. While CFIA managers and staff are making efforts to adjust to the emerging situation, there is missing still a comprehensive and coordinated risk review and analysis of CFIA surveillance, registration, inspection, monitoring and testing practices.