Language selection

Search

Archived - Evaluation of Administrative Monetary Penalties

This page has been archived

Information identified as archived is provided for reference, research or record-keeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Management Response and Action Plan

April 2012

Table of Contents

Executive Summary

Administrative Monetary Penalties (AMPs) are intended to address non-compliance in areas in which corrective action requests and simple warnings are not enough, but where prosecution, seizure, or license suspension or revocation are considered too harsh. AMPs at the Canadian Food Inspection Agency (CFIA) are currently applied to select areas of the Health of Animals Act (HAA) and Health of Animals Regulations (HAR) and of the Plant Protection Act and Plant Protection Regulations.

Senior management in the Operations Branch, and the Programs and Policy Branch of the CFIA requested an evaluation of AMPs to assess their effectiveness and efficiency, and provide guidance on the merits of expanding their application to other acts and regulations, as the AMPs Act currently allows. The evaluation assessed the relevance and performance of AMPs, conducting literature, document, file and case reviews, and analysis, in addition to interviews. The scope of the study included the application of AMPs at the CFIA from 2000 to 2010.

Conclusions

AMPs are an effective compliance tool that has been successfully employed at the CFIA as an appropriately-scaled response for cases in which prosecutions would be excessively time consuming and costly.

Effective Implementation

Existing inefficiencies and obstacles to the effective application of the AMPs tool will be amplified with any expanded use of AMPs for other acts and regulations. Unclear legislative and regulatory wording, leading to uncertainty in interpretation, has contributed to increased risk of appeal. As well, the collection of evidence without regard to risk of appeal limits capacity for reprioritization of resources.

The lack of clear objectives, systematic measurement systems, and periodic reviews, which would have permitted systematic learning from the early application of AMPs, has limited improvements in the use of the tool, and reduced certainty about its value. Additionally, the lack of reporting capability has limited the opportunity to build learning into delivery and to provide managers with information on the issues causing the greatest challenges for inspectors and investigators.

The use of inspectors for less complex and lower risk of appeal cases would allow for additional capacity if AMPs were expanded.

Recommendation 1

The VP, Operations Branch, should establish a plan to build learning into the delivery of current AMPs for their ongoing development, and for any expansion of the use of the tool.

The plan should include the following elements:

Strategic and Appropriate Use

There was limited integration of the AMPs tool into the CFIA's compliance strategies and little evidence of linkages between Programs and Policy Branch, and Operations Branch that would allow program and enforcement staff to identify problem areas early on, and adapt enforcement strategies according to the specifics of each case.

Key conditions, demonstrated internationally and at the CFIA, affect the success of administrative monetary penalty instruments (e.g., clarity of objectives, legislative wording, and inspection and enforcement roles). Consistency in understanding of the roles and obligations of all players, from regulated parties to government staff is also essential.

Recommendation 2

The VP, Operations, should work with Policy and Programs Branch, with guidance from legal services, to ensure that all existing AMPs are operating under the key conditions identified, and that AMPs processes, guidelines and strategies are included in program specific enforcement strategies, where relevant.

List of Abbreviations

ALRC
Australian Law Reform Commission
AMPs
Administrative Monetary Penalties
CART
Canada Agricultural Review Tribunal
CDSR
Cabinet Directive on Streamlining Regulations
CFIA
Canadian Food Inspection Agency
CLT
Certified Livestock Transport
EIS
Enforcement and Investigation Services
FCA
Federal Court of Appeal
HAA
Health of Animals Act
HAR
Health of Animals Regulations
ID
Identification
IT
Information Technology
NETS
National Enforcement Tracking System
PPA
Plant Protection Act
PPR
Plant Protection Regulations
RIAS
Regulatory Impact Analysis Statement
SCC
Supreme Court of Canada
US
United States
VP
Vice-President

Preamble

Introduction to the Evaluation of AMPs

The CFIA's Operations Branch, and Policy and Programs Branch requested that an evaluation of AMPs at the CFIA be carried out by the Audit, Evaluation and Risk Oversight (AERO) Branch. The request was part of a review of regulatory priorities which included consideration of the potential usefulness of expanding the application of the AMPs regulations to other Agency legislation.

Scope

The evaluation assessed the relevance and performance of AMPs at the CFIA since its introduction in 2000 through to 2010. The approach was to determine what works for whom under what conditions, focusing on how AMPs achieves outcomes in a given situation or context, with the objective of identifying areas where the use of this mechanism can be improved within the context of CFIA regulatory control and in consideration of any possible expansion of the use of AMPs to other program areas in the CFIA.

Methodology

Using interviews, a sample survey and file, document and literature review and analysis, the evaluation sought to assess the following:

Details on the evaluation framework, methodology and study limitations are provided in Annex B.

How AMPs Work

The CFIA introduced its Administrative Monetary Penalties (AMPs) program in 2000, with partial implementation of the traveler stream, applying them in a select few airports. Over the years, the AMPs program has expanded to its current scope, which allows for the issuance of penalties or warnings at border points (administered by the Canada Border Services Agency) and within Canada (referred to as Commercial AMPs) for violations of scheduled sections of the Health of Animals Act, Health of Animals Regulations, Plant Protection Act and Plant Protection Regulations.

Under the Agriculture and Agri-Food Administrative Monetary Penalties Act and Regulations, the CFIA may issue an AMP as an enforcement measure to encourage compliance with the Health of Animals Act and the Plant Protection Act and their associated regulations. Its application typically begins with CFIA inspectors identifying possible violations of the relevant legislation and providing this information to CFIA investigators who may then issue an AMP with warning or with penalty.

Penalties range from $500 to $15,000, depending on the nature and gravity of the offence and the history of the offender.Footnote 1 If an individual or a company receives a Notice of Violation, they may request a review of the facts. The review is made by the federal Minister of Agriculture and Agri-Food Canada or the Canada Agricultural Review Tribunal. In this report, reviews are also referred to as appeals.Footnote 2

This report is organized around the evaluation questions as outlined in the evaluation framework, which is provided in Annex B.

Introduction

Sections from 1.1 to 5.3 below present the findings by evaluation question as originally developed for the evaluation framework. Section 6 provides the overall conclusions and recommendations.Footnote 3

1.0 Need

1.1 What Needs are AMPs Supposed to Address?

AMPs are expected to fill a need for flexible sanctioning as part of the mid-range sanctionFootnote 4 of the compliance pyramidFootnote 5.

Figure 1: Compliance PyramidFootnote 6

Click on Image for Larger View
Figure 1: Compliance Pyramid. Description follows.

Description - Figure 1: Compliance Pyramid

This figure is a triangle that shows escalation of enforcements tools used to achieve compliance.

  • This figure is a triangle that is broken up into six different levels
  • At the base of the triangle, the word "Persuasion" is written
  • On the next level up, the words "Warning Letter" are written
  • On the next level up, the words "Civil Penalty" are written; this level has a line coming out from it that goes to a box. Inside the box are the words: "AMPs theoretically fill this spot in the pyramid"
  • On the next level up, the words "Criminal Penalty" are written
  • On the next level up, the words "Licence Suspension" are written
  • The very top level of the pyramid, which forms the triangular peak, has the words "License Revocation" written beside it

There is text beneath the pyramid that states: "Source: Ayers and Braithwaite (1992)."

AMPs add a level of penalty based in administrative law before criminal lawFootnote 7 is invoked. This is discussed in some detail in M. Prabhu's 2010 doctoral thesis from the Faculty of Law at the University of Ottawa:Footnote 8

"Using criminal law for even minor breaches of the Regulation has created unease and agitation among eminent scholars and regulation specialists, especially when a great many breaches are minor and overburden criminal courts…"

Furthermore, AMPs decriminalizes relatively minor transgressions. The distinction here between criminal and civil law is explored in J. Coffee's 1992 Yale Law Journal article: "Paradigms Lost: The Blurring of the Criminal and Civil Law Models - And What Can Be done About It":

I would suggest that in its characteristic operation, the civil law "prices", while the criminal law "sanctions'. … I would argue that the criminal law should be reserved to prohibiting conduct that society believes lacks any social utility, while civil penalties should be used to deter (or "price") many forms of misbehaviour (for example, negligence) where the regulated activity has positive social utility but is imposing externalities on othersFootnote 9.

1.2 AMPs at the CFIA

The original rationale for the use of AMPs is not fully articulated in any single document. A 2002 Regulatory Impact Analysis Statement (RIAS) suggests benefits in terms of cost effectiveness,Footnote 10 using figures from the Province of Ontario Attorney General's Office regulatory prosecution costs, estimated at $13,000 per prosecution. The 2002 RIAS suggests that AMPs represented an appropriately-scaled response needed for cases where prosecution would be too time consuming and costly (RIAS for Cattle ID, 2002). The development of an AMP program for the CFIA began in the early 1990s, before the Agency was created. According to interviewees, prior to the establishment of AMPs, there was a belief among those involved with its implementation at the CFIA that AMPs would encourage compliance without prosecution, and that their use would improve efficiency – interviewees confirmed that this belief persists at the CFIA.

A link between AMPs and increased compliance was expressed in the 1997 RIAS: "The experience in the United States, where administrative monetary penalties are used extensively, is that compliance rates will go up considerably because of the deterrent impact of AMPs." AMPs were initially applied in 2000 to travelers entering Canada at airports, an application subsequently moved to the Canada Border Services Agency. However, interviews also suggest that there was concern from the beginning that AMPs might not work.

This caution on the part of various parties may help explain why AMPs were then applied to the Health of Animals Act and Health of Animals Regulations and to the Plant Protection Act and Plant Protection Regulations in 2002, but not applied as anticipated (i.e. within one year) to all the other acts included in the AMPs Act.Footnote 11

1.3 Who is Affected?

Those affected by AMPs include the following:

1.4 What Has the Reaction Been to Date?

So far, the industry reaction to the issuance of AMPs by the CFIA has been mild, as noted by interviewees.Footnote 12 AMPs were not of very high interest to most of the industry associations contacted for this evaluation, though the most affected groups, as represented by association staff, note problems with consistency (for example, uncertainty regarding when AMPs are triggered), timing and fairness (such as the need for consistent treatment and consistent payment of penalties).

There are some opportunities to work with industry associations such as cattle, poultry and swine associations, which have taken steps to clarify animal care responsibilities.Footnote 13 These associations and others have supported the evolving Certified Livestock Transport (CLT) program, which is modeled on the US Transport Quality Assurance program and may begin setting some de facto animal transportation standards for Canada. Some processors have expressed an interest in using only certified transporters in future.

2.0 Alignment with Government Priorities

AMPs address the Cabinet Directive on Streamlining Regulations (CDSR): "…Departments and Agencies are to demonstrate that the regulatory response is proportional to the degree and type of risk." (CDSR, April 1, 2007).

Following from this notion of the need for a proportionate response, AMPs are an appropriately-scaled response for cases in which prosecutions would be excessively time consuming and costly. (RIAS for Cattle ID)

3.0 Alignment with Federal Roles and Responsibilities

The issuance of AMPs appears to be aligned with overall federal roles and responsibilities. However, among CFIA stakeholders, there are different perspectives of how AMPs work. The available information suggests that not all CFIA investigators and inspectors involved with AMPs understand them or their use in a consistent manner and, for this reason, application is inconsistent. There are also some differences in the perceptions among some CFIA stakeholders regarding the appropriate role for legal counsel in AMPs, especially concerning the appeal process.

3.1 Roles of Key Stakeholders

The roles of key stakeholders varied according to area,Footnote 14 though there were some common elements:

3.2 Differences in Understanding and Interpretation across Stakeholders

Inspectors

The majority of investigators and inspectors interviewed noted significant variation in the understanding of the roles of inspectors, even within areas. Investigation staff suggested that many inspectors, given their history of mainly issuing non-sanctioning notices such as Corrective Action Requests as opposed to Notices of Violation, did not see their job as being part of an enforcement system. For this reason, investigation staff noted that some inspectors did not issue AMPs or sufficiently collect and organize data and evidence in support of AMPs.

Of eight responsibilities for inspectors listed in the CFIA Compliance and Enforcement Operational Policy, seven were identified in interviews as not always being an accurate reflection of roles and activities when it comes to AMPs.

Inspectors
Inspector Roles Outlined In Compliance And Enforcement Operational Policy Inconsistencies And Variations From Expectations
1 carrying out inspections to verify, assess and monitor compliance with the law not always clear to inspectors how much or what type of information should be collected
2 communicating with regulated parties regarding the legislative requirements, how compliance will be assessed and what will be considered non-compliance inspector consistency in understanding the legislative requirements and in communicating those requirements
3 reviewing options for preventive and corrective actions and explaining legislative requirements to regulated parties consistency of understanding and communication with regulated parties;
4 directing that corrective measures be taken to address non-compliance that can be immediately corrected Notices of Violation can take months to issue – even for warnings
5 preparing non-compliance reports level of information collection (documentation) in preparation for possible appeals
6 making comprehensive notes, completing inspection reports level of information collection (documentation) in preparation for possible appeals
7 giving evidence in court and at reviews before the AMPs Review Tribunal legal counsel appear in court or in front of the Canada Agricultural Review Tribunal (CART) when investigators may often be sufficient.

Another concern expressed by CFIA interviewees was with the excessive internal (CFIA) requirements for the collection and documentation of evidence. The main concern expressed by industry association representatives was with regard to the consistency of AMPs application and the long time it takes to have AMPs issued.

Investigators

Note that most (22) of the interviews were conducted with Enforcement and Investigation Services staff. There were fewer problems noted with the roles of investigators than with the roles of inspectors, though only four inspectors were interviewed. Notwithstanding this point, four of 13 functions were noted as not always being an accurate reflection of roles and activities when it comes to AMPs.

Investigators
Inspector Roles Outlined In Compliance And Enforcement Operational Policy Inconsistencies And Variations From Expectations
1 review and prepare briefs of evidence recommending prosecution the level of evidence gathering and evidence to demonstrate violation (e.g. undue suffering, section 138 (2)(a) HAR)
2 prepare Administrative Monetary Penalty files despite stated intentions to move to prosecution after a few AMPs, some violators appear to receive repeated AMPs before CFIA moves on to prosecute (This raised questions regarding the quality of evidence, strength of penalties, costs, and understanding of escalation policy by CFIA staff)
3 liaise with CFIA Legal Services and the Public Prosecution Service of Canada with respect to investigations and prosecutions lack of training of investigators and inspectors
4 deliver enforcement training to CFIA staff That this is the appropriate role for EIS head office

3.3 Roles and Responsibilities Alignment

Legal Counsel

In the case of AMPs, legal counsel play a service role, representing the CFIA before the CART for all reviews, as well as before the Federal Court of Appeal for reviews of CART decisions.

Perspectives on the appropriate role of legal counsel differed. Some respondents suggested that the intent of AMPs was to have a system simple enough that legal counsel would not have to be involved to represent the CFIA before the CART. They also thought that the CFIA may be represented by a CFIA Enforcement and Investigation (EIS) officer at CART appeals, without legal counsel present, which was the original intent.

Interviewees also noted that some appeal cases had become more complex than originally envisaged. According to interviewees, the CART was intended to provide a simple and speedy administrative process, and not one requiring lawyers and large briefs. Others valued the support of legal counsel to interpret difficult clauses and to ensure that the CFIA made appropriate use of evidence.

Canada Agricultural Review Tribunal (CART)

A number of interviewees noted that because CART does not provide a speedy summary process, the CART process may not be aligned with the intent of AMPs. The CART typically takes from four to six months to reach a judgment.

Program Managers

Program managers were found to have a very limited role in terms of where, how and to what extent AMPs are being applied in their program areas. The extent of program involvement appears to be when they are consulted on specific AMP applications, at the discretion of investigators. Furthermore, their only information on AMPs in terms of system feedback appears to come from prominent complaints and anecdotal information. In addition to creating a gap in basic results knowledge, this means that, when considering operational decisions such as license renewals, program managers do not have consistent, if any information on whether and to what extent companies have received AMP warnings or penalties.

4.0 Achievement of Expected Outcomes

In general, despite a lack of detailed evidence, due to a paucity of reliable cost data and lack of performance targets and benchmarks, it appears that AMPs have succeeded in terms of providing the CFIA with the option to use a sanction which is less expensive or punitive than prosecution or licence revocation. Most AMPs are paid and most AMP sanctions are upheld upon appeal. Evidence suggests that on these terms, AMPs are more successful than criminal prosecutions.

Notwithstanding some overall positive achievements in terms of certain metrics, AMPs were found to vary in application consistency, in turn causing uncertainty across stakeholder communities. A more detailed systems review of AMPs in terms of how it operated as compared to the original theory is shown in Annex A.

Data such as an analysis of HAA and HAR cases taken from before, and then after the introduction of the AMPs option shows that on the whole, despite its problems, AMPs can serve as a legitimate and cost effective part of the compliance and enforcement toolkit for the CFIA. Scholars and interviewees identified a number of key factors and conditions important to the effectiveness of AMPs, outlined at the end of this section.

4.1 AMPs Actual Implementation Has Varied from What Was Planned

The following table summarizes how AMPs have varied as compared to expectations.
AMPs Expected Result (RIAS 2002) Observed Results
Emphasizing compliance vs. punitive action The penalties for AMPs can be higher than in prosecutions. Furthermore the Federal Court of Appeal commented that AMPs import some punitive elements of penal law. (Doyon decision 2009).
Provides for immediate enforcement and immediate corrective action Significant delays in issuance of AMPs were noted. It sometimes took two months or more just to issue a warning, and often more than a year to issue a penalty.Footnote 15

Records and interviews show the process has become more complicated, partially due to interpretations upon appeal which have called into question such elements as "undue suffering" leading to the imposition of a number of criteria by the CART against which the facts must be assessed to determine if the violation occurred.

Provides for the use of negotiated solutions to non-compliance for commercial violations Records and interviews suggest other compliance options are not exercised. Compliance agreements, an option which allows for a violator to apply the monetary penalty directly to make a corrective action (for example, fix a faulty trailer) are almost never used. Some interviewees expressed a concern that compliance agreements are overly complex and will not be entered into in good faith.

There were also variations found in some common perceptions among interviewees about how AMPs work. These misperceptions have been referred to by some as myths, which are dispelled below.

Myth/Reality
Myth Reality
Many AMPs are appealed

Relatively few AMPs are appealed, and most for a few specific violations. According to the evidence available, fewer AMPs are appealed than prosecutions.Footnote 16

Roughly 13% of AMPs are appealed (to both the CART and the Minister), and these appeals are concentrated in a few sections of the regulations. There are over 400 sections (including subsections) of the two CFIA acts and their regulations that outline possible violations where AMPs could have been applied over the time periods examined: 2000-2010. Only 135 have been used to issue an AMP with penalty. AMPs that were appealed represent 58 of the 135 sections. Three (3) of those 58 refer to "undue suffering" and represent 37% of all the appeals, or 144 out of 393 appeals. Another 2 of those 135 involve animal transport and tagging, and have had 86 AMPs issued, representing another 22%. A total of 59% of appeals come from just five animal transport sections.

Many appeals result in reversals

Most AMPs decisions are upheld: available data suggests more AMPs decisions are upheld than prosecution decisions.Footnote 17

Roughly 73% of AMPs are upheld on appeal.

Many violators do not pay AMPs Roughly 10% of AMPs are unpaidFootnote 18 (Source: National Accounts Receivable)

A lack of clarity in the wording of some of the infractions has caused significant challenges in the interpretation of their meaning, particularly in the appeal process, particularly those related to "undue suffering". As noted in the table above, three of those sections account for 37% of all appeals. What is "undue" is not always interpreted the same way by all parties. Note that the Doyon decision suggests specific tests for "undue suffering", but even these are subject to interpretation for the particularities of individual cases.

4.2 How AMPs Work in Different Circumstances

According to interviews and records, AMPs work better when there is clarity in the legislative and regulatory wording, when there are controlled inspection conditions, when there are few stakeholders, and when a regulated community generally wants to be part of a firm and stable marketplace, and is generally delivering medium to higher-end products. The problem players, prone to recurring violation, are often the so-called marginal actors. Interviews and file reviews pointed to dealers in cull animals and spent hens that collect animals to be slaughtered for marginal value attached.

AMPs do not work well when they are seen as a "cost of doing business", that is, where the AMP is a small cost of the load (e.g., chickens), and the system is geared for continuous production.

4.3 Achievement of Expected Results

The following factors limit the identification of strict causality related to the expected results:

It is expected that the increase in the maximum penalty amounts, which were more than doubled in October 2010, will increase the deterrence effect of AMPs. No evidence was available at this early stage to assess this degree to which this expectation is being realized.

However, there is sufficient evidence to make a number of conclusions about the achievement of results for AMPs, as outlined below. As noted above, the 1997 RIAS referred to increased compliance in the United States and it stated that AMPs would allow the CFIA to increase enforcement activities: "…Since AMPs are not as time consuming as prosecutions, it will allow the CFIA to increase its enforcement activities." No evidence is cited with these claims in the RIAS.

Analysis in the previously referenced 2010 Prabhu doctoral thesis suggests that the number of individuals prosecuted for offences under Health of Animals Act and Regulations dropped from 125 in the five years preceding 2001, to 66 in the five years from 2001 to 2006. Note that there is correlation, but no confirmed attribution in this analysis, that is, although this shows that prosecutions declined following the application of AMPs as was intended, there are other possible causes of a decrease in prosecutions that have not been investigated, such as changes in policies, investigation practices and resources.

Prabhu's thesis analysis also points to a strong decline in the number of failed prosecutions over the five years after the introduction of AMPs. This may indicate that with fewer prosecutions there were higher quality prosecutions and a higher success rate, but may also suggest that AMPs were previously applied to cases that were relatively weak as criminal prosecutions. He also notes that the success rate for AMPs in terms of violations being upheld was much higher than what was experienced by enforcement officers in criminal courts.

4.4 Integrity of Implementation

The level of AMP issuance varies significantly by area. This is partially due to the nature of the regulated communities (e.g. some areas have large cattle volumes leading to more cattle ID AMPs), but is also due to differing area practices about when and how to issue AMPs. Across areas, Area managers employ different strategies, and different degrees of thoroughness in evidence collection.

Notwithstanding differences in context and conditions, the level of AMPs investigation and evidence gathering varies extensively. The documentation process is sometimes performed at a high level even when a Notice of Violation is issued without penalty (i.e., a warningFootnote 19). Investigators have 15 days to prepare briefs for any appeal requested, and appeals can only be requested within 30 days of AMP issuance.Footnote 20 This limitation to the time to produce an evidence brief has led most investigation managers to require that full briefs be prepared for all AMPs at the outset, in case they are appealed – despite the fact that only 13 percent are appealed and most of the appeals are for a few specific types of violation, as noted in the previous table.

Some interviewees suggested that the CFIA is reluctant to escalate from AMPs to other compliance tools when parties repeat the same violations. However, clear evidence shows that the track records of regulated parties, and of inspection and investigation efforts, are consistently used to determine CFIA actions to achieve compliance and enforcement. The ability to use this history to determine the course of action is limited to the degree that this information is accessible.

The National Enforcement Tracking System (NETS), the primary CFIA Enforcement and Investigation Services database, does not have the capacity to provide consistent tracking information and has no reporting capabilities. Its significant limitations have led to inconsistent data entering which in turn has led to the use of numerous Excel files in each office across the country keeping duplicate information, little of which is linked.

4.5 Some Key Principles to Guide AMPs Application

ScholarsFootnote 21 have noted that AMPs are appropriate when the following elements are present, and their observations are supported by the interviews conducted for this evaluation:

Interviewees made the following suggestions of key factors affecting AMPs success:

5.0 Demonstration of Efficiency and Economy

The evidence to date, while not allowing for a precise estimation, suggests that AMPs can be cost-effective (that is, provide economy and efficiency) in the right circumstances. Among interviewees, views were split in terms of whether a summary conviction ticketing scheme would be a better alternative.Footnote 22 Views also varied in terms of how and to what extent evidence collection and appeals preparation and delivery could be streamlined. However, the evidence below suggests that there is an opportunity to improve the economy, efficiency and cost-effectiveness of AMPs.

5.1 AMPs vs. Alternatives

Some interviewees suggested that the CFIA should also seek alternatives to AMPs. In cases where remedies and sanctions such as detention or seizure of shipments were readily available and generally practiced, the contention was made that AMPs might not be necessary. Interviewees were split as to whether a summary conviction ticketing regime would work better than AMPs. Some pointed to Health Canada's Contraventions Act as an example because it allows for a determination of a violation, or a contravention, similar to a parking or speeding ticket, and appeals go before a special contraventions court. Others felt that this would only be relevant for certain types of AMPs that were straightforward, and that for complex AMPs, such as animal transport cases, a full investigation would be required.

5.2 More Cost-effective Ways of Achieving the Same Results

Cost-effectiveness is impossible to calculate precisely due to lack of strong data. However, it was an issue of interest to most interviewees. Many of the points raised about cost-effectiveness by interviewees related either to legal and administrative process improvements, or to training, though some related to the need for more strategic use of these instruments.

Efficiencies through legal and administrative process improvements

A number of interviewees expressed concern about the timeliness of issuing AMPs. The regulations provide up to six months to issue a minor AMP and up to two years for more serious AMPs. Beyond assuring the investigation staff that a violation has taken place, most evidence is prepared in case an AMP is appealed, as discussed earlier.

There are significant costs related to investigation, both in CFIA resources and in the frustration of AMP recipients for a violation that, after a year or more, they may no longer recall. The time elapsed may lead to repetitions of the violation before the AMP is issued, whereas a speedy issuance could deter additional violations. A number of interviewees noted the opportunity for cost savings if level of investigation were based on the risk of an appeal.Footnote 23

For legal process improvements, many interviewees discussed the use of lawyers to represent the CFIA before the CART. Most considered this representation was necessary only when cases are complex. It was noted that the original intent when AMPs was implemented at the CFIA was for lawyers to provide some guidance to investigators for handling appeals, but lawyers have always represented the CFIA before the CART. Some have pointed to the use of lawyers by the CFIA at the CART as a reason for excessive investigation. Conversely, a reduction in the use of legal counsel might not improve cost-effectiveness for complex cases.

Improving payment compliance was raised by many interviewees, despite the fact that the vast majority of AMPs penalties are collected and only about 10 percent (by number and 14% by value) are not. A number of interviewees noted that when penalties are not collected, word spreads to others and reduces the credibility of the system. The CFIA has been working with the Canada Revenue Agency to obtain some of the non-payments.

Need for greater training of inspectors

More training of inspectors was a common suggestion to improve the efficiency and effectiveness of AMPs. The CFIA's Ontario investigators developed a training program for inspectors, but this has resulted in only about five inspectors issuing AMPs, although 50 were trained. Many interviewees noted that any increase in the use of AMPs would be feasible only if inspectors were trained to issue them. The grouping of AMPs into simple vs. complex types was considered key for identifying which AMPs could be issued by inspectors.

Interviewees also recommended inspector training to increase awareness of alternative enforcement options. A number of investigators believe that many inspectors are not using their full range of enforcement options. For example, few inspectors use their authority to seize, confiscate and destroy property. For animal transport issues, they can also immobilize vehicles, as well as seize vehicles, but this is seldom done.

Strategic Use

Program manager interviewees noted a need to better integrate the use of AMPs into business line strategies. The suggestion was made by CFIA interviewees that AMPs should be considered part of staged initiatives and targeted blitzes. As a minimum, program managers interviewed expressed a desire to be more involved in the planning and monitoring of AMPs applications. For example, some managers are responsible for licensing companies, without any awareness of the companies' compliance history, such as its receipt of AMPs. Some managers develop inspection strategies based on risk, without awareness of non-compliance information that AMPs investigations have uncovered. Increased program manager investment could help in operational decisions such as company license renewals, as well as in strategic and tactical program management decisions regarding compliance

5.3 What Would Happen Without AMPs?

Every interviewee noted that without AMPs, the CFIA would be deprived of a key tool for compliance, and more prosecutions would be required. Interviewees consistently noted that prosecutions could never replace the number of AMPs issued and as such compliance would decrease.

Interviewees also noted that AMPs provide "early warnings" about program non-compliance. For example, when a number of AMPs have been issued for a particular situation, such as a particular vegetable consistently arriving at the border contaminated and therefore leading to the imposition of AMPs, tracking of the use of AMPs provides the CFIA with an awareness of a possible need to consider increased certification requirements, such as the addition of a phytosanitary certification in addition to regular certification.

6.0 Conclusions, Good Practices and Recommendations

While AMPs have shown some success in some conditions, they have not lived up to expectations in terms of speed, cost, and likely deterrence effect. This appears to be due to uncertainties in certain legislative and regulatory wording and the complicated multi-party nature of certain inspected enterprises such as animal transport. The nature of certain parts of the regulated sector, for example, certain chronic non-compliers that do not want or cannot afford to comply, and differing policies on evidence collection compound the difficulties in effective application of the AMPs tool.

6.1 The Process - Build Learning Into Delivery

Conclusions

AMPs at the CFIA have not grown beyond the Health of Animals Act and Plant Protection Act over the past decade. The CFIA never established the type of clear objectives, systematic measurement systems and key review point milestones needed to permit systematic learning from the early applications, which would have facilitated improvements in the use of the tool and an increased certainty about their value. This knowledge and capacity gap may have contributed to myths about the process which, in turn, have led to ineffective practices such as a lack of delegation and excessive evidence collection.

Good Practices

AMPs require an effective database, monitoring, evaluation and reporting system to ensure that CFIA management is more aware of the changing compliance conditions across all areas and ensure that investigators have ready access to the history of violations. This history is one of the criteria for setting the value of the penalty, requiring an assessment of past violations. Database reporting would also provide information on the types of AMPs causing inspectors and investigators the greatest challenges which would assist in the maintenance of a categorization of AMPs by risk levels.

Categorization of AMPs by complexity and risk of appeal would provide an opportunity to gauge the comprehensiveness of evidence collection required. Regular communication between AMPs Areas would allow for an effective and regular sharing of good practices and lessons learned.

Recommendation 1

The VP, Operations Branch, should establish a plan to build learning into the delivery of current AMPs for their ongoing development and for any expansion of the use of the tool.

The plan should include the following elements:

6.2 Strategy - An Integrated Approach to Compliance

Conclusions

Both the theory and the practical wisdom regarding the use of AMPs show that they need to be considered one instrument of policy among many, to be applied as part of an integrated compliance strategy. AMPs work best when integrated into a considered and consistently applied compliance strategy which clarifies the roles of inspectors and investigators, as well as legal counsel, review processes, communications and affected industry associations. There was little evidence of program strategies including any consideration of the different contexts and conditions where AMPs could be best applied. In addition, the determination of the potential role of inspection staff for the issuance of many of less complex AMPs is essential for any expansion of the application of AMPs to other Acts.

Good Practices

Integrating the needs of all players could provide a common understanding of the operations challenges, risks and problems facing specific food sectors and value chain segments. If connected to program (business line) strategies, AMPs could be better targeted and streamlined to address distinct compliance risks. Linkages of information between Programs and Policy Branch, and Operations Branch would allow program and enforcement staff to identify problem areas early on, and to adapt enforcement strategies according to the specifics of each case. For example, repeated violations in a particular area would be identified for review, possible education of stakeholders, or revisions of government policies.

6.3 Implementation - Setting the Right Conditions

Conclusions

Internationally and at the CFIA, it is apparent that there are key conditions which affect the success of this instrument. These include elements such as clarity of objectives, legislative wording, and inspection and enforcement roles. Consistency in understanding of the roles and obligations of all players, from regulated parties to government staff is also essential. Some of the key problems with AMPs at the CFIA relate to legislative wording of a few sections of relevant Acts (see section 4.1). A more complete list of conditions and principles is contained in section 4.5. Further suggestions for improvement are contained in section 5.2.

Good Practices

An assessment of all current and potential regulations and acts (as listed in the AMPs Act) to determine the degree to they meet the conditions and principles outlined in this study, would provide the CFIA with strong direction for choosing which parts of acts and regulations are most appropriate for applying AMPs. Such a determination would be most effective if developed cooperatively with operational and program staff, along with guidance from legal services.

Should any expansion of AMPs be considered, it should be assessed against these key conditions and principles for their success.

Recommendation 2

The VP, Operations, should work with Policy and Programs Branch, with guidance from legal services, to ensure that all existing AMPs are operating under the key conditions identified, and that AMPs processes, guidelines and strategies are included in program specific enforcement strategies, where relevant.

Annex A: AMPs Results Logic – Expected vs. Actual

There are various theories about the gaps which AMPs can address, and about the context appropriate for use of AMPs (see CFIA AMPs Evaluation Framework, July 2010). These can serve as background for understanding the results logic flow.

Figure A.1 starts with the perceived gaps and contextual factors then outlines the theory of AMPs use to promote compliance and food safety.

Figure A.1: The Perceived Gaps, Context and Results Logic of AMPs

Click on Image for Larger View
Figure A.1: The Perceived Gaps, Context and Results Logic of Administrative Monetary Penalties. Description follows.

Description - Figure A.1: The Perceived Gaps, Context and Results Logic of AMPs

This figure is a logic model for the Administrative Monetary Penalty System that illustrates the linkages between Activities / Outputs, Immediate Outcomes, Intermediate Outcomes and Ultimate Outcomes. To the left of the results logic are two columns of words called Perceived Gaps and Contextual Factors.

It is important to note that in this figure the columns and rows do not have lines. The results logic is made up of a collection of boxes and arrows on the right side of the page.

  • The figure has 4 rows going from the bottom to the top: Activities / Outputs, Immediate Outcomes, Intermediate Outcomes and Ultimate Outcomes. These 4 labels comprise a column on the far right of the page.
  • On the Activities / Outputs row in the Perceived Gaps column, the words "Regulators have inefficient existing means available (i.e. Need to improve efficiency in regulation)" are written
  • On the Intermediate Outcomes row in the Perceived Gaps column, the words "Regulators need an ability to moderate an otherwise harsh response (i.e. too big a jump between warning and criminal penalty)" are written
  • On the Activities / Outputs row in the Contextual Factors column (2nd column), the words "Level of internal capacity, resourcing and governance" are written
  • The Immediate Outcomes row has 2 levels
  • On the Immediate Outcomes row in the Contextual Factors column, the words "Physical environment of inspection / regulated party engagement" are written
  • Above that on the Immediate Outcomes row in the Contextual Factors column, the words "Surrounding support climate for regulated party (associations, marketplaces etc.) 'cultural' factors" are written
  • On the Intermediate Outcomes row in the Contextual Factors column the words " 'Nature' of the operations being regulated - levels of complexity, time, space, 'regularity' 'volume'" are written
  • On the Ultimate Outcomes row in the Contextual Factors column, the words "Social, Economic, Political, Technological, Environment Factors" are written
  • In the Results Logic column at the bottom of the Activities / Outputs row, there is a box which forms the base of the results logic, above which are 3 columns of smaller boxes. Small circled numbers 1-10 appear throughout, which are explained on the following page. Inside the large base box, there is a loop connecting the words Detect, Respond, Enforce, Assess and Modify; inside the loop is the number 1 and it has been circled; above the loop and inside the base box is another box with the words "Information (persuasion), warning, penalty, enforcement" written in it
  • Above the base box on the Immediate Outcomes row are three boxes that are parallel to each other; the box to the left has "Engagement of regulated party" written in it. This is the only box in this 1st of 3 column boxes. There is an arrow going to this box from the base box with a circled number 2 beside it and there is an arrow going from this box back to the lower box with a circled number 3 beside it. The middle box has the words "Notice of violation (AMPs) to regulated party" written in it, and there is an arrow going to this box from the base box with a circled number 4 beside it. This is the beginning of the second column of boxes. The box to the right has the words "Collections" written in it; there is an arrow going to this box from the lower base box with a circled number 5 beside it. This is the third column of boxes.
  • Above the "Collections" box, still in the Immediate Outcomes row, there is a box that has the words "Support to Tribunal / Ministerial review" written in it; there is an arrow going to this box from the base box. The number 7 is circled beside this arrow.
  • Above the "Notice of violation (AMPs) to regulated party" box in the Results Logic Column and in the Immediate Outcomes row, there is a box with the words "Regulated party reacts by paying penalty (or not)" written in it; there is an arrow with a circled number 5 beside it going from the "Notice of violation (AMPs) to regulated party" box to the "Regulated party reacts by paying penalty (or not)" box. There is an arrow going from the "Collections" box to the "Regulated party reacts by paying penalty (or not)" box. Also, there is an arrow going from the "Regulated party reacts by paying penalty (or not)" box to the base box.
  • Above the "Support to Tribunal / Ministerial review" box in the Results Logic column on the Intermediate Outcomes row there is a box with the words "Tribunal / Ministerial review decisions render clarity and fairness" written in it; there is an arrow going from the "Support to Tribunal / Ministerial review" box to the "Tribunal / Ministerial review decisions render clarity and fairness" box.
  • To the left and slightly above the "Tribunal / Ministerial review decisions render clarity and fairness" box is another box with the words "Regulated party understands infraction and changes capacity to comply". This box is in the Results Logic column on the Intermediate Outcomes row. There is an arrow going from the "Regulated party reacts by paying penalty (or not)" box to the "Regulated party understands infraction and changes capacity to comply" box. Also, there is an arrow going from the "Tribunal / Ministerial review decisions render clarity and fairness" box to the "Regulated party understands infraction and changes capacity to comply" box with a circled number 8 beside it.
  • Above the "Regulated party understands infraction and changes capacity to comply" box still in the Results Logic column and on the Intermediate Outcomes row is a box with the words "Improved compliance and stewardship 'actions' by regulated party" written in it. There is an arrow going from the "Regulated party understands infraction and changes capacity to comply" box to the "Improved compliance and stewardship 'actions' by regulated party" box with a circled number 9 beside it.
  • Above the "Improved compliance and stewardship 'actions' by regulated party" box, still in the Results Logic column and on the Ultimate Outcomes row, is a box with the words "Health, safety, security, environmental protection" written in it. There is an arrow going from the "Improved compliance and stewardship 'actions' by regulated party" to the "Health, safety, security, environmental protection" box with a circled number 10 beside it.

Given the perceived gaps, contextual factors and basic results logic shown in Figure A.1 above, the theory can be broken down into a series of premises:

  1. CFIA Operations Branch undertakes a systematic process of responsive compliance management activities which follows a systematic process. The process starts with consultation and information. As noted in Figure 1, this persuasion phase may escalate to warning, then to penalty, then to prosecution or licence revocation. In general, the framework of detection, response, enforcement, assessment and modification can be seen (the "DREAM" model, Baldwin and Black 2007)Footnote 24 can be seen throughout
  2. Verification: the clear verification of non-compliance.
  3. Regulated parties' responses render clear information to allow the CFIA to make a definitive choice regarding the most appropriate response to non-compliance.
  4. Notices of violation are defined clearly and unambiguously to key responsible parties.
  5. Notices of violations serve to foster respect for the process by virtue of regulated parties appropriately paying their penalties – or if not, proceed to step 6.
  6. Nonpayment triggers appropriate collection actions which may include steps 7 and 8.
  7. If a request for a review is made to either the Review Tribunal or the Minister, the investigator must assemble the case file within 15 days. The case file is used by the CFIA in presentation of their witnesses and evidence.
  8. Tribunals (or the Minister) deliver clear, logical, evidence-based decisions which constructively impact compliance (while respecting the rights of all parties).
  9. Regulated parties understand their violation and are provided with (or have) the knowledge, abilities, skills and aspirations to comply with requirements.
  10. All these elements combine to improve compliance, foster even further stewardship actions (i.e., actions beyond simple compliance to support desired outcomes) and lead to benefits for Canadians.

The following chart, Figure A.2, as read left to right, describes the perceived need for AMPs, some key contextual factors affecting their delivery and the results logic of the inspection, investigation and appeals processes in terms of an integrated 'system.' The 'comment' boxes represent study observations where conditions and practices have apparently affected the system.

Figure A.2: The Perceived Need, Context, Results Logic and Findings re: AMPs at CFIA

Click on Image for Larger View
Figure A.2: The Perceived Need, Context, Results Logic and Findings re: Administrative Monetary Penalties at Canadian Food Inspection Agency. Description follows.

Description - Figure A.2: The Perceived Need, Context, Results Logic and Findings re: AMPs at CFIA

This is the same as figure A.1. with a few key changes. There has been the addition of 10 yellow comment boxes plus one appeal process box. The comment boxes are labelled with numbers 1 through 8 (there are two box number 1s and 7s because they are related to each other). Also, the circled numbers next to the arrows have been removed. Another change is the title of the Perceived Gaps column has been changed to Perceived Need.

Please note, at the bottom of the figure there is a line that says "comment boxes suggest where observed conditions and practices appear to have affected the system"

  • Comment box number 1 says "High variance found… Certain groups tend to be high risk". It is attached to the "'Nature' of the operations being regulated - levels of complexity, time, space, 'regularity' 'volume'" box on the Intermediate Outcomes row in the Contextual Factors column.
  • The next comment box says "Different groups reached…dealers of cull animals, high volume producers and some others not 'prone' to compliance". This comment box is also labelled with a 1. It is attached to "'Nature' of the operations being regulated - levels of complexity, time, space, 'regularity' 'volume'" box on the Intermediate Outcomes row in the Contextual Factors column and the "Engagement of regulated party" box on the Immediate Outcomes row.
  • Comment box number 2 says "Significantly different engagement conditions found in different cases". It is attached to "Physical environment of inspection / regulated party engagement" box on the Immediate Outcomes row in the Contextual Factors column.
  • Comment box number 3 says "Engagement of Associations has been limited but shows early signs of success". It is attached to the "Surrounding support climate for regulated party (associations, marketplaces etc.) 'cultural' factors" box on the Immediate Outcomes row in the Contextual Factors column.
  • Comment box number 4 says "History and culture around inspection… CFIA as service providers vs. regulators"; it is attached to the arrow that goes from the lower base box to the "Engagement of regulated party box".
  • Comment box number 5 says "AMP issuance can take a long time…can be 'remote' - not a consistent learning opportunity". It is attached to the "Notice of Violation (AMPs) to regulated party" box on the Immediate Outcomes row.
  • Comment box number 6 says "regional variance in issuance". It is also attached to the "Notice of Violation (AMPs) to regulated party" box.
  • Comment box number 7 says "Legal interpretations around burdens of proof… higher than expected legal engagement"; it is attached to the "Tribunal / Ministerial review decisions render clarity and fairness" box.
  • The next box says "Tribunal and court decisions raise 'the bar' re: evidence gathering"; there is a circled number 7 in this comment box. It is to the right of the results logic column at the immediate outcomes level.
  • Comment box 8 says "Gap in knowledge and confidence re: when and how to apply AMPs… increased legal involvement and evidence burden may be causing delays". It is attached to the word Respond that is part of the loop in the base box.
  • Above the "Regulated party reacts by paying penalty (or not)" box and below the "Regulated party understands infraction and changes capacity to comply", in the second results logic column, is a diamond figure with the word "Appeal?" written in it. There is an arrow going from the diamond to the "Tribunal / Ministerial review decisions render clarity and fairness" box with the word "Yes" written on it. There is an arrow going from the "Appeal" diamond to the "Regulated party understands infraction and changes capacity to comply" box with word "No" written beside it. Also, there is an arrow going from the "Regulated party reacts by paying penalty (or not)" box to the appeal diamond.

Upon review, the following observations can be made with regard to this logic.

  1. A review of AMPs violations shows that the problematic cases such as cases appealed, and decisions not upheld on appeal tended to relate to specific situations of animal transport. In these situations, the violators might have been prone to non-compliance in both will (that is, a lack of desire to comply) and capacity (the economics of the enterprise made compliance unattractive – or, more to the point – make the potential cost of non-compliance relatively attractive or worth the risk). Low-value animals being transported by broker groups appear to be a high-risk group.
  2. The complexity versus simplicity of the transaction, location, context and culture of inspection and subsequently investigation engagement appears to have an effect on compliance success. For example, complicated transportation situations involving ambiguities in the law such as the idea of what constitutes "undue suffering" and who has custody of an animal during the time the suffering occurred, and what inspection or investigation conditions constitute "undue suffering", tend to be problematic.
  3. The engagement or lack of engagement of associations in creating the support conditions for compliance, through information, education and "peer pressure", appear to make a difference. Groups without a strongly committed association appear to exhibit more problems.
  4. The history or culture around inspection, in terms of whether inspectors were accustomed to issuing actual penalties, as compared to Corrective Action Requests, was claimed by several interviewees as a contributor to compliance or non-compliance behaviours on the part of the regulators and regulatees.
  5. Relating to points 2 and 4 (and "downstream" appeal decisions as "precedence" see 7 and 8) it can take a long time to issue Notices of Violation. This reduces the ability of CFIA to reinforce messages and to create deterrence.
  6. There were significant differences in approaches to AMPs issuance across areas. This is partly due to differences in context and conditions, related to inspection and enforcement philosophy and culture, but it is also apparently due to differences in interpretation.
  7. Legal interpretations raise the evidence bar in the minds of many investigation staff. The evidentiary bar cannot be raised because it is set by statute, but interviews and file reviews suggest that evidence gathering, even when no Notice of Violation is issued, has increased over the years. Investigators attribute this to the nature of tribunals, the Federal Court of Appeal and Ministerial Reviews which appear to work against the principle of absolute liability (violations simply need to be demonstrated without intent and due diligence is not a defence) as opposed to the more common "strict" liability (where a due diligence defence is available). It would appear that the application of absolute liability has been distasteful to the legal communities involved in appeals (such as the Tribunal Chair, the Federal Court of Appeal, those responding in Ministerial Reviews) and that this has been argued to have resulted in decisions which have rendered more stringent the elements to be proven and therefore increased evidence collection to levels approaching those for prosecution in many cases. The level of information and evidence to be collected in both inspection and investigation, in order to withstand appeal challenge, has been interpreted as significant.
  8. Due to the preceding points, there is a lack of consistency, certainty, speed and efficiency in the investigation and serving of AMPs. There may be the makings of a vicious circle in the system, as uncertainty leads to more evidence collection, and potential decision inconsistency which lead to delays reducing deterrence effectiveness and reducing regulatee trust and cooperation which lead to less compliance, more challenges and potentially more uncertainties, inconsistencies and more inefficiencies.

Annex B: Evaluation Framework, Methodology and Limitations

The July 2010 evaluation framework and plan showed that there was insufficient available data for detailed cost-benefit or cost-effectiveness analysis, and that there was a need to understand how (and to what extent) AMPs work, for whom, and in what conditions. The study included AMPs at the CFIA from its inception and addressed core evaluation questions of relevance and performance.

The approach of the evaluation was guided by an emerging TBS guideline related to assessing the contribution of programs to expected results in complex environments, based on the work of John Mayne and Ray Pawson, including the key evaluation question: What works for whom in what circumstances and in what respects, and how?Footnote 25 This is also known as a "Realist" evaluation approach, which addresses outcomes by assessing, to the extent possible given the state of existing evidence, the extent that objectives were being achieved in different contexts and conditions. This approach allowed the report to identify the key principles and conditions to the success of AMPs. A realist interpretation of TBS compulsory issues suggests the questions as shown in the table below, which was included in the evaluation framework.

Core TBS Evaluation Issues Interpreted to Address "Realist" CFIA Management Needs - Relevance
Core TBS Directive Evaluation Issues Realist Evaluation Questions
Issue #1: Continued Need for program Assessment of the extent to which AMPs continue to address a demonstrable need and are responsive to the needs of Canadians What needs does AMPs appear to be organized to address? Why was AMPs initiated at CFIA? Who is served by the AMPs program? What has been the reaction to date from regulated parties?
Issue #2: Alignment with Government Priorities Assessment of the linkages between AMPs objectives and (i) federal government priorities (e.g. CDSR) and (ii) departmental strategic outcomes (i.e. Food Safety and Quality) Are there differences in the understanding of the "theory" of AMPs? Under what assumptions and circumstances does it align with federal government priorities (e.g. CDSR) and the strategic outcomes of departments? (i.e. Food Safety and Quality)
Issue #3: Alignment with Federal Roles and Responsibilities Assessment of the role and responsibilities for the federal government in delivering the program What is the understanding of the roles of key actors in the application of AMPs? Are there differences in stakeholder understanding and interpretation of those roles? How well aligned are roles and responsibilities?
Core TBS Evaluation Issues Interpreted to Address "Realist" CFIA Management Needs - Performance (effectiveness, efficiency and economy)
Core TBS Directive Evaluation Issues Realist Evaluation Questions
Issue #4: Achievement of Expected Outcomes Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and reach, program design, including the linkage and contribution of outputs to outcomes Does the program theory for AMPs tend to bend in actual usage? How? Why? How do AMPs work in different contexts and circumstances? Does the theory fare better with particular individuals, interpersonal relations, sectors and situations? What are the best practices for given situations? What are some key principles to guide AMPs application?
Integrity of the implementation chain – are AMPs elements applied consistently and cumulatively? Why or why not? How can / should this be addressed?
Issue #5: Demonstration of Efficiency and Economy Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes How does AMPs compare in terms of what it delivers to alternatives (e.g. prosecutions)? Are there more cost-effective ways of achieving the same results? What would happen without AMPs?

An evaluation advisory committee, consisting of CFIA executives from Operations Branch, Policy and Programs Branch and Corporate Management Branch, plus ex officio members from Legal Services and Health Canada, provided strategic and technical advice to the Head of Evaluation and approved the evaluation framework and final draft report. An Executive Director from the Operations Branch co-chaired the committee with the Head of Evaluation. The questions shown in the table above, included in the framework, were approved by the Committee, and were used to guide the evaluation, collect data and report findings, as seen in the organization of this report.

Study limitations were reviewed and addressed in the evaluation framework phase of the study. They can be summarized as follows:

  1. Details on costs and details of AMPs for all incumbent activities are not readily available on a consistent pan-Canadian basis. This means that accurate estimates of the breadth and depth of coverage of AMPs was not possible. This also greatly hindered the ability to pursue a design which would have involved a generalizable sample or detailed costing estimates.
  2. Roles, responsibilities and processes for delivering AMPs have been documented in various ways across areas and across CFIA groups. Furthermore, limited discussions regarding the CFIA's rationale for AMPs and its fundamental policy for AMPs were found. This limitation means that the evaluation had to rely heavily on respondent memory and third party documentation to glean even fundamental expectations regarding the use of AMPs at the CFIA.
  3. Not all of the stakeholder groups were found to know much about AMPs. This limited the ability of the study team to garner knowledgeable viewpoints from a diverse set of stakeholders.

Given these three fundamental limitations, the study team chose to adopt a "generative" approach to knowledge production, starting with an early understanding of the theory of change related to AMPs, then testing and adjusting it as knowledge accumulated over the course of the study. The accumulation of knowledge in this fashion provides generative learning about what works for whom in what conditions and why.

Data collection was undertaken from July 2010 to July 2011 and consisted of the following:

Enforcement and Investigation Services (EIS) staff and the industry associations were identified at the outset as the key informants for this evaluation, representing those administering and issuing AMPs, and those who represent those receiving them, respectively. The greatest interview concentration was placed on the office of EIS. A total of 48% of EIS staff were interviewed (22 out of 46 at time of selection). Industry associations were chosen by the significance of their members' involvement with AMPs, i.e., those with high numbers of AMPs issued. The NETS database was not capable of providing specific percentages of AMPs issued by association representation. However, file reviews, interviews and reviews of the types of AMPs issued provided strong evidence for choosing those key association interviewed. Both the program and policy staff represented an important focus of the evaluation interview program.

The survey was a sample undertaken with one of the key industry associations, which distributed the questions to its members and received responses from, according to the association, "the more significant industry players". No statistical significance could be ascertained due to the small sample size, although most of the individual responses were comprehensive and supported this report's findings.

The following evaluation matrix was included in the planning document for the evaluation, or evaluation framework, approved by the evaluation's advisory committee.

Issues, Indicators, Sources and Collection Approaches

Issue #1 Continued Need for program
Issue Indicators (Metrics / Evidence) Source Collection Approaches
1.1 Why were AMPs created? (What needs have AMPs addressed?) Documentary descriptions Originating policy documents
Decision records
Academic articles
Reviews of AMPs (Prabhu)
Content analysis of documents
1.2 Why was AMPs initiated at CFIA? Decision records
Recalled reasons by key officers (interviews, emails, records of decision, RIAS)
Originating policy documents
Decision records
Content analysis of documents
1.3 Who is served by the AMPs program? Opinions expressed by key stakeholders
Documentary descriptions
CFIA, CART and industry
Originating policy documents
Interviews
Content analysis of documents
1.4 What has been the reaction to date from regulated parties? Self assessed reactions from regulated community
Observed levels of compliance and appeals (see issue #4)
Industry sector representatives
Other stakeholders
CFIA
NETS
Documented reactions in incident reports
Interviews
Content analysis of database
Content analysis of case files
Issue #2 Alignment with Government Priorities
Issue Indicators (Metrics / Evidence) Source Collection Approaches
2.1 Are there differences in the understanding of the theory of AMPs? Observed differences in stated understanding of the use of AMPs across stakeholders CFIA viewpoints
Descriptive documents
Interviews with CFIA
Content analysis
2.2 Under what assumptions and circumstances does it align with a) Federal government priorities (e.g. CDSR) and b) the strategic outcomes of the department (Food Safety and Quality) Degree of consistency between observed AMPs use and:
  • CDSR (i.e. streamlining regulatory processes and optimizing burden)
  • Food Safety and Quality objectives
  • What about animal health, including humane treatment?

NETS and CFIA documents

CFIA, CART and other stakeholders

Content analysis of documents
Interviews
Issue #3 Alignment with Federal Roles and Responsibilities
Issue Indicators (Metrics / Evidence) Source Collection Approaches
3.1 What is the understanding of roles of key actors in application of AMPs? Perceptions of roles by key stakeholders CFIA, CART sector regulatee representatives Interviews
3.2 Are there differences in stakeholder understanding and interpretations of those roles?
Are we talking about differences within certain stakeholder groups or between types of stakeholders?
Perceptions of roles by key stakeholders CFIA, CART sector regulatee representatives Interviews
3.3 How well aligned are roles and responsibilities? Observed degree of consistency with government roles and responsibilities CFIA regulatory documents re: policy and process Content analysis
Issue #4 Achievement of Expected Outcomes
Issue Indicators (Metrics / Evidence) Source Collection Approaches
4.1 Does the theory for AMPs tend to bend in actual usage? How and why? Evidence of theory in documents
vs.
Evidence of actual usage of AMPs

CFIA decision records
Reviews of AMPs (Prabhu)

NETS database
Legal decisions
CART decisions
Academic articles
Audit and Review working papers and reports
Observed use by CFIA

Content analysis of files and literature (synthesis)

Content analysis of documents (and synthesis)
Interviews with CFIA and experts

4.2 How do AMPs work in different contexts and circumstances? # of AMPs by region, Act, sector (other?)
% payment compliance
% appeals
Process time
Level of repeat AMPs
Changes to compliance level (% of those not in compliance returning to compliance)
NETS database
Reviews of AMPs (Prabhu)
Legal decisions
CART decisions
Audit and Review reports including legal reviews
Observed use by CFIA
Reviews of AMPs (Prabhu)
Content analysis of documents (and synthesis)
Interviews with key officers and experts
4.3 Do AMPs achieve expected results with certain individuals, interpersonal relations, sectors and situations? Level of compliance (reduced non-compliance)
% of those not in compliance returning to compliance
Number of infractions
% decreased enforcement $/costs Table Note t1 [All of the above considered by Authorizing Act, region and groups.]

CFIA
Regulatee observations

CART officials
Audit and Review reports including legal reviews
Review of AMPs (Prabhu)

Interviews with CFIA, regulatee representatives, CART officials and experts
Content analysis of reviews (and synthesis)
4.4 What are some key principles to guide AMPs application? (e.g. should some of the principles discussed by legal analysts such as Goldschmidt and Priest be applied to the use of AMPs at CFIA?) Demonstrated success rate associated with key factors CFIA employee observations
Audit and Review reports including legal reviews
Academic articles
Reviews of AMPs (Prabhu)
Review observations
Experts

Interviews with CFIA

Content analysis of reviews and literature

Interviews with other stakeholders (CART, Industry, courts etc.)

4.5 Integrity of the implementation chain - are AMPs elements applied consistently and cumulatively? Why or why not? How can / should this be addressed? Described situation for AMPs in cases
Level of AMPs per volume by region, act, sector (other)
Observed level of overturned AMPs and reasons for decisions by region, Act, sector (other) and what body overturns them at what point in the justice system?
Observed level of consistency in application of AMPs (decision processes, workflows) across regions, Acts, sector (other)

NETS
CART, Ministerial and court decisions

Descriptive documents

CFIA employee recall
Internal audit working papers

Content analysis of database
Content analysis of decision records Content analysis of documents

Interviews with CFIA
Content analysis of internal audit working papers

Table Notes

Table Note t1

Note that a full cost-benefit study is not feasible at this time.

Return to table note t1  referrer

Issue #5 Demonstration of Efficiency and Economy
Issue Indicators (Metrics / Evidence) Source Collection Approaches
5.1 How does AMPs compare, in terms of what they deliver, to alternatives? (e.g. regular ticketing, prosecutions, other) Level of similarity or difference observed between AMPs and other instruments. Within Canada? GoC? International? CFIA , CART, court officials and others
Review of AMPs (Prabhu)
Interviews with key stakeholders
Content analysis of reviews (and synthesis)
5.2 Are there more cost effective ways of achieving the same results? (e.g. in combination with other instruments) Comparison of level of observed compliance and compliance change in AMPs usage as compared to the use of other instruments as noted at CFIA and elsewhere
Level of perceived effectiveness compared to other instruments
General level of costs associated with use of AMPs as compared to other approach

NETS
Reviews of compliance
Regulatory reviews
Review of AMPs (Prabhu)

Experts

Cost / expenditure data

Content analysis (and synthesis)

Interviews

Content analysis of documents
CFIA cost data

5.3 What would happen without AMPs? Level of perception of AMPs impact as compared to use of other instruments
Observed conclusions of reviews

CFIA officers, CART, Court, Experts

Reviews of AMPs (Prabhu)

Interviews with key stakeholders

Content analysis of documents / reviews

Annex C: Bibliography

Academic / Industry

Ayres, Ian, and Braithwaite, John, Responsive Regulation: Transcending the Deregulation Debate, New York: Oxford University Press, 1992. p35

Baldwin, Robert, and Black, Julia, Really Responsive Regulation, LSE Law, Society and Economy Working Papers 15, London School of Economics and Political Science, 2007

Coffee, John, Paradigms Lost: The Blurring of Criminal and Civil Law Models - And What Can Be Done About It, The Yale Law Journal 101, no. 8 (1992): 1875-1893, As cited in Australian Law Reform Commission Background Paper 7, supra note at 12

Goldschmidt, Harvey. An Evaluation of the Present and Potential use of Civil Monetary Penalties as a Sanction by Federal Administrative Agencies, in 2 Recommendations and Reports of the Administrative Conference of the United States, 1972

Mayne, John, Contribution Analysis: Addressing Cause and Effect in Forss, Marra and Schwartz editors Evaluating the Complex Transactions Publishers, 2011

Pawson, Ray, Evidence Based Policy: A realist Perspective, SAGE, 2006

Prahbu, Mohan. Efficacy of Administrative Monetary Penalties in compelling compliance with Federal Agri-Food Statutes, Faculty of Law at the University of Ottawa, 2010

Tait, Amanda. The Use of Administrative Monetary Penalties in Consumer Protection. Ottawa, The Public Interest Advocacy Centre, 2007

World Society for the Protection of Animals, Curb the Cruelty: Canada's farm animal transport system in need of repair, 2010 http://www.wspa.ca/ati/CurbtheCrueltyReport.pdf

Certified Livestock Transport (CLT) training program http://www.livestocktransport.ca/

Government of Canada

Canadian Food Inspection Agency, Compliance and Enforcement Operational Policy (2010) http://www.inspection.gc.ca/english/agen/transp/comp/pole.shtml

Canadian Food Inspection Agency, Enforcement and Compliance Policy (1999)

Cabinet directive on streamlining regulation. Ottawa: Government of Canada, 2007 http://www.tbs-sct.gc.ca/ri-qr/directive/directive00-eng.asp

Priest, Margot, Penalties and Enforcement Techniques that Promote Compliance: International Best Practices, Prepared for the CMC Enforcement Best Practices Research Group, Industry Canada, Office of Consumer Affairs, 2009

Legal/Regulatory

"Canada Agricultural Review Tribunal" Decisions and Reports http://cart-crac.gc.ca/CART-CRAC/display-afficher.do?id=1277928993362&lang=eng

Doyon v. Canada (Attorney General), 2009 FCA 152

R. v. Jarvis 2002 SCC 73

R. v. Ling 2002 SCC 74

"Proposed Regulations" Agriculture and Agri-Food Administrative Monetary Penalties Regulations, Regulatory Impact Analysis Statement, Canada Gazette Part 1 131, August 9, 1997 http://www.collectionscanada.gc.ca/databases/canada-gazette/093/001060-119.01-e.php?document_id_nbr=1572&f=p&PHPSESSID=fun3k5cvdm2dg535aktcjrl9d1

"Regulations Amending the Agriculture and Agri-Food Administrative Monetary Regulations" Regulatory Impact Analysis Statement, Canada Gazette Part II 136, no. 11 May 2002, http://www.gazette.gc.ca/archives/p2/2002/2002-05-22/pdf/g2-13611.pdf

Annex D: Interview Guides

AMPs Interview Questions

  1. What has been your experience with AMPs? (time, types, what Acts, general roles played)
  2. How have regulated parties responded to AMPs implementation over time? Why do you say that? (Reference specific case examples and/or data analysis performed on compliance rates, payments, appeals etc.)
  3. In your view, how consistent is AMPs with CFIA's mandate?
  4. What are the key roles to be played in AMPs delivery by important actors? Please name each key actor and their role.
  5. Are there differences in stakeholder understanding and interpretations of those roles, including CFIA employees? What are these?
  6. How have AMPs been used over time? Has this changed? How?
  7. From your experience with AMPs - do they work differently for the different Acts and situations? How? Why?
  8. How well have AMPs worked in general and in specific areas? (individuals, interpersonal relations, sectors and situations)
  9. In general and in your view, where do AMPs work best? Why? Do you have recommendations regarding where, with whom and how to use AMPs?
  10. How do AMPs compare, in terms of what they deliver, to alternatives like regular ticketing, prosecutions, other? Why ? How does the comparison differ by area of application? (How do AMPs compare in each key area of context ?- i.e different Acts and different process situations.)
  11. Are there more cost effective ways of achieving the same results? (e.g. in combination with other instruments) Why?
  12. What would happen without AMPs? In your opinion, if AMPs were stopped tomorrow - what difference would it make?

AMPs Regional Interview Questions

  1. What has been your experience with AMPs? (time, types, what Acts, general roles played)
  2. Why were AMPs used in your situation? (Needs addressed, what was the basic intent? )
  3. How are AMPs applied in your Region? (Refer to background materials and previous write-ups where available.) How does this differ - to your knowledge - from other groups? Why?
  4. How have regulated parties responded to AMPs implementation over time? Why do you say that? (Reference specific case examples and/or data analysis performed on compliance rates, payments, appeals etc)
  5. How consistently is the intent of AMPs and its mechanisms understood? Why do you say that?
  6. In your view, how consistent is AMPs with CFIA's overall intent in your area? (i.e. the Key Acts you support)
  7. What are the key roles to be played in AMPs delivery by important actors? Please name each key actor and their role
  8. Are there differences in stakeholder understanding and interpretations of those roles? What are these?
  9. Are roles and responsibilities - as laid out in your situation - appropriately aligned? Why do you say that?
  10. How have AMPs been used over time? Has this changed? How?
  11. From your experience with AMPs - do they work differently for the different Acts you apply and situations you face? How? Why?
  12. How well have AMPs worked in general and in specific areas? (Probe for differences with certain individuals, interpersonal relations, sectors and situations.)
  13. In general and in your view, where do AMPs work best? Why? Do you have recommendations regarding where, with whom and how to use AMPs?
  14. How do AMPs compare, in terms of what they deliver, to alternatives like regular ticketing, prosecutions, other? Why ? How does the comparison differ by area of application? (How do AMPs compare in each key area of context ?- i.e. different Acts and different process situations.)
  15. Are there more cost effective ways of achieving the same results? (e.g. in combination with other instruments) Why?
  16. What would happen without AMPs? In your opinion, if AMPs were stopped tomorrow - what difference would it make?
  17. What improvement should be made to NETS to make it into a sufficient information management system to support AMPs?
Date modified: