Internal Audit Report
Audit of CFIA External Stakeholder Complaints Process
3.0 Findings and Recommendations

3.1 Complaints Handling Processes and Controls

The CAO has established complaint handling processes which incorporate controls to help ensure due diligence and a timely response. Key interactions with the complainant and branch representatives are made throughout the complaint process. In addition, the branch, CAO Director and Chief Redress Officer are engaged prior to the final outcome being made to help ensure technical accuracy and objectivity.

Process and control efficiency was assessed through a review of the standard operating procedures and information management practices, and a mapping of processes and controls. The audit found that complaint handling processes were free of unnecessary redundancies and excessive controls. Although the processes and controls were found to be generally efficient, the CAO's information management practices include the maintenance of both paper and electronic files. This practice represents a process inefficiency that could impact workload and records management within the CAO.

Policies and Procedures

The Complaints and Appeals Administrative Policy captures the roles and responsibilities of the CFIA branches, the CAO, and the Chief Redress Officer. The branch heads and their management teams are responsible for reviewing and gathering information on complaints referred to them by the CAO. The CAO requests that each branch identifies a contact (branch representative) responsible for sending feedback and information to the CAO. The CAO's established "single-window" communication approach is an effective practice given that it encourages responses in a timely and streamlined manner.

The Standard Operating Procedure (SOP) for Complaints and Appeals processed by the Complaints and Appeals Office (December 2015), provides guidance regarding the implementation of the Complaints and Appeals Administrative Policy. The SOP details the inputting of complaints into a database, triaging, reviewing complaints, roles and responsibilities, communications, service standards, and the process for closing complaint files and reporting.

CAO policies and procedures were found to be well-designed and complete to support the office and the discharge of staff responsibilities.

File Documentation

The audit reviewed a sample of 20 files to determine if processes and controls were operating in accordance with CAO standard operating procedures. The results of the sample testing found that standard operating procedures and controls were being followed, specifically in the areas of: soliciting input from branches, approvals from the CAO Director and Chief Redress Officer and informing the branch of the final outcome.

Files were well-organized and structured in a logical manner; however, not all documentation was consistently retained in the complaint files. This included evidence of initial telephone conversations with the complainant and, where applicable, evidence of technical review by the applicable branch. Although CAO paper files contain a checklist of analyst procedures, it did not contain a list of what final complaint review documents should be retained in the master file (paper or electronic).

Recommendation 1.0:

The Chief Redress Officer should clarify file documentation requirements and ensure all files adhere to these requirements.

Training and Development

Training and development is fundamental to establishing and maintaining capacity to deliver programs.

Upon hire, CAO Analysts are provided with the CAO Process Binder, which contains information on policies and procedures, applicable regulations, tools and templates. The CAO Analysts review this binder and familiarize themselves with their responsibilities. Following the review of this information, new CAO Analysts will "job shadow" existing Analysts in order to fully understand the complaints process from beginning to end.

Regarding ongoing training, CAO Analysts develop annual training plans. The annual training plans reviewed included courses pertaining to business writing, dealing with clients, and negotiations.

Information Security

The audit expected to find that the CAO files were appropriately labelled, stored and secured throughout the complaints handling process.

The Complaints and Appeals Office is located in a restricted area requiring authorized access to enter. Within the CAO offices, there are RCMP-approved security containers with an integral combination lock which store the hard copy of the complaints files.

Complaint files are classified as either Protected A or Protected B. The default classification for complaint files is Protected A (defined as unauthorized release could cause injury to an individual, an organization or the Government). A complaint file is classified Protected B if sensitive information on an employee is mentioned (unauthorized release could cause serious injury to an individual, an organization or the Government). Protected A or B files should be marked on the top right corner of the file folder containing hard copy complaint information.

Of the 20 files sampled, two had employee information that escalated the file to Protected B status. These two files were labelled Protected B, but the remaining 18 files that should have been labelled Protected A were not.

At the end of every complaint process the CAO Analyst writes a final report that summarizes the complaint, the process followed, and the result of the review. The names and contact information of the complainants must not be included in the final report. In the 20 files reviewed, all final reports excluded complainant names and contact information in accordance with the Privacy Act.

Information regarding the complaint is also stored electronically within an electronic database called ccmMercury/TRECS. A user access listing was reviewed to verify that only authorized users within the CAO had access to the database. The audit found one exception, for which management has taken immediate action to remove access.

Recommendation 2.0:

The Chief Redress Officer should ensure all complaint files meet the requirements of the CFIA Security Management Policy with respect to the security marking of files. Additionally, the Chief Redress Officer should periodically review system access to the CAO's electronic database.

3.2 Communication and Awareness

The CAO is responsible for delivering information sessions on complaints processes with internal and external stakeholders in order to raise awareness of the CAO's mandate and processes.

A primary benefit of the communication of the complaints process is that the CAO has the opportunity to increase the visibility of stakeholder complaints and help find solutions. Internally, educating CFIA employees about the CAO allows them to appropriately refer stakeholders with issues to the CAO. Educating external CFIA stakeholders about the complaints process enables them to bring their concerns forward to CFIA which can help improve CFIA's service delivery, enhance regulatory oversight and contribute to a stronger relationship between CFIA and its stakeholders.

The CAO engages both internal and external stakeholders through various types of outreach activities. These outreach activities include attending conferences where external stakeholders will be present, giving presentations to both internal and external stakeholders, publishing and distributing the CAO's annual report, and posting information regarding the CAO and its processes on CFIA's internal and external websites. The audit found that the CAO complaints review process, service standards and applicable forms were made available on both the external website and the intranet (Merlin) as a reference tool for stakeholders and CFIA employees.

While the communication and awareness function of the CAO is a requirement as outlined in the Complaints and Appeals Administrative Policy, the audit found that the CAO did not have a communications strategy in place for fiscal years 2014-15 and 2015-16. The lack of a communication strategy could result in stakeholders not being targeted by outreach activities, or outreach activities that are misaligned with CAO's mandate. Although the CAO complaints process is outlined on the external website, there is an opportunity to further enhance awareness and provide a more strategic distribution of outreach activities. Through discussion with the CAO, there was no documented communications strategy to determine and prioritize outreach objectives including targeted stakeholders.

Nevertheless, the outreach activities that have taken place are captured using a tracking tool that outlines the type of outreach activity, the stakeholder name and the date on which the outreach activity occurred. The outreach tracker indicated that a large majority of internal outreach activities occurred in the NCR and that no external outreach activities occurred during fiscal year 2015-16. CAO management indicated that the concentration of outreach activities within the NCR was due to a limited travel budget.

Recommendation 3.0:

The Chief Redress Officer should develop and implement a communications strategy to help better align communication activities with identified priorities.

3.3 Performance Measures and Service Standards

Performance Measures

A performance measure is a qualitative or quantitative means of measuring an output or outcome with the intention of gauging the performance of a program. As part of its Results and Delivery agenda, the Government of Canada has made performance measurement a top priority and is critical to the success of any organization. The audit found that the CAO has a well-defined performance measurement strategy framework in place that can facilitate evaluation of the complaints process.

To develop the CAO's performance measurement strategy framework, management first developed a logic model which is a visual expression of the rationale behind a program and serves as the program's road map. It outlines the intended results (i.e. outcomes) of the program, the activities the program will undertake and the outputs it intends to produce in achieving the expected outcomes. It is also expected that targets are established for each performance measure to help evaluate the complaints process.

The CAO identified 19 performance measures that were directly linked to the planned results. Performance measures were found to be measurable as a numerical value was associated with each measure. Performance against targets was calculated and reported at year-end.

As per Treasury Board's Guide to Developing Performance Measurement Strategies, following the first year of program implementation, a review of the performance measurement strategy framework should be undertaken to ensure that the appropriate information is being collected and captured to meet both program management and evaluation needs. The CAO first implemented their performance measurement strategy framework in fiscal year 2015-16. To-date, the CAO has not undertaken a review of its strategy, including performance measures and targets.

Service Standards

The Treasury Board of Canada Secretariat defines a service standard to be a public commitment to a measurable level of performance that clients can expect under normal circumstances. Service standards serve two key purposes: to provide staff with performance targets and to inform clients what to expect. They help clarify expectations for clients and employees, drive service improvement, and contribute to results-based management.

The CAO has established service standards for the acknowledgement of receipt of all complaints, comments and compliments as well as for the amount of time needed to review a complaint. These service standards are communicated to the general public and employees through the CFIA external and internal websites.

Measuring these services to ensure a timely and appropriate delivery is an important element of ensuring CAO's service management excellence. The service standards were found to be measureable as they are based on the number of business days outstanding from receipt of the complaint to the final closing letter. They also align with the overall objectives allowing for an efficient, accessible and timely system for processing of complaints.

CAO has established a performance target of 90% for both service standards. It was reported in the draft Complaints and Appeals Office 2014-2016 Biennial Report that the CAO met the service standard of acknowledging receipt of the complaint within two days 99% of the time; however, the service standard related to the 30 business days to complete a review of a complaint did not meet the established target. The CAO has not yet formally analyzed the root cause of the service standard not being met in order to identify opportunities for continuous improvement.

Recommendation 4.0:

The Chief Redress Officer should undertake a review of the CAO's performance measurement strategy framework to ensure that objectives and performance expectations are appropriate and being met.

3.4 Continuous Improvement

A CAO objective is to use the information gathered on complaints to facilitate continuous improvement. The CAO does this primarily in two manners: through "Branch Email Closings" and through "Opportunity for Improvement" management letters.

The Complaints and Appeals Administrative Policy indicates that the identified repetitive issues within CFIA and opportunities for improvement will be shared with senior management and used by the Agency to develop and enhance work plans for improving programs and policies, tool development and service delivery.

Branch Email Closings (BECs) are emails sent by the CAO to the applicable Branch Executive Director at the end of each complaint process. The purpose of the BEC is to inform the Branch that the complaint file has been closed and to provide a final summary of the complaint.

Annually, the CAO provides Opportunity for Improvement (OFI) management letters to a designated representative, at the Executive Director level, in branches that had been involved in complaints that year. The OFI management letters are intended to capture key observations that the CAO made throughout the year. The observations relate to complaints that could have a broader impact on the Agency or complaints that were received multiple times from different stakeholders. These letters contain key information that could be used to improve service delivery, enhance regulatory oversight or contribute to stronger relationships with stakeholders. The process to complete the OFI management letter involves reviewing all the complaint files for the year, identifying common themes and/or systemic issues and categorizing complaints as observations for the applicable branch. The CAO conducted extensive analysis of historical themes and trends which is used as an input to the OFI process. Although the OFI management letters were sent to impacted branches for the 2014-2015 fiscal year, the letters were not issued in a timely manner.

In 2014-2015, OFI management letters were sent to Executive Directors in Operations Branch and Programs and Policy Branch (PPB). One OFI letter identified six areas for improvement; only two areas were responded to, the remaining four areas were left with a nil response. The other OFI letter identified three OFI areas, of which all were responded to. The audit concluded that where OFI responses were not received, branch actions in support of continuous improvement could not always be determined. As at the end of the audit's examination phase (August 2016), the CAO's 2015-2016 OFI management letters had not been sent to applicable branches.

Specific accountabilities, roles and responsibilities pertaining to the tracking or follow-up of management actions to address observations communicated through the OFI management letter have not been established. Furthermore, there is no mechanism in place to inform Agency executives above the Executive Director level, including the President, Branch Vice Presidents and CFIA governance committees, of themes and systemic issues identified as a result of stakeholder complaints. Such a mechanism would facilitate Agency-wide discussions on significant issues.

Recommendation 5.0:

The Chief Redress Officer should implement a strategy and process to broadly inform senior management and CFIA oversight bodies on key themes and systemic issues raised through the complaints process to help promote continuous improvement.