ACOT Continuing Competence Program Review and Evaluation Survey

Introduction

Having a Continuing Competence Program (CCP) is a requirement within Alberta’s Health Professions Act (the HPA) and the Occupational Therapists Profession Regulation (the Regulation). A CCP is one tool regulatory bodies use to ensure that the public receives safe and competent services from regulated health professionals. Section 13 of the Regulation specifies the required components of a CCP for occupational therapists (OTs) in Alberta including the annual completion of:
a) a practice challenge log;
b) a self-assessment questionnaire;
c) a competence maintenance log; and,
d) a continuing competence portfolio.

Completion of all the CCP components does not directly measure a registrant’s competence. However, the reflective practice and demonstration of continuous learning that CCP completion requires is considered a proxy of OT competence. Although every OT is ultimately responsible for ensuring their professional practice adheres to ACOT’s Standards of Practice and Code of Ethics, the annual completion of the required CCP components by each registrant is ACOT’s way of indirectly monitoring the competence of OTs as part of a regulatory body’s mandate to protect the public.

Demonstrating that the methods/tools used by regulators to monitor the competence of self-regulating professionals are an accurate reflection of actual competence is a challenge facing all colleges and regulatory bodies internationally. ACOT is committed to ensuring that our CCP is the most accurate proxy of registrant competence as possible. Conducting reviews and evaluations of CCP submissions will help us to demonstrate that the reflective practice approach of ACOT’s CCP is a reliable proxy.

Review and Evaluation of CCP Submissions

The Regulation specifies:

14(1) The Registrar or Competence Committee must periodically select regulated members in accordance with criteria established by the Council for a review and evaluation of all or part of a regulated member’s continuing competence program.

Historically, the term audit was used to describe the “review and evaluation” of a regulated member’s CCP submission. Going forward, ACOT will resume the use of the "review and evaluation" language to reflect ACOT's focus on supporting OTs to submit CCP information that accurately reflects their efforts to improve their practice each year.

ACOT’s History of CCP Submission Audits:

From 2007-2011, ACOT’s process included the review of approximately twenty percent of registrant CCP files each year. Registrants were randomly selected for review and were notified by mail to submit their CCP files from the past two years for review. Reviewers were recruited from an initial CCP submission review pilot in 2007 from OTs who had “exemplary” CCP submissions. Reviewers were trained beforehand to ensure inter-rater reliability. Reviewers would indicate whether a submission was complete or incomplete; additional information was requested and feedback was offered if required/appropriate.

Prior to the launch of the online version of the CCP for the 2014-2015 registration year, a legal opinion was sought to interpret Section 14(1) of the Regulation to determine if individual-level audits of CCP submissions were required or if an anonymized, program-level audit would suffice to meet the “review and evaluation” requirements of the Regulation. Legal Counsel advised that program-level audits would suffice, and individual-level audits were discontinued at that time.

Program-level audits occurred in 2016 and 2018. The results of the 2016 audit lead to the creation of resources to support registered members to complete the CCP components as intended (CCP Completion Guide and video tutorial). The 2018 audit revealed a broad variance in how the CCP components were completed by the selected registrants and the reviewers recommended the resumption of individual-level audits. In response to this recommendation, ACOT’s Council asked for the Competence Committee to develop and/or refresh the policies and procedures for CCP submission review and evaluations.

Purpose of Survey

Prior to the development of review and evaluation policies and procedures, ACOT’s Competence Committee would like to gather your input and ideas about the process for CCP submission review used in the past and your suggestions on what would be useful to incorporate in the future. Please note that you will be given ample notice prior to the resumption of CCP submission reviews.

All responses to this survey will be used in aggregate and will remain anonymous.

Question Title

* 1. How long have you practiced as an OT?

0-2 years

3-5 years

6-10 years

11-20 years

21-30 years

31-40 years

>40 years

Question Title

* 2. How long have you been registered to practice OT in Alberta?

0-2 years

3-5 years

6-10 years

11-20 years

21-30 years

31-40 years

>40 years

Processes Used in the Past

Question Title

* 3. Have you had your CCP submission individually reviewed by ACOT in the past?

Yes

Don't Recall

If you answered yes to #3, please answer questions 4 through 6, otherwise skip to question 7.

Question Title

* 4. Were the communications (letter, feedback) about your CCP submission useful?

Yes

Please elaborate:

Question Title

* 5. What impact, if any, did the review process have on your OT practice?

None

Very Little

Moderate

Significant

Don't Recall

Please elaborate:

Question Title

* 6. Please include any additional feedback you'd like to share about your prior CCP submission review and evaluation experience.

Question Title

* 7. Have you been involved in a review of your CCP submission in other provinces related to your OT practice?

Yes

If no to question 7, skip question 8.

Question Title

* 8. If yes to question 7, was there anything in that process that you'd like ACOT to incorporate? (Please describe)

What Would be Useful Going Forward

We plan to use a systematic and transparent randomization process, so that people won’t be selected for CCP submission review and evaluation in successive years unless support was provided to complete a submission during a prior year’s review.

Current literature indicates that use of reflections on learning as a component of continuing competence programs is correlated to practice maintenance and improvement. As such we anticipate the initial and goal completion status reflections, as well the summary of learning in each Learning Record, could be the main focus of a CCP submission review.

We will use a detailed and objective CCP submission review rubric to guide evaluation of reflections and summaries of learnings. Please refer to the draft CCP Submission Rubric before answering the following questions.

Question Title

* 9. Would the draft rubric adequately guide you in completing your CCP components to an acceptable rating?

Yes

Please elaborate:

Question Title

* 10. Is there anything else you’d like to see on the rubric beyond what is in the draft?

Yes

If you selected yes, please specify additions you'd like to see in the draft rubric:

Question Title

* 11. Please indicate any other ways ACOT’s CCP submission evaluation and review processes could support you in maintaining/evolving your OT practice in alignment with ACOT’s Standards of Practice and Code of Ethics (Select all that interest you):

Ability to meet with reviewer/ACOT staff to receive feedback directly (versus in writing)

Ability to request CCP submission review outside of random selection process

Monthly or quarterly virtual meetups/Q&A forums with ACOT staff specific to CCP completion

Additional CCP Resources beyond what is already available on the ACOT website (e.g. “how-to” webinars, podcasts, section-by-section guides)

Other (please specify)

Thank you for completing the survey! Aggregate survey results will be used to inform development of ACOT's CCP review and evaluation policies and procedures.