Centre for University Teaching Resources for Program Review
Program Review at the University of Ottawa
The strategies, resources, and tools used at the University of Ottawa in the context of cyclical program review and curriculum development have evolved quite a bit over the last 5-6 years. The basis of the pedagogical support we use today at the Centre for Innovative Pedagogies and Digital Learning (CIPDL) was initially developed by Bob Parson. Since 2012, many practices including the SWOC/SOAR analysis focus groups (for faculty and students) and curriculum analysis data collection now have an online option. As such, we have developed surveys, templates and guides to assist programs with the collection and analysis of data which will help them describe the structure and function of their program. We outline a brief snapshot of our process and tools below.
Each academic program at the University of Ottawa must follow the protocol for cyclical review every 7-8 years. Consisting of an initial self-evaluation of the program, internal recommendations by a Senate committee, external review and a final recommendation report, the timeline is structured as follows:
Figure 1: University of Ottawa timeline for the cyclical review of undergraduate programs
What is Curriculum Mapping?
Curriculum mapping is the visual representation of the underlying logic of curricular design for a given program (Maki, 2004). In higher education, it is most frequently used for two main purposes:
- To ensure the alignment and sequencing of learning outcomes and assessments across courses when developing a new program; and
- To evaluate the current alignment and look for any gaps, redundancies and inconsistencies in order to enhance an existing program (Uchiyama & Radin, 2009; Kopera-Frye, Mahaffy & Svare, 2008).
Curriculum mapping is “a deliberate process of curriculum deconstruction in order to understand better how the sum of the parts relates to the whole” (Jackson, 2000, p.144).
Curriculum Mapping in Five Stages
An evidence-based approach to collaborative curricular enhancement can commonly be characterised by five stages:
Figure 2: Five stages of curriculum enhancement
Stage 1 – Data Collection and Analysis
Using a Canadian based internet survey tool (Fluidsurvey), instructors identify which program learning outcomes are covered by their courses, at which level these outcomes are presented (introductory, intermediate, advanced), how the outcome is covered (taught, practiced, assessed), and if assessed, by what means.
Figure 3: Screen capture of the learning outcome section of the survey
Figure 4: Screen capture of the evaluation methods section of the survey
Stage 2 – Interpretation and Discussion
Once the course information is collected, tables can be generated within a spreadsheet to better visualize the program’s underlying framework. Current tables include the integration of learning outcomes across program courses, assessment methods used throughout the program, grade weight and distribution of assessments across the program, favoured instructional approaches, etc.
Figure 5: Sample learning outcome distribution table
Figure 6: Sample assessment distribution table
Stage 3 – Identification of Areas for Improvement
As a catalyst for reflection and interdepartmental discussion, a series of questions are provided (Kenny, 2014). These include:
- What learning outcomes are most/least emphasized?
- Where are the strengths and gaps in teaching and assessment across the program?
- Do the instructional and assessment methods used best align with the intended learning outcomes?
- What instructional/assessment strategies are most/least used?
- Are the instructional and assessment methods used in the courses congruent with the discipline and the program’s/Institution’s mission/vision?
- In terms of supporting student learning, how well are the selected/utilized instructional and assessment methods actually working?
Stages 4 and 5 - Development of an Action Plan and Implementation
An action plan is most effective when it is faculty driven, data informed and supported by curriculum design specialists, and includes specific responsibilities and timelines as well as mechanisms to sustain the curriculum review and enhancement process by systematically revisiting the five stage cycle (Wolf, 2007).
Resources / Curriculum Analysis Templates
Seamless Data Collection: uoSyllabus
As these practices evolve at the Centre for Innovative Pedagogies and Digital Learning (CIPDL), the questionnaire continues to grow and become more robust. Of particular interest is how to collect information regarding student achievement of program learning outcomes and track progress on a continuous basis.
A tool called uoSyllabus being tested in a pilot phase may help standardise the collection of this information. uoSyllabus is an online module allowing instructors to produce syllabi that conform with the rules outlined by Senate and offer the possibility of adding content which aligns with best practices. Once the information is entered, the module produces a syllabus in a PDF format.
The module offers the possibility to specify which learning outcomes, accreditation standards or attributes are covered by each course within a program, thus facilitating data collection for various accreditation reports which many programs (including programs in professional schools) are subject to. The module equally enables curriculum design specialists to continuously collect information, thus facilitating the analysis of curricular alignment for the purpose of program evaluation.
uoSyllabus is now being tested by four professional Masters programs from the School of Rehabilitation Sciences: Audiology, Speech-Language Pathology, Occupational Therapy and Physiotherapy. All programs have accreditation standards that are specified in uoSyllabus, thus making it possible to identify how these standards are met by program courses.
List of uoSyllabus Advantages
|Course syllabi are accessible (WCAG 2.0, level 2A)||Course syllabi are accessible (WCAG 2.0, level 2A)||Course syllabi are accessible (Accessibility for Ontarians with Disabilities Act, 2005)|
|Information is easily identifiable||uoSyllabus online module is accessible (WCAG 2.0, level 2A)||Centralised database of PLOs|
|Automatically sent to students via uoZone (online student portal)||Online management of past, present and future syllabi||Control panel for Program Chairs with pre-formatted statistic tables of PLOs|
|Public syllabi are searchable||User-friendly navigation||Responds to Senate requirements (uOttawa academic regulation 8.5)|
|Public syllabi are searchable||Accurate image of how PLOs are covered by programs|
|Multiple professors can contribute to a course syllabus||Longitudinal data (instead of a snapshot taken every 7-8 years)|
List of Pre-Formatted Data Accessible Through the Control Panel
|All data and crosstabs below are by course code letters (e.g. PHT)
By course code letters AND 1st number of course code numbers (1, 2, 3, 4 and so on)
|Descriptive (1 variable)
|Crosstabs (2 variables):
List of courses in uoSyllabus (Letters, numbers, section) by semester (e.g. 20149)
Number of times each PLO is covered by courses
Methods of assessment
Value of assessments
Week of assessment
Course learning outcomes x PLO
For each PLO, crosstab with:
Type of assessment
Level of inclusion (Principal or Secondary) OR (Introduction, Reinforcement or Advanced)
Development (7 combinations of taught practiced and assessed= NT-NP-A, NT-P-A, NT-P-NA, T-NP-NA, T-NP-A, T-P-NA, T-P-A)
PLOs (count) x Level of inclusion
PLOs (count) x Development
Value of assessment x Type of assessments
Value of assessment x Week of assessments
Crosstabs (3 variables):
PLOs (count) x type of assessment x week
PLOs (count) x type of assessment x percentage
PLOs (count) x week x percentage
PLOs (count) x Level of inclusion x Development
Upcoming Development of uoSyllabus
- All information regarding PLOs will be centralised under one tab (Curriculum Tab) allowing for the collection of more refined data about how they are taught and evaluated.
- Ability for faculty to enter the average grade for each course assessment.
- Faculty members will be able to enter the average result obtained for each evaluation (giving us a first look at which PLOs are most or least achieved by students).
- Through a central system, we will be able to link the mean course grade to PLOs
- Manager access for administrative staff in departments will allow one to:
- Add user / prof
- Ability to see and edit all course syllabi for their programs
- Add program learning outcomes
- Add instructional approaches
- Add assessment methods
- Two types of access to uoSyllabus: Regular mode and Course Design mode.
Resources / uoSyllabus
Online SWOC Analysis
In this two-hour session, participants identify the elements that are most important to their stakeholders and take part in an enriching dialogue amongst peers, discussing the program in a constructive fashion. As well, participants produce concrete suggestions on what could or should be done to improve the program.
In previous SWOC workshops for programs under review, participants indicated that the SWOC :
- Created a concise vision of the program
- Brought new ideas to the discussion
- Generated active participation
- Encouraged recognition of what was positive about the program
- Confirmed common goals and aims
By invitation, the CIPDL will facilitate a SWOC analysis discussions with full time/part time faculty or students and provide a written report of findings. A meeting space may also be provided by the Centre for Innovative Pedagogies and Digital Learning (CIPDL).
For certain programs, reaching students/faculty can be challenging. Increasingly, programs are choosing to move this process online and thus gather even more input from students and Faculty members, using an online SWOC to ask questions that normally are not part of a SWOC analysis.
Resources SWOC / Templates and Examples
For greater detail or if you have questions or comments, please get in touch with us.
As Curriculum Design and Quality of Learning Specialist, Jovan Groen, Acting Director of the Centre for Innovative Pedagogies and Digital Learning (CIPDL), works closely with the various faculties and departments in different processes of assessment, program development and review at the undergraduate level. In addition to working at the program level, Jovan collaborates with faculty in different professional development activities such as educational workshops, personalized support and guidance, and ongoing research on innovative pedagogical practices. With a background in both the physical sciences and educational psychology, Jovan has worked at various levels within the public school system and as an instructor and consultant within the university setting.
Patrick Milot is a Graduate Curriculum Design and Professional Skills Development Specialist at the Centre for Innovative Pedagogies and Digital Learning (CIPDL) and at the Faculty of Graduate and Postdoctoral Studies. He provides support services for graduate programs in evaluation and for the creation and development of masters and doctoral programs. Patrick has over 20 years of experience in education, having taught at the high school, college and university level. He also managed higher education programs and faculty development projects.
Jackson, N. (2000). Programme specification and its role in promoting an outcomes model of learning. Active Learning in Higher Education, 1(2), 132-151.
Kenny, N. (2014). Curriculum Mapping. Retrieved from http://www.uoguelph.ca/vpacademic/avpa/outcomes/curriculummap.php
Kopera-Frye, K., Mahaffy, J. & Svare, G.M. (2008). The map to curriculum alignment and improvement. Collected Essays on Teaching and Learning, 1, 8-14.
Maki, P. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.
Uchiyama, K.P. & Radin, J.L. (2009). Curriculum mapping in higher education: A vehicle for collaboration. Innovative Higher Education, 33, 271-280.
Wolf, P. (2007). A model for facilitating curriculum development in higher education. New Directions for Teaching and Learning, 112, 15-20.