Pilot Background
In the early 2000s, The Claremont Colleges deployed a variety of Learning Management Systems (LMS) solutions – Moodle at Pomona College, WebCT at Claremont McKenna College, and Sakai at Harvey Mudd College. Sometime around 2005 or 2006, the Consortium standardized on one LMS, Sakai, and it has been in use continually by most of the Consortium since then. However, over the past seven years, several schools, Claremont Graduate University, Keck Graduate Institute, and significantly, the first undergraduate school, Scripps College, moved to Canvas. Given the value of consortial collaboration and supporting cross-registration, Claremont Consortium faculty and administrative governing bodies, have been interested in returning the Consortium to a single LMS platform in order to provide a common teaching and learning environment for our students and faculty.
In late Spring 2022 Pomona College ITS began planning a pilot of Instructure’s Canvas LMS to gain valuable information on the experience of Pomona instructors and students, test the Canvas environment, and build ITS capacity. A total of 25 courses, 300 students and 16 instructors used Canvas during the Fall 2022 semester. Four instructors were brand new to Canvas and worked as a cohort with ITS staff over the summer and twelve were incoming new instructors who were already familiar with using Canvas and expressed interest in joining the pilot.
Evaluation
To evaluate the implementation of Canvas, ITS designed three surveys: one for the four Cohort instructors, one for the twelve “Do it yourself” (DIY) instructors, and one for all the students enrolled in a Canvas course at Pomona this past fall. Each survey included questions on satisfaction, use, impact, and support as well as areas to leave open-ended responses.
Feedback from all participants will inform us on the technological and pedagogical implications of Canvas implementation and help us determine future directions. Together with our faculty, students, administrators, and staff, we can explore costs and benefits of a potential move to Canvas and a return to a common LMS experience.
Survey Results
Participation on all three surveys was good, especially from the instructors. ITS obtained a response rate of 100% from the Cohort instructors (4/4), 67% from DIY instructors (8/12) and 24% from the students 71/300).
Below are some highlights and interesting findings from the various quantitative and qualitative responses.
Faculty Feedback
Use
There were similar findings between DIY and Cohort instructors when it came to Canvas use. When asked how they use Canvas, 100% of instructors reported having used Canvas primarily to upload course readings and other document files as well as adding assignments and giving back feedback on those assignments. Across the board Canvas was least used for quizzes and/or exams and for virtual office hours.
Figure 1: Survey results from DIY Instructors and Cohort Instructors on Canvas use.
Satisfaction
Figure 2: Survey results on recommending Canvas to other colleagues who are using Sakai.
When asked if they would recommend Canvas to their colleagues who are using Sakai, an overwhelming 91.66% of faculty respondents to the Canvas Pilot Surveys overwhelmingly stated that they strongly agreed with this statement, while the remaining 8.33% of faculty agreed with the statement (figure 2).
Overall, satisfaction with Canvas is high among instructors. A combined 100% of instructor respondents indicated that they were either “very satisfied” or “satisfied” with Canvas (figure 3). When asked to elaborate on this rating instructors mostly pointed out that Canvas was “easiest to use and re-use”, “intuitive”, and helped “improve course design.”
I love how easy it is to integrate tools with Canvas.
It has everything I could need!
The Canvas experience is seamless!…Having everything integrated has made for effective teaching and learning.
Canvas is easy and intuitive to use…
Impact
Figure 4: Survey results on whether Canvas has had a positive impact on teaching.
Data indicates that Canvas had an overwhelmingly positive impact on instructors in their teaching and learning experiences with the majority of instructor respondents (75%) choosing “strongly agreed” on the Likert scale (figure 4).
Qualitative responses indicate the biggest impact was file management as instructors reported that Canvas LMS was easier to upload and manage files. Another area where Canvas had a positive impact was managing assignments; the ability to organize and track assignments more easily.
Using Canvas helped me make both classes more resource and media rich.
Streamlined the course so all assignments, readings, documents, etc. for the week were in one place. Much easier for students to find everything and stay on top of the course. Much easier and less time consuming for me to manage.
Support
Both groups of instructors felt well supported by both RITG and Instructure. 75% of Cohort instructors reported they were very satisfied with support from Instructure and 100% were either very satisfied or satisfied with the support from RITG. DIY instructors reported similar results with 62.5% stating they were very satisfied with support from Instructure and 75% were either very satisfied or satisfied with the support from RITG.
Please rate your overall satisfaction with Canvas support from Pomona ITS-RITG.
Please rate your overall satisfaction with Canvas support (from Instructure, not Pomona ITS).
Great experience. Thank you for your time and work!
Thank you for all your help and support! Working with RITG has been wonderful!
I’m very happy with the outcome. Thank you so much!
The ITS Canvas Team has been very responsive and helpful! Thank you so much!…
Additional Faculty Feedback
The following list includes some comments instructors left throughout different parts of the survey:
- “I appreciate you opening it up to new faculty. Having come here from another institution that switched to Canvas, some years ago I was surprised to learn Pomona was still using Sakai.”
- In general I would say I’m very satisfied. One feature I still miss from Sakai is the Dropbox, but otherwise Canvas worked very well for my course.
- I wish access to the grade book was not as wonky. When you are in the grade book, you have to re-enter the class to access other things.”
- “I’d like to keep with the pilot! Using Canvas has made it much easier to teach!”
- “There were some challenges with access for students from the other 5Cs, but they were worked out and I would imagine that these would not be an issue once Canvas is more fully adopted and integrated.”
- “I am so happy to not have to return to Sakai after several years on Canvas!”
Student Feedback
Use
Like instructors, students were also asked to indicate how they use Canvas. Over 95% of students reported having used it to access course readings and submit assignments which is similar to instructor findings. Between 50% and 75% of students use Canvas for viewing grades, the course calendar with due dates as well as streaming media and/or other course media. Canvas was least used for quizzes and/or exams and for communication between professors and classmates.
Satisfaction
The majority of students were positive when asked to rate their overall satisfaction with a combined 88.4% selecting either “very satisfied” or “satisfied”. When asked to elaborate on this rating many students praised Canvas for its user-friendliness, with comments like “no problems” and “easy to use” being common. A few students even compared Canvas favorably to Sakai by pointing out added benefits.
Figure 7: Survey results on overall satisfaction with Canvas.
I find myself more likely to use the class site and review materials on Canvas rather than Sakai. I also think the way Canvas has in-site architecture to view documents to store different file types on my personal devices is better.”
…Canvas also has a mobile app which makes checking my grades from my phone easier. I’ve really enjoyed being able to check my assignments from my phone.
Impact
Figure 8: Survey results on whether Canvas has had a positive impact on the learning experience.
When asked about the impact of Canvas on the quality of their learning, nearly 80% of students responded positively. Many students highlighted how Canvas helped them stay organized and easily track and complete assignments, with some making direct comparisons to Sakai. Overall, it’s clear that Canvas has had a significant impact on students’ learning experiences.
Canvas is simple to use and makes viewing my grades much easier. It’s very organized and it’s made uploading assignments and checking feedback from my professor better.
It makes learning more interactive.
Additional Student Feedback
The following list includes some comments students left throughout different parts of the survey:
- “Please fully switch to Canvas, everyone I’ve talked to wants this.”
- In the beginning I had issues logging in and Canvas thinking I did not have an account, but I think it has been resolved.”
- “Please switch all course to Canvas for spring. My mental health is on a decline because of Sakai.”
- “I found it less useful than Sakai.”
- “Please let more professors have the option to use Canvas instead of Sakai for their courses, I have enjoyed using it in mine.”
- “It’s easier than Sakai, but inconvenient since it’s my only class not on Sakai.”
- “No strong preference either direction, I think I like the design of the Canvas website more, but I’m used to Sakai.”
Conclusion
The evaluation of the Canvas Learning Management System pilot at Pomona College has yielded encouraging results, as participants responded positively overall. Survey data indicates that Canvas had a positive impact on teaching and learning, and both instructors and students reported high levels of satisfaction with the platform, even when compared to the College’s current LMS, Sakai. In addition, support from both ITS and Instructure has been rated positively by instructors, with most being satisfied or very satisfied.
One particularly exciting finding is that the majority of instructor respondents expressed a strong willingness to recommend Canvas to their colleagues who are currently using Sakai. Overall, the pilot participants have demonstrated a positive attitude toward Canvas, with the vast majority expressing interest in continuing to use the platform.
Next Steps
In order to fully evaluate the pilot of Canvas LMS and help Pomona determine future directions, ITS has developed the following plan:
- Monitor and evaluate the Spring 2023 semester implementation: Distribute surveys to instructor and student participants and evaluate the implementation of the Canvas platform to ensure that it is meeting our goals and objectives.
- Continue to onboard new and interested faculty to Canvas: Given the positive feedback from instructors and students, Pomona College ITS will begin to offer Canvas as a new service in future semesters.
- Analyze usage data: ITS will analyze usage data from Canvas in order to gain insight into how it is being used, what features are the most popular, and how Canvas might be optimized to meet the needs of instructors and students.
- Integrate the Student Information System (SIS): Connect our current SIS, Jenizabar CX to Canvas so that courses, faculty, and students can be preloaded. This will save time and ensure that data is accurate.
- Explore additional integrations: ITS will explore additional integrations with other platforms and tools to streamline and enhance the teaching and learning experience.
- Provide additional training and support for instructors: While most instructors felt supported by ITS and Instructure, some may require additional training and support. ITS is prepared to offer more workshops or one-on-one consultations for instructors. Additionally, instructors may be interested in webinars and training opportunities from Instructure.
- Communicate the benefits of Canvas: As all faculty pilot participants indicated that they would likely suggest Canvas to their colleagues, ITS will continue to update our FAQs to describe the benefits of Canvas more widely to encourage adoption of the LMS.
- Develop a roadmap for future development: Based on the analysis and evaluation of the Spring 2023 pilot, ITS will develop a roadmap for future development of the Canvas LMS at Pomona. This will help guide future decision-making and ensure that the LMS continues to meet the evolving needs of the institution.
Special thanks to Nick Weber and Nancy Rodriguez, ITS instructional technologists for their work on this pilot evaluation.
Appendix: Faculty and Student Surveys
The survey instruments used in the evaluation are provided for your reference.