About the Spring 2023 Pilot
During the Spring 2023 semester, Canvas was utilized by a growing number of students, instructors and courses, building on the success of the Fall 2022 pilot. Specifically, over 500 students and 21 instructors across 35 courses used Canvas. Seven new instructors joined the pilot, while fourteen existing users continued to leverage their familiarity with the platform from the previous semester.
Evaluation
To evaluate the implementation of Canvas, ITS designed three surveys: one for the seven new instructors, one for the fourteen existing instructors, and one for all the students enrolled in a Canvas course at Pomona this past spring. Each survey included questions on satisfaction, use, impact, and support as well as areas to leave open-ended responses.
To fully understand the technological and pedagogical implications of implementing Canvas, feedback from all pilot participants in the Spring 2023 semester will be combined with feedback from the previous Fall 2022 semester to gain a comprehensive understanding of the platform’s impact. With the input of our faculty, students, administrators, and staff, we will explore the costs and benefits of potentially adopting Canvas to help the College make an informed decision about the future directions.
Survey Results
While the response rate for the Spring semester was lower than the previous Fall semester, it was still respectable, particularly from the instructors. ITS received feedback from 62% of the instructors and 11% of the students. This valuable input will allow us to make any necessary adjustments to improve the use of Canvas for both students and instructors.
Below are some highlights and interesting findings from the various quantitative and qualitative responses.
Faculty Feedback
Use
Consistent patterns were observed in the usage of Canvas between instructors from the previous Fall semester and the new instructors of this current Spring semester. Notably, 100% of new pilot instructors reported utilizing Canvas for key activities such as uploading course readings and other document files, creating and managing assignments, and providing valuable feedback on student submissions. Data indicates Canvas was least used for virtual office hours (figure 1).
I’ve used Canvas in the following ways [check all that apply].
Among the existing instructors who participated in the survey, a significant majority (66.6%) have been utilizing Canvas for over three years. When asked to provide examples of particularly effective or compelling uses of Canvas, here are a some of their responses:
- “It easily collates all course materials–assignments, readings, multimedia–in one place through the modules feature. Links easily to external websites, and integrates to e-publications on the library website. Seamless and user friendly.”
- “Accessibility is a great thing to make the course accessible to anyone.”
- “Canvas puts everything on the Calendar so students can easily follow the Calendar to submit their works.”
- “Using modules to organize assignments, files, etc. by week.”
- “Yes. If one wants to flip the classroom, ie video lectures at home, class time used for psets, practice problems, etc, Canvas makes that really doable. The other main use is that it is a convenient way to go paperless and have documents available for all the students.”
Satisfaction
Figure 2: Survey results on overall satisfaction with Canvas.
Overall, satisfaction with Canvas is very high among instructors. A combined 100% of instructor respondents indicated that they were either “very satisfied” or “satisfied” with Canvas (figure 2). These 2023 findings are very similar to the findings from the Fall 2022 Canvas Pilot survey results.
When new instructors were asked about their likelihood of recommending Canvas to their colleagues currently using Sakai, the response was overwhelmingly positive. A remarkable 100% of respondents either “strongly agreed” or “agreed” with this statement, demonstrating an unequivocal endorsement of Canvas as a preferred platform over Sakai.(figure 3).
Figure 3: Survey results on recommending Canvas to other colleagues who are using Sakai.
Canvas makes it so easy to organize my coursework (I love using the Modules feature) and stay on top of grades with the notifications when there are new submissions (even for past assignments). The interface is clean and easy to navigate/pretty to look at and infinitely customizable! The app makes it easy to access on the go for both me and the students. As an adjunct, I appreciate that I can import course materials from other courses that I teach at schools that use Canvas!
It is soooo much easier than Sakai. I love the speed grader option!
Some operations took awhile to figure out and are still not intuitive to me, e.g. creating test banks and quizzes–seems bulky process. But, most operations were pretty easy. Many students were already familiar with Canvas from HS so that helped. Re-arranging and revising the modules was very easy.
Impact
Figure 4: Survey results on whether Canvas has had a positive impact on teaching.
Data indicates that Canvas had a positive impact on new pilot instructor’s teaching and learning experiences with the majority of instructor respondents (75%) choosing “strongly agree” or “agree” on the Likert scale (figure 4). Existing instructors were not asked this question.
Qualitative responses indicate the biggest impact was time management as instructors reported that Canvas LMS helped to do things more efficiently. Another area where Canvas had a positive impact was the ability to organize content and make it more accessible.
I feel that students were able to engage with the material in a meaningful way due to the organizational structure and the variety of materials I could easily embed on pages.
It had made my life easier thus leaving me more time to prep classes.
When existing instructors were asked if they’ve noticed any changes to their teaching since having started using Canvas, here are a some of their responses:
- “No. It just helps me manage the course. Students seem to prefer it to Sakai; they say it is more user-friendly.”
- “Yes, students engagement have improved!”
- “Easy to change”
- “Scheduling becomes easier. Notification is automatic, very convenient.”
- “I prefer Canvas to Sakai and Moodle, and probably to Blackboard too. Canvas is more intuitive and I think it has helped keep each week more organized for me and more clear for students.”
Support
New pilot instructors felt well supported by both RITG and Instructure. 100% of instructors reported they were either “very satisfied” or “satisfied” with support from ITS-RITG and 75% were satisfied with the support from Instructure (figure 5).
I feel supported, because I’ve received quick and helpful replies to all of my inquiries.
I would like to thank the ITS Canvas team for their time and efforts.
Tech support for Canvas was prompt and super helpful!
Additional Faculty Feedback
The following list includes some comments instructors left throughout different parts of the survey:
- “I hope I can use Canvas in both of my classes in the Fall!”
- “It has served as a single central hub for my students to go for the course and has increased how accessible my course is.“
- “The discussions tool can be a bit frustrating at times (seeing the sequence of messages in order of posting, for example, as well as the chains of replies to messages). I wish Canvas had a drop box tool like Sakai does.”
- “I prefer Canvas to Sakai and Moodle, and probably to Blackboard too. Canvas is more intuitive and I think it has helped keep each week more organized for me and more clear for students.”
- “I think developing training for making tests and quizzes in Canvas would be useful, as would developing training about how to use canvas to encourage collaborative learning (via google docs, collaborative note-taking, mind-mapping, etc.).”
- “Faculty should be able to add student to the class.”
Student Feedback
Use
Like instructors, students were also asked to indicate how they use Canvas. The majority of students (over 94%) reported having used it to access course readings and submit assignments which is similar to instructor findings. Close to 78% of students use Canvas for viewing grades and about half of the student respondents use the course calendar with due dates. Canvas was least used for quizzes and/or exams and for communication between professors and classmates.
Satisfaction
The majority of students were positive when asked to rate their overall satisfaction with a combined 82.4% selecting either “very satisfied” or “satisfied” (figure 7). When asked to elaborate on their rating a few positive comments included easy navigation and good user interface.
Figure 7: Survey results on overall satisfaction with Canvas.
I spend a lot less time looking for resources and completing assignments when it’s on Canvas, which makes it way better for me.“
It has everything I want it to have.
Impact
Figure 8: Survey results on whether Canvas has had a positive impact on the learning experience.
When asked about the impact of Canvas on the quality of their learning, over 70% of students responded positively (figure 8). Many students highlighted how Canvas is intuitive and easy to navigate, with most making direct comparisons to Sakai. Overall, it’s clear that Canvas has had a significant impact on students’ learning experiences.
I think Canvas is more intuitive and easier to navigate than Sakai. It helps me stay organized as a student.
It’s easy to navigate, clean in UI, and importantly, it *actually works*.
Additional Student Feedback
The following list includes some comments students left throughout different parts of the survey:
- “I would love to see Pomona transition to Canvas completely in the near future.”
- “Canvas has been pretty easy to use and fairly intuitive, it doesn’t feel that different from Sakai but it does look nicer.”
- I found the portal clunky and difficult to use. The app is very convenient, but distinguishing between pending, submitted, and graded assignments is complicated on both app and webpage sites.”
- If the question is if we should migrate from Sakai to Canvas, let my voice give a *resounding* “YES!!”
- “I’ve heard bad things about canvas, but I liked it a lot more than sakai.”
- “I got sad when I had to switch over to using Sakai for my other classes this semester.“
- “It’s annoying and difficult to use.”
- “please, please, please make an institutional switch to canvas!!”
Conclusion
Following in the footsteps of the Fall 2022 Canvas Pilot evaluation, the Spring 2023 Canvas Pilot evaluation results reflect similarly very positive feedback from both faculty and students. Survey data highlights the positive impact Canvas had on teaching and learning, and both instructors and students reported high levels of satisfaction with the platform, even when compared to the College’s current LMS, Sakai. Furthermore, the support provided by both ITS and Instructure was highly rated by instructors, with the majority expressing satisfaction or high levels of satisfaction. These findings provide strong evidence in support of continuing to explore the potential of Canvas as an effective learning management system for the College.
Next Steps
ITS has developed a plan, focusing on the following areas over the summer and beyond:
- Analyze usage data: ITS will analyze usage data from Canvas in order to gain insight into how it is being used, what features are the most popular, and how Canvas might be optimized to meet the needs of instructors and students.
- Integrate the Student Information System (SIS): Connect our current SIS, Jenizabar CX to Canvas so that courses, faculty, and students can be preloaded. This will save time and ensure that data is accurate.
- Explore additional integrations: ITS will explore additional integrations with other platforms and tools to streamline and enhance the teaching and learning experience.
- Continue to onboard new and interested faculty to Canvas: Given the positive feedback from instructors and students, Pomona College ITS will begin to offer Canvas as a new service in future semesters.
- Provide additional training and support for instructors: While most instructors felt supported by ITS and Instructure, some may require additional training and support. ITS is prepared to offer more workshops or one-on-one consultations for instructors. Additionally, instructors may be interested in webinars and training opportunities from Instructure.
- Communicate the benefits of Canvas: As all faculty pilot participants indicated that they would likely suggest Canvas to their colleagues, ITS will continue to update our FAQs to describe the benefits of Canvas more widely to encourage adoption of the LMS.
- Develop a roadmap for future development: Based on the analysis and evaluations of the Fall 2022 and Spring 2023 pilots, ITS will develop a roadmap for future development of the Canvas LMS at Pomona. This will help guide future decision-making and ensure that the LMS continues to meet the evolving needs of the institution.
Special thanks to Nick Weber and Nancy Rodriguez, ITS instructional technologists for their work on this pilot evaluation.
Appendix: Faculty and Student Surveys
The survey instruments used in the evaluation are provided for your reference.