A Comparison of Learning Management System Accessibility

Introduction

If not designed with accessibility in mind, Learning Management Systems (LMS) can pose accessibility problems for students and instructors with disabilities. Depending on the features enabled for a given course, students with disabilities could find that participating independently and effectively is nearly impossible. Some LMS tools—Discussions, Quizzes, Chat, or Wiki tools, for instance—can be more problematic than others. Learning Management Systems are becoming richer and more complex applications, and if they are not designed with accessibility in mind, it can be next to impossible to make them accessible and usable to users with various needs.

Increasing Awareness

Thanks to the hard work of accessibility advocates, in particular the LMS collaboration/working groups across the country, accessibility has become part of the design criteria for most commercial and open-source LMS projects. This achievement was hard fought and was not obtained overnight. Accessibility advocates have worked on at least three fronts:

  1. Convincing IT administration to include accessibility in the purchase process. IT administrators needed to learn that accessibility, as with privacy and security, must be supported by the LMS by default, and that they must require accessibility in the same way they require any other feature, adding accessibility to the purchasing criteria in the RFP processes.
  2. Working with vendors to incorporate accessibility as part of the design phase. Vendors needed to learn that accessibility is not a "touch-up” after the completion of development but must instead be an integral part of the design process.
  3. Pressuring for standards. Work with legislative and judicial entities helped define and pave the way for accessibility standards for applications used by the public, including those used in educational systems.

The good news is that we have achieved a respectable degree of public awareness for accessibility on all levels. Many IT administrators are now considering accessibility features in their product selection or at least ask about accessibility in the RFP processes. LMS vendors and open-source developers have realized that by considering accessibility in their design and implementation processes, they can make their products more accessible and usable for everyone, while increasing the likelihood that their products will be more readily adopted. We also now have several important state accessibility purchasing regulations like the Illinois Information Technology Accessibility Act (IITAA), as well as other Federal regulations, in addition to Section 508, that are currently pending. Over the past few years, commercial LMS vendors, like Blackboard and Desire2Learn, and open-source communities, like Moodle and SAKAI, have invested significant resources into making their products more accessible. Indeed, these products are more accessible now than they have ever been.

The Need to Consider Accessibility as Universal Usability

At the same time, however, and in spite of significant strides that have made many Learning Management Systems more accessible, using these applications can still be a challenge for users with disabilities. A key reason for this is that significant usability factors that would have directly benefited all users' interaction and experience have been overshadowed by the more immediate, "pure” accessibility problems, such as ensuring alternative text for images. These essential fixes can sometimes obscure broader usability concerns and can even distract vendors from core problems that affect usability universally.

As a case in point, let's consider notification mechanisms. Most of us have experienced the scenario where we have submitted a form and don't receive any obvious notification whether the submission was successful. While this usability issue can be manageable for typical users, it can be extremely difficult for users with disabilities because they often end up reading the entire page from top to bottom just to find out if there is any acknowledgement of their form submission. Another example is when users are timed out of the LMS session but receive no notification of it—they simply find themselves at a login prompt, often having lost what they were working on. Alerting users that an active session is about to time out and providing a mechanism to renew the session is another usability feature everyone benefits from, particularly users with disabilities.

There are numerous examples such as these that are not "pure” accessibility issues and that are in fact general usability issues, but that have an overwhelmingly negative impact on the accessibility of the LMS. Ultimately, although supporting pure accessibility features in an application is essential, it does not guarantee an effective user experience. To accomplish this, we must consider basic usability factors that expedite the user experience and improve productivity, all the while guaranteeing that these are implemented with accessibility in mind.

Based on these considerations, we compiled criteria that affect the functional accessibility of four learning management systems—two commercial systems, Blackboard and Desire2learn; and two open source systems, Moodle and SAKAI.

Author Credits

This project was initiated and spearheaded by Hadi Rangin, Information Technology & Collaboration Coordinator from the University of Illinois, who has established and run collaboration groups with many vendors, including the Blackboard Accessibility Interest Group, that led to improving accessibility of the Blackboard LMS over the past years.

Other members of this project include Ken Petri, Director of the Web Accessibility Center at The Ohio State University, Marc Thompson, an instructional designer with Online & Continuing Education at the University of Illinois, who is actively involved in accessibility, and has been designing and developing courses for the Moodle LMS for a number of years, Joe Humbert, Adaptive Technology and Accessibility Specialist from Indiana University, and Dan Hahn, eLearning Professional at University of Illinois who helped me with re-testing and verifying the Blackboard accessibility evaluation. Ken and Joe are experts in accessibility and are currently involved in the Desire2Learn Accessibility Interest Group and the SAKAI Accessibility Working Group, respectively.

About the Evaluation and Results

Each LMS presents a different learning environment with unique features and implementation techniques. While all tested LMS offer all common and basic tools/modules, the level of functionality and their implementations of the same tool can be significantly different. For example, Moodle offers a basic discussion tool with standard functionality, whereas Blackboard offers a more complex tool that is also potentially more difficult to make accessible.

In this round of LMS accessibility evaluation, we focused on functional tasks instead of testing specific pages of tools. For example, in the Login, Configuration, and Compatibility Testing category, we ask, "Can users detect error messages and recover from errors with certainty?" instead of checking if the login page is accessible. By focusing on functional tasks like these, we examine the accessibility of a tool with all related interaction and processes as a whole, regardless what implementation technique has been used in that tool.

We would like to underscore that we have tested only the core and common features of selected tools for accessibility and that our accessibility rating does not convey any information about the overall functionality of the selected tools. We are hoping to educate the public and product development teams about how the presence or absence of certain key usability/accessibility features can significantly impact users' experience. Toward this end, we provide a rationale for each evaluation category before defining the testing criteria. This testing protocol and rationale are then followed by specific testing results for each LMS, after which we draw our conclusions and present a side-by-side comparison.

The Learning Management Systems We Compared

We tested the following Learning Management Systems:

We would like to thank the University of Illinois, The Ohio State University, and Indiana University and their respective LMS administrators for facilitating the necessary testing environments and helping us to understand the administrative view of these LMS. We would also like to thank Blackboard and Desire2learn for working with us to find answers to a number of questions we were not sure about.

Disclaimers

All statements and opinions of this report are entirely those of the authors. Our testing does not reflect a full or complete accessibility evaluation. This report does not represent the view or opinion of the presenters' respective institutions. Our results are preliminary, and we focus only on selected modules/tools. It is possible that we did not test for some relevant functional accessibility features. If you have suggestions for improving the testing protocol, please send your feedback to hadi@illinois.edu.

Accessibility/usability testing and evaluation were performed at different times in 2012, and our comparisons are thereby uneven in this regard. Testing that was more recent, as with the latest version of Desire2Learn version 10 (released December 2012), is thus more likely to reflect recent accessibility improvements. By contrast, testing that was less recent does not always reflect the newest enhancements introduced in the most recent LMS versions.

Blackboard revised its discussion board tool since our evaluation, for example, and is thus not included in the version we tested. Similarly, since the public release date of Moodle 2.4 (December 2012) after our evaluation, a number of accessibility enhancements have been added—most notably the incorporation of ARIA landmarks and roles—that were not included in our evaluation results.

Our evaluation of SAKAI was focused on version 2.8.0, and did not include the following accessibility improvements introduced in SAKAI 2.9 (released December 2012):

Categories of Testing/Evaluation

No doubt our testing categories are incomplete, owing to limited time and resources. We were determined to focus on commonly used (and thus high priority) aspects of the various LMS—primary modes of interacting with the systems and typically used "tools." If you have suggestions on how to expand our testing categories and related criteria, please contact Hadi Rangin at hadi@illinois.edu.

Considering the primary interactions in a typical Learning Management System environment—between end-users and the LMS, between students, and between instructors and students—we identified the following major categories for testing. then we divided each category in logical task groups.

  1. Login, Configuration, and Compatibility Testing
  2. Personalization and Customization
  3. Navigation
  4. Forms
  5. Help and Documentation
  6. Common Student Facing Modules/Tools
  7. Authoring Tools and Content Creation

As mentioned earlier, each LMS offers unique functionality and uses different technology to implement that functionality.

1. Login and Configuration/Compatibility Testing

Rationale

The login page is usually the first point of contact for a user interacting with an LMS, often through a localized/customized page set up by the hosting institution. In addition to the accessibility of the login page itself, login pages often have some form of application and/or browser test to check for specific tools/applications required by the LMS, such as Java, or for browser settings that enable JavaScript or pop ups. When third party components or plug ins are required, the accessibility of the LMS testing procedure and test results page are essential for providing users with vital information about any necessary configuration and software components required by the LMS.

Criteria

Results Summary and Comparison

2. Personalization and Customization

Rationale

Users have different needs and ways of viewing and interacting with the LMS. As such, it is vital that users be able to easily configure the LMS environment to their individual needs, instead of adapting to the application. Although they allow users to customize their experience to one degree or another, there are many global settings and personalization/customization features that can significantly augment or limit the usability/accessibility of complex applications and the LMS.

Criteria

Results Summary and Comparison

3. Navigation

Rationale

Navigation is the most important element of accessibility. Proper use of structural markup is critical to achieving effective, consistent, and logical navigation that allows keyboard users to navigate within the application effectively and efficiently, and to obtain necessary information quickly and easily.

Criteria

Results Summary and Comparison

4. Forms

Rationale

Once in the the LMS, real interaction with the LMS starts with forms and entering data. To work effectively and efficiently in the LMS, the user needs to concentrate on the data being entered and not be challenged or distracted by the complexity of the interface and navigation. In this context, it is critical that all the form controls be accessible so the user can enter data with certainty. It is equally important that the user be notified about any warning, error, or successful form submission, and that the user be able to identify easily places where errors have occurred and navigate to them.

Criteria

Results Summary and Comparison

5. Help and Documentation

Rationale

The concept of online learning and using an Learning Management system can be challenging to new users, in particular users with disabilities. Application developers use various styling and formatting effects to convey certain information to their users which is not necessarily apparent to users with disabilities. For example, some LMS use tab-style design for their layout. Sighted users can immediately see what tab they are on, but this might not be easily accessible to blind or low-vision users even though the necessary information is provided in the system. Screen reader users, for instance, might not be able to find the necessary information if they don't know where and how to look for it. Or keyboard users might not be able to utilize the built-in accessibility features if they don't know about them.

There are also tools and tasks that are not easy to use and require clear and straightforward explanation of the processes involved in those tasks. Documentation of supported accessibility features, a quick accessibility guide in fully accessible format outside the LMS, and accessible inline help/instruction are all essential for helping users learn, troubleshoot, and work effectively in the LMS. Whether internal or external to the LMS, the accessibility/usability of such help systems and documentation materials, as well as effective documentation of LMS-specific accessibility features and general accessibility guidelines, are vital to all users. Ultimately, it is very important for users with disabilities, and blind users in particular, to learn about the basic functions of the LMS in advance before they actually start using it. Moreover, understanding the general layout of the LMS can be extremely useful to some users with disabilities, and it is an important step in helping users with disabilities develop a strategy for navigating and interacting with the LMS.

Criteria

Results Summary and Comparison

6. Common Modules/Tools (Student Facing)

Rationale

The accessibility/usability of the common modules and tools of the LMS provide the necessary means by which student users create and interact with course content. The most essential and common modules/tools have been selected for evaluation, and the criteria for each tool is predicated on its functionality.

6.1 Announcements

Criteria
Results Summary and Comparison

6.2 Discussion

Criteria
Results Summary and Comparison

6.3 E-mail

Criteria
Results Summary and Comparison

6.4 Chat

Criteria
Results Summary and Comparison

6.5 Assignments, Activities, Course Content, Learning Modules

Criteria
Results Summary and Comparison

6.6 Grade Book

Criteria
Results Summary and Comparison

6.7 Quizzing/Testing Components

Criteria
Results Summary and Comparison

7. Authoring Tools and Content Creation

Rationale

Authoring tools, including HTML editors, file uploading tools, and helper tools for specific content creation, and multimedia handling, all significantly impact the way course content is designed in the LMS, as well as the accessibility of the content created by those authoring tools. The best time to address the accessibility of the content is when it is created. Toward this end, authoring tools should be accessible, feature-ful, easy to use, and prevent invalid and or inaccessible HTML code. The LMS can play an vital role in educating users about the accessibility of the content they create, upload, or reference. For example, authoring tools can warn or test for accessibility on the files being uploaded, prevent users from saving graphics without first specifying the type of image (stylistic/informative), prompt the user for alternative text in the case of informative images, or alert users about captions for video clips, etc.

Criteria

Results Summary and Comparison