This is a follow-up to my previous post that described my second structured assessment of cognitive Web accessibility. The work’s progression can be seen via this blog’s Category of Cognitive Web Accessibility Assessments.
Assessment Scoring System Needed Revision
The result of the second structured assessment was that the Web site was inaccessible to people with developmental disabilities. Had I followed my original assessment plan, the result would have been the opposite. This is because of the plan’s scoring system.
I had intended to score one point if any Web site feature met even one guideline of each of the sections of WebAIM’s Cognitive Web Accessibility Checklist. While following this system during my second structured assessment, I realized it was too generous. Points were adding up although it was obvious to me the site was likely inaccessible by people with developmental disabilities.
Consequently, I decided to average the number of guidelines that were met (successes) with those that were not (failures) for each checklist section. Arbitrarily but within reason, I also decided an average success of 80% or higher would score one point. I will apply this standard to future assessments unless I learn it is unworkable.
Assessments Require Significant Effort
When I developed my original assessment plan for 100 sites, I had been hoping the work could be performed quickly. That was naive. I now recognize much more work is needed. Every relevant guideline in all of the checklist sections must be evaluated to portray a site’s cognitive Web accessibility as well as possible. Indeed, because I evaluated all the relevant guidelines in the last (and only) structured assessments, comprehensive portrayals of the sites’ cognitive Web accessibility were produced.
Assessments Should Be Performed By Users
The cognitive accessibility of Web sites would be best assessed by users with cognitive disabilities. I was reminded of this by Joe Chidzik after I posted my second structured assessment. Specifically, his message was, “A site may be *likely* to cause accessibility issues, but to claim it does so without user testing isn’t helpful”. Point taken.
Yet this assessment work, in part, is an attempt to find a uniform way for developers to help determine the cognitive accessibility of Web sites. Of the many automated tools that help assess general Web accessibility, none are focused on cognitive Web accessibility. WebAIM is pursuing funding to incorporate such assessment into its WAVE Web accessibility evaluation tool. My work is an unofficial precursor to that. I hope it will be helpful.
That said, I would like to include people with cognitive disabilities in this work. Honestly though, I don’t know how to do it simply and economically. (This project’s work is performed primarily on my own time and is unfunded.) I am open to constructive suggestions. Please post a comment with one.
Very interesting read on your cognitive assessment articles and on this site in general. It’s great to see work being done on evaluating the cognitive accessibility of websites.
I’d like to add to the point raised above by myself on the usefulness of non-user based testing. User testing is indeed the best way of discovering the biggest range of usability\accessibility issues, however expert reviews such as the ones you are carrying out, complement any user testing well, and are a great way of establishing benchmarks across sites, as testing will be carried out consistently.
Additionally, as you rightly point out, even basic user testing is not always practical from an economic point of view.
Whilst a comprehensive usability\accessibility review should always have some user involvement, reviews such as this can pick up on more common issues and have them addressed before a more in depth review is carried out. This would be particularly useful in the design stages of a project, where multiple iterations of designs are gone through consecutively. It would be unrealistic to expect user testing to be carried out on all the seperate design iterations.
Additionally, establishing a methodology for carrying out cognitive assessments means that web designers themselves can pre-empt user testing by carrying out basic checks themselves, which serves the dual purpose of streamlining any potential user testing by enabling a focus on more in depth issues, and also gives some level of awareness training to web designers on typical issues faced by users with cognitive difficulties. In the long run this can only be a good thing.
I would say that a holistic approach to web accessibility (cognitive, physical and sensory) can and should include not only user testing but expert assessments to known guidelines.
Cheers
Joe