I have fundamentally revised the scoring of my cognitive Web accessibility assessments. The impetus was my troubling assessment of the Web site for Bipolar Scotland. It achieved a good score, but I judged it to be inaccessible to people with cognitive disabilities.

Scoring System

My 10-point assessments are based upon WebAIM’s Cognitive Web Accessibility Checklist. Three of its sections relate to site content and four to site design. Each section is an assessment criterion. I record one point for each of those that are met, plus a possible point for each of three simple design-related criteria I added to the assessments.

Scoring System Problems

Until now, I had not considered a couple aspects of my scoring system.

• The design criteria outnumber the content criteria. This meant I was judging sites to be accessible that, like Bipolar Scotland’s, were strong in design but had content accessibility problems. This was not good practice.
• Because meeting each criterion means one point, my simple design-related criteria had the same significance as the complex, detailed design criteria of WebAIM’s checklist. They should not have.

Scoring System Revision

Site content should have significantly-greater impact on my assessments; the simple design-related criteria should have less. To accomplish this, I now do not consider any site accessible unless, at a minimum, it meets all three content criteria and all four design criteria of WebAIM’s checklist. Any points recorded for the simple design-related criteria improve the total assessment score.

New Site Scoring System, By Points:

• minimum for “accessible” = 3 content + 4 design
• total score = those 7 + up to 3 simple design-related

Notes:

• This post is part of a continuing series on Cognitive Web Accessibility Assessments.
• I have published on The Clear Helper site an index of aggregate results.
• I will soon publish detailed results for each assessed site.

I assessed the Web site of Bipolar Scotland. It was designed for people with cognitive disabilities. Summary assessment results show the site met all seven design-related criteria, but not the three content criteria. In my judgment, a site with significant content-accessibility problems can not be considered accessible to people with cognitive disabilities.

The bullets below list the sections of WebAIM’s Cognitive Web Accessibility Checklist. Each section is an assessment criterion. Each is considered met if at least 75% of its applicable guidelines are implemented on the Web site.

Site Design Strengths

• Consistency (100%)
• Transformability (75%)
• Orientation and Error Prevention/Recovery (75%)
• Assistive Technology Compatibility (86%)

The site also attempts to meet W3C accessibility guidelines, has an accessibility statement, and explains at least one accessibility feature (the font size changer).

Site Content Problems

• Multi-modality (0%)
  • Text is not offered in video- or audio alternatives.
  • With few exceptions, contextually-relevant images are not used, and icons/graphics are not paired with text.
• Focus and Structure (67%)
  • Design does not focus attention on page-body content.
  • On every page there is a large, distracting element (a constantly-changing image inside an Amazon widget).
• Readability and Language (50%)
  • According to my tests with the Standards-Schmandards Readability Index Calculator, text is not understandable by people with a lower secondary education.
  • Other related problems include the use of jargon.  For example, simple explanations could have replaced  “hypomania”  and “dysthymia” in Types of Bipolar.

Statement Site Was Designed For People With Cognitive Disabilities

Bipolar Scotland’s accessibility statement, in part, says:

In web accessibility terms, bipolar disorder falls under the category of cognitive/intellectual disabilities, one of the five needs that web accessibility aims to address. We tend to think of web accessibility as ways to help blind or deaf users view the web, but people with cognitive disabilities have particular needs involving memory, comprehension, attention span, and logic skills. Creating a site for people with bipolar disorder, who may or may not be experiencing those issues on any given day, poses a particular challenge: we are designing to accomodate the restrictions within the brain, not the body.

Retrieved from: http://www.bipolarscotland.org.uk/accessibility

Conclusion

I agree with that statement. The one exception I take is with its last line. I think the reverse is true: the site’s design is good for physical disabilities (“the body”) but its content is not for cognitive disabilities (“restrictions within the brain”).

Notes

• My next blog post will discuss the contradiction between the site’s good assessment score and my judgment it is inaccessible to people with cognitive disabilities.
• This post is part of a continuing series on Cognitive Web Accessibility Assessments.
• I have published on The Clear Helper site a description of the assessments and an index of aggregate results.

Results of my cognitive Web accessibility assessments, for the 12 sites I have evaluated to date, show an average score of 5 out of 10 points.  That datum is the launch point for this post, in which I consider the assessments’ consistency, accuracy, and related implications.

I hope the average score improves as I increase the sample size of assessed sites, but it will be unlikely if I encounter more like that of The International Dyslexia Association.  It is the first Web site for which no points were scored.

I think the zero-point score is an accurate portrayal of the site’s accessibility.  Comparing it to the two sites that scored all points and to the other assessed sites indicates to me my assessment system is internally consistent.  It is obvious, for example, that the top scorers are much more accessible to people with cognitive disabilities than those sites with five points or fewer.

I suspect the top scores were achieved because the two sites were designed for people with intellectual disabilities and because my assessments are for the broader, perhaps-more-capable group of people with cognitive disabilities.

Given my experiences observing people with intellectual disabilities navigate Web sites, I am concerned even the efforts of the top-scoring sites may not mean they are truly relatively-accessible.  I don’t know how my assessments could better judge such sites, but that is my main interest.

Extensive testing by people with intellectual disabilities may be a good indicator of accessibility.  However, there is such a range of abilities within the population that I am unsure any Web site could be accessible to a significant portion of them. This may mean in practice I must produce criteria for minimum abilities needed and try to make the future Clear Helper site accessible to people who meet them.

Note: This post is part of a continuing series on Cognitive Web Accessibility Assessments.

I created a set of extensive pages that display aggregate results from my assessments of cognitive Web accessibility.  I also evaluated three more sites.

Results Pages

Seven of the results pages are based upon the sections of WebAIM’s Cognitive Web Accessibility Checklist.  Amongst the pages are aggregated results from assessments of the 44 guidelines that comprise the sections.  The guidelines are defined using WebAIM’s descriptions.

Dynamic Data

In-text results data are are always up-to-date. They are extracted dynamically from a draft database, as are those for the related charts created with Google Chart Tools.

Assessed Web Sites

These are the Web sites I just assessed, and their scores.

• Indian Head Injury Foundation, 2 of 10 points
• The Autism Acceptance Project, 3 of 10 points
• Learning Disabilities Association of America, 4 of 10 points

View a page of results for all Web sites assessed to date.

I performed cognitive Web accessibility assessments on five more sites.  Two of them, both of organizations located in the United Kingdom, received all possible points.  The results for the rest were varied.

Assessed Sites and Scores

• Alzheimer’s Association, 3 of 10 points
• Council for Exceptional Children, 7 of 10 points
• Mencap, 10 of 10 points
• People First, 10 of 10 points
• Speaking For Ourselves, 6 of 10 points

View detailed results, assessment criteria and methodology.

Site Highlights

People First

The People First site features a large site-navigation menu (pictured below). For menu options, there are contextually-relevant icons, which are also used throughout the site.

Mencap

The Mencap site incorporates many captioned videos (example pictured below) as an alternative to text content, and relevant images to augment it.  The site’s My Life section is specifically designed for constituents, with plain language; simple navigation, and lots of images and videos.

Conclusion

Though the Web sites of Mencap and People First have minor problems, it is apparent the two organizations expended great effort to make them accessible to their constituencies.  I offer my congratulations.

Note: This post is part of a series on cognitive Web accessibility assessments.

I now have a database of results from my assessments of cognitive Web accessibility.  It is a work in progress.

I am displaying the database data on The Clear Helper Web site.  Because, to date, I have assessed only 2 of my planned 100 Web sites, few data are presented.  Yet it seems a good idea to start the database development and the data presentation now, partly in hope of soliciting feedback from followers of this project.

The assessments home page references two pages that display the same database data differently.  Its main purpose is to describe the assessment criteria and methodology.  I will use it to record any related changes I make as a result of what I learn from performing subsequent assessments.

Via table and pie chart, the first page shows Results By Numbers of Web Sites that meet the assessment criteria.

The second page shows summaries of Results By Web Site.  Each has the criteria met, the number of points recorded, and the conclusion. Seven of the criteria are based upon the sections of WebAIM’s Cognitive Web Accessibility Checklist.

A future page will show results by the guidelines that form the checklist’s sections.  It might be interesting to create a page of results by country.  It may be, for example, that cognitive-disability organizations in the countries of the U.K. are more likely than those in other countries to make their Web sites accessible to their constituencies.

I would like feedback.  If you have suggestions for how I can improve my related efforts, or for other ways to display the results, please post a comment or contact me.

This is a follow-up to my previous post that described my second structured assessment of cognitive Web accessibility.  The work’s progression can be seen via this blog’s Category of Cognitive Web Accessibility Assessments.

Assessment Scoring System Needed Revision

The result of the second structured assessment was that the Web site was inaccessible to people with developmental disabilities.  Had I followed my original assessment plan, the result would have been the opposite.  This is because of the plan’s scoring system.

I had intended to score one point if any Web site feature met even one guideline of each of the sections of WebAIM’s Cognitive Web Accessibility Checklist. While following this system during my second structured assessment, I realized it was too generous.  Points were adding up although it was obvious to me the site was likely inaccessible by people with developmental disabilities.

Consequently, I decided to average the number of guidelines that were met (successes) with those that were not (failures) for each checklist section.  Arbitrarily but within reason, I also decided an average success of 80% or higher would score one point.  I will apply this standard to future assessments unless I learn it is unworkable.

Assessments Require Significant Effort

When I developed my original assessment plan for 100 sites, I had been hoping the work could be performed quickly.  That was naive.  I now recognize much more work is needed.  Every relevant guideline in all of the checklist sections must be evaluated to portray a site’s cognitive Web accessibility as well as possible.  Indeed, because I evaluated all the relevant guidelines in the last (and only) structured assessments, comprehensive portrayals of the sites’ cognitive Web accessibility were produced.

Assessments Should Be Performed By Users

The cognitive accessibility of Web sites would be best assessed by users with cognitive disabilities.  I was reminded of this by Joe Chidzik after I posted my second structured assessment.  Specifically, his message was, “A site may be *likely* to cause accessibility issues, but to claim it does so without user testing isn’t helpful”.  Point taken.

Yet this assessment work, in part, is an attempt to find a uniform way for developers to help determine the cognitive accessibility of Web sites.  Of the many automated tools that help assess general Web accessibility, none are focused on cognitive Web accessibility.  WebAIM is pursuing funding to incorporate such assessment into its WAVE Web accessibility evaluation tool.  My work is an unofficial precursor to that.  I hope it will be helpful.

That said, I would like to include people with cognitive disabilities in this work.  Honestly though, I don’t know how to do it simply and economically.  (This project’s work is performed primarily on my own time and is unfunded.)  I am open to constructive suggestions.  Please post a comment with one.

This post is my second structured assessment of cognitive Web accessibility.  I describe how it is performed in my assessment plan.  It is less-detailed than my first assessment, but it again addresses every relevant guideline of WebAIM’s Cognitive Web Accessibility Checklist.

Web Site: National Association of Councils on Developmental Disabilities

Assessment

• Consistency. One point is awarded.
  • Success
    • Navigation is consistent throughout the site.
    • Similar interface elements and similar interactions do produce predictably similar results.
• Transformability. No point is awarded.
  • Success
    • Images are readable and comprehensible when enlarged (scaled to 200% and 300%).
    • Color alone is not used to convey content.
  • Failure
    • Increased text sizes (200% and 300%) are not supported by the navigation menu.
    • The disabling of styles is not supported.  On the home page, Latin (Lorem Ipsum) text appears, as well as non-contextually relevant links (“Sub-Link 1”, etc.).
• Multi-Modality. No point is recorded.
  • Success
    • Icons of top menu are contextually-relevant.
  • Failure
    • No video- or audio alternatives are provided for textual content.
    • No images are used to convey or to enhance content.
• Focus and Structure. One point is awarded.
  • Success
    • Distractions are avoided.
    • Stylistic differences are used conservatively to highlight important content.
    • Content is organized into well-defined groups.  Headings and lists are used.
    • White space is used for separation.
    • Background sounds are not used.
  • Failure
    • White space and visual design elements are not used to focus user attention.  Particularly because of the red background color of the menus, attention is instead focused on them.
• Readability and Language. No point is recorded.
  • Success
    • There is no tangential-, extraneous-, or irrelevant information.
    • Grammar and spelling are correct.
    • Tables of contents are provided for complex or lengthy content.
    • Text-readability criteria are met.
  • Failure
    • Language is not as simple as is appropriate for the content.
    • The reading level is inadequate for the audience (assuming it is people with developmental disabilities).
    • Jargon is used.
    • Expansion of abbreviations and acronyms is inconsistently implemented.
    • Text is not succinct.
• Orientation and Error Prevention/Recovery. No point is recorded.
  • One form was found.  It has only one field (password).  It fails assessment of the related checklist criteria.
• Assistive Technology Compatibility. No point is recorded.
  • Success
    • A logical heading structure is used consistently.
    • The navigation order is essentially logical.
  • Failure
    • Use of alternative text is inconsistent.  There is none for the images of the text-size changer.
    • Form labels: the only field on the one form is missing a label.
    • Links do not make sense out of context.  There are multiple “Learn More” links on the home page.
    • Keyboard accessibility is problematic.  Navigation menus are not visible via keyboard navigation.
    • Descriptive and informative titles are missing from many pages.
• The site attempts to meet W3C accessibility standards. One point is awarded.
• There is no accessibility statement. No point is awarded.
• There is no explanation about how to use accessibility features. No point is awarded.

Results

Three of ten points possible are recorded.

Conclusion

The Web site of The National Association of Councils on Developmental Disabilities is inaccessible to people with developmental disabilities.