Multimodal Summary of Complex Sentences for People with Cognitive Disabilities

The following is a synopsis of work on creating multimodal summaries of complex sentences.  A poster of that work, performed by The Hajim School of Engineering and Applied Sciences at The University of Rochester, is the source of all the quoted information in this blog post. I plan to employ this approach on the future Clear Helper Web Site.

The Approach

We propose Multimodal summary of complex sentences. It gives readers the main idea of sentences using pictures and compressed text structured according to simplified text.

The general steps in the MMS approach are:

  • Identify both the main idea of the sentence and related entities and use them to create a compressed summary.
  • Extract pictures for the compressed summary.
  • Add structure to the pictures and text.


Input sentence: In 1492, Genoese explorer Christopher Columbus, under contract to the Spanish crown, reached several Caribbean islands, making first contact with the indigenous people.

Identify event and related entities: In 1492, Genoese explorer Christopher Columbus, under
contract to the Spanish crown, reached several Caribbean islands, making first contact with the
indigenous people.

Extract picture and add structure:

Sentence text, parsed by event and entities, alongside representative pictures.

Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. “Multimodal Summarization for People with Cognitive Disabilities in Reading, Linguistic and Verbal Comprehension” poster presented at “All Together Now: The Power of Partnerships In Cognitive Disability & Technology.” Tenth Annual Conference of The Coleman Institute for Cognitive Disabilities. Westminster, Colorado. 21 October 2010.

Note: No endorsement of The Hajim School of Engineering and Applied Sciences at The University of Rochester is intended or implied.

New U.S. Plain-Language Law Good for People with Cognitive Disabilities

The Plain Writing Act of 2010 was signed into U.S. federal law on October 13, 2010. Essentially, it requires federal agencies to create documents using plain language.

The law also requires a section, on each federal-agency Web site, that follows best practices of plain-language writing. If indeed that happens, I anticipate people with cognitive disabilities will find such information much easier to understand. This is particularly good for the people with intellectual disabilities whom I have interviewed. A common thread of our conversations related to their self-advocacy interest in contacting their government representatives, and for determining how their government could help them.

I hope U.S. federal agencies set a standard that others will follow. For people with cognitive disabilities, the accessibility of Web site content is just as important as the accessibility of a site’s design. Text must be written in plain, simple language. There are efforts all over the world to encourage the use of plain language, which helps everyone.

For more information, see:

Conferences Related to Technology, Web Accessibility and Cognitive Disabilities

Two upcoming- and two recent conferences are listed below along with their related topics and presenters.

Upcoming Conferences

All Together Now: The Power of Partnerships In Cognitive Disability & Technology

  • October 21, 2010 – Westminster, Colorado, U.S.
  • Forty Years after PARC v The Commonwealth of Pennsylvania: Is there a right to technology access? – Gilhool, Thomas
  • A Partnership for Technology to Improve Quality of Life – Pietrangelo, Renee
  • Accessible TeleMEd and eHealth Strategies for People with Cognitive Disabilities – O’Hara, David
  • Technologies to Improve Quality of Life for People with Cognitive Disabilities – Kautz, Henry
  • Developing an Accessible National Information Infrastructure for People with Cognitive Disabilities – Coleman, Bill

Web Accessibility London 2010 Unconference

  • September 21, 2010 – London, England
    • “The unconference will have a motor impairment theme … ” but “… will also consider cognitive impairments and the wider-disability population.”

Recent Conferences

12th International Conference on Computers Helping People with Special Needs

  • July 14 to 16, 2010 – Vienna, Austria
  • Track IV, Session C: People with Specific Learning and Cognitive Problems: ICT, AT and HCI
    • Developing a Multimedia Environment to Aid in Vocalization for People on the Autism Spectrum: A User-Centered Design Approach – Al-Wabil, Areej
    • EasyICT: a Framework for Measuring ICT -Skills of People with Cognitive Disabilities – Dekelver, Jan
    • Involving users in the design of ICT aimed to improve education, work, and leisure for users with intellectual disabilities – Gutiérrez y Restrepo, Emmanuelle
    • Methodological Considerations for Involving SpLD Practitioners in the Design of Interactive Learning Systems  – Karim, Latifa
    • PDA software aimed at improving workplace adaptation for people with cognitive disabilities  – Ferreras, Alberto
    • The Performance of Mouse Proficiency for Adolescents with Intellectual Disabilities – Wu, Ting-Fang
    • Towards an Interactive Screening Program for Developmental Dyslexia: Eye Movement Analysis in Reading Arabic Texts – Al-Wabil, Areej
    • When Words Fall Short: Helping People with Aphasia to Express – Al Mahmud, Abdullah
  • Track IV, Session D: Easy – to – Web
    • Adaptive Reading: A Design of Reading Browser with Dynamic Alternative Text Multimedia Dictionaries for the Text Reading Difficulty Readers – Chu, Chi Nung
    • Easy-to-web search for people with learning disabilities as part of an integrated conception of cognitive web accessibility – Erle, Markus
    • EasyWeb – A Study How People with Specific Learning Difficulties Can Be Supported on Using the Internet – Matausch, Kerstin
    • In-Folio: An Open Source Portfolio for students with learning disabilities – Ball, Simon
    • Supporting the web experience of young people with learning disabilities – Weber, Harald
    • The need for Easy-to-Read information on web sites – Bohman, Ulla

Annual Conference of The American Association on Intellectual and Developmental Disabilities

Know of another such conference? Please post a comment.

New Readability Tool Built Into Safari 5

Reader, a new feature of Safari 5, removes visual distractions from Web pages. This is a boon for people with cognitive disabilities, indeed everyone distracted by advertisements, contextually-irrelevant images, etc..

How Reader Works

For any Web page Safari 5 recognizes as an article, a gray button labeled “Reader” appears to the right of the Web address at the top of the screen. (The button is indicated by a red arrow in the image below of a Wikipedia page.)

Wikipedia page displayed within Safari 5.

Clicking the button, which changes its color to purple, activates Reader. The following image shows the result: a view of only the page’s primary text.

Three columns. The middle one contains the article's text. The others are black bars that mask distracting elements.

Clicking the button again or pressing the “Esc” key deactivates Reader.


  • A toolbar appears near the bottom of the screen. It presents options to reduce- and enlarge text size, forward the article via e-mail, and print it.
  • Safari 5 remembers the selected text size the next time the article is viewed.
  • Every page of the article is displayed within Reader.


  • Neither the toolbar nor the Reader button are keyboard accessible.
  • The toolbar appears for just seconds, so using it means acting fast and with accuracy.
  • Clicking a link to an external page, even if Safari 5 recognizes it as an article, displays it outside of Reader.


This is the first time such a readability tool has been built into a popular Web browser. I hope it is adopted by all the others. For now, equivalent tools can be added to browsers via plug-ins.  Three I have reviewed are listed below.

I also hope these readability tools show Web designers how difficult the reading experience can be. Large- or animated advertisements and other distractions can drive people from Web sites. Simple page layouts designed for readability can have the opposite effect. An example of this is Craig’s List.

Other Readability Tools

Free Readability Tool for iPhone & Desktop Web Browsers

Readable Tool Better Than One David Pogue Says Is Best Tech Idea Of 09

Readability: Free Tool Strips All Distractions From Web Pages

Note: The version of Safari referenced above is 5.0 (7533.16). No endorsement of it is intended or implied.

Free Readability Tool for iPhone & Desktop Web Browsers

TidyRead, similar to the other such tools I have described, is a free bookmarklet that strips the clutter from Web pages and otherwise makes them easier to read.  Unlike them, it works on the iPhone and the iPod Touch.  It also works with Firefox, Chrome, Safari, Opera and Internet Explorer.

Configuration Options

While TidyRead does not have as many configuration options as those offered by Readability and Readable, it does offer them on demand.  With the latter two tools, users configure how they want the main content of Web pages to appear, and then they add the bookmarklet to their Web browsers.  TidyRead has no pre-configuration.  After its bookmarklet is installed, a click to it presents the toolbar, shown below, at the top of each page it displays.  It adjusts the presentation of content as options are selected.

toolbar with 4 buttons for changing page background color, 4 for text size and 3 for page width

The toolbar has buttons to change background color, font size and margin width.  Its “More” menu includes settings to change the font family and the text alignment.

Error Handling

TidyRead does not have the great feature Readable has, which enables users to select the content they want to display.  If TidyRead can not determine the main content of a page, it displays the error, “This page doesn’t look like an article, and TidyRead couldn’t extract”.

iPhone & iPod Touch

TidyRead can be installed on the iPhone and the iPod Touch by syncing Firefox- or Safari bookmarks with iTunes.  Alternatively, there are step-by-step installation instructions.

Relevancy To Clear Helper Project

The reason I am investigating these readability tools is that they could be quite useful to people with cognitive disabilities who are distracted by extraneous content on all Web sites.  Maybe, in addition to offering easy- and standard versions of the future Clear Helper Web site, I could offer on it one of these tools to install.


Upcoming Web Site to Include Accessibility for People with Cognitive Disabilities

I am working on a Web site that will incorporate two significant features with which I have experimented: text-to-speech (TTS) and plain language. The site will have other accessibility features for people with cognitive disabilities, text enlargement and text highlighting among them.

The site will be a report for The Massachusetts Department of Developmental Services (DDS).  It will be published by The Eunice Kennedy Shriver Center, for which I work.  Because the constituency of The DDS is people with intellectual disabilities, Shriver project staff would like the report to be as accessible to them as can be afforded at this point.  I have thus been in discussions with representatives of Web-accessibility technology companies.

Web Accessibility Technologies

An accessible content management system (CMS) from WebCredible has been purchased for the Web site. WebCredible reports that, in addition to its purpose of creating accessible Web pages, the CMS includes two other features, ones that attracted me to it.

  • Its back-end, content-management interface is itself accessible; and
  • “… content editors are forced to … produce accessible and well-written page content …”.

I will begin using the WebCredible CMS next week.  Future blog posts will describe what I learn about it.

The Shriver project staff and I are considering two other products from The United Kingdom: BrowseAloud and ROKTalk.  Each provides TTS and text-accessibility features for Web sites. I have mentioned both products in previous blog posts, and reviewed the one from ROKTalk.  A future post will describe which we choose, and why.

Plain Language

The report to be published is long and contains complex information.  For the home page of each section, we plan to write a plain-language version of the section’s main points.  I am concerned about doing this well because, as I have said before, writing “easy” text is not so easy.  Our related efforts will also be the subject of future blog posts.

Note: No endorsement of the above-mentioned products is expressed or implied.

Cognitive Web Accessibility Assessment: First Attempt, Part 2 of 3

This post is the second part of my first structured attempt to evaluate cognitive Web accessibility.  I am using WebAIM’s Cognitive Web Accessibility Checklist and its WAVE accessibility evaluation toolbar to assess the Web site of Down’s Syndrome Scotland.  For details, see Part 1.

This post covers the checklist sections of:

  • Multi-Modality;
  • Focus and Structure;
  • Readability and Language.


  • Checklist Section: Multi-Modality
    • Guideline: Provide content in multiple mediums
      • I could find no instances of video- or audio alternatives to textual content.
    • Guideline: Use contextually-relevant images to enhance content
      • Many, particularly the header images, are not contextually relevant to pages’ textual content.  There is some contextually-relevant imagery.  Examples: Meet Keith, Titan Abseil.
    • Guideline: Pair icons or graphics with text to provide contextual cues and help with content comprehension
  • Checklist Section: Focus and Structure
    • Guideline: Use white space and visual design elements to focus user attention
      • Picture of Resources Page. Shows disproportionately-large header image.The header images focus user attention to themselves, not to the content of page bodies.  An example (pictured), is the Resources Information page.
    • Guideline: Avoid distractions
      • On assessed pages, the header images pull attention away from page-body content.  The home page has an element of text that is animated in the site’s default / standard view and in its optional views.
    • Guideline: Use stylistic differences to highlight important content, but do so conservatively
      • Important textual content is bold.  It is frequently large and red in color.  One point is recorded.
    • Guideline: Organize content into well-defined groups or chunks, using headings, lists, and other visual mechanisms
      • Pages have short paragraphs.  Headings are used, but incorrectly on some pages.
    • Guideline: Use white space for separation
      • White space is used to separate page elements.
    • Guideline: Avoid background sounds
      • There are no background sounds.
  • Checklist Section: Readability and Language
    • Guideline: Use language that is as simple as is appropriate for the content
      • I am ignoring this guideline. I do not understand how it is different from the one (below): “Maintain a reading level that is adequate for the audience”.
    • Guideline: Avoid tangential, extraneous, or non-relevant information
      • This guideline is met throughout the site.  One point is recorded.
    • Guideline: Use correct grammar and spelling
    • Guideline: Maintain a reading level that is adequate for the audience
    • Guideline: Be careful with colloquialisms, non-literal text, and jargon
      • This guideline is met throughout the site.
    • Guideline: Expand abbreviations and acronyms
    • Guideline: Provide summaries, introductions, or a table of contents for complex or lengthy content
      • This guideline is not applicable.
    • Guideline: Be succinct
      • This guideline is met throughout the site.
    • Guideline: Ensure text readability
      • These criteria meet this guideline: line height; text spacing and justification; sans-serif fonts; adequate text size; content-appropriate fonts; paragraph length; and adequate color contrast.
      • These criteria do not meet this guideline: Line length (exceeds 80 characters); and horizontal scrolling (necessary if text size is increased by 200% to 300%).


Two of three possible points are recorded.  Combined with the points from Part 1, the subtotal is 4 of 5 points.


  • A point is recorded only if a site or a significant part of it consistently follows a guideline.  The Down’s Syndrome Scotland site did not meet this criterion for any of the Multi-Modality guidelines, so no related point is recorded.
  • I assessed Web pages only, not the many linked PDFs.

50+ Readability Resources Related To Cognitive Web Accessibility

I have created an index of readability resources related to plain language; measurement tools; guidelines, research; content; symbols; and  free- and commercial products and services. At the time of this writing, there are over fifty. I will add more as I find them.

Characteristics Of Readability Listings

  • All have links to the original sources.
  • All are annotated with related information, primarily edited quotes from source pages.
  • The majority are free- and commercial products and services.  The rest are research articles.
  • The publication dates of original studies and articles range from 2001 to 2009 / present.

Links to Readability Index & RSS Feed


Technorati Verification Code = 63S9AZXDSA9K

Ray Kurzweil’s Blio eReader: New, Free & Accessible to People with CD

Ray Kurzweil is a giant in the accessibility industry.  He has been inventing reading machines and devices used by people with visual- and reading disabilities for 35 years.  His newest creation is the Blio eReader, digital-book-reading software.

Note: At the time of this writing, the Blio eReader is not yet available to the public.  However, in a CNET interview (video below), Ray Kurzweil says it will be within one month.

Blio eReader Feature Highlights

  • It combines full-color, digital content with Web content, video, and audio narration.
  • It runs on Windows computers, tablets and mobile devices such as the iPhone.
  • It is free, and has access to a million free books. (Presumably, there will be a store of books for sale.)
  • Its catalog includes “cookbooks, travel guides, how-to books, schoolbooks, art books, children’s stories, and magazines”.
  • Books can have interactive, multi-media content and quizzes.

Accessibility Features Good for People with Cognitive Disabilities

The Blio eReader:

  • reads books aloud via either an accompanying, human-read audio track or via a text-to-speech reader;
  • synchronizes its synthesized voices with “follow-along word highlighting”;
  • has adjustable reading speed and font size;
  • has a text-only mode good for minimizing distractions and also for displaying on small screens;
  • uses a “3D book view which includes realistic page turning”; and
  • can be connected to a personalized set of reference Web sites for “one-touch look-up of highlighted phrases”.

In the YouTube video below, CNET interviews Ray Kurzweil about the Blio eReader.  A demonstration of it begins at about 2 minutes, 23 seconds (point 2:23).  This video is not closed captioned.


Note: No endorsement of the Blio eReader is intended or implied.

Readable Tool Better Than One David Pogue Says Is Best Tech Idea Of 09

Readability is a free Web-browser bookmarklet that strips all distractions from Web pages. David Pogue, the personal-technology columnist for The New York Times, called it the “… single best tech idea of 2009 …” and a “… real life-changer …”.

Pogue, D. (2009-12-31). The Pogie Awards for the Year’s Best Tech Ideas. New York Times. Retrieved from

It is indeed a promising tool.  Yet, as I pointed out in my review of Readability, it has a significant problem. For many Web pages I tested, it could not determine which was the main text content, or it displayed only a snippet of it.

Readable (created by Gabriel Coarna)

Readable, also a free Web-browser bookmarklet, includes the same configuration features, has more of them, and has a feature that solves Readability’s problem.  If Readable can not determine a page’s main text content, it enables users to identify the text they want to read.  A user holds down the Control (Ctrl) key, selects the text with the mouse, and clicks it.  Readable then shows that text free of extraneous content. (Instructions are on the “Tutorial” page.)

Readable can be used with Firefox, Safari, Chrome, Opera and Internet Explorer.