|
Project R4-1: Computer Facilitated
Evaluation User-Machine Performance
These projects seek to develop computer-assisted assessment
techniques to measure the performance characteristics and usability
of AAC technologies.
Universal Logfile Format and ACQUA - We have completed
a preliminary version of a universal logfile format, for collecting
human-computer interaction data. This is a flexible logfile
format that is powerful enough to support the most common data
collection requirements while also providing an extendable framework
for customized logging needs. We have continued to develop the
Augmentative Communication Quantitative Analysis (ACQUA) program
that reduces and analyzes logfiles. Among the features included
in ACQUA are over 30 statistical tests, and the ability to analyze
multiple logfiles; output to Microsoft Excel and statistical
applications; and analyze data at machine event, user event,
word, and sentence levels.
AAC Simulator - We completed work on an initial version
of the simulator - a reconfigurable AAC application that includes
extensive logging capabilities. The most recent enhancement
provides a "human simulator" which allows automated
processing of text materials. This is important for studying
the efficiency of different AAC features.
Manufacturer's Working Group - The RERC-Manufacturers
Working Group on Automated Data Logging is a discussion group
focused on specifying the function and structure of logfiles.
Currently, the group includes six corporate entities (Dynavox,
PRC, Liberator UK, Applied Human Engineering, Saltillo, Enkidu,
Words+) and four universities (Purdue, Arizona State, UB, New
Hampshire). Each manufacturer is planning to incorporate logfiles
into their products. The universities are conducting research
projects. We are developing a text corpus with statistical characteristics
for use in device performance evaluation.
Design Salon - We will develop a procedure to generate
and prototype devices using Flash software to explore device
techniques that would promote interactive communication.
Assessment Materials and Protocol - We are developing
a library of texts and procedures for device performance assessments.
This entails organizing existing and new texts (obtained through
conventional sources and through the web-crawler) in a repository
for use by scientists and clinicians, and formalizing some of
our analysis techniques. A standardized method for text storage
and retrieval is also being developed.
Webcrawler - We are researching and developing an autonomous
web-crawler that can collect vocabulary and text materials from
web pages. Rather than mindlessly collecting pages, the web-crawler
is trained to retrieve only specific types of text. For example,
the web-crawler might be instructed to collect only essays,
sports stories, or narrative accounts. The domain categories
may be topic, genre, style, language, or any other classification
scheme that can be reliably correlated with language statistics.
A sophisticated pattern recognition network is used to differentiate
between domains. Collected texts will be archived and tagged
in a general format compatible with most commercial and academic
AAC software. The web-crawler project will build upon preliminary
research performed by Enkidu under NIDRR SBIR grant ED-98-CO-0031,
during the course of which a prototype web-crawler was used
to collect a 100 million word corpus used for word prediction
studies.
Project R4-2: Augmentative Communication
Performance Analysis
These projects study the different aspects of communication
rate and performance.
Comprehension and Acceptability of Pause-Altered Synthetic
Speech Discourse - We completed a project to determine the
minimum comprehension rate required by listeners to comprehend
synthetic speech. We found systematic changes in comprehension
and subjective judgment performance as a function of increases
on communication rate. Seventy words per minute appear to be
the optimal communication rate for listening to synthesized
discourse.
Production Rate - Scanning - As an extension of the
data-logging project, we are studying the communication output
of 15 subjects learning to use AAC scanning systems. This represents
the first multi-subject, long-term study of device acquisition.
Each subject practices with 2 devices, 15 hours each. This research
will help us to test our data logging and analysis programming
and provide baseline measures for assessing communication rates
that are so desperately needed in our field. Analysis is being
completed.
Production Rate - Field Methods - We are working with
Arizona State University to develop automated techniques to
study the communication of device users in the field. We are
completing the collection of communication logs of 8 AAC users
during everyday communication activities and analyzing the files
to determine the performance measures that can be reliably used
to assess device operation in the field. Data analysis is currently
underway.
Production Rate Minspeak - We are working with the
University of New Hampshire to analyze a longitudinal study
of 4 individuals learning to use Minspeak devices (50 hours
training). We have been working to modify the data to be analyzed
with ACQUA.
Interaction - This project seeks to develop a valid
approach towards analyzing interactive dialogue and to determine
the rates of communication associated with AAC users during
conversations. We have developed protocols for transcription,
grammar, and common ground coding of multimodal, device-mediated
interaction; detailed the transcription of a set of 4 videos
for agreement and reliability assessment; and are in the process
of making agreement estimates using another transcriber and
rater. Next steps include perform reliability testing using
five other raters, research via a focus group to refine the
technique, and evaluate the ability of the technique to differentiate
communication by individuals using different communication interfaces
and communicating at different rates.
Project R4-3: Communication Performance
Model
We are developing a workable, joint action analysis technique
and specifications for a transcription tool. We have run some
pilot analyses, and generated a field-initiated grant in 2002.
We will use data and theoretical work from the previously described
projects to develop a model of communication performance (ala
SOS talk). In year 5, we will begin to work with a simulating
device using proprietary and commercial software and are looking
to emphasize the simulation aspects of this project in the future.
Deliverables for the project include Joint action analysis
scheme, field-initiated grant and pilot by Fall 2002, a theoretical
framework for studying augmented communication and device design,
and an initial simulation of AAC performance using software
being developed by Enkidu and commercial simulation software.
To date, we have developed a prototype joint action analysis
scheme, wrote and submitted a manuscript to AAC on the initial
theoretical framework, reanalyzed data from previous a study
using a new perspective and submitted results to AAC, and advanced
the simulator to testing stage.
Project R4-4: Comparative Study
in Communication Performance: Facilitating Communication Performance
in the Workplace
This project focuses on designing performance analysis techniques,
collecting comparative device performance use (Frametalker project),
piloting the procedure for completing joint action analysis,
and developing a testable model of communication performance.
It is co-funded through an SBIR Phase II (Frametalker) and an
ESPIR grant to John Todman in Scotland.
The RERC side of the project will focus on the design of a
communication performance evaluation protocol, and data analysis
of communication performance of device users in workplace context.
Year 4 will focus on protocol development and initial data
collection (5 device users, 5 natural speakers). Year 5 will
focus on a comparative study of the 5 device users using their
own devices vs. an utterance-based communication device.
Deliverables for the project include a Communication Performance
evaluation protocol, language materials (utterance and lexical
information) related to workplace communication and information
pertaining to system design, and a comparative analysis of device
use in the workplace.
To date, we are in the process of hiring a data collector
for the Temple site, have developed the data collection protocol,
and are analyzing data from the University of Dundee study site.
|
