|Research goals and objectives|
|General plan of work|
|Project 1: Establishing factors influencing subjective telepresence|
|Project 2: Developing an objective measure of telepresence|
The primary goal of the research component of this career plan is to formulate an integrated technological and psychological model of telepresence on which design guidelines for teleoperators and methods of teleoperation, aimed at enhancing performance in the named application arenas, can be generated. This goal is to be accomplished through a series of (three) studies having the following objectives:
1. identifying technological system and human psychological factors that influence subjectively assessed telepresence,These objectives are to be achieved by conducting human studies involving experimental subject performance in teleoperation (remote control robot) tasks developed by MSU in collaboration with ORNL. For example, remote pick and place tasks and part assembly/disassembly tasks will be used. The tasks will be designed such that the results of the study will have relevance to real-world teleoperator applications.
2. developing an objective measure of the strength of telepresence,
3. determining the strength of association between objective telepresence and performance, and
4. establishing states of teleoperation system features based on a model of telepresence that may enhance performance.
GENERAL PLAN OF WORK
Four research projects are to be conducted over the course of a four year
period. Each project is proposed as having a 12 month duration for completion
including experimental design (beyond that offered in the following sections)
and setup, subject recruiting, intense experimental training and testing (data
collection), data analysis, and dissemination of results. The projects are
scheduled such that the results of one can be used to guide work on a following
project; thereby, advancing the knowledge of telepresence on the basis of the
most current data available. Information on system and user factors having an
influence on subjective telepresence will be used in selecting variables
meriting control in correlative study of subjective versus objective
telepresence. Established objective measures of telepresence will then be used
in investigating the relation of telepresence to performance. A model of this
relationship will be used to develop teleoperation system and VE design
guidelines for optimizing telepresence and performance.
The experimental methods to be employed in the research projects will conform to the policies of the MSU Institutional Review Board (IRB). Human subject recruiting and participation in the described experiments will be handled according to IRB guidelines and “good professional practice,” as described by Martin (1991).
In general, classical designs of experiments will be employed to ensure wide acceptance of the results of the proposed studies. All data collected will be analyzed using analyses of variance (ANOVAs) to differentiate, for example, the influence of technological factors on telepresence from chance. Pearson’s product-moment correlation coefficients will also be used in the validation of objective measures of telepresence through subjective measures. Lastly, technical writing skills will be used to document results from data analyses and inferences and conclusions based on these results.
Project 1: Establishing factors influencing subjective telepresence
This project will establish specific technological teleoperation system and
human psychological factors that significantly effect subjectively assessed
telepresence. The project will involve relating changes in subjective
telepresence to manipulations of these types of factors in a controlled
The possible categories of factors to be investigated in this study for an effect on telepresence include:
1. teleoperator feedback fidelity and system dexterity,Methodology. Three experiments will be conducted in which as many as 30 human subjects will be required to perform a teleoperation task under different settings of specific factors falling within each of the above categories. Both experienced and naive users of teleoperators will be recruited from the technical staff of the RPSD of ORNL and the MSU student population, respectively, for participation. Teleoperator feedback fidelity and system dexterity will be manipulated in an experiment by implementing different visual and mechanical interfaces. Further, the quality of visual feedback will be altered via spatial and temporal perturbations in the form of visual display and motor control misalignments, as well as time delays induced in response of the teleoperator to control commands. Operator psychological state will be assessed for its impact on telepresence in another experiment by establishing user preconceived notions of how a teleoperation system should function via structured queries. User mental models will be considered in this study by exposing certain groups of subjects to various training protocols. The protocols will be related to changes in user telepresence. (User mental “pictures” of teleoperation system behavior are considered to be dynamic; therefore, different forms and durations of training are anticipated to have a substantial influence on the refinement of a model.) In addition to these experiments, task importance factors including reward associated with error-free performance, such as bonus compensation for subjects, will be studied in a separate investigation. Task difficulty will also be manipulated in this experiment by controlling task object type and features.
2. operator psychological state including predisposition to teleoperation and mental models of system behavior (based on training), and
3. task importance and difficulty.
Task. The teleoperation to be conducted
in these experiments will involve using MSU’s IE teleoperator for controlling
pick and place motions of a robotic arm (a Puma 560) for assembly of a small
parts structure/simple assembly. Subjects will be required to use 2-degree of
freedom (DOF) and 6-DOF hand-controllers to manipulate the arm in grasping and
moving small blocks in sequence to assemble a vertical structure. The structure
will be built on a base block with a hole for peg insertion. The assembly will
be completed by placement of additional identical blocks on top of the base
The actual task and manipulator will be presented to subjects through the use of closed circuit cameras in the remote environment integrated with Silicon Graphics Indy? systems in the local control room and at the remote site. Further, a virtual representation of the real-task will be provided to subjects, as an interface for control of the robotic arm. This interface has been developed by MSU using Sense8 WorldUp? VR development software and a Pentium Pro? personal computer (PC) to allow for accurate control of the manipulator. The VE is of high fidelity including scaling to the real-task environment and incorporating perspective adjusted textures on the manipulator and task objects.
Dependent Variables. Operator performance in the teleoperation task will be assessed in terms of time-to-completion of the structure assembly and the number of errors incurred in mismating parts. Beyond performance, three subjective measures of telepresence will be extracted in these studies through the use of structured questionnaires and rating scales. In order to gain insight into a subject’s predisposition to the teleoperation task in terms of immersive tendencies, Witmer & Singer’s (1994) IQ will be administered. The PQ (Witmer & Singer, 1994) will also be employed as a post-experiment measure of telepresence along with two visual analog scales (VAS) in the degree to which users feel involved in the remote environment and the degree to which the teleoperation experience involves unity of the user with the remote environment. (These are essentially the same scale; however, two responses will be collected to identify inconsistencies in subject perception of telepresence.)
Procedures. All subjects recruited for these experiments will initially complete the immersive tendencies questionnaire to establish their predisposition towards the teleoperation assembly. In the technological system factors study, subjects will be randomly assigned to use one of two visual display types including a helmet mounted display (HMD) or a large screen projection. As well, the participants will be randomly assigned to use either a 2-DOF joystick or a SpaceBall? for controlling the robotic arm. The fidelity of visual feedback from the system will be manipulated by, for example, using wire-frame, rendered or textured models of the manipulator and task objects in the virtual control interface. The dexterity of the teleoperator will be controlled by, for example, adjusting the incremental step size of arm motion and the responsiveness of the manipulator to control commands. Control time lags will be induced during task performance. In addition, spatial disturbances involving misalignments of video images (due to camera position) and displays of the virtual representation of the real-task environment with the real environment will be encouraged. (Investigation of the effect of these disturbances on telepresence is important because too great a focus has been placed on the impact of display type in previous research and there is significant room for improvement in the manipulation capabilities of teleoperators.)
In the psychological factors study, subjects will be randomly assigned to one of two training groups involving either limited task experience or training to asymptotic performance. Limited task experience will amount to familiarization with the remote environment setup, the functional capability of the MSU teleoperator, and its visual and mechanical interfaces (e.g., HMD and SpaceBall?). Those subjects provided with the opportunity to achieve asymptotic performance will be exposed to the same system information; however, they will be permitted to continue task training until flattening of their learning curve is observed. During this training period, subjects will be exposed to the system settings that they are to encounter in actual experimental testing.
Just prior to testing, subjects will be interviewed using the cognitive task analysis technique of concept mapping to characterize their mental model of system behavior based on the training experience. This map will later be related to teleoperation performance and telepresence by correlating concept nodes and links in a user’s map with the response measures.
The study of task importance and difficulty will allow for examination of
speed/accuracy trade-offs in the teleoperation task and effects on user
involvement and telepresence. Subjects in this experiment will be randomly
assigned to groups characterized by differences in combinations of monetary
multipliers assigned to productivity versus quality in the block structure
assembly. All subjects will be instructed in their payoff matrix prior to
training and actual testing, and will be informed of the results of training
sessions in terms of compensation earned. In this experiment, subjects will also
be randomly assigned to groups based on the size of the blocks to be used in the
teleoperation task. This grouping will be considered in devising the payoff
matrices motivating task importance.
During all of these experiments, each volunteer will participate in multiple test trials involving assembly of the block structure. Each trial will run until a subject has fully assembled the structure or an upper-time limit is reached; whichever occurs first.
Project 2: Developing an objective measure of telepresence
This project will develop an objective measure of telepresence for determining
whether the cognitive concept can be used to improve overall performance in
teleoperated systems. The project will involve performing correlative studies of
the impact of technological system and human psychological factors on subjective
measures of telepresence, as investigated in the first project, with changes in
objective measures. As well, factors hypothesized in the literature (Draper et
al., in press; Usher et al., in review) to be influential in telepresence will
The objective measures to be examined fall into the categories of measures of task involvement and mental constructs. Specific involvement measures might include EEG signals and heart rate variability (HRV), which could be dynamically assessed during teleoperator performance. For example, reduced HRV associated with particular teleoperation system factor settings may be indicative of increased user involvement in the teleoperation (Bryne, 1992). However, such a measure may not allow for a clear separation of user concentration on the local control environment from the remote manipulator environment.
Mental constructs, possibly having utility for assessing telepresence, to be considered in this project include SA (Endsley, 1995) and attentional resources (Draper & Blair, 1996). As previously mentioned, the strength of telepresence may be scaled by comparing the relative strengths of local and remote SA or attentional resource allocation. However, physical and electronic connections between the remote and local environments raise the same difficulty, as encountered with physiological measures, of separating, say, SA across the environments.
Methodology. An experiment will be conducted in which as many as 30 human subjects will perform a teleoperation task using the MSU IE teleoperator with system feedback fidelity and manipulator dexterity settings established based on the results of Project 1. Experienced participants will be recruited from the study population of Project 1. (Technical staff from ORNL will be excluded from this sampling to reduce participant support costs.) Additional subjects will be recruited from the MSU student population. Teleoperation system settings having produced both high and low subjective telepresence will be used. Additional, technological factors to be controlled in this experiment will be identified through the teleoperation literature.
Task. The teleoperation task will
involve controlling pick and place motions of a robotic arm for parts handling.
Subjects will be required to use a SpaceBall? to manipulate the arm in moving
small wood blocks from an input pallet located on a workstation (within the
robot work envelope) to a series of drop positions forming an established
pattern. Initially, the arm will be moved from a “ready” state to the input
pallet where a single block will be retrieved and moved to a sequenced drop
position under the control of the operator. This task will be repeated until all
parts have been moved from the input pallet to fill all vacant drop positions.
Dependent Variables. Operator performance in the task will be measured in terms of the number of successful (error-free) part retrievals and drops, where “error-free” is defined as retrievals absent of manipulator collisions with workstation surfaces and parts not currently being processed. Further, average part processing time will be determined along with overall task completion time.
The measures of telepresence to be recorded during this experiment include those subjective responses to be collected in the experiments of the first project and some of the objective measures of telepresence described above. The IQ will be administered at the outset of all trials and the PQ and VAS will be completed by subjects subsequent to task performance. The objective measures of telepresence will include HRV, extracted using the method described by Byrne (1992), and attentional resource allocation across local control and remote task environments. The latter measure will be quantified as a ratio of sensory perception and decision-making performance in local versus remote monitoring and alternative selection tasks, etc.
Procedures. In addition to performing the teleoperation task, participants will be required to perform two identical secondary monitoring tasks involving subject detection of warning signals in the form of a light presented on physical display panels in both the remote and local control environments. Operators will be required to respond to these signals by depressing keys on a keyboard corresponding to the location of signals (i.e., either a remote or local signal). Performance in the monitoring tasks will be recorded in terms of the percentage of signals detected in both environments.
The experimental intent of these tasks is to provide measures of operator residual attention, as distributed across the environments in which the teleoperation scenario plays-out. The ratio of residual attention allocated to the remote versus local environment is to be assessed as a predictor of telepresence by association with the subjective measures of telepresence. In addition, since the use of teleoperators often involves human performance of other tasks simultaneously, the inclusion of the monitoring tasks in the experiment may provide for a more realistic workload of subjects.
During experimental testing subjects will perform both the teleoperation task and monitoring tasks simultaneously in multiple trails until all parts have been processed. The data on subjective and objective telepresence resulting from these tests will be related to each other and to the controlled system variables using Pearson’s product moment correlation coefficients and canonical correlations.
Project 3: Assessing the strength of association between telepresence and performance
To date there have been few empirical investigations of experiential telepresence, either to demonstrate that it is a cognitive concept distinct from technological factors, or to assess the relationship between it and other outcomes of teleoperation. Sheridan (1992b) commented, “It has yet to be shown how important is the sense of (telepresence) per se as compared to simply having . . . good sensory feedback” for performance.
According to most technological explanations of telepresence, the likelihood and strength of the experience are largely determined by the quality of the human-machine interface. However, teleoperator performance, independent of telepresence, will also be largely determined by the quality of the human-machine interface. Therefore, it is difficult to imagine an experiment which could evaluate the benefits of telepresence on mission performance per se. An experimental manipulation of telepresence determinants would also change the quality of the human machine interface. For example, one might hypothesize that if display quality affects telepresence, subjects would rate telepresence higher while using a stereoscopic display than while using a mono-image display. Suppose that there is a similar improvement in performance. Is the improvement caused by telepresence, does it happen because the stereoscopic display affords more information than the mono-image display? Insight into this issue will be gained through the first experiment as part of Project 1; wherein, jointly observed changes in performance and subjective telepresence due to the use of the HMD versus the large screen projection unit may or may not correspond. If the latter is true, performance could occur independently of telepresence. In either case, study will be still be required to independently assess telepresence for its relation to performance.
It is possible to conduct an experiment examining the effect of task difficulty and factors internal to operators on telepresence and task performance, such as that proposed for the third experiment in Project 1, by maintaining the quality of the human-machine interface, in terms of displays and controls, across task conditions while manipulating operator cognitive involvement and practice. This study is expected to reveal differences or similarities in the magnitude of the rate of change in performance versus telepresence. This will provide insight into the hypothesized confound of performance with subjective telepresence, but will not directly relate teleoperation performance to an objectively assessed telepresence.
Methodology. Based on the above reasoning, an experiment will be conducted to make comparison of an objective measure of the strength of telepresence, validated in Project 2, with teleoperation task performance. The intent of this comparison will be to establish a foundation for a model of telepresence that is predictive of task performance. (The model will be defined and assessed in the final project).
This experiment will involve as many as 30 human subjects performing the teleoperation task proposed for Project 1. Subject recruiting will be undertaken in a manner similar to that of Project 2. The MSU IE teleoperation system will also be used in this study. Configurations of the system, assessed in terms of objective telepresence, during the experiment as part of Project 2 will be employed. System factor settings having produced low and high telepresence will be used in an attempt to explain performance in terms of telepresence; that is, an objective measure of telepresence will serve as a lone predictor of teleoperation system functioning.
Dependent Variables. Participants in this experiment will be required to perform in multiple teleoperation test trials during which performance will be assessed in terms of time-to-task completion and the number of errors committed. Objective telepresence in the task will also be evaluated in terms of HRV or attentional resource allocation across the local control and remote task environments to validate the settings selected as being representative of different levels of telepresence.
Project 4: Formulating and validating a model of telepresence for use in teleoperator design
This project will involve formulating an integrated technological and psychological model of telepresence. The model will relate changes in telepresence (or virtual presence), dictated by system and human factors, to performance. The model will be validated through an experiment in which performance in a teleoperation task will be predicted by the telepresence model and then empirically assessed using the MSU IE teleoperation system.
With respect to the model formulation aspect of this project, a psychological approach to telepresence taking an attentional perspective will initially be considered. Up to this point, such a perspective has not been examined. The usefulness of attentional resource allocation as a cognitive concept for objectively representing telepresence has been hypothesized in the proposal of Project 3; however, Draper et al. (in press) have offered that a structured attentional resource model may be useful as a heuristic for not only understanding the relationship among telepresence and performance, but characterizing the experience itself. Draper & Blair (1996) proposed an attentional model to remote and virtual environments featuring a structured resource approach based on a model by Wickens (1980). The approach identifies various stimuli that a human operator may be exposed to in a teleoperation scenario. Stimuli present in the computer-mediated world are presented within the local environment by computer-mediated displays. At the same time, there may be stimuli inherent in the local environment. In both the remote and local environments, stimuli may or may not be related to the task at hand; stimuli that are not task-related are distracters. Therefore, there are four types of stimuli present for processing by the human user: remote task-related, remote distracters, local task-related, and local distracters. In the attentional model, telepresence may be interpreted as a state arising from commitment of attentional resources to the stimuli of the computer-mediated world. The more attentional resources that a user devotes to these stimuli presented on displays, the greater the identification with it and the stronger the sense of telepresence.
This structured resource model has the potential benefit of being able to explain the incidence and strength of telepresence and the failure to observe a relationship between telepresence and performance. When attentional resources are totally committed to the computer-mediated world, the user will report a strong sense of telepresence. To the degree that attention is split between the remote and local environments, the sense of telepresence will be weakened. When attentional resources are totally committed to the local environment, the user will not feel at all telepresent in the computer-mediated world. However, even when completely immersed, not all of the information about the computer-mediated world that is provided to the user is task-related. There are distracters in the computer-mediated world as well as in the local environment. When attentional resources are committed to dealing with information provided by the computer-mediated world that is not task-related, telepresence is maintained but task performance is degraded. Therefore, a user may feel strongly telepresent but still exhibit poor task performance. In addition, there may be information provided by the local environment that can be used to enhance task performance. Allocating processing resources to this information enhances performance but degrades telepresence.
The attentional model appears to have good explanatory power for both telepresence and teleoperator performance. It has the advantage of being firmly grounded in a well-known and widely accepted psychological framework, and one that has been successfully used to understand human-machine interaction in the past. The model provides an excellent starting point for developing an integrated technological and psychological approach to telepresence in which specific system and task factors may be identified as promoting the salience of perceptual stimuli demanding operator attentions. In support of assessing an attentional model, Taylor & Rushton (1993) report that measures of telepresence and attentional “Absorption” are in fact positively correlated.
Methodology. An empirical investigation
will be conducted into the predictive utility of the structured attentional
resource model for describing the relation between attention and telepresence or
performance. Initially, Draper & Blair’s (1996) model will be further
defined in terms of quantitative/mathematical expressions of the named
relationships. The expressions will identify attentional resource pools (Navon
& Gopher, 1979) allocated to the different task and distracter stimuli for
Subsequently, an experiment will be conducted to validate these expressions. A setup identical to that employed in Project 2 will be used. This makes sense, as Project 2 will seek to validate attentional resource allocation as an objective measure of telepresence. The parts handling task and the secondary monitoring tasks will be used to quantify subject allocation of attentional resources to task stimuli or distracters across the local control and remote task environments.
The experiment will involve as many as 30 human subjects simultaneously
performing the teleoperation and monitoring tasks. Subjects for the proposed
experiment will be recruited from the MSU undergraduate and graduate student
populations. Participation will occur on a voluntary basis. The independent
variables to be examined in the study will include level of operator training
and task difficulty. The variables should reveal any effect of user expertise or
task involvement on the hypothesized relationships between attention,
telepresence and performance. This will be accomplished by exposing subjects to
the two different training protocols to be employed in the second experiment as
part of Project 1 and using wood blocks of varying sizes in the teleoperation
task. During the experiment the quality of the human-machine interface, in terms
of displays and controls, will be maintained across all task conditions.
Dependent Variables. The performance measures employed in Project 2 including the number of successful part retrievals and drops and the average part processing time will be recorded along with overall task completion time. Performance in the monitoring tasks will be recorded in terms of the percentage of signals detected in the local or remote environments. A mathematical expression of telepresence formulated on the basis of the structured attentional resource model will be resolved in terms of the performance measures.
Procedure. On the basis of the predictive utility of the attentional model, to be demonstrated through the planned experiment, teleoperation system design guidelines and configurations for optimizing telepresence and performance will be established. The guidelines are expected to be useful in decisions concerning retrofitting existing teleoperation systems for telepresence, or developing new systems capable of provoking in a user the psychological state producing superior performance.
EXPECTED RESEARCH RESULTS
The above described projects are anticipated to produce the following results:
1. identification of technological system and human psychological factors influencing telepresence in teleoperations,Impact of the proposed projects
2. establishment of an objective measure of telepresence for use in relating the strength of telepresence to teleoperation system performance, and
3. development of a comprehensive model of telepresence in attention for predicting performance and formulating system design guidelines.
The impact of this research on the pursuit of telepresence as a design ideal for teleoperators will be significant. The proposed projects offer experiments to investigate teleoperation system feedback fidelity and manipulator dexterity, as well as spatial and temporal disturbances in feedback on subjectively assessed telepresence and performance. Experiments are also proposed to investigate the effects of human operator predisposition to teleoperation, user mental models of system behavior, and task importance and difficulty on telepresence and system functioning. These studies will reveal significant teleoperation system and task factors or human factors for consideration in designing or configuring teleoperators for subjective telepresence and performance.
Additional empirical studies will demonstrate the potential usefulness of physiological variables, such as HRV, or established mental constructs, including SA, for objectively representing telepresence. The relation of these possible measures of the strength of telepresence to changes in teleoperation task performance, based system and human factors, will be established and a foundation for a comprehensive model of telepresence will be laid.
Lastly, the research projects will produce a structured attentional resource model as a heuristic for understanding telepresence and predicting teleoperation performance on its basis. The model will be validated and will yield the promised design guidelines for teleoperation systems and VEs intended to facilitate telepresence.
Documentation and dissemination of research results
In an effort to disseminate the knowledge gained through the proposed research,
technical reports will be prepared summarizing all literature reviewed,
experimental testing procedures, data analyses, results and inferences, and
experimental conclusions. On the basis of these technical reports, conference
proceeding papers and journal articles will be prepared. Major technical
conferences will be considered for presentation of the research results
including the Annual Meeting of the Human Factors and Ergonomics Society, the
IEEE Conference on Systems, Man and Cybernetics, the IEEE Conference on Robotics
and Automation, as well as the American Nuclear Society Meeting on Robotics and
Remote Systems. Journals to be considered for publication of scholarly writings
produced based on this research include the Human Factors Journal, Presence,
Ergonomics, the International Journal of Cognitive Ergonomics, etc. Beyond these
measures of publication, a world wide web (WWW) site documenting the results of
this career plan will be developed under the WWW home page of the Cognitive
Engineering and Decision-Making Technical Group of the Human Factors and
The material on this page describes work supported by the National Science Foundation (NSF) under Grant No. IIS-9734504. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the NSF.
For information about this page, contact Dr.
For information about Industrial Engineering, contact Dr. Jim Wilson.
Last modified: 05 Feb 2002 21:26