4. DISCUSSION

⇐ ПредыдущаяСтр 15 из 16Следующая ⇒

F₀+F

₀+F₀₀

F₀ +

^F0+F₀0

(1)

(2)

(3)

(4)

(5)

(a) Approach

(b) Adhesion

Figure 11. Capture by adhesion of a 50-jU. m-diameter object. Reprinted with permission from [32], D. S. Haliyo and S. Regnier, " Advanced Applications Using [mtiJMAD, the Adhesion Based Dynamic Micro-Manipulator, " IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Kobe, Japan, 2003, pp. 880-885. © 2003, IEEE.

Virtual Reality and Haptics in Nano- and Bionanotechnology

(a) Adhesion

(b) Capture

Figure 12. Physically based simulation, in virtual environment, of dynamic effects during releasing of an object. Reprinted with permission from [32], D. S. Haliyo and S. Regnier, " Advanced Applications Using [mtiJMAD, the Adhesion Based Dynamic Micro-Manipulator, " IEEE/ASME International Conference on Advanced Intelligent Mechatronics, " Kobe, Japan, 2003, pp. 880-885. © 2003, IEEE.

Hence, solutions, like an intuitive, VR-based interface hiding the details of performing complex 3D tasks in combination with 3D topography display, seem to be an attractive solution to some extent. The introduction of direct human interaction creates not only an enhanced measurement capability but also an automated technology presaging the nanofabrication or repair of nanostructures. The intuitive interface would include virtual tools or virtual effective probes together with a 3D representation used as a functional intermediary to map operator actions to the nanoworld. Its purpose is to assist a human operator with the different levels of planning and decision making involved in performing remote nanomanipulation operations. A number of features could be provided to make this possible:

• Virtual barriers to " fence off" areas or objects to protect against collisions (repulsive force fields) or contamination with other materials

Figure 13. In this sequence of images, from left to right, a 15-nm gold ball (circled) is moved, avoiding collisions, into a test rig. Force feedback is used to feel as the ball is pushed and when it has slipped off the tip. Reprinted with permission from [7], R. M. Taylor and M. Falvo, " Pearls Found on the Way to the Ideal Interface for Scanned-probe Microscopes, " IEEE Visualization Conference, 1997. © 1997, IEEE.

Virtual Reality and Haptics in Nano- and Bionanotechnology 15

• 3D stencils to constrain teleoperated motions to within predefined boundaries, improv
ing the reliability and safety of the probe tip

• 3D tape measure to determine straight-line distances between any two points within
the virtual 3D image

• 3D Protactor to quickly measure angles between any two intersecting 3D lines

• Color change indication of dynamic changes in object color to warn of impending over
turn, collisions, and so forth

• Surface coloring to indicate location and quantity of nongeometric information, such
as temperature and radiation

However, building patterns with large numbers of particles interactively is tedious and tends to be inaccurate because humans have difficulty in performing precise positioning tasks. Automating the process of moving a large number of objects (potentially hundreds or thousands) in near real time is necessary to make such nanorobotic tasks possible [36, 37]. Up to now, automatic motion and manipulation planning (termed nanoplanner) are not efficient at the nanoscale because of scale effects. Another extension is human/planner cooperation using a haptic and 3D VR simulator that would solve the complex strategy of nanoautomation. Introducing hints from human operators could provide a means of increasing the efficiency of automatic methods while still relieving the operator from the tedious, time-consuming work of producing the path manually. The human operator communicates with the automatic planner by manipulating virtual nano-objects using a haptic interface, capturing some paths' configurations that seem feasible, and passing them to the planner for execution.

2. 3. Experimental Virtual Reality Systems for Nanotechnology

In this section, we give an overview of a few VR systems developed for nanotechnology-related applications.

Researchers at the Institute of Industrial Science, University of Tokyo, have proposed a user interface for teleoperated nanometer-scale object manipulation [2, 8]. A 3D VR computer graphics display presents the topology of the nanoworld to the user, and a 1-DOF haptic device used together with a 2-D conventional mouse is the master manipulator, which also enables force and tactile feedback from the nanoworld.

Preliminary experiments on the user interface show that the interface can be used for telenanorobotics applications such as 2D assembly of nanoparticles and biological object manipulation.

The researchers also proposed that the force feedback haptic interface for such manipulation tasks can also be realized by using a haptic device constructed in Hashimoto's Laboratory called Hand Shake Device [38]. This interface is used to sense the nanometric forces in the order of nN on the operator's hand.

Scientists at the National Institute of Standards and Technology are developing an Open Architecture for Virtual Reality in Nanoscale Manipulation, Measurement, and Manufacturing [39]. The goal of this project is to develop an open architecture for users to incorporate powerful VR functionality in nanorelated research in a standard way. The architecture covers generalized VR related issues and, more important, covers issues that are specific to nanotechnology.

The Collaborator for Advanced Computing and Simulations at the University of Southern California is using immersive and interactive virtual environments for explorative visualization of multiscale simulation data for nanosystems [40]. They have implemented this system in an ImmersaDesk [41] virtual environment. The ImmersaDesk consists of a pivotal screen, an Electrohome Marquee stereoscopic projector, a head-tracking system, an eyewear kit, infrared emitters, a wand with a tracking sensor, and a wand-tracking input/output subsystem. A programmable wand with three buttons and a joystick allows interactions between the viewer and simulated objects. The rendering system is an SGI Onyx with two R10000 processors, 4 Gbytes of system RAM, and an InfiniteReality2 [101] graphics pipeline; it can render walkthroughs of multimillion-atom systems.

Virtual Reality and Haptics in Nano- and Bionanotechnology

3-D VR Computer Graphics (OpenGL)

Haptic Device

SGI (IMPACT10000 IRIXG. 2)

1. 66MHz, 32MB, Windows),

Figure 14. Telenanorobotics system. Reprinted with permission from S. Horiguchi et al., " Virtual Reality User Interface for Teleoperated Nanometer Scale Object Manipulation, Proceedings of the 7th International IEEE Workshop on Robot and Human Communication, " ROMAN'98, Takamatsu, Japan, 1998, pp. 142-147. © 1998, IEEE.

Also, scientists of the Argonne Futures Lab at the Argonne National Laboratory are exploring the use of spatially immersive virtual reality systems (e. g., CAVE [42, 102] and ImmersaDesk) for interactive modeling and visualization of nanotechnology-relevant molecular systems [43]. The goal of their work is to characterize the role that immersive virtual reality can play in improving the user's effectiveness in conceiving, modeling, and understanding large-scale molecular nanostructures. Spatially immersive display (SID) devices surround the user in real space with a 3D computer-generated visual and audio scene that is responsive to the user's point of view, orientation, and action.

Researchers at the University of North Carolina at Chapel Hill have developed a nanoma-nipulator system [44]. Scanning-probe microscopes (SPMs) allow the investigation and manipulation of surfaces down to the atomic scale. The nanoManipulator system provides an improved, natural interface to SPMs, including STM and AFM. The NM couples the microscope to a VR interface that gives the scientist virtual telepresence on the surface, scaled by a factor of about a million to one. It provides new ways of interacting with materials and objects at the nanometer scale, placing the scientist on the surface, and in control while an experiment is happening.

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 15. A scientist immersed in an atomistic model of a fractured ceramic nanocomposite. Reprinted with permission from [40], A. Nakano et al., IEEE Comput. Sci. Eng. 3, 56 (2001). © 2001, IEEE.

Nanomotion planning of complex nanotasks implies the integration of nanorobotics, 3D sensing, VR systems, and computer-aided design. Hybrid motion planning approach, which combines vision-based global path planning and model-based local motion planning, is currently investigated at KIST of Seoul [45].

Researchers at the Laboratoire Vision et Robotique of Bourges have developed human/ NM interfaces for real-time automatic motion planning. A 3D topology of the nanoworld is reconstructed, in real-time, from real 2D images from SEM images. The visual and 3D reconstruction recycling frames are less than 20 ms.

Visual Feedback

Graphics

Engine and

Host Processor

Intel-based PHANToM Controller

Intel-based

AFM Controller

Atomic Force Microscope

Figure 16. NM system using VE interface to SPMs. A student uses the NM to examine carbon nanotubes. Reprinted with permission from [7], R. M. Taylor and M. Falvo, " Pearls Found on the Way to the Ideal Interface for Scanned-probe Microscopes, " IEEE Visualization Conference, 1997. © 1997, IEEE.

Virtual Reality and Haptics in Nano- and Bionanotechnology

Repulsive sphere obstacle

Figure 17. Haptic and VR interfaces for real-time human/planner cooperation using physically based models for micro- and nanomanipulation. Integration of repulsive force fields for boundary of the workspace and microsphere obstacle for guidance of nanoplanning tasks. Courtesy of Professor Antoine Ferreira, Laboratoire Vision et Robo-tique of Bourges, France.

Thus, a truly interactive virtual environment (VE) that incorporates efficient, dynamic, physically based simulation (adhesive, repulsive, attractive nanoforces) and motion-planning techniques applicable to complex nanotasks has been experienced. In this system, a haptic (PHANToM) [99, 100] device and visual interfaces (3D head-mounted display with a head-tracking system) are used together to enable the user skills, and an automatic motion planner to cooperatively solve a motion task query [46] physically placing the scientist at the scale of a nanopart. The operator's planning difficulty is evaluated with easily measurable quantities such as the number of collisions produced by the n-forces, the NM clearance, and the contact forces acting on the cantilever tip.

Recently, VR has been used to define an intuitive interface to a nanoscale OT-based manipulation device [47]. To understand the underlying physical nature of the device, a VE that can simulate the physics of the laser beam and particle interactions is shown in Fig. 18a. The solid sphere at the center is the particle and is being illuminated by a dithered laser beam shooting in the direction of the plane. The small sphere to the left of the particle is the cursor for the operator. The operator steers the particle with a cursor, which is a small spherical ball that is position controlled by the stylus of the haptic device. The operator moves the cursor to the particle of interest and presses the button attached to the stylus, and then the particle is rigidly glued to the cursor. Figure 18b illustrates the instant snapshot when the particle is steered to the right and the displacement amount exceeds the physical capability of the potential field. The virtual environment reacts with a force that is in the direction counter to the steering motion.

Figure 18. Intuitive interface using virtual environment to control the motion of OT: (a) initial screen representing the solid sphere at the center as a particle and an ellipse representing the dithered laser beam shooting in the plane; (b) the virtual environment reacts with a force when the displacement amount exceeds the physical capability of the potential field. Courtesy Kevin Lyons, J. Res. Natl. Inst. Standards Technol. 108, 2003.

Virtual Reality and Haptics in Nano- and Bionanotechnology 19

3. BIONANOTECHNOLOGY

3. 1. Objectives and Related Problems

Bionanotechnology is an emerging area of scientific and technological opportunity. It is a new and rapidly growing interdisciplinary field addressing the assembly, construction, and utilization of biomolecular devices on the basis of nanoscale principles or dimensions. Research and product development at the interface of physical sciences and biology as applied to this area require multiskilled teams and often novel technical approaches for material synthesis, characterization, and applications. This section gives an overview spanning fundamental considerations of applications, with an emphasis on the development and construction of molecular motors. One such motor, the viral protein linear (VPL) motor [48], is shown in Fig. 19.

Almost all physical illnesses that affect the human body, such as cancer and AIDS, are caused pathology at the atomic level. Bulk chemical reactions were earlier used to study these molecules and their function. The problem with this approach is that the reactions occur stochastically, and we can only measure average quantities. As nanotechnology will allow for atomic assembly, disassembly, and rearrangement, the task of treating a disease can be viewed as simply a task of atomic engineering. Bionanotechnology applies the tools and processes of nano-/microfabrication to build devices for studying such biosystems. For example, it can be used to disassemble cancer cells or invalidate cancer viral material, alter DNA [49, 50], and RNA chains, or, in the near future, repair damaged tissue or perform restorative dental procedure on a patient's tooth. Precise manipulation could help scientists better understand the principles of molecular motors [51, 52], design chips for gene sequencing and engineering, and develop biosensors and lab-on-chip diagnostic devices.

To achieve this, it is essential for the molecular scientists to be able to visualize the atom-to-atom interaction in real time and see the results in a fully immersive 3D environment. Also, to facilitate user input in bionano systems it is essential to develop voice-, gesture-, and touch-recognition features in addition to the conventional visualization and manipulation techniques. VR technology is applied here that not only provides immersive visualization but also gives an added functionality of navigation and interactive manipulation of molecular graphical objects.

One of the active areas of research in bionanotechnology is the computer-aided drug design. The goal of drug design is to find a small molecule (ligand) that docks with a receptor cavity in a specific protein [53]. This docking can stimulate or inhibit some biological activity leading to the desired pharmalogical or drug effect, so the strategy of drug design is to accurately determine the protein (receptor) structure. Once this structure is determined, it may be possible to design new compounds (ligands)—candidates for new drugs—that can

Figure 19. (a) Three titin fibers can be used as passive spring elements to join two platforms and form a single degree of freedom parallel platform that is actuated by a viral protein linear (VPL) actuator (center). (b) The VPL actuator has stretched out, which results in the upward linear motion of the platform. The three-titin fibers acting as a spring are also stretched out. Their elastic behavior can be used as a passive control element or as the restitution force that will bring the platform back to its original position. Courtesy of Bio-Nano Robotics Laboratory, Northeastern University.

20 Virtual Reality and Haptics in Nano- and Bionanotechnology

specifically bind to this protein [54]. Conventional tools from computer science and engineering have been developed for drug-design systems, but they have their limitations. Such tools have no support for the interactive manipulation of molecules, and their functions are insufficient to express interactions between drug molecules and a protein [55]. These problems can, however, be solved using VR techniques because VR supports haptic interaction with force-feedback technology.

3. 2. Virtual Reality-Based Perception Solutions at the Bionanoscale

3. 2. 1. Virtual Reality for Structural Biology

Structural information on biological macromolecules is an essential requirement for our understanding of biological functions. From the very beginning of structural biology, visualization was essential for determining and understanding structures.

The initial work in this area was the use of Visualization and Virtual Reality for Bioin-formatics, especially for the 3D structural analysis of the biomolecular system. Surface- and volume-based visualization provide 3D concepts of biomolecular structure. VR offers a channel to reach into the molecular space in an immersive and interactive environment.

Progress in the x-ray and nuclear magnetic resonance (NMR) instrumentation and computer and software technology has led to an increasing rate of accumulation of new biological structures such as proteins and DNA. The Protein Data Bank [56] has a database of coordinate entries of 21, 248 proteins. Proteins are complex biological structures made up of several hundred atoms. The usual approach to visualizing such a molecule is to retrieve the coordinate files from the database and then use one of the molecular graphics software packages. These packages largely provide monorepresentations of molecular images without any interactivity. Use of VR Modeling Language (VRML) can provide a 3D and immersive environment with provision for stereo representations of the molecules [57]. This not only helps in better understanding the molecular structure but also provides insight into the chemical and biochemical properties of the molecule, such as the number of hydrogen bonds.

3. 2. 2. Complex Molecular Structure Modelling for Design and Analysis

The biosystems are made up of atoms, and as atoms cannot be observed directly, we can see them only in our imaginations. Commonly available physical models provide an intuitive representation of structural molecular biology. When applied to large molecules, such as proteins and the molecular assemblies found in cells, computer graphics simulation can be used to accurately portray various molecular computational models, with their varying complexities. Electrostatic field data around the molecules of interest, interatomic forces, laws of quantum mechanics, and so on represent the variety of biomolecular nanointeractions [58]. However, these methods lack the beneficial tactile and kinaesthetic attributes of real physical models. By coupling an accurate molecular dynamics (MD) simulation code to an immersive VR display with interactive capabilities and manual force feedback, " immersive" visualization of molecular atoms could be improved [59]. To create multimodality enhancements of such tangible models, VR-based technology is an interesting tool. By superimposing additional graphical information on top of fabricated models, for example, augmented reality, by incorporating support for voice commands and by providing haptic interface for sensing the electrostatic charges and interatomic collisions, the user would be able to interact with these virtual enhancements haptically while manipulating the physical model.

3. 2. 3. Unification Problem of Enhanced Vision and Force Display for
Automatic Bionanomanipulation

To precisely control and manipulate biomolecules, we need tools that can interact with these objects at the nanoscale in their native environments [60, 61]. Existing bionanomanipulation techniques can be classified as noncontact manipulation, including laser trapping [62, 63] and electro-rotation [64], and contact manipulation referred to as mechanical stylus-, AFM-, or STM-based nanomanipulation [65]. The rapid expansion of AFM studies

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 20. Secondary structure representation of the Bacillus circulans xylanase (PDB code, 1bcx) complexed with sulfate and cyclic xylose in the VRML viewer WebSpace (helix-tube, sheet-ribbon). Reprinted with permission from [57], J. Siihnel, Virtual Reality Modeling for Structural Biology, " Proceedings of the German Conference on Bioinformatics, " GCB '96, Lecture notes in computer science (R. Hofestadt, T. Lengauer, M. Loffler, and D. Schomburg, Eds. ), Vol. 1278, pp. 189-198. Springer, Berlin, 1997. © 1997, Springer.

in biology/biotechnology results from the fact that AFM techniques offer several unique advantages: first, they require little sample preparation, with native biomolecules usually being imaged directly; second, they can provide a 3D reconstruction of the sample surface in real space at ultrahigh resolution; third, they are less destructive than other techniques (e. g., electron microscopy) commonly employed in biology; and fourth, they can operate in several environments, including air, liquid, and vacuum. Rather than drying the sample, one can image quite successfully with AFM in fluid. The operation of AFM in aqueous solution offers an unprecedented opportunity for imaging biological molecules and cells in their physiological environments and for studying biologically important dynamic processes in real time [66].

Figure 21. Visual tracking of tip pattern (on the left) and real-time construction of the virtual biologic environment (on the right) with a continuously updated object model. Courtesy of Dr. Toshio Fakuda, University of Nagoya, Japan.

Virtual Reality and Haptics in Nano- and Bionanotechnology

At present, these bionanomanipulations are conducted manually; however, long training, disappointingly low success rates from poor reproducibility in manual operations, and contamination call for the elimination of direct human involvement. Furthermore, there are many sources of spatial uncertainty in AFM manipulation (e. g., tip effects, thermal drift, slow creeping motion, and hysteresis). To improve the bionanomanipulation techniques, automatic manipulation must be addressed. Visual tracking of patterns from multiple views is a promising approach, that is currently investigated in autonomous embryo pronuclei DNA injection [66]. Interactive nanomanipulation can be improved by imaging a 3D viewpoint in a virtual environment. Construction of a VR space in an off-line operation mode for trajectory planning combined with a real-time operation mode for vision tracking of environmental change ensures a complete " immersed" visual display. Figure 22 shows an example of a 3D biomicro-/nanomanipulation system with a 3D VR model of the environment including the bio cell, and carrying out the user viewpoint change in the virtual space [67].

Figure 22. Haptic force measurement process on the Zona Pellucida of mouse oocytes and embryos. Reprinted with permission from [66], Y. Sun et al., IEEE Trans. NanoBioSci. 2 (2003). © 2003, IEEE.

Since its discovery by Binning [4], AFM has been the main contact mode tool and is used to manipulate macromolecular biological assemblies as well as individual biomolecules [68, 69]. However, the contact mode can also be used where higher force is desired, as in cutting or dissection. As application, elastic properties of DNA, and identification of the different force-extension regimes in DNA are currently controlled with a haptic interface. However, contact manipulation can damage specimens as a result of uncontrolled forces at the contact interface between the tip and the surface. Introduction of physically based models of contact nanoforces (friction, rugosity, elasticity of cell membrane), liquid manipulation forces (viscosity, density), and continuum models (electrostatic, adhesive) into 3D virtual objects will generate realistic virtual nanoenvironments for guided simulations. As an example, aided by nanomanipulation systems and MEMS multiaxis cellular force sensors, biomembrane mechanical characterization and modeling have been conducted [70] at the University of Minnesota. Figure 23 shows the force measurement process on the Zona Pellu-cida of mouse oocytes and embryos. The proposed biomembrane mechanical model enables vision-based biomembrane force sensing, assimilating vision, and force-sensing modalities. A real-time 3D virtual simulator with force display based on measurement of actual biocells will enable motion planning, tactile data feeling, and controlled tip/surface interactions.

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 23. (a) Low-magnification of the living cell using optical microscope. The AFM tip is adjusted to the top of the cell surface. (b) Low-magnification imaging of the living cell using tapping mode AFM with scanning range of 60 fxm. Reprinted with permission from [35], G. Li et al., " 3-D Nanomanipulation Using Atomic Force Microscopy, " IEEE International Conference on Robotics and Automation, Taipei, Taiwan, 2003, pp. 3642-3647. © 2003, IEEE.

3. 2. 4. Molecular Dynamics Simulations in Virtual Environment

MD is a computer simulation technique in which the time evolution of a set of interacting atoms is followed by integrating their equations of motion. In MD, we follow the laws of classical mechanics, and most notably Newton's law:

= ma

(15)

for each atom i in a system constituted of N atoms. Here, m_i is the atom mass, a_i = d2ri/dt²its acceleration, and F_i the force acting upon it, caused by the interactions with other atoms. MD simulations are based on the calculation of the free energy that is released during the transition from native to fusogenic state and compute the atomic trajectories and coordinates by numerically solving the equations of motions using an empirical force field that mimics the actual interatomic forces in the molecular system. MD simulations of complex molecular systems require enormous computational power and produce large amount of data in each step. The resulting data include the number of atoms per unit volume, atomic positions, velocity of each atom, force applied on each atom, and the energy contents. These results of MD simulations need to be visualized to give the user a more intuitive feel of what is happening. Haptic interaction used in conjunction with VR visualization helps the scientist to control/monitor the simulation progress and to get feedback from the simulation process as well [71]. Figure 24 shows the VR visualization of MD simulation. Figure 25 shows a VR representation of MD simulations in CAVE VE.

3. 2. 5. Computational Steering and Visualization of Complex Molecular Systems

Computational steering is the ability of the user to design or modify a simulation interactively in a VE during run time, which gives the user a tremendous advantage over postsimulation visualization and analysis of results. In run-time steering, the user does not have to wait until the end of the simulation to see the results of his modifications; instead, he can immediately see the result of the interactively changed parameters, giving him an opportunity to detect and modify them to steer the simulation to a desired output. In computational steering, the user can steer MD simulation by applying external forces into the computations. These external forces can help a complex molecular system overcome a potential energy barrier and can even steer the system to a new geometric conformation for further analysis. Thus, it provides a great advantage over targeted MD [72], which also targets a MD simulation to a desired output, but in which the user has no control over the simulations once it has started. Another advantage of steered MD over conventional MD is the possibility of inducing relatively large conformational changes in molecules on nanosecond timescales [73]. Figure 26 shows the force-induced unfolding of the protein titin [73].

Computational steering of MD simulation with the aid of VR visualization can thus help molecular scientists and researchers explore new models and their structural behavior, and

Virtual Reality and Haptics in Nano- and Bionanotechnology

Molecular Dynamics Simulation Virtual Environment

Molecular Dynamics Simulation

Figure 24. Block diagram showing virtual environment for MD simulations. Reprinted with permission from [88], Z. Ai and T. Frohlich, Comput. Graphics Forum 17, 1998. © 1998, Eurographics Association.

also to study the elastic/mechanical behavior of bionanotechnology-relevant complex molecular systems.

3. 3. Experimental Virtual Reality Systems for Bionanotechnology

The Virtual Molecular studio at Imperial College London [74-76] and the Technische Hochschule Darmstadt [77] have done some pioneering research in the bionanotechnology VR field. In addition, researchers at the Institut fur Molekulare Biotechnologie have set up an Internet-based Image Library of Biological Macromolecules [78]. This was one of the

Figure 25. MD simulation visualized in CAVE. Courtesy of Dr. Zhuming Ai, VRMedLab, University of Illinois at Chicago. Available at: http: //plum. bhis. uic. edu/zai/research/mol/molimg.gif.

Virtual Reality and Haptics in Nano- and Bionanotechnology

(a)(b) I и

2500 M

0 50 100 150 200 300 250

extension (A)

Figure 26. (a) Force extension profile from SMD simulations of titin I27 domain with a pulling velocity v = 0: 5 A/ps. The extension domain is divided into four sections: I, preburst; II, major burst; III, postburst; IV, pulling of fully extended chain. (b) Intermediate stages of the force-induced unfolding. All I27 domains are drawn in the cartoon representation of the folded domain; solvating water molecules are not shown. The four figures at extensions 10, 17, 150, and 285 A correspond, respectively, to regions I—IV in (a). Reprinted with permission from [73], Sergei Izrailev et al., in " Computational Molecular Dynamics: Challenges, Methods, Ideas" (P. Deuflhard, J. Hermans, B. Leimkuhler, A. E. Mark, S. Reich, and R. D. Skeel, Eds. ), Lecture Notes in Computational Science and Engineering, Vol. 4, pp. 39-65. Springer, Berlin, 1998. © 1998, Springer-Verlag.

first VRML applications in biology. The VRML division of this library contains more than 650 VRML representations of biopolymer structures.

One of the challenges in computational structural biology is enabling the efficient use and interoperation of a diverse set of techniques to simulate, analyze, model, and visualize the complex architecture and interactions of macromolecular systems. Scientists at the Scripps Research Institute, California, in collaboration with the Central Institute for Applied Mathematics, are developing a 3D graphics package termed SenSitus [79] that is capable of supporting VR devices such as stereo glasses, 3D trackers, and force-feedback (haptic) devices. This development effort will permit scientists to build models, combine atomic and volumetric data, and perform morphing and warping (flexible docking) interactively within a single computational environment. A force-feedback device measures a user's hand position and exerts a precisely controlled force on the hand. The software supports this process by calculating forces according to the correlation coefficient of density maps and crystallo-graphic data. The high sampling frequency required for force feedback (refresh rate > 1 kHz)

Figure 27. Interacting with a room-sized display of a three-dimensional protein in CAVE. Courtesy Dr. Karl, V Steiner, Delaware Biotechnology Institute, University of Delaware. Available at: http: //www. dbi. udel. edu/ bioinformcic. html.

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 28. Geometrical representation model for the molecule data. Courtesy of Dr. Juergen Pleiss, HIMM project, University of Stuttgart, Germany. Available at: http: //www. informatik. uni-stuttgart. de/ifi/gr/projekte/himm/ himm. html.

is achieved by means of the vector quantization algorithm developed by the group, which reduces the complexity of the data representation to manageable levels.

In addition to automated fitting, microscopists have a need to evaluate and manipulate docking models interactively, " by eye". Three-dimensional capabilities and the physics of touch offer tangible benefits for modelers who wish to explore a variety of docking situations in a VR environment.

In the Visualization Studio of the Delaware Biotechnology Institute at the University of Delaware, research is focused on molecular biology, genomics, proteomics, structural and computational biology, and biomedical imaging. The institute's interactive, immersive Visualization Studio consists of a SGI Reality Center system linked to a 15 x 7-foot FakeSpace [80] rear-projection screen. A six-processor Onyx 3200 visualization supercomputer with two graphics pipes drives a pair of Mirage 2000 projectors to deliver an edge-blended image with a total resolution of approximately 2560 x 1024 pixels. The studio is currently being outfitted with a haptic feedback system to further improve the immersive interface with the visualized environment.

The Institute of Technical Biochemistry at the University of Stuttgart has an ongoing project regarding the application of virtual reality technology in the area of molecular modeling [81]. The Highly Immersive Molecular Modeling (HIMM) [82] project is aimed at the integration of computer-aided molecular modeling tools and virtual reality systems like COVISE [83].

Recently, however, another way of visualizing atoms has become available. Researchers in the NAS data analysis group have developed an application called Virtual Mechanosynthesis, or VMS [84]. The user of this application sees various collections of atoms floating in the space above the NAS Visualization Laboratory's Immersive Workbench, made by Fakespace Inc. The VMS utility allows the user to see, move, and even " feel" simulated molecular structures in 3D space (Fig. 29).

A haptic, or force-feedback, device can also be used in VMS to interact with the simulation. This device is essentially a mouse mounted on a mechanical arm. It can be pushed around in three dimensions and, by way of a clever arrangement of small motors, can push back. Once users " attach" to an atom using the haptic interface, they can feel the attractive and repulsive forces as this atom pushes and pulls on its neighbors.

The Theoretical Biophysics Group at the University of Illinois at Urbana Champaign have implemented a system termed Interactive Molecular Dynamics (IMD) [85], which permits

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 29. In the graphite sheet being explored, gray spheres represent carbon atoms, and green spheres represent hydrogen. Using the pointer, the user has " grabbed" the pink atom for manipulation. Courtesy of Dr. Christopher Henze, NASA Ames Research Center. Available at: http: //www. nas. nasa. gov/About/Gridpoints/ PDF/nasnews_V04_N02_1999. pdf.

manipulation of molecules in MD simulations with real-time force feedback and graphical display. IMD consists of three primary components: a haptic device controlled by a server that generates the force environment, a MD simulation for determining the effects of force application, and a visualization program for display of the results. Communication is achieved through an efficient socket connection between the visualization program Visual Molecular Dynamics (VMD) [86] and the molecular dynamics program (NAMD) [87] running on single or multiple machines. A natural force-feedback interface for molecular steering is provided by a haptic device.

Another kind of VR interface for MD simulation has been developed at the Fraunhofer Institute for Computer Graphics, Germany. This system, called RealMol [88], is implemented to run on CAVE or any other computer system with a head-mounted display or a stereo projection screen. The MD simulation program for RealMol is NAMD, and the communication between the two for exchanging molecular data is achieved by RAPP [89], developed at the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign. The IDEAL (interaction device abstraction layer) system [90] is used to interface RealMol with the CAVE rendering system. IDEAL is independent of the choice of rendering system, provides an easy to use interface, and handles devices for interaction in a virtual environment. With the use of IDEAL, a cyberglove can be displayed in the virtual environment as a mapping of the hand (Fig. 30).

Modeling proteins as biological motors, as well as designing proteins not found in nature, is a rapidly developing field. Keeping this in mind, researchers at the University of North Carolina at Chapel Hill have built SMD [91, 98], a system for interactively steering MD calculations by adding user-specified external forces into the computation on the fly. Steering implies that the user is able to " tug" an atom or a group of atoms toward a desired target position. SMD consists of two software components. The first one is the software component for performing MD simulations on a molecular system and studying its response to an externally applied steering force. The display component of SMD is VMD (Fig. 31).

An example from the University of Washington is given in Figure 32, where the super-imposition of multimodal interaction can greatly complement and accelerate the learning process of the operator. For initial demonstration, a superoxide dismutase (SOD), an essential enzyme for cellular functioning that exhibits a strong electrostatic funneling effect, is modeled. In this scenario, the user holds the superoxide radical with the haptic device probe, and as it nears the charge field of SOD, strong forces pull the superoxide free radical toward the Cu and Zn ions at the active site of SOD. At the same time, the user sees

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 30. (a) Molecules can be grasped with a cyberglove and moved to a desired position. (b) 3D menu can be activated and selected by a gesture of the cyberglove. Reprinted with permission from [88], Z. Ai and T. Frohlich, Comput. Graphics Forum 17 (1998). © 1998, Eurographics Association.

the secondary structure of the SOD enzyme as an augmented reality overlay on top of the physical model [59].

The researchers have also developed a Virtual Reality Peripheral Network (VRPN) [92]. The VRPN is a set of classes within a library and a set of servers that are designed to implement a network-transparent interface between application programs and the set of physical devices (tracker, etc. ) used in a VR system. The idea is to have a PC or other host at each VR station that controls the peripherals (tracker, button device, haptic device, analog inputs, sound, etc. ). VRPN provides connections between the application and all of the devices using the appropriate class-of-service for each type of device sharing this link.

STALK [93] is a VR-based system developed at the University of Illinois at Urbana-Champaign and Argonne National Lab for studying the docking of a ligand molecule with a protein binding site. STALK, shown in Figure 33, uses a parallel genetic algorithm library to search for a low-energy conformation. An interface to the CAVE virtual reality system allows a scientist to visualize the genetic algorithm's progress and to interact with the algorithm by, for example, changing the position or structure of the drug molecule and restarting the algorithm so that the new parameters are incorporated [94].

Figure 31. (a) Specifying a " tug": selected atom is highlighted as the red sphere at upper right. (b) Moving the tug: target position follows pointer. (c) System response after 300 fs of simulation. Reprinted with permission from [91], J. Leech et al. IEEE Comput. Sci. Eng. 3, 38 (1996). © 1996, IEEE.

The Virtual Biomolecular Environment (VIBE) [95] is a VR-based system proposed at Argonne National Lab that would provide an environment for drug design. It consists of a massively parallel computing system to simulate the physical and chemical properties of a molecular system, CAVE for immersive display and interaction with the molecular system, and a high-speed network interface to exchange data between the simulation and CAVE. VIBE enables molecular scientists to have a visual, auditory, and haptic experience with

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 32. Display of force vector field around active site of superoxide dismutase (SOD), and (b) user interacting with SOD models using HMD and PHANToM. Reprinted with permission from [59], G. Sankaranarayanan et al., " Role of Haptics in Teaching Structural Molecular Biology, " 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Los Angeles, CA, 2003. © 2003, IEEE.

a chemical system while simultaneously manipulating its physical properties by steering, in real-time, a simulation executed on a supercomputer.

The Protein Interactive Theater (PIT) [96] is an integrated virtual environment for steered MD constructed at the University of North Carolina at Chapel Hill. The PIT is a dual-screen stereo display system for two operators seated at a table, as shown in Figure 34. Each operator wears a pair of CrystalEyes liquid crystal shutter glasses with an attached tracking sensor and views a stereo image projected on the screen across the table. The two operator's views are in registration, so that the operators agree on the apparent position of objects in the physical workspace and can augment their discussions using hand gestures. Each operator also has a tracked, six degree-of-freedom, handheld controller that provides pointing, picking, and other scene manipulations. Dials and buttons at the table corner shared by the two operators provide top-level control over the display parameters

Figure 33. A user in STALK Courtesy of MCS Division, Argonne National Laboratory. Available at: http: //www-fp. mcs. anl. gov/ccst/research/reports_pre1998/comp_bio/stalk/docking. html.

Virtual Reality and Haptics in Nano- and Bionanotechnology

Figure 34. (a) Details of PIT environment, (b) Two users interacting in the PIT. Courtesy of Dr. Mary Whitton, Photograph by Todd Gaul, Nanoscale Science Research Group, University of North Carolina at Chapel Hill. © Dr. Russell M. Taylor.

and the dynamics simulation. A separate, flat-panel LCD monitor in front of each user, visible through their stereo glasses, provides access to a conventional windows, keyboard, and mouse interface for detailed control of the visual representation of the molecule and the MD simulation.

4. DISCUSSION

It is hoped that this review will give students and professionals a feel of the applications and utility of VR in computational biology, nanotechnology, and bionanotechnology, where it is starting to be applied, in some cases, with spectacular results. VR interfaces to realtime simulations over high-speed networks hold a great deal of potential for molecular and nanotechnology scientists. The advantages of a VR interface are twofold. First, it allows the scientist to gain a deeper understanding of the micro-/nanoworld by allowing immersive visualization in three dimensions. Second, a scientist can use personal intuition to steer a simulation toward more favorable or realistic results.

However, some problems still remain to be solved. The time delay, or the lag between a user action in the real world and the corresponding update of the state of the object in the virtual world, is a major concern because it leads to unrealistic visualization, and thus to simulation sickness [97]. Modeling nanoforces in realistic-based models is necessary for reliable and guided nanomanipulation. Real-time VR nanodynamic simulations (generic continuum models for van der Waals, capillary, contact deformation, and electrostatic forces) are needed to be developed for complex tip/object geometry shapes. The lack of force-feedback input and output modality through haptic devices such as a force-feedback CyberGrasp [103] glove and so forth is another area for future research and development. These haptic devices are important because they can offer resistance to the movement of the molecule during simulation, and thus can reflect the force it can exert, and they offer resistance proportional to the local temperature in the system to reflect the dissipation of work through friction. In practical, telenanomanipulation systems and dedicated nano-haptic interfaces should be developed. Real-time capable force and kinesthetic-feedback interfaces with multi-degree-of-freedom nanoforce sensing axes are indispensable and should be an area of future research. The required " feeling" of the nanoworld calls for additional force display information from, for example, 3D acoustic interfaces, noncontact vision techniques, temperature, and so on. To keep pace with real-time interaction, VR technology must be supported by high-performance computers, associated software, and high-bandwidth network capabilities. VR also requires the development of new technologies such as displays that update in real time with head motion; advances in sensory feedback such as force, touch, texture, temperature, and smell; and intelligent models of environments.

These limitations must be overcome before a fully functional VR system is useful in practice. Additional development and refinement of the VR interface are needed to make a robust tool for the study of a real nanosystem.

Virtual Reality and Hap tics in Nano- and Bionano technology 31

⇐ Предыдущая 7 8 9 10 11 12 13 141516 Следующая ⇒