Seeing in Space: Shuttle Astronauts Get a Better View with ASVS

Jennifer Coombes

Space is probably the most extreme visual environment you can get. There are few visual cues to gauge depth, distance, or speed, and light levels range from blinding sunlight to pitch black. Under these conditions, how do astronauts see to perform such delicate operations as launching satellites into orbit or docking the space shuttle? Until recently, the answer was: with difficulty. But now, with the help of ASVS, the QNX-based Advanced Space Vision System from Neptec Design Group, astronauts no longer have to battle the rapidly changing light conditions and skimpy views offered by small shuttle windows. With ASVS, astronauts get a great view of what they're working on, from the ideal angle - without necessarily seeing anything directly at all.

A room with a view

Prior to ASVS, mission specialists responsible for moving things in and out of the shuttle's payload bay had a pretty tough time. Not only did they have to move things around remotely from inside the cabin using the shuttle's main robotic telemanipulator (a.k.a. Canadarm), but their only visual information was supplied by external TV cameras or the shuttle's small windows. For some operations, like releasing a satellite into space, cameras and windows were enough. But what about when the payload had to be reattached to something else? Not being able to see the mating surfaces with anything but a TV camera made this sort of operation extremely difficult if not impossible, especially if the cameras didn't happen to catch the object of interest at a useful angle.

Help has arrived, however, in the form of ASVS. The system quite literally turns the shuttle cabin into a room with a view, providing astronauts with exact location, orientation, and motion information through a computer-generated view of the payload and any important reference points. No matter what the position of the camera or payload, ASVS shows mission specialists what the object looks like from an ideal location. Used with the Canadarm, the system becomes a precision guidance system allowing astronauts to accurately position and orient a payload. Not just a system for guiding the Canadarm, however, the system can be used for all manner of shuttle operations - everything from piloting the shuttle to providing relative range, bearing, and elevation information that helps operators monitor clearances between a payload and the shuttle.

In the beginning

Currently part of the space program, ASVS has come a long way from its original use. The system, which is essentially machine vision based on realtime photogrammetry techniques, was originally developed at the National Research Council of Canada (NRCC) by Dr. Lloyd Pinkney and Charles Perratt in the 1970s to study car collisions. The SVS technology, as it was first called, is now licensed to Neptec Design Group, a Kanata, Ontario-based company of about 30 scientists, engineers, and software developers that provides ongoing support services to the Canadian Space Agency (CSA) and staffs the Space Vision Laboratory at NASA's Johnson Space Center in Houston. Says John Schneider, Neptec Software Manager, "We inherited ASVS technology from our predecessor, Leigh Instruments, the people who originally built the vision system for the NRCC way back when. When we first started with ASVS, it was a piece of lab software with potential. At the time, we had no idea it would turn into a piece of shuttle equipment."

No matter what the position of the camera or payload, ASVS shows mission specialists what the object looks like from an ideal location.

The technology came to the attention of NASA scientists in the '80s who thought it might satisfy the need for a guidance system for the Canadarm. Neptec has since expanded the operational capabilities and performance of the vision system, and has miniaturized it to the size of a portable computer. Today, a more robust and versatile system, ASVS is undergoing extensive testing and space qualification for continued use on the shuttle fleet and for eventual use in assembling the International Space Station.

ASVS in action

To work properly, ASVS requires certain key pieces of information that are obtained during what's known as a "ground survey." While the payload is still on the ground, target dots are placed in precise locations on the object and carefully measured. Other measurements taken at this point include the position and orientation of the cameras relative to the shuttle, the distance from the camera to some known fixed reference or coordinate system, and the focal length of the camera.

Once the shuttle is in orbit, ASVS uses one or more of the shuttle's external cameras as sensors to monitor the pattern of target dots placed on the payload during the ground survey. The targets now have to be "acquired" or locked on to by the system. Acquisition is performed by pointing at the individual dots on the shuttle display with a light pen, a keyboard, or with a cursor control device. ASVS draws a small box around each dot and tracks it, centering the box over the dot as it moves around on the screen. The system tracks the apparent position of the target dots on the image plane of the camera (the view you see on the monitor). As the object moves, the ASVS computer measures the changing position of the dots. An autoranging feature permits ASVS to switch from one set of targets to another when one moves out of its field of vision.

At this point, photogrammetric analysis is used to calculate the three-dimensional position and orientation of the object relative to the camera, based on the two-dimensional image. By measuring the precise positions of each dot on the image plane, and knowing the distance from the image plane to the projection center, and the geometry of the real target dots, ASVS calculates the distance from the camera to the payload.

The calculated position, orientation, and rate data is then synthesized into a realtime computer-generated display that shows the results of the photogrammetry as either x, y, z, roll, pitch, or yaw Cartesian information and provides the rate information for each value. The operator refers to this information as the payload is maneuvered.

The system can track three target arrays per camera in real time, where each array comprises a minimum of three and a maximum of 25 target dots. At least three dots are needed to produce six-degree-of-freedom information. Although the tracking process is very robust and can function even when shadows obscure part of the target array or light levels change drastically, adding more than three dots provides additional redundancy for improved accuracy and for those times when another object obscures the view.

ASVS can also control many of the camera's functions. It controls pan, tilt, focus, and zoom to best track the target array and keep as many dots in the field of view as possible. It can also control the camera's exposure to ensure the best image quality.

The computer-generated display produced by ASVS can be easily modified to suit any number of tasks or operator preferences for visual cues. For example, the graphical interface can be configured to represent a standard peg target, a standard rendezvous target, or an aircraft altitude indicator - whatever is best suited to the task at hand. ASVS can display operational data in a graphical display with or without alphanumerical data, or in strip-chart graphs.

Connecting the dots

Naturally, the dots themselves are of primary importance to the proper functioning of the system, so a great deal of care and effort is taken to ensure the dots are highly visible, that they're positioned close to the operating points of the object (i.e. in a mating operation, targets will be close to the interfaces), and that the correct number of dots are applied.

First and foremost the dots must be large enough for the camera to pick up and have high contrast with the background they're positioned on. The actual size of the dots is a function of the focal length of the camera, which is determined by the distance of the object from the camera, and by how far the astronauts need to watch it travel. On average, the dots measure about four inches across - about 10-15 pixels on the camera image.

Black dots on a white background are usually chosen for high contrast, but under harsh direct sunlight even black paint can appear bright in a camera image. So whenever possible, the target dots are made from thin films of silicon dioxide layered with inconel to form an inconel interference stack. A stack of this material has almost zero reflectivity in the visible spectrum. The result is a black that appears "blacker" than flat black paint.

Once it's determined how many dots should be placed on the object, it then becomes a matter of negotiation to choose locations for the dots that don't require any structural modifications, and that are acceptable to the payload's interested parties.

Fault-tolerant configuration

On the shuttle, two identical ASVS systems - 486-based IBM Thinkpads mounted on special expansion pans - are networked in a dual-redundant configuration. This dual redundancy is important in space where high-energy radiation can sometimes play havoc with integrated circuits. Although the radiation effects are usually transitory, if one system goes down, shuttle astronauts can still carry on with the secondary unit without having to reset the system.

Video signals from the exterior-mounted shuttle cameras are passed to the ASVS where the photogrammetric processes determine the position and orientation information of the payload. ASVS then feeds video signals directly to the shuttle's video system. Displays appear on the shuttle's own monitors.

Improving the original

When Neptec took over the vision system technology in 1990, the system was built around a homegrown OS designed by Leigh Instruments; the hardware consisted of two 8086 processors and 128K of RAM. Neptec took the core software - including photogrammetry, tracking, database, and control features - and put it into a single-processor PC configuration, adding peripheral interfaces and drivers.

They also ported the system to QNX 2. After investigating other operating systems, Neptec chose QNX because it was the only Intel-based OS that was established and stable - other OS vendors were only starting to come out with Intel-based systems. Also, QNX had everything the Neptec design team needed and wanted: realtime performance (ASVS takes a sample once every 30 milliseconds or once per video frame), messaging, interprocess communication capabilities, and native development. C-Scape was used to develop the user interface.

Cross-development It's a testament to QNX that when it came time to upgrade the system last spring, Neptec didn't even look at other operating systems. They chose QNX once again and have decided to upgrade the user interface by adopting Photon. The port to QNX 4.2, which began in April 1996, from a Windows 95 workstation using Phindows, was complete in September and is currently undergoing testing.

An intuitive point-and-click user interface is particularly important for this application. Says Iain Christie, Neptec's ASVS Specialist, "Astronauts are jacks-of-all-trades. They simply don't have time to learn every detail of all the various pieces of equipment they'll need to use on missions. What they want is an interface that's very much the standard Windows-style, user-friendly GUI. Photon will help us there." A lot of what Neptec's doing right now involves revamping the way a user interacts with the unit, and having the system do a lot more things automatically - like controlling the cameras. Currently, astronauts do this manually. Neptec is also separating the user interfaces so that development functions have a different interface than user functions. Right now, both aredone with one user interface, which is not ideal for either.

Other enhancements to the system are happening all the time. For instance, on some payloads it's impossible to install the round target dots, so Neptec is working on a new method to track irregular-shaped targets. With this enhancement, natural features of the object could be used as targets. For example, corners, ends of lines, bolt heads, and so on could be used to calculate the position information. Another planned enhancement will help the vision system deal better with the harsh lighting conditions of space.

Test flight

SVS, ASVS's predecessor, had its first test as a space-worthy tool onboard shuttle mission STS-52 in October 1992. Operated by Canadian astronaut Steve MacLean, it was used with the Canadarm and the Canadian Target Assembly, a six-foot-long, 150-pound satellite developed specially to test the operation of the system. The success of that first mission set the stage for ASVS's most dramatic experiment to date, which took place in November 1995 onboard STS-74. On STS-74 astronaut Chris Hadfield used ASVS to provide guidance information as he operated the Canadarm to perform a nerve-jangling operation; he had to unberth, re-orient, and install a Russian-built docking module on the Mir space station. The module, left attached to Mir after this mission, now serves as a permanent connecting link for visiting space shuttles.

The future of ASVS: the International Space Station

STS-74 was just one in a series of shuttle missions that will be taken by NASA and the Canadian Space Agency to test the effectiveness of vision system technology for assembling large structures in space. One such flight, STS-80, which was launched in November 1996, carried the Orpheus SPAS, a sort of flying telescope, and the Wake Shield Facility, a satellite especially designed to be deployed and retrieved by the Canadarm.

ASVS will begin helping in the assembly of the International Space Station starting in early 1997.

The Wake Shield Facility was released and flew free of the shuttle for about 48 hours, creating a wake behind it and an extremely high vacuum that was used to test new techniques for growing pure semiconductor crystals. This technique may eventually benefit the semiconductor manufacturing industry.

The greatest success of ASVS, however, is still to come. Fueled by ASVS's performance on recent missions, NASA has made it one of the baseline components for on-orbit assembly of the International Space Station (ISS). ASVS will begin helping in the assembly of ISS starting in early 1997 with the launch of the core module.

Using ASVS: A realworld scenario

Just how hard is it for astronauts to see what they're doing in space? Well, let's put a twist on a familiar example and see.

Consider the routine task of inserting your key into the lock of your front door. This task is straightforward for most of us, not only because we do it so frequently and therefore get lots of practice, but because we have several other factors in our favor:

1. The key is a relatively small object that's easy to pick up and move around.
2. While inserting the key into the lock you usually stand facing the door, directly in front of the lock.
3. Your hand is behind the key, which is an extension of your thumb and index finger.
4. Usually you have a sightline that is parallel to, and on the same vertical axis as, the keyhole.
5. Once you get the key started into the barrel of the lock, it usually goes the rest of the way without much effort.

Now, let's change some of the parameters and see what happens.

1. Your key is a meter long and weighs 50 kilograms.
2. You are standing to one side of your front door and can't hold the key directly.
3. You must pick up the key and maneuver it around with an extension arm that's also a meter long.
4. You cannot see the keyhole in your line of sight - you view a reflection of it in a mirror positioned half a meter in front of, and off to one side of, the lock.
5. Oh yes, and don't forget to close one eye!

This front door predicament could be solved if you could find out the exact position and attitude of the now gigantic key at any point in time, and if you could get a familiar view of your key and keyhole.

Thanks to the Advanced Space Vision System this can easily be solved. We would place several target dots on your key - normally we use a minimum of three, but because you, your arm, or some other obstruction might get in the way of one of them, we would probably opt for half a dozen or so. The distances and angles between the dots would be carefully measured, as would the geometric relationship between the dots and the business end of your key. We'd then position a couple of TV cameras outside your front door, one on either side of the lock, probably at different heights. Although these cameras are in fixed positions, we can vary the viewing angle and zoom setting as needed.

We'd then connect the TV cameras to ASVS and power everything up. After closing the "Front Door" mission configuration, we'd acquire, or lock on, to a set grouping of target dots placed on your key. ASVS would first measure the dot positions in the camera image plane, and from this, determine the position and attitude of the key with respect to the camera.

Finally, ASVS would create a display showing the key's location with respect to the lock from a vantage point that is on axis with the lock and about a meter away. From here on, unlocking your front door would be simple. You'd keep one eye on the display and move the key slowly in the direction of the keyhole according to the computer-generated picture of the key and lock.

A few moments later, you'd be through the door. Not so bad, was it?

- Neptec Design Team