ECA and QNX Deliver State-of-the-Art Bar-Code Sorters to the US Postal Service

Rus Duderstadt & Christina Dahling, ElectroCom Automation L.P.

Every business day, the United States Postal Service (USPS) processes millions of pieces of mail at over 250 major facilities nationwide. Given ever-increasing volumes of mail, the USPS workforce of over 600,000 depends heavily on state-of-the-art machinery to help process the mail quickly and efficiently.

One of these state-of-the-art machines is the Delivery Bar Code Sorter (DBCS), which can sort 35,000 to 40,000 letters per hour. QNX is a vital part of the DBCS, serving as the OS that controls and monitors the machine's functions.

The DBCS is designed and manufactured by ElectroCom Automation L.P. (ECA) in Arlington, Texas in partnership with AEG ElectroCom GmbH in Konstanz, Germany. Since 1990, ECA has been awarded two major USPS contracts for manufacturing DBCS machines, with a combined value of US $460 million. To date, ECA has deployed over 1700 DBCS machines to the USPS.

ECA also supplies DBCS machines to commercial presort bureaus, which presort the outgoing mail of companies such as banks and department stores. In turn, these customers receive a discount in mailing costs from the USPS.

The DBCS sorts mail pieces bearing the POSTNET bar code, a series of short and tall vertical bars preprinted on the mail piece or added by other mail-processing equipment. The bar code corresponds to the USPS zip code, which can consist of 5, 9, or 11 digits. A five-digit or nine-digit zip code identifies the general destination of the mail piece, while an 11-digit zip code identifies the specific destination address. Using 11-digit zip codes, the DBCS can sort mail pieces into the order in which postal carriers walk their routes. This saves carriers from spending hours each day hand sorting the mail they deliver.

But there's more to the DBCS than just sorting mail. The machine exchanges data with a central file server over a TCP/IP network and also contains a broad range of diagnostic tools. All this functionality requires a full-featured, multitasking operating system, together with reliable TCP/IP support and a comprehensive graphical user interface. QNX supports all this, and more.

System demands

The DBCS must process mail pieces within a wide range of physical characteristics--minimum and maximum limits are set for mail-piece length, height, thickness, and weight. The USPS also sets strict performance requirements for sorting accuracy and mechanical handling of the mail.

The system must be capable of interfacing with several external serial devices while still providing an immediate response to user inputs and machine events. The system must simultaneously print reports, print tray-routing labels, interpret diagnostics data received from the machine, and update a graphical display every second, all while accurately sorting letters. The system must also support a modem link to a remote PC so that off-site technicians can monitor the machine's performance and provide troubleshooting support.

Modular design

The DBCS has two major components--a system PC (running QNX) and the actual machine. The system PC controls the machine and provides the user interface, destination pocket lookup, report generation, diagnostics support, and many other software functions.

The DBCS machine consists of several major modules, each performing a basic mail-processing function:

The feeder, reader, and stacker modules are often referred to as "firmware modules" because each one is controlled by an embedded processor (not running QNX). These processors scan photocells and control diverter gates that direct mail pieces to their destination pockets.

Feeding and reading

Bar-coded letters are fed into the feeder module by an operator and moved through the reader and stacker modules by a series of roller-driven belts. The reader module includes the POSTNET bar-code reader as well as the electrical and mechanical hardware necessary to track and divert letters to the level of the destination pocket.

The DBCS supports a number of sort pockets, allowing different configurations to be deployed based on a site's requirements. The current DBCS model ranges from 62 to 302 pockets and has four levels.

The reader module scans the bar-coded information on the mail piece and sends the zip code to the PC. The PC determines the destination pocket for that zip code and sends it back to the reader module. The reader module then diverts the mail piece to the proper level and sends an associated tracking message to the first stacker module. The tracking message follows the mail piece through the rest of the machine.

Each stacker module contains four pockets per level. Mail pieces are transported through the stacker modules and diverted into their destination pockets. When a pocket becomes full, the operator removes the letters and puts them in a tray that's set aside for that pocket.

Network of processors

The DBCS may be viewed as a network of processors with several different communication interfaces. The reader module serves as master of an RS-485 network with the stacker modules as slaves. A unidirectional parallel interface, the descriptor bus, carries the tracking messages associated with mail pieces as they are transported from module to module. Other types of messages sent to and received from the stacker modules are carried over the RS-485 network.

All message traffic between the PC and the reader or stacker modules passes through a proprietary, high-speed parallel computer interface. The reader module forwards PC-to-stacker and stacker-to-PC messages via the parallel computer interface and the RS-485 network. The PC also communicates with the feeder module via an RS-232 connection.

Adapter boards

The PC uses several adapter boards to interface with different modules and the outside world. A custom board provides the parallel computer interface with the reader module. Zip code and pocket assignment data are exchanged using this interface. A second custom adapter board provides a high-speed (10 Mbit/sec) serial link to the bar-code reader. This board is not active during mail processing. It's used for downloading control code to the bar-code reader at startup and for performing off-line bar-code reader diagnostics.

The current DBCS model uses Connect Tech's Dflex-8 serial board to communicate with several serial devices, including the feeder module, two thermal label printers, and a separate data collection computer that stores zip code information for all mail pieces. An internal modem is used for remote diagnostics and Kermit file transfers.

The PC includes an SMC WD8003 series Ethernet interface board for use in TCP/IP-based file transfers to and from a central file server. USPS personnel create sort-plan files on a VAX computer and download them to the DBCS PCs at their site. A sort plan contains the data necessary to determine a destination pocket, given a mail-piece zip code. Many USPS facilities maintain over a hundred site-specific sort plans, which may change daily.

Two Microcom M-410EC thermal label printers are controlled by the PC via serial ports. The operator may command batch printing of labels from PC-hosted menus or individually request labels using a push button located near each sort pocket. Once printed, each label is attached to a tray of sorted mail. Automated tray handling systems and personnel with hand scanners use the labels to route trays through the mail-processing facility.

Mail processing with QNX processes

The current DBCS model is controlled by a 486/25SX PC equipped with 16M RAM. All QNX-based software--the OS, TCP/IP, QNX Windows, and our application--consumes only 13M of disk space. Try squeezing another full-featured OS into that much space!

The design of the DBCS software was heavily influenced by one of QNX's hallmark features--very fast interprocess communication. With another OS, we would have had to load more functionality into each process and rethink our communication methods. In a large system like ours, distributing the work among a number of smaller processes is one way to help manage complexity. The software is functionally partitioned into groups of processes, or domains, described below.

mc (machine control)

These processes provide the minimum software functionality necessary to sort mail to a destination pocket. Higher-level processes perform mail-piece pocket lookup and manage the overall state of the DBCS machine. Lower-level processes are responsible for communicating with the firmware modules. One example of a lower-level machine-control process is the driver for the parallel computer interface.

mp (mail processing)

Processes in this domain gather sort and run statistics and generate reports. These reports are vital because management uses them to monitor machine performance. Daily statistics are maintained on the system for 45 days, and a variety of reports are generated from this data. Typical reports include statistics for mail-processing throughput, bar-code read rate, and machine behavior. At the completion of each mail-processing run, data is appended to a file containing prior-run statistics. This way, only current-run data will be lost if the machine reboots or if power goes out.

dg (diagnostics)

Diagnostics processes receive and interpret status and statistical messages from firmware modules. For each module type, there are more than twenty different messages. Status messages contain low-level information about components such as boards, lamps, push buttons, and photocells. In addition to sending current-state information to the system PC, firmware modules can test most components and send the results in a status message.

Status and statistical messages may be sent spontaneously by the firmware (e.g. when a mail piece jam occurs) or when requested by the PC. The diagnostics processes examine the data to determine if a fault has occurred. If there is a fault, then an error message is logged to a database and displayed on the screen.

Most statistical messages contain data accumulated by firmware modules during a mail-processing run. This data is collected and stored on disk for later use by trend analysis diagnostics software. Maintenance personnel may view machine behavior on a module level over periods of up to one year. We use the Tilcon Real-Time Developer package to present this trend analysis data in the form of bar and line graphs.

ui (user interface)

This set of processes manages the graphical user interface. To simplify development of the user interface, we used the Klondike QWCTool Toolkit. The windows designed for the DBCS are composed of simple objects; screens are uncomplicated and intuitive. Detailed online help is available within each screen, as is access to machine diagnostics. A realtime display, updated every second during mail processing, provides feedback to operators.

lp (label printer)

Label printer processes control the printing of tray-routing labels. These bar-coded labels include the zip-code range, destination-pocket number, mail-delivery day, and other information obtained from the mail-processing run data. Label data is sent to one of two label printers via 9600 baud RS-232 serial links. Printer status is monitored, and events such as "out-of-paper" are displayed on the user interface.

rp (remote data collection computer)

These processes interface with a host data-collection computer through a 9600 baud serial link. Machine state and zip code data are sent to the collection computer while the DBCS is processing mail. The collection computer accumulates this information for all DBCS machines in a mail facility.

fs (file services)

File services processes perform Ethernet and floppy disk control, including user requests to copy files to and from the VAX file server or diskettes. TCP/IP for QNX provides the file transfers over Ethernet. Both DOS and QNX file formats are supported in the diskette utilities.

rd (remote diagnostics)

The remote diagnostics processes allow USPS technicians access to a DBCS PC via modem. The remote PC contains a user interface, similar to the DBCS GUI, but with different functions. Through menus on the remote PC, technicians may dial up the DBCS PC and view the machine's reports, diagnostics, or mail-processing status. The QNX modem utility answers the call and passes control to a remote diagnostics process running on the DBCS side. Other remote diagnostics processes resident on both PCs interface with the modem and other application software to service remote requests. Remote logins are supported by temporarily passing control to qtalk for the duration of the login session.

de (debug)

Debug processes read and print the contents of various shared-memory segments. Access to these segments is controlled using semaphores. Recent mail-piece events, mail-processing statistics, diagnostics data, and queue information can be easily examined and verified during testing of the application software.

st (startup and shutdown)

Processes in this domain start the DBCS software, handle death notices when other processes die, and perform a software shutdown.

Message passing

Most message passing in the DBCS software system uses a modified version of QNX's queue administrator, available on QUICS. This intermediary process provides indirect, non-blocking communication by maintaining message queues for each application process. We chose this approach to ease the integration of high-priority machine-control processes with lower-priority diagnostics and data-collection processes. High-priority processes always send without waiting and will drop a message if the destination queue is full. A software error is logged when this occurs. Lower-priority processes will wait until the destination queue has an empty message slot. Code was added to the queue administrator to record and display the high-water mark in each queue. This helps programmers tune queue sizes to prevent bottlenecks and dropped messages. Besides message passing, queues are used to implement semaphores for various shared-memory segments.

Information about all DBCS application processes is contained in a configuration file. Each line of data in the file includes the process name and priority, control flags, input queue parameters, and command line arguments.

A startup process reads this file and places the information in a shared-memory process table. The startup process then creates all queues and most other processes in the system. Synchronization problems are reduced by creating all message queues before any application processes are started.

The startup process is indirectly informed whenever an application process dies. Depending on control flags in shared memory, the startup process may log an error message containing the dead process' name, exit point or exception address, and pending signals. A stub process may also be created to take over the dead process' queue. This usually keeps other application processes from generating a lot of side-effect error messages that obscure the initial problem.

The queue functions provided by QNX are encapsulated within an application-specific set of mailbox functions that access the shared-memory process table. A call to open a mailbox involves scanning the table for the process name of interest and using the associated queue name in a call to QNX's queue_open() function.

Processes know their own name and the names of other processes they communicate with, but all queue mechanisms are hidden by the mailbox functions. Combining these messaging functions with the shared-memory process table allows us to implement some important features. For example, by setting a flag in the startup configuration, we can create a stub process (instead of a real application process) to read and discard messages from the associated input queue. This is very useful during the early and middle stages of development when processes are in various states of completion. We were able to demonstrate mail sorting very early in the move to QNX 4 by first porting the machine-control processes and stubbing out all other processes. This success encouraged us to port other processes one at a time.

Development and maintenance

The DBCS software was developed by a team of software engineers on a QNX network consisting of a boot server, a hot backup node, developer workstations, and integration floor workstations.

The DBCS runtime application software consists of over thirty resident processes. These executables are produced from more than five hundred .c and .h files. Source code, shell scripts, system files, and various configuration files are maintained using the Revision Control System (RCS) shipped with QNX. Besides RCS, we established a file-naming convention for all development and deliverable files to help manage our code.

We keep old versions of system binaries, libraries, and INCLUDE files in separate directories for rebuilds during maintenance. QNX's prefix utility is used to point to the old directories. This lets us compile executables with the same checksums as those in a release based on previous versions of the OS, compiler, TCP/IP, or QNX Windows.

QNX delivers

Currently, ECA is developing a multimedia expert system to run under QNX Rundos and Microsoft Windows. This will provide DBCS operators and supervisors with online training as well as a knowledge-based system tied into the machine diagnostics software.

QNX Software Systems gave us great support throughout the software development cycle. Knowledgeable engineers answered questions promptly and accurately, while QUICS proved to be an invaluable information resource and an easy way to obtain software updates.

The Delivery Bar Code Sorter was the first major USPS contract to use QNX. The USPS response to the machine's performance has been extremely positive. Based on this success, ECA will continue to look to QNX to help us deliver automation projects.