How Sweet it Is!
Cogent and QNX Boost Productivity for Cadbury Chocolate

Paul Benford, Products and Services Director, Cogent Real-Time Systems
Frank Cipparrone, Engineering Manager, Cadbury Chocolate Canada

Canadians are a dedicated bunch when it comes to chocolate. They devoured over 48 million pounds of Caramilk, Crunchie, and other Cadbury chocolate bars last year alone. While this demand for chocolate was great news for Canada's largest chocolate manufacturer, it also caused them some anxiety. Simply put, Cadbury's control system, which used MS Windows for a control platform, InTouch for a user interface, and a Windows-based PLC driver to talk to a Siemens PLC, was hindering instead of helping productivity.

Windows woes

The chocolate-making process is no piece of cake. It is, in fact, a complex procedure. At Cadbury Chocolate, large-volume ingredients are fed via inlet pipes to two industrial mixers running in parallel while small-volume ingredients are added by hand. These mixers produce a chocolate paste, which is transported by a shared conveyor to four large heated rollers, called refiners, running in parallel. The refiners churn out powdered chocolate flake, which is, in turn, conveyed to large heated conditioning tanks, called conches. The conches blend the chocolate by heating and stirring it for several hours. From there, liquid chocolate is pumped through heated pipes to a storage farm of 65 tanks, each tank holding up to 26,000 pounds of chocolate. The chocolate is then pumped on demand to small tanks near production lines where it is molded into chocolate bars, wrapped, and packaged, and then moved to a warehouse.

The process was made even more complex because the Windows-based system, used to monitor and control this entire process, was unstable, completely ineffective at controlling the process, and poorly suited to specific process requirements like pumping and mixing control. For instance, MS Windows would often slow to a crawl and the link to the PLC would time-out, causing the system to reset and valuable data to be lost. The operator would then have to run the system manually for the remainder of the cycle, resulting in large product variances, slower production, and wasted product. Further, the lack of a standard approach made operator training complex and arduous.

Cadbury spent a year trying to get this Windows-based system to work, during which time they were continually reassured that a smoothly running system was just two weeks away. Eventually they turned to us for a better solution.

Cogent takes control

The chocolate-making process is divided into two distinct but related systems: one for mixing and the other for distribution. Cadbury's goals for the former included increasing total output while reducing product variability. Their goals for the latter included minimizing operator error to avoid accidents such as blending different recipes or overfilling tanks. They also wanted operators to have easier access to process data and they hoped to make it easier for new operators to learn the chocolate-making process without intense training.

To preserve their original investment, Cadbury requested we keep the InTouch interface in MS Windows. We decided the best way to isolate the Windows platform and, at the same time, deliver increased functionality was to separate this interface from the control system. QNX, with its ability to connect independent process areas with a high-speed transparent network, allowed us to do just this.

Cadbury spent a year trying to get their Windows-based system to work. Eventually they turned to us for a better solution.

First taste of success

Our QNX-based control system runs on a 486. The control program, written using our own SCADALisp programming language, communicates with the PLC through a serial line device driver implemented in QNX. We took advantage of the priority scheduling of QNX to run this driver at a higher priority than other system tasks. Consequently, less important processes can no longer disconnect the driver.

The original InTouch interface is on another 486 running Windows 3.11. The QNX and MS Windows computers are connected to a fiber-optic network and communicate using our own Cascade Connect software. Cascade Connect provides a bidirectional realtime data connection between the two machines and safely handles any break in connection and subsequent reconnection. It uses TCP/IP as the network protocol as well as native messaging in both QNX (Send/Receive/ Reply) and MS Windows (DDE).

With QNX providing the stable, high-speed control platform the system so badly needed, we transferred all aspects of the system, apart from the user interface, to it. (The MS Windows machine is now stripped down so that it only runs InTouch and the Windows side of Cascade Connect.) As a result, MS Windows has more system resources and is more stable. Things have certainly come a long way! Operators need only reboot MS Windows once every few days and, when they do, QNX continues to do its job of keeping the entire system under control.

Savoring the results

Since two mixers feed a single conveyer, the greatest improvement in production at the mixing stage was achieved by running both mixers in a closely timed control strategy. One mixer now mixes and discharges chocolate paste while the other receives ingredients. Because QNX runs virtually unassisted, operator error has been significantly reduced while the number of batches handled by the mixing system has increased. Moreover, the QNX network supports communication between this system, an analytical lab computer, and the distribution control program, giving managers and operators easy access to a range of process and historical data.

Because the distribution system includes 65 tanks, 63 pumps, 109 monitored valves, and 300 unmonitored valves, there are over 10,000 different routes for pumping chocolate from source to destination tanks. We eliminated operator guesswork with a sophisticated SCADALisp program in QNX that suggests optimal routes using a model of interconnected tanks, pipes, and valves. The SCADALisp control program also monitors tank limits and automatically stops pumps when high levels are reached. This has helped prevent accidents where operators mistakenly select routes that would blend recipes or overflow tanks. SCADALisp was also used to provide a range of graphical screens that depict the piping network and inventory in the storage farm.

Sweet ending

A reliable, protected, and standardized control system has helped Cadbury significantly improve productivity and quality in all areas of their chocolate-making process. Results include:

• a 50% increase in production simply by using existing equipment more efficiently
• a reduction in product variance to less than 1%
• virtually no operator errors or system downtime
• increased access to plant data for operators and managers

What's more, the mixing stage is no longer the bottleneck in the process. Cadbury, in fact, has had to install a new refiner to keep up with added production from running the two mixers in parallel!

Cadbury Chocolate made close to $66 million Cdn in profit last year from their chocolate-making system. No wonder our QNX-based control system is quite a success story.

SCADALisp has been refined and is now at the heart of Cogent"s new Slang programming language for the Photon microGUI.