QNX and McObject Put a Cap on Oil Rig Blowouts

by Eric L. Milne, Hydril Pressure Control

As industrial accidents go, a blowout in offshore natural gas or oil exploration surely ranks among the worst. A drill pipe or “drill string” extends from the drilling rig down thousands of feet to the wellhead and, directly beneath it, into the wellbore on the sea floor. This drill string is contained within a riser, or solid casing, to create an enclosed space. Inside the riser is drilling mud, a fluid mixture whose sheer weight is intended to contain any upsurge from the highly pressurized formations that are the exploration’s target.

Oil rig blowouts can threaten lives, pollute the environment, and cost drilling operators millions of dollars per day in lost revenue. Using the QNX Neutrino RTOS and McObject eXtremeDB database, Hydril blowout preventers help keep oil rigs safe and in business.

But sometimes the “kick” from the newly released hydrocarbons can literally push the drilling mud up the drill string and riser. If not blocked by heavier mud or stopped by a pressure control system, the oil or gas can rush up the pipe and emerge like a geyser on top of the rig. This is a blowout, and it can release toxic gasses and even ignite. Fatalities can result, along with severe environmental damage.

Thankfully, blowouts are rare today. This stems in large part from the emergence of sophisticated and robust pressure control devices called blowout preventers (BOPs), controlled by realtime computing networks. Hydril Pressure Control invented the first hydraulically operated BOP as well as the annular BOP (a widely-used BOP featuring an opening, lined with high-strength rubber, that narrows or widens to control oil or gas flow) in the 1930s.

Multiple levels of protection

Today, Hydril is a leader in the design and manufacture of BOPs and drilling control systems used worldwide, at drilling sites with the most extreme environments in terms of depth and wellbore pressure and temperature. A major development goal is to provide multiple levels of backup protection. Our newest technology achieves this by integrating three separate software disciplines — real-time operating systems (RTOSs), embedded high-availability database systems, and enterprise-style SQL server database management — into a single control system. The enterprise database, Microsoft’s SQL Server, lives on the rig and is used for archiving, reporting and troubleshooting functions. The embedded database, McObject’s eXtremeDB High Availability, runs on the QNX® Neutrino® RTOS within the individual controllers, both on the rig and subsea, and provides duplicate “working copies” of data to support real-time processes.

“McObject’s eXtremeDB High Availability runs on the QNX Neutrino RTOS within the individual controllers, both on the rig and subsea.”

At Hydril, we strive to produce the safest and most reliable BOPs and drilling control systems in our industry. Exploratory drilling is conducted by specialist contractors to determine whether or not there is a resource worth extracting. Compared to the later production phase, much less is known during exploration about what’s underground, and surprises can be greatest — requiring failsafe solutions should the oil or gas within the wellbore deliver an unanticipated kick.

A pressure control system’s key hardware is the BOP, which is essentially a large, heavy valve to contain pressure. A system will use multiple, specialized BOPs, configured in a vertical “Stack” that sits on top of the well head. Above the Stack are the redundant BOP control “Pods,” each consisting of a lower unit containing hydraulics to control the BOPs, and an upper electronics housing (EH). The EH is contained in a three-inch thick steel domed container to protect the sensitive electronics gear from the surrounding water pressure. The Pod also controls solenoids that operate the BOPs’ hydraulic valves. During operation, each redundant Pod continuously gathers data from remote sensors, including:

• Temperature and pressure in the wellbore. Obviously it is very cold at depths of 10,000 feet, but the energy used in drilling can generate temperatures as high as 180 – 350 degrees Fahrenheit (82 – 176 C) or higher. High-Pressure/High-Temperature sensors in the wellbore monitor pressure spikes to help predict and control kicks.
• Positioning. The Stack should remain vertical, with the riser extending straight to the surface, but environmental conditions and positioning of the drilling vessel in relation to the Stack can make this a challenge. A KVH fiber optic gyro within the Pod’s electronics housing monitors Stack rotation, while inclinometers measure tilt. If the riser is in an unacceptable position with regards to either the Stack or the surface vessel, a dynamic positioning system on the rig corrects the problem.
• Fluid Levels. Electronics and saltwater do not mix well, thus the need for the 3-inch-thick, one-atmosphere electronics housing. If water monitors and liquid level detectors within the Pod indicate water ingress, the operators can power down the electronics from the surface prior to damaging any equipment. The standby Pod then takes over primary control of the Stack.
• Solenoid current. Every Stack function is controlled via solenoid actuation within the Pods. For example, the Pod sends a signal to fire any one of a number of solenoids to open or close BOP valves, and current through every solenoid must be monitored to verify proper operation.
• Pressure. BOP hydraulic pressure must exceed the hydrostatic pressure exerted by the surrounding water to ensure proper operation, and even higher pressure is required to ensure shearing of the pipe if and when severe measures are needed. Each Pod monitors up to 20 pressure transducers to maintain proper operating pressures.
• Electrical states. Power for the Pods is generated on the surface, cleaned of spikes and noise, converted to either 720V (or 480V) and transmitted subsea to the Pod, where transformers reduce it to 120V and 60V (or 48V). This is further converted to the necessary voltages to operate the electronics (±12, +5, +24, etc.) Power levels are constantly monitored.
• Pod temperature. Electronics are also sensitive to heat, and will begin to degrade in performance above the rated temperature. Each Pod monitors air temperature within the electronics housing as well as processor core temperature. Temperatures in the electronics housing are typically not a problem subsea due the presence of the world’s largest heat sink (the ocean). However when the Stack and Pods are on the surface, temperatures within the dome can exceed 60 degrees Celsius.

Realtime data collection

All of this collected data has to be in the right place, at the right time. For our software upgrade, which emphasized data management efficiency, we found it useful to think of the system in terms of three data flows:

First, data collected by sensors is moved to local data storage on the controllers. The primary controllers, within the Pods, are single-board and Intel x86-based, with serial, digital, and analog I/O boards for communication. Gear on the surface includes industrial-rated, passive-backplane single-board controllers, some of which include PCI- and ISA-based serial, digital, and analog I/O boards. The surface-based display station controllers typically use touchscreen monitors with the QNX Photon® microGUI® graphical user interface.

Second, data must travel from the controllers to the archival Microsoft SQL Server database. Access to SQL Server is facilitated using Easysoft’s Open Database Connectivity (ODBC) bridge technology for the QNX Neutrino RTOS.

The third flow is directed from the operator working at the display station controller, to the controllers that are responsible for operating parts of the system. For example, when the operator pushes a button, that action is recorded in the display station's local data storage, and it also triggers a remote procedure call, causing the appropriate controller to do something such as opening a valve. Local data storage on the controllers is provided by McObject eXtremeDB-HA. Using a third-party’s database means that the database logic and application logic are inherently separate, which enforces software modularity. This promises to ease future upgrades and maintenance to the system. It will be easier to add new features, which customers are always requesting.

Automatic failover for ultimate reliability

High Availability is an important feature we gain. The database itself is an in-memory DBMS: it stores data records in memory at all times, for realtime access (disk and file I/O are eliminated). However, databases are replicated on every controller, and eXtremeDB’s high availability subsystem automatically propagates changes from the primary database to the secondary, backup database. Any database update occurs within the scope of a database transaction, so that physically separate data stores are guaranteed to remain in synch.

The HA mechanism also replicates selected data between controllers. In addition, the entire control system is designed for redundancy via a secondary network of controllers, with the database managing the updates between networks. This replication of vital system data, at multiple levels and with automatic failover, provides the highest degree of reliability.

“QNX Transparent Distributed Processing (TDP) simplifies the coordination of system nodes by enabling devices to share information in a peer-to-peer fashion.”

The database also provides the core messaging component in our system by using remote access interfaces, which enable controllers and software components to read from and write to databases at every network node. The database high availability mechanism is integral to this update process, which ensures that whatever happens on one controller gets reflected on all other parts of the system. Underneath the messaging layer, the Transparent Distributed Processing (TDP) provided by the QNX Neutrino RTOS simplifies the coordination of system nodes by enabling all network devices to recognize one another and share information in a peer-to-peer fashion, regardless of their location.

Minimizing downtime

This redundancy, and the need to update system elements with mission critical information, makes for software complexity. But a well-functioning pressure control system keeps the drilling process on track and cuts downtime — an important priority, since drilling operators can lose millions of dollars per day for a rig that is not drilling. Therefore high reliability and availability along with safety are Hydril’s key design considerations; our new system’s seamless integration of RTOS technology, multiple replicated HA embedded databases, and an archival SQL Server database, advances these goals to a point that can justifiably be termed state-of-the-art.

Eric Milne is Chief Engineer of Electrical and Software Engineering at Hydril.