| Updated: April 19, 2023 |
This appendix describes how to add 2-Step Precision Time Protocol (PTP) Ethernet packet timestamping functionality into an io-sock driver.
PTP is documented in IEEE 1588-2002 and IEEE 1588-2008. The 2-Step solution involves simpler hardware than the 1-step solution and it is used on many more hardware platforms. 2-Step is also currently the only option that supports an Ethernet speed of 10 Gigabit or higher, which provides limited time to encode messages.
Because this discussion is not meant to be locked to a particular PTP profile or a particular hardware platform, the information about initialization and timestamp collection from the transmit and receive PTP packets does not provide any details about hardware register handling.
For a general discussion of writing io-sock drivers, see “Writing Network Drivers for io-sock.”
For easier reading, the calling and PTP functions are presented and discussed in the order in which they are used. Functions that are used to initialize the hardware are discussed first, followed by the ones that process timestamps.
This driver design assumes that the hardware uses a free-running internal oscillator. No adjustments are made in the hardware to try to control the internal oscillator's frequency. Adjusting the frequency of an oscillator (e.g., using a Phase-Locked Loop (PLL)) is slow and prone to gain peaking effects. Instead of hardware frequency adjustments, the driver uses the addend value it receives from the PTP daemon in the sample_ptp_set_compensation() call to change how much time is added to the PTP hardware clock after each tick of the internal oscillator. This solution allows the internal oscillator to free run and its frequency varies over time due to temperature and other variables. However, the PTP daemon adjusts to the changes and keeps the PTP client (slave) synchronized to the PTP server (master) by adjusting the addend passed to the driver. The PTP daemon sends the addend clock adjustments to the driver with an ioctl() call and the PTP_SET_COMPENSATION parameter.
Adding PTP support requires ptp.h and additional header files, some of which are hardware dependent. For example:
#include <netinet/udp.h> #include <netinet/in.h> #include <netinet/ip.h> #include <qnx/ptp.h>
static int sample_ptp_set_time(struct sample_softc *sc, ptp_time_t *time);
static void sample_ptp_get_time(struct sample_softc *sc, ptp_time_t *time);
static int sample_set_addend(struct sample_softc *sc, uint32_t addend);
static int sample_ptp_init_hw(struct sample_softc *sc);
static void sample_ptp_init_sw(struct sample_softc *sc);
static int sample_ptp_start(struct sample_softc *sc);
static int sample_ptp_get_timestamp(struct sample_softc *sc, ptp_extts_t *ts, uint8_t tx);
void sample_ptp_add_rx_timestamp(struct sample_softc *sc, const ptpv2hdr_t *ptp_hdr, sample_bd_t *bd);
static int sample_ptp_set_compensation(struct sample_softc *sc, const ptp_comp_t *ptc);
int sample_ptp_ioctl(struct sample_softc *sc, const struct ifdrv *ifd);
int sample_ptp_add_tx_timestamp(struct sample_softc *sc, const ptpv2hdr_t *ptp_hdr, sample_bd_t *bd);
#define PTP_TIMESTAMP_BUF_SZ 32
struct sample_softc {
/* Add the specific data structures required for the PTP driver. */
pthread_mutex_t hw_rw_mutex;
uint32_t addend;
pthread_mutex_t ptp_rx_mutex;
pthread_mutex_t ptp_tx_mutex;
ptp_extts_t rx_ts[PTP_TIMESTAMP_BUF_SZ];
ptp_extts_t tx_ts[PTP_TIMESTAMP_BUF_SZ];
uint32_t rx_ts_cnt;
uint32_t tx_ts_cnt;
uint32_t ptp_clk_rate;
int ptp_disable;
};
The device_t structure is the pointer type for the device structure. All API functions have the device pointer as a first parameter.
When an Ethernet device is found during initialization, the compatible sample_attach() function is called.
From the PTP perspective, the attach function is executed for each Ethernet device detected. The attach function should contain everything needed to initialize the hardware, allocate resources, initialize the MAC and PHY, attach to the interrupts required to transmit and receive packets, and so on.
An attach function attaches a driver to a device. The following example is a simplified example of an Ethernet driver connecting to an Ethernet device and adding in the PTP functionality. The PTP driver functionality is enabled in sample_attach(), and sample_ptp_start() is usually called after the Ethernet interface has been created. All the Ethernet initialization is done inside sample_attach() and not before it is called.
static int sample_attach(device_t dev) {
sc = device_get_softc(dev);
/* Initialize the hardware access mutex (io-sock is multi-threaded). */
if (pthread_mutex_init(&sc->hw_rw_mutex, NULL)) {
device_printf(dev, "Could not initialize the hardware access mutex\n");
return ENXIO;
}
/* The Ethernet interface is created, and fully configured. */
sc->is_attached = true;
/* Initialize the PTP software and hardware. */
error = sample_ptp_start(sc);
if (error != EOK) {
device_printf(dev, "PTP initialization failed\n");
}
}
static int sample_ptp_start(struct sample_softc *sc)
{
int status = EOK;
/* A mutex is required to protect hardware accesses to timestamp data. */
status = pthread_mutex_init(&sc->ptp_rx_mutex, NULL);
if (status != EOK){
goto done;
}
status = pthread_mutex_init(&sc->ptp_tx_mutex, NULL);
if (status != EOK){
goto done;
}
status = sample_ptp_init_hw(sc);
if (status != EOK){
goto done;
}
sample_ptp_init_sw(sc);
done:
if(status != EOK) {
device_printf(sc->dev, "devs-sample %s ERROR %d",__func__ status);
}
return status;
}
static int sample_ptp_init_hw(struct sample_softc *sc)
{
pthread_mutex_lock(&sc->hw_rw_mutex);
/* Initialize the clocking hardware. */
/* The addend parameter is the time adjustment used (the increment added to clock on every
tick) in the PTP clock handling. The initial value should be calculated based on the PTP
clocking hardware and the clock initialization above.
Note: The sample_set_addend is also called frequently on a PTP client (slave) with
adjustment requests from the PTP daemon.
The adjustment call is made through the IOCTL interface.
This is discussed below with sample_ptp_set_compensation().
*/
pthread_mutex_unlock(&sc->hw_rw_mutex);
/* the mutex is set/cleared in sample_set_addend */
sc->addend = STARTUP_ADDEND;
sample_set_addend(sc, sc->addend);
pthread_mutex_lock(&sc->hw_rw_mutex);
/* Configure the Rx and Tx timestamping hardware. */
pthread_mutex_unlock(&sc->hw_rw_mutex);
return EOK;
}
The sample_ptp_init_sw() function initializes the variables used to capture the PTP timestamps. The rx_ts array is used to capture the receive timestamps and the tx_ts array is used to capture the transmit timestamps. The handling of these arrays is discussed in the sections on transmit and receive packet handling below.
The sample_ptp_init_sw() function also initializes the indices used to track the active timestamp data.
static void sample_ptp_init_sw(struct sample_softc *sc)
{
pthread_mutex_lock(&sc->ptp_rx_mutex);
memset(sc->ptp_rx_ts_list, 0, sizeof(sc->ptp_rx_ts_list));
sc->ptp_rx_ts_rptr = 0;
sc->ptp_rx_ts_wptr = 0;
memset(sc->rx_ts, 0, sizeof(sc->rx_ts));
sc->rx_ts_cnt = 0;
pthread_mutex_unlock(&sc->ptp_rx_mutex);
pthread_mutex_lock(&sc->ptp_tx_mutex);
memset (sc->ptp_tx_hdr_list, 0, sizeof(sc->ptp_tx_hdr_list));
sc->ptp_tx_hdr_rptr = 0;
sc->ptp_tx_hdr_wptr = 0;
memset(sc->tx_ts, 0, sizeof(sc->tx_ts));
sc->tx_ts_cnt = 0;
pthread_mutex_unlock(&sc->ptp_tx_mutex);
}
The two functions sample_ptp_set_time() and sample_ptp_get_time() support the ioctl() interface into the PTP driver from the PTP daemon. The sample_ptp_get_time() function reads the PTP hardware time in seconds and nanoseconds and returns the value in the ptp_time_t structure. The sample_ptp_set_time() function updates the PTP hardware time from the daemon.
Frequently implementations also use these functions during hardware initialization to verify the PTP clock is performing as expected. Depending on the platform hardware, there may be other, better methods to verify the timestamping hardware initialization.
sample_ptp_stop(device_t dev)
{
/*
* Disable the timestamp interrupt, if used, and
* release the mutex resources.
*/
pthread_mutex_destroy(&sc->ptp_rx_mutex);
pthread_mutex_destroy(&sc->ptp_tx_mutex);
return EOK;
}
For the code details of a driver without PTP added, see the sample_detach() function in “A Hardware-Independent Sample Driver: sample.c.” In most drivers, PTP support is one of the last pieces of functionality turned on in the sample_attach(). Therefore, it should be one of the first pieces of functionality turned off in the sample_detach(). sample_detach(device_t dev)
{
/* Stop the PTP processing. */
sample_ptp_stop(dev);
/*
* Disable the hardware read/write mutex,
* This mutex may already exist in the driver you are modifying.
*/
pthread_mutex_destroy(&sc->hw_rw_mutex);
return EOK;
}
The transmit packet is processed in two stages.
In the first stage, the packet must be enqueued into the transmit queue. If the packet is a PTP event message, the transmit descriptor must be configured to have the hardware generate a timestamp when the packet is finally transmitted. Usually, the completed transmission of the packet and the generation of the timestamp generate an interrupt so that the software can quickly collect the generated timestamp and store it for later retrieval by the PTP daemon. The PTP daemon sends this timestamp to the far end in the Follow Up message. The Follow Up message is the second step of the two-step process.
When the PTP daemon requests the timestamp, it provides the header information of the specific event message that it is requesting data for. Providing the header information ensures that the correct timestamp is returned to the PTP daemon for each packet request.
static void
sample_txstart_locked(struct sample_softc *sc)
{
ptp_packet = 0;
for (i=0; i < nsegs; i++) {
/* First packet */
/* This is the first buffer required for packet data. It may also be last buffer required. */
if (tq_pkt_xbytes == 0) {
ptpv2hdr_t *ptp_hdr = NULL;
if (sample_ptp_is_eventmsg(m, &ptp_hdr)) {
ptp_packet = 1;
/* Event message
* Write into the Tx Descriptor to configure this PTP packet
* for timestamping when transmitted.
*/
}
}
/* Last packet or buffer required */
if (m == NULL || sc->tq_pkt_xbytes >= sc->tq_pkt_len) {
/* Process the last descriptor of the packet, as required. */
if (ptp_packet) {
/* Apply any extra hardware processing needed for a PTP
event message. */
}
}
}
}
The timestamp is collected just before the transmit descriptor is released back to the application.
static void
sample_txfinish_locked(struct sample_softc *sc)
{
uint32_t ts_nsec;
uint32_t ts_nsec;
sample_bd_t *bd;
ptpv2hdr_t *ph = NULL;
/* Just before releasing the mbuf, check whether there is a timestamp to collect.
If there is a Tx timestamp, collect it from the PTP hardware and store it in
the tx_ts array. It is stored in an array along with the
relevant Tx packet header information. Later, when the daemon calls
sample_ptp_get_timestamp() with an ioctl() call, the Tx packet header
information is used to ensure that the correct timestamp is returned.
*/
/* If this packet is timestamped, collect the timestamp information and save it. */
if (sample_ptp_is_eventmsg(m, &ph)) {
sample_ptp_add_tx_timestamp(sc, ph, bd);
}
if (tdesc->m != NULL){
m_free(tdesc->m);
tdesc->m = NULL;
}
}
int sample_ptp_add_tx_timestamp(struct sample_softc *sc, const ptpv2hdr_t *ptp_hdr, sample_bd_t *bd)
{
uint32_t length;
pthread_mutex_lock(&sc->ptp_tx_mutex);
if ( ((ptp_hdr->version & 0x0f) != 0x2) || /* Only PTPv2 is currently supported. */
(ptp_hdr == NULL) || (bd == NULL) ) {
pthread_mutex_unlock(&sc->ptp_tx_mutex);
return
}
/* Save the details. */
tx_ts[tx_ts_cnt].msg_type = ph->messageId & 0x0f;
tx_ts[tx_ts_cnt].sequence_id = ntohs(ph->sequenceId);
memcpy(tx_ts[tx_ts_cnt].clock_identity, ph->clockIdentity,
sizeof(tx_ts[tx_ts_cnt].clock_identity));
tx_ts[tx_ts_cnt].sport_id = ntohs(ph->sportId);
/* The timestamp is taken from the hardware. */
tx_ts[tx_ts_cnt].ts.nsec = bd->nsec;
/*
* If required, handle rollovers of the timer and asynchronous
* updates of the seconds stored in software.
*/
tx_ts[tx_ts_cnt].ts.sec = bd->sec;
/* Advance the counter, including wrapping. */
sc->tx_ts_cnt = (sc->tx_ts_cnt + 1) % PTP_TIMESTAMP_BUF_SZ;
pthread_mutex_unlock(&sc->ptp_tx_mutex);
return EOK;
}
The timestamp is collected before the receive packet is put into the io-sock stack.
static int sample_receive (sample_dev_t *sample, struct nw_work_thread *wtp)
sample_bd_t *bd
if (sample_ptp_is_eventmsg(m, &ptp_hdr)) {
/* Any required hardware processing */
sample_ptp_add_rx_timestamp (sample, ptp_hdr, *bd);
/* Set flag if further processing is required. */
timestamp_received = 1;
} else {
/* No further timestamp processing is required.
timestamp_received = 0;
}
}
void sample_ptp_add_rx_timestamp (struct sample_softc *sc, const ptpv2hdr_t *ptp_hdr, sample_bd_t *bd)
{
pthread_mutex_lock(&sc->ptp_rx_mutex);
if ( ((ptp_hdr->version & 0x0f) != 0x2) || /* Only PTPv2 is currently supported. */
(ptp_hdr == NULL) || (bd == NULL) ) {
pthread_mutex_unlock(&sc->ptp_rx_mutex);
return;
}
ptp_extts_t *ptp_entry = &sc->rx_ts[sc->rx_ts_cnt];
/* Add the details. */
ptp_entry->msg_type = ptp_hdr->messageId & 0x0f;
ptp_entry->sequence_id = ntohs(ptp_hdr->sequenceId);
memcpy(ptp_entry->clock_identity, ptp_hdr->clockIdentity,
sizeof(ptp_entry->clock_identity));
ptp_entry->sport_id = ntohs(ptp_hdr->sportId);
ptp_entry->ts.nsec = bd->nsec;
ptp_entry->ts.sec = bd->sec;
/* Advance the counter including wrapping. */
sc->rx_ts_cnt = (sc->rx_ts_cnt 1) % PTP_TIMESTAMP_BUF_SZ;
pthread_mutex_unlock(&sc->ptp_rx_mutex);
}
A driver must support the following ioctl() commands for the PTP protocol daemon:
#define PTP_GET_RX_TIMESTAMP 0x100 /* Get Rx timestamp. */ #define PTP_GET_TX_TIMESTAMP 0x101 /* Get Tx timestamp. */ #define PTP_GET_TIME 0x102 /* Get time. */ #define PTP_SET_TIME 0x103 /* Set time. */ #define PTP_SET_COMPENSATION 0x104 /* Set compensation. */ #define PTP_GET_COMPENSATION 0x105 /* Get compensation. */
In the sample_ioctl() function, the commands are passed to the driver in the cmd parameter. The data structures that define the format of the data transferred by each command are discussed below.
sample_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data)
{
switch (cmd) {
case SIOCSDRVSPEC:
case SIOCGDRVSPEC:
ifd = (struct ifdrv *)data;
error = sample_ptp_ioctl(sc, ifd);
break;
}
}
int sample_ptp_ioctl (struct sample_softc *sc, const struct ifdrv *ifd)
{
ptp_time_t time;
ptp_comp_t comp;
ptp_extts_t ts;
uint8_t tx;
int status = EOK;
if (ifd != NULL) {
switch(ifd->ifd_cmd) {
case PTP_GET_TX_TIMESTAMP:
case PTP_GET_RX_TIMESTAMP:
if (ifd->ifd_len != sizeof(ts)) {
return EINVAL;
}
if (copyin((((uint8_t *)ifd) + sizeof(*ifd)),
&ts, sizeof(ts))) {
return EINVAL;
}
if (ifd->ifd_cmd == PTP_GET_RX_TIMESTAMP) {
tx = 0;
} else {
tx = 1;
}
if (sample_ptp_get_timestamp(sc, &ts, tx)) {
return (copyout(&ts, (((uint8_t *)ifd) + sizeof(*ifd)),
sizeof(ts)));
}
return ENOENT;
break;
case PTP_GET_TIME:
if (ifd->ifd_len != sizeof(time)) {
return EINVAL;
}
sample_ptp_get_time(sc, &time);
return (copyout(&ts, (((uint8_t *)ifd) + sizeof(*ifd)),
sizeof(ts)));
break;
case PTP_SET_TIME:
if (ifd->ifd_len != sizeof(time)) {
return EINVAL;
}
if (copyin((((uint8_t *)ifd) + sizeof(*ifd)),
&time, sizeof(time))) {
return EINVAL;
}
status = sample_ptp_set_time(sc, &time);
if(status != EOK) {
return status;
}
/* Clock has changed so all old timestamps are invalid. */
sample_ptp_init_sw(sc);
return EOK;
break;
case PTP_SET_COMPENSATION:
if (ifd->ifd_len != sizeof(comp)) {
return EINVAL;
}
if (copyin((((uint8_t *)ifd) + sizeof(*ifd)),
&comp, sizeof(comp))) {
return EINVAL;
}
return sample_ptp_set_compensation(sc, &comp);
break;
case PTP_GET_COMPENSATION:
return ENOTTY;
default:
break;
}
}
return ENOTTY;
}
static int sample_ptp_get_timestamp (struct sample_softc *sc, ptp_extts_t *ts, uint8_t tx)
{
int i;
ptp_extts_t * buf;
if (ts == NULL) {
return 0;
}
/* Lock access to the list holding the timestamps. */
if (tx) {
buf = sc->tx_ts;
pthread_mutex_lock(&sc->ptp_tx_mutex);
} else {
buf = sc->rx_ts;
pthread_mutex_lock(&sc->ptp_rx_mutex);
}
for (i = 0; i < PTP_TIMESTAMP_BUF_SZ; i++) {
if ((ts->msg_type == buf[i].msg_type) &&
(ts->sequence_id == buf[i].sequence_id) &&
(ts->sport_id == buf[i].sport_id) &&
!memcmp(ts->clock_identity, buf[i].clock_identity,
sizeof(ts->clock_identity))) {
/* Collect the timestamp and release the mutex. */
ts->ts.nsec = buf[i].ts.nsec;
ts->ts.sec = buf[i].ts.sec;
if (tx) {
pthread_mutex_unlock(&sc->ptp_tx_mutex);
} else {
pthread_mutex_unlock(&sc->ptp_rx_mutex);
}
return 1;
}
}
/* Allow access to the list holding the timestamps. */
if (tx) {
pthread_mutex_unlock(&sc->ptp_tx_mutex);
} else {
pthread_mutex_unlock(&sc->ptp_rx_mutex);
}
return 0;
}
The sample_ptp_set_time() function writes the current time in seconds and nanoseconds into the PTP clock. It is not an adjustment to the free-running system clock. The free-running PTP hardware clock is logically syntonized using the addend value supplied by the PTP daemon (see IEEE Std 802.1AS™-2011, section “7.3.3 Logical syntonization”).
static int sample_ptp_set_time (struct sample_softc *sc, ptp_time_t *time)
{
pthread_mutex_lock(&sc->hw_rw_mutex);
/* Write the time into the PTP clock. */
pthread_mutex_unlock(&sc->hw_rw_mutex);
return EOK;
}
The sample_ptp_set_compensation() function has the most impact on your driver’s performance.
static int sample_ptp_set_compensation (struct sample_softc *sc, const ptp_comp_t *ptc)
{
uint32_t new_addend;
uint32_t delta;
uint64_t adjustment;
adjustment = ((uint64_t)sc->addend)*((uint64_t)ptc->comp);
delta = (uint32_t)(adjustment/1000000000ULL);
if (ptc->positive) {
new_addend = sc->addend + delta;
} else {
new_addend = sc->addend - delta;
}
/* the mutex is set in sample_set_addend */
return sample_set_addend (sc, new_addend);
}
The sample_set_addend() function updates the Logical Clock addend hardware. The IEEE Std 802.1AS™-2011 section “7.5 Differences between gPTP and PTP” states that all time-aware systems in a gPTP domain are logically syntonized (i.e., they all measure time intervals using the same frequency). This adjustment is described in the “7.3.3 Logical syntonization” section of the standard, and is mandatory. Syntonization in IEEE 1588 is optional.
static int sample_set_addend(struct sample_softc *sc, uint32_t addend)
{
pthread_mutex_lock(&sc->hw_rw_mutex);
/* Update the clock adjustment hardware.
This is required functionality when running 802.1AS */
pthread_mutex_unlock(&sc->hw_rw_mutex);
return EOK;
}