PTP Guide
PTP Profiles Guide
Overview
The Ouster sensor’s PTP profile must match your master clock’s profile for time synchronization to work. This guide explains how to set the profile.
PTP Profiles
There are several PTP profiles that are commonly used. The supported profiles on the Ouster sensor are listed below: The supported profiles on the Ouster sensor are:
default— IEEE 1588 Default profile. Supported by most PTP-capable devices.gptp— IEEE 802.1AS-2011 (gPTP). Simplifies PTP for improved interoperability. Use with gPTP-compatible hardware such as AVB devices, e.g., the MOTU AVB.automotive-slave— Common in automotive applications. BMCA is disabled, the slave suppresses announce messages, and the convergence controller is ~8× faster than the Default profile.
PTP HTTP API
Change the PTP profile with an HTTP PUT request using any HTTP client (HTTPie, curl, etc.). The full API is in http-v1-get-ptp-profile.
The request URL is: http://<sensor_hostname>/api/v1/time/ptp/profile/
-
Valid values are (““, are included):
-
“default” -
“gptp” -
“automotive-slave”
Note: Changing the PTP profile takes effect immediately on a valid PUT response. No reinitialization or config file write is required.
Enabling the PTP profiles
Below are some examples using popular command-line tools.
Example using cURL
Set the PTP profile to "gptp" using cURL:
Command:
Response:
Example using HTTPie
Set the PTP profile to "default" using HTTPie:
Command:
Response:
Sync Verification
Please see the sensor-ptp-sync-verify section for details on how to verify the sensor is synchronized.
PTP Quickstart Guide
This guide covers basic PTP network configuration using Ubuntu 18.04. It includes settings for a commercial PTP grandmaster clock and instructions for configuring a Linux machine as a PTP grandmaster.
The linuxptp project provides a suite of PTP tools that can be used to serve as a PTP master clock for a local network of sensors.
Assumptions
- Command line Linux knowledge (e.g., package management, command line familiarity, etc.).
- Ethernet interfaces that support hardware timestamping.
- Ubuntu 18.04 is assumed for this tutorial, but any modern distribution should suffice.
- Knowledge of systemd service configuration and management.
- Familiarity with Linux permissions.
Physical Network Setup
Connect the sensor to the PTP grandmaster with at most one network switch. A direct connection is ideal. A single layer-2 gigabit Ethernet switch is acceptable. Multiple switches add jitter and are not recommended.
Third Party Grandmaster Clock
Use a dedicated grandmaster clock with a GPS receiver for highest absolute accuracy.
It must be configured with the following parameters which match the linuxptp
client defaults:
- Transport:
UDP IPv4 - Delay Mechanism:
E2E - Sync Mode:
Two-Step - Announce Interval:
1- sent every 2 seconds - Sync Interval:
0- sent every 1 second - Delay Request Interval:
0- sent every 1 second
For more settings, review the port_data_set field returned from the sensor’s HTTP /time/ptp interface.
Linux PTP Grandmaster Clock
When absolute accuracy allows, run a local Linux PTP master clock instead of an external grandmaster. This is commonly done on vehicle computers that interface directly with lidar sensors.
This section outlines how to configure a master clock.
Example Network Setup
This section assumes the following network setup as it has elements of a local master clock and the option for an upstream PTP time source.
The focus is configuring the Linux PTP service to distribute a common clock to all downstream Ouster OS1 sensors, using the Ubuntu host system time.
Optionally, add a grandmaster clock to discipline the Linux system time.
Installing Necessary Packages
Install the following packages for PTP functionality:
-
linuxptp- Linux PTP package with the following components:ptp4ldaemon to manage hardware and participate as a PTP nodephc2systo synchronize the Ethernet controller’s hardware clock to the Linux system clock or shared memory regionpmcto query the PTP nodes on the network.
-
chrony- A NTP and PTP time synchronization daemon. It can be configured to listen to both NTP time sources via the Internet and a PTP master clock such as one provided by a GPS with PTP support. This will validate the time configuration makes sense given multiple time sources. -
ethtool- A tool to query the hardware and driver capabilities of a given Ethernet interface.
Ethernet Hardware Timestamp Verification
Identify the Ethernet interface (e.g., eno1) and run ethtool to confirm hardware timestamping support:
Output of ethtool -T for a functioning Intel i210 Ethernet interface:
Configurations
Configuring ptp4l for Multiple Ports
On a system with multiple Ethernet ports (e.g., Intel i210), configure /etc/linuxptp/ptp4l.conf to include all relevant interfaces:
Note: Add the above required modification at the end of the existing file. Deleting or editing the default settings section of the ptp4l.conf file will result in an error.
The default Ubuntu 18.04 systemd service hardcodes eth0. Override it to use the configuration file instead:
Create a systemd drop-in directory to override the system service file:
Create /etc/systemd/system/ptp4l.service.d/override.conf with the following contents:
Restart the ptp4l service so the change takes effect:
The above systemctl status ptp4l console output shows systemd correctly reading the override file created earlier before starting several seconds after the restart command.
The log output shows that a grandmaster clock has been discovered on port 1 (eno1) and port 2 (eno2) is currently disconnected and in the faulty state as expected. In the test network a Trimble Thunderbolt PTP GM100 Grandmaster Clock is attached on eno1.
Logs can be monitored (i.e. followed) like so:
Configuring ptp4l as a Local Master Clock
The IEEE-1588 Best Master Clock Algorithm (BMCA) selects the grandmaster from all available masters. In most networks only one master should exist. Configure the Ubuntu machine with a lower clockClass value so it wins the BMCA.
Replace the default value with a lower clock class (higher priority) and restart linuxptp. Edit /etc/linuxptp/ptp4l.conf and comment out the default clockClass value and insert a line setting it 128.
Restart ptp4l so the configuration change takes effect.
This will configure ptp4l to advertise a master clock on eno2 as a clock that will win the BMCA for an Ouster OS1 sensor.
However, the ptp4l service is only advertising the Ethernet controller’s PTP hardware clock, not the Linux system time as is often expected.
Configuring phc2sys to Synchronize the System Time to the PTP Clock
Run phc2sys to sync the Linux system time to the PTP hardware clock. The following configuration writes CLOCK_REALTIME to the PTP hardware clock on eno2, which connects to Ouster OS1 sensors.
Create a systemd drop-in directory to override the system service file:
Create /etc/systemd/system/phc2sys.service.d/override.conf with the following contents:
Note: If multiple interfaces need to be synchronized from
CLOCK_REALTIMEthen multiple instances of thephc2sysservice need to be run as it only accepts a single slave (i.e.-c) argument.
Restart the phc2sys service so the change takes effect:
Configuring Chrony to Set System Clock Using PTP
To set the system time from an upstream GPS-disciplined PTP grandmaster, use Chrony. Chrony reads from both NTP and PTP and selects the most accurate source. When the PTP grandmaster is functioning correctly, it will be selected over NTP.
The following phc2shm service will synchronize the time from eno1 (where the external grandmaster is attached) to the system clock.
Create /etc/systemd/system/phc2shm.service with the following contents:
Start the service and verify it started:
Add the PTP time source to the chrony configuration. Append to /etc/chrony/chrony.conf:
Restart chrony so the updated configuration takes effect:
After waiting a minute for the clock to synchronize, review the chrony client timing accuracy:
Note that the Reference ID matches the ptp reference ID from the chrony.conf file and that the sources output shows the ptp reference ID as selected (signified by the * state in the second column). Additionally, the NTP time sources show a small relative error to the high accuracy PTP time source.
In this case the PTP grandmaster is properly functioning.
If this error is large, chrony will select the NTP time sources and mark the PTP time source as invalid. This typically signifies that something is mis-configured with the PTP grandmaster upstream of this device or the linuxptp configuration.
Verifying Operation
After setting up a new PTP grandmaster, power-cycle the sensor. This causes it to jump directly to the correct time rather than slowly slewing from an incorrect initial value.
Sensor PTP Sync Verification
Query the sensor’s local PTP state via the http-v1-get-system-time-ptp request.
JSON response fields to check:
parent_data_set.grandmaster_identityshould list the identity of the local grandmasterport_data_set.port_stateshould beSLAVEtime_status_np.gm_presentshould betruetime_status_np.master_offsetwhich is given in nanoseconds, should be less than250000. This equates to 250 microseconds.
PTP Example JSON Response
LinuxPTP PMC Tool
The sensor responds to PTP management messages. Use the linuxptp pmc utility (man pmc) to query all PTP devices on the local network.
On the Linux host, run:
Tested Grandmaster Clocks
- Trimble Thunderbolt PTP GM100 Grandmaster Clock
- Firmware version:
20161111-0.1.4.0, November 11 2016 15:58:25 - PTP configuration:
- Ubuntu 18.04 + Linux PTP as a master clock
- Intel i210 Ethernet interface
- PCI hardware identifiers:
8086:1533 (rev 03) - Ubuntu 18.04 kernel package:
linux-image-4.18.0-16-generic - Ubuntu 18.04 linuxptp package:
linuxptp-1.8-1
Analyzing Linux Networking Issues
Note: Follow this section only when experiencing intermittent packet drops or reordering. First verify
udp_destsettings—this section does not apply if you are receiving zero data. For zero-data issues, contact our Field Application Team.
This section covers tools and procedures for debugging networking issues in systems with a PC/Workstation, L2 switch, and one or more Ouster sensors. Examples use Linux, but the concepts apply equally to Windows.
Debugging the Workstation Data Path
Diagnose packet loss by examining per-layer statistics in the network stack. Always start at the lowest layer—L1 issues cascade upward and can appear as higher-layer problems.
Link Layer Statistics and Configuration
ethtool
ethtool queries NIC statistics and configuration. While Linux offers more generic kernel filesystem interfaces, ethtool is the most complete tool for NIC debug. Note that output and options are vendor-specific and vary by NIC.
Line Interface Statistics
Start link-layer debugging with ethtool -S <ethX>, where ethX is the NIC identifier from ifconfig. If the system has multiple NICs and you are unsure which is receiving sensor traffic, run traffic and observe which NIC shows incrementing stats.
Note: The output of
ethtool -S <ethX>is 100% NIC vendor specific and will be quite different depending on NIC vendor used in your system.
Example output of ethtool -S:
MAC Errors
Focus on rx (receive) statistics. Any stat labeled rx.*error may indicate a problem. These MAC-level errors typically point to L1 issues: loose connectors, faulty transceivers, or out-of-spec cables. They can also indicate a link-partner MAC problem.
Internal System Errors
Stats like rx_dma_failed and rx_no_buffer_count lack an “error” suffix but indicate real failures in the NIC driver handoff.
Solving MAC Errors
MAC errors most likely indicate a cabling issue. Replace the cable first. If errors persist, test against a different node using iPerf/iPerf3 (see below). As a final step, swap out the sensor.
Solving Internal System Errors
These errors are often counterintuitive—the MAC receives all traffic, yet frames are still dropped. The root cause is typically processor overload at peak rate. Even if average throughput is modest, all NIC traffic arrives at line rate. On a 10G NIC, this means bursts of back-to-back frames that the CPU cannot process in time.
Just how many frames arrive depends on the behavior of the sensors. Ouster sensor attempts to transmit the entire LIDAR frame all at once. Assuming a 40K (on the wire) LiDAR frame and 10 sensors, the worst case load will be 40K x 10 = 400K at 10G (since the peak transmit rate of each sensor is 1G x 10 = 10G.) 400K is a lot of 10G data to process all at once, and without hardware buffering things will certainly fail.
The NIC maintains a hardware ring-buffer or on advanced hardware, potentially multiple ring-buffers. The entries in the ring-buffer are pointers into kernel packet-buffer structures. This mechanism enables the NIC to efficiently deliver packets to the kernel at line rate. For our specific use-case the default size of this ring-buffer may be too small.
To update this value user can use ethtool:
ethtool -g <ethX>— display current ring-buffer settings and device limitsethtool -G <ethX> rx <value>— update the rx ring-buffer size
Example: Using a laptop/system, ring-buffer has 256 entries by default:
To find out how much buffer is sufficient we can apply the burst-tolerance equation:
fill_rate = NIC_line_speed - max_measured_throughput fill_time = rx_buffer_size * 1518 * 8 / fill_rate MBS = fill_time * NIC_line_speed
Note: It is not always easy to obtain max_measured_throughput, and in a busy workstation it can be subject to variable delay.
As a rule-of-thumb we need to at least accommodate one max-burst (one LiDAR packet) from the sensor. Assuming a 40KB LiDAR packet that’s 40KB/1518=27 frames. So 256 should be more than adequate.
However, even with the default buffer of 256, user can observe packet loss due to DMA errors. This is because the work-station is not a real-time system and the delay can be quite variable. Linux uses a technique called interrupt coalescence that determines how often it will service the driver, when it gets very busy.
Interrupt coalescence is controlled by the kernel filesystem key:
/proc/sys/net/core/netdev_budget_usecs and by default it's 8000us!
If the problem is not resolved by increasing the buffer size, it’s possible to reduce netdev_budget_usecs in order to favor moving data over other activities that the system could be doing. It’s also possible to increase the maximum number of frames the OS is willing to process when the line interface does get serviced which is controlled by:
/proc/sys/net/core/netdev_budget
Note: On some systems the user need to make the rx-ring-buffer quite large or disable interrupt coalescence all together.
The NIC also delays hardware interrupts. Use ethtool -c to view the ACQ107’s default settings:
Another useful parameter is the /proc/sys/net/core/netdev_max_backlog. The backlog queue, is a FIFO on the other side of the NIC ring-buffer. Increasing the backlog buffer is one more way to add capacity earlier in the data-path. It’s difficult to determine when to increase netdev_max_backlog vs increasing the rx ring-buffer. Certainly the ring-buffer is the only place where we can add capacity that can absorb traffic bursts at line rate.
Troubleshooting Advanced NICs
Advanced hardware interfaces have multiple ring-buffers that are typically mapped to different CPU cores (a technique known as RSS.) Each NIC has its own proprietary scheme for mapping input traffic flows to ring-buffers, and sometimes a NIC will incorrectly split a traffic flow into multiple FIFOs. If you see this behavior it means that the NIC itself will cause frames to be reordered in a way that will horribly disrupt the IP stack above it. The ACQ107 is one such NIC. The problem can be identified by looking at ethtool -S <ethX>. The NIC will list stats for each FIFO, and by sending a single large traffic flow we can see that device errantly split the flow into all of the different FIFOs.
Below you can see that this NIC has stats labeled Queue[0] … Queue[7].
Example:
The vendor provided a workaround in their README .
Note: RSS for UDP Currently, NIC does not support RSS for fragmented IP packets, which leads to an incorrect handling of RSS for fragmented UDP traffic. To disable RSS for UDP one can use the following RX Flow L3/L4 rule: ethtool -N eth0 flow-type udp4 action 0 loc 32
When Stats Fail
Sometimes a NIC will drop frames without any error stats incrementing. When this happens, the issue can be detected by inserting a managed L2 switch in between the sensor and the workstation. The managed switch will report receive and transmit stats, which can be correlated against the rx stats of the NIC to determine that the NIC has dropped frames without incrementing any stat.
IP Statistics
After the link layer the next layer up is IP. IP errors can be identified with the netstat tool:
netstat -s
This tool outputs a lot of information; focus on the IP section.
In this report you can see that there are a few different error categories:
Let’s look at each class of error and consider it’s implications:
- Packets received with invalid address means that they were sent to our MAC, but with an incorrect source IP.
- Packets dropped because of missing route indicates that the packet was sent to the correct IP address but no client program was listening on the destination port.
- Fragments dropped after timeout means that we received some data but subsequent data didn’t arrive in time to be processed.
- Fragments reassemblies failed means that some data was missing due to an Ethernet frame being aborted by the stack or being lost in transit and the IP layer was not able to reassemble a complete datagram.
Debugging a Layer 3 Issue
The best way to debug issues in the IP layer is to find them in the link layer, because generally speaking layer-2 issues are caused by layer-3 bugs, but this is not always the case.
For instance, packets received with invalid address are probably indicative of stale ARP table entries or some other external network bug or temporal state that will most likely clear up on its own. This sort of problem is probably not worth debugging unless its persistent. Packets dropped because of missing route is more indicative of an issue at the application layer (the client or server simply wasn’t listening when the packets arrived).
If a problem is detectable by L3 and not by L2, then its most likely a problem in the NIC itself, and if the NIC isn’t providing a FIFO or DMA stat that explains it. One possibility is packet reordering by the NIC. This can be detected by modifying
/proc/sys/net/ipv4/ipfrag_max_dist
This kernel attribute determines the systems tolerance to receiving out-of-order IPv4 frames. Nominally L2 networks do not reorder packets, so you should be able to configure a value of 1 and not observe a change in behavior. However, if setting a low threshold exacerbates the issue, or setting a high value makes the problem less severe then the NIC is most likely to blame.
Useful network debugging tools
iPerf
iPerf validates whether a system can sustain a target throughput. Configure it to mimic the sensor’s expected data rate. See the iPerf documentation for details.
Use iPerf to rule out sensor-side failures and reproduce errors under high network load. Run it from two machines:
- Server (receiving data)
- Client (sending data)
Both sides report packet counts and loss percentage.
Example: Test sender sending 300 Mbps of UDP packets with 20 KB blocks to receiver:
Receiver (server side):
Receiver arguments:
--server— Required. Indicates this machine receives data.--port 5300— Port to listen on. Useful for testing with multiple sources.
Sender (client side):
Sender arguments:
--client 192.168.88.248— IP address of the receiver.--port 5300— Port to send to. Must match--porton the receiver.--udp— Sends UDP traffic. Without this flag, TCP is used.--bitrate 300M— Send rate in bits/sec. Supports K, M, G suffixes.--length 20K— UDP datagram size (20 KB).