How to Build a Real-Time Object Detection and Tracking Pipeline with ROS 2 and YOLOv11

By Iyanuoluwa Enoch Oke

July 25, 2026 | #robotics

If you've ever tried to build a robotics system that can see, track, and respond to its environment, you know the real challenge isn't training a detection model—it's making that model run reliably within a real robotic software stack, in real time, without breaking under hardware constraints or timing issues.

In this tutorial, you'll build a complete real-time object detection and tracking pipeline using ROS 2 and YOLOv11. You'll learn how to:

• Publish camera frames from a simulator into ROS 2.
• Run YOLO inference in a dedicated thread.
• Integrate ByteTrack for multi-object tracking across frames.
• Export your model to ONNX for faster inference on constrained hardware (as of 2026, ONNX Runtime remains a top choice for edge deployment).

By the end, you'll understand not just how to wire these tools together, but why each architectural decision matters for a perception system built for production—not just a notebook demo.

Table of Contents

1. Prerequisites
2. What We Are Building and Why
3. Project Structure
4. How to Set Up Your ROS 2 Workspace
5. How to Install Dependencies
6. How to Publish Camera Frames into ROS 2
7. How to Build the Perception Node with Threaded Inference
8. How to Integrate ByteTrack for Multi-Object Tracking
9. How to Add a Confidence Validation Layer



---




Prerequisites



• ROS 2 (Humble or newer; as of 2026, ROS 2 Jazzy is recommended)
• Python 3.10+ with OpenCV, NumPy, and PyTorch (or ONNX Runtime)
• A working Gazebo or Webots simulator setup (or a real camera feed)
• Basic understanding of ROS 2 nodes, topics, and services



---




What We Are Building and Why




We're building a perception pipeline that:

1. Captures camera images from a simulated robot.
2. Publishes them as ROS 2 sensor_msgs/Image messages.
3. Runs YOLOv11 inference (exported to ONNX for speed) in a separate thread to avoid blocking the ROS 2 callback.
4. Applies ByteTrack to maintain consistent object IDs across frames.
5. Publishes detected object bounding boxes, class labels, and tracking IDs as custom ROS 2 messages.



Why this architecture? Real-time performance on embedded hardware often requires separating I/O (ROS 2 callbacks) from compute (inference). In 2026, ROS 2's executors and threading model make this cleaner than ever, but the core design remains critical.




---




Project Structure




ros2_perception_pipeline/
├── launch/
│   └── perception_pipeline.launch.py
├── src/
│   ├── camera_publisher/
│   │   ├── package.xml
│   │   └── camera_publisher_node.py
│   └── perception_node/
│       ├── package.xml
│       ├── perception_node.py
│       ├── yolo_inference.py
│       └── tracker.py
└── models/
    └── yolov11n.onnx




---




How to Set Up Your ROS 2 Workspace




mkdir -p ~/ros2_perception_ws/src
cd ~/ros2_perception_ws/src
ros2 pkg create --build-type ament_python camera_publisher
ros2 pkg create --build-type ament_python perception_node




Update setup.py and package.xml files as needed. For 2026 compliance, ensure your package.xml includes depend>sensormsgsdepend> and depend>cvbridgedepend>.




---




How to Install Dependencies




pip install ultralytics onnxruntime-bbox-tracker opencv-python




For ROS 2 interface:




sudo apt install ros-humble-cv-bridge ros-humble-sensor-msgs-py




(Replace humble with your ROS 2 distro, e.g., jazzy.)




---




How to Publish Camera Frames into ROS 2




Create camerapublishernode.py:




import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import cv2

class CameraPublisher(Node):
    def __init__(self):
        super().__init__('camera_publisher')
        self.publisher_ = self.create_publisher(Image, 'camera/image_raw', 10)
        self.bridge = CvBridge()
        self.timer = self.create_timer(0.033, self.timer_callback)  # ~30 FPS
        self.cap = cv2.VideoCapture(0)  # Replace with simulator camera topic if needed

    def timer_callback(self):
        ret, frame = self.cap.read()
        if ret:
            msg = self.bridge.cv2_to_imgmsg(frame, 'bgr8')
            self.publisher_.publish(msg)
            self.get_logger().info('Published camera frame')

def main(args=None):
    rclpy.init(args=args)
    node = CameraPublisher()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()




---




How to Build the Perception Node with Threaded Inference




The perception node subscribes to /camera/image_raw, runs YOLO inference in a background thread, and publishes results.




import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import threading
import queue
import time
from .yolo_inference import YOLOInference
from .tracker import ObjectTracker

class PerceptionNode(Node):
    def __init__(self):
        super().__init__('perception_node')
        self.subscription = self.create_subscription(
            Image, 'camera/image_raw', self.image_callback, 10)
        self.publisher_ = self.create_publisher(
            DetectionArray, 'perception/detections', 10)
        self.bridge = CvBridge()
        self.inference_queue = queue.Queue(maxsize=2)
        self.result_queue = queue.Queue()
        self.yolo = YOLOInference('models/yolov11n.onnx')
        self.tracker = ObjectTracker()
        
        # Start inference thread
        self.inference_thread = threading.Thread(target=self.inference_loop, daemon=True)
        self.inference_thread.start()
        
        # Start result publishing thread
        self.publish_thread = threading.Thread(target=self.publish_loop, daemon=True)
        self.publish_thread.start()

    def image_callback(self, msg):
        cv_image = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        if not self.inference_queue.full():
            self.inference_queue.put(cv_image)

    def inference_loop(self):
        while rclpy.ok():
            if not self.inference_queue.empty():
                frame = self.inference_queue.get()
                detections = self.yolo.infer(frame)
                tracks = self.tracker.update(detections)
                self.result_queue.put(tracks)
            time.sleep(0.001)  # Prevent busy-wait

    def publish_loop(self):
        while rclpy.ok():
            if not self.result_queue.empty():
                tracks = self.result_queue.get()
                msg = DetectionArray()
                # Populate msg from tracks
                self.publisher_.publish(msg)
            time.sleep(0.001)

def main(args=None):
    rclpy.init(args=args)
    node = PerceptionNode()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()




---




How to Integrate ByteTrack for Multi-Object Tracking




ByteTrack is ideal for real-time tracking because it associates detections based on bounding box overlap rather than appearance features, making it lightweight.




Install via:




pip install onnxruntime-bbox-tracker




Then create tracker.py:




from bbox_tracker import BYTETracker

class ObjectTracker:
    def __init__(self, track_thresh=0.5, match_thresh=0.8):
        self.tracker = BYTETracker(track_thresh=track_thresh, match_thresh=match_thresh)

    def update(self, detections):
        # detections: list of [x1, y1, x2, y2, score, class_id]
        tracks = self.tracker.update(detections)
        return tracks




Tracks are returned with unique IDs persistent across frames.




---




How to Add a Confidence Validation Layer




To reduce false positives, implement a validation layer that filters detections based on temporal consistency and confidence threshold.




Add this to perception_node.py:




class ConfidenceValidator:
    def __init__(self, min_confidence=0.4, history_size=5):
        self.min_confidence = min_confidence
        self.history = {}

    def validate(self, tracks, frame_id):
        valid_tracks = []
        for track in tracks:
            track_id = track['id']
            if track['confidence'] < self.min_confidence:
                continue
            if track_id not in self.history:
                self.history[track_id] = []
            self.history[track_id].append(frame_id)
            if len(self.history[track_id]) >= 3:  # must be seen in 3+ frames
                valid_tracks.append(track)
        return valid_tracks




This ensures that only consistently appearing objects are published, critical for real-world robotics.




---




Running the Pipeline




Build and source your workspace:




cd ~/ros2_perception_ws
colcon build
source install/setup.bash




Launch both nodes:




ros2 run camera_publisher camera_publisher_node &
ros2 run perception_node perception_node




Or create a launch file for convenience.




---




Conclusion




You've built a robust real-time object detection and tracking pipeline using ROS 2 and YOLOv11. By threading inference separately from ROS 2 callbacks, integrating ByteTrack for consistent multi-object IDs, and adding a confidence validation layer, your system is ready for deployment on robots in 2026.




Key takeaways:

• Separate I/O from compute for real-time reliability.
• ONNX export boosts inference speed on edge hardware.
• ByteTrack provides lightweight but effective multi-object tracking.
• Temporal validation filters out noisy detections.



Now go try this on a real robot—or in simulation—and watch your perception system come to life.




---




Questions or improvements? Reach out to the author.

via FreeCodeCamp