luxonis_ml.data.datasets.annotation

Record Model

Dataset ingestion starts with DatasetRecord. A record points to media, optionally assigns a task name, and optionally carries an annotation payload validated by Detection.

Single-source records use "file":

{
    "file": "path/to/image.jpg",
    "task_name": "detection",
    "annotation": {
        "class": "car",
        "boundingbox": {"x": 0.1, "y": 0.2, "w": 0.3, "h": 0.4},
    },
}

Multi-source records use "files":

{
    "files": {
        "rgb": "path/to/rgb.png",
        "depth": "path/to/depth.png",
    },
    "task_name": "detection",
    "annotation": {
        "class": "person",
        "boundingbox": {"x": 0.1, "y": 0.1, "w": 0.3, "h": 0.4},
    },
}

Task names group annotations that should be consumed together. If no task_name is provided, the empty string "" is used. Loader label keys therefore follow "task_name/task_type" and default-task keys start with "/".

Coordinates and Instances

Spatial annotations use image-normalized coordinates. For an image with width W and height H, an absolute point (x, y) is stored as (x ⁄ W, y ⁄ H).

When multiple annotation types describe the same physical object, use the same instance_id so bounding boxes, keypoints, and instance masks can be associated even when yielded as separate records.

Detection.scale_to_boxes supports box-relative annotations. When enabled, keypoints and segmentation values are interpreted relative to the bounding box and rescaled to image-normalized coordinates before storage.

Classification

Classification assigns a class to the whole sample or instance:

{"class": "vehicle"}

Classification is represented internally as the classification task type. Any detection that provides a class name contributes a classification target, even when the same detection also contains boxes, masks, keypoints, arrays, or metadata. When loaded, classes are usually returned as one-hot vectors with shape (C).

Bounding Boxes

BBoxAnnotation stores normalized xywh boxes, where x and y are the top-left corner and w and h are width and height:

{
    "class": "car",
    "instance_id": 17,
    "boundingbox": {
        "x": 0.20,
        "y": 0.10,
        "w": 0.35,
        "h": 0.25,
    },
}

Loader output combines boxes into (N, 5) arrays with rows [c, x, y, w, h], where c is the class index.

Keypoints

KeypointAnnotation stores keypoints as (x, y, visibility) triplets. Coordinates are normalized and visibility follows the COCO convention:

• 0: not visible or not labeled.
• 1: occluded.
• 2: visible.

{
    "class": "car",
    "instance_id": 17,
    "keypoints": {
        "keypoints": [
            (0.10, 0.20, 2),
            (0.30, 0.40, 1),
        ],
    },
}

For K keypoints and N instances, loader output uses shape (N, 3⋅K).

Segmentation

SegmentationAnnotation supports polygon, binary-mask, and run-length encoded inputs.

Polyline segmentation stores normalized polygon points. The final point is implicitly connected to the first one:

{
    "class": "road",
    "segmentation": {
        "height": 720,
        "width": 1280,
        "points": [
            (0.10, 0.10),
            (0.90, 0.10),
            (0.80, 0.80),
            (0.20, 0.80),
        ],
    },
}

Binary masks are two-dimensional arrays where foreground pixels are 1 and background pixels are 0:

{
    "class": "road",
    "segmentation": {
        "mask": binary_mask,
    },
}

Run-length encoded masks use COCO RLE. The counts value may be an uncompressed list of integers or a compressed byte string:

{
    "class": "road",
    "segmentation": {
        "height": 720,
        "width": 1280,
        "counts": [120, 8, 200, 12],
    },
}

Semantic segmentation loader output uses channel-first masks with shape (C, H, W).

Instance Segmentation

InstanceSegmentationAnnotation uses the same mask encodings as semantic segmentation, but stores one mask per instance. A detection may include both a bounding box and an instance mask:

{
    "class": "car",
    "instance_id": 17,
    "boundingbox": {"x": 0.20, "y": 0.10, "w": 0.35, "h": 0.25},
    "instance_segmentation": {
        "height": 720,
        "width": 1280,
        "points": [
            (0.20, 0.10),
            (0.55, 0.10),
            (0.55, 0.35),
            (0.20, 0.35),
        ],
    },
}

Instance-mask loader output uses shape (N, H, W).

Arrays

ArrayAnnotation references arbitrary .npy data synchronized with a sample:

{
    "class": "embedding",
    "array": {
        "path": "path/to/embedding.npy",
    },
}

Arrays are useful for modality-specific targets or auxiliary data that should be stored with the dataset but does not fit standard spatial schemas.

Metadata and Categories

Metadata stores flexible key-value values. Use Category to mark a string as categorical metadata rather than free-form text:

from luxonis_ml.data import Category

{
    "metadata": {
        "text": "ABC-123",
        "text_color": Category("white"),
        "track_id": 42,
    },
}

Categorical metadata can be encoded as integers by LuxonisLoader, or kept as strings when loader configuration requests it.

OCR annotations commonly store recognized text and categorical visual properties:

{
    "metadata": {
        "text": "ABC-123",
        "color": Category("red"),
    },
}

Embedding and re-identification datasets commonly store identifiers or other lookup keys:

{
    "metadata": {
        "id": 42,
        "color": Category("red"),
    },
}