class LuxonisDataset(BaseDataset):
Constructor: LuxonisDataset(dataset_name, team_id, bucket_type, bucket_storage, ...)
Luxonis Dataset Format (LDF) dataset handle.
LDF is a flexible and feature-rich dataset format designed for use within the Luxonis MLOps ecosystem.
| Static Method | exists |
Check whether a dataset exists. |
| Static Method | list |
List available datasets. |
| Method | __eq__ |
Compare datasets for equivalence. |
| Method | __init__ |
Create a Luxonis Dataset Format dataset handle. |
| Method | __len__ |
Return the number of records in the dataset. |
| Method | add |
Add data to the dataset from a generator of records. |
| Method | clone |
Create a local copy of the current dataset. |
| Method | delete |
Delete the dataset from local storage and optionally the cloud. |
| Method | export |
Export the dataset into one of the supported formats. |
| Method | get |
Get the categorical encodings for the dataset grouped by task. |
| Method | get |
Get class names and IDs per task. |
| Method | get |
Get the metadata types for each metadata annotation in the dataset. |
| Method | get |
Return keypoint skeletons for each task. |
| Method | get |
Return input source names for the dataset. |
| Method | get |
Get the dataset splits definitions. |
| Method | get |
Return dataset statistics for a view or the full dataset. |
| Method | get |
Return task names and task types. |
| Method | make |
Create dataset splits for training, validation, and testing. |
| Method | merge |
Merge another dataset into this or a new dataset. |
| Method | pull |
Synchronize the dataset from a remote bucket to a local storage. |
| Method | push |
Push the local dataset to a remote bucket. |
| Method | remove |
Remove duplicate files and annotations from the dataset. |
| Method | set |
Set class order for specific tasks. |
| Method | set |
Set classes for one or more tasks. |
| Method | set |
Set keypoint skeleton metadata. |
| Method | set |
Set dataset tasks. |
| Method | update |
Update the dataset source definition. |
| Instance Variable | bucket |
Underlying storage backend for the dataset. |
| Instance Variable | bucket |
Whether the dataset uses internal or external buckets. |
| Instance Variable | dataset |
Name of the dataset. |
| Instance Variable | team |
Optional cloud team identifier. |
| Property | identifier |
The unique identifier for the dataset. |
| Property | is |
Whether the dataset is stored remotely (in a cloud bucket) or locally. |
| Property | metadata |
Get the dataset metadata. |
| Property | source |
Get the source information for the dataset. |
| Property | version |
The version of the underlying LDF that the dataset adheres to. |
| Static Method | _construct |
Undocumented |
| Static Method | _scan |
Undocumented |
| Method | _add |
Undocumented |
| Method | _get |
Get secret credentials from the credentials file or environment. |
| Method | _get |
Load unique file entries from annotation data. |
| Method | _get |
Load metadata from local storage or cloud. |
| Method | _ignore |
Undocumented |
| Method | _init |
Undocumented |
| Method | _init |
Configure local paths or a bucket directory. |
| Method | _load |
Load the dataset DataFrame from local storage. |
| Method | _load |
Undocumented |
| Method | _merge |
Merge relevant metadata from other into self. |
| Method | _merge |
Undocumented |
| Method | _process |
Undocumented |
| Method | _save |
Save annotations DataFrame into parquet files. |
| Method | _save |
Undocumented |
| Method | _warn |
Undocumented |
| Method | _write |
Undocumented |
| Instance Variable | _annotations |
Undocumented |
| Instance Variable | _arrays |
Undocumented |
| Instance Variable | _base |
Undocumented |
| Instance Variable | _bucket |
Undocumented |
| Instance Variable | _bucket |
Undocumented |
| Instance Variable | _bucket |
Undocumented |
| Instance Variable | _credentials |
Undocumented |
| Instance Variable | _dataset |
Undocumented |
| Instance Variable | _fs |
Undocumented |
| Instance Variable | _is |
Undocumented |
| Instance Variable | _local |
Undocumented |
| Instance Variable | _media |
Undocumented |
| Instance Variable | _metadata |
Undocumented |
| Instance Variable | _metadata |
Undocumented |
| Instance Variable | _path |
Undocumented |
| Instance Variable | _team |
Undocumented |
| Property | _progress |
Undocumented |
Inherited from BaseDataset:
| Method | get |
Return class names per task. |
| Method | get |
Return number of classes per task. |
| Method | get |
Return the number of keypoints for each task. |
| Method | get |
Return task names for the dataset. |
@override
str, team_id: str | None = None, bucket_storage: BucketStorage = BucketStorage.LOCAL, bucket: str | None = None) -> bool:
¶
Check whether a dataset exists.
| Parameters | |
datasetstr | Dataset name to check. |
teamstr | None | Optional team identifier. |
bucketBucketStorage | Storage backend to inspect. |
bucket:str | None | Optional bucket name for remote storage. |
| Returns | |
bool | True if the dataset exists, False otherwise. |
| Raises | |
ValueError | If bucket storage is remote but no bucket name is provided. |
str | None = None, bucket_storage: BucketStorage = BucketStorage.LOCAL, bucket: str | None = None) -> list[ str]:
¶
List available datasets.
| Parameters | |
teamstr | None | Optional team identifier. |
bucketBucketStorage | Storage backend to inspect. |
bucket:str | None | Optional bucket name for remote storage. |
| Returns | |
list[ | List of dataset names. |
| Raises | |
ValueError | If bucket storage is remote but no bucket name is provided. |
ValueError | If the dataset is stored remotely but no bucket parameter is provided or no LUXONISML_BUCKET environment variable is set. |
str, team_id: str | None = None, bucket_type: BucketType | Literal[ 'internal', 'external'] = BucketType.INTERNAL, bucket_storage: BucketStorage | Literal[ 'local', 'gcs', 's3', 'azure'] = BucketStorage.LOCAL, *, delete_local: bool = False, delete_remote: bool = False):
¶
Create a Luxonis Dataset Format dataset handle.
| Parameters | |
datasetstr | Dataset name. |
teamstr | None | Optional cloud team identifier. |
bucketBucketType | Literal[ | Whether the dataset uses internal or external buckets. |
bucketBucketStorage | Literal[ | Underlying storage backend. |
deletebool | Whether to delete a local dataset with the same name before initialization. |
deletebool | Whether to delete the remote dataset as well. |
| Raises | |
ValueError | If the dataset exists and deletion flags are not set. |
ValueError | If the dataset is remote but no bucket is configured. |
NotImplementedError | If Azure Blob Storage is selected as the bucket storage. |
luxonis_ml.data.datasets.BaseDataset.addAdd data to the dataset from a generator of records.
| Parameters | |
generator:DatasetIterator | The generator should yield either dictionaries that can be converted to DatasetRecord objects or actual DatasetRecord instances. Each record must contain at least a file path and can optionally include an annotation and a task name. For example: def record_generator(): yield { "file": f"/path/to/image.jpg", "task_name": "animals", "annotation": { "instance_id": 1, "class_name": "cat", "boundingbox": { "x": 10, "y": 20, "w": 100, "h": 150, } "keypoints": { "keypoints": [ (15, 25, 1), (50, 60, 1), (70, 80, 0), ], }, "instance_segmentation": { "mask": "/path/to/mask.png", } }, } |
batchint | The number of records to process in a batch before writing to storage. Larger batch sizes may be more efficient but will use more memory. |
| Returns | |
Self | Undocumented |
| Raises | |
ValueError | If the records yielded by the generator are not in the expected format. |
ValueError | If the dataset contains metadata annotations with conflicting types. |
str, push_to_cloud: bool = True, splits_to_clone: list[ str] | None = None, team_id: str | None = None) -> LuxonisDataset:
¶
Create a local copy of the current dataset.
Warning
The cloned dataset overwrites any existing dataset with the same name.
| Parameters | |
newstr | Name of the cloned dataset. |
pushbool | Whether to push the cloned dataset to the cloud when the current dataset is remote. |
splitslist[ | Optional split names to clone. If omitted, all data is cloned. |
teamstr | None | Optional team identifier for the cloned dataset. |
| Returns | |
LuxonisDataset | Cloned dataset handle. |
| Raises | |
FileNotFoundError | If the current dataset is empty and splits_to_clone is specified. |
Delete the dataset from local storage and optionally the cloud.
| Parameters | |
deletebool | Whether to delete the remote dataset. |
deletebool | Whether to delete the local dataset files. |
| Raises | |
ValueError | If neither delete_remote nor delete_local is set to True. |
PathType, dataset_type: DatasetType = DatasetType.NATIVE, max_partition_size_gb: float | None = None, zip_output: bool = False) -> Path | list[ Path]:
¶
Export the dataset into one of the supported formats.
| Parameters | |
outputPathType | Directory where the dataset should be exported. |
datasetDatasetType | Export format. |
maxfloat | None | Optional maximum partition size. If the dataset exceeds this size, it is split into partitions named {dataset_name}_part{partition_number}. |
zipbool | Whether to zip the exported dataset or each partition after export. |
| Returns | |
Path | list[ | Export directory, or ZIP archive paths when zip_output is enabled. |
| Raises | |
NotImplementedError | If the specified export format is not supported. |
ValueError | If the output path already exists. |
Get the categorical encodings for the dataset grouped by task.
Example output:
{
"vehicles": {
"color": {"red": 0, "green": 1, "blue": 2},
"brand": {"audi": 0, "bmw": 1, "mercedes": 2},
}
}Get class names and IDs per task.
| Returns | |
Mapping from class names to class IDs grouped by task name |
{
"color": {"red": 0, "green": 1, "blue": 2},
"brand": {"audi": 0, "bmw": 1, "mercedes": 2},
} |
Get the metadata types for each metadata annotation in the dataset.
Example output:
{
"id": "int",
"time_of_day": "Category",
"temperature": "float",
}Return keypoint skeletons for each task.
| Returns | |
dict[ | Keypoint labels and edges keyed by task name. |
Get the dataset splits definitions.
| Returns | |
dict[ | A mapping of split names to list of UUIDs, or None if no splits are defined. |
int | None = None, view: str | None = None) -> dict[ str, Any]:
¶
Return dataset statistics for a view or the full dataset.
The returned statistics include:
- "duplicates": Analysis of duplicated content.
- "class_distributions": Class frequencies organized by task name and task type. Classification tasks are excluded.
- "missing_annotations": File paths that lack annotations.
- "heatmaps": Spatial annotation distributions.
| Parameters | |
sampleint | None | Optional number of samples used for heatmap generation. |
view:str | None | Optional split name to analyze. If omitted, the entire dataset is analyzed. |
| Returns | |
dict[ | Dataset statistics. |
@override
Mapping[ str, Sequence[ PathType]] | Mapping[ str, float] | tuple[ float, float, float] | None = None, *, ratios: dict[ str, float] | tuple[ float, float, float] | None = None, definitions: dict[ str, list[ PathType]] | None = None, replace_old_splits: bool = False):
¶
Create dataset splits for training, validation, and testing.
Note
Although "train", "val", and "test" are the conventional split names, you can use any split names you want by providing a mapping to the splits argument. This can be useful for combining records from multiple sources ("train_real", "train_synth") or for creating fully custom splits.
| Parameters | |
splits:Mapping[ | A mapping defining the splits. Can be one of the following:
|
ratios:dict[ | A mapping of split names to float ratios or a tuple of three float ratios for train, val, and test splits.
Deprecated since version 0.4.0: Use splits instead.
|
definitions:dict[ | A mapping of split names to lists of file paths.
Deprecated since version 0.4.0: Use splits instead.
|
replacebool | Whether to replace old splits with new ones. If False` (default), new splits will be added to old splits, and duplicate group IDs will be filtered out. If ``True, old splits will be replaced with new splits. |
| Raises | |
ValueError | If both ratios and definitions are provided. |
ValueError | If neither splits, ratios, nor definitions is provided. |
ValueError | If both splits and ratios/definitions are provided. |
ValueError | If splits is provided but is empty. |
ValueError | If ratios is provided but does not sum to 1. |
ValueError | If definitions is provided but the total number of files in definitions exceeds the dataset size. |
ValueError | If definitions are provided but all of them are already included in old splits, resulting in no new files to add to splits while replace_old_splits is False. |
FileNotFoundError | If the dataset is empty. |
TypeError | If the splits definitions are not in the expected format. |
LuxonisDataset, inplace: bool = True, new_dataset_name: str | None = None, splits_to_merge: list[ str] | None = None, team_id: str | None = None) -> LuxonisDataset:
¶
Merge another dataset into this or a new dataset.
| Parameters | |
other:LuxonisDataset | Dataset to merge into this dataset. |
inplace:bool | Whether to merge into this dataset. If False, a new dataset is created. |
newstr | None | Name of the new dataset when inplace is False. |
splitslist[ | Optional split names to merge. |
teamstr | None | Optional team identifier for a newly created dataset. |
| Returns | |
LuxonisDataset | Dataset containing the merged data. |
| Raises | |
ValueError | If the datasets have different bucket storage types. |
ValueError | If inplace is False but no name for the new dataset is provided. |
Synchronize the dataset from a remote bucket to a local storage.
Annotations and metadata are always pulled. Media files are pulled either when missing locally or always, depending on update_mode.
| Parameters | |
updateUpdateMode | Media synchronization mode. |
BucketStorage | None, update_mode: UpdateMode = UpdateMode.MISSING):
¶
Push the local dataset to a remote bucket.
Annotations and metadata are always pushed. Media files are pushed either when missing remotely or always, depending on update_mode.
| Parameters | |
bucketBucketStorage | None | Remote storage backend to push to. If unset, uses the dataset's current bucket storage type. |
updateUpdateMode | Media synchronization mode. |
| Raises | |
ValueError | If the dataset is empty or not initialized. |
FileNotFoundError | If any media files are missing locally when attempting to push. |
Remove duplicate files and annotations from the dataset.
| Raises | |
FileNotFoundError | If the dataset is empty. |
Set class order for specific tasks.
| Parameters | |
classdict[ | Mapping from task names to class names in the desired order. |
| Raises | |
ValueError | If a task is missing or the provided class names do not match the dataset classes for that task. |
list[ str] | dict[ str, int], task: str | None = None, rewrite_metadata: bool = True):
¶
Set classes for one or more tasks.
| Parameters | |
classes:list[ | Class names, or class IDs keyed by class name. If class names are provided, IDs are assigned alphabetically starting from 0. A class named "background" is always assigned ID 0. |
task:str | None | Optional task to update. If omitted, all tasks are updated. |
rewritebool | Undocumented |
list[ str] | None = None, edges: list[ tuple[ int, int]] | None = None, task: str | None = None):
¶
Set keypoint skeleton metadata.
| Parameters | |
labels:list[ | Optional keypoint names. |
edges:list[ | Optional keypoint edges as 0-based index pairs. |
task:str | None | Optional task to update. If omitted, all tasks are updated. |
| Raises | |
ValueError | If neither labels nor edges is provided. |
Get the source information for the dataset.
| Returns | |
| Dataset source metadata. |
| Raises | |
ValueError | If source metadata is missing. |
BucketStorage, bucket: str, team_id: str, dataset_name: str) -> str:
¶
Undocumented
list[ DatasetRecord], pfm: ParquetFileManager, index: pl.DataFrame | None):
¶
Undocumented
Literal[ False] = ..., raise_when_empty: Literal[ False] = ...) -> pl.DataFrame | None:Literal[ False] = ..., raise_when_empty: Literal[ True] = ...) -> pl.DataFrame:Literal[ True] = ..., raise_when_empty: Literal[ False] = ...) -> pl.LazyFrame | None:Literal[ True] = ..., raise_when_empty: Literal[ True] = ...) -> pl.LazyFrame:Load unique file entries from annotation data.
| Parameters | |
lazy:bool | Whether to return a LazyFrame that can be further processed before collecting. |
raisebool | Whether to raise an error if the dataset is empty. If False, returns None when the dataset is empty. |
| Returns | |
pl.DataFrame | pl.LazyFrame | None | DataFrame or LazyFrame containing the dataset records. |
| Raises | |
FileNotFoundError | If the dataset is empty and raise_when_empty is True. |
PathLike[ str] | str, names: list[ str], uuids_to_keep: set[ str]) -> set[ str]:
¶
Undocumented
Literal[ False] = ..., raise_when_empty: Literal[ False] = ..., attempt_migration: bool = ...) -> pl.DataFrame | None:Literal[ False] = ..., raise_when_empty: Literal[ True] = ..., attempt_migration: bool = ...) -> pl.DataFrame:Literal[ True] = ..., raise_when_empty: Literal[ False] = ..., attempt_migration: bool = ...) -> pl.LazyFrame | None:Literal[ True] = ..., raise_when_empty: Literal[ True] = ..., attempt_migration: bool = ...) -> pl.LazyFrame:Load the dataset DataFrame from local storage.
| Parameters | |
lazy:bool | Whether to return a LazyFrame that can be further processed before collecting. |
raisebool | Whether to raise an error if the dataset is empty. If False, returns None when the dataset is empty. |
attemptbool | Whether to attempt internal migration of the
DataFrame if the LDF version of the dataset does not match
the supported LDF version of the current luxonis-ml
installation. If True, will perform internal migration
using migrate_dataframe. If False, will return the
DataFrame as-is without migration, which may lead to errors
if the LDF versions do not match. |
| Returns | |
pl.DataFrame | pl.LazyFrame | None | The dataset annotations as a Polars DataFrame or LazyFrame, or None if the dataset is empty and raise_when_empty is False. |
| Raises | |
FileNotFoundError | If the dataset is empty and raise_when_empty is True. |