luxonis_ml.data.utils.ParquetFileManager

class documentation

class ParquetFileManager:

Constructor: ParquetFileManager(directory, num_rows)

View In Hierarchy

Manage append-style writes across partitioned parquet files.

Rows are buffered in memory and flushed to the current parquet file. A new file is selected every num_rows writes. Filenames are zero-padded numeric counters such as 0000000000.parquet.

Method __enter__ Undocumented
Method __exit__ Undocumented
Method __init__ Manage writing rows into partitioned parquet files.
Method remove_duplicate_uuids Undocumented
Method write Write a row to the current parquet file.
Instance Variable buffer In-memory column buffer for the current file.
Instance Variable current_file Path to the current parquet file.
Instance Variable dir Directory containing parquet files.
Instance Variable num Current parquet file index.
Instance Variable num_rows Maximum rows written to one parquet file.
Instance Variable parquet_files Existing parquet files discovered in dir.
Instance Variable row_count Number of rows currently buffered or loaded from the current file.
Static Method _ensure_metadata_column Undocumented
Method _find_num Undocumented
Method _flush Write buffered data to parquet.
Method _generate_filename Undocumented
Method _initialize_data Undocumented
Method _read Undocumented
def __enter__(self) -> Self:

Undocumented

def __exit__(self, *_):

Undocumented

def __init__(self, directory: PathType, num_rows: int = 100000):

Manage writing rows into partitioned parquet files.

Example

>>> record = {
...     "file": "image.jpg",
...     "source_name": "image",
...     "task_name": "detection",
...     "class_name": "car",
...     "instance_id": 0,
...     "task_type": "boundingbox",
...     "annotation": "{}",
... }
>>> manager = ParquetFileManager("/tmp/ldf-parquet-example", num_rows=2)
>>> manager.num_rows
2
Parameters
directory:PathTypeLocal directory where parquet files are stored.
num_rows:intMaximum rows per parquet file before a new file is created.
def remove_duplicate_uuids(self, overwrite_uuids: set[str]):

Undocumented

def write(self, uuid: str, data: ParquetRecord, group_id: str):

Write a row to the current parquet file.

Parameters
uuid:strUnique row identifier.
data:ParquetRecordAnnotation row data.
group_id:strUnique identifier for the sample group.
buffer =

In-memory column buffer for the current file.

current_file =

Path to the current parquet file.

dir =

Directory containing parquet files.

num =

Current parquet file index.

num_rows =

Maximum rows written to one parquet file.

parquet_files =

Existing parquet files discovered in dir.

row_count =

Number of rows currently buffered or loaded from the current file.

@staticmethod
def _ensure_metadata_column(df: pl.DataFrame) -> pl.DataFrame:

Undocumented

def _find_num(self) -> int:

Undocumented

def _flush(self):

Write buffered data to parquet.

def _generate_filename(self, num: int) -> Path:

Undocumented

def _initialize_data(self, data: ParquetRecord):

Undocumented

def _read(self):

Undocumented