Hello, I apologize for the long answer.
Why you’re seeing segmentation = annotation['segmentation'][0] fail
That traceback means the converter is iterating over COCO annotations, sees a "segmentation" field, and assumes it’s a polygon list ([[x1,y1,x2,y2,...], ...]). But in COCO, segmentation can be either:
- Polygons (list of lists), or
- RLE masks (a dict, commonly when
iscrowd=1)
In the converter code, RLE is detected when annotation.get("iscrowd") == 1, and it explicitly logs that RLE is not supported for COCO import into Label Studio: the code flags RLE and prints an error ("RLE in segmentation is not yet supported in COCO") while iterating annotations.
Source: label-studio-sdk converter COCO import code
This matches what people hit in the field: COCO mixes polygon + RLE segmentations, and the converter only handles polygon segmentation reliably; when it hits RLE/different shapes, you can get failures like KeyError: 0 / indexing errors.
Source: GitHub issue #8707
Are you using the right COCO file?
Yes: instances_val2017.json is the right type of COCO file for object detection/segmentation import.
But if you instead try files like captions_train2017.json, those aren’t instance annotations and won’t have categories/annotations in the expected structure—leading to errors like KeyError: 'categories'.
Source: GitHub issue #8707
What to do next (practical paths)
1) If you only need bounding boxes (YOLO training, etc.)
Avoid segmentation conversion entirely: create a COCO file that includes only bbox annotations (or ignore segmentation), then import. The converter supports bbox conversion via create_bbox(...).
You can also keep only annotations where "bbox" exists and drop "segmentation" to avoid hitting the segmentation path. (You’d do this with a small preprocessing script.)
This is usually the simplest if your goal is “get COCO into LS to edit boxes”.
2) If you need segmentations, you must avoid RLE entries
Filter your COCO annotations to only polygon segmentations:
- Keep only annotations where
iscrowd == 0 (polygon segmentations are generally there; RLE often correlates with iscrowd==1)
- And ensure
annotation["segmentation"] is a non-empty list of lists
The failure you saw (annotation['segmentation'][0]) is exactly what happens when segmentation is not a list (RLE dict) or is empty.
This is consistent with another common failure mode when segmentation lists are empty.
Source: GitHub issue #5898
3) If your source is RLE masks, use LS “Brush” RLE workflows instead of COCO-import
Label Studio supports importing brush segmentation pre-annotations in RLE format via predictions workflows (this is often the correct route for mask datasets), as referenced in the support reply for the same problem.
Source: GitHub issue #8707
Two quick clarifying questions
- Do you actually need segmentations, or are bounding boxes enough for your workflow?
- Can you paste one failing annotation object (the JSON for a single entry in
annotations[] where it crashes), especially its segmentation and iscrowd fields?
With that, I can tell you exactly which filtering/preprocess rule will fix your dataset.
Further information: