I posted this also on github #6593: check it out for a complet description with shreeshot and video.
Description
I created a video with OpenCV at 10 FPS and labeled its first 10 frames in Label Studio. When I refresh the page and open the task, the label aligns perfectly on the first frame. However, when I advance to the second frame, the video quickly skips the second frame and displays the third frame, but still shows the bounding box intended for the second frame (see the example below).
To reproduce:
Configuration
UI Labeling Configuration::
<View>
<Labels name="videoLabels" toName="video" allowEmpty="true">
<Label value="Man" background="blue"/>
<Label value="Woman" background="red"/>
<Label value="Other" background="green"/>
</Labels>
<!-- Please specify FPS carefully, it will be used for all project videos -->
<Video name="video" value="$video" framerate="$fps"/>
<VideoRectangle name="box" toName="video"/>
</View>
Task with Annotation
{
"id": 15,
"data": {
"video": "/data/upload/1/2d96aee1-20241025-145032-937345_day.mp4",
"fps": 10
},
"annotations": [
{
"id": 4,
"result": [
{
"value": {
"framesCount": 43,
"duration": 4.3,
"sequence": [
{"enabled": true, "frame": 1, "height": 1.39, "rotation": 0, "time": 0.1, "width": 3.41, "x": 51.42, "y": 50.35},
{"enabled": true, "frame": 2, "height": 1.39, "rotation": 0, "time": 0.2, "width": 3.27, "x": 50.43, "y": 50.35},
{"enabled": true, "frame": 3, "height": 1.56, "rotation": 0, "time": 0.3, "width": 3.41, "x": 49.57, "y": 50.17},
{"enabled": true, "frame": 4, "height": 1.56, "rotation": 0, "time": 0.4, "width": 3.41, "x": 48.58, "y": 50.17},
{"enabled": true, "frame": 5, "height": 1.39, "rotation": 0, "time": 0.5, "width": 3.27, "x": 47.73, "y": 50.17},
{"enabled": true, "frame": 6, "height": 1.39, "rotation": 0, "time": 0.6, "width": 3.27, "x": 46.88, "y": 50.17},
{"enabled": true, "frame": 7, "height": 1.56, "rotation": 0, "time": 0.7, "width": 3.27, "x": 45.88, "y": 50},
{"enabled": true, "frame": 8, "height": 1.56, "rotation": 0, "time": 0.8, "width": 3.41, "x": 45.03, "y": 50},
{"enabled": true, "frame": 9, "height": 1.56, "rotation": 0, "time": 0.9, "width": 3.27, "x": 44.18, "y": 50},
{"enabled": true, "frame": 10, "height": 1.56, "rotation": 0, "time": 1, "width": 3.27, "x": 43.18, "y": 50}
]
},
"id": "xNNNVLwq5Y",
"from_name": "box",
"to_name": "video",
"type": "videorectangle",
"origin": "manual"
}
],
"created_username": "di@mail.com",
"created_ago": "3 hours, 56 minutes",
"completed_by": {
"id": 1,
"email": "di@mail.com",
"initials": "di"
},
"was_cancelled": false,
"ground_truth": false,
"created_at": "2024-11-01T18:05:06.388864Z",
"updated_at": "2024-11-01T18:45:51.407308Z",
"lead_time": 2214.572,
"task": 15,
"project": 1
}
],
"predictions": []
}
Expected Behavior:
Navigating the video back and forth, the labels should always remain aligned with the images.
Actual Behavior
-
After refreshing the page on the first frame: The label aligns correctly with the displayed image.
image -
When advancing to the second frame: The second frame is quickly skipped, and the third frame is displayed, but it shows the label intended for the second frame. Note that the timeline is poinintg on the 2nd frame here
image -
When returning to the first frame: The video displays the second frame, but with the label from the first frame. From this point onward, labels are consistently one frame behind the displayed image, regardless of the selected frame. This misalignment continues if I advance further, and the actual first frame is never shown again until I refresh the page.
After this misalignment starts, the labels stay one frame behind the images until I refresh the page. However, after refreshing, the issue recurs as soon as I advance by one frame, following the same pattern.
The root of the problem seems to be in the video display, which skips a frame as described, while the label display remains correctly sequenced.
Video used
see the github issue #6593 to view and download the video
Summary
This appears to be a bug that causes video frames and labels to become misaligned, making accurate video labeling challenging. This misalignment significantly impacts labeling precision, so addressing it would improve usability and reliability for video annotation tasks.
Environment (please complete the following information):
- OS: ubuntu 22.04
- Label Studio Version: docker 1.13.1