How can I restore the labeling from individual files stored in Target Storage?

Label Studio Support
1

Hello, I annotate pictures for the detection task in Label Studio v1.13.1.

I have connected the S3 storages to this task:

  • The Source Cloud Storage which stores the original images.
    I activate option “Treat every bucket object as a source file” for this storage.

  • Target Cloud Storage which stores the annotations in some Labels Studios JSON format files without extensions.
    Here, I have a number of separate markup files for each image.

The data is stored in S3 repositories and I wanted to try to test the possibility of restoring my labeling project in Label Studio.

I copy my pictures from Source Cloud Storage to new bucket.
Then I copy annotations files from Target Cloud Storage to other new bucket.
After that, I successfully connected the new buckets to my new project as The Source Cloud Storage and Target Cloud Storage.
I have successfully synchronized the project with these storages.

All images were successfully uploaded to the Label studio project, but the labeling from the files in the Target Source was not.

I also tried downloading labeling files from Target Source, manually adding the json extension to them and uploading them via the Data manager UI.
And I got this Error:

Runtime error
Validation error

Error at item 0: "image" key is expected in task data [assume: item["data"] = task root with values] :: {'data': {'id': 8, 'result': [{'id': '4e00d354f6', 'type': 'rectanglelabels', 'value': {'x': 19.366741617267444, 'y': 80.85469948080951, 'width': 0.9425660448662186, 'height': 0.3924094337491686, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'f3edb13034', 'type': 'rectanglelabels', 'value': {'x': 20.06262438930789, 'y': 81.64151520373808, 'width': 1.1654671308440148, 'height': 0.3812244463319253, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '501e513da8', 'type': 'rectanglelabels', 'value': {'x': 22.304097238813934, 'y': 78.55719830195243, 'width': 1.1513524896814302, 'height': 0.4116285671843882, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '3ffc726279', 'type': 'rectanglelabels', 'value': {'x': 24.622000909810808, 'y': 75.92536788377356, 'width': 1.1900473507297453, 'height': 0.3822590362445197, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '68a1673a68', 'type': 'rectanglelabels', 'value': {'x': 22.44054489151523, 'y': 75.03936918536088, 'width': 0.9989273942466875, 'height': 0.35552660539308123, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'e8d2980230', 'type': 'rectanglelabels', 'value': {'x': 24.429020258341122, 'y': 74.62957454057182, 'width': 1.801152747050434, 'height': 0.5361344957497208, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'a415be122e', 'type': 'rectanglelabels', 'value': {'x': 21.272408173586914, 'y': 73.92547283270952, 'width': 1.2529958013281945, 'height': 0.4255061390738936, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'e8bb275c92', 'type': 'rectanglelabels', 'value': {'x': 26.22421987415616, 'y': 68.59716447379685, 'width': 1.1581876957547104, 'height': 0.41249557895763034, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'e45dcc8d61', 'type': 'rectanglelabels', 'value': {'x': 35.01015815982987, 'y': 64.67310257261903, 'width': 1.1558576119777013, 'height': 0.3966812161543487, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'd7f71ba2b9', 'type': 'rectanglelabels', 'value': {'x': 34.49267059439376, 'y': 62.91367669865887, 'width': 1.4707882508439951, 'height': 0.4561192568070211, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'c5bc9b9b3c', 'type': 'rectanglelabels', 'value': {'x': 33.55213833681982, 'y': 60.13401363524057, 'width': 1.4922182927119965, 'height': 0.43117523495037546, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '5ec3352aa0', 'type': 'rectanglelabels', 'value': {'x': 37.47102559287029, 'y': 59.99603240902801, 'width': 1.0649568093608044, 'height': 0.34595622216297983, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'e0da0852db', 'type': 'rectanglelabels', 'value': {'x': 36.07928721051732, 'y': 59.673670002121284, 'width': 0.5660765776444101, 'height': 0.14075555190530475, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '3c6231f709', 'type': 'rectanglelabels', 'value': {'x': 34.91414392632331, 'y': 59.284642529873075, 'width': 1.0063911567571087, 'height': 0.43302928846061656, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '9d5b5c2aec', 'type': 'rectanglelabels', 'value': {'x': 38.547511999614656, 'y': 57.96990300346268, 'width': 1.1469899213038228, 'height': 0.42520345794626635, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'a546c8c6ae', 'type': 'rectanglelabels', 'value': {'x': 40.737892752556135, 'y': 56.84037479922473, 'width': 1.6057858898253694, 'height': 0.5686770173760395, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '9382486011', 'type': 'rectanglelabels', 'value': {'x': 38.16271538085472, 'y': 53.895253915657925, 'width': 1.0803905065184318, 'height': 0.39390013588897277, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '555d2d167e', 'type': 'rectanglelabels', 'value': {'x': 37.797872456031016, 'y': 52.69030336338992, 'width': 1.2064507061046221, 'height': 0.45469388412953055, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '9694eab228', 'type': 'rectanglelabels', 'value': {'x': 41.352866724966916, 'y': 53.00501011761678, 'width': 1.1978871249683605, 'height': 0.40158884603356937, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '2bde94e1bc', 'type': 'rectanglelabels', 'value': {'x': 41.14250590503634, 'y': 50.99575308134218, 'width': 1.3565620842835742, 'height': 0.4527339966979574, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'b1f7360444', 'type': 'rectanglelabels', 'value': {'x': 41.626198123612845, 'y': 50.64493323108709, 'width': 1.034100605232618, 'height': 0.28222379014935745, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '037e39556c', 'type': 'rectanglelabels', 'value': {'x': 45.18133037046969, 'y': 47.75435993451618, 'width': 1.6011880339084918, 'height': 0.3900175988868527, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '588afe74c7', 'type': 'rectanglelabels', 'value': {'x': 45.64834354702631, 'y': 47.299666050386726, 'width': 1.2620475128376112, 'height': 0.3684588371394436, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '673e1f61fa', 'type': 'rectanglelabels', 'value': {'x': 46.020842152136936, 'y': 47.07819877061675, 'width': 0.950705395133205, 'height': 0.21362773004362104, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '7d65aa1f28', 'type': 'rectanglelabels', 'value': {'x': 47.884044996264755, 'y': 42.61201568943787, 'width': 1.2071733392799429, 'height': 0.400517942444323, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '5ad3776f33', 'type': 'rectanglelabels', 'value': {'x': 49.78722568267885, 'y': 40.404344395660786, 'width': 1.314106340960299, 'height': 0.43409076997252083, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '48b614c8ec', 'type': 'rectanglelabels', 'value': {'x': 51.93298673592515, 'y': 39.9686338840093, 'width': 1.038419696003563, 'height': 0.42437231989861035, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '470a2f21f7', 'type': 'rectanglelabels', 'value': {'x': 51.413776887923405, 'y': 39.395245329642485, 'width': 1.0981518024108021, 'height': 0.38549851960251397, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'f7c2595519', 'type': 'rectanglelabels', 'value': {'x': 53.83556685569085, 'y': 36.032447908274726, 'width': 1.579146123911112, 'height': 0.5365911122672298, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '5bacd73efe', 'type': 'rectanglelabels', 'value': {'x': 61.4545399071386, 'y': 31.35520987086526, 'width': 1.3512281266456583, 'height': 0.44763750542078634, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '1749070e65', 'type': 'rectanglelabels', 'value': {'x': 61.34872083697962, 'y': 30.3968064938746, 'width': 1.1802896286964923, 'height': 0.5538079393987978, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'b8fd4234ea', 'type': 'rectanglelabels', 'value': {'x': 64.49887315632823, 'y': 26.121294422868345, 'width': 1.2128493425915674, 'height': 0.4533764477979744, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '0b867ca134', 'type': 'rectanglelabels', 'value': {'x': 65.27216636133628, 'y': 24.7209924828341, 'width': 0.9849313453260171, 'height': 0.34146707144277555, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '666f12c37d', 'type': 'rectanglelabels', 'value': {'x': 68.324541348975, 'y': 24.41599887455377, 'width': 1.180387818721174, 'height': 0.41116404225836933, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '9694a9a873', 'type': 'rectanglelabels', 'value': {'x': 69.43465028187146, 'y': 22.498065774143722, 'width': 1.5338047041481255, 'height': 0.7375252393826425, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '072f3a7e28', 'type': 'rectanglelabels', 'value': {'x': 70.50354799282135, 'y': 21.173715475297588, 'width': 1.1617897912560802, 'height': 0.4429179733851639, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '3b6eb0ba1a', 'type': 'rectanglelabels', 'value': {'x': 70.23850693753387, 'y': 20.556915803058786, 'width': 1.4611392182707605, 'height': 0.4502881867625536, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '99d9466c79', 'type': 'rectanglelabels', 'value': {'x': 70.59682891107317, 'y': 17.84479953356051, 'width': 1.4808967611590589, 'height': 0.4115763606344523, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'cd5ba2aecd', 'type': 'rectanglelabels', 'value': {'x': 72.16364800991086, 'y': 17.2229807029915, 'width': 1.2787265548576756, 'height': 0.5113524480610296, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '8d1e530f52', 'type': 'rectanglelabels', 'value': {'x': 75.04431980377444, 'y': 14.996932685375175, 'width': 1.0080202895987842, 'height': 0.4137582050043116, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '48a82be51b', 'type': 'rectanglelabels', 'value': {'x': 74.98226124866747, 'y': 13.886331081263178, 'width': 1.1152777909611868, 'height': 0.40346292731509154, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'db7f699484', 'type': 'rectanglelabels', 'value': {'x': 82.99833850041189, 'y': 7.366319209897662, 'width': 1.0154458089238474, 'height': 0.39035774463951767, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': '30366ed6c1', 'type': 'rectanglelabels', 'value': {'x': 83.74269244179499, 'y': 7.056948549208237, 'width': 1.1119361346587218, 'height': 0.39359722799753694, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'f44ecae8ad', 'type': 'rectanglelabels', 'value': {'x': 83.88405090977301, 'y': 5.976246623138925, 'width': 1.0321851595547316, 'height': 0.37238238450387834, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}, {'id': 'bcb5b463a0', 'type': 'rectanglelabels', 'value': {'x': 86.35857333659523, 'y': 4.303533112170379, 'width': 1.2787265548575304, 'height': 0.447210677572502, 'rotation': 0, 'rectanglelabels': ['scallop']}, 'to_name': 'image', 'from_name': 'label_rectangles', 'image_rotation': 0, 'original_width': 6125, 'original_height': 17375}], 'created_username': 'Anton Shiryaev med_phisiker@mail.ru, 1', 'created_ago': '8\xa0minutes', 'completed_by': {'id': 1, 'first_name': 'Anton', 'last_name': 'Shiryaev', 'email': 'med_phisiker@mail.ru'}, 'task': {'id': 53, 'data': {'image': '/data/local-files/?d=/Zone_1_10_1.png'}, 'meta': {}, 'created_at': '2024-10-10T17:33:02.181050Z', 'updated_at': '2024-10-10T17:33:02.181064Z', 'is_labeled': True, 'overlap': 1, 'inner_id': 7, 'total_annotations': 1, 'cancelled_annotations': 0, 'total_predictions': 0, 'comment_count': 0, 'unresolved_comment_count': 0, 'last_comment_updated_at': None, 'project': 5, 'updated_by': None, 'file_upload': 13, 'comment_authors': []}, 'was_cancelled': False, 'ground_truth': False, 'created_at': '2024-10-10T17:33:02.182563Z', 'updated_at': '2024-10-10T17:33:02.182575Z', 'draft_created_at': None, 'lead_time': None, 'import_id': None, 'last_action': None, 'project': 5, 'updated_by': None, 'parent_prediction': None, 'parent_annotation': None, 'last_created_by': None}, 'file_upload_id': 26}

I wanted to ask how to restore the labeling for a project from files that were saved in Target Source?

2

Hello,

I understand that you’re trying to restore your labeling project in Label Studio using the files saved in your Target Cloud Storage, but the annotations are not being imported as expected. The error you’re encountering indicates that the imported JSON files are missing the required "image" key in the "data" field.

Explanation:

By default, Label Studio’s Target Cloud Storage saves annotations in a format that contains only the annotation results, without including the task data such as the "data" field required for importing tasks. This means that the files in your Target Storage lack the information about which image each annotation corresponds to.

When you try to import these annotation files directly into a new Label Studio project, the importer expects each item to have a "data" field containing the input data (e.g., the image path), but it only finds the annotation results.

Solution:

To restore your labeling project using the annotation files from your Target Cloud Storage, you need to reconstruct the tasks by combining the images from your Source Cloud Storage with the annotations from the Target Cloud Storage.

Here’s how you can do this:

1. Create a New Project and Connect Source Storage

  • Create a New Project:
    • In Label Studio, create a new project.
  • Connect Source Storage:
    • Go to the project settings and add your new bucket (where you copied your images) as Source Cloud Storage.
    • Ensure that the option “Treat every bucket object as a source file” is enabled for this storage.
    • Click “Sync Storage” to import all your images as tasks.

2. Prepare Annotations for Import

The annotation files in your Target Cloud Storage need to be transformed into a format suitable for importing into Label Studio. Since the files currently only contain annotation results without task data, you need to add the "data" field with the image path to each annotation.

3. Create a Script to Process Annotation Files

You’ll need to write a script to process the annotation files and include the necessary task data.

Here are the steps:

  • Read Annotation Files:
    • For each annotation file in your Target Cloud Storage:
      • Read the contents of the file.
  • Add Task Data:
    • Create a new JSON object for each annotation file with the following structure:
{
  "data": {
    "image": "<your image path>"
  },
  "annotations": [
    {
      "result": [ ... ] // the annotation results from your file
    }
  ]
}
  • Replace <your image path> with the path to the corresponding image in your Source Cloud Storage. This should match the path used in the tasks created during the Source Storage sync, e.g., s3://your-new-source-bucket/image1.jpg.
  • Copy the "result" field from your annotation file into the "annotations" array.
  • Example Python Script: Here’s a simple example of how you might write this script in Python:
import os
import json

# Directory where your annotation files are stored
annotations_dir = 'path/to/your/annotations'

# Base path for images in your Source Cloud Storage
image_base_path = 's3://your-new-source-bucket/'

# List to hold the new task data
tasks = []

# Iterate over each annotation file
for filename in os.listdir(annotations_dir):
    if not filename.endswith('.json'):
        continue  # Skip files without .json extension

    file_path = os.path.join(annotations_dir, filename)
    with open(file_path, 'r') as f:
        annotation = json.load(f)

    # Extract the image filename from the annotation filename or content
    image_filename = filename.replace('.json', '') + '.jpg'  # Adjust if your images have different extensions

    # Construct the full image path
    image_path = image_base_path + image_filename

    # Create the new task JSON
    task = {
        "data": {
            "image": image_path
        },
        "annotations": [
            {
                "result": annotation['result']  # Assuming 'result' key exists
            }
        ]
    }

    tasks.append(task)

# Save the tasks to a new JSON file
with open('tasks_with_annotations.json', 'w') as f:
    json.dump(tasks, f)

Note: Adjust the script based on the actual structure of your annotation files and image paths.

4. Import the Prepared Annotations

  • Import Tasks:
    • In Label Studio, go to the Data Manager of your project.
    • Click “Import” and select the JSON file (tasks_with_annotations.json) you prepared.
    • This will import the tasks along with their annotations into your project.

5. Verify the Annotations

  • Check Tasks:
    • In the Data Manager, verify that the tasks now have the annotations applied.
    • Open a task to ensure that the annotations are correctly displayed on the images.

Important Notes:

  • Match Image Paths:
    • Ensure that the "image" paths in your tasks match the paths used in the tasks created from the Source Storage sync. Consistency is crucial for Label Studio to associate annotations with the correct images.
  • File Extensions:
    • Add the .json extension to your annotation files if they don’t have it. This helps in processing and importing the files.
  • Data Structure:
    • The error you encountered indicates that the importer expects a specific data structure. By adding the "data" field with the "image" key, you comply with the expected format.

Alternative Solution for Future Projects:

In future projects, you can configure Label Studio to save annotations in a format that is directly importable, simplifying the restoration process.

  • Enable Task Format for Target Storage:
    • Start Label Studio with the environment variable FUTURE_SAVE_TASK_TO_STORAGE=1:
FUTURE_SAVE_TASK_TO_STORAGE=1 label-studio

This changes the behavior of the Target Storage, causing it to save annotations in the Label Studio task format, which includes both the "data" and "annotations" fields.

  • Direct Import:
    • With this setting enabled, the annotation files saved in your Target Storage will be in a format suitable for direct import into Label Studio without additional processing.

Additional Resources:

Conclusion:

By processing your existing annotation files to include the necessary task data, you can restore your labeling project in Label Studio using the annotations saved in your Target Cloud Storage.

3

Hello @makseq

I was trying to do the same thing and stumbled upon this discussion.

I added the env variable to my docker run command as such:
docker run --rm -p 80:8080 -v /home/ec2-user/mydata:/label-studio/data heartexlabs/label-studio:latest -e FUTURE_SAVE_TASK_TO_STORAGE=1

The saved annotations don’t seem to differ though. Before and after adding the env variable, the format is the same and seems to match what you were recommending so I was wondering if something had been implemented in a recent version of LS to include that (I am on 1.14.0).

Here is the format I get:

{
    "id": 5,
    "result": [
        {
            "original_length": 3,
            "value": {
                "start": 0.20736288504883546,
                "end": 2.129977460555973,
                "channel": 0,
                "labels": [
                    "Siren"
                ]
            },
            "id": "smw56",
            "from_name": "label",
            "to_name": "audio",
            "type": "labels",
            "origin": "manual"
        }
    ],
    "created_username": " antoine.purier@microdb.fr, 1",
    "created_ago": "0\u00a0minutes",
    "completed_by": {
        "id": 1,
        "first_name": "",
        "last_name": "",
        "email": "antoine.purier@microdb.fr"
    },
    "task": {
        "id": 21859,
        "data": {
            "audio": "s3://dbflash/audio/ref_mic/Paris17_2022_04_12_09_39_02_1649749066_ref_mic.wav"
        },
        "meta": {},
        "created_at": "2024-12-06T14:53:45.190681Z",
        "updated_at": "2024-12-11T08:30:27.196447Z",
        "is_labeled": true,
        "overlap": 1,
        "inner_id": 10926,
        "total_annotations": 1,
        "cancelled_annotations": 0,
        "total_predictions": 0,
        "comment_count": 0,
        "unresolved_comment_count": 0,
        "last_comment_updated_at": null,
        "project": 2,
        "updated_by": 1,
        "file_upload": null,
        "comment_authors": []
    },
    "was_cancelled": false,
    "ground_truth": false,
    "created_at": "2024-12-11T09:10:48.736560Z",
    "updated_at": "2024-12-11T09:10:48.736595Z",
    "draft_created_at": null,
    "lead_time": 5.257,
    "import_id": null,
    "last_action": null,
    "project": 2,
    "updated_by": 1,
    "parent_prediction": null,
    "parent_annotation": null,
    "last_created_by": null
}

I then tried to import it add this file (+ json extension) to my target source storage. It imports without any error but I can’t see the audio or annotation.
I tried to import it via the import button but I get the following error:

image
image580×425 99.4 KB

Note: My source storage is dbflash/tasks and the referenced audio is in another bucket dbflash/audio/ref_mic. But I tried with a more standard json task file without annotations that was in this dbflash/tasks bucket, referencing audio in the dbflash/audio/ref_mic and it works. It is just with this annotated task file that it is not working.

  • Is this format correct or does it need further processing?
  • Do I still need to include the env variable at launch or has something been implemented in the latest LS version?
  • What is the proper way to import such an annotation file?
4

The format you got is incorrect, it’s annotation json, not task json.
Seems your env var FUTURE_SAVE_TASK_TO_STORAGE=1 didn’t work for your LS deployment. Try to go on the /version page as superadmin (it’s django superuser; usually, the first user who you create in label studio) and check the value of this variable on that page.