We’re designing annotation at Landvex for a document-review pipeline. The review itself has six separate answers: integrity, origin, authority, validity, facts, and linkage. The boxes we want people to draw are more boring than that: stamp, seal, signature, table, photo zone, that kind of object.
I don’t want annotators to start drawing a “forged area” class, because that quietly trains a genuine/fake detector. But I also need to know whether two annotators even agree on what they saw.
What I’m unsure about:
- For document pages, do you treat “this signature looks overwritten” as an attribute on the signature box, a separate class, or something you keep out of the visual labels entirely and handle in a later review step?
- How do you measure agreement when the task is “find these objects” plus a later dimensional review? Pixel IoU on boxes feels incomplete, and a single accuracy number would hide the part I care about.
- If two reviewers disagree on a dimension (say validity is unclear vs a deviation) but they boxed the same stamp, is that agreement or not? I keep going in circles here.
Genuinely looking for how other teams have set this up without the labels collapsing into authentic vs forged.