# Measuring annotator agreement when the boxes are document objects, not genuine or fake **URL:** \ **Category:** Machine Learning\ **Created:** [November 8, 2026, 6:29am UTC](https://community.labelstud.io/t/measuring-annotator-agreement-when-the-boxes-are-document-objects-not-genuine-or-fake/771 "2026-11-08T06:29:09Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![wolfoftyresodebug](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.labelstud.io/wolfoftyresodebug/32/601_2.png) [@wolfoftyresodebug](https://community.labelstud.io/u/wolfoftyresodebug)\ **Post date:** [November 8, 2026, 6:29am UTC](https://community.labelstud.io/t/measuring-annotator-agreement-when-the-boxes-are-document-objects-not-genuine-or-fake/771/1 "2026-11-08T06:29:09Z")
We’re designing annotation at Landvex for a document-review pipeline. The review itself has six separate answers: integrity, origin, authority, validity, facts, and linkage. The _boxes_ we want people to draw are more boring than that: stamp, seal, signature, table, photo zone, that kind of object. I don’t want annotators to start drawing a “forged area” class, because that quietly trains a genuine/fake detector. But I also need to know whether two annotators even agree on what they saw. What I’m unsure about: - For document pages, do you treat “this signature looks overwritten” as an attribute on the signature box, a separate class, or something you keep out of the visual labels entirely and handle in a later review step? - How do you measure agreement when the task is “find these objects” plus a later dimensional review? Pixel IoU on boxes feels incomplete, and a single accuracy number would hide the part I care about. - If two reviewers disagree on a dimension (say validity is unclear vs a deviation) but they boxed the same stamp, is that agreement or not? I keep going in circles here. Genuinely looking for how other teams have set this up without the labels collapsing into authentic vs forged.