Property Finder identified a bias in its evaluation process while assessing alternatives for its vision-AI workload on Amazon Bedrock, according to a post published by AWS. The digital real estate application found that 96.8% of its evaluation labels had been inherited from the predictions of its incumbent model rather than independently verified by human experts.

AWS says this created a circular comparison in which competing models were judged on how closely they matched the existing system. The original dataset also contained labeling issues, including character-level errors in watermark extraction and single labels for images showing multiple room types.

To create a new benchmark, Property Finder’s team independently annotated 356 UAE property images, with 256 used for prompt development and all 356 used for the final comparison. The annotations were reviewed by AWS specialists, and the team adopted multi-label classification for images containing overlapping room or quality characteristics.

The teams then iteratively refined prompts using error analysis and Micro F1 scores. AWS reports that Llama 4 Maverick 17B Instruct reached a Micro F1 score of 0.80 on the 356-image evaluation set with the final prompt, compared with 0.56 for Property Finder’s proprietary model using its existing prompt. AWS also reports an 86% lower cost after selecting the alternative model, although the source does not provide the underlying cost figures.

AWS notes that the evaluation results are optimistic because both models’ prompts were tuned using the development images included in the evaluation set.