GPT and Claude go to heraldry school

Thoughtbot tests how ChatGPT and Claude handle heraldry identification from photographs, starting with a confidently wrong answer and working towards what the models can actually do. A grounded read on where visual identification breaks down.

While on vacation in Europe I came across a coat of arms I didn’t recognize on the back of a statue. I pulled out my phone, snapped a picture, and asked ChatGPT to identify it for me. It gave me a hilariously wrong answer: “this isn’t a coat of arms, it’s a statue of a dog in Spain”. Unsatisfied, I started playing around with different models to see who could get it right. Some did but most didn’t. But what I found most interesting is why . Today’s thinking models output a thought trace that ...