Heard some good reviews about how good Claude 3.5 Sonnet is. So I bought some credits and tried it out.
Just some simple tests, I will feed the API with a picture and a question, with these settings:
"model": "claude-3-5-sonnet-20240620"
"max_tokens": 3000
Test 1
<div class="max-w-sm mx-auto bg-orange-50 rounded-lg shadow-md p-6">
<h1 class="text-2xl font-bold mb-4">RAG without knowing RAG?</h1>
<p class="text-gray-600 mb-4">2024-06-25 00:08:37</p>
<div class="flex flex-wrap gap-2">
<span class="bg-orange-200 text-orange-800 px-3 py-1 rounded-full text-sm font-semibold">#rag</span>
<span class="bg-orange-200 text-orange-800 px-3 py-1 rounded-full text-sm font-semibold">#notebooklm</span>
<span class="bg-orange-200 text-orange-800 px-3 py-1 rounded-full text-sm font-semibold">#Google</span>
</div>
</div>
Which is equivalent to the following. Not too bad?
Test2
Could you count the quantity of the plates by their color?
And we got this response.
Certainly! I'll count the plates by color based on what I can see in the image:
1. Blue: 2 plates at the top of the left stack
2. Red: 4 plates below the blue ones in the left stack
3. Orange/Peach: 4 plates at the bottom of the left stack
4. Purple: 9 plates in the right stack
There's also 1 plate at the very top of the right stack that appears to be a different color, possibly white or light purple.
In total, there are at least 20 plates visible in the stacks, with the majority being purple. The colors create a vibrant and eye-catching display of dinnerware.
Is this accurate? 🤠Not at all. But let's try some other tests when I have time.
Sample API response
{
"id": "msg_015Bw5W2o8EfweNH4e5jFrYR",
"type": "message",
"role": "assistant",
"model": "claude-3-5-sonnet-20240620",
"content": [
{
"type": "text",
"text": "Certainly! I'll count the plates by color based on what I can see in the image:\n\n1. Blue: 2 plates at the top of the left stack\n2. Red: 4 plates below the blue ones in the left stack\n3. Orange/Peach: 4 plates at the bottom of the left stack\n4. Purple: 9 plates in the right stack\n\nThere's also 1 plate at the very top of the right stack that appears to be a different color, possibly white or light purple.\n\nIn total, there are at least 20 plates visible in the stacks, with the majority being purple. The colors create a vibrant and eye-catching display of dinnerware."
}
],
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 1031,
"output_tokens": 157
}
}