Exam CCAR-F Topic 1 Question 38 Discussion
Actual exam question for Anthropic's CCAR-F exam
Question #: 38
Topic #: 1
Question #: 38
Topic #: 1
You are building a structured data extraction system using Claude. The system extracts information from unstructured documents, validates the output using JSON schemas, and maintains high accuracy. It must handle edge cases gracefully and integrate with downstream systems.
After implementing tool use with strict schema definitions, JSON syntax errors are eliminated, but 5% of extractions still contain empty arrays or null values for required fields such as citations and methodology.
Spot-checking reveals that the source documents contain this information, but in varied formats-inline citations versus bibliographies, and methodology sections versus details embedded in introductions.
What is the most effective way to address these failures?
After implementing tool use with strict schema definitions, JSON syntax errors are eliminated, but 5% of extractions still contain empty arrays or null values for required fields such as citations and methodology.
Spot-checking reveals that the source documents contain this information, but in varied formats-inline citations versus bibliographies, and methodology sections versus details embedded in introductions.
What is the most effective way to address these failures?
Suggested Answer: B Vote an answer
Option B targets the remaining failure mode: semantic recognition across heterogeneous document structures.
Strict schemas eliminate malformed JSON and can guarantee that tool inputs conform to declared types, but they cannot force Claude to locate evidence that appears under unfamiliar headings or in atypical sections.
Anthropic's prompting guidance says that a few relevant, diverse, structured examples are among the most reliable ways to improve accuracy and consistency. Examples should therefore show inline citations, reference lists, numbered bibliographies, methodology sections, and methods embedded in introductions, each paired with the correct extracted structure. This teaches the intended evidence-location and granularity rules rather than merely repeating the same request. Option A retries an unchanged prompt and can reproduce the same omission. Option C introduces brittle regex rules that may miss nonstandard citations and mistake keyword mentions for methodology content. Option D suppresses validation failures by weakening the contract, but it does not improve extraction and would convert recoverable omissions into incomplete records.
The examples should be drawn from real failure cases, evaluated on a held-out set, and expanded when monitoring reveals new layouts. Schema constraints and few-shot coverage solve different layers of reliability and should be used together.
Strict schemas eliminate malformed JSON and can guarantee that tool inputs conform to declared types, but they cannot force Claude to locate evidence that appears under unfamiliar headings or in atypical sections.
Anthropic's prompting guidance says that a few relevant, diverse, structured examples are among the most reliable ways to improve accuracy and consistency. Examples should therefore show inline citations, reference lists, numbered bibliographies, methodology sections, and methods embedded in introductions, each paired with the correct extracted structure. This teaches the intended evidence-location and granularity rules rather than merely repeating the same request. Option A retries an unchanged prompt and can reproduce the same omission. Option C introduces brittle regex rules that may miss nonstandard citations and mistake keyword mentions for methodology content. Option D suppresses validation failures by weakening the contract, but it does not improve extraction and would convert recoverable omissions into incomplete records.
The examples should be drawn from real failure cases, evaluated on a held-out set, and expanded when monitoring reveals new layouts. Schema constraints and few-shot coverage solve different layers of reliability and should be used together.
by Mamie at Aug 27, 2026, 02:53 AM
0
0
0
10
Comments
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
Report Comment
Commenting
You can sign-up / login (it's free).