Exam CCAR-F Topic 1 Question 66 Discussion
Actual exam question for Anthropic's CCAR-F exam
Question #: 66
Topic #: 1
Question #: 66
Topic #: 1
You built an LLM-powered code-review tool that analyzes pull requests and returns structured findings. Each finding is a JSON object containing file_path, line_number, issue_category-such as security or style-and description. Developers can dismiss findings they consider unhelpful, and currently 35% of findings are dismissed. You want to analyze these dismissals to understand what the system is getting wrong and improve the prompts accordingly. What change to the output structure would best support this analysis?
Suggested Answer: B Vote an answer
Option B creates the granular error taxonomy needed to understand recurring false-positive patterns. The existing issue_category field is too broad: a high dismissal rate for style does not reveal whether developers object to single-letter variables, line-length warnings, naming conventions, or something else. Recording the detected construct allows dismissal rates to be grouped by trigger and connected to targeted prompt revisions or project-specific examples.
Anthropic's evaluation guidance recommends measurable, task-specific criteria and evaluation cases that reflect real production behavior and edge cases. Its structured-output documentation supports schema- constrained, parseable records suitable for this type of downstream analysis.
Option A can help rank findings, but a confidence score does not explain why developers reject them and may be poorly calibrated across issue types. Option C produces more text without adding a stable dimension for aggregation. Option D removes useful category information and makes systematic analysis harder. A normalized detected_pattern field enables dashboards, pattern-level dismissal metrics, representative-example sampling, and controlled evaluations of prompt changes while retaining the broader category for higher-level reporting.
Anthropic's evaluation guidance recommends measurable, task-specific criteria and evaluation cases that reflect real production behavior and edge cases. Its structured-output documentation supports schema- constrained, parseable records suitable for this type of downstream analysis.
Option A can help rank findings, but a confidence score does not explain why developers reject them and may be poorly calibrated across issue types. Option C produces more text without adding a stable dimension for aggregation. Option D removes useful category information and makes systematic analysis harder. A normalized detected_pattern field enables dashboards, pattern-level dismissal metrics, representative-example sampling, and controlled evaluations of prompt changes while retaining the broader category for higher-level reporting.
by Ron at Oct 02, 2026, 02:05 AM
0
0
0
10
Comments
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
Report Comment
Commenting
You can sign-up / login (it's free).