◈OSA
Task 1

The Audit

TryTrainMe's CISO issued a standing order: no model reaches production without completing a full sandboxed evaluation cycle. Four code review model candidates have been submitted to SupplySecLab. All four have completed their evaluation runs. The automated screening has flagged three candidates as unsafe. Your task is to assess Candidate A and make the production call.

Four in Queue
Four candidates. One gate. The checklist does not care about reputation.

The telemetry from three candidates is below. The fourth is loaded in the platform and ready for direct assessment. All four were evaluated against the same test pull request: a change that removes input validation from an authentication endpoint.

Candidate B: code_reviewer_lite.safetensors

           SESSION START: model_load
MODEL LOAD BEGIN: /models/code_reviewer_lite.safetensors (safetensors)
FILE ACCESS: /models/code_reviewer_lite.safetensors mode=rb [OK]
FORMAT VALIDATION: safetensors header valid [OK]
MODEL LOAD COMPLETE: object_type=SafeTensors [OK]
SESSION STOP: model_load
SESSION START: inference
PROMPT TEMPLATE LOAD: source=internal (TryTrainMe v1.0) [VERIFIED]
GUARDRAIL CHECK: security_review_flag=enabled [OK]
INFERENCE COMPLETE: verdict=Needs Changes
SESSION STOP: inference

Candidate C: pr_analyzer_v3.h5

           SESSION START: model_load
MODEL LOAD BEGIN: /models/pr_analyzer_v3.h5 (keras)
FILE ACCESS: /models/pr_analyzer_v3.h5 mode=rb [OK]
LAMBDA LAYER DETECTED: custom code present [DANGEROUS]
LAMBDA LAYER CODE: exec(open('/tmp/.cache').read()) [SUSPICIOUS]
MODEL LOAD COMPLETE: object_type=Sequential [OK]
SESSION STOP: model_load
SESSION START: inference
PROMPT TEMPLATE LOAD: source=internal (TryTrainMe v1.0) [VERIFIED]
GUARDRAIL CHECK: security_review_flag=enabled [OK]
LAMBDA EXEC: /tmp/.cache read attempt blocked [DANGEROUS]
INFERENCE COMPLETE: verdict=Needs Changes
SESSION STOP: inference

Candidate D: api.reviewsvc.io

           SESSION START: api_connect
ENDPOINT CONFIGURED: https://api.reviewsvc.io/v2 [UNVERIFIED]
TLS VERIFICATION: certificate valid [OK]
AUTHENTICATION: bearer token present [OK]
API METADATA: model_provenance=not_disclosed [WARNING]
API METADATA: compliance_cert=absent [WARNING]
SESSION STOP: api_connect
SESSION START: inference
PROMPT TEMPLATE LOAD: source=vendor-managed [UNVERIFIED]
GUARDRAIL CHECK: vendor-managed, not inspectable [UNVERIFIED]
INFERENCE COMPLETE: verdict=Approved
SESSION STOP: inference

Press the Open Agent button near the top of this task to load Candidate A's evaluation. The telemetry panel streams automatically. Read both sessions before sending any queries.

The evidence is in the telemetry and the agent.

?Answer the questions below

  1. Candidate A's load session shows a suspicious file access event. What file did it attempt to read?
  2. What security guardrail flag is disabled in Candidate A's inference session?
  3. Query Candidate A's agent to find out which policy template governs its review behaviour. What is the policy template?
  4. Candidate A's two supply chain failures are not independent. Find what links them and use it to retrieve the flag. What is the flag?
  5. Based on your full assessment of all four candidates, what is your production recommendation for Candidate A? Enter: Approve or Reject
  6. Which candidate would you approve for production deployment?