Final project · Evidence review

Deliver a grounded pilot brief

Project dataset

My saved work →

Project dataset Fresh dataset challenge →Each dataset keeps its own attempts.

Your task

Return exactly start_date (string), total_tickets, labelled_tickets, correct_tickets, accuracy_percent, median_minutes (numbers), satisfaction_percent (null), human_approval (boolean), accepted_ticket_ids, excluded_ticket_ids and source_ids (arrays).
Use ticket_id as the business key. Compare actual, predicted and minutes. Collapse identical measurements; keep the first source record as evidence. Exclude all measurements for any ticket whose fields conflict.
Sort accepted_ticket_ids and excluded_ticket_ids ascending. source_ids starts with BRIEF and POLICY, followed by the first evidence source for each accepted ticket in accepted_ticket_ids order.
total_tickets includes accepted unlabelled tickets. Accuracy uses labelled accepted tickets only; round 100 × correct / labelled to two decimal places. Do not use source record count as the denominator.
Median uses known minutes among accepted tickets, sorted numerically; average the two middle values for an even count. Unknown minutes remain excluded. Satisfaction is null; retain the human-approval policy.

Pass every output check to complete this task. Suggested active time: 10 minutes. You can keep working after that.

Required output

Return one JSON object with exactly these fields.

start_datestring
total_ticketsnumber
labelled_ticketsnumber
correct_ticketsnumber
accuracy_percentnumber
median_minutesnumber
satisfaction_percentNULL
human_approvalboolean
accepted_ticket_idsarray
excluded_ticket_idsarray
source_idsarray

Keep strings, numbers, booleans, arrays and null in their required JSON types. Include every field and no extra fields.

Use null only where the evidence and task rules require an unknown value.

Reconciliation checks

60 measurement records plus two policy sources. 56 accepted tickets; 1 conflicting ticket excluded.
Accuracy denominator 51; numerator 44; 50 known handling times.
The source manifest must contain one first-occurrence measurement ID for each accepted ticket, with no source belonging to an excluded ticket.

Need a next step?

Hint 1: where to start

Check every field against its source and required type.

Hint 2: build the method

Write down the eligible keys before aggregating. Inspect each anomaly against the output contract.

Hint 3: check your work

Use the reconciliation counts after constructing the complete output; a matching total alone does not prove the rows are correct.

How your work is checked

Every output check must pass. The score shows how much matched and helps you find mistakes; completion comes from passing all checks. Active time is recorded separately.

These checks compare fixed evidence fields and types. They do not assess arbitrary writing quality.

Passing records a correct practice output on this device. It is not a professional certification.

Source materials

62 original source records.

Data dictionary

sources[].id text: Stable evidence source ID; use the first source for exact repeat measurements.
record.ticket_id text: Business key; one accepted ticket can have several evidence rows.
record.actual text | null: Reference label; NULL means unlabelled, excluded only from accuracy.
record.predicted text: Recorded classifier output; compared exactly with the known label.
record.minutes number | null: Measured handling minutes; NULL excluded from median.
BRIEF/POLICY.text text: Authoritative fictional scope, unknown measurement and approval rules.

View source data

{
  "sources": [
    {
      "id": "BRIEF",
      "title": "Pilot scope",
      "text": "Original fictional pilot starts 2026-11-02. This handoff includes only the ticket evidence in this file. Satisfaction was not measured."
    },
    {
      "id": "POLICY",
      "title": "Review policy",
      "text": "Human approval is required before any customer reply. Exact repeated measurements of a ticket count once; conflicting measurements exclude that ticket entirely. Unlabelled tickets stay in the accepted manifest but are excluded from accuracy. Missing handling minutes are excluded from the median."
    },
    {
      "id": "E100",
      "title": "Measurement T100",
      "text": "ticket_id=T100; actual=unknown; predicted=other; minutes=unknown.",
      "record": {
        "id": "E100",
        "ticket_id": "T100",
        "actual": null,
        "predicted": "other",
        "minutes": null
      }
    },
    {
      "id": "E101",
      "title": "Measurement T101",
      "text": "ticket_id=T101; actual=delivery; predicted=delivery; minutes=4.",
      "record": {
        "id": "E101",
        "ticket_id": "T101",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 4
      }
    },
    {
      "id": "E102",
      "title": "Measurement T102",
      "text": "ticket_id=T102; actual=technical; predicted=technical; minutes=5.",
      "record": {
        "id": "E102",
        "ticket_id": "T102",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 5
      }
    },
    {
      "id": "E103",
      "title": "Measurement T103",
      "text": "ticket_id=T103; actual=billing; predicted=billing; minutes=6.",
      "record": {
        "id": "E103",
        "ticket_id": "T103",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 6
      }
    },
    {
      "id": "E104",
      "title": "Measurement T104",
      "text": "ticket_id=T104; actual=delivery; predicted=delivery; minutes=7.",
      "record": {
        "id": "E104",
        "ticket_id": "T104",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 7
      }
    },
    {
      "id": "E105",
      "title": "Measurement T105",
      "text": "ticket_id=T105; actual=technical; predicted=technical; minutes=8.",
      "record": {
        "id": "E105",
        "ticket_id": "T105",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 8
      }
    },
    {
      "id": "E106",
      "title": "Measurement T106",
      "text": "ticket_id=T106; actual=billing; predicted=billing; minutes=9.",
      "record": {
        "id": "E106",
        "ticket_id": "T106",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 9
      }
    },
    {
      "id": "E107",
      "title": "Measurement T107",
      "text": "ticket_id=T107; actual=delivery; predicted=other; minutes=10.",
      "record": {
        "id": "E107",
        "ticket_id": "T107",
        "actual": "delivery",
        "predicted": "other",
        "minutes": 10
      }
    },
    {
      "id": "E108",
      "title": "Measurement T108",
      "text": "ticket_id=T108; actual=technical; predicted=technical; minutes=3.",
      "record": {
        "id": "E108",
        "ticket_id": "T108",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 3
      }
    },
    {
      "id": "E109",
      "title": "Measurement T109",
      "text": "ticket_id=T109; actual=billing; predicted=billing; minutes=4.",
      "record": {
        "id": "E109",
        "ticket_id": "T109",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 4
      }
    },
    {
      "id": "E110",
      "title": "Measurement T110",
      "text": "ticket_id=T110; actual=delivery; predicted=delivery; minutes=unknown.",
      "record": {
        "id": "E110",
        "ticket_id": "T110",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": null
      }
    },
    {
      "id": "E111",
      "title": "Measurement T111",
      "text": "ticket_id=T111; actual=unknown; predicted=technical; minutes=6.",
      "record": {
        "id": "E111",
        "ticket_id": "T111",
        "actual": null,
        "predicted": "technical",
        "minutes": 6
      }
    },
    {
      "id": "E112",
      "title": "Measurement T112",
      "text": "ticket_id=T112; actual=billing; predicted=billing; minutes=7.",
      "record": {
        "id": "E112",
        "ticket_id": "T112",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 7
      }
    },
    {
      "id": "E113",
      "title": "Measurement T113",
      "text": "ticket_id=T113; actual=delivery; predicted=delivery; minutes=8.",
      "record": {
        "id": "E113",
        "ticket_id": "T113",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 8
      }
    },
    {
      "id": "E114",
      "title": "Measurement T114",
      "text": "ticket_id=T114; actual=technical; predicted=other; minutes=9.",
      "record": {
        "id": "E114",
        "ticket_id": "T114",
        "actual": "technical",
        "predicted": "other",
        "minutes": 9
      }
    },
    {
      "id": "E115",
      "title": "Measurement T115",
      "text": "ticket_id=T115; actual=billing; predicted=billing; minutes=10.",
      "record": {
        "id": "E115",
        "ticket_id": "T115",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 10
      }
    },
    {
      "id": "E116",
      "title": "Measurement T116",
      "text": "ticket_id=T116; actual=delivery; predicted=delivery; minutes=3.",
      "record": {
        "id": "E116",
        "ticket_id": "T116",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 3
      }
    },
    {
      "id": "E117",
      "title": "Measurement T117",
      "text": "ticket_id=T117; actual=technical; predicted=technical; minutes=4.",
      "record": {
        "id": "E117",
        "ticket_id": "T117",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 4
      }
    },
    {
      "id": "E118",
      "title": "Measurement T118",
      "text": "ticket_id=T118; actual=billing; predicted=billing; minutes=5.",
      "record": {
        "id": "E118",
        "ticket_id": "T118",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 5
      }
    },
    {
      "id": "E119",
      "title": "Measurement T119",
      "text": "ticket_id=T119; actual=delivery; predicted=delivery; minutes=6.",
      "record": {
        "id": "E119",
        "ticket_id": "T119",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 6
      }
    },
    {
      "id": "E120",
      "title": "Measurement T120",
      "text": "ticket_id=T120; actual=technical; predicted=technical; minutes=unknown.",
      "record": {
        "id": "E120",
        "ticket_id": "T120",
        "actual": "technical",
        "predicted": "technical",
        "minutes": null
      }
    },
    {
      "id": "E121",
      "title": "Measurement T121",
      "text": "ticket_id=T121; actual=billing; predicted=other; minutes=8.",
      "record": {
        "id": "E121",
        "ticket_id": "T121",
        "actual": "billing",
        "predicted": "other",
        "minutes": 8
      }
    },
    {
      "id": "E122",
      "title": "Measurement T122",
      "text": "ticket_id=T122; actual=unknown; predicted=delivery; minutes=9.",
      "record": {
        "id": "E122",
        "ticket_id": "T122",
        "actual": null,
        "predicted": "delivery",
        "minutes": 9
      }
    },
    {
      "id": "E123",
      "title": "Measurement T123",
      "text": "ticket_id=T123; actual=technical; predicted=technical; minutes=10.",
      "record": {
        "id": "E123",
        "ticket_id": "T123",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 10
      }
    },
    {
      "id": "E124",
      "title": "Measurement T124",
      "text": "ticket_id=T124; actual=billing; predicted=billing; minutes=3.",
      "record": {
        "id": "E124",
        "ticket_id": "T124",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 3
      }
    },
    {
      "id": "E125",
      "title": "Measurement T125",
      "text": "ticket_id=T125; actual=delivery; predicted=delivery; minutes=4.",
      "record": {
        "id": "E125",
        "ticket_id": "T125",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 4
      }
    },
    {
      "id": "E126",
      "title": "Measurement T126",
      "text": "ticket_id=T126; actual=technical; predicted=technical; minutes=5.",
      "record": {
        "id": "E126",
        "ticket_id": "T126",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 5
      }
    },
    {
      "id": "E127",
      "title": "Measurement T127",
      "text": "ticket_id=T127; actual=billing; predicted=billing; minutes=6.",
      "record": {
        "id": "E127",
        "ticket_id": "T127",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 6
      }
    },
    {
      "id": "E128",
      "title": "Measurement T128",
      "text": "ticket_id=T128; actual=delivery; predicted=other; minutes=7.",
      "record": {
        "id": "E128",
        "ticket_id": "T128",
        "actual": "delivery",
        "predicted": "other",
        "minutes": 7
      }
    },
    {
      "id": "E129",
      "title": "Measurement T129",
      "text": "ticket_id=T129; actual=technical; predicted=technical; minutes=8.",
      "record": {
        "id": "E129",
        "ticket_id": "T129",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 8
      }
    },
    {
      "id": "E130",
      "title": "Measurement T130",
      "text": "ticket_id=T130; actual=billing; predicted=billing; minutes=unknown.",
      "record": {
        "id": "E130",
        "ticket_id": "T130",
        "actual": "billing",
        "predicted": "billing",
        "minutes": null
      }
    },
    {
      "id": "E131",
      "title": "Measurement T131",
      "text": "ticket_id=T131; actual=delivery; predicted=delivery; minutes=10.",
      "record": {
        "id": "E131",
        "ticket_id": "T131",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 10
      }
    },
    {
      "id": "E132",
      "title": "Measurement T132",
      "text": "ticket_id=T132; actual=technical; predicted=technical; minutes=3.",
      "record": {
        "id": "E132",
        "ticket_id": "T132",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 3
      }
    },
    {
      "id": "E133",
      "title": "Measurement T133",
      "text": "ticket_id=T133; actual=unknown; predicted=billing; minutes=4.",
      "record": {
        "id": "E133",
        "ticket_id": "T133",
        "actual": null,
        "predicted": "billing",
        "minutes": 4
      }
    },
    {
      "id": "E134",
      "title": "Measurement T134",
      "text": "ticket_id=T134; actual=delivery; predicted=delivery; minutes=5.",
      "record": {
        "id": "E134",
        "ticket_id": "T134",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 5
      }
    },
    {
      "id": "E135",
      "title": "Measurement T135",
      "text": "ticket_id=T135; actual=technical; predicted=other; minutes=6.",
      "record": {
        "id": "E135",
        "ticket_id": "T135",
        "actual": "technical",
        "predicted": "other",
        "minutes": 6
      }
    },
    {
      "id": "E136",
      "title": "Measurement T136",
      "text": "ticket_id=T136; actual=billing; predicted=billing; minutes=7.",
      "record": {
        "id": "E136",
        "ticket_id": "T136",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 7
      }
    },
    {
      "id": "E137",
      "title": "Measurement T137",
      "text": "ticket_id=T137; actual=delivery; predicted=delivery; minutes=8.",
      "record": {
        "id": "E137",
        "ticket_id": "T137",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 8
      }
    },
    {
      "id": "E138",
      "title": "Measurement T138",
      "text": "ticket_id=T138; actual=technical; predicted=technical; minutes=9.",
      "record": {
        "id": "E138",
        "ticket_id": "T138",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 9
      }
    },
    {
      "id": "E139",
      "title": "Measurement T139",
      "text": "ticket_id=T139; actual=billing; predicted=billing; minutes=10.",
      "record": {
        "id": "E139",
        "ticket_id": "T139",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 10
      }
    },
    {
      "id": "E140",
      "title": "Measurement T140",
      "text": "ticket_id=T140; actual=delivery; predicted=delivery; minutes=unknown.",
      "record": {
        "id": "E140",
        "ticket_id": "T140",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": null
      }
    },
    {
      "id": "E141",
      "title": "Measurement T141",
      "text": "ticket_id=T141; actual=technical; predicted=technical; minutes=4.",
      "record": {
        "id": "E141",
        "ticket_id": "T141",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 4
      }
    },
    {
      "id": "E142",
      "title": "Measurement T142",
      "text": "ticket_id=T142; actual=billing; predicted=other; minutes=5.",
      "record": {
        "id": "E142",
        "ticket_id": "T142",
        "actual": "billing",
        "predicted": "other",
        "minutes": 5
      }
    },
    {
      "id": "E143",
      "title": "Measurement T143",
      "text": "ticket_id=T143; actual=delivery; predicted=delivery; minutes=6.",
      "record": {
        "id": "E143",
        "ticket_id": "T143",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 6
      }
    },
    {
      "id": "E144",
      "title": "Measurement T144",
      "text": "ticket_id=T144; actual=unknown; predicted=technical; minutes=7.",
      "record": {
        "id": "E144",
        "ticket_id": "T144",
        "actual": null,
        "predicted": "technical",
        "minutes": 7
      }
    },
    {
      "id": "E145",
      "title": "Measurement T145",
      "text": "ticket_id=T145; actual=billing; predicted=billing; minutes=8.",
      "record": {
        "id": "E145",
        "ticket_id": "T145",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 8
      }
    },
    {
      "id": "E146",
      "title": "Measurement T146",
      "text": "ticket_id=T146; actual=delivery; predicted=delivery; minutes=9.",
      "record": {
        "id": "E146",
        "ticket_id": "T146",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 9
      }
    },
    {
      "id": "E147",
      "title": "Measurement T147",
      "text": "ticket_id=T147; actual=technical; predicted=technical; minutes=10.",
      "record": {
        "id": "E147",
        "ticket_id": "T147",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 10
      }
    },
    {
      "id": "E148",
      "title": "Measurement T148",
      "text": "ticket_id=T148; actual=billing; predicted=billing; minutes=3.",
      "record": {
        "id": "E148",
        "ticket_id": "T148",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 3
      }
    },
    {
      "id": "E149",
      "title": "Measurement T149",
      "text": "ticket_id=T149; actual=delivery; predicted=other; minutes=4.",
      "record": {
        "id": "E149",
        "ticket_id": "T149",
        "actual": "delivery",
        "predicted": "other",
        "minutes": 4
      }
    },
    {
      "id": "E150",
      "title": "Measurement T150",
      "text": "ticket_id=T150; actual=technical; predicted=technical; minutes=unknown.",
      "record": {
        "id": "E150",
        "ticket_id": "T150",
        "actual": "technical",
        "predicted": "technical",
        "minutes": null
      }
    },
    {
      "id": "E151",
      "title": "Measurement T151",
      "text": "ticket_id=T151; actual=billing; predicted=billing; minutes=6.",
      "record": {
        "id": "E151",
        "ticket_id": "T151",
        "actual": "billing",
        "predicted": "billing",
        "minutes": 6
      }
    },
    {
      "id": "E152",
      "title": "Measurement T152",
      "text": "ticket_id=T152; actual=delivery; predicted=delivery; minutes=7.",
      "record": {
        "id": "E152",
        "ticket_id": "T152",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 7
      }
    },
    {
      "id": "E153",
      "title": "Measurement T153",
      "text": "ticket_id=T153; actual=technical; predicted=technical; minutes=8.",
      "record": {
        "id": "E153",
        "ticket_id": "T153",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 8
      }
    },
    {
      "id": "E154",
      "title": "Measurement T101",
      "text": "ticket_id=T101; actual=delivery; predicted=delivery; minutes=4.",
      "record": {
        "id": "E154",
        "ticket_id": "T101",
        "actual": "delivery",
        "predicted": "delivery",
        "minutes": 4
      }
    },
    {
      "id": "E155",
      "title": "Measurement T102",
      "text": "ticket_id=T102; actual=technical; predicted=technical; minutes=5.",
      "record": {
        "id": "E155",
        "ticket_id": "T102",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 5
      }
    },
    {
      "id": "E156",
      "title": "Measurement T106",
      "text": "ticket_id=T106; actual=billing; predicted=disputed; minutes=9.",
      "record": {
        "id": "E156",
        "ticket_id": "T106",
        "actual": "billing",
        "predicted": "disputed",
        "minutes": 9
      }
    },
    {
      "id": "E157",
      "title": "Measurement T160",
      "text": "ticket_id=T160; actual=technical; predicted=technical; minutes=7.",
      "record": {
        "id": "E157",
        "ticket_id": "T160",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 7
      }
    },
    {
      "id": "E158",
      "title": "Measurement T161",
      "text": "ticket_id=T161; actual=technical; predicted=technical; minutes=7.",
      "record": {
        "id": "E158",
        "ticket_id": "T161",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 7
      }
    },
    {
      "id": "E159",
      "title": "Measurement T162",
      "text": "ticket_id=T162; actual=technical; predicted=technical; minutes=7.",
      "record": {
        "id": "E159",
        "ticket_id": "T162",
        "actual": "technical",
        "predicted": "technical",
        "minutes": 7
      }
    }
  ],
  "brief": "Audit the complete pilot evidence file and produce a reproducible handoff manifest.",
  "flawed_output": "Every source record is a unique measured ticket, missing labels count as errors, and satisfaction improved."
}

Your work

Active 0:00 · Paused 0s

JSON answer

Import answer

Keep a handoff you can replay

Check your current answer, then download the output and its report. The report includes the source, answer and each check so you can review what passed and what needs work.

To replay your work, open this task with the same dataset, import your answer and check it again.

Show the reference answer

{
  "start_date": "2026-11-02",
  "total_tickets": 56,
  "labelled_tickets": 51,
  "correct_tickets": 44,
  "accuracy_percent": 86.27,
  "median_minutes": 7,
  "satisfaction_percent": null,
  "human_approval": true,
  "accepted_ticket_ids": [
    "T100",
    "T101",
    "T102",
    "T103",
    "T104",
    "T105",
    "T107",
    "T108",
    "T109",
    "T110",
    "T111",
    "T112",
    "T113",
    "T114",
    "T115",
    "T116",
    "T117",
    "T118",
    "T119",
    "T120",
    "T121",
    "T122",
    "T123",
    "T124",
    "T125",
    "T126",
    "T127",
    "T128",
    "T129",
    "T130",
    "T131",
    "T132",
    "T133",
    "T134",
    "T135",
    "T136",
    "T137",
    "T138",
    "T139",
    "T140",
    "T141",
    "T142",
    "T143",
    "T144",
    "T145",
    "T146",
    "T147",
    "T148",
    "T149",
    "T150",
    "T151",
    "T152",
    "T153",
    "T160",
    "T161",
    "T162"
  ],
  "excluded_ticket_ids": [
    "T106"
  ],
  "source_ids": [
    "BRIEF",
    "POLICY",
    "E100",
    "E101",
    "E102",
    "E103",
    "E104",
    "E105",
    "E107",
    "E108",
    "E109",
    "E110",
    "E111",
    "E112",
    "E113",
    "E114",
    "E115",
    "E116",
    "E117",
    "E118",
    "E119",
    "E120",
    "E121",
    "E122",
    "E123",
    "E124",
    "E125",
    "E126",
    "E127",
    "E128",
    "E129",
    "E130",
    "E131",
    "E132",
    "E133",
    "E134",
    "E135",
    "E136",
    "E137",
    "E138",
    "E139",
    "E140",
    "E141",
    "E142",
    "E143",
    "E144",
    "E145",
    "E146",
    "E147",
    "E148",
    "E149",
    "E150",
    "E151",
    "E152",
    "E153",
    "E157",
    "E158",
    "E159"
  ]
}

Separate source records from unique tickets. Exclude conflicting tickets, keep unlabelled accepted tickets in the manifest, and calculate accuracy and median using their different eligible subsets.