Tools Nov 16, 2026 · Originally published Nov 13

SageMaker HyperPod adds Data Capture, logging live inference inputs and outputs to S3 at three tiers; in AWS's synthetic demo a 60-second LoRA fine-tune lifts held-out accuracy from 83.9% to 100%

Teams serving their own models usually lack the real traffic that arrives after launch. The three capture tiers, one full capture-label-fine-tune loop, and the shortcuts the demo exposed.

Image: AWS Machine Learning Blog
Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

  • The three tiers can be combined freely: the SageMaker endpoint tier records full requests and responses for compliance and Model Monitor; the load balancer tier records only access-log fields for low-cost traffic and error monitoring; the model pod tier records full payloads plus event metadata for debugging, analytics and training data.
  • The demo uses a synthetic home-appliance support router: after a product launch, the Other share rose from 4.8% to 36.8%; Athena queries surfaced the blind spot, a SQL policy labeled the captures, and a 60-second rank-16 LoRA fine-tune on one L40S trained about 29.9 million parameters, under 1% of the 3.1B model.
  • Held-out accuracy on 280 tickets went from 235 (83.9%) to 280 (100%) with no regression on the five existing classes, but the model scored 20 of 24 on hand-written probes, having learned two shortcuts: cooling means air conditioner, and any mention of Arctis means the AC category.
  • AWS measured near-identical p50 and p99 latency and throughput through the capture proxy versus direct to vLLM; the post warns that full payloads can include names, addresses and order numbers, and recommends sampling, KMS encryption, lifecycle rules and PII redaction.

Builder's takeThe most useful part is the 24 hand-written probes: an eval set from the same generator says 100%, and a change of phrasing exposes the shortcuts. For intent routing in AI Cloud Drive I would keep two eval sets the same way, one from captured traffic and one written by hand in deliberately different wording. If you capture production data, settle redaction and retention first, then think about the flywheel.

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →