Join host Hamza Mimi for a deep conversation on model customization with two guests tackling it from opposite vantage points. Oliver Gracer, Head of Data & Insights at Perform.AI (formerly Parcel Perform) — the Singapore-based delivery experience platform processing billions of shipments across 160+ countries — shares how his team fine-tuned Amazon Nova and achieved something counter-intuitive: a smaller model that outperformed larger ones on accuracy, cost, and speed. Joining him is Karan Pandharkar, Principal Product Manager for Forge and model customization at AWS, who reveals the patterns, anti-patterns, and tools he sees across customers worldwide.Together they unpack why customization is an economics question before it’s a technology question, and give a practical roadmap for teams moving from prompting to production customization.
This episode covers critical topics including:
• Why and when to customize vs. prompt engineer, RAG, or build an agentic harness
• Building a robust evaluation framework and defining what "good" looks like
• The ROI conversation: fine-tuning cost vs. inference cost (and talking to your CFO)
• The hallucination problem and how fine-tuning tackled it
• Pre-training, mid-training, fine-tuning, and reinforcement learning explained
• Data quality over quantity — and how little data you may actually need
• Catastrophic forgetting and the role of general-purpose data mixing
• LoRA / parameter-efficient fine-tuning and the pay-as-you-go economics of serving
• Customizing with SageMaker AI, Amazon Bedrock, and Nova Forge
Whether you’re a startup, enterprise, or public sector organization running high-volume AI workloads, this episode offers invaluable guidance on building scalable, cost-efficient, production-ready AI solutions in today’s rapidly evolving landscape.
OUTLINE:
00:00:00 : Introduction
00:00:43 : Meet Oliver — Perform.AI
00:03:34 : Meet Karan — AWS Forge
00:07:04 : Why bother customizing when frontier models keep winning?
00:11:51 : The only 3 reasons to customize: quality, speed, cost
00:13:58 : Does the math actually work? The ROI reality
00:19:47 : When you should NOT customize
00:22:11 : How a small model beat the giants in production
00:25:36 : Making the model stop hallucinating
00:28:04 : Pre-training to RL: the 4 stages demystified
00:31:39 : You need way less data than you think
00:39:29 : Catastrophic forgetting (and how to avoid it)
00:44:08 : Why evals matter more than your model
00:48:48 : Prompt vs. fine-tune vs. agent — it’s not either/or
00:50:34 : How to actually do it on AWS
00:56:00 : From notebook to production
00:56:15 : Quickfire round
01:01:16 : A CTO’s playbook for customization
01:04:53 : Closing remarks
To learn more about Perform.AI : https://bit.ly/4ywZ6OM
Parcel Perform case study blog : https://aws.amazon.com/blogs/machine-learning/fine-tune-amazon-nova-models-for-accurat
Subscribe to AWS: https://go.aws/subscribe
Create a free AWS account: https://go.aws/signup
Try AWS for free: https://go.aws/free
Connect with an expert: https://go.aws/contact
Explore more: https://go.aws/more
Next steps:
Explore on AWS in Analyst Research: https://go.aws/reports
Discover, deploy, and manage software that runs on AWS: https://go.aws/marketplace
Join the AWS Partner Network: https://go.aws/partners
Learn more on how Amazon builds and operates software: https://go.aws/library
Do you have technical AWS questions?
Ask the community of experts on AWS re:Post: https://go.aws/3lPaoPb
Why AWS?
Amazon Web Services is the world’s most comprehensive and broadly adopted cloud, enabling customers to build anything they can imagine. We offer the greatest choice of innovative cloud capabilities and expertise, on the most extensive global infrastructure with industry-leading security, reliability, and performance.
#AWS #AmazonWebServices #CloudComputing #AWS #AmazonWebServices #CloudComputing #AWSForAI #artificial-intelligence #ModelCustomization #FineTuning #SmallLanguageModels #AmazonNova #SageMaker #AmazonBedrock #LoRA #MachineLearning #TechTalks
