Lyft moved LyftLearn's offline training and batch-processing workloads off a fully self-managed Kubernetes stack and onto AWS SageMaker, while deliberately keeping online model serving on Kubernetes — requiring zero changes to any team's actual ML code.

LyftLearn Compute — the offline half of Lyft’s ML platform, handling model training, batch processing, and hyperparameter optimization for hundreds of data scientists — ran entirely as custom Kubernetes workloads. A K8s Orchestration Service translated every job request into full Kubernetes resource specs: pods, ConfigMaps, Secrets, PersistentVolumeClaims, Services, Ingresses, and RBAC policies, while a fleet of background watcher scripts continuously polled the cluster to keep Lyft’s own job-tracking database in sync with Kubernetes’ actual state.

Lyft’s core migration principle was strict: replace the execution engine underneath — Kubernetes to SageMaker — while every user’s actual ML code, the Python that trains models and processes data, worked identically on both platforms with no modifications. Forcing hundreds of users to rewrite business-critical workflows wasn’t an option Lyft was willing to consider; the entire compatibility burden had to sit with the platform team, not spread across every team that used it.

Lyft evaluated AWS SageMaker across both halves of LyftLearn. For LyftLearn Serving, the online half handling real-time inference, adopting SageMaker would have meant re-architecting deployment, promotion, and monitoring systems deeply integrated with Lyft’s own tooling — observability built on internal infrastructure rather than CloudWatch, client communication over Envoy rather than SageMaker’s invocation patterns. Lyft’s own existing Kubernetes-based serving stack was already meeting latency requirements reliably, so the team chose to keep it exactly as it was. For LyftLearn Compute, the calculus reversed: this was where the operational complexity actually lived — eventually-consistent job state, heterogeneous cluster capacity planning, and custom orchestration for every new feature — and SageMaker’s managed infrastructure addressed each of those directly.

The trickiest technical gap was job launch speed — Kubernetes jobs started in 30-45 seconds on warm, already-idle cluster nodes, while SageMaker provisioned instances on demand with no idle waste but slower cold starts. Lyft closed the gap for notebooks using SOCI lazy-loading container indexes, cutting notebook startup times 40-50% by fetching only the filesystem layers needed immediately; for the most latency-sensitive training jobs — some retraining every 15 minutes — SageMaker’s warm pools kept instances alive between runs instead. A separate networking gap for interactive Spark, where the driver now ran in SageMaker Studio while executors stayed on Lyft’s EKS cluster, required partnering directly with AWS to open inbound connections that SageMaker Studio’s default networking blocked. Rolled out repository by repository with both infrastructures running in parallel, the migration reduced compute costs by eliminating idle cluster capacity and made infrastructure-related incidents rare, freeing the platform team to build capabilities instead of managing low-level orchestration.

“Build versus buy” isn’t a single company-wide policy — it can be the right call for one half of a platform and the wrong call for the other half, decided by where the actual operational complexity lives, not by a general preference for managed services.