AI Infrastructure Engineer

ABOUT MERCOR Mercor's mission is to organize human intelligence to power the AI economy. We're a leading AI data company, building the layer between human expertise and frontier models. Millions of domain experts on the platform are paid over $4 million per day to train frontier AI models. Mercor's APEX benchmark family measures AI's real-world impact on professional work. Mercor Enterprise brings this same infrastructure to Fortune 500 companies: helping companies capture how their best people actually work, translating that expertise directly back into agents.   Mercor is creating a new category of work where expertise powers AI advancement. Achieving this requires an ambitious, fast-paced and deeply committed team. You’ll work alongside researchers, operators, and AI companies at the forefront of shaping the systems that are redefining society. Mercor is a profitable Series C company valued at $10 billion. We work in-person five days a week in our San Francisco, NYC, or London offices. Summary of role: All of Mercor’s AI work, from training and serving models to building agents, runs on GPUs. Your job is to make that compute fast, fault-tolerant and fully utilized. You’ll own the accelerator platform end to end. That means the clusters and schedulers across AWS, GCP, neoclouds and on-prem, the training and LLM serving on top, and the observability and cost data that show where every GPU-hour goes. This is a software engineering role, not an ops role. You’ll write production code every day, own systems from design through on-call, and build the platform that sets how fast Mercor can move. Representative projects - Stand up a multi-cloud GPU fleet spanning hyperscalers, neoclouds and on-prem, orchestrated through custom Kubernetes and Slurm operators so researchers never have to think about where their job lands. - Build a scheduling layer with gang scheduling, bin-packing and fair-share quotas that measurably raises accelerator utilization across teams. - Ship a high-throughput vLLM serving stack with prefill/decode disaggregation, KV cache offload and FP8/FP4 quantization. Back it with a benchmarking and traffic-replay harness that tracks TTFT, TPOT and latency percentiles against per-config baselines. - Build a self-healing health-check and remediation system that rides out fault storms across hardware, network and software without paging a human. - Build an end-to-end cost-attribution pipeline. It ingests utilization, occupancy and billing data into the warehouse, attributes every GPU-hour to a team and workload, and drives capacity planning. - Build internal tools and dashboards that researchers and non-engineers use every day.   Responsibilities: - Design, deploy and operate GPU clusters across AWS, GCP, and other hyperscalers/neoclouds/on-premise data centers, including custom Kubernetes and Slurm operators and the metadata services behind them. - Build scheduling that keeps accelerators busy: gang scheduling for distributed jobs, bin-packing to reduce fragmentation, and quotas and fair share for multi-tenant teams. - Run distributed training infrastructure on PyTorch Distributed, Ray, or Megatron, with fast checkpointing and model loading. - Deploy and tune high-throughput LLM serving with vLLM or similar: tensor and model/expert parallelism, prefill/decode disaggregation, KV cache offload, FP8/FP4 quantization. - Build benchmarking and traffic-replay tools that measure TTFT, TPOT and latency percentiles, and set per-config performance baselines. - Build health checks across hardware, network and software layers, and automated remediation that handles fault storms without human intervention. - Instrument fleet utilization with DCGM, Prometheus and Grafana. Write recording rules and alerts that teams rely on. - Ingest utilization, occupancy and billing data into a warehouse, attribute spend to teams and workloads, and support capacity planning. - Drive autoscaling, rightsizing and idle-capacity recovery programs with measurable cost results. - Write clear runbooks and documentation, and explain infrastructure tradeoffs to researchers, finance and leadership. - Build internal tools or dashboards that researchers and non-engineers use daily. You may be a good fit if you - Have 5+ years building and operating production distributed systems or ML infrastructure - Write production-quality Python plus Go or C++, and are fluent in shell scripting for system administration. - Have deep hands-on Kubernetes and container experience, plus working knowledge of Slurm or another HPC resource manager. - Have run GPU workloads at scale, including scheduling, orchestration and troubleshooting. - Have hands-on experience with at least one distributed training or serving stack: PyTorch Distributed, Ray, vLLM or Megatron-LM. - Have deep experience with at least one major cloud provider. - Know observability tools like DataDog, Prometheus, PromQL and Grafana, and are comfortable on-call for critical systems. - Own projects end to end and gather your own requirements with little direction. Strong candidates may also have experience with (or a strong desire to learn deeply) - LLM inference optimization: KV cache systems, disaggregated serving, quantization, speculative decoding - High-performance networking: RDMA, InfiniBand, NCCL, NVLink - GPU profiling with Nsight or similar, and CUDA or Triton kernel work - Accelerators beyond NVIDIA: AMD MI-series, TPU, Trainium - Capacity planning, billing normalization or cost attribution across providers - Automated fault diagnosis, log mining or AIOps at fleet scale - Configuration management (Ansible, Puppet) and secure or air-gapped deployments Benefits - Bi-annual performance bonus structure - Generous equity grant vested over 4 years - Up to $15k Relocation bonus - $10K housing bonus (if you live within 0.5 miles of our office) - $1.5K monthly stipend for meals - Free Equinox membership - $200 monthly laundry reimbursement - $200 monthly personal wellness reimbursement - Health, Dental, Vision insurance