Principal AI Engineering Architect

Title: Principal AI Engineering Architect Location: US Remote

Principal AI Engineering Architect

We're looking for a Principal AI Engineering Architect to lead the design and delivery of complex, multi-domain systems spanning cloud, data, and AI — with deep, hands-on mastery of multi-agent agentic AI solutions. This role is ideal for a deeply experienced engineer who owns the hardest architectural challenges, sets technical direction, and serves as the senior technical voice on the engagements they support, while also being able to roll up their sleeves and lead model development, agentic system design, and production delivery end-to-end.

In this role, you will operate as the senior technical authority on a cross-functional team, defining architecture across cloud infrastructure, data platforms, and AI/ML workloads, with a strong bias toward AWS-native services and AWS GenAI offerings. You'll partner closely with leadership and clients on technical strategy, lead the design and delivery of complex, production-grade multi-agent systems, mentor experienced engineers, and own high-stakes decisions that shape the long-term success of the systems you build.

Why This Role Matters

At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.

Every role here contributes directly to that mission. Here, you shape how AI systems integrate into enterprise operations, how teams move at real velocity, and how products create measurable impact for clients and the people they serve. We ship production-ready AI in 30 to 45 days. That pace demands people who take ownership, lead with craft, and care deeply about what they put their name on.

What You'll Do

Craft & Delivery

Define technical strategy and lead architectural design across cloud, data, and AI/ML systems for end-to-end engagements, owning architecture decisions and driving solutions from research through production at scale
Architect and ship production-grade multi-agent agentic AI systems, including agent orchestration, tool use, memory, and inter-agent communication patterns
Design and build with Amazon Bedrock AgentCore and complementary AWS GenAI services to deploy, scale, and operate agentic workloads securely in production
Architect scalable cloud-native solutions with a strong bias toward AWS, including multi-cloud and hybrid strategies where needed (AWS primary, with Azure, GCP, Kubernetes as secondary)
Design data architectures including warehouses, data lakes, and pipelines for batch and streaming workloads (e.g., Snowflake, Redshift, BigQuery, Spark, Kafka)
Design AI/ML systems including model serving, MLOps pipelines, feature stores, and LLM-based applications (e.g., SageMaker, Bedrock, AgentCore, Vertex AI, MLflow, Hugging Face)
Build and evolve scalable ML platforms, pipelines, and infrastructure that support reliable, repeatable model development and deployment across teams
Define infrastructure as code, CI/CD, and DevOps standards across engagements (e.g., Terraform, CloudFormation, GitHub Actions)
Drive performance, scalability, cost, and reliability optimization across deployed systems
Ensure architecture meets security, governance, and compliance requirements (e.g., GDPR, HIPAA, SOC2)
Lead cloud migrations and platform modernization initiatives
Set the standard for AI-forward engineering, using tools like Claude and Cursor with sophistication and helping the team adopt them effectively

Collaboration & Communication

Partner with senior leadership and clients as the principal technical voice on strategy and direction
Translate complex AI tradeoffs, risks, and opportunities into clear narratives that drive decision-making across technical and non-technical stakeholders
Lead design reviews and technical discussions, raising the bar for engineering rigor and constructive challenge across the team
Engage closely with engineering, data, AI, and product teams to align architecture with broader business priorities
Develop and maintain architecture documentation, standards, and guidelines

Leadership & Influence

Define and champion architectural standards and best practices across the engagements you support, bringing depth on tradeoffs, long-term implications, and responsible AI practices
Mentor and grow engineers at all levels, multiplying impact through coaching, code reviews, and pairing on hard problems
Own the most difficult architectural, integration, and agentic-system challenges, serving as the senior technical decision-maker and driving them through to production with care for reliability, cost, and safety
Evaluate emerging technologies — especially in the agentic AI and AWS ecosystems — and recommend tools, frameworks, and patterns that improve architecture over time

What You'll Bring

8+ years of software engineering experience, with at least 5 years in technical leadership roles and 4+ years focused on AI/ML systems in productionExpert software engineering background (Python or similar) with strong design sensibilities for scalable, maintainable systems
Deep, hands-on expertise designing and shipping production multi-agent agentic AI systems, including agent orchestration, planning, tool use, and multi-agent coordination patterns
Deep expertise with AWS, including in-depth knowledge of AWS GenAI offerings and hands-on experience with Amazon Bedrock AgentCore; broader multi-cloud experience (Azure, GCP) is a plus
Strong background in microservices, serverless, containers, and event-driven systems (e.g., Kubernetes, Docker, Lambda, EventBridge)
Proficiency with infrastructure as code and CI/CD (e.g., Terraform, CloudFormation, Pulumi, GitHub Actions)
Strong data architecture expertise across relational, NoSQL, and big data systems (e.g., PostgreSQL, MongoDB, Snowflake, BigQuery, Spark, Kafka)
Hands-on experience with data modeling, ETL/ELT pipelines, and orchestration (e.g., Airflow, Prefect, dbt)
Mastery of AI frameworks and orchestration tools for building agentic systems (e.g., LangChain, LangGraph, AgentCore, CrewAI, AutoGen, or equivalents)
Strong experience designing AI/ML systems for production, including LLMs, MLOps, and model serving (e.g., SageMaker, Bedrock, Vertex AI, MLflow, Hugging Face, PyTorch, TensorFlow)
Strong experience with evaluation frameworks and observability tools for LLM and agentic apps, including building these capabilities where they don't yet exist
Deep understanding of AI safety, responsible AI principles, prompt injection defenses, and PII handling
Extensive experience building RAG pipelines: chunking strategies, embedding models, vector databases, and advanced retrieval techniques
API design experience, including architecting and integrating with internal and third-party services at scale
Advanced cost optimization expertise: token economics, caching strategies, model routing, quantization
Solid understanding of networking, security, identity, and access management in cloud environments
Experience with governance, compliance, and observability frameworks
Track record of senior technical leadership and mentoring experienced engineers
Strong stakeholder communication skills, with the ability to translate technical depth across audiences
Demonstrable, day-to-day usage and expert knowledge of AI-forward coding tools such as Claude Code and Cursor
Multi-cloud architecture experience, AI ethics or responsible AI experience, or enterprise architecture certifications (e.g., TOGAF, AWS/Azure/GCP) is a plus

Our salary range is $180,375 – $230,625 USD