Principal AI Engineer, Agentic Systems and LLM Platforms

Why You’ll Actually Care

District Partners is recruiting for an established leader in commercial finance that is reinvesting in itself, building a dedicated AI transformation team and a proprietary agentic platform to power new products and service lines. The architecture decisions being made now, for how agents call tools, hand off work, route across models and pause for human review, will shape everything the firm builds with AI for years. You will set those patterns, own the hardest initiatives end to end and raise the bar for every engineer who joins after you. The work goes straight into high-value production processes, so the systems you design get real users and real stakes quickly. It is principal scope on a greenfield AI platform, backed by the data, capital and executive commitment most AI engineers never get.

The Basics

The company is a long-established leader in its segment of financial services that treats technology as a growth engine, not a cost center. Its AI transformation team runs like an in-house R&D lab, building a shared agentic platform alongside AI products aimed at high-value, document-intensive work such as intake, review and ongoing servicing. The firm also runs an internal AI collaboration platform used across the business. This is where the company’s next products and service lines are being built.

What You’ll Be Owning

• End-to-end ownership of the most complex AI and platform initiatives, from architecture and design through production rollout and ongoing reliability
• Production agentic workflows that automate intake, document review and servicing, turning slow, manual processes into fast, auditable systems
• Reusable platform patterns and interfaces that other teams build on, including tool servers, agent-to-agent handoffs and human-in-the-loop checkpoints
• A model selection and routing strategy that balances cost, latency and quality across a growing set of use cases
• Evaluation, testing, observability and guardrail practices built for non-deterministic systems, so the firm can ship AI with confidence
• A stronger engineering bench, through design reviews, technical direction and hands-on mentorship of senior engineers

The Environment

This is a senior individual contributor role on a newly formed team, with high autonomy and minimal oversight. Multiple teams will depend on the patterns you set, so your architectural decisions carry weight across the organization. The business is regulated and the processes are high value, which raises the bar for guardrails, observability and auditability. You will be expected to make technical calls independently and bring others along with them.

What You Bring to the Table

• 12+ years in software development, including senior or staff-level individual contributor work on production systems
• Deep, hands-on production experience with machine learning systems, LLM applications and agentic workflows, beyond prototypes
• Experience with modern agent frameworks and interoperability protocols for tool use and agent-to-agent communication
• Strong context engineering across retrieval-augmented generation, embeddings, search and reranking
• A practical approach to evaluation, observability and guardrails for AI systems, including testing non-deterministic behavior
• Expert Python and modern backend API development, strong SQL and database design, and experience with event sourcing and domain-driven design
• Bachelor’s or master’s degree in computer science or a related field, or equivalent experience; financial services or capital markets background is a plus

A Few Things Worth Knowing

• Compensation: $245K to $305K base, plus discretionary bonus and full benefits including 401(k) match
• Location: Remote, US based
• Work model: Fully remote
• Engagement type: Full-time, direct hire