Sr. Machine Learning Engineer

bout You

• You have 7+ years of experience working with Machine Learning (ML) in production (with at least 1-2 years experience specific to Large Language Models (LLM)). You have experience writing clean Python code with reproducible results. You have knowledge of ML/LLM data structures and modeling, software architecture, libraries and frameworks to create models that sufficiently meet the needs of project goals.
• You combine your strong applied ML/LLM knowledge with statistical expertise (e.g. confidence intervals, regression modeling, significance testing). You have strong mathematics skills, especially in statistics, to create algorithms. This includes demonstrated experience with common ML techniques for deep data mining and exploration (e.g. predictive modeling, time series modeling, classification, and clustering techniques).  Understand how to determine the right machine learning algorithms for a given task and evaluate their performance, including understanding the trade-offs between different models and selecting the most suitable one based on the data and problem at hand.
• You have a strong understanding of how LLMs and ML models succeed in a production environment. You have experience developing and improving upon evaluation frameworks for LLMs which focus on performance, reliability, and bias assessment/mitigation. You have extensive experience working with user data to engineer features for models, select appropriate model architectures, tune model parameters, and evaluate model quality. You have familiarity with fine-tuning models to achieve their maximum performance, i.e. optimizing hyperparameters, handling overfitting, A/B testing, and improving the efficiency of algorithms.
• You’re familiar with various prompt techniques for LLM grounding such as chain-of-thought, static few-shot examples, and dynamic few-shot. You’re familiar with Retrieval-Augmented Generation (‘RAG’) systems. You have an understanding of how to optimize knowledge retrieval for improved model accuracy and speed.  You can apply your knowledge of different indexing/chunking strategies on the data to achieve desired goals. You understand semantic search and vector databases, and how they differ from classic retrieval methods and databases. Experience with reranking and knowledge graphs is a plus.
• You have a strong engineering sense. You’re familiar with automating ModelOps/DataOps processes which include creating pipelines and understand how CI/CD processes work.  You’re a stellar data wrangler and have experience cleaning and structuring data for modeling purposes.
• You are a skilled written communicator. Zapier is a 100% remote team and writing is our primary means of communication. You communicate complex technical topics clearly and in an approachable way.
• You have proven stakeholder management expertise. As a collaborative thought partner, you’re gifted at explaining your findings clearly to a non-technical audience. You have experience partnering with product and engineering to deliver business value through scalable ML/LLM development and deployment.
• You enjoy collaboration and knowledge sharing. You appreciate our team’s values of eagerness to collaborate with teammates with any level of statistical and ML/LLM knowledge, iterating over your deliverables, and being curious.
• You are an out of the box thinker. You employ your creative thinking skills to come up with new solutions and approaches that meet business objectives. You have sound problem-solving skills to refine prototypes and troubleshoot performance issues. You remain up to date on the latest innovations in machine learning in order to develop solutions that scale.
• You understand that perfect is the enemy of good. You will default to action by initially shipping solutions that simply work and work simply while iterating as needed.
• You’ve used AI tooling for work or personal use—or you are willing to dive in and learn fast. You explore new tools, workflows, and ideas to make things more efficient, and are eager to deepen your understanding of AI and use it regularly.

Things You’ll Do

Zapier is a fast-growing and remote-first company, so you’ll get experience on many different projects to support our stakeholders. Here are some things you might get to help our teams with:

• Work with Engineering teams to develop and improve ML and LLM models, as well as refine our systems and processes in the space.
• Design, implement, deploy, and improve upon prompting for LLMs with comprehensive evaluation suites to assess model performance and reliability.
• Collaborate with other Data teams to develop ML projects in various sectors of the business.
• Create LLM and ML model prototypes based on project specifications.
• Process data for model fine tuning and fine-tune models for maximum performance that meet business needs.
• Build and deploy data models as well as RAG systems.
• Implement changes to algorithms to improve ML model performance as well as troubleshoot and address problems with deployed ML models to improve user experience.
• Develop data literacy programs and provide easy-to-use tools with clear documentation, examples, and tutorials.
• Participate in the on-call rotation (about 1 week per quarter) to support our production services.

Our stack is best summed up by: Databricks, AWS(S3), Python/Typescript, Airflow, dbt, Kafka, Braintrust.