AI/ML Data Engineer (Databricks)
QuidelOrtho is a leading in vitro diagnostics company formed by the merger of Quidel Corporation and Ortho Clinical Diagnostics. They are seeking an AI/ML Data Engineer to design, build, and optimize data pipelines and infrastructure using Databricks to support AI and machine learning initiatives, working closely with business stakeholders to translate requirements into technical solutions.
Responsibilities
- Work directly with business stakeholders to identify and define AI/ML use cases, translating business needs into technical requirements
- Design, develop, and optimize scalable data pipelines in Databricks for AI/ML applications, ensuring efficient data ingestion, transformation, and storage
- Build and manage Apache Spark-based data processing jobs in Databricks, ensuring performance optimization and resource efficiency
- Implement ETL/ELT processes and orchestrate workflows using Azure Data Factory, integrating various data sources such as Azure Data Lake, Blob Storage, and Microsoft Fabric
- Collaborate with Data Engineering teams to meet data infrastructure needs for model training, tuning, and deployment within Databricks and Azure Machine Learning
- Monitor, troubleshoot, and resolve issues within Databricks workflows, ensuring smooth operation and minimal downtime
- Implement best practices for data security, governance, and compliance within Databricks and Azure environments
- Automate data and machine learning workflows using CI/CD pipelines through Azure DevOps
- Maintain documentation of workflows, processes, and best practices to ensure knowledge sharing across teams
- Perform other work-related duties as assigned
Skills
- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience)
- 1-3 years of experience in data engineering, with a strong focus on Databricks and AI/ML applications
- Proven experience working directly with business stakeholders to identify and implement AI/ML use cases
- Expertise in Apache Spark and hands-on experience with Databricks for building and optimizing data pipelines
- Strong programming skills in Python and Scala for data engineering and machine learning workflows in Databricks
- Experience with Azure Data Factory, Azure Data Lake, Azure Blob Storage, and Azure Synapse Analytics
- Proficiency with Databricks Delta Lake for data reliability and performance optimization
- Familiarity with MLflow and Databricks Runtime for Machine Learning for model management and deployment
- Knowledge of Azure DevOps for implementing CI/CD pipelines in Databricks-based projects
- Strong understanding of data governance, security practices, and compliance requirements in cloud environments
- Familiarity with emerging Databricks features such as Delta Live Tables and Unity Catalog
- Ability to travel up to 5-10%
- This position is not currently eligible for visa sponsorship
- Experience with real-time data processing using Apache Kafka or Azure Event Hubs
- Master's degree in Computer Science or related technical fields
Benefits
- Bonus eligible
- Medical insurance
- Dental insurance
- Vision insurance
- Life insurance
- Disability insurance
- 401(k) plan
- Employee assistance program
- Employee Stock Purchase Plan
- Paid time off (including sick time)
- Paid Holidays
Company Overview