Staff Software Engineer, Data Ingestion

Remote role Full-time Open position

The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible fordesigning collection strategies, developing, and maintaining robust and scalable data pipelines.This role is at the heart of our data ecosystem, deliver new analytical software solution to access timely, accurate, and complete data for insights, products, and operational efficiency.

Key Responsibility

Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python.

Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.).

Implement data transformation and cleansing logic during ingestion to ensure data quality.

Monitor and troubleshoot data ingestion pipelines, identifying and resolving issues promptly.

Collaborate with database engineers to optimize data models for fast consumption.

Evaluate and propose new technologies or frameworks to improve ingestion efficiency and reliability.

Develop and implement self-healing mechanisms for data pipelines to ensure continuity.

Define and uphold SLAs and SLOs for data freshness, completeness, and availability.

Participate in on-call rotation as needed for critical data pipeline issues.

Required Skills

6+ years experience in software development industry from computer science background

Extensive Python Expertise: Extensive experience in developing robust, production-grade applications with Python.

Data Collection & Integration: Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).

Distributed Systems & Scalability: Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.

Cloud Platforms: Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).

Database Fundamentals: Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)

Monitoring & Alerting: Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.

Version Control: Proficiency with Git.

Containerization (Plus): Experience with Docker and Kubernetes.

Streaming Technologies (Plus): Experience with real-time data processing using Kafka, Flink, Spark Streaming

Apply to this Job

Apply

Staff Software Engineer, Data Ingestion

Key Responsibility

Required Skills

Further positions

Implementation Project Manager

Manager Quality Automation - BOT

Adobe AEM Solutions Manager, Presales

SF Developer

Manager, Accounts Payable

Senior Software Engineer – Real-Time Data Applications

CST Operations - Senior Associate/ Associate - Paytm Travel CST (Flights)

Backend Engineer-SSE

State Head - EDC - Bangalore

State Head - Chandigarh

Amazon Career Opportunities Work From Home ? At Careermilard

Experienced Bilingual Customer Service Specialist – Spanish/English: Unlock a Rewarding Career at blithequark

Experienced Part-Time Remote Data Entry Clerk – Flexible Typing Position with arenaflex

Virtual Educational Diagnostician

IoT Solution Architect Home Office, IoT Technical Sales Posted ago

Experienced Full Stack Data Analyst – Web & Cloud Application Development

Experienced Part-Time Remote Data Entry Clerk – Customer Service & Administrative Support with Flexible Hours at Blithequark

Experienced Amazon Remote Customer Service Representative - Work from Anywhere with a Leading E-commerce Company

Business Plan Writer (EB-5)

[Remote] Remote Customer Service Representative (Call Center) | Work From Home | Kansas Residents Only