Senior Data Engineer

We are hiring a Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role leverages AWS services to support Analytics and BI teams with reliable data delivery. Responsibilities include designing and maintaining batch and real-time pipelines that ingest data from APIs, databases, files, and external sources using Apache Kafka and AWS tools. Workflows are developed and orchestrated using Apache Airflow, with built-in monitoring, retries, and alerting to ensure system reliability. The engineer will build and optimize distributed processing solutions using PySpark, Spark SQL, and AWS EMR to handle large-scale data workloads. Management of Data Lake and Data Warehouse environments is required, utilizing Amazon S3 and Amazon Redshift to store and query structured data. Analytical data models, including Star and Snowflake schemas, are designed and implemented to support downstream use cases. Data preparation and optimization are performed to meet Analytics and Business Intelligence needs, particularly for Qlik integration. Solutions are developed using Python and SQL, adhering to best practices in Git version control, code reviews, testing, and thorough documentation. Data quality, security, governance, and compliance are prioritized through AWS IAM, encryption, and access control mechanisms. Production pipelines are actively monitored, with proactive identification and resolution of operational and performance issues to minimize downtime. The role requires proven experience in a Senior Data Engineer position or a comparable role with advanced technical expertise. Mandatory requirements include advanced proficiency in Python and SQL, along with hands-on experience using Apache Spark, PySpark, Apache Kafka, and Apache Airflow. Strong AWS proficiency is essential, especially in S3, Redshift, and EMR for large-scale data processing environments. Experience with ETL/ELT processes, data lakes, warehouses, and data modeling is required to ensure robust and scalable architectures. The candidate must understand both batch and streaming data architectures and be capable of optimizing performance across the entire data lifecycle. Familiarity with Git, testing methodologies, and production support practices is necessary to maintain code quality and system reliability. The position operates within a dynamic engineering team focused on delivering secure, compliant, and high-performance data solutions.

Source: Himalayas