Job Overview

education: bachelor's or master's degree in computer science, information technology, engineering, or a related field.

experience: 8-10 years

• 8+ years of experience in data engineering or a related field.
• strong hands-on experience with azure databricks, spark, python/scala, cicd, scripting for data processing.
• experience working in multiple file formats like parquet, delta, and iceberg.
• knowledge of kafka or similar streaming technologies for real-time data ingestion.
• experience with data governance and data security in azure.
• proven track record of building large-scale data ingestion and etl pipelines in cloud environments, specifically azure.
• deep understanding of azure data services (e.g., azure blob storage, azure data lake, azure sql data warehouse, event hubs, functions etc.).
• familiarity with data lakes, data warehouses, and modern data architectures.
• experience with ci/cd pipelines, version control (git), jenkins and agile methodologies.
• understanding of cloud infrastructure and architecture principles (especially within azure).

technical skills:

• expert-level proficiency in spark, spark streaming, including optimization, debugging, and troubleshooting spark jobs.
• solid knowledge of azure databricks for scalable, distributed data processing.
• strong coding skills in python and scala for data processing.
• experience working with sql, especially for large datasets.
• knowledge of data formats like iceberg, parquet, orc, and delta lake.

leadership skills:

• proven ability to lead and mentor a team of data engineers, ensuring adherence to best practices.
• excellent communication skills, capable of interacting with both technical and non-technical stakeholders.
• strong problem-solving, analytical, and troubleshooting abilities.