Job Overview

key responsibilities

• design and implement streaming data pipelines integrating kafka with databricks using structured streaming.

• architect and maintain medallion architecture with well-defined bronze, silver, and gold layers.

• implement efficient ingestion using databricks autoloader for high-throughput data loads.

• work with large volumes of structured and unstructured data, ensuring high availability and performance.

• apply performance tuning techniques such as partitioning, caching, and cluster resource optimization.

• collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions.

• establish best practices for code versioning, deployment automation, and data governance.

required technical skills:

• strong expertise in azure databricks and spark structured streaming

• 7+ years' experience in data engineering

• processing modes (append, update, complete)

• output modes (append, complete, update)

• checkpointing and state management

• experience with kafka integration for real-time data pipelines

• deep understanding of medallion architecture

• proficiency with databricks autoloader and schema evolution

• deep understanding of unity catalog and foreign catalog

• strong knowledge of spark sql, delta lake, and dataframes

• expertise in performance tuning (query optimization, cluster configuration, caching strategies)

• must have data management strategies

• excellent with governance and access management

• strong with data modelling, data warehousing concepts, databricks as a platform

• solid understanding of window functions.