Job Title: Azure Databricks Engineer
Experience: 8+ Years
Location: Hyderabad
Employment Type: Full-Time/ Onsite
Domain: Insurance
Job Summary:
We are seeking an experienced Azure Databricks Engineer with 8+ years of IT experience and strong expertise in building scalable data platforms on Azure. The ideal candidate should possess hands-on experience with Lakehouse architecture, Delta Lake, Azure Event Hub, and Databricks-based data engineering solutions. Experience in the Insurance domain is highly preferred. Databricks certification is mandatory.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using Azure Databricks.
- Implement and optimize Lakehouse architecture leveraging Delta Lake for high-performance analytics.
- Build batch and streaming data ingestion solutions using Azure Event Hub.
- Develop ETL/ELT workflows using PySpark, Spark SQL, and Databricks Workflows.
- Integrate data from multiple sources and ensure data quality, reliability, and governance.
- Optimize cluster performance, partitioning strategies, and Delta tables for efficient processing.
- Collaborate with data architects, analysts, and business stakeholders to deliver data solutions aligned with Insurance business requirements.
- Implement CI/CD processes and automate deployment pipelines.
- Monitor and troubleshoot production issues to ensure platform stability and performance.
- Follow best practices for security, scalability, and cost optimization within the Azure ecosystem.
Required Skills:
- 8+ years of overall IT experience with strong Data Engineering expertise.
- Hands-on experience with Azure Databricks and Apache Spark.
- Strong knowledge of Lakehouse Architecture and Delta Lake.
- Experience with Azure Event Hub for real-time data ingestion.
- Proficiency in PySpark, Spark SQL, Python, and SQL.
- Experience with Azure services such as ADLS Gen2, Azure Data Factory, Key Vault, and Azure Monitor.
- Knowledge of batch and streaming data processing concepts.
- Experience in performance tuning and optimization of Databricks workloads.
- Familiarity with Git, CI/CD pipelines, and DevOps practices.
- Excellent analytical, troubleshooting, and communication skills.
Preferred Qualifications:
- Experience in the Insurance domain.
- Exposure to real-time analytics and event-driven architectures.
- Knowledge of data governance and security frameworks.
- Experience with Agile/Scrum methodologies.
Mandatory Certification
- Databricks Certified Data Engineer Associate or Professional Certification (Mandatory).