Azure Databricks Engineer

Our requirements

• Bachelor's degree or higher in Computer Science, Engineering or Science
• 8–10 years of experience, including 5+ years with Azure cloud-native services
• Strong hands-on experience with Azure Databricks, Delta Lake, Python / PySpark and Spark
• Knowledge of Lakehouse, Data Mesh, Data Virtualisation, dimensional modelling and data aggregation techniques
• Experience building distributed data platforms (e.g. Spark, Kafka, Flink, HBase) at large scale
• Hadoop-based implementations, SQL / T-SQL, Unix and Hive
• Azure Data Factory, ADLS Gen2 and Azure SQL Database
• Good understanding of RDBMS, Data Lake, Data Warehouse, Lakehouse and Medallion Architecture
• At least one workflow orchestration tool (Apache Airflow, Luigi, Oozie, AWS Glue or similar)
• Knowledge of Microsoft Fabric and basic understanding of data quality dimensions
• Fluent English
• Polish or EU citizenship, or a permanent residence / work permit valid for at least 12 months

Your responsibilities

• Design and implement big data solutions in Azure Databricks for batch and real-time analytics use cases
• Build Spark pipelines that populate downstream data marts for BI and generate feeds for Data Science
• Integrate real-time services that process business events from Kafka and persist them in ADLS Gen2 using Databricks
• Develop near real-time stream processing to derive features for ML model inference
• Optimise Databricks performance (liquid clustering, vacuum, Z-ordering etc.)
• Work with DevOps teams to drive operationally excellent infrastructure

About the project

This is how we organize our work

This is how we work on a project

• documentation
• issue tracking tools

• One interview with the IT services partner, followed by 1–3 interviews with the bank
• Technical tasks are done live: please have your programming environment set up and tested beforehand. The use of AI tools, coding assistants or code generators during interviews is not allowed
• The selected candidate will go through a background check by an external company (HireRight), covering identity, right to work, education and employment history, so the dates and employers in your CV must be accurate
• Your name in the application and CV must match your official ID document exactly (Polish citizens: please include all first names)
• Candidates who took part in a recruitment process for this IT services partner or this bank in the last 12 months cannot be considered

Development opportunities we offer

• external training
• mentoring

Recruitment stages

• 1.
  IT Partner & Bank Interviews
• 2.
  Technical Interview & Live Coding
• 3.
  Background Check
• 4.
  Identity Verification
• 5.
  Eligibility Review

OPTIVEUM SPÓŁKA Z OGRANICZONĄ ODPOWIEDZIALNOŚCIĄ