Senior Data Engineer

Our requirements

• 5+ years of experience in data engineering or a related field, with a strong focus on ETL processes and cloud data tools.
• Bachelor’s degree in Computer Science, Engineering or a related field (Master’s preferred).
• Required certifications: Certified Azure Data Engineer and Databricks Certified Data Engineer.
• Proficiency in Python, Java or Scala.
• Hands-on experience with MS Cloud SQL Server, Azure Data Factory, Azure Databricks, PySpark, Spark SQL, ADLS, SQL, Power BI, SSIS and SSRS.
• Strong SQL knowledge and experience with relational databases.
• Familiarity with big data technologies and data processing frameworks.
• Significant expertise in the Financial Services domain, ideally in a captive finance company, with in-depth knowledge of local, regional and global financial processes and of data related to customers, dealers and suppliers, including personal data.
• Knowledge of the client’s infrastructure and security policies.

Your responsibilities

• Extract, combine, transform, load and cleanse data from various sources to deliver high-quality datasets for analysis, including data migration from legacy systems.
• Work with data modelers and business analysts to understand the data model and source-to-staging mapping.
• Create data structures suitable for analysis, including DDLs for MS Cloud SQL Server tables and views in the Pond database (a subset of the Data Lake).
• Design and develop Databricks notebooks that extract data from input files and join related entities into Shared File Formats (SFFs).
• Configure Ingestion Framework tables to load staging and Data Lake Core (DLC) tables using the existing framework.
• Build large-scale batch and real-time data pipelines in cloud or on-premises environments, and develop performance-efficient algorithms for analytical and data platforms.
• Apply CI/CD best practices and recommend and implement automation in existing and future integration flows.
• Track data consumption patterns, cooperate with data governance and information stewards, and help vet and promote content for the curated data catalog.
• Drive improvements in data acquisition, testing and tooling to raise data quality.
• Work in cross-functional agile teams, take part in requirement analysis and project estimates.
• Monitor performance and optimize SQL queries.
• Prepare detailed design documents with prototypes and screenshots based on Business Requirement Documents and System Requirement Documents.

About the project

What we offer

• Work 100% onsite at Wroclaw office.

Benefits

• sharing the costs of sports activities
• private medical care
• life insurance
• integration events

Recruitment stages

• 1.
  Rozmowa z rekruterem ok. 15 min
• 2.
  Spotkanie rekrutacyjne z klientem ok. 60 min

IT CONNECT Sp. z o.o. Sp. k.