Data Engineer (PySpark)

Our requirements

• 2–4 years of experience as a Data Engineer or in a similar role.
• Practical, hands-on experience with PySpark / Apache Spark.
• Solid understanding of key Spark concepts, including partitioning, shuffle, repartition vs coalesce, broadcast joins, and data skew impact.
• Practical experience with AWS services, especially AWS Glue, Amazon S3, and Amazon CloudWatch.
• Experience building, maintaining, or enhancing ETL/data processing pipelines.
• Ability to diagnose common Glue and PySpark issues from logs and basic metrics.
• Experience handling ETL error scenarios: retries, reprocessing, partial failures, and preventing duplicate processing.
• Basic understanding of idempotency and safe restart strategies after failures.
• Experience processing large data volumes or many files and familiarity with batch migration processes.
• Strong SQL skills and familiarity with joins, aggregations, filtering, grouping, and ranking.

Your responsibilities

• Implement data transformations and ETL logic using PySpark.
• Maintain and develop existing AWS Glue jobs and related pipelines.
• Investigate and resolve pipeline failures using logs, error messages, and basic metrics.
• Diagnose and mitigate Spark performance issues (partitioning, shuffle, skew, joins).
• Choose appropriate partitioning and join strategies for given workloads.
• Handle error scenarios in ETL processes, including retries, reprocessing, and partial failures to avoid duplicate processing.
• Work with large volumes of data and many files without creating uncontrolled parallelism.
• Follow established architectural patterns and implement solutions that integrate with the current system design.
• Support data migration or batch processing tasks as needed.

About the project

This is how we organize our work

This is how we work on a project

• team-level deployment
• documentation
• issue tracking tools
• testing environments

What we offer

• Remote working.
• Unique TEAL culture, relationship- and respect-driven community, non-corporate atmosphere.
• Agile approach and no bureaucracy.
• Outstanding integration trips to various places in Europe.
• Activities to support your well-being and health.
• Luxmed Gold Extended medical care and Multisport Plus benefit.

Benefits

• sharing the costs of sports activities
• private medical care
• remote work opportunities
• integration events

Recruitment stages

• 1.
  Screening Call
• 2.
  Cultural Fit Interview
• 3.
  Technical Interview
• 4.
  Final Round Interview

YOUR ITEAMS sp. z o.o.