Senior Data Quality Engineer

We are seeking a skilled Senior Data Quality Engineer to join our team. The successful candidate will play a crucial role in ensuring the integrity, reliability, and quality of our data engineering pipelines and platforms.
Responsibilities
Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies
Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms
Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes
Conduct and execute performance tests to assess the efficiency and scalability of data processing systems
Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives
Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues
Requirements
Minimum of 3 years of experience in Data Quality Engineering or a related field
Solid understanding of automated testing frameworks and data engineering principles
Proficiency with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs. Solid understanding of automated testing frameworks and data engineering principles
Experience in integrating testing frameworks with Azure DevOps and managing source control with Git
Knowledge of pipeline configuration and deployments utilizing Terraform
Familiarity with Jira
Strong analytical abilities with a proven track record in troubleshooting and resolving complex data and testing challenges
Fluency in English with a B2+ level
Nice to have
Experience with Power BI, Azure Data Lake, Spark Streaming, and other data processing and visualization tools
Prior experience in a data-intensive industry or with large-scale data environments
Proficient in implementing solutions through Infrastructure as Code (IaC)