Big Data Software Engineer [Senior/Lead]

We are looking for an experienced Senior/Lead Big Data Software Engineer to join our innovative team.
The ideal candidate will support the development of a customer-centric strategy designed to increase automation and the use of data and analytics throughout the customer journeys. The candidate will be responsible for identifying relevant data and utilizing data tools, technologies, and processes to develop continuous, data-driven, and automated customer communications across marketing and servicing.
Responsibilities
Create and manage AWS cloud resources
Implement data ingestion and processing using Big Data technologies from diverse data sources such as RDBMS and REST HTTP API
Carry out data transformation and processing utilizing Spark and other cloud services to support business logic
Develop and uphold automated data quality checks to ensure data accuracy and integrity
Build infrastructure for collecting, transforming, combining, and distributing customer data
Explore process improvement opportunities for optimal data collection and insight generation
Guarantee that data solutions are scalable, efficient, accurate, and flexible
Analyze complex data sets to identify patterns and trends
Design frameworks for data visualization to present analytical results to stakeholders
Actively engage in Scrum ceremonies with agile teams
Develop queries, produce reports, and present findings
Mentor junior team members and promote best industry practices
Requirements
5+ years of experience in data engineering within consumer finance or related fields
Demonstrated expertise in Python, Pyspark, or AWS-specific technologies including Lambda, Airflow, and Athena
Production experience with Big Data tools such as HDFS, YARN, Hive, Spark, Kafka, and AWS
Proficiency in utilizing Docker, Kubernetes, and Snowflake in a production environment
Strong foundation in mathematics, statistics, computer science, data science, or similar
Proficiency in business intelligence, analytical tools, and data visualization techniques
Excellent organizational skills with the ability to manage multiple projects and meet deadlines
Effective communication skills to articulate results to non-technical audiences
Continuous learning and integration of new technologies into solutions
Nice to have
AWS certification
Experience with Kafka Streaming and Kafka Connect
Familiarity with ELK Stack
Background in managing Cassandra or MongoDB databases