Descrição de Vaga
| Código: | 14958 |
| Título da vaga: | AWS Data Engineer |
| Local: | São Paulo, SP |
| Região | Outra |
| Nível Profissional: | Analista |
| Nível Acadêmico: | Ensino Superior Completo |
| Áreas de Atuação Profissional: | TI - Projetos |
| Descrição: | Project Description We are looking for an experienced AWS data engineer, who will join the team responsible for the second phase of data warehouse cloud migration project. The initial "lift-and-shift" of an on-premise MS SQL data warehouse to AWS RDS has been completed. This next phase focuses on refactoring the legacy system into a modern, serverless data architecture. The primary goal is to migrate all data transformation logic from MS SQL to AWS Glue and prepare the data for consumption by business intelligence tools like AWS QuickSight. Key Responsibilities • Analyze existing MS SQL stored procedures and data structures within the PMM data warehouse to understand the current business and transformation logic. • Design, develop, and deploy scalable and efficient ETL/ELT pipelines using AWS Glue and PySpark to replace the legacy SQL processes. • Ingest and process data from various sources currently landing in Amazon S3, including SAP extracts, operational system feeds, and flat files. • Implement robust data validation, error handling, and logging mechanisms within Glue jobs. • Collaborate with architects to define data models and schemas suitable for the new data lakehouse architecture. • Optimize Glue jobs for performance and cost-efficiency. • Document the new data pipelines and processes. Skills • Must-Haves: o High proficiency in AWS Glue, including hands-on experience writing and deploying PySpark scripts. o Strong experience with Python and the Spark framework (PySpark), particularly for data manipulation and transformation. o Knowledge of MS SQL Server, including the ability to read, understand, and reverse-engineer T-SQL, stored procedures, and functions. o Hands-on experience with Amazon S3 as a data source and target. o Good understanding of IAM roles and policies for managing secure access to AWS resources. o Strong proficiency in English |
| Habilidades: | • Highly Desirable: o Experience with AWS data orchestration services like AWS Step Functions and Lambda. o Familiarity with data warehousing concepts and dimensional modeling. o Knowledge of monitoring and logging using Amazon CloudWatch. o Experience working with data from SAP systems is a significant plus. Atuação remota |