Alexandre is available for hire

Alexandre França de Magalhães

Verified Expert in Engineering

Data Warehousing Developer

Location

Salvador - State of Bahia, Brazil

Toptal Member Since

February 4, 2022

Alexandre是一位拥有超过六年专业经验的高级数据工程师. 他的主要经验是设计和构建数据湖和数据仓库，并利用可用资源处理数据, such as Spark, SQL from their databases, Pandas, etc. Alexandre熟悉Azure和AWS堆栈，但对使用其他云持开放态度.

Data Engineering Data Modeling Data Warehousing Azure Data Lake SQL Python Data Pipelines REST APIs Spark SQL Spark Databricks Apache Airflow PySpark Pandas Data Cleaning Data Lakehouse MLflow

Portfolio

PepsiCo

数据工程、Python、广告、媒体、Azure、OTT...

BCG - Gamma

SQL, PySpark，自动化，Azure，亚马逊网络服务(AWS)，微软Azure...

Via Varejo

Azure, Databricks, Azure数据工厂，Azure数据湖，Azure Synapse...

Experience

Data Pipelines - 5 years Data Warehousing - 5 years Data Engineering - 5 years Data Modeling - 5 years SQL - 5 years Apache Airflow - 3 years PySpark - 3 years Databricks - 3 years

Availability

Part-time

Preferred Environment

Spark SQL, Spark, SQL, Azure, Databricks, Python, Amazon Web Services (AWS), Apache Airflow, Azure Data Factory, Amazon Elastic MapReduce (EMR)

The most amazing...

...我开发的项目是一个基于云的数据湖架构, on-premise, and API data sources.

Work Experience

Data Engineer

2022 - PRESENT

PepsiCo

从零开始开发百事公司的全球媒体数据仓库, 目标是将所有媒体测量数据集中在一个公司和集中的环境中. 这些源混合在api、ODBC/JDBC和云存储之间.
致力于PySpark代码优化，以提高性能和标准化.
在Databricks上使用MLflow开发简单的机器学习模型来跟踪性能, metrics, and artifacts.

Technologies: 数据工程、Python、广告、媒体、Azure、OTT, Databricks, Samba, Roku, YouTube, Dynamic Data, Snowflake, APIs, Azure Data Factory, Data Warehousing, Data Lakes, Azure Data Lake, Amazon S3 (AWS S3), Google Cloud Storage, Azure Blobs, PySpark, Spark, MLflow, Machine Learning

Lead Data Engineer

2022 - 2022

BCG - Gamma

与数据科学家一起开发数据管道，使实验产品化, data extractions, data modeling, data cleaning, 以及对多个云环境的质量检查.
处理大型数据集，使用Spark作为处理工具.
Developed SQL queries to query, 在多个平台上分析和操作数据, such as Spark, Hive, and relational data sources.

Technologies: SQL, PySpark，自动化，Azure，亚马逊网络服务(AWS)，微软Azure, Apache Airflow, Pandas, Docker, Amazon S3 (AWS S3), Spark, Python, Parquet, CSV, JSON, Delta Lake, CI/CD Pipelines, Azure Blobs, Data Extraction, Data Cleaning, Hue, Apache Hive, Amazon Elastic MapReduce (EMR)

Senior Data Engineer

2021 - 2022

Via Varejo

重构欺诈分析管道，为2021年黑色星期五的业绩改进做好准备, 在增加的批处理数据负载上实现恒定的执行时间.
参与公司欺诈数据市场的开发工作.
开发各种管道，解决摄取和数据处理需求.

Technologies: Azure, Databricks, Azure数据工厂，Azure数据湖，Azure Synapse, Azure Event Hubs, Data Engineering, Data Warehousing, Data Modeling, ETL, ETL Tools, SQL, Data Management

Senior Data Engineer

2021 - 2021

Radix

为关系数据源开发了通用的摄取数据管道, 使用简单的配置文件加速新摄取的过程.
从头开始开发Delta Lake架构，用于安全高效的数据处理.
在Azure Synapse平台上开发和维护企业数据仓库.

Technologies: Azure, Azure Data Factory, Databricks, Azure Data Lake, Azure Synapse, Google Cloud Storage, Data Engineering, Data Modeling, Data Warehousing, ETL, ETL Tools, SQL, PySpark, Synapse, Apache Kafka, Microsoft Azure, Amazon Web Services (AWS)

Senior Data Engineer

2020 - 2021

Bridgestone

支持和增强的企业数据湖，构建在Azure云服务上，具有内部部署数据源, such as SQL Server, Oracle, and Kafka streams for sensor data.
使用SQL、PL/SQL和T-SQL为数据管道开发SSIS包.
管理负责现场软件和数据支持需求的第三方团队.

Technologies: Azure, Azure Data Factory, Databricks, Oracle, Pandas, SQL Server 2016, Data Engineering, Data Warehousing, ETL, ETL Tools, Data Modeling, SQL, PySpark, Oracle PL/SQL

Software Developer

2019 - 2020

Chemtech

开发数据提取和功能，帮助数据科学团队训练和验证基于Python构建的机器学习模型, Pandas, 和Scikit-learn技术，用于公司的各种项目.
为多个客户公司开发数据管道，以服务于数据湖和数据仓库架构.
使用Jira作为报告工具跟踪和开发用户历史.

Technologies: Oracle, SQL Server 2016, Python, Pandas, ETL Tools, ETL, Data Engineering, Data Warehousing, Data Modeling, Oracle PL/SQL, T-SQL (Transact-SQL), Data Pipelines, Microsoft Azure, Amazon Web Services (AWS), HDFS, Spark

Software Developer

2017 - 2019

Braskem

为企业数据仓库中的数据ETL开发SQL脚本.
为满足业务分析师需求的生产报告创建复杂查询.
开发用于制造执行系统的c#后端应用程序.

Technologies: Python, SQL, Oracle, SQL Server 2016, MongoDB, Data Warehousing, ETL Tools, ETL, Data Engineering, Data Modeling, Oracle PL/SQL, Amazon Web Services (AWS), Microsoft Azure, Data Pipelines, Pandas, C#

Experience

Data Lakehouse For an Educational Company

我为各种数据源(如本地Oracle和SQL Server数据库)开发了一个带有动态可配置摄取管道的Azure数据湖和数据仓库, Google Cloud Platform (GCP) storage, and external API providers. 云基础设施以前是不存在的, 因此，我为每个数据源建立了分层存储模式和分区, 使用数据湖配置在Databricks平台上处理大部分数据.

Data Lake for Rubber and Tire Industry

将企业数据湖从内部部署Hadoop迁移到Azure ADLS2. 开发了用于数据摄取的管道，并开始了数据仓库解决方案的数据建模，其中使用Spark Databricks执行处理. 数据仓库托管在一个专用于突触的SQL池上.

零售企业欺诈检测管道的重构

我重构了一个现有的欺诈检测管道，该管道用于使用ML模型分析信用卡购买的异常行为. 问题的主要根源在于许多计算可以并且应该并行执行. 有了这个变化，还需要重新考虑计算值. 这将执行时间从两个小时减少到20分钟, consistent even on last year's Black Friday.

Education

2013 - 2018

Bachelor's Degree in Engineering

巴伊亚-萨尔瓦多联邦大学，巴西

Certifications

SEPTEMBER 2022 - PRESENT

Certified Data Engineer Associate

Databricks

Skills

Libraries/APIs

PySpark, Pandas, REST APIs

Tools

Spark SQL, Apache气流，Synapse, Hue, Amazon Elastic MapReduce (EMR)

Frameworks

Spark, Hadoop, Data Lakehouse

Languages

SQL, Python, T-SQL (Transact-SQL)，批处理，c#，雪花

Paradigms

ETL, Automation, Samba

Platforms

Azure, Databricks, Oracle, Azure Synapse, Azure Event Hubs, Apache Kafka, Amazon Web Services (AWS), Docker, YouTube

Storage

Data Pipelines, SQL Server 2016, Oracle PL/SQL, MongoDB, Data Lake Design, Data Lakes, HDFS, Amazon S3 (AWS S3), JSON, Azure Blobs, PostgreSQL, Apache Hive, Google Cloud Storage

Other

Azure Data Factory, Azure Data Lake, Data Engineering, Data Modeling, ETL Tools, Data Warehousing, Data Management, Data Cleaning, Microsoft Azure, Streaming, Parquet, CSV, Delta Lake, CI/CD Pipelines, Data Extraction, Advertising, Media, Over-the-top Content (OTT), Roku, Dynamic Data, APIs, MLflow, Machine Learning

Collaboration That Works

How to Work with Toptal

在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.

Share your needs

在与Toptal领域专家的电话中讨论您的需求并细化您的范围.

Choose your talent

在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.

Start your risk-free talent trial

与你选择的人才一起工作，试用最多两周. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring