This is my profileContact via CazVidVedant
Senior Data Engineer
Contact
Competencies
Demonstrated experience with Azure Data Factory (ADF).
Demonstrated experience with ADLS Gen2.
Demonstrated experience with Azure Synapse Analytics.
Demonstrated experience with Delta Lake.
Demonstrated experience with Python.
Education
Summary
Senior Data Engineer open to new opportunities
Experience
Senior Data Engineer
Designed and orchestrated end-to-end Sales Analytics ETL pipeline using Azure Data Factory (ADF) and PySpark in Databricks, ingesting data from APIs, CSVs, and SQL sources into Azure Data Lake.. Implemented incremental loads in ADF and Databricks using watermarking and Delta Lake MERGE to efficiently capture new and updated records.. Enhanced performance of PySpark ETL jobs.. Developed checkpointing and retry mechanisms in PySpark jobs to ensure reliability during large-scale data loads.. Built monitoring and alerting mechanisms for ETL pipelines using ADF and Databricks, proactively detecting failures and anomalies.. Optimized SQL queries in Databricks using built-in performance tuning strategies.. Integrated CI/CD pipelines in Azure DevOps for Databricks & Power BI deployments.. Automated SharePoint data retrieval using Microsoft Graph API and Python, processed with PySpark in Databricks to maintain up-to-date tables for analytics and reporting.
Data Engineer
Implemented incremental refresh in Databricks notebooks.. Built ETL pipelines in Azure Data Factory and Databricks to load data from CSV files and SQL databases into Azure Data Lake.. Developed transformation scripts (PySpark + SQL) in Databricks to clean and prepare raw data for reporting.. Created Gold layer notebooks in Databricks (SQL) for reporting and analytics.. Designed a PreCheck Validation Framework (SQL + Python) in Databricks to validate row counts and thresholds, ensuring reliable data loads.. Resolved SQL data quality issues, improving KPI accuracy and consistency across reports.