Abhijeet

Data Engineer

Designed and implemented scalable ETL solutions, processing 10M+ structured records from diverse upstream sources, with orchestration and monitoring via SQL Server and ADF.. Migrated PySpark-based legacy workflows from Dataiku to Azure Databricks, implementing Medallion Architecture to improve pipeline performance, scalability, and cost efficiency.. Wrote clean, reusable, and testable PySpark, Spark SQL, and Python (Pandas, NumPy) logic to build analytical data solutions, including complex queries, transformations, and data profiling.. Monitored and troubleshot production ETL workflows, investigating data failures, job execution issues, and downstream data discrepancies to maintain reliable data processing and reporting.. Developed scalable Databricks-based data pipelines to preprocess structured and unstructured customer reviews and social media comments, leveraging LLM-powered classification to generate business categories for downstream analytics and reporting.. Optimized complex SQL procedures and ETL jobs through indexing, query tuning, and efficient transformation logic.. Developed and optimized SQL-based data validation and reconciliation processes across data warehouse layers, validating source-to-target data, identifying discrepancies, and improving data quality while reducing manual troubleshooting effort.. Collaborated daily with stakeholders to lead and automate Meltwater licensing management using Azure IAM, integrating the solution into the client's system to strengthen data governance and access control.. Collaborated with the data science team to curate, transform, and structure social-listening data for an AI-powered chatbot, enabling business users to retrieve relevant brand and consumer insights through natural-language queries.. Automated Power BI reporting workflows by eliminating manual refreshes and data preparation steps, reducing analyst intervention and delivery time.