Sai

Senior Data Engineer

Building real-time trade data pipelines with Kafka and Spark Structured Streaming to handle equity and fixed-income trade events that risk teams need within minutes, not hours.. Bringing in Bloomberg and Reuters market data feeds and standardizing reference data across asset classes in Snowflake, which several downstream teams now pull from instead of maintaining their own copies.. Migrated legacy SSIS ETL pipelines to Azure Data Factory, improving scalability, maintainability, and cloud adoption.. Designed and optimized AWS Redshift data warehouse solutions for analytics and reporting workloads.. Utilized AWS Athena to perform serverless SQL analytics directly against data stored in Amazon S3.. Writing PySpark jobs that calculate intraday position exposures for market and counterparty risk — one of the highest-stakes pieces of data work on the team, since a bad number here gets noticed fast.. Designed event-driven data pipelines integrating Kafka with GCP Dataflow for near-real-time ingestion and processing.. Working with the regulatory reporting group on the data feeding Dodd-Frank and MiFID II transaction reports.. Integrated Azure Data Factory with Azure Databricks to orchestrate complex ETL and ELT workflows.. Developed PySpark-based Azure Databricks pipelines for data cleansing, transformation, aggregation, enrichment, and validation.. Built a Python-based reconciliation framework comparing trade capture against front- and back-office systems, which has already flagged several mismatches before they reached reporting.. Handling end-of-day settlement files from custodians and clearing houses through Glue and Lambda-based workflows.. Set up row-level security and masking on trade and client data in Snowflake as part of an internal data governance push.. Designed end-to-end Azure ETL/ELT pipelines integrating ADF, ADLS Gen2, Databricks, Synapse, SQL Server, Oracle, and REST APIs.. Azure DevOps CI/CD pipelines to automate deployment of ADF pipelines, Databricks notebooks, and data engineering components.. Implemented secure access to GCP data resources using IAM roles, service accounts, and least-privilege access controls.. Tuned Snowflake warehouse sizing and clustering keys on the largest trade tables after query costs on the risk dashboards started climbing, which brought both runtime and spend back down.. Reworked a few of the slower PySpark transformation stages — mostly fixing skewed joins and unnecessary shuffles — after they started missing the intraday SLA window.. Implemented AWS Glue Data Catalog to maintain metadata and support centralized data discovery.. Built automated AWS ETL/ELT workflows integrating S3, Glue, Lambda, Redshift, and downstreamBuilt a data quality framework in Python that checks row counts, null rates, and value ranges against expected thresholds before trade data is released downstream, catching bad loads that used to surface as complaints from risk analysts instead.. Running Airflow for the end-of-day batch jobs, with SLA alerting tuned tightly given how time-sensitive the downstream reporting.. Used Azure DevOps and Git for source control, branching strategies, code reviews, and automated deployment across environments.. Implemented automated data quality and validation frameworks using Python, PySpark, and Azure Databricks.. Built Power BI dashboards so the platform team can check pipeline health and SLA status without pinging engineering directly.. Migrated legacy ETL workloads to GCP BigQuery, Cloud Storage, Dataflow, and Cloud Composer, improving scalability and maintainability.. First point of contact when trade data pipelines break during market hours, since delays here ripple into risk and settlement reporting almost immediately.. Designed and implemented GCP-based data pipelines using BigQuery, Cloud Storage, and Cloud Composer to support scalable trade and risk data processing.. Designed and architected scalable Databricks Lakehouse solutions using Delta Lake, Apache Spark, PySpark, and SQL for enterprise data platforms.. Implemented CI/CD using Git, Cloud Build, and Terraform for automated deployment of GCP data engineering components across environments. Integrated GCP Dataflow and Dataproc with existing Azure Databricks, Snowflake, and AWS data platforms for hybrid and multi-cloud processing.. Used Google Cloud Dataflow and PySpark for batch and near-real-time processing, integrating cloud data with existing Kafka and Snowflake environments.