Aniket

Data Engineer

Make python script to take data from database and put in s3 bucket. Wrote a Python script (using PyCharm IDE) to automate connections to AWS services and handle file movement, cleaning, and logging.. Used Pandas to clean and transform raw data (removing, renaming columns) before uploading to S3 Final Bucket.. Used Python with Boto3 to interact with AWS (upload/download/check files). Check whether Data is ingested in Landing Bucket or not. Worked with Business to understand requirements and propose technical implementation approach.. Developed data ingestion pipelines for different source systems.. Work with testing team on building test cases and provide support on testing phase.. Used SQL in Redshift to clean, transform, and validate the loaded data for analytics and reporting.. Work with Business during User Acceptance testing.. I create glue job to copy files from Landing S3 Bucket and paste it into Final S3 Bucket. Create IAM role for Amazon Glue and give it full access of S3, Glue. Run the glue job manually. Check whether files got copied in Final Bucket or not. Create Lambda Function to Trigger Glue job. Create separate IAM role for Lambda and give it full access of S3, Glue, Lambda. Upload file in Landing Bucket and check Lambda Job got automatically invoked or not and check glue job got triggered or not.. Create Redshift Cluster.. Use Query editor or any SQL Workbench to connect with Redshift Cluster.. Create Table inside Redshift Cluster.. Use copy command to copy data from S3 Final Bucket to Redshift table.. Check all data got inserted or not.