Alb Log Parser is a tool built on serverless framework which parses the ALB logs and load them in Athena. Logs can be fetched by simple SQL query on Athena table.
This tool deploys 3 Lambda functions.
Copy ALB Logs
This function triggeres on the put event of S3 bucket where ALb logs are dumped. When ever a new file is pushed in the bucket from ALB, This func copies that file to project bucket which is created at the time of deployment. It then creates the DB and table in Athena.
Repair Disk Athena
This function triggeres once in a day at 12:06 am. Athena creates a new partition for every year then every month and day. Eg if date is 17/02/2018, log file path will be 2018/02/17/{filename}. So every day new partition is created. This function is responsible for loading that new partition by executing a query in athena.
Fetch Data From Athena
This function triggeres after every 5 minutes as ALB push logs every 5 minutess. It executes query to fetch data(by default non 200 http status urls) and post them to logz io.
By Amboj Goyal and Karandeep Singh Johar
