Pursuant
How MEV Automated ETL Pipelines with AWS Glue, EventBridge, SNS, and Lambda
Client:
Pursuant
Industry:
Fundraising and marketing
Focus:
Event-based ETL pipeline automation / AWS service orchestration / Scheduled reporting data refresh
Technologies:
AWS EventBridge / AWS Lambda / AWS DMS / Amazon SNS / AWS Glue Crawler / AWS Glue Jobs
[ about the client/ ]
Pursuant is a fundraising and marketing agency that helps nonprofit organizations improve donor engagement through analytics and direct response services. Its account teams relied on internal reports built from client data such as donor demographics, donation amounts, donation frequency, and campaign response rates.
What Needed to Be Done
Pursuant needed more than a data warehouse MVP. The data pipeline had to become repeatable, scheduled, and easier to run without manual coordination across multiple AWS services.
The job was to automate the ETL flow: start data imports, detect when each step finished, trigger the next step, and reduce manual pipeline work.
The job was to automate the ETL flow: start data imports, detect when each step finished, trigger the next step, and reduce manual pipeline work.
Why It Mattered
A reporting warehouse is only useful if fresh data can move through it reliably. Without automation, the team would still need to manually run imports, crawlers, transformations, and checks.
For Pursuant, that would have kept the reporting process dependent on technical staff and made future client onboarding harder to manage at scale.
For Pursuant, that would have kept the reporting process dependent on technical staff and made future client onboarding harder to manage at scale.
What MEV Built
MEV designed an event-based pipeline automation flow using AWS EventBridge, SNS, Lambda, AWS DMS, Glue Crawlers, and Glue Jobs.
The automation listened to status changes from DMS tasks and Glue Crawlers, then triggered the next pipeline step through Lambda functions.
The automation listened to status changes from DMS tasks and Glue Crawlers, then triggered the next pipeline step through Lambda functions.
Technical Flow
01:
Scheduled pipeline start
EventBridge triggered a Lambda function responsible for starting AWS DMS import tasks. In the documented setup, the initial EventBridge rule ran every Tuesday at 7 AM UTC.
02:
Data import with AWS DMS
AWS DMS exported source data from Microsoft SQL Server into AWS S3.
03:
DMS completion event
DMS event subscriptions posted task status events to SNS after the import step completed.
04:
Glue Crawler trigger
SNS passed the event to a Lambda function that triggered the relevant Glue Crawler for the completed DMS task.
05:
Schema refresh
The Glue Crawler scanned the newly exported files and recreated or updated the schema used by downstream Glue Jobs.
06:
Glue Job trigger
EventBridge listened for successful Glue Crawler state changes. Once a crawler succeeded, another Lambda function triggered the relevant Glue Jobs.
07:
Data transformation
Glue Jobs transformed the source data into reporting-ready warehouse models.
08:
Automatic continuation after manual runs
Because the pipeline was event-based, a manual run of a DMS task or crawler could still trigger the next steps automatically.
Outcome
Pursuant gained a more repeatable ETL process with fewer manual pipeline steps. The automation connected data import, schema discovery, and transformation into a coordinated AWS workflow that could support scheduled reporting refreshes and future client expansion.
This case is part of the full Pursuant case study:
MEV Built an AWS Data Warehouse and Automated Reporting Pipeline for Pursuant[ portfolio/ ]
