I recently migrated a few core components of my daily Chinese vocab app - including contact lists, subscribe/unsubscribe API, and email sending functionality - from SendGrid to a custom built backend that uses DynamoDB, SES, Lambda, and API Gateway. Here I'll share about my architecture and some of the considerations in the migration.
Jump to:
Why build my own backend
When I began building my daily vocab service out into a public application, I used SendGrid's APIs and console for contact list management and sending emails. This was a great choice, since it allowed me to focus on the core, unique components of my application as opposed to getting bogged down and overwhelmed by building contact management at the same time.
Now that my app is more mature, I decided to build my own backend. The first motivation was reducing costs. SendGrid's free tier is 100 emails a day, and my current subscriber count is up to 70 (🎉). SendGrid's first paid tier is $15/mo., which was steep for my level of usage, whereas my new architecture is still well within the AWS free tier.
Another factor was increasing my ability to add more custom logic into my backend. When I thought about features that I wanted to add to my app, I was beginning to feel restricted by needing to fit into SendGrid's model.
For example, SendGrid email workflows are divided into 'marketing' and 'transactional' emails, neither of which 100% fit my use case. With the marketing model, I could group users into contact lists by the vocab level they subscribed to, but I needed the transactional email functionality of dynamically substituting in email content (ie, handlebars). I created a workaround to be able to use a bit of both (I used SendGrid marketing contact lists, but I packaged my email templates within the Lambda function zip files so I could manipulate them, as opposed to using SendGrid's template management), but I could see my SendGrid integration getting more hacky over time.
One new feature that I'm excited to roll out soon is support for traditional Chinese characters. I shared my app with the r/ChineseLanguage community on Reddit, and I got a few requests for this. It would have been complicated with my old architecture, but it was easy with my new backend - some slight tweaks to my contact list data structure and adding some if statements (if char_set == "traditional"... if char_set == "simplified"...) into my functions that compose emails.
Architecture before
This was my previous architecture using SendGrid for subscribing/unsubscribing users and sending emails.
I had a subscribe Lambda function that posted to SendGrid's subscribe API. Within SendGrid, I created prod and staging contact lists for each of the 6 vocab levels. I needed to use the contact list IDs to subscribe and send emails, so I stored the contact list IDs in S3. SendGrid managed unsubscribes.
My daily email send function looped through each contact list. I called SendGrid's API to create an email campaign, get the campaign ID, and then send the campaign to each contact list.
Architecture after
For my new backend, I needed to:
- Build a contact list database,
- Build an unsubscribe workflow, and
- Direct my existing subscribe and daily email sending functions to use the new database and send email through SES.
The subscribe function now writes to a DynamoDB contact table. I have contact tables for prod and staging. The data structure looks like this:
I needed to create a new API endpoint for unsubscribing since this was handled by SendGrid previously. I created an unsubscribe UI with the option to unsubscribe from an individual list or all lists. If the user comes from the unsubscribe link in their daily email, the dropdowns prepopulate with the level they are subscribed to.
The send email function now scans the Dynamo contact table and loops through each contact. Assembling the campaign contents is a similar process as before, with a few less steps than it took using SendGrid's workflow. In both architectures, my email template is an HTML file packaged with the Lambda function zip file.
The send email function calls the SES API, which has been straightforward to implement. On my to-do list is automating unsubscribing bounced emails, which SendGrid used to handle for me. SES has built in functionality for receiving emails (like bounce notifications) and saving them to S3. The next step is writing a Lambda that processes the received emails, identifies bounces, and unsubscribes the contact.
What I learned
I was surprised by how fast I was able to build this! I attribute this to growing familiarity - I'm using services that I've used at least once before (with the exception of SES) and combining them in new ways. Even though building this backend was a big change for my app, it was less of a learning curve for me.
I'm also much more comfortable in Python. After spending a lot of time recently on a frontend refactor from vanilla Javascript to Vue.js and then switching back to a primarily Python-based backend project, I realized just how much more intuitive and easy Python feels for me. This makes me want to spend more time with the Javascript and Vue.js basics so I can have more fun building frontends. Still ❤️ Python.
I'm excited about the opportunities the new backend opens up, from traditional character support to eventually the ability for users to create profiles and custom vocab lists.
Visit Haohaotiantian.com
Check out the source code
🌻