How to Speed up the Processing and Sending of Bulk but Personalized Emails in Drupal

Introduction

One of our long-standing clients is a leading player in the financial news space in the US. A financial site dealing with the latest financial news from across the world, the site provides the latest stock market news. The site also offers live web-based news streaming.

The live news is delivered via a JavaScript live feed application and the site is also integrated with multiple third-party APIs for the aggregation of content and the delivery of content.

Challenge

There are around 15,000 active portfolios in the system and the summary emails are to be sent to holders of all these portfolios daily. It is a complex process to build the summary email content and the mail content has to be done for each portfolio. This is a financial news site, so it is really important to deliver the content without too much delay.

Solution

The emails being personalized and independent, it is not required to handle its processing and sending in a normal sequential fashion. It could also be done parallelly. So we implemented it in a parallel fashion, with the help of Drupal queue system and the module ultimate_cron.

We created a cron job which fetches all the active portfolio IDs and adds them to a queue. This queue is capable of parallel processing more items at the same time. After processing the portfolio summary from the queue, these items get added to another queue which is also capable of parallel processing. The second queue is to send the summary to its corresponding users via email. So sending the emails are also done as a parallel process in the second queue.

Thus we were able to deliver all the contents in an hour with the current server capabilities. Increasing the number of parallel requests by improving the capability of the server can further improve the speed.

Result

The email sending process improved 10 times faster than its normal sequential flow.