Serply Notifications Part 1: Search Engine Result Pages (SERP)

Learn to develop a serverless application with Slack, Serply API, AWS CDK and Python.

Profile picture of Osvaldo Brignoni
Osvaldo Brignoni
Cover Image for Serply Notifications Part 1: Search Engine Result Pages (SERP)

What is Serply Notifications?

Serply Notifications is an open-source notification scheduler for the Serply API. It allows you to schedule and receive notifications for Google Search Engine Result Pages (SERP) on your Slack account.

The application is developed with AWS serverless services: API Gateway, Lambda, DynamoDB and EventBridge. All of its resources are defined as a Cloud Development Kit Stack. This is a detailed walk-through of all its components. First, let's look at the application features from the user's point of view.

As a Slack user, I can...

  • Schedule SERP notifications for specific search queries.
  • Receive notifications daily, weekly or monthly.
  • List notification schedules.
  • Disable and re-enable scheduled notifications.

How can you track the SERP position of a website?

To track the position of a website in Google Search Engine Result Pages (SERP), you can use a few methods:

1. Manual Search: You can manually search for a specific keyword related to your website and note the position of your website in the SERP.

2. SERP Tracking Tool: There are various online tools available that can track the position of your website in the SERP. Some popular tools include Ahrefs, SEMrush, Moz, and SERPstat.

3. Google Search Console: This is a free tool provided by Google that allows you to track your website's performance in Google search. It provides information about the keywords for which your website is ranking and the position of your website for each keyword.

Regardless of the method you use, it's important to track your website's position regularly to monitor its performance and make necessary adjustments to improve its ranking in the SERP.

Example manual search

https://www.google.com/search?q=professional+network&num=100&domain=linkedin.com

Slack /serply command

The command structure is really simple. It starts with the slash command /serply, followed by the sub-command and specific parameters. There are two sub-commands available: serp and list.

/serply serp - schedule a SERP notification

The serp sub-command creates a scheduled notification for the linkedin.com domain with the search query "professional+network". You will see a confirmation message with the Schedule details, shortly after entering the command. You will receive the notification on the specified interval: daily, weekly, or monthly.

/serply list - list all scheduled notifications

The list sub-command returns a list of all schedules.

Disable a scheduled notification

You can disable a scheduled notification by clicking the Disable button on a given Slack message. The history of notifications will remain on the database. You can also re-enable it by clicking the Enable button.

Application Design

Let's start from the request patterns

Since this is an event-driven application, it will be very helpful to plan for all the request patterns before coding. You won't need to code because you can just clone the Serply Notifications repository. However, I do want to go over the thought process for designing this application. In simple terms, this is the sequence of requests.

After a user enters the /serply command...

1. Slack will perform a POST request to our application webhook with the message payload.

2. The webhook will respond to Slack with a 200 status code, acknowledging the receipt of the request.

3. The backend will parse the payload and parameters from the slash command string.

4. The backend will transform that payload to a Schedule object and send it to the event bus.

5. A function will receive that event and perform 2 tasks:

  • Create the Schedule for triggering the notification.
  • Save the Schedule data to a Notifications database.

6. Another function will send a confirmation message to Slack letting the user know that the schedule was created.

7. The Schedule will call another function that performs 3 tasks:

  • Gets the SERP data from the Serply API.
  • Transforms and saves the response as a timestamped notification.
  • Sends the SERP data to the event bus.

8. Another function will receive the SERP data from the event bus and send the notification back to Slack for users to see.

DynamoDB: Planning for access patterns

We should also anticipate how we will model, index and query the data. Let's make a list of DynamoDB access patterns and the query conditions that we will use to accommodate them.

Adjacency list design pattern

When different entities of an application have a many-to-many relationship between them, the relationship can be modeled as an adjacency list. In this pattern, all top-level entities (synonymous with nodes in the graph model) are represented using the partition key. Any relationships with other entities (edges in a graph) are represented as an item within the partition by setting the value of the sort key to the target entity ID (target node).

The advantages of this pattern include minimal data duplication and simplified query patterns to find all entities (nodes) related to a target entity (having an edge to a target node).

~ Amazon DynamoDB Developer Guide

Primary Key, Sort Key Design and EventBridge Schedules

In addition to the request and DynamoDB access patterns, we need to take into consideration other service constraints.

  • Map the Slack command to the Schedule database key.
  • The Schedule Primary Key is the concatenated attributes prefixed with schedule_.
  • The Schedule Sort Key is intentionally the same as the Schedule Primary Key.
  • The SERP Notification is similar to the Schedule Primary Key prefixed with notification_ and suffixed with a ISO 8601 datetime string.
  • Map the Schedule database key to the EventBridge Schedule Name.
  • The EventBridge Schedule Name has a maximum limit of 64 characters.
  • The Slack command string and database key attributes might exceed 64 characters.
  • Generate a deterministic hash from the database key for the EventBridge Schedule Name that is always 64 characters in length.

Example Keys

The Slack command will always produce the same keys. The EventBridge Schedule Name hash is generated from the Schedule PK.

EventBridge Schedule API Reference: Name Length Constraint

Ready for Multi-Tenancy: The Global Secondary Index

In this case, the main reason for creating a Global Secondary Index is that we need to query all the schedules for a given Slack account when we enter the /serply list command. Another benefit of this index is that we could develop multi-tenancy for this application if we need it later. We could scope notifications for each Slack account. For now, there is only one default account ID.

SerplyStack

All the AWS resources are defined in the SerplyStack. When we run the cdk deploy command, the CDK builds the application's infrastructure as a set of AWS CloudFormation templates and then uses these templates to create or update a CloudFormation stack in the specified AWS account and region. During deployment, the command sets up the required AWS resources as defined in the CDK application's code and also configures any necessary connections between the resources. The result is a fully functioning, deployed application in the target AWS environment.

# src/cdk/serply_stack.py

class SerplyStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, config: SerplyConfig, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        RUNTIME = _lambda.Runtime.PYTHON_3_9

        lambda_layer = _lambda.LayerVersion(
            self, f'{config.STACK_NAME}LambdaLayer{config.STAGE_SUFFIX}',
            code=_lambda.Code.from_asset(config.LAYER_DIR),
            compatible_runtimes=[RUNTIME],
            compatible_architectures=[
                _lambda.Architecture.X86_64,
                _lambda.Architecture.ARM_64,
            ]
        )

        event_bus = events.EventBus(
            self, config.EVENT_BUS_NAME,
            event_bus_name=config.EVENT_BUS_NAME,
        )

        event_bus.apply_removal_policy(RemovalPolicy.DESTROY)

        scheduler_managed_policy = iam.ManagedPolicy.from_aws_managed_policy_name(
            'AmazonEventBridgeSchedulerFullAccess'
        )

        scheduler_role = iam.Role(
            self, 'SerplySchedulerRole',
            assumed_by=iam.ServicePrincipal('scheduler.amazonaws.com'),
        )
        
        scheduler_role.add_to_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                actions=['lambda:InvokeFunction'],
                resources=['*'],
            )
        )

        slack_receive_lambda = _lambda.Function(
            self, 'SlackReceiveLambdaFunction',
            runtime=RUNTIME,
            code=_lambda.Code.from_asset(config.SLACK_DIR),
            handler='slack_receive_lambda.handler',
            timeout=Duration.seconds(5),
            layers=[lambda_layer],
            environment={
                'STACK_NAME': config.STACK_NAME,
                'STAGE': config.STAGE,
            },
        )

        event_bus.grant_put_events_to(slack_receive_lambda)

        slack_respond_lambda = _lambda.Function(
            self, 'SlackRespondLambdaFunction',
            runtime=RUNTIME,
            code=_lambda.Code.from_asset(config.SLACK_DIR),
            handler='slack_respond_lambda.handler',
            timeout=Duration.seconds(5),
            layers=[lambda_layer],
            environment={
                'SLACK_BOT_TOKEN': config.SLACK_BOT_TOKEN,
                'STACK_NAME': config.STACK_NAME,
                'STAGE': config.STAGE,
            },
        )

        slack_notify_lambda = _lambda.Function(
            self, 'SlackNotifyLambdaFunction',
            runtime=RUNTIME,
            code=_lambda.Code.from_asset(config.SLACK_DIR),
            handler='slack_notify_lambda.handler',
            timeout=Duration.seconds(5),
            layers=[lambda_layer],
            environment={
                'SLACK_BOT_TOKEN': config.SLACK_BOT_TOKEN,
                'STACK_NAME': config.STACK_NAME,
                'STAGE': config.STAGE,
            },
        )

        slack_notify_lambda.role.add_managed_policy(scheduler_managed_policy)

        # More resources...

List of CloudFormation resources

  • AWS::ApiGateway::RestApi
  • AWS::ApiGateway::Account
  • AWS::ApiGateway::Deployment
  • AWS::ApiGateway::Method
  • AWS::ApiGateway::Resource
  • AWS::ApiGateway::Stage
  • AWS::DynamoDB::Table
  • AWS::Events::EventBus
  • AWS::Events::EventRule
  • AWS::IAM::Policy
  • AWS::IAM::Role
  • AWS::Lambda::Function
  • AWS::Lambda::LayerVersion
  • AWS::Lambda::Permission
  • AWS::Scheduler::ScheduleGroup

Useful tip

All the CloudFormation resources are well indexed on Google. Searching for any of them will quickly return the relevant documentation. In the documentation, you will be able to see all the available parameters and validation criteria.

API Gateway Rest API /slack/{proxy+}

The CDK creates an AWS::ApiGateway::RestApi resource that will act as a catch-all webhook for all Slack events. The request is forwarded to the slack_receive_lambda function via the LAMBDA_PROXY Integration Request.

Lambda functions

The business logic lives entirely in Lambda functions developed with Python. These functions are triggered by three sources: API Gateway, EventBridge Events and EventBridge Schedules.

src/slack/

  • slack_notify_lambda
  • slack_receive_lambda
  • slack_respond_lambda

src/schedule/

  • schedule_disable_lambda
  • schedule_enable_lambda
  • schedule_save_lambda
  • schedule_target_lambda

slack_receive_lambda

This function has a few important tasks:
1. Receives and processes the event forwarded by API Gateway.
2. Checks the Slack Challenge string if present in the request body.
3. Verifies the `X-Slack-Signature` header to make sure it is the Slack app calling our Rest API.
4. Parses the `/serply` command string with the `SlackCommand` class.
5. Forwards all the data to the `SerplyEventBus`.
6. Returns an acknowledgement response to Slack within 3 seconds.

This confirmation must be received by Slack within 3000 milliseconds of the original request being sent, otherwise, a "Timeout reached" will be displayed to the user. If you couldn't verify the request payload, your app should return an error instead and ignore the request.

~ Confirming receipt | api.slack.com

import json
from slack_receive_response import (
    command_response,
    default_response,
    event_response,
    interaction_response,
)


def get_challenge(body):
    if not body.startswith('{') and not body.endswith('}'):
        return False
    return json.loads(body).get('challenge', '')


responses = {
    '/slack/commands': command_response,
    '/slack/events': event_response,
    '/slack/interactions': interaction_response,
}


def handler(event, context):

    challenge = get_challenge(event.get('body'))

    if challenge:
        return {
            'statusCode': 200,
            'body': challenge,
            'headers': {
                'Content-Type': 'text/plain',
            },
        }

    return responses.get(event.get('path'), default_response)(event=event)

Serply Event Bus

An event bus is a channel that allows different services to communicate and exchange events. EventBridge enables you to create rules that automatically trigger reactions to events, such as sending an email in response to an event from a SaaS application. If you are sending emails in bulk, use email template creator to create the emails effortlessly. The event bus is capable of triggering multiple targets simultaneously.


Receiving the Slack command

After receiving the initial request from Slack, the slack_receive_lambda function puts an event into the SerplyEventBus.


The event bus triggers 2 functions:

  • slack_respond_lambda
  • schedule_save_lambda

slack_respond_lambda

This function sends a SERP Notification Scheduled message to the Slack channel. I could have done this in the slack_receive_lambda function. However, I needed to keep that function as light as possible to acknowledge receipt within 3 seconds. The response message is delegated to the slack_respond_lambda function intentionally.

import boto3
from pydash import objects
from slack_api import SlackClient, SlackCommand
from slack_messages import ScheduleMessage, ScheduleListMessage
from serply_config import SERPLY_CONFIG
from serply_database import NotificationsDatabase, schedule_from_dict


notifications = NotificationsDatabase(boto3.resource('dynamodb'))
slack = SlackClient()


def handler(event, context):

    detail_type = event.get('detail-type')
    detail_input = event.get('detail').get('input')
    detail_schedule = event.get('detail').get('schedule')

    if detail_type == SERPLY_CONFIG.EVENT_SCHEDULE_SAVE:
        schedule = schedule_from_dict(detail_schedule)
        message = ScheduleMessage(
            channel=detail_input.get('channel_id'),
            user_id=detail_input.get('user_id'),
            command=schedule.command,
            interval=schedule.interval,
            type=schedule.type,
            domain=schedule.domain,
            domain_or_website=schedule.domain_or_website,
            query=schedule.query,
            website=schedule.website,
            enabled=True,
            replace_original=False,
        )
        slack.respond(
            response_url=detail_input.get('response_url'),
            message=message,
        )

    elif detail_type == SERPLY_CONFIG.EVENT_SCHEDULE_LIST:
        schedules = notifications.schedules()
        message = ScheduleListMessage(
            channel=detail_input.get('channel_id'),
            schedules=schedules,
        )
        slack.notify(message)

    elif detail_type in [
        SERPLY_CONFIG.EVENT_SCHEDULE_DISABLE_FROM_LIST,
        SERPLY_CONFIG.EVENT_SCHEDULE_ENABLE_FROM_LIST,
    ]:
        schedules = notifications.schedules()
        message = ScheduleListMessage(
            schedules=schedules,
            replace_original=True,
        )
        slack.respond(
            response_url=detail_input.get('response_url'),
            message=message,
        )

    elif detail_type == SERPLY_CONFIG.EVENT_SCHEDULE_DISABLE:
        schedule = SlackCommand(
            command=objects.get(detail_input, 'actions[0].value'),
        )
        message = ScheduleMessage(
            user_id=detail_input.get('user').get('id'),
            command=schedule.command,
            interval=schedule.interval,
            type=schedule.type,
            domain=schedule.domain,
            domain_or_website=schedule.domain_or_website,
            query=schedule.query,
            website=schedule.website,
            enabled=False,
            replace_original=True,
        )
        slack.respond(
            response_url=detail_input.get('response_url'),
            message=message,
        )

    elif detail_type == SERPLY_CONFIG.EVENT_SCHEDULE_ENABLE:
        schedule = SlackCommand(
            command=objects.get(detail_input, 'actions[0].value'),
        )
        message = ScheduleMessage(
            user_id=detail_input.get('user').get('id'),
            command=schedule.command,
            interval=schedule.interval,
            type=schedule.type,
            domain=schedule.domain,
            domain_or_website=schedule.domain_or_website,
            query=schedule.query,
            website=schedule.website,
            enabled=True,
            replace_original=True,
        )
        slack.respond(
            response_url=detail_input.get('response_url'),
            message=message,
        )

    return {'ok': True}

schedule_save_lambda

This function saves the schedule data to the DynamoDB Notifications table. EventBridge Schedules are not provisioned via the CDK. Instead, there is a specific schedule for each /serply serp command that is created by the schedule\_save\_lambda function via the boto3.client('scheduler') client.

import boto3
import json
from serply_database import NotificationsDatabase, schedule_from_dict
from serply_scheduler import NotificationScheduler

notifications = NotificationsDatabase(boto3.resource('dynamodb'))
scheduler = NotificationScheduler(boto3.client('scheduler'))


def handler(event, context):

    schedule = schedule_fro_dict(event.get('detail').get('schedule'))
    
    notifications.save(schedule)

    scheduler.save_schedule(
        schedule=schedule,
        event=event,
    )
    
    return {'ok': True}

schedule_target_lambda

This function is triggered by its corresponding schedule and it has 3 tasks.
1. Get the SERP data from the Serply API https://api.serply.io/v1/serp endpoint.
2. Save the event and response data to the database as a SerpNotification.
3. Forward all the data to the SerplyEventBus.

import boto3
import json
from dataclasses import asdict
from serply_api import SerplyClient
from serply_config import SERPLY_CONFIG
from serply_database import NotificationsDatabase, SerpNotification, schedule_from_dict
from serply_events import EventBus

event_bus = EventBus(boto3.client('events'))
notifications = NotificationsDatabase(boto3.resource('dynamodb'))
serply = SerplyClient(SERPLY_CONFIG.SERPLY_API_KEY)


def handler(event, context):

    detail_headers = event.get('detail').get('headers')
    detail_schedule = event.get('detail').get('schedule')
    detail_input = event.get('detail').get('input')

    schedule = schedule_from_dict(detail_schedule)

    if schedule.type not in [SERPLY_CONFIG.SCHEDULE_TYPE_SERP]:
        raise Exception(f'Invalid schedule type: {schedule.type}')

    if schedule.type == SERPLY_CONFIG.SCHEDULE_TYPE_SERP:

        response = serply.serp(
            domain=schedule.domain,
            website=schedule.website,
            query=schedule.query,
            mock=schedule.interval == 'mock',
        )

        notification = SerpNotification(
            command=schedule.command,
            domain=schedule.domain,
            domain_or_website=schedule.domain_or_website,
            query=schedule.query,
            interval=schedule.interval,
            serp_position=response.position,
            serp_searched_results=response.searched_results,
        )

    notifications.save(notification)

    notification_input = asdict(notification)

    event_bus.put(
        source=schedule.source,
        detail_type=SERPLY_CONFIG.EVENT_SCHEDULE_NOTIFY,
        schedule=schedule,
        input={
            **detail_input,
            **notification_input,
        },
        headers=detail_headers,
    )

    return {'ok': True}


Scheduled notification


slack_notify_lambda

This function builds the notification message and sends it to Slack. The SerpNotificationMessage data class serves as a template for formatting the message using Slack Message Blocks.

import boto3
import json
from serply_config import SERPLY_CONFIG
from slack_api import SlackClient
from slack_messages import SerpNotificationMessage
from serply_database import schedule_from_dict
from serply_scheduler import NotificationScheduler


slack = SlackClient(SERPLY_CONFIG.SLACK_BOT_TOKEN)
scheduler = NotificationScheduler(boto3.client('scheduler'))


def handler(event, context):

    detail_schedule = event.get('detail').get('schedule')
    detail_input = event.get('detail').get('input')

    schedule = schedule_from_dict(detail_schedule)

    if schedule.type not in [SERPLY_CONFIG.SCHEDULE_TYPE_SERP]:
        raise Exception(f'Invalid schedule type: {schedule.type}')

    if schedule.type == SERPLY_CONFIG.SCHEDULE_TYPE_SERP:
        message = SerpNotificationMessage(
            channel=detail_input.get('channel_id'),
            serp_position=detail_input.get('serp_position'),
            serp_searched_results=detail_input.get('serp_searched_results'),
            command=schedule.command,
            domain=schedule.domain,
            domain_or_website=schedule.domain_or_website,
            interval=schedule.interval,
            query=schedule.query,
            website=schedule.website,
        )
        slack.notify(message)

    if schedule.interval in SERPLY_CONFIG.ONE_TIME_INTERVALS:
        scheduler.delete_schedule(schedule)

    return {'ok': True}

SerpNotificationMessage Slack Message Blocks

@dataclass
class SerpNotificationMessage:

    blocks: list[dict] = field(init=False)
    domain: str
    domain_or_website: str
    command: str
    interval: str
    query: str
    serp_position: int
    serp_searched_results: str
    website: str
    channel: str = None
    num: int = 100
    replace_original: bool = False

    def __post_init__(self):

        TEXT_ONE_TIME = f'This is a *one-time* notification.'
        TEXT_YOU_RECEIVE = f'You receive this notification *{self.interval}*. <!here>'

        website = self.domain if self.domain else self.website
        total = int(self.serp_searched_results or 0)
        google_search = f'https://www.google.com/search?q={self.query}&num={self.num}&{self.domain_or_website}={website}'
        results = f'<{google_search}|{total} results>' if total > 0 else f'0 results'

        self.blocks = [
            {
                'type': 'section',
                'text': {
                    'type': 'mrkdwn',
                    'text': f'> `{website}` in position `{self.serp_position or 0}` for `{self.query}` from {results}.'
                },
            },
            {
                'type': 'context',
                'elements': [
                    {
                        'type': 'mrkdwn',
                        'text': f':bell: *SERP Notification* | {TEXT_ONE_TIME if self.interval in SERPLY_CONFIG.ONE_TIME_INTERVALS else TEXT_YOU_RECEIVE}'
                    }
                ]
            },
        ]


Serply API

The schedule_target_lambda function makes a GET request to the Serply API that is equivalent to the following CURL request.

Request

curl --request GET \
  --url 'https://api.serply.io/v1/serp/q=professional+network&num=100&domain=linkedin.com' \
  --header 'Content-Type: application/json' \
  --header 'X-Api-Key: API_KEY'

Response

{
    "searched_results": 100,
    "result": {
        "title": "Why professional networking is so important - LinkedIn",
        "link": "https://www.linkedin.com/pulse/why-professional-networking-so-important-jordan-parikh",
        "description": "7 de nov. de 2016 — Networking becomes a little clearer if we give it a different name: professional relationship building. It's all about getting out there ...",
        "additional_links": [
            {
                "text": "Why professional networking is so important - LinkedInhttps://www.linkedin.com › pulse",
                "href": "https://www.linkedin.com/pulse/why-professional-networking-so-important-jordan-parikh"
            },
            {
                "text": "Traduzir esta página",
                "href": "https://translate.google.com/translate?hl=pt-BR&sl=en&u=https://www.linkedin.com/pulse/why-professional-networking-so-important-jordan-parikh&prev=search&pto=aue"
            }
        ],
        "cite": {
            "domain": "https://www.linkedin.com › pulse",
            "span": " › pulse"
        }
    },
    "position": 8,
    "domain": ".linkedin.com",
    "query": "q=professional+network&num=100"
}

Github Repository

You can find the repository here. It includes all the installation steps: to set up your Slack app, Serply account and AWS CDK deployment. Deploy it on your AWS account within the AWS Free Tier. You can also fork it and customize it to your own needs.

serply-inc/notifications

Reference

Possibilities

Serply Notifications is structured such that all the scheduling is decoupled from the messaging logic using an event bus. More integrations and notification types could be added such as email notifications, other chatbot platforms and more notification types from the Serply API.

Stay tuned for Part 2 of this Serply Notifications series!