Day 7 - DynamoDB - and Working with 2 Services - Lambda



John Rankin @ August 25, 2018

So Today, we will be learning the basics of DynamoDB...  

We will be using AWS CLI and NodeJS + Lambda (because why not?) ...   

We are going to create a simple table that resembles a blog post table.     

This table will have the following fields:

  • id
  • title
  • content

So where NoSQL is different than Relational DB's like MySQL is it schema-less.  This means the only thing that is required to create our table is to simply define the indexes for it.   

 

Before we get started... lets get some basic understanding:

DynamoDB supports two types of primary keys:

  • Partition key: Also known as a hash key, the partition key is composed of a single attribute. Attributes in DynamoDB are similar in many ways to fields or columns in other database systems.
  • Partition key and sort key: Referred to as a composite primary key or hash-range key, this type of key is composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key. Here is an example:

DynamoDB stores data as groups of attributes, known as items. Items are similar to rows or records in other database systems. DynamoDB stores and retrieves each item based on the primary key value which must be unique.  

DynamoDB uses the partition key’s value as an input to an internal hash function. The output from the hash function determines the partition in which the item will be stored. Each item’s location is determined by the hash value of its partition key.

All items with the same partition key are stored together, and for composite partition keys, are ordered by the sort key value. DynamoDB will split partitions by sort key if the collection size grows bigger than 10 GB.

 

TLDR;  It Just allows for faster accessing because it forces the data to be stored in an organized fashion

So anyways let's get started.

For DynamoDB ... we are going to simply create a json file for easier reading than making 1 very long command.

lets create RekousTestTable.json

{
    "TableName": "RekousTestTable",
    "KeySchema": [
      { "AttributeName": "id", "KeyType": "HASH" },
      { "AttributeName": "title", "KeyType": "RANGE" }
    ],
    "AttributeDefinitions": [
      { "AttributeName": "id", "AttributeType": "N" },
      { "AttributeName": "title", "AttributeType": "S" }
    ],
    "ProvisionedThroughput": {
      "ReadCapacityUnits": 5,
      "WriteCapacityUnits": 5
    }
}

So Keep Note:  We have not added the "content"  column to our table definition.   NoSQL is SchemaLess.  This means if we need to add a column to our table, we simply add the column when we submit data.  So the only required definitions is our index'es.

So our table definition is now created... Lets take this json and create the table and use this table definition:

root@damrkul2:~/aws/dynamodb# aws dynamodb create-table --cli-input-json file://RekousTestTable.json --region us-west-2
{
    "TableDescription": {
        "TableArn": "arn:aws:dynamodb:us-west-2:841181805441:table/RekousTestTable",
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "N"
            },
            {
                "AttributeName": "title",
                "AttributeType": "S"
            }
        ],
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "WriteCapacityUnits": 5,
            "ReadCapacityUnits": 5
        },
        "TableSizeBytes": 0,
        "TableName": "RekousTestTable",
        "TableStatus": "CREATING",
        "TableId": "5d1eecbb-8d8a-4d15-9adc-281dbcc34c60",
        "KeySchema": [
            {
                "KeyType": "HASH",
                "AttributeName": "id"
            },
            {
                "KeyType": "RANGE",
                "AttributeName": "title"
            }
        ],
        "ItemCount": 0,
        "CreationDateTime": 1535216135.112
    }
}


Lets Now play with Lambda. I've created a blank serverless - nodejs template directory. Please look at the previous blog post to get
a lambda envirornment setup if you need instructions.

Here is the serverless.yaml where I've configured 3 functions for Lambda.

#
# Rekous DynamoDB Lambda
#

service: rekous-dynamodb-test
provider:
  name: aws
  runtime: nodejs8.10
  region: us-west-2

functions:
  list:
    handler: handler.list
    events:
      - http:
          path: list/
          method: get
  put:
    handler: handler.put
    events:
      - http:
          path: put/
          method: post

  show:
    handler: handler.show
    events:
      - http:
          path: show/
          method: post

Here is the handler.js

'use strict';

const AWS = require('aws-sdk');
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});


module.exports.list =  function(event, context, callback) {
var table = "RekousTestTable";
var params = {
    TableName: table,
};

dynamodb.scan(params, function(err, data) {
var results = [];
  if (err) {
    var temp = {
        error: err.stack
    };
    results.push(temp);
  } else {
    data.Items.forEach(function(element, index, array) {
      var temp = {
            "id" : element.id.N,
            "title" : element.title.S
      }

      results.push(temp);
    });
  }

 callback(null, {
    statusCode: 200,
    body: JSON.stringify({
      reply: results,
    }),
  });
});

};


module.exports.put = function(event, context, callback) {
  // Parse the Body that contains our text POST JSON
  var input =  JSON.parse(event.body);

var table = "RekousTestTable";

// Incase Not submitted and left blank
var content =  (input.content == undefined ) ? '' : input.content;
var url =  (input.content == undefined ) ? '' : input.content;

var item = {
    "id": { N: String(input.id) },
    "title": { S: input.title },
    "content": { S: content },
    "url": { S: url },
} ;

var params = {
    "TableName": table,
    "Item": item
};

dynamodb.putItem(params, function(err, data) {
var results = [];
 if (err) {
    results.push(err.stack);
    console.log(err, err.stack);
  } else {
    results.push(params.Item);
 }


 callback(null, {
    statusCode: 200,
    body: JSON.stringify({
      reply: results,
    }),
  });

});


};



module.exports.show= function(event, context, callback) {

// Parse the Body that contains our text POST JSON
var input =  JSON.parse(event.body);

var table = "RekousTestTable";
var key = {
    "id": { N: String(input.id) },
    "title": { S: input.title },
} ;

var params = {
    "TableName": table,
    "Key": key
};

dynamodb.getItem(params, function(err, data) {
var results = [];
 if (err) {
    results.push(err.stack);
    console.log(err, err.stack);
  } else {
    results.push(data);
 }


 callback(null, {
    statusCode: 200,
    body: JSON.stringify({
      reply: results,
    }),
  });

});


};

To Show a List of Items:

root@damrkul2:~/aws/dynamodb/reklambdatest# curl https://77acb0x6te.execute-api.us-west-2.amazonaws.com/dev/list
{"reply":[{"id":"1","title":"RekTitle1"}]}

To Add/Put an Item:

root@damrkul2:~/aws/dynamodb/reklambdatest# curl -d '{"title":"RekTitle2", "id":2,  "url": "rekous2.com" , "content": "my first content2"}' -X POST https://77acb0x6te.execute-api.us-west-2.amazonaws.com/dev/put
{"reply":[{"id":{"N":"2"},"title":{"S":"RekTitle2"},"content":{"S":"my first content2"},"url":{"S":"my first content2"}}]}

To View/Show an Item and all its fields:

root@damrkul2:~/aws/dynamodb/reklambdatest# curl -d '{"title":"RekTitle2", "id":2}' -X POST https://77acb0x6te.execute-api.us-west-2.amazonaws.com/dev/show
{"reply":[{"Item":{"content":{"S":"my first content2"},"id":{"N":"2"},"url":{"S":"my first content2"},"title":{"S":"RekTitle2"}}}]}

 

There ya have it... we have now implemented a solution that has a AWS service (Lambda) accessing another AWS service (DynamoDB).  

Summary

Now this took me quite a while todo... as one of the reasons is that I had to figureout that I had to change the function definiton in the handler.js to take in a callback function argument, to be used in the dynamoDB in order to display data to the user.  

module.exports.list =  function(event, context, callback)

I had to use CloudWatch to view logs to check for errors.  But I eventually, did find this utility that came with Lambda very helpful:

^^^ Was super helpful

One of the early issues while learning this was figuring out how to use lambda to connect to the DynamoDB service.  The solution is to attatch an IAM policy to the Lambda service that allows Lambda to connect to Dynamo DB.  I took the extra time to create my own policy for fun using the generator.  
Once I attatched the policy, the Lambda Dashboard gives a nice representation of what services it currently has access to.  See below:

 

So anyways --- there ya have it... we have now learned how to connect to 2 AWS services and have them interact with one another.  If I wanted

Lambda to connect to the S3 service, I would simply need to attatch another Policy to Lambda to give it access.

 

Thanks for reading.

 

 

 

 

Most Recent Posts


Blog Test

Read This
October 16, 2018

Day 7 - DynamoDB - and Working with 2 Services - Lambda

Read This
August 25, 2018

Day 6 - Lambda - Creating an API

Read This
August 23, 2018

AWS - Day 5 - S3 - Simple Storage Service

Read This
August 22, 2018

AWS - Day 4 - AWS CLI Useful Scripts

Read This
August 21, 2018

AWS - Day 3 - Create Container from another container

Read This
August 20, 2018

Day 2 - Docker Intro

Read This
August 17, 2018

AWS - Day1 - Tutorial - Launching my first Docker Container

Read This
August 16, 2018

AWS - Day 1 - Signing up and testing out their tutorials

Read This
August 16, 2018

Dynamic Programming - Edit Distance

Read This
December 19, 2016

Dynamic Programming - Fibonacci

Read This
December 19, 2016

What I Do

Read This
June 23, 2015

First Blog Post

Read This
June 23, 2015