So Today, we will be learning the basics of DynamoDB...
We will be using AWS CLI and NodeJS + Lambda (because why not?) ...
We are going to create a simple table that resembles a blog post table.
This table will have the following fields:
So where NoSQL is different than Relational DB's like MySQL is it schema-less. This means the only thing that is required to create our table is to simply define the indexes for it.
Before we get started... lets get some basic understanding:
DynamoDB supports two types of primary keys:
DynamoDB stores data as groups of attributes, known as items. Items are similar to rows or records in other database systems. DynamoDB stores and retrieves each item based on the primary key value which must be unique.
DynamoDB uses the partition key’s value as an input to an internal hash function. The output from the hash function determines the partition in which the item will be stored. Each item’s location is determined by the hash value of its partition key.
All items with the same partition key are stored together, and for composite partition keys, are ordered by the sort key value. DynamoDB will split partitions by sort key if the collection size grows bigger than 10 GB.
TLDR; It Just allows for faster accessing because it forces the data to be stored in an organized fashion
So anyways let's get started.
For DynamoDB ... we are going to simply create a json file for easier reading than making 1 very long command.
lets create RekousTestTable.json
{ "TableName": "RekousTestTable", "KeySchema": [ { "AttributeName": "id", "KeyType": "HASH" }, { "AttributeName": "title", "KeyType": "RANGE" } ], "AttributeDefinitions": [ { "AttributeName": "id", "AttributeType": "N" }, { "AttributeName": "title", "AttributeType": "S" } ], "ProvisionedThroughput": { "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 } }
So Keep Note: We have not added the "content" column to our table definition. NoSQL is SchemaLess. This means if we need to add a column to our table, we simply add the column when we submit data. So the only required definitions is our index'es.
So our table definition is now created... Lets take this json and create the table and use this table definition:
root@damrkul2:~/aws/dynamodb# aws dynamodb create-table --cli-input-json file://RekousTestTable.json --region us-west-2 { "TableDescription": { "TableArn": "arn:aws:dynamodb:us-west-2:841181805441:table/RekousTestTable", "AttributeDefinitions": [ { "AttributeName": "id", "AttributeType": "N" }, { "AttributeName": "title", "AttributeType": "S" } ], "ProvisionedThroughput": { "NumberOfDecreasesToday": 0, "WriteCapacityUnits": 5, "ReadCapacityUnits": 5 }, "TableSizeBytes": 0, "TableName": "RekousTestTable", "TableStatus": "CREATING", "TableId": "5d1eecbb-8d8a-4d15-9adc-281dbcc34c60", "KeySchema": [ { "KeyType": "HASH", "AttributeName": "id" }, { "KeyType": "RANGE", "AttributeName": "title" } ], "ItemCount": 0, "CreationDateTime": 1535216135.112 } }
Lets Now play with Lambda. I've created a blank serverless - nodejs template directory. Please look at the previous blog post to get
a lambda envirornment setup if you need instructions.
Here is the serverless.yaml where I've configured 3 functions for Lambda.
# # Rekous DynamoDB Lambda # service: rekous-dynamodb-test provider: name: aws runtime: nodejs8.10 region: us-west-2 functions: list: handler: handler.list events: - http: path: list/ method: get put: handler: handler.put events: - http: path: put/ method: post show: handler: handler.show events: - http: path: show/ method: post
Here is the handler.js
'use strict'; const AWS = require('aws-sdk'); const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'}); module.exports.list = function(event, context, callback) { var table = "RekousTestTable"; var params = { TableName: table, }; dynamodb.scan(params, function(err, data) { var results = []; if (err) { var temp = { error: err.stack }; results.push(temp); } else { data.Items.forEach(function(element, index, array) { var temp = { "id" : element.id.N, "title" : element.title.S } results.push(temp); }); } callback(null, { statusCode: 200, body: JSON.stringify({ reply: results, }), }); }); }; module.exports.put = function(event, context, callback) { // Parse the Body that contains our text POST JSON var input = JSON.parse(event.body); var table = "RekousTestTable"; // Incase Not submitted and left blank var content = (input.content == undefined ) ? '' : input.content; var url = (input.content == undefined ) ? '' : input.content; var item = { "id": { N: String(input.id) }, "title": { S: input.title }, "content": { S: content }, "url": { S: url }, } ; var params = { "TableName": table, "Item": item }; dynamodb.putItem(params, function(err, data) { var results = []; if (err) { results.push(err.stack); console.log(err, err.stack); } else { results.push(params.Item); } callback(null, { statusCode: 200, body: JSON.stringify({ reply: results, }), }); }); }; module.exports.show= function(event, context, callback) { // Parse the Body that contains our text POST JSON var input = JSON.parse(event.body); var table = "RekousTestTable"; var key = { "id": { N: String(input.id) }, "title": { S: input.title }, } ; var params = { "TableName": table, "Key": key }; dynamodb.getItem(params, function(err, data) { var results = []; if (err) { results.push(err.stack); console.log(err, err.stack); } else { results.push(data); } callback(null, { statusCode: 200, body: JSON.stringify({ reply: results, }), }); }); };
To Show a List of Items:
root@damrkul2:~/aws/dynamodb/reklambdatest# curl https://77acb0x6te.execute-api.us-west-2.amazonaws.com/dev/list {"reply":[{"id":"1","title":"RekTitle1"}]}
To Add/Put an Item:
root@damrkul2:~/aws/dynamodb/reklambdatest# curl -d '{"title":"RekTitle2", "id":2, "url": "rekous2.com" , "content": "my first content2"}' -X POST https://77acb0x6te.execute-api.us-west-2.amazonaws.com/dev/put {"reply":[{"id":{"N":"2"},"title":{"S":"RekTitle2"},"content":{"S":"my first content2"},"url":{"S":"my first content2"}}]}
To View/Show an Item and all its fields:
root@damrkul2:~/aws/dynamodb/reklambdatest# curl -d '{"title":"RekTitle2", "id":2}' -X POST https://77acb0x6te.execute-api.us-west-2.amazonaws.com/dev/show {"reply":[{"Item":{"content":{"S":"my first content2"},"id":{"N":"2"},"url":{"S":"my first content2"},"title":{"S":"RekTitle2"}}}]}
There ya have it... we have now implemented a solution that has a AWS service (Lambda) accessing another AWS service (DynamoDB).
Summary
Now this took me quite a while todo... as one of the reasons is that I had to figureout that I had to change the function definiton in the handler.js to take in a callback function argument, to be used in the dynamoDB in order to display data to the user.
module.exports.list = function(event, context, callback)
I had to use CloudWatch to view logs to check for errors. But I eventually, did find this utility that came with Lambda very helpful:
^^^ Was super helpful
One of the early issues while learning this was figuring out how to use lambda to connect to the DynamoDB service. The solution is to attatch an IAM policy to the Lambda service that allows Lambda to connect to Dynamo DB. I took the extra time to create my own policy for fun using the generator.
Once I attatched the policy, the Lambda Dashboard gives a nice representation of what services it currently has access to. See below:
So anyways --- there ya have it... we have now learned how to connect to 2 AWS services and have them interact with one another. If I wanted
Lambda to connect to the S3 service, I would simply need to attatch another Policy to Lambda to give it access.
Thanks for reading.