(Erratum) VPC Endpoint increases DynamoDB latency by 30%

Andreas Wittig – 20 Aug 2020

Our reader Tom wrote in to tell me, that the latency for read requests to DynamoDB increased significantly after enabling a VPC endpoint a few weeks ago. Someone else reported a similar problem in the AWS discussion forums as well. Therefore, I started to investigate to write this article. After many hours of benchmarking, I came to the conclusion that using a VPC endpoint to connect to DynamoDB increase latency by 30% compared to connections through a internet gateway or NAT gateway. Shortly, after publishing the article, Petar send me a message via Twitter to tell me that something must be wrong with my benchmark. Unfortunatly, that was correct. I made a mistake when measuring the read latency for DynamoDB from EC2.

Therefore, I have depublished the original blog post. Instead you will find a post mortem analysis in the following.

VPC Endpoint increases DynamoDB latency by 30%

The problem starts with Node.js and the AWS SDK. I was using the following script to measure the read latency for DynamoDB.

const AWS = require('aws-sdk')
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = new Date().getTime();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = new Date().getTime();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that code, requests through a VPC endpoint took about 10 ms - about 30% - longer than when connecting through an Internet gateway.

The problem with that Node.js code? Each DynamoDB API request create a new TCP connection. Doing so adds latency, and it seems like establishing a new TCP connection is taking about 30% longer when using a VPC endpoint.

However, you should not use the Node.js default in production. Instead, you need to tell the AWS SDK to reuse exsisting connections. To do so, I modified my code as described in Reusing Connections with Keep-Alive in Node.js.

const AWS = require('aws-sdk')
const https = require('https');
const agent = new https.Agent({
keepAlive: true
});
const dynamodb = new AWS.DynamoDB({httpOptions: {agent}});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = process.hrtime.bigint();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = process.hrtime.bigint();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that modification I could no longer measure any significant differences between connecting to DynamoDB through an Internet gateway, NAT gateway, or VPC endpoint.

I’m sorry about my mistake. Michael and I are working hard on publishing high-quality content.

Andreas Wittig

Andreas Wittig

I’ve been building on AWS since 2012 together with my brother Michael. We are sharing our insights into all things AWS on cloudonaut and have written the book AWS in Action. Besides that, we’re currently working on bucketAV,HyperEnv for GitHub Actions, and marbot.

Here are the contact options for feedback and questions.