From 167e961e617db32b9bbd9e70648bab48f689ea6d Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Sun, 30 Oct 2016 13:26:14 -0700 Subject: [PATCH 001/128] Add S3 gotcha; CLI/SDK Authentication --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index a8da6be..2b1a5d8 100644 --- a/README.md +++ b/README.md @@ -699,6 +699,7 @@ S3 - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- ❗If a S3 file operation using CLI or SDK doesn't work in one region but works correctly in another region, make sure you are using the latest [Authentication Signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). See also [Specifying Signature Version in Request Authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). ### Storage Durability, Availability, and Price From b783085b03d27148aa930f52da95b0d4d9130aed Mon Sep 17 00:00:00 2001 From: "jesper.eneberg" Date: Sun, 30 Oct 2016 23:39:28 +0100 Subject: [PATCH 002/128] Added Cloudfront Gotchas about 2048 bit ssl cert --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7b63221..ca5ed5b 100644 --- a/README.md +++ b/README.md @@ -1332,6 +1332,7 @@ CloudFront - 🔸If using S3 as a backing store, remember that the endpoints for website hosting and for general S3 are different. Example: “bucketname.s3.amazonaws.com” is a standard S3 serving endpoint, but to have redirect and error page support, you need to use the website hosting endpoint listed for that bucket, e.g. “bucketname.s3-website-us-east-1.amazonaws.com” (or the appropriate region). - 🔸By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. +- If you want to use a self-obtained SSL certificate with Cloudfront. Make sure it's 2048 bits since Cloudfront only support 2048 bits SSL certificates. [Ongoing discussion](https://forums.aws.amazon.com/thread.jspa?threadID=148783) DirectConnect ------------- From 39de2b41fa938aacf3e4bb581ad511c1b8cd5d55 Mon Sep 17 00:00:00 2001 From: Jesper Eneberg Date: Mon, 31 Oct 2016 00:17:54 +0100 Subject: [PATCH 003/128] Added Cloudfront Gotchas about 2048 bit ssl cert --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ca5ed5b..c0b8571 100644 --- a/README.md +++ b/README.md @@ -1332,7 +1332,7 @@ CloudFront - 🔸If using S3 as a backing store, remember that the endpoints for website hosting and for general S3 are different. Example: “bucketname.s3.amazonaws.com” is a standard S3 serving endpoint, but to have redirect and error page support, you need to use the website hosting endpoint listed for that bucket, e.g. “bucketname.s3-website-us-east-1.amazonaws.com” (or the appropriate region). - 🔸By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. -- If you want to use a self-obtained SSL certificate with Cloudfront. Make sure it's 2048 bits since Cloudfront only support 2048 bits SSL certificates. [Ongoing discussion](https://forums.aws.amazon.com/thread.jspa?threadID=148783) +- 🔸If you want to use a self-obtained SSL certificate with Cloudfront. Make sure it's 2048 bits since Cloudfront only support 2048 bits SSL certificates. [Ongoing discussion](https://forums.aws.amazon.com/thread.jspa?threadID=148783) DirectConnect ------------- From 7daaded46bfdbf6ec5dccae338b4f425a4df39ac Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Fri, 4 Nov 2016 15:08:18 -0700 Subject: [PATCH 004/128] added redshift distribution column type info affects query performance --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 04f536e..83eadb2 100644 --- a/README.md +++ b/README.md @@ -1376,7 +1376,7 @@ Redshift - Redshift uses columnar storage, hence it does not have indexing capabilities. You can, however, use distribution key [distkey](http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html) and sort key [sortkey](docs.aws.amazon.com/redshift/latest/dg/c_best-practices-sort-key.html) to improve performance. Redshift has two type of sort keys: compounding sort key and interleaved sort key. - A compound sort key is made up of all columns listed in the sort key definition. It is most useful when you have queries with operations using prefix of the sortkey. - An interleaved sort key on the other hand gives equal weight to each column or a subset of columns in the sort key. So if you don't know ahead of time which column you want to choose for sorting and filtering, this is a much better choice than the compound key.[Here](https://aws.amazon.com/blogs/aws/quickly-filter-data-in-amazon-redshift-using-interleaved-sorting/) is an example using interleaved sort key. - +- Redshift includes a ability to condigure a [distribution key](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) for each table. The distribution key that you create on a table sets the type of physical data distribution for that table in your cluster and can have a significant impact on query performance. There are three possible distribution key settings -- even (which is the default), key or all. Use 'key' to collocate join key columns for tables which are joined in queries. Use 'all' to place the data in small-sized tables on all cluster nodes. ### Redshift Gotchas and Limitations From cac46aef1ac04220c33deb7cddf3510baddb127e Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Mon, 7 Nov 2016 13:53:23 -0800 Subject: [PATCH 005/128] #279; Incorporate changes from comments --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2b1a5d8..64bb818 100644 --- a/README.md +++ b/README.md @@ -699,7 +699,7 @@ S3 - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. -- ❗If a S3 file operation using CLI or SDK doesn't work in one region but works correctly in another region, make sure you are using the latest [Authentication Signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). See also [Specifying Signature Version in Request Authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). +- 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). ### Storage Durability, Availability, and Price From 8440a74af687fffd04529719abaa2d05ef87569e Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Mon, 7 Nov 2016 14:12:51 -0800 Subject: [PATCH 006/128] Resolve conflict --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 6ea95d5..2836305 100644 --- a/README.md +++ b/README.md @@ -699,12 +699,8 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. -<<<<<<< HEAD - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. - 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). -======= -- ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. ->>>>>>> master ### Storage Durability, Availability, and Price From 5c93e63c434cfa5910f2a4f2a23e0f6275aa3e73 Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Mon, 7 Nov 2016 14:27:24 -0800 Subject: [PATCH 007/128] fix merge conflict --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 439e0b8..ece8220 100644 --- a/README.md +++ b/README.md @@ -699,7 +699,12 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. +<<<<<<< 7f5ccf5d8bb6eea6e33a071dfb5d792da92c833f - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +======= +- ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). +>>>>>>> #279; Incorporate changes from comments ### Storage Durability, Availability, and Price From c288d8f4174e3b83e3dc5c6ad87411e0aaf7e9d0 Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Mon, 7 Nov 2016 14:32:10 -0800 Subject: [PATCH 008/128] fix added lines --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index ece8220..668ce42 100644 --- a/README.md +++ b/README.md @@ -699,12 +699,8 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. -<<<<<<< 7f5ccf5d8bb6eea6e33a071dfb5d792da92c833f - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. -======= -- ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. - 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). ->>>>>>> #279; Incorporate changes from comments ### Storage Durability, Availability, and Price From 35e29240db255003e61d3692386a908a58c837eb Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Mon, 7 Nov 2016 14:27:24 -0800 Subject: [PATCH 009/128] fix merge conflict --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 439e0b8..ece8220 100644 --- a/README.md +++ b/README.md @@ -699,7 +699,12 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. +<<<<<<< 7f5ccf5d8bb6eea6e33a071dfb5d792da92c833f - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +======= +- ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). +>>>>>>> #279; Incorporate changes from comments ### Storage Durability, Availability, and Price From dfecf5614ece7da62bca75b4962a3623e3a70641 Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Mon, 7 Nov 2016 14:32:10 -0800 Subject: [PATCH 010/128] fix added lines --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index ece8220..668ce42 100644 --- a/README.md +++ b/README.md @@ -699,12 +699,8 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. -<<<<<<< 7f5ccf5d8bb6eea6e33a071dfb5d792da92c833f - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. -======= -- ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. - 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). ->>>>>>> #279; Incorporate changes from comments ### Storage Durability, Availability, and Price From be30ea3443d5c55769813cb5ebbecf408617d50f Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Sun, 30 Oct 2016 21:07:01 -0700 Subject: [PATCH 011/128] Clarify gotcha severities. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 853adcd..04f536e 100644 --- a/README.md +++ b/README.md @@ -114,8 +114,8 @@ Legend - 📒 Marks standard/official AWS pages and docs - 🔹 Important or often overlooked tip -- ❗ Gotcha or warning (where risks or time or resource costs are significant) -- 🔸 Limitation or quirk (where it’s not quite so bad) +- ❗ “Serious” gotcha (used where risks or time or resource costs are significant: critical security risks, mistakes with significant financial cost, or poor architectural choices that are fundamentally difficult to correct) +- 🔸 “Regular” gotcha, limitation, or quirk (used where where consequences are things not working, breaking, or not scaling gracefully) - 📜 Undocumented feature (folklore) - 🐥 Relatively new (and perhaps immature) services or features - ⏱ Performance discussions From e1b1607adfd34ca48aa3762cbef89ee02c700e50 Mon Sep 17 00:00:00 2001 From: Abdul Mohammed Date: Tue, 25 Oct 2016 17:08:13 -0500 Subject: [PATCH 012/128] added user-data limit of 16K to EC2 limitation section added user-data limit of 16K to EC2 limitation section --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 04f536e..7e1de40 100644 --- a/README.md +++ b/README.md @@ -772,6 +772,7 @@ EC2 - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. - ❗⏱ Instances come in two types: **Fixed Performance Instances** (e.g. M3, C3, and R3) and [**Burstable Performance Instances**](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. +- Instance user-data is (limited to 16KB)[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data](This limit applies to the data in raw form, not base64-encoded form). If larger data is needed, it can be downloaded from S3 by a user-data script. AMIs ---- From ea4986672cc4defab2ba2264d2ff5a9894b2586f Mon Sep 17 00:00:00 2001 From: Abdul Mohammed Date: Thu, 27 Oct 2016 09:49:59 -0500 Subject: [PATCH 013/128] updated EC2 16 KB user-data limits updated EC2 16 KB user-data limits --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7e1de40..79bc412 100644 --- a/README.md +++ b/README.md @@ -772,7 +772,7 @@ EC2 - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. - ❗⏱ Instances come in two types: **Fixed Performance Instances** (e.g. M3, C3, and R3) and [**Burstable Performance Instances**](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. -- Instance user-data is (limited to 16KB)[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data](This limit applies to the data in raw form, not base64-encoded form). If larger data is needed, it can be downloaded from S3 by a user-data script. +- Instance user-data is (limited to 16 KB)[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data](This limit applies to the data in raw form, not base64-encoded form). If more data is needed, it can be downloaded from S3 by a user-data script. AMIs ---- From ee63b665764718c1d2ed55e6b0dad7c25776b81a Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sun, 30 Oct 2016 12:59:12 -0700 Subject: [PATCH 014/128] add info on non-default cluster param groups similar to info on this topic in RDS, just ran into it with a client (again!) last week, so wanted to add it here --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 79bc412..ca32802 100644 --- a/README.md +++ b/README.md @@ -1368,6 +1368,7 @@ Redshift - Although Redshift is mostly Postgres-compatible, its SQL dialect and performance profile are different. - Redshift supports only [12 primitive data types](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html). ([List of unsupported Postgres types](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-datatypes.html)\) - It has a leader node and computation nodes (the leader node distributes queries to the computation ones). Note that some functions [can be executed only on the lead node.](https://docs.aws.amazon.com/redshift/latest/dg/c_SQL_functions_leader_node_only.html) +- 🔹Make sure to create a new [cluster parameter group](http://docs.aws.amazon.com/redshift/latest/mgmt/working-with-parameter-groups.html) and option group for your database since the default parameter group does not allow dynamic configuration changes. - Major third-party BI tools support Redshift integration (see [Quora](https://www.quora.com/Which-BI-visualisation-solution-goes-best-with-Redshift)). - [Top 10 Performance Tuning Techniques for Amazon Redshift](https://blogs.aws.amazon.com/bigdata/post/Tx31034QG0G3ED1/Top-10-Performance-Tuning-Techniques-for-Amazon-Redshift) provides an excellent list of performance tuning techniques. - [Amazon Redshift Utils](https://github.com/awslabs/amazon-redshift-utils) contains useful utilities, scripts and views to simplify Redshift ops. From 14f83ee6509dd465d7cdad44c8d4da2e050f54a5 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sun, 6 Nov 2016 19:04:34 -0800 Subject: [PATCH 015/128] added link to AWS well-architected framework guide (#289) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index ca32802..8298ab3 100644 --- a/README.md +++ b/README.md @@ -1690,6 +1690,7 @@ This section covers a few unusually useful or “must know about” resources or - [Resilience and Reliability on AWS](http://shop.oreilly.com/product/0636920026839.do) - [AWS documentation as Kindle ebooks](https://www.amazon.com/Amazon-Web-Services/e/B007R6MVQ6) - General references + - [AWS Well Architected Framework Guide](https://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf): Amazon's own 56 page guide to operational excellence - guidelines and checklists to validate baseline security, reliability, performance (including high availability) and cost optimization practices - [Awesome Microservices](https://github.com/mfornos/awesome-microservices): A curated list of tools and technologies for microservice architectures. Worth browsing to learn about popular open source projects. - [Is it fast yet?](https://istlsfastyet.com/): Ilya Grigorik’s TLS performance overview - [High Performance Browser Networking](https://hpbn.co/): A full, modern book on web network performance; a presentation on the HTTP/2 portion is [here](https://docs.google.com/presentation/d/1r7QXGYOLCh4fcUq0jDdDwKJWNqWK1o4xMtYpKZCJYjM/edit?usp=sharing). From ccae227df0c23490e2e7d819414ed9fc63a48f02 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Sun, 6 Nov 2016 19:06:39 -0800 Subject: [PATCH 016/128] PR 289 nit: missing period at end of sentence. (#297) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8298ab3..13fe40e 100644 --- a/README.md +++ b/README.md @@ -1690,7 +1690,7 @@ This section covers a few unusually useful or “must know about” resources or - [Resilience and Reliability on AWS](http://shop.oreilly.com/product/0636920026839.do) - [AWS documentation as Kindle ebooks](https://www.amazon.com/Amazon-Web-Services/e/B007R6MVQ6) - General references - - [AWS Well Architected Framework Guide](https://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf): Amazon's own 56 page guide to operational excellence - guidelines and checklists to validate baseline security, reliability, performance (including high availability) and cost optimization practices + - [AWS Well Architected Framework Guide](https://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf): Amazon's own 56 page guide to operational excellence - guidelines and checklists to validate baseline security, reliability, performance (including high availability) and cost optimization practices. - [Awesome Microservices](https://github.com/mfornos/awesome-microservices): A curated list of tools and technologies for microservice architectures. Worth browsing to learn about popular open source projects. - [Is it fast yet?](https://istlsfastyet.com/): Ilya Grigorik’s TLS performance overview - [High Performance Browser Networking](https://hpbn.co/): A full, modern book on web network performance; a presentation on the HTTP/2 portion is [here](https://docs.google.com/presentation/d/1r7QXGYOLCh4fcUq0jDdDwKJWNqWK1o4xMtYpKZCJYjM/edit?usp=sharing). From 65c70391201af604344c5668dd375b4d8c0efbef Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Sun, 6 Nov 2016 19:09:03 -0800 Subject: [PATCH 017/128] Add Network Performance to EC2 Basics (#296) fix merge conflict fix added lines fix added lines --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 13fe40e..668ce42 100644 --- a/README.md +++ b/README.md @@ -700,6 +700,7 @@ S3 - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). ### Storage Durability, Availability, and Price @@ -725,6 +726,7 @@ EC2 - **EC2** (Elastic Compute Cloud) is AWS’ offering of the most fundamental piece of cloud computing: A [virtual private server](https://en.wikipedia.org/wiki/Virtual_private_server). These “instances” can run [most Linux, BSD, and Windows operating systems](https://aws.amazon.com/ec2/faqs/#What_operating_system_environments_are_supported). Internally, they use [Xen](https://en.wikipedia.org/wiki/Xen) virtualization. - The term “EC2” is sometimes used to refer to the servers themselves, but technically refers more broadly to a whole collection of supporting services, too, like load balancing (CLBs/ALBs), IP addresses (EIPs), bootable images (AMIs), security groups, and network drives (EBS) (which we discuss individually in this guide). - 💸**[EC2 pricing](https://aws.amazon.com/ec2/pricing/)** and **[cost management](#ec2-cost-management)** is a complicated topic. It can range from free (on the [AWS free tier](https://aws.amazon.com/free/)) to a lot, depending on your usage. Pricing is by instance type, by hour and changes depending on AWS region and whether you are purchasing your instances [On-Demand](https://aws.amazon.com/ec2/pricing/on-demand/), on the [Spot market](https://aws.amazon.com/ec2/spot/) or pre-purchasing ([Reserved Instances](https://aws.amazon.com/ec2/pricing/reserved-instances/)). +- **Network Performance:** For some instance types, AWS uses general terms like Low, Medium, and High to refer to network performance. Users have done [benchmarking](http://stackoverflow.com/questions/18507405/ec2-instance-typess-exact-network-performance) to provide expectations for what these terms can mean. ### EC2 Alternatives and Lock-In From edfe6f6e93ae4dc96d64bd354f0be581832cad8f Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Mon, 7 Nov 2016 17:44:39 -0800 Subject: [PATCH 018/128] updated to include @jlevy requested changes --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 83eadb2..0aade45 100644 --- a/README.md +++ b/README.md @@ -1376,7 +1376,7 @@ Redshift - Redshift uses columnar storage, hence it does not have indexing capabilities. You can, however, use distribution key [distkey](http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html) and sort key [sortkey](docs.aws.amazon.com/redshift/latest/dg/c_best-practices-sort-key.html) to improve performance. Redshift has two type of sort keys: compounding sort key and interleaved sort key. - A compound sort key is made up of all columns listed in the sort key definition. It is most useful when you have queries with operations using prefix of the sortkey. - An interleaved sort key on the other hand gives equal weight to each column or a subset of columns in the sort key. So if you don't know ahead of time which column you want to choose for sorting and filtering, this is a much better choice than the compound key.[Here](https://aws.amazon.com/blogs/aws/quickly-filter-data-in-amazon-redshift-using-interleaved-sorting/) is an example using interleaved sort key. -- Redshift includes a ability to condigure a [distribution key](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) for each table. The distribution key that you create on a table sets the type of physical data distribution for that table in your cluster and can have a significant impact on query performance. There are three possible distribution key settings -- even (which is the default), key or all. Use 'key' to collocate join key columns for tables which are joined in queries. Use 'all' to place the data in small-sized tables on all cluster nodes. +- 🔸⏱ **Distribution strategies:** Since data in Redshift is physically distributed among nodes, choosing the right data **distribution key** and [distribution style](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) is crucial for adequate query performance. There are three possible distribution style settings settings — **EVEN** (the default), **KEY**, or **ALL**. Use KEY to collocate join key columns for tables which are joined in queries. Use ALL to place the data in small-sized tables on all cluster nodes. ### Redshift Gotchas and Limitations From 2aa81d5c63a55dfeea0816cc59f9ee284ab48f70 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Fri, 11 Nov 2016 02:54:07 +0100 Subject: [PATCH 019/128] Add more details about ACM (#298) * Add more details about ACM * Fixes for #298 --- README.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 690d7c5..db9ae2a 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ Table of Contents | [Device Farm](#device-farm) | [📗](#device-farm-basics) | [📘](#device-farm-tips) | [📙](#device-farm-gotchas-and-limitations) | | [IoT](#iot) | [📗](#iot-basics) | [📘](#iot-tips) | [📙](#iot-gotchas-and-limitations) | | [SES](#ses) | [📗](#ses-basics) | [📘](#ses-tips) | [📙](#ses-gotchas-and-limitations) | -| [Certificate Manager](#certificate-manager) | [📗](#certificate-manager-basics) | | | +| [Certificate Manager](#certificate-manager) | [📗](#certificate-manager-basics) | [📘](#certificate-manager-tips) | [📙](#certificate-manager-gotchas-and-limitations) | **Special Topics** @@ -1582,9 +1582,17 @@ Certificate Manager ### Certificate Manager Alternatives and Lock-in -- Certificates issued by the **Certificate Manager** can't be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. +- ⛓Certificates issued by the Certificate Manager can't be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. -🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) +### Certificate Manager Tips + +- 🔹**Supported services:** Elastic Load Balancer and CloudFront. +- 🔸During certificate validation process Certificate Manager will send an email to every contact address specified in WHOIS and up to five common administrative addresses. Some anti-spam filters can mark emails as spam because of this. You should check the spam folder of your email if you don't receive a confirmation email. + +### Certificate Manager Gotchas and Limitations + +- 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from US East (N. Virginia) region. Certificates from other regions can be used for Elastic Load Balancer only. See [link](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). +- 🔸**IoT** use their own process for work with certificates. Details described [here](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html). High Availability ----------------- From 775e1c1d402bac3e37516a6d292dc2c5ffcbe4b4 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Fri, 11 Nov 2016 20:25:05 -0800 Subject: [PATCH 020/128] Fixing ACM nits --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index fc3b620..8943b90 100644 --- a/README.md +++ b/README.md @@ -1583,17 +1583,17 @@ Certificate Manager ### Certificate Manager Alternatives and Lock-in -- ⛓Certificates issued by the Certificate Manager can't be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. +- ⛓Certificates issued by the Certificate Manager can’t be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. ### Certificate Manager Tips -- 🔹**Supported services:** Elastic Load Balancer and CloudFront. -- 🔸During certificate validation process Certificate Manager will send an email to every contact address specified in WHOIS and up to five common administrative addresses. Some anti-spam filters can mark emails as spam because of this. You should check the spam folder of your email if you don't receive a confirmation email. +- 🔹**Supported services:** Managed [Load Balancers](#load-balancers) and [CloudFront](#cloudfront). +- 🔸During the domain validation process, Certificate Manager will send an email to every contact address specified in the domain’s WHOIS record and up to five common administrative addresses. Some anti-spam filters can mark emails as spam because of this. You should check the spam folder of your email if you don’t receive a confirmation email. ### Certificate Manager Gotchas and Limitations -- 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from US East (N. Virginia) region. Certificates from other regions can be used for Elastic Load Balancer only. See [link](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). -- 🔸**IoT** use their own process for work with certificates. Details described [here](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html). +- 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from us-east-1 (N. Virginia) region. Certificates from other regions can [only be used with Elastic Load Balancers](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). +- 🔸**IoT** has its [own way](http://docs.aws.amazon.com/iot/latest/developerguide/create-device-certificate.html) of setting up certificates. High Availability ----------------- From f55aa69fd639c88e0f73d66b5743c9250c65f68e Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Fri, 11 Nov 2016 22:12:21 -0800 Subject: [PATCH 021/128] added links to Lambda code samples from my 'big link list' --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index fc3b620..c2db588 100644 --- a/README.md +++ b/README.md @@ -1189,6 +1189,13 @@ Lambda - 🔸Currently, [Lambda does not natively provide for environment variables](https://forums.aws.amazon.com/thread.jspa?messageID=686261), which are a [common way](https://12factor.net/config) of storing configuration, so you must rely on additional mechanisms to achieve this (such as the [Serverless framework](https://serverless.com/framework/docs/guide/environment-variable-handling/#environment-variables-in-serverless) or [apex](http://apex.run/#environment-variables)). - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. +### Lambda Code Samples + +- Example of using Lambda to 'fan-out' or copy data from one service, in this case Kinesis to multiple other AWS data service as [Fan-out](https://github.com/awslabs/aws-lambda-fanout). Destinations for fan-out data in the sample include IoT, SQS and more. +- AWS Service Limit Monitor [using Lambdas](https://github.com/awslabs/aws-limit-monitor). Example code shows use of multiple lambdas for monitoring. +- Example of using Lambda with ECS in an 'extended worker' pattern via [Lambda ECS Worker Pattern](https://github.com/awslabs/lambda-ecs-worker-pattern). Example code shows use of lambda in a workflow, data from S3 is picked up by the lambda, pushed to a queue, then sent to ECS for more processing. +- Sample Java application which uses Lambda and API Gateway with Cognito (for user identity) via [Secure Pet Store](https://github.com/awslabs/api-gateway-secure-pet-store). Example shows services working together in a complete application. + 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) API Gateway From 542f8d32f34f40056d932f311f90c2eeccaa3d0b Mon Sep 17 00:00:00 2001 From: Dan Bower Date: Mon, 14 Nov 2016 18:04:02 +0000 Subject: [PATCH 022/128] Remove word repetitions The repetitions were "where where" and "settings settings". --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fc3b620..389ac3a 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ Legend - 📒 Marks standard/official AWS pages and docs - 🔹 Important or often overlooked tip - ❗ “Serious” gotcha (used where risks or time or resource costs are significant: critical security risks, mistakes with significant financial cost, or poor architectural choices that are fundamentally difficult to correct) -- 🔸 “Regular” gotcha, limitation, or quirk (used where where consequences are things not working, breaking, or not scaling gracefully) +- 🔸 “Regular” gotcha, limitation, or quirk (used where consequences are things not working, breaking, or not scaling gracefully) - 📜 Undocumented feature (folklore) - 🐥 Relatively new (and perhaps immature) services or features - ⏱ Performance discussions @@ -1403,7 +1403,7 @@ Redshift - Redshift uses columnar storage, hence it does not have indexing capabilities. You can, however, use distribution key [distkey](http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html) and sort key [sortkey](docs.aws.amazon.com/redshift/latest/dg/c_best-practices-sort-key.html) to improve performance. Redshift has two type of sort keys: compounding sort key and interleaved sort key. - A compound sort key is made up of all columns listed in the sort key definition. It is most useful when you have queries with operations using prefix of the sortkey. - An interleaved sort key on the other hand gives equal weight to each column or a subset of columns in the sort key. So if you don't know ahead of time which column you want to choose for sorting and filtering, this is a much better choice than the compound key.[Here](https://aws.amazon.com/blogs/aws/quickly-filter-data-in-amazon-redshift-using-interleaved-sorting/) is an example using interleaved sort key. -- 🔸⏱ **Distribution strategies:** Since data in Redshift is physically distributed among nodes, choosing the right data **distribution key** and [distribution style](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) is crucial for adequate query performance. There are three possible distribution style settings settings — **EVEN** (the default), **KEY**, or **ALL**. Use KEY to collocate join key columns for tables which are joined in queries. Use ALL to place the data in small-sized tables on all cluster nodes. +- 🔸⏱ **Distribution strategies:** Since data in Redshift is physically distributed among nodes, choosing the right data **distribution key** and [distribution style](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) is crucial for adequate query performance. There are three possible distribution style settings — **EVEN** (the default), **KEY**, or **ALL**. Use KEY to collocate join key columns for tables which are joined in queries. Use ALL to place the data in small-sized tables on all cluster nodes. ### Redshift Gotchas and Limitations From 6a7fab8a10b9015cc272e344c184effeb3ef67f4 Mon Sep 17 00:00:00 2001 From: Bogdan Luput Date: Sat, 12 Nov 2016 08:22:23 +0200 Subject: [PATCH 023/128] Rewrite EFS tips section --- README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index fc3b620..835c2cf 100644 --- a/README.md +++ b/README.md @@ -876,14 +876,17 @@ EFS - High availability is achieved by having [mount targets in different subnets / availability zones](http://docs.aws.amazon.com/efs/latest/ug/images/overview-flow.png). ### EFS Tips -- ⏱ Carefully consider using one EFS for multiple applications. Any directory on the EFS can be mounted, it doesn't have to be the root directory. One application could mount fs-12345678:/prog1, another fs-12345678:/prog2. [User and group level permissions](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html) can be used to limit access to some extent. + +- With EFS being based on NFSv4.1, any directory on the EFS can be mounted directly, it doesn't have to be the root directory. One application could mount *fs-12345678:/prog1*, another *fs-12345678:/prog2*. +- [User and group level permissions](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html) can be used to control access to certain directories on the EFS file system. +- ⏱ One EFS file system can be used for multiple applications or services, but it should be considered carefully: Pros: - - Because performance is based on total size of stored files, having everything on one drive will increase performance for everyone and might offset one application consuming credits faster than it can accumulate by another application that just stores files on EFS and rarely accesses them. + - Because performance is based on total size of stored files, having everything on one drive will increase performance for everyone. One application consuming credits faster than it can accumulate might be offset by another application that just stores files on EFS and rarely accesses them. Cons: - - Since credits are shared, if one application gets out of control it will affect the others. - - Security of the drive is compromised. All clients will have to have network access to the drive and any of the clients can mount another application's folder. Someone with root access on one client instance can easily read and write to any application's mount. + - Since credits are shared, if one application over-consumes them, it will affect the others. + - A compromise is made with regards to [security](http://docs.aws.amazon.com/efs/latest/ug/security-considerations.html). All clients will have to have network access to the drive. Someone with root access on one client instance can mount any directory on the EFS and they have read-write access to all files on the drive, even if they don't have access to the applications hosted on other clients. ### EFS Gotchas and Limitations From 8eafa8940ccf1fc0aceed2ae6be7d341539a9f6c Mon Sep 17 00:00:00 2001 From: Noah Zoschke Date: Tue, 15 Nov 2016 19:24:59 -0800 Subject: [PATCH 024/128] KMS basics, tips, gotchas and link to tutorial blog (#307) * KMS basics, tips, gotchas and link to tutorial blog * feedback on KMS formatting and links * copy editing and remove stray line --- README.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 389ac3a..d217671 100644 --- a/README.md +++ b/README.md @@ -1319,15 +1319,24 @@ KMS ### KMS Basics - 📒 [Homepage](https://aws.amazon.com/kms/) ∙ [Developer guide](http://docs.aws.amazon.com/kms/latest/developerguide/) ∙ [FAQ](https://aws.amazon.com/kms/faqs/) ∙ [Pricing](https://aws.amazon.com/kms/pricing/) -- **KMS** (Key Management Service) is secure service for storing keys, such encryption keys for [EBS](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) and [S3](http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). +- **KMS** (Key Management Service) is a secure service for creating, storing and auditing usage of cryptographic keys. +- **Service integration:** KMS [integrates with other AWS services](http://docs.aws.amazon.com/kms/latest/developerguide/service-integration.html): EBS, Elastic Transcoder, EMR, Redshift, RDS, SES, S3, WorkMail and Workspaces. +- **Encryption APIs:** The [Encrypt](http://docs.aws.amazon.com/kms/latest/APIReference/API_Encrypt.html) and [Decrypt API](http://docs.aws.amazon.com/kms/latest/APIReference/API_Decrypt.html) allow you to encrypt and decrypt data on the KMS service side, never exposing the master key contents. +- **Data keys:** The [GenerateDataKey](http://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys) API generates a new key off of a master key. The data key contents are exposed to you so you can use it to encrypt and decrypt any size of data in your application layer. KMS does not store, manage or track data keys, you are responsible for this in your application. +- 🔹**Auditing:** Turn on CloudTrail to audit all KMS API events. +- **Access:** Use [key policies](http://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) and [IAM policies](http://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html) to grant different levels of KMS access. For example, you create an IAM policy that only [allows a user to encrypt and decrypt with a specific key](http://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html#iam-policy-example-encrypt-decrypt-specific-cmks). ### KMS Tips - 🔹It’s very common for companies to manage keys completely via home-grown mechanisms, but it’s far preferable to use a service such as KMS from the beginning, as it encourages more secure design and improves policies and processes around managing keys. - A good motivation and overview is in [this AWS presentation](http://www.slideshare.net/AmazonWebServices/encryption-and-key-management-in-aws). - The cryptographic details are in [this AWS whitepaper](https://d0.awsstatic.com/whitepapers/KMS-Cryptographic-Details.pdf). +- [This blog from Convox](https://convox.com/blog/encryption-at-rest/) demonstrates why and how to use KMS for encryption at rest. -🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) +### KMS Gotchas and Limitations + +- 🔸The Encrypt API only works with < 4KB of data. Larger data requires generating and managing a [data key](http://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys) in your application layer. +- 🔸KMS audit events are not available in the [CloudTrail Lookup Events API](http://docs.aws.amazon.com/awscloudtrail/latest/APIReference/API_LookupEvents.html). You need to look find them in the raw .json.gz files that CloudTrail saves in S3. CloudFront ---------- From 76e49aba09f34111c50d83d1ef312923bcaab6a2 Mon Sep 17 00:00:00 2001 From: Vincent DM Date: Sat, 19 Nov 2016 18:23:11 +0100 Subject: [PATCH 025/128] Route 53: domain registration != hosted zones Issue #107 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2462f8a..df72834 100644 --- a/README.md +++ b/README.md @@ -1242,6 +1242,7 @@ Route 53 - Somewhat confusingly, you can have CNAME and A aliases, depending on the type of the target. - Because aliases are extensions to regular DNS records, if exported, the output [zone file](https://en.wikipedia.org/wiki/Zone_file) will have additional non-standard “ALIAS” lines in it. - Take advantage of AWS Route 53 latency based routing. This means that your users around the globe are automatically directed to the nearest AWS region where you are running in terms of having the shortest latency. +- Understand that domain registration and DNS management (hosted zones) are two separate Route 53 services. When you buy/transfer a domain, Route 53 automaticcaly assigns four name servers to it (e.g. ns-2.awsdns-00.com). Route 53 also offers to automatically create a hosted zone for DNS management, but you are not required do do your DNS management in the same account or even in Route 53. You just need to create an NS record pointing to the servers assigned to your domain in Route 53. A use case would be to put your domain registration (very mission critical) in a [bastion account] (https://cloudonaut.io/your-single-aws-account-is-a-serious-risk/) while managing the hosted zones within another account which is accessible by your applications. CloudFormation -------------- From 7899e30e68dcca8c51288e9a0a702a6ccbddafe5 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Fri, 11 Nov 2016 20:25:05 -0800 Subject: [PATCH 026/128] Fixing ACM nits --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index fc3b620..8943b90 100644 --- a/README.md +++ b/README.md @@ -1583,17 +1583,17 @@ Certificate Manager ### Certificate Manager Alternatives and Lock-in -- ⛓Certificates issued by the Certificate Manager can't be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. +- ⛓Certificates issued by the Certificate Manager can’t be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. ### Certificate Manager Tips -- 🔹**Supported services:** Elastic Load Balancer and CloudFront. -- 🔸During certificate validation process Certificate Manager will send an email to every contact address specified in WHOIS and up to five common administrative addresses. Some anti-spam filters can mark emails as spam because of this. You should check the spam folder of your email if you don't receive a confirmation email. +- 🔹**Supported services:** Managed [Load Balancers](#load-balancers) and [CloudFront](#cloudfront). +- 🔸During the domain validation process, Certificate Manager will send an email to every contact address specified in the domain’s WHOIS record and up to five common administrative addresses. Some anti-spam filters can mark emails as spam because of this. You should check the spam folder of your email if you don’t receive a confirmation email. ### Certificate Manager Gotchas and Limitations -- 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from US East (N. Virginia) region. Certificates from other regions can be used for Elastic Load Balancer only. See [link](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). -- 🔸**IoT** use their own process for work with certificates. Details described [here](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html). +- 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from us-east-1 (N. Virginia) region. Certificates from other regions can [only be used with Elastic Load Balancers](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). +- 🔸**IoT** has its [own way](http://docs.aws.amazon.com/iot/latest/developerguide/create-device-certificate.html) of setting up certificates. High Availability ----------------- From 07ee1663473d0d2aef59c9726e5d6574d88d637a Mon Sep 17 00:00:00 2001 From: Dan Bower Date: Mon, 14 Nov 2016 18:04:02 +0000 Subject: [PATCH 027/128] Remove word repetitions The repetitions were "where where" and "settings settings". --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8943b90..ee38069 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ Legend - 📒 Marks standard/official AWS pages and docs - 🔹 Important or often overlooked tip - ❗ “Serious” gotcha (used where risks or time or resource costs are significant: critical security risks, mistakes with significant financial cost, or poor architectural choices that are fundamentally difficult to correct) -- 🔸 “Regular” gotcha, limitation, or quirk (used where where consequences are things not working, breaking, or not scaling gracefully) +- 🔸 “Regular” gotcha, limitation, or quirk (used where consequences are things not working, breaking, or not scaling gracefully) - 📜 Undocumented feature (folklore) - 🐥 Relatively new (and perhaps immature) services or features - ⏱ Performance discussions @@ -1403,7 +1403,7 @@ Redshift - Redshift uses columnar storage, hence it does not have indexing capabilities. You can, however, use distribution key [distkey](http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html) and sort key [sortkey](docs.aws.amazon.com/redshift/latest/dg/c_best-practices-sort-key.html) to improve performance. Redshift has two type of sort keys: compounding sort key and interleaved sort key. - A compound sort key is made up of all columns listed in the sort key definition. It is most useful when you have queries with operations using prefix of the sortkey. - An interleaved sort key on the other hand gives equal weight to each column or a subset of columns in the sort key. So if you don't know ahead of time which column you want to choose for sorting and filtering, this is a much better choice than the compound key.[Here](https://aws.amazon.com/blogs/aws/quickly-filter-data-in-amazon-redshift-using-interleaved-sorting/) is an example using interleaved sort key. -- 🔸⏱ **Distribution strategies:** Since data in Redshift is physically distributed among nodes, choosing the right data **distribution key** and [distribution style](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) is crucial for adequate query performance. There are three possible distribution style settings settings — **EVEN** (the default), **KEY**, or **ALL**. Use KEY to collocate join key columns for tables which are joined in queries. Use ALL to place the data in small-sized tables on all cluster nodes. +- 🔸⏱ **Distribution strategies:** Since data in Redshift is physically distributed among nodes, choosing the right data **distribution key** and [distribution style](http://docs.aws.amazon.com/redshift/latest/dg/c_choosing_dist_sort.html) is crucial for adequate query performance. There are three possible distribution style settings — **EVEN** (the default), **KEY**, or **ALL**. Use KEY to collocate join key columns for tables which are joined in queries. Use ALL to place the data in small-sized tables on all cluster nodes. ### Redshift Gotchas and Limitations From 2ef36c92e78101c68c98fc2a20779abbbd9ec1af Mon Sep 17 00:00:00 2001 From: Bogdan Luput Date: Sat, 12 Nov 2016 08:22:23 +0200 Subject: [PATCH 028/128] Rewrite EFS tips section --- README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index ee38069..0305f74 100644 --- a/README.md +++ b/README.md @@ -876,14 +876,17 @@ EFS - High availability is achieved by having [mount targets in different subnets / availability zones](http://docs.aws.amazon.com/efs/latest/ug/images/overview-flow.png). ### EFS Tips -- ⏱ Carefully consider using one EFS for multiple applications. Any directory on the EFS can be mounted, it doesn't have to be the root directory. One application could mount fs-12345678:/prog1, another fs-12345678:/prog2. [User and group level permissions](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html) can be used to limit access to some extent. + +- With EFS being based on NFSv4.1, any directory on the EFS can be mounted directly, it doesn't have to be the root directory. One application could mount *fs-12345678:/prog1*, another *fs-12345678:/prog2*. +- [User and group level permissions](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html) can be used to control access to certain directories on the EFS file system. +- ⏱ One EFS file system can be used for multiple applications or services, but it should be considered carefully: Pros: - - Because performance is based on total size of stored files, having everything on one drive will increase performance for everyone and might offset one application consuming credits faster than it can accumulate by another application that just stores files on EFS and rarely accesses them. + - Because performance is based on total size of stored files, having everything on one drive will increase performance for everyone. One application consuming credits faster than it can accumulate might be offset by another application that just stores files on EFS and rarely accesses them. Cons: - - Since credits are shared, if one application gets out of control it will affect the others. - - Security of the drive is compromised. All clients will have to have network access to the drive and any of the clients can mount another application's folder. Someone with root access on one client instance can easily read and write to any application's mount. + - Since credits are shared, if one application over-consumes them, it will affect the others. + - A compromise is made with regards to [security](http://docs.aws.amazon.com/efs/latest/ug/security-considerations.html). All clients will have to have network access to the drive. Someone with root access on one client instance can mount any directory on the EFS and they have read-write access to all files on the drive, even if they don't have access to the applications hosted on other clients. ### EFS Gotchas and Limitations From 0f5e8545eb940ff193247f16e148479db8c5d920 Mon Sep 17 00:00:00 2001 From: Noah Zoschke Date: Tue, 15 Nov 2016 19:24:59 -0800 Subject: [PATCH 029/128] KMS basics, tips, gotchas and link to tutorial blog (#307) * KMS basics, tips, gotchas and link to tutorial blog * feedback on KMS formatting and links * copy editing and remove stray line --- README.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0305f74..2462f8a 100644 --- a/README.md +++ b/README.md @@ -1322,15 +1322,24 @@ KMS ### KMS Basics - 📒 [Homepage](https://aws.amazon.com/kms/) ∙ [Developer guide](http://docs.aws.amazon.com/kms/latest/developerguide/) ∙ [FAQ](https://aws.amazon.com/kms/faqs/) ∙ [Pricing](https://aws.amazon.com/kms/pricing/) -- **KMS** (Key Management Service) is secure service for storing keys, such encryption keys for [EBS](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) and [S3](http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). +- **KMS** (Key Management Service) is a secure service for creating, storing and auditing usage of cryptographic keys. +- **Service integration:** KMS [integrates with other AWS services](http://docs.aws.amazon.com/kms/latest/developerguide/service-integration.html): EBS, Elastic Transcoder, EMR, Redshift, RDS, SES, S3, WorkMail and Workspaces. +- **Encryption APIs:** The [Encrypt](http://docs.aws.amazon.com/kms/latest/APIReference/API_Encrypt.html) and [Decrypt API](http://docs.aws.amazon.com/kms/latest/APIReference/API_Decrypt.html) allow you to encrypt and decrypt data on the KMS service side, never exposing the master key contents. +- **Data keys:** The [GenerateDataKey](http://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys) API generates a new key off of a master key. The data key contents are exposed to you so you can use it to encrypt and decrypt any size of data in your application layer. KMS does not store, manage or track data keys, you are responsible for this in your application. +- 🔹**Auditing:** Turn on CloudTrail to audit all KMS API events. +- **Access:** Use [key policies](http://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) and [IAM policies](http://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html) to grant different levels of KMS access. For example, you create an IAM policy that only [allows a user to encrypt and decrypt with a specific key](http://docs.aws.amazon.com/kms/latest/developerguide/iam-policies.html#iam-policy-example-encrypt-decrypt-specific-cmks). ### KMS Tips - 🔹It’s very common for companies to manage keys completely via home-grown mechanisms, but it’s far preferable to use a service such as KMS from the beginning, as it encourages more secure design and improves policies and processes around managing keys. - A good motivation and overview is in [this AWS presentation](http://www.slideshare.net/AmazonWebServices/encryption-and-key-management-in-aws). - The cryptographic details are in [this AWS whitepaper](https://d0.awsstatic.com/whitepapers/KMS-Cryptographic-Details.pdf). +- [This blog from Convox](https://convox.com/blog/encryption-at-rest/) demonstrates why and how to use KMS for encryption at rest. -🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) +### KMS Gotchas and Limitations + +- 🔸The Encrypt API only works with < 4KB of data. Larger data requires generating and managing a [data key](http://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys) in your application layer. +- 🔸KMS audit events are not available in the [CloudTrail Lookup Events API](http://docs.aws.amazon.com/awscloudtrail/latest/APIReference/API_LookupEvents.html). You need to look find them in the raw .json.gz files that CloudTrail saves in S3. CloudFront ---------- From 9f9549d952d4d5a8cccc3bc7d2070d7940a623a9 Mon Sep 17 00:00:00 2001 From: Ram Nadella Date: Sat, 19 Nov 2016 22:12:55 -0500 Subject: [PATCH 030/128] Update the note about nginx name resolution in CLB gotchas Mention the way to get around open-source nginx's limitation of resolving backends only on start-up --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2462f8a..0f809b2 100644 --- a/README.md +++ b/README.md @@ -927,7 +927,7 @@ Load Balancers ### Load Balancer Gotchas and Limitations - ❗CLBs and ALBs have **no fixed external IP** that all clients see. For most consumer apps this doesn’t matter, but enterprise customers of yours may want this. IPs will be different for each user, and will vary unpredictably for a single client over time (within the standard [EC2 IP ranges](http://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html)). And similarly, never resolve an CLB name to an IP and put it as the value of an A record — it will work for a while, then break! -- ❗Some web clients or reverse proxies cache DNS lookups for a long time, which is problematic for CLBs and ALBs, since they change their IPs. This means after a few minutes, hours, or days, your client will stop working, unless you disable DNS caching. Watch out for [Java’s settings](http://docs.oracle.com/javase/8/docs/api/java/net/InetAddress.html) and be sure to [adjust them properly](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html). Another example is nginx as a reverse proxy, which [resolves backends only at start-up](https://www.jethrocarr.com/2013/11/02/nginx-reverse-proxies-and-dns-resolution/). +- ❗Some web clients or reverse proxies cache DNS lookups for a long time, which is problematic for CLBs and ALBs, since they change their IPs. This means after a few minutes, hours, or days, your client will stop working, unless you disable DNS caching. Watch out for [Java’s settings](http://docs.oracle.com/javase/8/docs/api/java/net/InetAddress.html) and be sure to [adjust them properly](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html). Another example is nginx as a reverse proxy, which [normally resolves backends only at start-up](https://www.jethrocarr.com/2013/11/02/nginx-reverse-proxies-and-dns-resolution/) (although there is [a way to get around this](https://tenzer.dk/nginx-with-dynamic-upstreams/)). - ❗It’s not unheard of for IPs to be recycled between customers without a long cool-off period. So as a client, if you cache an IP and are not using SSL (to verify the server), you might get not just errors, but responses from completely different services or companies! - 🔸As an operator of a service behind an CLB or ALB, the latter phenomenon means you can also see puzzling or erroneous requests by clients of other companies. This is most common with clients using back-end APIs (since web browsers typically cache for a limited period). - ❗CLBs and ALBs take time to scale up, it does not handle sudden spikes in traffic well. Therefore, if you anticipate a spike, you need to “pre-warm” the load balancer by gradually sending an increasing amount of traffic. From 709c44fdd3939a894d1a34e25a5ac6fcec6827c1 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Sun, 20 Nov 2016 18:43:34 -0800 Subject: [PATCH 031/128] Make it clearer how everyone can help. Few other minor tweaks. --- CONTRIBUTING.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a8c6dcf..2be8041 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,17 +8,17 @@ Contributions of all kinds, including discussion, corrections, additions, and im Please Help ----------- -If you’ve found this guide useful, please join us: +If you’ve found this guide useful, please see if you can help (in increasing levels of commitment and expertise): -- The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) and **ask or answer questions** or **discuss**, which +- **Discussion:** The easiest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) and ask or answer questions. As we discuss, see if it points to new things you or others can contribute to the Guide. helps the community and guides what contributors can focus on. -- [**File issues**](https://github.com/open-guides/og-aws/issues) if it’s clear something needs to be improved and you’re not able to make a pull request. -- [**Pull requests**](https://github.com/open-guides/og-aws/pulls) with changes are always welcome. Please keep them small and focused, so we can add items individually, and review the conventions below. If you want to make a larger change, try to discuss it in Slack. -- **Review** or **comment** on existing issues and pull requests if you have expertise. -- If you have deep expertise, we may ask you to be an **editor** or **expert**. Editors and experts are assigned roles that [help us review](#editorial-process) the Guide. Join Slack to discuss this. +- **Focused pull requests:** [Pull requests](https://github.com/open-guides/og-aws/pulls) with focused changes like typos, specific tips, and corrections are always welcome and fast to review and merge in. Keep them small and focused, and *use multiple PRs for unrelated changes*. (See [writing conventions](#writing-conventions) below.) +- **Track issues:** [File issues](https://github.com/open-guides/og-aws/issues) to aggregate ideas or links if it’s clear something needs to be improved, but it’s not possible to file a PR immediately. +- **Major pull requests:** Take a look at areas [where we need help](https://github.com/open-guides/og-aws/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22). If you want to make a larger change, such as rewriting a lot of content, changing style, or adding a section, discussion in Slack is helpful and usually necessary. For new additions, we often create and edit the first draft in [Quip](https://quip.com/). +- **Deeper expertise:** If you have deep expertise, let one of the project leads know if you’re interested in being an **editor** or an **expert**. Editors and experts are assigned roles that [help us review](#editorial-process) the Guide. -Making Contributions --------------------- +Review Process +-------------- ### Pull Request Etiquette @@ -49,7 +49,7 @@ When creating a PR or reviewing one, it’s helpful to consider a few questions: - Roles: - **Project leads:** Own overall quality of the Guide, direction, and process. - **Editors:** Contributors own specific sections or aspects of the Guide, reviewing PRs and/or writing. Requires expert knowledge. - - **Experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs. + - **Experts:** People with expert knowledge in various areas, who assist editors and have agreed to review or help on demand with tougher questions or PRs. - **Contributors:** Everyone who contributes content or helps one way or another. - All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. - In addition, anyone with relevant knowledge is encouraged to review/comment on PRs. @@ -59,7 +59,7 @@ When creating a PR or reviewing one, it’s helpful to consider a few questions: Writing Conventions ------------------- -When you contribute, keep in mind these conventions: +To keep a polished, consistent style we list a bunch of our conventions. Try to follow these and/or enforce them in reviews: - **Abbreviations:** For AWS service names, we use the abbreviation throughout the guide if it is more common, e.g. EC2 and not Elastic Compute Cloud. We also omit “Amazon” at the front of product names, e.g. EMR and not Amazon EMR. If an abbreviation is convenient but not always used, e.g. AZ instead of Availability Zone, either use the full term once per section/paragraph and abbreviate subsequent usages or do not abbreviate it at all. - Terms that appear for the first time in **boldface** are defined there in a brief summary, with a link if possible to what is probably the best page for that concept. It’s also fine to boldface **key statements** that guide the eye. @@ -80,7 +80,7 @@ When you contribute, keep in mind these conventions: - Not all sections need to follow the above conventions exactly. - Note we try to make sections uniquely titled, so GitHub links to Markdown section anchors don’t collide and are stable. -Note we keep consistent formatting in Markdown via [markdownfmt](https://github.com/shurcooL/markdownfmt). We run **admin/reformat.sh** to do this, but you don’t have to worry about it unless you really want to. +Occasionally, we keep consistent formatting in Markdown via [markdownfmt](https://github.com/shurcooL/markdownfmt). (One of the project leads might run **admin/reformat.sh** to do this, but you can safely ignore that.) Contact ------- From 4179ea515381cbb89ea4471fcf8437349c2510d3 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Sun, 20 Nov 2016 20:05:32 -0800 Subject: [PATCH 032/128] Copy editing on #318 and Route 53. --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 11462a9..cffc9ca 100644 --- a/README.md +++ b/README.md @@ -877,13 +877,13 @@ EFS ### EFS Tips -- With EFS being based on NFSv4.1, any directory on the EFS can be mounted directly, it doesn't have to be the root directory. One application could mount *fs-12345678:/prog1*, another *fs-12345678:/prog2*. +- With EFS being based on NFSv4.1, any directory on the EFS can be mounted directly, it doesn't have to be the root directory. One application could mount *fs-12345678:/prog1*, another *fs-12345678:/prog2*. - [User and group level permissions](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html) can be used to control access to certain directories on the EFS file system. -- ⏱ One EFS file system can be used for multiple applications or services, but it should be considered carefully: +- ⏱ One EFS file system can be used for multiple applications or services, but it should be considered carefully: Pros: - Because performance is based on total size of stored files, having everything on one drive will increase performance for everyone. One application consuming credits faster than it can accumulate might be offset by another application that just stores files on EFS and rarely accesses them. - + Cons: - Since credits are shared, if one application over-consumes them, it will affect the others. - A compromise is made with regards to [security](http://docs.aws.amazon.com/efs/latest/ug/security-considerations.html). All clients will have to have network access to the drive. Someone with root access on one client instance can mount any directory on the EFS and they have read-write access to all files on the drive, even if they don't have access to the applications hosted on other clients. @@ -1248,8 +1248,9 @@ Route 53 - You can use them for CLBs/ALBs or any other resource where AWS supports it. - Somewhat confusingly, you can have CNAME and A aliases, depending on the type of the target. - Because aliases are extensions to regular DNS records, if exported, the output [zone file](https://en.wikipedia.org/wiki/Zone_file) will have additional non-standard “ALIAS” lines in it. -- Take advantage of AWS Route 53 latency based routing. This means that your users around the globe are automatically directed to the nearest AWS region where you are running in terms of having the shortest latency. -- Understand that domain registration and DNS management (hosted zones) are two separate Route 53 services. When you buy/transfer a domain, Route 53 automaticcaly assigns four name servers to it (e.g. ns-2.awsdns-00.com). Route 53 also offers to automatically create a hosted zone for DNS management, but you are not required do do your DNS management in the same account or even in Route 53. You just need to create an NS record pointing to the servers assigned to your domain in Route 53. A use case would be to put your domain registration (very mission critical) in a [bastion account] (https://cloudonaut.io/your-single-aws-account-is-a-serious-risk/) while managing the hosted zones within another account which is accessible by your applications. +- [**Latency-based routing**](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html#routing-policy-latency) allows users around the globe to be automatically directed to the nearest AWS region where you are running, so that latency is reduced. +- Understand that domain registration and DNS management (hosted zones) are two separate Route 53 services. When you buy/transfer a domain, Route 53 automaticaly assigns four name servers to it (e.g. ns-2.awsdns-00.com). Route 53 also offers to automatically create a hosted zone for DNS management, but you are not required do your DNS management in the same account or even in Route 53; you just need to create an NS record pointing to the servers assigned to your domain in Route 53. + - One use case would be to put your domain registration (very mission critical) in a [bastion account](https://cloudonaut.io/your-single-aws-account-is-a-serious-risk/) while managing the hosted zones within another account which is accessible by your applications. CloudFormation -------------- From 4615d2bd92f68097bf28e446e74da02fbe610e7a Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Sun, 20 Nov 2016 20:11:34 -0800 Subject: [PATCH 033/128] Few copy edits following #310. --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index cffc9ca..3dc2e7f 100644 --- a/README.md +++ b/README.md @@ -1194,10 +1194,10 @@ Lambda ### Lambda Code Samples -- Example of using Lambda to 'fan-out' or copy data from one service, in this case Kinesis to multiple other AWS data service as [Fan-out](https://github.com/awslabs/aws-lambda-fanout). Destinations for fan-out data in the sample include IoT, SQS and more. -- AWS Service Limit Monitor [using Lambdas](https://github.com/awslabs/aws-limit-monitor). Example code shows use of multiple lambdas for monitoring. -- Example of using Lambda with ECS in an 'extended worker' pattern via [Lambda ECS Worker Pattern](https://github.com/awslabs/lambda-ecs-worker-pattern). Example code shows use of lambda in a workflow, data from S3 is picked up by the lambda, pushed to a queue, then sent to ECS for more processing. -- Sample Java application which uses Lambda and API Gateway with Cognito (for user identity) via [Secure Pet Store](https://github.com/awslabs/api-gateway-secure-pet-store). Example shows services working together in a complete application. +- [Fan-out](https://github.com/awslabs/aws-lambda-fanout) is an example of using Lambda to “fan-out” or copy data from one service, in this case Kinesis, to multiple other AWS data services. Destinations for fan-out data in the sample include IoT, SQS and more. +- This [AWS limit monitor using Lambdas](https://github.com/awslabs/aws-limit-monitor) shows use of multiple Lambdas for monitoring. +- This [Lambda ECS Worker Pattern](https://github.com/awslabs/lambda-ecs-worker-pattern) shows use of Lambda in a workflow where data from S3 is picked up by the Lambda, pushed to a queue, then sent to ECS for more processing. +- The [Secure Pet Store](https://github.com/awslabs/api-gateway-secure-pet-store) is a sample Java application which uses Lambda and API Gateway with Cognito (for user identity). 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 3d51e3ea89f3789720f4700d0840508ac04805be Mon Sep 17 00:00:00 2001 From: Jason Barry Date: Sun, 20 Nov 2016 23:30:50 -0500 Subject: [PATCH 034/128] CloudWatch Section (#320) * initial cloudwatch section commit * move cloudwatch section after EC2 per jlevy comments * move text section after EC2 section * remove italicized links * fix extra word * link up cloudwatch section in WSTU * adjust language for triggered event actions * add a couple periods. * add statsd, collectd, graphite as options * remove custom metrics disclaimer * add cost link to custom metrics * add basic about instance checks and failures * add elastic stack to monitoring and log alternatives --- README.md | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3dc2e7f..f7e235d 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Table of Contents | [Security and IAM](#security-and-iam) | [📗](#security-and-iam-basics) | [📘](#security-and-iam-tips) | [📙](#security-and-iam-gotchas-and-limitations) | | [S3](#s3) | [📗](#s3-basics) | [📘](#s3-tips) | [📙](#s3-gotchas-and-limitations) | | [EC2](#ec2) | [📗](#ec2-basics) | [📘](#ec2-tips) | [📙](#ec2-gotchas-and-limitations) | +| [CloudWatch](#cloudwatch) | [📗](#cloudwatch-basics) | [📘](#cloudwatch-tips) | [📙](#cloudwatch-gotchas-and-limitations) | | [AMIs](#amis) | [📗](#ami-basics) | [📘](#ami-tips) | [📙](#ami-gotchas-and-limitations) | | [Auto Scaling](#auto-scaling) | [📗](#auto-scaling-basics) | [📘](#auto-scaling-tips) | [📙](#auto-scaling-gotchas-and-limitations) | | [EBS](#ebs) | [📗](#ebs-basics) | [📘](#ebs-tips) | [📙](#ebs-gotchas-and-limitations) | @@ -191,7 +192,7 @@ General Information - [Route 53](#route-53): DNS and domain registration - [VPC](#vpcs-network-security-and-security-groups): Virtual networking, network security, and co-location; you automatically use - [CloudFront](#cloudfront): CDN for hosting content - - [CloudWatch](https://aws.amazon.com/cloudwatch/): Alerts, paging, monitoring + - [CloudWatch](#cloudwatch): Alerts, paging, monitoring - **Managed services:** Existing software solutions you could run on your own, but with managed deployment: - [RDS](#rds): Managed relational databases (managed MySQL, Postgres, and Amazon’s own Aurora database) - [EMR](#emr): Managed Hadoop @@ -776,6 +777,47 @@ EC2 - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. - Instance user-data is (limited to 16 KB)[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data](This limit applies to the data in raw form, not base64-encoded form). If more data is needed, it can be downloaded from S3 by a user-data script. +CloudWatch +------------------- + +### CloudWatch Basics + +* 📒 [Homepage](https://aws.amazon.com/cloudwatch/) ∙ [Documentation](https://aws.amazon.com/documentation/cloudwatch/) ∙ [FAQ](https://aws.amazon.com/cloudwatch/faqs/) ∙ [Pricing](https://aws.amazon.com/cloudwatch/pricing/) +* **CloudWatch** monitors resources and applications, captures logs, and sends events. +* CloudWatch monitoring is the standard mechanism for keeping tabs on AWS resources. A wide range of [**metrics and dimensions**](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CW_Support_For_AWS.html) are available via CloudWatch, allowing you to create time based graphs, **[alarms](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html)**, and **[dashboards](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html)**. + * Alarms are the most practical use of CloudWatch, allowing you to trigger notifications from any given metric. + * Alarms can trigger [SNS notifications](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ConsoleAlarms.html), [Auto Scaling actions](http://docs.aws.amazon.com/autoscaling/latest/userguide/policy_creating.html), or [EC2 actions](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html). + * Publish and share graphs of metrics by creating [customizable dashboard views](https://aws.amazon.com/blogs/aws/cloudwatch-dashboards-create-use-customized-metrics-views/). + * Monitor and report on EC2 [instance system check failure alarms](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#creating_status_check_alarms). +* **Using CloudWatch Events:** + * Events create a mechanism to automate actions in various services on AWS. You can create [event rules](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html) from instance states, AWS APIs, Auto Scaling, Run commands, deployments or time-based schedules (think Cron). + * [Triggered events](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_GettingStarted.html) can can invoke Lambda functions, send SNS/SQS/Kinesis messages, or perform instance actions (terminate, restart, stop, or snapshot volumes). + * Custom payloads can be sent to targets in JSON format, this is especially useful when triggering Lambdas. +* **Using CloudWatch Logs:** + * [CloudWatch Logs](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) is a streaming log storage system. By storing logs within AWS you have access to unlimited paid storage, but you also have the option of streaming logs directly to ElasticSearch or custom Lambdas. + * A [log agent installed](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html) on your servers will process logs over time and send them to CloudWatch Logs. + * You can [export logged data to S3](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html) or stream results to other AWS services. +* **Detailed monitoring:** [Detailed monitoring](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) for EC2 instances must be enabled to get granular metrics, and is [billed under CloudWatch](https://aws.amazon.com/cloudwatch/pricing/). + +### CloudWatch Alternatives and Lock-In + +* CloudWatch offers fairly basic functionality that doesn't create significant (additional) AWS lock-in. Most of the metrics provided by the service can be obtained through APIs that can be imported into other aggregation or visualization tools or services (many specifically provide CloudWatch data import services). +* 🚪 Alternatives to CloudWatch monitoring services include [NewRelic](http://newrelic.com/), [Datadog](http://datadog.com/), [Sumo Logic](http://sumologic.com/), [Zabbix](http://zabbix.com/), [Nagios](http://nagios.org/), [Ruxit](http://ruxit.com/), [Elastic Stack](https://www.elastic.co/v5), open source options such as [StatsD](https://github.com/etsy/statsd) or [collectd](https://collectd.org/) with [Graphite](https://graphiteapp.org/), and many others. +* 🚪 CloudWatch Log alternatives include [Splunk](http://splunk.com/), [Sumo Logic](http://sumologic.com/), [Loggly](http://loggly.com/), [Logstash](https://www.elastic.co/products/logstash), [Papertrail](https://papertrailapp.com/), [Elastic Stack](https://www.elastic.co/v5), and other centralized logging solutions. + +### CloudWatch Tips + +* Some very common use cases for CloudWatch are **[billing alarms](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html)**, **instance** **or [load balancer up/down alarms](http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html)**, and **disk usage alerts**. +* You can use [EC2Config](http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#send_logs_to_cwl) to monitor watch memory and disk metrics on Windows platform instances. +* You can [publish your own metrics](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html) using the AWS API. [Incurs additional cost](https://aws.amazon.com/cloudwatch/pricing/). +* You can stream directly from CloudWatch Logs to a Lambda or ElasticSearch cluster by creating [subscriptions](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) on Log Groups. +* Don't forget to take advantage of the [CloudWatch non-expiring free tier](https://aws.amazon.com/free/#Amazon_CloudWatch). + +### CloudWatch Gotchas and Limitations + +* 🔸You can not use [more than one metric for an alarm](https://forums.aws.amazon.com/thread.jspa?threadID=94984). +* 🔸Notifications you receive from alarms will not have any contextual detail; they have only the specifics of the threshold, alarm state, and timing. + AMIs ---- From a562c5fa074ca689003b9d0c13549c3fa932157a Mon Sep 17 00:00:00 2001 From: Luke de Oliveira Date: Sun, 20 Nov 2016 23:08:39 -0800 Subject: [PATCH 035/128] Fix malformed link in EC2 gotchas --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f7e235d..3f44474 100644 --- a/README.md +++ b/README.md @@ -775,7 +775,7 @@ EC2 - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. - ❗⏱ Instances come in two types: **Fixed Performance Instances** (e.g. M3, C3, and R3) and [**Burstable Performance Instances**](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. -- Instance user-data is (limited to 16 KB)[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data](This limit applies to the data in raw form, not base64-encoded form). If more data is needed, it can be downloaded from S3 by a user-data script. +- Instance user-data is [limited to 16 KB](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data)(This limit applies to the data in raw form, not base64-encoded form). If more data is needed, it can be downloaded from S3 by a user-data script. CloudWatch ------------------- From c1061c84abbe373709b632e44b062d49b6ba3c0e Mon Sep 17 00:00:00 2001 From: Luke de Oliveira Date: Sun, 20 Nov 2016 23:19:16 -0800 Subject: [PATCH 036/128] add info about GPU-focused AMI --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f7e235d..a3a0c8a 100644 --- a/README.md +++ b/README.md @@ -757,7 +757,7 @@ EC2 - Avoid sharing keys and [add individual ssh keys](http://security.stackexchange.com/questions/87480/managing-multiple-ssh-private-keys-for-a-team) for individual users. - **GPU support:** You can rent GPU-enabled instances on EC2 for use in machine learning or graphics rendering workloads. - There are [three generations](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html) of GPU-enabled instances available: - - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. + - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CuDNN, TensorFlow) installed. - Second generation G2 series offers NVIDIA K520 GPUs in 1 or 4 GPU configurations targeting graphics and video encoding. - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. From 72187226694d63865fe168686555587edbe2e77d Mon Sep 17 00:00:00 2001 From: Luke de Oliveira Date: Mon, 21 Nov 2016 16:14:40 -0800 Subject: [PATCH 037/128] convert some spaces => tabs, move bullet point one level up, and add lock-in info --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a3a0c8a..6ddadc4 100644 --- a/README.md +++ b/README.md @@ -756,11 +756,13 @@ EC2 - Avoid reusing the original boot keys except by administrators when creating new instances. - Avoid sharing keys and [add individual ssh keys](http://security.stackexchange.com/questions/87480/managing-multiple-ssh-private-keys-for-a-team) for individual users. - **GPU support:** You can rent GPU-enabled instances on EC2 for use in machine learning or graphics rendering workloads. - - There are [three generations](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html) of GPU-enabled instances available: - - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CuDNN, TensorFlow) installed. + + - There are [three generations](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html) of GPU-enabled instances available: + - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. - Second generation G2 series offers NVIDIA K520 GPUs in 1 or 4 GPU configurations targeting graphics and video encoding. - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. - - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. + - ⛓ AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CUBLAS, CuDNN, TensorFlow) installed to lower the barrier to usage. Note, however, that this leads to lock-in due to Amazon Linux and the fact that you have no direct access to software configuration or versioning. + - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. ### EC2 Gotchas and Limitations From d0bda0048565129ed04efb6d19202966ed335eca Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Mon, 21 Nov 2016 19:40:41 -0800 Subject: [PATCH 038/128] Update authors. --- AUTHORS.md | 29 +++++++++++++++++------------ admin/authors-info.yml | 1 + 2 files changed, 18 insertions(+), 12 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 4c9b747..13dce8d 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -6,24 +6,25 @@ but the content is the result of a community of contributors, editors, and exper Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for notes on roles and editorial process. -* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [10+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) — _editor (Certificate Manager)_ +* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [11+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) — _editor (Certificate Manager)_ * [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) — _expert (IoT)_ -* [Bogdan Luput (bgdnlp)](https://github.com/bgdnlp) — [1+](https://github.com/open-guides/og-aws/commits?author=bgdnlp)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abgdnlp) — _editor (EFS)_ -* [Dan Hermann (danhermann)](https://github.com/danhermann) — [4+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) — _expert (RDS)_ +* [Bogdan Luput (bgdnlp)](https://github.com/bgdnlp) — [3+](https://github.com/open-guides/og-aws/commits?author=bgdnlp)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Abgdnlp) — _editor (EFS)_ +* [Dan Hermann (danhermann)](https://github.com/danhermann) — [4+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[7+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) — _expert (RDS)_ * [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ * [Eric Hammond (ehammond)](https://github.com/ehammond) — _expert (Lambda, serverless)_ * [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [99+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[98+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ -* [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — [1+](https://github.com/open-guides/og-aws/commits?author=kimschmidtsbrain)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Akimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ -* [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — _expert (machine learning, Terraform, orchestration)_ -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [17+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[13+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ +* [Jason Barry (JCBarry)](https://github.com/JCBarry) — [2+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) — _editor (CloudWatch)_ +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [104+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[104+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — [1+](https://github.com/open-guides/og-aws/commits?author=kimschmidtsbrain)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Akimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ +* [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — [1+](https://github.com/open-guides/og-aws/commits?author=lukedeo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alukedeo) — _expert (machine learning, Terraform, orchestration)_ +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [22+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[14+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [Max Zanko (max-zanko)](https://github.com/max-zanko) — [12+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [23+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[26+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [26+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[27+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ * [0xmohit](https://github.com/0xmohit) — [3+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) -* [Abdul Mohammed (abdulirfan3)](https://github.com/abdulirfan3) — [3+](https://github.com/open-guides/og-aws/commits?author=abdulirfan3)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aabdulirfan3) +* [Abdul Mohammed (abdulirfan3)](https://github.com/abdulirfan3) — [5+](https://github.com/open-guides/og-aws/commits?author=abdulirfan3)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aabdulirfan3) * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) * [Bo Bayles (bbayles)](https://github.com/bbayles) — [1+](https://github.com/open-guides/og-aws/commits?author=bbayles)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abbayles) @@ -36,6 +37,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Chris Lennon (chrislennon)](https://github.com/chrislennon) — [1+](https://github.com/open-guides/og-aws/commits?author=chrislennon)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrislennon) * [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) * [Rainer Eli (claushellsing)](https://github.com/claushellsing) — [1+](https://github.com/open-guides/og-aws/commits?author=claushellsing)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aclaushellsing) +* [Dan Bower (danbower)](https://github.com/danbower) — [2+](https://github.com/open-guides/og-aws/commits?author=danbower)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanbower) * [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) * [Matthew Lapworth (dragonndev)](https://github.com/dragonndev) — [5+](https://github.com/open-guides/og-aws/commits?author=dragonndev)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adragonndev) * [Dmitry Guyvoronsky (dreamiurg)](https://github.com/dreamiurg) — [1+](https://github.com/open-guides/og-aws/commits?author=dreamiurg)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adreamiurg) @@ -50,7 +52,6 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Theo Hultberg (iconara)](https://github.com/iconara) — [1+](https://github.com/open-guides/og-aws/commits?author=iconara)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aiconara) * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) * [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) -* [Jason Barry (JCBarry)](https://github.com/JCBarry) — [1+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) * [Jose Luis Ordiales Coscia (jlordiales)](https://github.com/jlordiales) — [1+](https://github.com/open-guides/og-aws/commits?author=jlordiales)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlordiales) * [Jaanus Torp (jnsaff)](https://github.com/jnsaff) — [1+](https://github.com/open-guides/og-aws/commits?author=jnsaff)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajnsaff) * Jurgen Philippaerts @@ -67,17 +68,19 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Mathias Leppich (muhqu)](https://github.com/muhqu) — [3+](https://github.com/open-guides/og-aws/commits?author=muhqu)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Amuhqu) * [nickbarclay](https://github.com/nickbarclay) — [2+](https://github.com/open-guides/og-aws/commits?author=nickbarclay)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anickbarclay) * [Justin Mullis (nonadmin)](https://github.com/nonadmin) — [1+](https://github.com/open-guides/og-aws/commits?author=nonadmin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anonadmin) +* [Noah Zoschke (nzoschke)](https://github.com/nzoschke) — [2+](https://github.com/open-guides/og-aws/commits?author=nzoschke)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anzoschke) * [Ola Wiberg (olawiberg)](https://github.com/olawiberg) * [Pascal Borreli (pborreli)](https://github.com/pborreli) — [1+](https://github.com/open-guides/og-aws/commits?author=pborreli)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apborreli) * [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) * Praveen Patnala * [Raajhesh Kannaa Chidambaram (raajheshkannaa)](https://github.com/raajheshkannaa) — [1+](https://github.com/open-guides/og-aws/commits?author=raajheshkannaa)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Araajheshkannaa) +* [Ram Nadella (ram-nadella)](https://github.com/ram-nadella) — [1+](https://github.com/open-guides/og-aws/commits?author=ram-nadella)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aram-nadella) * [Richard Birkby (rbirkby)](https://github.com/rbirkby) — [2+](https://github.com/open-guides/og-aws/commits?author=rbirkby)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arbirkby) * [Federico Hernandez (recollir)](https://github.com/recollir) — [1+](https://github.com/open-guides/og-aws/commits?author=recollir)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arecollir) -* [Rich Hintz (rjhintz)](https://github.com/rjhintz) — [2+](https://github.com/open-guides/og-aws/commits?author=rjhintz)/[8+](https://github.com/open-guides/og-aws/issues?q=author%3Arjhintz) +* [Rich Hintz (rjhintz)](https://github.com/rjhintz) — [10+](https://github.com/open-guides/og-aws/commits?author=rjhintz)/[12+](https://github.com/open-guides/og-aws/issues?q=author%3Arjhintz) * [Russell Power (rjpower)](https://github.com/rjpower) +* [David Schott (schottsfired)](https://github.com/schottsfired) — [1+](https://github.com/open-guides/og-aws/commits?author=schottsfired)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aschottsfired) * [Ashley Davis (SgtCoDFish)](https://github.com/SgtCoDFish) — [1+](https://github.com/open-guides/og-aws/commits?author=SgtCoDFish)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3ASgtCoDFish) -* [David Schott (shott85)](https://github.com/shott85) — [1+](https://github.com/open-guides/og-aws/commits?author=shott85)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashott85) * [Shubham Aggarwal (shubham391)](https://github.com/shubham391) — [1+](https://github.com/open-guides/og-aws/commits?author=shubham391)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashubham391) * [Adam Nelson (spudstuff)](https://github.com/spudstuff) — [4+](https://github.com/open-guides/og-aws/commits?author=spudstuff)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aspudstuff) * [Steven Maude (StevenMaude)](https://github.com/StevenMaude) — [1+](https://github.com/open-guides/og-aws/commits?author=StevenMaude)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AStevenMaude) @@ -88,10 +91,12 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Tom Schlick (tomschlick)](https://github.com/tomschlick) — [3+](https://github.com/open-guides/og-aws/commits?author=tomschlick)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atomschlick) * [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite) * [Uli Stroetz (ustroetz)](https://github.com/ustroetz) — [1+](https://github.com/open-guides/og-aws/commits?author=ustroetz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Austroetz) +* [Vincent DM (vincent-dm)](https://github.com/vincent-dm) — [1+](https://github.com/open-guides/og-aws/commits?author=vincent-dm)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Avincent-dm) * [Stefan Zier (weirded)](https://github.com/weirded) * [Michael Ortali (xethorn)](https://github.com/xethorn) — [1+](https://github.com/open-guides/og-aws/commits?author=xethorn)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Axethorn) * [Jifeng Zhang (zjfroot)](https://github.com/zjfroot) — [1+](https://github.com/open-guides/og-aws/commits?author=zjfroot)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Azjfroot) +Contributors are listed alphabetically with assigned roles first. Please don't PR the AUTHORS.md or authors-info.yml files! It is auto-generated regularly by the project leads. Please let the project leads know of any errors or omissions on this list. diff --git a/admin/authors-info.yml b/admin/authors-info.yml index 0b3b18b..7ca2bb1 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -25,6 +25,7 @@ roles: richadams: editor (VPC) artemnikitin: editor (Certificate Manager) bgdnlp: editor (EFS) + JCBarry: editor (CloudWatch) donnemartin: expert (tools) merrells: expert (cloud infrastructure, when to use AWS) benkehoe: expert (IoT) From 83b419fcd1cf96be91236814091d9d650304c2b7 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Mon, 21 Nov 2016 19:55:40 -0800 Subject: [PATCH 039/128] Update authors. --- AUTHORS.md | 4 ++-- admin/authors-info.yml | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 13dce8d..bef6de7 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -14,13 +14,14 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Eric Hammond (ehammond)](https://github.com/ehammond) — _expert (Lambda, serverless)_ * [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ * [Jason Barry (JCBarry)](https://github.com/JCBarry) — [2+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) — _editor (CloudWatch)_ -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [104+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[104+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [105+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[105+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ * [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — [1+](https://github.com/open-guides/og-aws/commits?author=kimschmidtsbrain)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Akimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ * [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — [1+](https://github.com/open-guides/og-aws/commits?author=lukedeo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alukedeo) — _expert (machine learning, Terraform, orchestration)_ * [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [22+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[14+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [Max Zanko (max-zanko)](https://github.com/max-zanko) — [12+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ +* [Noah Zoschke (nzoschke)](https://github.com/nzoschke) — [2+](https://github.com/open-guides/og-aws/commits?author=nzoschke)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anzoschke) — _editor (KMS)_ * [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ * [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [26+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[27+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ * [0xmohit](https://github.com/0xmohit) — [3+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) @@ -68,7 +69,6 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Mathias Leppich (muhqu)](https://github.com/muhqu) — [3+](https://github.com/open-guides/og-aws/commits?author=muhqu)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Amuhqu) * [nickbarclay](https://github.com/nickbarclay) — [2+](https://github.com/open-guides/og-aws/commits?author=nickbarclay)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anickbarclay) * [Justin Mullis (nonadmin)](https://github.com/nonadmin) — [1+](https://github.com/open-guides/og-aws/commits?author=nonadmin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anonadmin) -* [Noah Zoschke (nzoschke)](https://github.com/nzoschke) — [2+](https://github.com/open-guides/og-aws/commits?author=nzoschke)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anzoschke) * [Ola Wiberg (olawiberg)](https://github.com/olawiberg) * [Pascal Borreli (pborreli)](https://github.com/pborreli) — [1+](https://github.com/open-guides/og-aws/commits?author=pborreli)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apborreli) * [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) diff --git a/admin/authors-info.yml b/admin/authors-info.yml index 7ca2bb1..1e4d220 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -26,6 +26,7 @@ roles: artemnikitin: editor (Certificate Manager) bgdnlp: editor (EFS) JCBarry: editor (CloudWatch) + nzoschke: editor (KMS) donnemartin: expert (tools) merrells: expert (cloud infrastructure, when to use AWS) benkehoe: expert (IoT) From d28bd54040e5a645b3f409969456d0c8b468d535 Mon Sep 17 00:00:00 2001 From: Bogdan Luput Date: Thu, 24 Nov 2016 04:10:05 +0200 Subject: [PATCH 040/128] Add to CloudWatch tips and gotchas. (#329) * Add to CloudWatch tips and gotchas. * Implement fix from PR --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3f44474..02ea011 100644 --- a/README.md +++ b/README.md @@ -808,15 +808,18 @@ CloudWatch ### CloudWatch Tips * Some very common use cases for CloudWatch are **[billing alarms](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html)**, **instance** **or [load balancer up/down alarms](http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html)**, and **disk usage alerts**. -* You can use [EC2Config](http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#send_logs_to_cwl) to monitor watch memory and disk metrics on Windows platform instances. +* You can use [EC2Config](http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#send_logs_to_cwl) to monitor watch memory and disk metrics on Windows platform instances. For Linux, there are [example scripts](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html) that do the same thing. * You can [publish your own metrics](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html) using the AWS API. [Incurs additional cost](https://aws.amazon.com/cloudwatch/pricing/). * You can stream directly from CloudWatch Logs to a Lambda or ElasticSearch cluster by creating [subscriptions](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) on Log Groups. * Don't forget to take advantage of the [CloudWatch non-expiring free tier](https://aws.amazon.com/free/#Amazon_CloudWatch). ### CloudWatch Gotchas and Limitations +* 🔸Metrics in CloudWatch originate [on the hypervisor](https://forums.aws.amazon.com/message.jspa?messageID=403578). The hypervisor doesn't have access to OS information, so certain metrics (most notably memory utilization) are not available unless pushed to CloudWatch from inside the instance. * 🔸You can not use [more than one metric for an alarm](https://forums.aws.amazon.com/thread.jspa?threadID=94984). * 🔸Notifications you receive from alarms will not have any contextual detail; they have only the specifics of the threshold, alarm state, and timing. +* 🔸Minimum granularity in CloudWatch is 1 minute. That means that multiple values of a metric that are pushed to CloudWatch within the same minute are aggregated into minimum, maximum, average and total (sum) per minute. +* 🔸Data about metrics is kept in CloudWatch [for 15 months](https://aws.amazon.com/blogs/aws/amazon-cloudwatch-update-extended-metrics-retention-user-interface-update/), starting November 2016 (used to be 14 days). Minimum granularity increases after 15 days. AMIs ---- From b68f894f6a76f438f2b01ad1a9fb91391f69ec07 Mon Sep 17 00:00:00 2001 From: Costa Michailidis Date: Wed, 23 Nov 2016 23:39:33 -0500 Subject: [PATCH 041/128] AWS does now (Nov 2016) support environment variables, gotcha removed (#334) --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 02ea011..102d826 100644 --- a/README.md +++ b/README.md @@ -1234,7 +1234,6 @@ Lambda - 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications. - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. -- 🔸Currently, [Lambda does not natively provide for environment variables](https://forums.aws.amazon.com/thread.jspa?messageID=686261), which are a [common way](https://12factor.net/config) of storing configuration, so you must rely on additional mechanisms to achieve this (such as the [Serverless framework](https://serverless.com/framework/docs/guide/environment-variable-handling/#environment-variables-in-serverless) or [apex](http://apex.run/#environment-variables)). - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. ### Lambda Code Samples From 55248e26a113c76803dc98daabb2d6938b1c3da5 Mon Sep 17 00:00:00 2001 From: Rich Hintz Date: Fri, 25 Nov 2016 15:29:31 -0800 Subject: [PATCH 042/128] Add ec2 gotcha - undocumented limit of zero for new accounts (#333) Document an undocumented restriction applicable to very new accounts, such as might be used in instruction. A teacher attempting to have students use their own brand new accounts to each launch one g2 instance reported that the student launches failed because of an undocumented soft limit of zero. This may be associated with AWS trying to mitigate the use of GPU instances for Bitcoin mining, though that's speculation. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 37d6ce6..5f8c703 100644 --- a/README.md +++ b/README.md @@ -778,6 +778,7 @@ EC2 - ❗⏱ Instances come in two types: **Fixed Performance Instances** (e.g. M3, C3, and R3) and [**Burstable Performance Instances**](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. - Instance user-data is [limited to 16 KB](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data)(This limit applies to the data in raw form, not base64-encoded form). If more data is needed, it can be downloaded from S3 by a user-data script. +- Very new accounts may not be able to launch some instance types, such as GPU instances, because of an initially imposed “soft limit” of zero. This limit can be raised by making a support request. See [AWS Service Limits](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the method to make the support request. Note that this limit of zero is [not currently documented](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ec2). CloudWatch ------------------- From def9d117e375240d3540e71b6691b9f523d58bdd Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sat, 26 Nov 2016 18:23:33 -0800 Subject: [PATCH 043/128] added info about IoT Thing Types --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5f8c703..c4680bc 100644 --- a/README.md +++ b/README.md @@ -1583,7 +1583,7 @@ IoT * Send, receive, and process messages: For example, a simple processing board, such as a **Raspberry Pi** ([quick start guide](http://docs.aws.amazon.com/iot/latest/developerguide/iot-device-sdk-c.html)), or an AWS device, such as [Echo or Echo Dot](https://developer.amazon.com/echo), which are designed to work with the [AWS Alexa skills kit](https://developer.amazon.com/alexa-skills-kit) (a programmable voice-enabled service from AWS). - AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. * **IoT terms:** - * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). + * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). Device metadata can also be stored in [**IoT Thing Types**](http://docs.aws.amazon.com/iot/latest/developerguide/thing-types.html). This aids in device metadata management by allowing for reuse of device description and configuration for more than one device. Note that IoT Thing Types can be deprecated, but not changed -- they are immutable. * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). From 05948c4a46c8d6ec49e67caf302aaa47955d9cbe Mon Sep 17 00:00:00 2001 From: AJ Kerrigan Date: Sun, 27 Nov 2016 22:36:26 -0500 Subject: [PATCH 044/128] Add S3 gotcha - incomplete multi-part upload costs Each uploaded part of a multi-part upload begins accruing storage charges immediately, even if the upload as a whole fails. This can really add up for certain usage patterns, and can fly under the radar easily unless you know to look for it. Include quick summary of this gotcha and some links (Amazon and third-party) for more detailed information. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5f8c703..c08e5eb 100644 --- a/README.md +++ b/README.md @@ -699,6 +699,7 @@ S3 - 🔸After uploading, any change that you make to the object causes a full rewrite of the object, so avoid appending-like behavior with regular files. - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). +- 🔸**Incomplete multi-part upload costs:** Incomplete multi-part uploads accrue [storage charges](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpuploadpricing) even if the upload fails and no S3 object is created. [Amazon](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config) ([and](http://www.deplication.net/2016/06/aws-tip-save-s3-costs-with-abort.html) [others](https://www.sumologic.com/aws/s3/s3-cost-optimization/)) recommend using a lifecycle policy to clean up incomplete uploads and save on storage costs. - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 regions now support read-after-write consistency. Amazon S3 also renamed the US Standard region to the US East (N. Virginia) region to be consistent with AWS regional naming conventions. - ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjunction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. - 🔸**S3 authentication versions and regions:** In newer regions, S3 [only supports the latest authentication](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version). If an S3 file operation using CLI or SDK doesn't work in one region, but works correctly in another region, make sure you are using the latest [authentication signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). From a04641f3803ea1d9841acb7ef3971fc4797eee10 Mon Sep 17 00:00:00 2001 From: Russell Davies Date: Sun, 27 Nov 2016 17:29:27 +0000 Subject: [PATCH 045/128] Updated API Gateway tips Added note about support of binary data. --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5f8c703..eea4a66 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -![An Open Guide](figures/signpost-horiz1-1600.jpg) +![An Open Guide](figures/signpost-horiz1-1600.jpg) The Open Guide to Amazon Web Services ===================================== @@ -45,7 +45,7 @@ Table of Contents | [DynamoDB](#dynamodb) | [📗](#dynamodb-basics) | [📘](#dynamodb-tips) | [📙](#dynamodb-gotchas-and-limitations) | | [ECS](#ecs) | [📗](#ecs-basics) | [📘](#ecs-tips) | | | [Lambda](#lambda) | [📗](#lambda-basics) | [📘](#lambda-tips) | [📙](#lambda-gotchas-and-limitations) | -| [API Gateway](#api-gateway) | [📗](#api-gateway-basics) | | [📙](#api-gateway-gotchas-and-limitations) | +| [API Gateway](#api-gateway) | [📗](#api-gateway-basics) | [📘](#api-gateway-tips) | [📙](#api-gateway-gotchas-and-limitations) | | [Route 53](#route-53) | [📗](#route-53-basics) | [📘](#route-53-tips) | | | [CloudFormation](#cloudformation) | [📗](#cloudformation-basics) | [📘](#cloudformation-tips) | [📙](#cloudformation-gotchas-and-limitations) | | [VPCs, Network Security, and Security Groups](#vpcs-network-security-and-security-groups) | [📗](#vpc-basics) | [📘](#vpc-and-network-security-tips) | [📙](#vpc-and-network-security-gotchas-and-limitations) | @@ -1263,6 +1263,10 @@ API Gateway - [Kong](https://getkong.org) is an open-source, on-premises API and microservices gateway built on nginx with Lua. Kong is extensible through “plugins”. - [Tyk](https://tyk.io) is an open-source API gateway implemented in Go and available in the cloud, on-premises or hybrid. +### API Gateway Tips + +- 🔹Prior to 2016-11, you could only send and receive plain text data (so people would base64-encode binary data), but binary data is [now](https://aws.amazon.com/about-aws/whats-new/2016/11/binary-data-now-supported-by-api-gateway/) supported. + ### API Gateway Gotchas and Limitations - 🔸API Gateway only supports encrypted (https) endpoints, and does not support unencrypted HTTP. (This is probably a good thing.) From 17eef7af486e4e7f3a89abc008c91abc32609b5c Mon Sep 17 00:00:00 2001 From: Dorrin Date: Wed, 7 Dec 2016 14:32:38 -0500 Subject: [PATCH 046/128] Added logging details for ECS --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c4680bc..65523ef 100644 --- a/README.md +++ b/README.md @@ -1200,6 +1200,7 @@ ECS - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. - A container's health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. +- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use `awslogs` for cloudwatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips From 3367b9bc736e4091bc36834e9b79da85340d9b6a Mon Sep 17 00:00:00 2001 From: Jesper Eneberg Date: Wed, 7 Dec 2016 23:29:12 +0100 Subject: [PATCH 047/128] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c0b8571..a8ce49c 100644 --- a/README.md +++ b/README.md @@ -1332,7 +1332,7 @@ CloudFront - 🔸If using S3 as a backing store, remember that the endpoints for website hosting and for general S3 are different. Example: “bucketname.s3.amazonaws.com” is a standard S3 serving endpoint, but to have redirect and error page support, you need to use the website hosting endpoint listed for that bucket, e.g. “bucketname.s3-website-us-east-1.amazonaws.com” (or the appropriate region). - 🔸By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. -- 🔸If you want to use a self-obtained SSL certificate with Cloudfront. Make sure it's 2048 bits since Cloudfront only support 2048 bits SSL certificates. [Ongoing discussion](https://forums.aws.amazon.com/thread.jspa?threadID=148783) +- 🔸4096-bit SSL certificates: CloudFront do not support 4096-bit SSL certificates as of late 2016. If you are using an externally issued SSL certificate, you’ll need to make sure it’s 2048 bits. See [ongoing discussion](https://forums.aws.amazon.com/thread.jspa?threadID=148783). DirectConnect ------------- From d5274a09954c90ec4e3027e0f8f16e3da4a687a2 Mon Sep 17 00:00:00 2001 From: Dorrin Date: Wed, 7 Dec 2016 23:15:12 -0500 Subject: [PATCH 048/128] linked to awslogs documentation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 65523ef..eaeda19 100644 --- a/README.md +++ b/README.md @@ -1200,7 +1200,7 @@ ECS - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. - A container's health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. -- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use `awslogs` for cloudwatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. +- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for cloudwatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips From 8c2088128c0cb93c0e698f11c2ec8b332f255db8 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 7 Dec 2016 22:08:41 -0800 Subject: [PATCH 049/128] Minor copy editing on #343. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eaeda19..62e12d3 100644 --- a/README.md +++ b/README.md @@ -1200,10 +1200,10 @@ ECS - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. - A container's health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. -- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for cloudwatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips +- **Log drivers:** ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, ...). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. - [This blog from Convox](https://convox.com/blog/ecs-challenges/) (and [commentary](https://news.ycombinator.com/item?id=11598058)) lists a number of common challenges with ECS as of early 2016. ### ECS Alternatives and Lock-in From b74e6a82ddace21489bbe6532d2bb3f62435b71c Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 7 Dec 2016 22:24:27 -0800 Subject: [PATCH 050/128] Misc copy editing. --- README.md | 50 +++++++++++++++++++++++++------------------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 62e12d3..db40adf 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,7 @@ General Information - In general, “[cloud computing](https://en.wikipedia.org/wiki/Cloud_computing)” can refer to one of three types of cloud: “public,” “private,” and “hybrid.” AWS is a public cloud provider, since anyone can use it. Private clouds are within a single (usually large) organization. Many companies use a hybrid of private and public clouds. - The core features of AWS are [infrastructure-as-a-service](https://en.wikipedia.org/wiki/Cloud_computing#Infrastructure_as_a_service_.28IaaS.29) (IaaS) — that is, virtual machines and supporting infrastructure. Other cloud service models include [platform-as-a-service](https://en.wikipedia.org/wiki/Cloud_computing#Platform_as_a_service_.28PaaS.29) (PaaS), which typically are more fully managed services that deploy customers’ applications, or [software-as-a-service](https://en.wikipedia.org/wiki/Cloud_computing#Software_as_a_service_.28SaaS.29) (SaaS), which are cloud-based applications. AWS does offer a few products that fit into these other models, too. - In business terms, with infrastructure-as-a-service you have a variable cost model — it is [OpEx, not CapEx](http://www.investopedia.com/ask/answers/020915/what-difference-between-capex-and-opex.asp) (though some [pre-purchased contracts](https://aws.amazon.com/ec2/purchasing-options/reserved-instances/) are still CapEx). -- AWS's annual revenue was [$7.88 billion as of 2015 according to their SEC 10-K filing](http://phx.corporate-ir.net/phoenix.zhtml?c=97664&p=irol-sec&control_selectgroup=Annual%20Filings), or roughly 7% of Amazon.com’s total 2015 revenue. +- AWS’s annual revenue was [**$7.88 billion**](http://phx.corporate-ir.net/phoenix.zhtml?c=97664&p=irol-sec&control_selectgroup=Annual%20Filings) as of 2015 according to their SEC 10-K filing, or roughly **7%** of Amazon.com’s total 2015 revenue. - **Main reasons to use AWS:** - If your company is building systems or products that may need to scale - and you have technical know-how @@ -172,7 +172,7 @@ General Information - **Hybrid cloud:** In larger enterprises, it is common to have [hybrid deployments](https://aws.amazon.com/enterprise/hybrid/) encompassing private cloud or on-premises servers and AWS — or other enterprise cloud providers like [IBM](https://www.ibm.com/cloud-computing/solutions/hybrid-cloud)/[Bluemix](http://www.ibm.com/cloud-computing/bluemix/hybrid/), [Microsoft](https://www.microsoft.com/en-us/cloud-platform/hybrid-cloud)/[Azure](https://azure.microsoft.com/en-us/overview/azure-stack/), [NetApp](http://www.netapp.com/us/solutions/cloud/hybrid-cloud/), or [EMC](http://www.emc.com/en-us/cloud/hybrid-cloud-computing/index.htm). - **Major customers:** Who uses AWS and Google Cloud? - AWS’s [list of customers](https://aws.amazon.com/solutions/case-studies/) includes large numbers of mainstream online properties and major brands, such as Netflix, Pinterest, Spotify (moving to Google Cloud), Airbnb, Expedia, Yelp, Zynga, Comcast, Nokia, and Bristol-Myers Squibb. - - Azure's [list of customers](https://azure.microsoft.com/en-us/case-studies/) includes companies such as NBC Universal, 3M and Honeywell Inc. + - Azure’s [list of customers](https://azure.microsoft.com/en-us/case-studies/) includes companies such as NBC Universal, 3M and Honeywell Inc. - Google Cloud’s [list of customers](https://cloud.google.com/customers/) is large as well, and includes a few mainstream sites, such as [Snapchat](http://www.businessinsider.com/snapchat-is-built-on-googles-cloud-2014-1), Best Buy, Domino’s, and Sony Music. ### Which Services to Use @@ -586,7 +586,7 @@ We cover security basics first, since configuring user accounts is something you - [**AWS WAF**](https://aws.amazon.com/waf) is a web application firewall to help you protect your applications from common attack patterns. - **Security auditing:** - [Security Monkey](https://github.com/Netflix/security_monkey) is an open source tool that is designed to assist with security audits. - - [Scout2](https://github.com/nccgroup/Scout2) is an open source tool that uses AWS APIs to assess an environment's security posture. Scout2 is stable and actively maintained. + - [Scout2](https://github.com/nccgroup/Scout2) is an open source tool that uses AWS APIs to assess an environment’s security posture. Scout2 is stable and actively maintained. - 🔹**Export and audit security settings:** You can audit security policies simply by exporting settings using AWS APIs, e.g. using a Boto script like [SecConfig.py](https://gist.github.com/jlevy/cce1b44fc24f94599d0a4b3e613cc15d) (from [this 2013 talk](http://www.slideshare.net/AmazonWebServices/intrusion-detection-in-the-cloud-sec402-aws-reinvent-2013)) and then reviewing and monitoring changes manually or automatically. ### Security and IAM Gotchas and Limitations @@ -605,7 +605,7 @@ S3 ### S3 Basics - 📒 [Homepage](https://aws.amazon.com/s3/) ∙ [Developer guide](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) ∙ [FAQ](https://aws.amazon.com/s3/faqs/) ∙ [Pricing](https://aws.amazon.com/s3/pricing/) -- **S3** (Simple Storage Service) is AWS’ standard cloud storage service, offering file (opaque “blob”) storage of arbitrary numbers of files of almost any size, from 0 to 5 TB. (Prior to [2011](https://aws.amazon.com/releasenotes/Amazon-S3/1917932037969964) the maximum size was 5 GB; larger sizes are now well supported via [multipart support](https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html).) +- **S3** (Simple Storage Service) is AWS’ standard cloud storage service, offering file (opaque “blob”) storage of arbitrary numbers of files of almost any size, from 0 to **5TB**. (Prior to [2011](https://aws.amazon.com/releasenotes/Amazon-S3/1917932037969964) the maximum size was 5 GB; larger sizes are now well supported via [multipart support](https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html).) - Items, or **objects**, are placed into named **buckets** stored with names which are usually called **keys**. The main content is the **value**. - Objects are created, deleted, or updated. Large objects can be streamed, but you cannot access or modify parts of a value; you need to update the whole object. - Every object also has [**metadata**](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html), which includes arbitrary key-value pairs, and is used in a way similar to HTTP headers. Some metadata is system-defined, some are significant when serving HTTP content from buckets or CloudFront, and you can also define arbitrary metadata for your own use. @@ -614,12 +614,11 @@ S3 ### S3 Tips -- For most practical purposes, you can consider S3 capacity unlimited, both in total size of files and number of objects. +- For most practical purposes, you can consider S3 capacity unlimited, both in total size of files and number of objects. The number of objects in a bucket is essentially also unlimited. Customers routinely have millions of objects. - **Bucket naming:** Buckets are chosen from a global namespace (across all regions, even though S3 itself stores data in [whichever S3 region](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) you select), so you’ll find many bucket names are already taken. Creating a bucket means taking ownership of the name until you delete it. Bucket names have [a few restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html) on them. - Bucket names can be used as part of the hostname when accessing the bucket or its contents, like `.s3-us-east-1.amazonaws.com`, as long as the name is [DNS compliant](http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html). - A common practice is to use the company name acronym or abbreviation to prefix (or suffix, if you prefer DNS-style hierarchy) all bucket names (but please, don’t use a check on this as a security measure — this is highly insecure and easily circumvented!). - 🔸Bucket names with '.' (periods) in them [can cause certificate mismatches](https://forums.aws.amazon.com/thread.jspa?threadID=169951) when used with SSL. Use '-' instead, since this then conforms with both SSL expectations and is DNS compliant. -- The number of objects in a bucket is essentially unlimited. Customers routinely have millions of objects. - **Versioning:** S3 has [optional versioning support](https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html), so that all versions of objects are preserved on a bucket. This is mostly useful if you want an archive of changes or the ability to back out mistakes (it has none of the features of full version control systems like Git). - **Durability:** Durability of S3 is extremely high, since internally it keeps several replicas. If you don’t delete it by accident, you can count on S3 not losing your data. (AWS offers the seemingly improbable durability rate of [99.999999999%](https://aws.amazon.com/s3/faqs/#How_durable_is_Amazon_S3), but this is a mathematical calculation based on independent failure rates and levels of replication — not a true probability estimate. Either way, S3 has had [a very good record](https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanently) of durability.) Note this is *much* higher durability than EBS! If durability is less important for your application, you can use [S3 Reduced Redundancy Storage](https://aws.amazon.com/s3/reduced-redundancy/), which lowers the cost per GB, as well as the redundancy. - 💸**S3 pricing** depends on [storage, requests, and transfer](https://aws.amazon.com/s3/pricing/). @@ -761,7 +760,7 @@ EC2 - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. - Second generation G2 series offers NVIDIA K520 GPUs in 1 or 4 GPU configurations targeting graphics and video encoding. - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. - - ⛓ AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CUBLAS, CuDNN, TensorFlow) installed to lower the barrier to usage. Note, however, that this leads to lock-in due to Amazon Linux and the fact that you have no direct access to software configuration or versioning. + - ⛓ AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CUBLAS, CuDNN, TensorFlow) installed to lower the barrier to usage. Note, however, that this leads to lock-in due to Amazon Linux and the fact that you have no direct access to software configuration or versioning. - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. ### EC2 Gotchas and Limitations @@ -777,7 +776,7 @@ EC2 - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. - ❗⏱ Instances come in two types: **Fixed Performance Instances** (e.g. M3, C3, and R3) and [**Burstable Performance Instances**](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. -- Instance user-data is [limited to 16 KB](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data)(This limit applies to the data in raw form, not base64-encoded form). If more data is needed, it can be downloaded from S3 by a user-data script. +- Instance user-data is [limited to 16 KB](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data). (This limit applies to the data in raw form, not base64-encoded form.) If more data is needed, it can be downloaded from S3 by a user-data script. - Very new accounts may not be able to launch some instance types, such as GPU instances, because of an initially imposed “soft limit” of zero. This limit can be raised by making a support request. See [AWS Service Limits](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the method to make the support request. Note that this limit of zero is [not currently documented](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ec2). CloudWatch @@ -914,7 +913,7 @@ EFS - 🐥**EFS** is Amazon’s new (general release 2016) network filesystem. - It is designed to be highly available and durable and each EFS file system object is redundantly stored across multiple availability zones. - EFS is designed to be used as a shared network drive and it can automatically scale up to petabytes of stored data and thousands of instances attached to it. -- It's presented as an [NFSv4.1](https://en.wikipedia.org/wiki/Network_File_System#NFSv4) server, so any compatible NFS client can mount it. +- It’s presented as an [NFSv4.1](https://en.wikipedia.org/wiki/Network_File_System#NFSv4) server, so any compatible NFS client can mount it. - EFS can offer [higher throughput](http://docs.aws.amazon.com/efs/latest/ug/performance.html) (multiple gigabytes per second) and better durability and availability than EBS (see [the comparison table](#storage-durability-availability-and-price)), but with higher latency. - EFS is priced based on the amount of data stored and it costs [much more than EBS](#storage-durability-availability-and-price), about three times as much compared to general purpose gp2 EBS volumes. - ⏱ [Performance](http://docs.aws.amazon.com/efs/latest/ug/performance.html) depends on the amount of data stored on it, which also determines the price: @@ -927,7 +926,7 @@ EFS - With EFS being based on NFSv4.1, any directory on the EFS can be mounted directly, it doesn't have to be the root directory. One application could mount *fs-12345678:/prog1*, another *fs-12345678:/prog2*. - [User and group level permissions](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html) can be used to control access to certain directories on the EFS file system. -- ⏱ One EFS file system can be used for multiple applications or services, but it should be considered carefully: +- ⏱ **Sharing EFS filesystems:** One EFS filesystem can be used for multiple applications or services, but it should be considered carefully: Pros: - Because performance is based on total size of stored files, having everything on one drive will increase performance for everyone. One application consuming credits faster than it can accumulate might be offset by another application that just stores files on EFS and rarely accesses them. @@ -944,7 +943,7 @@ EFS - 🔸 Mounting EFS over VPN connection, VPC peering, or AWS Direct Connect is not supported. - 🔸 Using an EFS volume on Windows is not supported, apparently due to Microsoft implementing NFS differently. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. -- 🔸 ⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. +- 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. - 🔸 Mount points are AZ-based. In an Auto scaling group spread across zones, you can end up with instances in one zone mounting EFS from a different zone. That might decrease performance and would create an unintended single point of failure. One way to fix it would be [a shell script](https://gist.github.com/bgdnlp/9fd326dc4a23f46bab93a1eade023fe4) that runs before network drives are mounted and edits /etc/fstab with the proper AZ. @@ -998,7 +997,7 @@ CLB - In general, CLBs are not as “smart” as some load balancers, and don’t have fancy features or fine-grained control a traditional hardware load balancer would offer. For most common cases involving sessionless apps or cookie-based sessions over HTTP, or SSL termination, they work well. - 🔸By default, CLBs will refuse to route traffic from a load balancer in one Availability Zone (AZ) to a backend instance in another. This [will cause 503s](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/ts-elb-error-message.html#ts-elb-errorcodes-http503) if the last instance in an AZ becomes unavailable, even if there are healthy instances in other zones. If you’re running fewer than two backend instances per AZ, you almost certainly want to [enable cross-zone load balancing](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-disable-crosszone-lb.html#enable-cross-zone). -- Complex rules for directing traffic are not supported. For example, you can’t direct traffic based on a regular expression in the URL, like [HAProxy](http://www.haproxy.org/) offers. +- 🔸Complex rules for directing traffic are not supported. For example, you can’t direct traffic based on a regular expression in the URL, like [HAProxy](http://www.haproxy.org/) offers. - **Apex DNS names:** Once upon a time, you couldn’t assign an CLB to an apex DNS record (i.e. example.com instead of foo.example.com) because it needed to be an A record instead of a CNAME. This is now possible with a Route 53 alias record directly pointing to the load balancer. - 🔸CLBs use [HTTP keep-alives](https://en.wikipedia.org/wiki/HTTP_persistent_connection) on the internal side. This can cause an unexpected side effect: Requests from different clients, each in their own TCP connection on the external side, can end up on the same TCP connection on the internal side. Never assume that multiple requests on the same TCP connection are from the same client! - 🔸 Traffic between CLBs and back-end instances in the same subnet **will** have [Network ACL](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html) rules evaluated (EC2 to EC2 traffic in the same subnet would not have Network ACL rules evaluated). If the default '0.0.0.0/0 ALLOW' rule is removed from the Network ACL applied to the subnet, a rule that allows traffic on both the health check port and any listener port must be added. @@ -1023,7 +1022,7 @@ ALB - 🔸ALBs only support HTTP/2 to external clients and not to internal resources (instances/containers). - ALBs support HTTP routing but not port-based TCP routing. - ALBs do not (yet) support routing based on HTTP “Host” header or HTTP verb. -- Instances in the ALB's target groups have to either have a single, fixed healthcheck port (“EC2 instance”-level healthcheck) or the healthcheck port for a target has to be the same as its application port (“Application instance”-level healthcheck) - you can't configure a per-target healthcheck port that is different than the application port. +- Instances in the ALB’s target groups have to either have a single, fixed healthcheck port (“EC2 instance”-level healthcheck) or the healthcheck port for a target has to be the same as its application port (“Application instance”-level healthcheck) - you can't configure a per-target healthcheck port that is different than the application port. - ALBs are VPC-only (they are not available in EC2 Classic) - In a target group, if there is no healthy target, all requests are routed to all targets. For example, if you point a listener at a target group containing a single service that has a long initialization phase (during which the health checks would fail), requests will reach the service while it is still starting up. @@ -1038,7 +1037,7 @@ Elastic IPs ### Elastic IP Tips - 🔹**Prefer load balancers to elastic IPs:** For single-instance deployments, you could just assign elastic IP to an instance, give that IP a DNS name, and consider that your deployment. Most of the time, you should provision a [load balancer](#load-balancers) instead: - - It’s easy to add and remove instances from load balancers. It's also quicker to add or remove instances from a load balancer than to reassign an elastic IP. + - It’s easy to add and remove instances from load balancers. It’s also quicker to add or remove instances from a load balancer than to reassign an elastic IP. - It’s more convenient to point DNS records to load balancers, instead of pointing them to specific IPs you manage manually. They can also be Route 53 aliases, which are easier to change and manage. - But in some situations, you do need to manage and fix IP addresses of EC2 instances, for example if a customer needs a fixed IP. These situations require elastic IPs. - Elastic IPs are limited to 5 per account. It’s possible to [request more](https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-elastic-ips-ec2-classic). @@ -1089,9 +1088,9 @@ RDS - ⏱RDS instances run on EBS volumes (either general-purpose or provisioned IOPS), and hence are constrained by EBS performance. - 🔸Verify what database features you need, as not everything you might want is available on RDS. For example, if you are using Postgres, check the list of [supported features and extensions](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#SQLServer.Concepts.General.FeatureSupport). If the features you need aren't supported by RDS, you'll have to deploy your database yourself. -- 🔸If you use the failover support offered by RDS, keep in mind that it is based on DNS changes, and make sure that your client reacts to these changes appropriately. This is particularly important for Java, given how its DNS resolver's TTL is [configured by default](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html). +- 🔸If you use the failover support offered by RDS, keep in mind that it is based on DNS changes, and make sure that your client reacts to these changes appropriately. This is particularly important for Java, given how its DNS resolver’s TTL is [configured by default](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html). - 🔸**DB migration to RDS:** While importing your database into RDS ensure you take into consideration the maintenance window settings. If a backup is running at the same time, your import can take a considerably longer time than you would have expected. -- [Database sizes are limited](https://aws.amazon.com/about-aws/whats-new/2015/06/amazon-rds-increases-storage-limits-to-6TB-for-piops-and-gp2/) to 6TB for all database engines except for SQL Server which has a 4TB limit and Aurora which supports up to 64TB databases. +- [Database sizes are limited](https://aws.amazon.com/about-aws/whats-new/2015/06/amazon-rds-increases-storage-limits-to-6TB-for-piops-and-gp2/) to **6TB** for all database engines except for SQL Server which has a **4TB** limit and Aurora which supports up to **64TB** databases. RDS MySQL and MariaDB --------------------- @@ -1117,7 +1116,7 @@ RDS Aurora ### RDS Aurora Basics -- Amazon's proprietary fork of MySQL intended to scale up for high concurrency workloads. Generally speaking, individual query performance under Aurora is not expected to improve significantly relative to MySQL or MariaDB, but Aurora is intended to maintain performance while executing many more queries concurrently than an equivalent MySQL or MariaDB server could handle. +- Amazon’s proprietary fork of MySQL intended to scale up for high concurrency workloads. Generally speaking, individual query performance under Aurora is not expected to improve significantly relative to MySQL or MariaDB, but Aurora is intended to maintain performance while executing many more queries concurrently than an equivalent MySQL or MariaDB server could handle. - [Notable new features](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine) include: - Log-structured storage instead of B-trees to improve write performance - Out-of-process buffer pool so that databases instances can be restarted without clearing the buffer pool @@ -1126,7 +1125,7 @@ RDS Aurora ### RDS Aurora Tips -- In order to take advantage of Aurora's higher concurrency, applications should be configured with large database connection pools and should execute as many queries concurrently as possible. For example, Aurora servers have been tested to produce increasing performance on some OLTP workloads with [up to 5,000 connections](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine/31). +- In order to take advantage of Aurora’s higher concurrency, applications should be configured with large database connection pools and should execute as many queries concurrently as possible. For example, Aurora servers have been tested to produce increasing performance on some OLTP workloads with [up to 5,000 connections](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine/31). - [Aurora scales well with multiple CPUs](https://www.percona.com/blog/2016/05/26/aws-aurora-benchmarking-part-2/) and may require a large instance class for optimal performance. - Because Aurora is based on MySQL 5.6.10, avoiding any MySQL features from 5.7 or later will ease the transition from a MySQL-compatible database into Aurora. - The easiest migration path to Aurora is restoring a database snapshot from MySQL 5.6. The next easiest method is restoring a dump from a MySQL-compatible database such as MariaDB. For [low-downtime migrations](http://cantrill.io/howto/aws/2016/06/06/migrating-from-mysql-to-aurora-with-almost-no-downtime.html) from other MySQL-compatible databases, you can set up an Aurora instance as a replica of your existing database. If none of those methods are options, Amazon offers a fee-based data migration service. @@ -1151,7 +1150,7 @@ RDS SQL Server - 🔸The user is granted only db_owner privileges for each database on the instance. - 🔸Storage cannot be expanded for existing databases. If you need more space, you must restore your database on a new instance with larger storage. -- 🔸4TB database size limit for non-Express editions. +- 🔸There is a **4TB** database size limit for non-Express editions. - 🔸Limited to [30 databases per instance](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html) DynamoDB @@ -1199,7 +1198,7 @@ ECS - Doesn’t support cross-region replication of images. - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. -- A container's health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. +- A container’s health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. ### ECS Tips @@ -1513,7 +1512,7 @@ EMR ### EMR Gotchas and Limitations - 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. -- 💸 Beware of “double-dipping”. With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. +- 💸 Beware of “double-dipping”. With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN’s [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. Kinesis Streams --- @@ -1584,12 +1583,13 @@ IoT * Send, receive, and process messages: For example, a simple processing board, such as a **Raspberry Pi** ([quick start guide](http://docs.aws.amazon.com/iot/latest/developerguide/iot-device-sdk-c.html)), or an AWS device, such as [Echo or Echo Dot](https://developer.amazon.com/echo), which are designed to work with the [AWS Alexa skills kit](https://developer.amazon.com/alexa-skills-kit) (a programmable voice-enabled service from AWS). - AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. * **IoT terms:** - * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). Device metadata can also be stored in [**IoT Thing Types**](http://docs.aws.amazon.com/iot/latest/developerguide/thing-types.html). This aids in device metadata management by allowing for reuse of device description and configuration for more than one device. Note that IoT Thing Types can be deprecated, but not changed -- they are immutable. + * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). Device metadata can also be stored in [**IoT Thing Types**](http://docs.aws.amazon.com/iot/latest/developerguide/thing-types.html). This aids in device metadata management by allowing for reuse of device description and configuration for more than one device. Note that IoT Thing Types can be deprecated, but not changed — they are immutable. * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). * Shown below is a [diagram](https://aws.amazon.com/iot/how-it-works/) which summarizes the flow of messages between the AWS IoT services: - * ![How AWS IoT Works](https://d0.awsstatic.com/IoT/diagrams/awsiot-how-it-works_HowITWorks_1-26.png "How AWS IoT Works") + +![How AWS IoT Works](https://d0.awsstatic.com/IoT/diagrams/awsiot-how-it-works_HowITWorks_1-26.png "How AWS IoT Works") ### IoT Alternatives and Lock-in @@ -1601,7 +1601,7 @@ IoT - **Getting started with Buttons:** One way to start is to use an [**AWS IoT Button**](https://aws.amazon.com/iot/button/). AWS provides a number of code samples for use with their IoT Button, you can use the AWS IoT console, click the “connect AWS IoT button” link and you'll be taken to the AWS Lambda console. There you fill out your button’s serial number to associate it with a Lambda. (As of this writing, AWS IoT buttons are only available for sale in the US.) - **Connections and protocols:** It is important to understand the details of about the devices you wish to connect to the AWS IoT service, including how you will secure the device connections, the device protocols, and more. Cloud vendors differ significantly in their support for common IoT protocols, such as MQTT, AMQP, XMPP. AWS IoT supports **secure MQTT**, **WebSockets** and **HTTPS**. - Support for **device security** via certificate processing is a key differentiator in this space. In August 2016, AWS added [just-in-time registrations](https://aws.amazon.com/blogs/iot/just-in-time-registration-of-device-certificates-on-aws-iot/) for IoT devices to their services. -- **Combining with other services:** It's common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). +- **Combining with other services:** It’s common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). - **Testing tools:** * To get started, AWS includes a lightweight MQTT client in the AWS IoT console. Here you can create and test sending and receiving messages to and from various MQTT topics. * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators @@ -1792,7 +1792,7 @@ This section covers a few unusually useful or “must know about” resources or - [Resilience and Reliability on AWS](http://shop.oreilly.com/product/0636920026839.do) - [AWS documentation as Kindle ebooks](https://www.amazon.com/Amazon-Web-Services/e/B007R6MVQ6) - General references - - [AWS Well Architected Framework Guide](https://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf): Amazon's own 56 page guide to operational excellence - guidelines and checklists to validate baseline security, reliability, performance (including high availability) and cost optimization practices. + - [AWS Well Architected Framework Guide](https://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf): Amazon’s own 56 page guide to operational excellence - guidelines and checklists to validate baseline security, reliability, performance (including high availability) and cost optimization practices. - [Awesome Microservices](https://github.com/mfornos/awesome-microservices): A curated list of tools and technologies for microservice architectures. Worth browsing to learn about popular open source projects. - [Is it fast yet?](https://istlsfastyet.com/): Ilya Grigorik’s TLS performance overview - [High Performance Browser Networking](https://hpbn.co/): A full, modern book on web network performance; a presentation on the HTTP/2 portion is [here](https://docs.google.com/presentation/d/1r7QXGYOLCh4fcUq0jDdDwKJWNqWK1o4xMtYpKZCJYjM/edit?usp=sharing). From 3a3a159304fd1348242780c7dcaef4d9a12ad778 Mon Sep 17 00:00:00 2001 From: Dorrin Date: Thu, 8 Dec 2016 14:37:05 -0500 Subject: [PATCH 051/128] added reference for ECS basics --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eaeda19..0336582 100644 --- a/README.md +++ b/README.md @@ -1200,7 +1200,7 @@ ECS - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. - A container's health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. -- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for cloudwatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. +- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for cloudwatch (make sure a group is made for the logs first). [Drivers such as fluentd are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips From 40df49e34505ec3027de64fad98e1ebe3d5c71d0 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 18:15:56 -0800 Subject: [PATCH 052/128] Add IPv6 in VPC details. --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 668d19d..cd9a8a0 100644 --- a/README.md +++ b/README.md @@ -1363,6 +1363,11 @@ VPCs, Network Security, and Security Groups - **VPC Flow Logs** allow you to monitor the network traffic to, from, and within your VPC. Logs are stored in CloudWatch Logs groups, and can be used for security monitoring (with third party tools), performance evaluation, and forensic investigation. - See the [VPC Flow Logs User Guide](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) for basic information. - See the [flowlogs-reader](https://github.com/obsrvbl/flowlogs-reader) CLI tool and Python library to retrieve and work with VPC Flow Logs. +- **IPv6** [is available in VPC.](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/) Along with this announcement came the introduction of the Egress-Only Internet Gateway. In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. +- Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already. +- New and existing VPCs can both use IPv6. Existing VPCs will need to be configured to have an IPv6 CIDR block associated with them, just as new VPCs do. +- ❗Security Groups and Route Tables apply entries separately for IPv4 and IPv6, so one must ensure they add entries for both protocols accordingly. +- ❗At launch, IPv6 for VPC is only available in the Ohio (us-east-2) region, though the launch announcement makes mention that other regions will see the same in the future. ### VPC and Network Security Gotchas and Limitations From 206a38a08aaf4ae8ef4da2e4076a6ee059cdc512 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 21:15:43 -0800 Subject: [PATCH 053/128] Move limitation-related lines for VPC IPv6 feature to limitations section. --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index cd9a8a0..bda4c93 100644 --- a/README.md +++ b/README.md @@ -1364,10 +1364,8 @@ VPCs, Network Security, and Security Groups - See the [VPC Flow Logs User Guide](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) for basic information. - See the [flowlogs-reader](https://github.com/obsrvbl/flowlogs-reader) CLI tool and Python library to retrieve and work with VPC Flow Logs. - **IPv6** [is available in VPC.](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/) Along with this announcement came the introduction of the Egress-Only Internet Gateway. In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. -- Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already. -- New and existing VPCs can both use IPv6. Existing VPCs will need to be configured to have an IPv6 CIDR block associated with them, just as new VPCs do. -- ❗Security Groups and Route Tables apply entries separately for IPv4 and IPv6, so one must ensure they add entries for both protocols accordingly. -- ❗At launch, IPv6 for VPC is only available in the Ohio (us-east-2) region, though the launch announcement makes mention that other regions will see the same in the future. +- Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already. +- New and existing VPCs can both use IPv6. Existing VPCs will need to be configured to have an IPv6 CIDR block associated with them, just as new VPCs do. ### VPC and Network Security Gotchas and Limitations @@ -1377,6 +1375,8 @@ VPCs, Network Security, and Security Groups - ❗New RDS instances require a [subnet group](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Subnets) within your VPC. If you’re using the [default VPC](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html) this isn’t a concern, it will contain a subnet for each availability zone in your region. However, if you’re creating your own VPC and plan on using RDS, make sure you have at least two subnets within the VPC to act as the subnet group. - ❗If you delete the default VPC, the only way to create another VPC marked as “default” is to contact AWS technical support. See this [note](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html#deleting-default-vpc) in the documentation. - ❗Be careful with VPC VPN credentials! If lost or compromised, the VPN endpoint must be deleted and recreated. See the instructions for [Replacing Compromised Credentials](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html#CompromisedCredentials). +- ❗Security Groups and Route Tables apply entries separately for IPv4 and IPv6, so one must ensure they add entries for both protocols accordingly. +- ❗At launch, IPv6 for VPC is only available in the Ohio (us-east-2) region, though the launch announcement makes mention that other regions will see the same in the future. KMS --- From 46e7038be44e10efaa51519816f20a4f88bc23ee Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 21:23:56 -0800 Subject: [PATCH 054/128] Add line that EC2 instances can use IPv6 now. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index bda4c93..ae7788f 100644 --- a/README.md +++ b/README.md @@ -763,7 +763,8 @@ EC2 - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. - ⛓ AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CUBLAS, CuDNN, TensorFlow) installed to lower the barrier to usage. Note, however, that this leads to lock-in due to Amazon Linux and the fact that you have no direct access to software configuration or versioning. - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. - +- EC2 instances can take advantage of IPv6 addressing, so long as they are launched in a subnet with an allocated CIDR range in an IPv6-enabled VPC. + ### EC2 Gotchas and Limitations - ❗Never use ssh passwords. Just don’t do it; they are too insecure, and consequences of compromise too severe. Use keys instead. [Read up on this](https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2) and fully disable ssh password access to your ssh server by making sure 'PasswordAuthentication no' is in your /etc/ssh/sshd_config file. If you’re careful about managing ssh private keys everywhere they are stored, it is a major improvement on security over password-based authentication. From b5d4d46c54308f439a24df19025fd501b6551b27 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 21:28:57 -0800 Subject: [PATCH 055/128] Add line about CLB IPv6 support. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ae7788f..fd82073 100644 --- a/README.md +++ b/README.md @@ -763,7 +763,7 @@ EC2 - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. - ⛓ AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CUBLAS, CuDNN, TensorFlow) installed to lower the barrier to usage. Note, however, that this leads to lock-in due to Amazon Linux and the fact that you have no direct access to software configuration or versioning. - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. -- EC2 instances can take advantage of IPv6 addressing, so long as they are launched in a subnet with an allocated CIDR range in an IPv6-enabled VPC. +- All current EC2 instance types can take advantage of IPv6 addressing, so long as they are launched in a subnet with an allocated CIDR range in an IPv6-enabled VPC. ### EC2 Gotchas and Limitations @@ -1003,6 +1003,7 @@ CLB - **Apex DNS names:** Once upon a time, you couldn’t assign an CLB to an apex DNS record (i.e. example.com instead of foo.example.com) because it needed to be an A record instead of a CNAME. This is now possible with a Route 53 alias record directly pointing to the load balancer. - 🔸CLBs use [HTTP keep-alives](https://en.wikipedia.org/wiki/HTTP_persistent_connection) on the internal side. This can cause an unexpected side effect: Requests from different clients, each in their own TCP connection on the external side, can end up on the same TCP connection on the internal side. Never assume that multiple requests on the same TCP connection are from the same client! - 🔸 Traffic between CLBs and back-end instances in the same subnet **will** have [Network ACL](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html) rules evaluated (EC2 to EC2 traffic in the same subnet would not have Network ACL rules evaluated). If the default '0.0.0.0/0 ALLOW' rule is removed from the Network ACL applied to the subnet, a rule that allows traffic on both the health check port and any listener port must be added. +- As of December 2016, CLBs launched in VPCs do not support IPv6 addressing. CLBs launched in EC2-Classic support both IPv4 and IPv6 [with the "dualstack" DNS name.](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internet-facing-load-balancers.html#internet-facing-ip-addresses) ALB --- From 68877c82ff731ec4c78148e55f954d99f702a809 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 21:41:59 -0800 Subject: [PATCH 056/128] Add ALB IPv6 announced support line. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index fd82073..afa3b23 100644 --- a/README.md +++ b/README.md @@ -1028,6 +1028,7 @@ ALB - Instances in the ALB’s target groups have to either have a single, fixed healthcheck port (“EC2 instance”-level healthcheck) or the healthcheck port for a target has to be the same as its application port (“Application instance”-level healthcheck) - you can't configure a per-target healthcheck port that is different than the application port. - ALBs are VPC-only (they are not available in EC2 Classic) - In a target group, if there is no healthy target, all requests are routed to all targets. For example, if you point a listener at a target group containing a single service that has a long initialization phase (during which the health checks would fail), requests will reach the service while it is still starting up. +- Although it has been announced as coming soon, [ALBs do not presently support IPv6.](https://aws.amazon.com/about-aws/whats-new/2016/12/announcing-internet-protocol-version-6-support-for-ec2-instances-in-amazon-virtual-private-cloud/) Even if the ALB is launched in an IPv6-enabled VPC and subnet, it will only have IPv4 addressing. Elastic IPs ----------- From d12d0b5a95e9180717ca750ebc40d548f81f1067 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 22:55:10 -0800 Subject: [PATCH 057/128] Expand details on IPv6 with CloudFront and WAF. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index afa3b23..17f7389 100644 --- a/README.md +++ b/README.md @@ -1421,7 +1421,7 @@ CloudFront ### CloudFront Tips -- 🐥**IPv6** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/)! +- 🐥**IPv6** is [supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). This is a configurable setting, and is enabled by default on new CloudFront distribution. - 🐥**HTTP/2** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/09/amazon-cloudfront-now-supports-http2/)! Clients [must support TLS 1.2 and SNI](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesSupportedHTTPVersions). - While the most common use is for users to browse and download content (GET or HEAD methods) requests, CloudFront also supports ([since 2013](https://aws.amazon.com/blogs/aws/amazon-cloudfront-content-uploads-post-put-other-methods/)) uploaded data (POST, PUT, DELETE, OPTIONS, and PATCH). - You must enable this by specifying the [allowed HTTP methods](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesAllowedHTTPMethods) when you create the distribution. @@ -1437,6 +1437,7 @@ CloudFront - 🔸If using S3 as a backing store, remember that the endpoints for website hosting and for general S3 are different. Example: “bucketname.s3.amazonaws.com” is a standard S3 serving endpoint, but to have redirect and error page support, you need to use the website hosting endpoint listed for that bucket, e.g. “bucketname.s3-website-us-east-1.amazonaws.com” (or the appropriate region). - 🔸By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. - 🔸4096-bit SSL certificates: CloudFront do not support 4096-bit SSL certificates as of late 2016. If you are using an externally issued SSL certificate, you’ll need to make sure it’s 2048 bits. See [ongoing discussion](https://forums.aws.amazon.com/thread.jspa?threadID=148783). +- Although connections from clients to CloudFront edge servers can make use of IPv6, [connections to the origin server will continue to use IPv4.](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/) DirectConnect ------------- From 0822e53c9f0cde549c406c4f8377fe1c765c74bc Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 22:57:13 -0800 Subject: [PATCH 058/128] Clarify CloudFront and WAF IPv6 support verbiage. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 17f7389..673f851 100644 --- a/README.md +++ b/README.md @@ -1421,7 +1421,7 @@ CloudFront ### CloudFront Tips -- 🐥**IPv6** is [supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). This is a configurable setting, and is enabled by default on new CloudFront distribution. +- 🐥**IPv6** is [supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). This is a configurable setting, and is enabled by default on new CloudFront distribution. IPv6 support extends to the use of WAF with CloudFront. - 🐥**HTTP/2** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/09/amazon-cloudfront-now-supports-http2/)! Clients [must support TLS 1.2 and SNI](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesSupportedHTTPVersions). - While the most common use is for users to browse and download content (GET or HEAD methods) requests, CloudFront also supports ([since 2013](https://aws.amazon.com/blogs/aws/amazon-cloudfront-content-uploads-post-put-other-methods/)) uploaded data (POST, PUT, DELETE, OPTIONS, and PATCH). - You must enable this by specifying the [allowed HTTP methods](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesAllowedHTTPMethods) when you create the distribution. From b2da2cd4092a37798de3e93ff411945b8b2b61c3 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 23:28:21 -0800 Subject: [PATCH 059/128] Introduce WAF section with IPv6 information. This commit completes Issue #160. --- README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/README.md b/README.md index 673f851..37bafca 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,7 @@ Table of Contents | [IoT](#iot) | [📗](#iot-basics) | [📘](#iot-tips) | [📙](#iot-gotchas-and-limitations) | | [SES](#ses) | [📗](#ses-basics) | [📘](#ses-tips) | [📙](#ses-gotchas-and-limitations) | | [Certificate Manager](#certificate-manager) | [📗](#certificate-manager-basics) | [📘](#certificate-manager-tips) | [📙](#certificate-manager-gotchas-and-limitations) | +| [WAF](#waf) | [📗](#waf-basics) | [📘](#waf-tips) | [📙](#waf-gotchas-and-limitations) | **Special Topics** @@ -1675,6 +1676,25 @@ Certificate Manager - 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from us-east-1 (N. Virginia) region. Certificates from other regions can [only be used with Elastic Load Balancers](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). - 🔸**IoT** has its [own way](http://docs.aws.amazon.com/iot/latest/developerguide/create-device-certificate.html) of setting up certificates. +Web Application Firewall (WAF) +------------------- + +### WAF Basics + +- 📒 [Homepage](https://aws.amazon.com/waf/) ∙ [Documentation](https://aws.amazon.com/documentation/waf/) ∙ [FAQ](https://aws.amazon.com/waf/faq/) ∙ [Pricing](https://aws.amazon.com/waf/pricing) +- WAF is used in conjunction with the CloudFront and ALB services to inspect and block/allow web requests based on user-configurable conditions. +- HTTPS and HTTP requests are supported with this service. +- WAF's strength is in detecting malicious activity based on pattern-matching inputs for attacks such as SQL injections, XSS, etc. +- WAF supports inspection of requests [received through both IPv6 and IPv4.](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/) + +### WAF Tips + +- Getting a WAF API call history can be done through CloudTrail. This is enabled through the CloudTrail console. + +### WAF Gotchas and Limitations + +- As of December 2016, WAF is available in the US East (Northern Virginia), US West (Oregon), Asia Pacific (Tokyo) and EU (Ireland) regions. + High Availability ----------------- From ccb4248eaf8b5576e2b6fad70f558c671e0d041c Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Thu, 8 Dec 2016 23:29:43 -0800 Subject: [PATCH 060/128] Update WAF section title. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 37bafca..e50d0a4 100644 --- a/README.md +++ b/README.md @@ -1676,13 +1676,13 @@ Certificate Manager - 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from us-east-1 (N. Virginia) region. Certificates from other regions can [only be used with Elastic Load Balancers](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). - 🔸**IoT** has its [own way](http://docs.aws.amazon.com/iot/latest/developerguide/create-device-certificate.html) of setting up certificates. -Web Application Firewall (WAF) +WAF ------------------- ### WAF Basics - 📒 [Homepage](https://aws.amazon.com/waf/) ∙ [Documentation](https://aws.amazon.com/documentation/waf/) ∙ [FAQ](https://aws.amazon.com/waf/faq/) ∙ [Pricing](https://aws.amazon.com/waf/pricing) -- WAF is used in conjunction with the CloudFront and ALB services to inspect and block/allow web requests based on user-configurable conditions. +- WAF (Web Application Firewall) is used in conjunction with the CloudFront and ALB services to inspect and block/allow web requests based on user-configurable conditions. - HTTPS and HTTP requests are supported with this service. - WAF's strength is in detecting malicious activity based on pattern-matching inputs for attacks such as SQL injections, XSS, etc. - WAF supports inspection of requests [received through both IPv6 and IPv4.](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/) From ba61e2a3912c52f787f397bec95ef8aa0b2bd3ef Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Fri, 9 Dec 2016 00:15:17 -0800 Subject: [PATCH 061/128] Added line about S3 Transfer Acceleration supporting IPv6 for clarity. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e50d0a4..da44bb0 100644 --- a/README.md +++ b/README.md @@ -688,7 +688,7 @@ S3 - [ObjectiveFS](https://objectivefs.com/) ([discussion](https://news.ycombinator.com/item?id=10117506)) is a commercial solution that supports filesystem features and concurrent clients. - If you are primarily using a VPC, consider setting up a [VPC Endpoint](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html) for S3 in order to allow your VPC-hosted resources to easily access it without the need for extra network configuration or hops. - **Cross-region replication:** S3 has [a feature](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) for replicating a bucket between one region and another. Note that S3 is already highly replicated within one region, so usually this isn’t necessary for durability, but it could be useful for compliance (geographically distributed data storage), lower latency, or as a strategy to reduce region-to-region bandwidth costs by mirroring heavily used data in a second region. -- **IPv4 vs IPv6:** For a long time S3 only supported IPv4 at the default endpoint `https://BUCKET.s3.amazonaws.com`. However, [as of Aug 11, 2016](https://aws.amazon.com/blogs/aws/now-available-ipv6-support-for-amazon-s3/) it now supports both IPv4 & IPv6! To use both, you have to [enable dualstack](http://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html) either in your preferred API client or by directly using this url scheme `https://BUCKET.s3.dualstack.REGION.amazonaws.com`. +- **IPv4 vs IPv6:** For a long time S3 only supported IPv4 at the default endpoint `https://BUCKET.s3.amazonaws.com`. However, [as of Aug 11, 2016](https://aws.amazon.com/blogs/aws/now-available-ipv6-support-for-amazon-s3/) it now supports both IPv4 & IPv6! To use both, you have to [enable dualstack](http://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html) either in your preferred API client or by directly using this url scheme `https://BUCKET.s3.dualstack.REGION.amazonaws.com`. This extends to S3 Transfer Acceleration as well. - **S3 event notifications:** S3 can be configured to send an [SNS notification](https://aws.amazon.com/blogs/aws/introducing-the-amazon-simple-notification-service/), [SQS message](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/Welcome.html), or [AWS Lambda function](http://docs.aws.amazon.com/lambda/latest/dg/welcome.html) on [bucket events](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html). ### S3 Gotchas and Limitations From b83d9e1771bb25641006e5c1cf7892d751f37248 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Fri, 9 Dec 2016 15:30:03 -0800 Subject: [PATCH 062/128] Update authors. --- AUTHORS.md | 19 ++++++++++++------- admin/authors-info.yml | 1 + 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index bef6de7..10d113f 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -8,38 +8,42 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [11+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) — _editor (Certificate Manager)_ * [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) — _expert (IoT)_ -* [Bogdan Luput (bgdnlp)](https://github.com/bgdnlp) — [3+](https://github.com/open-guides/og-aws/commits?author=bgdnlp)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Abgdnlp) — _editor (EFS)_ +* [Bogdan Luput (bgdnlp)](https://github.com/bgdnlp) — [4+](https://github.com/open-guides/og-aws/commits?author=bgdnlp)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Abgdnlp) — _editor (EFS)_ * [Dan Hermann (danhermann)](https://github.com/danhermann) — [4+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[7+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) — _expert (RDS)_ * [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ * [Eric Hammond (ehammond)](https://github.com/ehammond) — _expert (Lambda, serverless)_ * [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ * [Jason Barry (JCBarry)](https://github.com/JCBarry) — [2+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) — _editor (CloudWatch)_ -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [105+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[105+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [108+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[111+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ * [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — [1+](https://github.com/open-guides/og-aws/commits?author=kimschmidtsbrain)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Akimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ -* [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — [1+](https://github.com/open-guides/og-aws/commits?author=lukedeo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alukedeo) — _expert (machine learning, Terraform, orchestration)_ -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [22+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[14+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ -* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [12+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ +* [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — [3+](https://github.com/open-guides/og-aws/commits?author=lukedeo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alukedeo) — _expert (machine learning, Terraform, orchestration)_ +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [23+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[15+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ +* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [12+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[10+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Noah Zoschke (nzoschke)](https://github.com/nzoschke) — [2+](https://github.com/open-guides/og-aws/commits?author=nzoschke)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anzoschke) — _editor (KMS)_ * [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ * [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [26+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[27+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Min (wumintendy)](https://github.com/wumintendy) — _expert (ECS, CloudWatch, SQS, RDS, WAF)_ * [0xmohit](https://github.com/0xmohit) — [3+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) * [Abdul Mohammed (abdulirfan3)](https://github.com/abdulirfan3) — [5+](https://github.com/open-guides/og-aws/commits?author=abdulirfan3)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aabdulirfan3) +* [AJ Kerrigan (ajkerrigan-mdsol)](https://github.com/ajkerrigan-mdsol) — [1+](https://github.com/open-guides/og-aws/commits?author=ajkerrigan-mdsol)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aajkerrigan-mdsol) * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) * [Bo Bayles (bbayles)](https://github.com/bbayles) — [1+](https://github.com/open-guides/og-aws/commits?author=bbayles)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abbayles) * [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk) * [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer) +* [Jesper Eneberg (Bizzelicious)](https://github.com/Bizzelicious) — [3+](https://github.com/open-guides/og-aws/commits?author=Bizzelicious)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3ABizzelicious) * [Bradly Feeley (bradly)](https://github.com/bradly) — [2+](https://github.com/open-guides/og-aws/commits?author=bradly)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Abradly) * [Brady Dowling (bradydowling)](https://github.com/bradydowling) — [1+](https://github.com/open-guides/og-aws/commits?author=bradydowling)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abradydowling) * [Mårten Gustafson (chids)](https://github.com/chids) — [4+](https://github.com/open-guides/og-aws/commits?author=chids)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Achids) -* [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) +* [Chris Griffin (chris-griffin)](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) * [Chris Lennon (chrislennon)](https://github.com/chrislennon) — [1+](https://github.com/open-guides/og-aws/commits?author=chrislennon)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrislennon) * [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) * [Rainer Eli (claushellsing)](https://github.com/claushellsing) — [1+](https://github.com/open-guides/og-aws/commits?author=claushellsing)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aclaushellsing) * [Dan Bower (danbower)](https://github.com/danbower) — [2+](https://github.com/open-guides/og-aws/commits?author=danbower)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanbower) * [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) +* [Dorrin (DorrinPk)](https://github.com/DorrinPk) — [2+](https://github.com/open-guides/og-aws/commits?author=DorrinPk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3ADorrinPk) * [Matthew Lapworth (dragonndev)](https://github.com/dragonndev) — [5+](https://github.com/open-guides/og-aws/commits?author=dragonndev)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adragonndev) * [Dmitry Guyvoronsky (dreamiurg)](https://github.com/dreamiurg) — [1+](https://github.com/open-guides/og-aws/commits?author=dreamiurg)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adreamiurg) * [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy) @@ -77,7 +81,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Ram Nadella (ram-nadella)](https://github.com/ram-nadella) — [1+](https://github.com/open-guides/og-aws/commits?author=ram-nadella)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aram-nadella) * [Richard Birkby (rbirkby)](https://github.com/rbirkby) — [2+](https://github.com/open-guides/og-aws/commits?author=rbirkby)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arbirkby) * [Federico Hernandez (recollir)](https://github.com/recollir) — [1+](https://github.com/open-guides/og-aws/commits?author=recollir)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arecollir) -* [Rich Hintz (rjhintz)](https://github.com/rjhintz) — [10+](https://github.com/open-guides/og-aws/commits?author=rjhintz)/[12+](https://github.com/open-guides/og-aws/issues?q=author%3Arjhintz) +* [Rich Hintz (rjhintz)](https://github.com/rjhintz) — [11+](https://github.com/open-guides/og-aws/commits?author=rjhintz)/[13+](https://github.com/open-guides/og-aws/issues?q=author%3Arjhintz) * [Russell Power (rjpower)](https://github.com/rjpower) * [David Schott (schottsfired)](https://github.com/schottsfired) — [1+](https://github.com/open-guides/og-aws/commits?author=schottsfired)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aschottsfired) * [Ashley Davis (SgtCoDFish)](https://github.com/SgtCoDFish) — [1+](https://github.com/open-guides/og-aws/commits?author=SgtCoDFish)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3ASgtCoDFish) @@ -92,6 +96,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite) * [Uli Stroetz (ustroetz)](https://github.com/ustroetz) — [1+](https://github.com/open-guides/og-aws/commits?author=ustroetz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Austroetz) * [Vincent DM (vincent-dm)](https://github.com/vincent-dm) — [1+](https://github.com/open-guides/og-aws/commits?author=vincent-dm)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Avincent-dm) +* [Costa Michailidis (wamoyo)](https://github.com/wamoyo) — [1+](https://github.com/open-guides/og-aws/commits?author=wamoyo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Awamoyo) * [Stefan Zier (weirded)](https://github.com/weirded) * [Michael Ortali (xethorn)](https://github.com/xethorn) — [1+](https://github.com/open-guides/og-aws/commits?author=xethorn)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Axethorn) * [Jifeng Zhang (zjfroot)](https://github.com/zjfroot) — [1+](https://github.com/open-guides/og-aws/commits?author=zjfroot)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Azjfroot) diff --git a/admin/authors-info.yml b/admin/authors-info.yml index 1e4d220..f813912 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -33,6 +33,7 @@ roles: ehammond: expert (Lambda, serverless) danhermann: expert (RDS) kimschmidtsbrain: expert (AWS Marketplace, Aurora) + wumintendy: expert (ECS, CloudWatch, SQS, RDS, WAF) lukedeo: expert (machine learning, Terraform, orchestration) marcello3d: golyshev: From 513d7d597d8352ad57376525ebfb63ae690a4bdd Mon Sep 17 00:00:00 2001 From: Aron Woost Date: Sat, 10 Dec 2016 18:24:45 +0100 Subject: [PATCH 063/128] Fix typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 668d19d..7055be6 100644 --- a/README.md +++ b/README.md @@ -1178,7 +1178,7 @@ DynamoDB ### DynamoDB Gotchas and Limitations -- 🔸 DynamoDB doesn’t provide an easy way to bulk-load data (it is possible through [Data Pipeline](http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html), and this has some [unfortunate consequences](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.AvoidExcessivePTIncreases). Since you need to use the regular service APIs to update existing or create new rows, it is common to temporarily turn up a destination table’s write throughput to speed import. But when the table’s write capacity is increased, DynamoDB may do an irreversible split of the partitions underlying the table, spreading the total table capacity evenly across the new generation of tables. Later, if the capacity is reduced, the capacity for each partition is also reduced but the total number of partitions is not, leaving less capacity for each partition. This leaves the table in a state where it much easier for hotspots to overwhelm individual partitions. +- 🔸 DynamoDB doesn’t provide an easy way to bulk-load data (it is possible through [Data Pipeline](http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html)) and this has some [unfortunate consequences](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.AvoidExcessivePTIncreases). Since you need to use the regular service APIs to update existing or create new rows, it is common to temporarily turn up a destination table’s write throughput to speed import. But when the table’s write capacity is increased, DynamoDB may do an irreversible split of the partitions underlying the table, spreading the total table capacity evenly across the new generation of tables. Later, if the capacity is reduced, the capacity for each partition is also reduced but the total number of partitions is not, leaving less capacity for each partition. This leaves the table in a state where it much easier for hotspots to overwhelm individual partitions. - 🔸 It is important to make sure that DynamoDB [resource limits](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-data-types) are compatible with your dataset and workload. For example, the maximum size value that can be added to a DynamoDB table is 400 KB (larger items can be stored in S3 and a URL stored in DynamoDB). - 🔸 Dealing with **time series data** in DynamoDB can be challenging. A global secondary index together with down sampling timestamps can be a possible solution as explained [here](https://blogs.aws.amazon.com/bigdata/post/Tx3KPZDXIBJEQ4B/Scaling-Writes-on-Amazon-DynamoDB-Tables-with-Global-Secondary-Indexes). - 🔸 DynamoDB does [not allow](https://forums.aws.amazon.com/thread.jspa?threadID=90137) an empty string as a valid attribute value. The most common work-around is to use a substitute value instead of leaving the field empty. From 24293fcafe4ac6dcb8a70b8ef46cf44ee65786ed Mon Sep 17 00:00:00 2001 From: Ben Bridts Date: Mon, 12 Dec 2016 14:31:33 +0100 Subject: [PATCH 064/128] Add gotcha about Lambda DLQ regional availability This limitation isn't mentioned in the official docs. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7055be6..874da6b 100644 --- a/README.md +++ b/README.md @@ -1239,6 +1239,7 @@ Lambda - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. +- 🔸 At the time of writing (12 December 2016) Death Letter Queues are only available in the Ohio (us-east-2) region. ### Lambda Code Samples From 8ec2310ea096b1606ac000df35f037fc581a49cf Mon Sep 17 00:00:00 2001 From: Ben Bridts Date: Tue, 13 Dec 2016 10:57:26 +0100 Subject: [PATCH 065/128] Write 'dead' the right way --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 874da6b..fbcb8ac 100644 --- a/README.md +++ b/README.md @@ -1239,7 +1239,7 @@ Lambda - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. -- 🔸 At the time of writing (12 December 2016) Death Letter Queues are only available in the Ohio (us-east-2) region. +- 🔸 At the time of writing (12 December 2016) Dead Letter Queues are only available in the Ohio (us-east-2) region. ### Lambda Code Samples From f0f69632e303c93524856ff36b114b6288702f31 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Tue, 13 Dec 2016 15:11:07 -0800 Subject: [PATCH 066/128] Copy editing. --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 913f75f..2232da5 100644 --- a/README.md +++ b/README.md @@ -1203,9 +1203,8 @@ ECS - Doesn’t support cross-region replication of images. - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. - - A container’s health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. -- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). [Drivers such as fluentd are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. +- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, etc.). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd [are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips From 3bd6b8d84dea8250e0348d8397dc29351d9011ec Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Tue, 13 Dec 2016 15:18:39 -0800 Subject: [PATCH 067/128] Copy edits on #347. --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2232da5..d526fac 100644 --- a/README.md +++ b/README.md @@ -1004,7 +1004,7 @@ CLB - **Apex DNS names:** Once upon a time, you couldn’t assign an CLB to an apex DNS record (i.e. example.com instead of foo.example.com) because it needed to be an A record instead of a CNAME. This is now possible with a Route 53 alias record directly pointing to the load balancer. - 🔸CLBs use [HTTP keep-alives](https://en.wikipedia.org/wiki/HTTP_persistent_connection) on the internal side. This can cause an unexpected side effect: Requests from different clients, each in their own TCP connection on the external side, can end up on the same TCP connection on the internal side. Never assume that multiple requests on the same TCP connection are from the same client! - 🔸 Traffic between CLBs and back-end instances in the same subnet **will** have [Network ACL](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html) rules evaluated (EC2 to EC2 traffic in the same subnet would not have Network ACL rules evaluated). If the default '0.0.0.0/0 ALLOW' rule is removed from the Network ACL applied to the subnet, a rule that allows traffic on both the health check port and any listener port must be added. -- As of December 2016, CLBs launched in VPCs do not support IPv6 addressing. CLBs launched in EC2-Classic support both IPv4 and IPv6 [with the "dualstack" DNS name.](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internet-facing-load-balancers.html#internet-facing-ip-addresses) +- As of December 2016, CLBs launched in VPCs do not support IPv6 addressing. CLBs launched in EC2-Classic support both IPv4 and IPv6 [with the "dualstack" DNS name](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internet-facing-load-balancers.html#internet-facing-ip-addresses). ALB --- @@ -1029,7 +1029,7 @@ ALB - Instances in the ALB’s target groups have to either have a single, fixed healthcheck port (“EC2 instance”-level healthcheck) or the healthcheck port for a target has to be the same as its application port (“Application instance”-level healthcheck) - you can't configure a per-target healthcheck port that is different than the application port. - ALBs are VPC-only (they are not available in EC2 Classic) - In a target group, if there is no healthy target, all requests are routed to all targets. For example, if you point a listener at a target group containing a single service that has a long initialization phase (during which the health checks would fail), requests will reach the service while it is still starting up. -- Although it has been announced as coming soon, [ALBs do not presently support IPv6.](https://aws.amazon.com/about-aws/whats-new/2016/12/announcing-internet-protocol-version-6-support-for-ec2-instances-in-amazon-virtual-private-cloud/) Even if the ALB is launched in an IPv6-enabled VPC and subnet, it will only have IPv4 addressing. +- As of December 2016, although it has been announced as coming soon, ALBs [do not yet support IPv6](https://aws.amazon.com/about-aws/whats-new/2016/12/announcing-internet-protocol-version-6-support-for-ec2-instances-in-amazon-virtual-private-cloud/). Even if the ALB is launched in an IPv6-enabled VPC and subnet, it will only have IPv4 addressing. Elastic IPs ----------- @@ -1368,7 +1368,7 @@ VPCs, Network Security, and Security Groups - **VPC Flow Logs** allow you to monitor the network traffic to, from, and within your VPC. Logs are stored in CloudWatch Logs groups, and can be used for security monitoring (with third party tools), performance evaluation, and forensic investigation. - See the [VPC Flow Logs User Guide](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) for basic information. - See the [flowlogs-reader](https://github.com/obsrvbl/flowlogs-reader) CLI tool and Python library to retrieve and work with VPC Flow Logs. -- **IPv6** [is available in VPC.](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/) Along with this announcement came the introduction of the Egress-Only Internet Gateway. In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. +- **IPv6** [is available in VPC](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/). Along with this announcement came the introduction of the [Egress-Only Internet Gateway](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/egress-only-internet-gateway.html). In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. - Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already. - New and existing VPCs can both use IPv6. Existing VPCs will need to be configured to have an IPv6 CIDR block associated with them, just as new VPCs do. @@ -1423,7 +1423,7 @@ CloudFront ### CloudFront Tips -- 🐥**IPv6** is [supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). This is a configurable setting, and is enabled by default on new CloudFront distribution. IPv6 support extends to the use of WAF with CloudFront. +- 🐥**IPv6** is [supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). This is a configurable setting, and is enabled by default on new CloudFront distributions. IPv6 support extends to the use of WAF with CloudFront. - 🐥**HTTP/2** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/09/amazon-cloudfront-now-supports-http2/)! Clients [must support TLS 1.2 and SNI](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesSupportedHTTPVersions). - While the most common use is for users to browse and download content (GET or HEAD methods) requests, CloudFront also supports ([since 2013](https://aws.amazon.com/blogs/aws/amazon-cloudfront-content-uploads-post-put-other-methods/)) uploaded data (POST, PUT, DELETE, OPTIONS, and PATCH). - You must enable this by specifying the [allowed HTTP methods](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesAllowedHTTPMethods) when you create the distribution. @@ -1686,7 +1686,7 @@ WAF - WAF (Web Application Firewall) is used in conjunction with the CloudFront and ALB services to inspect and block/allow web requests based on user-configurable conditions. - HTTPS and HTTP requests are supported with this service. - WAF's strength is in detecting malicious activity based on pattern-matching inputs for attacks such as SQL injections, XSS, etc. -- WAF supports inspection of requests [received through both IPv6 and IPv4.](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/) +- WAF supports inspection of requests [received through both IPv6 and IPv4](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). ### WAF Tips From 12b7198082a03030de89d5c144d74413bd309570 Mon Sep 17 00:00:00 2001 From: Benjamin Wilson Date: Tue, 13 Dec 2016 20:53:53 -0500 Subject: [PATCH 068/128] Fix incorrect EFS information --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d526fac..e29733b 100644 --- a/README.md +++ b/README.md @@ -943,8 +943,8 @@ EFS - 🔸 A number of NFSv4.1 features are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html) and there are some [limits](http://docs.aws.amazon.com/efs/latest/ug/limits.html) to the service. - 🔸 As of 2016-11, EFS does not offer disk level encryption, though it is on the roadmap. - ❗ Some applications, like SQLite and IPython, [might not work properly](https://sqlite.org/faq.html#q5) on EFS when accessed from multiple clients. This is because lock upgrades and downgrades are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html). There might be [workarounds](https://github.com/ipython/ipython/issues/882) for some issues. -- 🔸 Mounting EFS over VPN connection, VPC peering, or AWS Direct Connect is not supported. -- 🔸 Using an EFS volume on Windows is not supported, apparently due to Microsoft implementing NFS differently. +- 🔸 Mounting EFS over VPN connection, VPC peering, ~~or AWS Direct Connect~~ is not supported. EFS [does support](https://aws.amazon.com/efs/faq/#on-premises) mounting over Direct Connect. +- 🔸 Using an EFS volume on Windows is not supported because while the server service supports NFS v3 and v4.1, the client only supports NFS v3. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. - 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. - 🔸 Mount points are AZ-based. In an Auto scaling group spread across zones, you can end up with instances in one zone mounting EFS from a different zone. That might decrease performance and would create an unintended single point of failure. One way to fix it would be [a shell script](https://gist.github.com/bgdnlp/9fd326dc4a23f46bab93a1eade023fe4) that runs before network drives are mounted and edits /etc/fstab with the proper AZ. From a488563e89b449848fbd89681e746b2995bc3e9d Mon Sep 17 00:00:00 2001 From: Benjamin Wilson Date: Wed, 14 Dec 2016 14:39:48 -0500 Subject: [PATCH 069/128] Clarify Windows NFS client support; formatting --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e29733b..3537d20 100644 --- a/README.md +++ b/README.md @@ -943,8 +943,8 @@ EFS - 🔸 A number of NFSv4.1 features are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html) and there are some [limits](http://docs.aws.amazon.com/efs/latest/ug/limits.html) to the service. - 🔸 As of 2016-11, EFS does not offer disk level encryption, though it is on the roadmap. - ❗ Some applications, like SQLite and IPython, [might not work properly](https://sqlite.org/faq.html#q5) on EFS when accessed from multiple clients. This is because lock upgrades and downgrades are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html). There might be [workarounds](https://github.com/ipython/ipython/issues/882) for some issues. -- 🔸 Mounting EFS over VPN connection, VPC peering, ~~or AWS Direct Connect~~ is not supported. EFS [does support](https://aws.amazon.com/efs/faq/#on-premises) mounting over Direct Connect. -- 🔸 Using an EFS volume on Windows is not supported because while the server service supports NFS v3 and v4.1, the client only supports NFS v3. +- 🔸 Mounting EFS over a VPN connection or VPC peering is not supported. EFS [does support](https://aws.amazon.com/efs/faq/#on-premises) mounting over Direct Connect. +- 🔸 Using an EFS volume on Windows is not supported because the Window NFS client does not support NFS v4 or v4.1. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. - 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. - 🔸 Mount points are AZ-based. In an Auto scaling group spread across zones, you can end up with instances in one zone mounting EFS from a different zone. That might decrease performance and would create an unintended single point of failure. One way to fix it would be [a shell script](https://gist.github.com/bgdnlp/9fd326dc4a23f46bab93a1eade023fe4) that runs before network drives are mounted and edits /etc/fstab with the proper AZ. From d115c3bab9190f696bf1470fe8eccf02f2ee80d6 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Wed, 14 Dec 2016 15:22:28 -0800 Subject: [PATCH 070/128] added warning about root user (#354) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index d526fac..303a478 100644 --- a/README.md +++ b/README.md @@ -561,6 +561,7 @@ We cover security basics first, since configuring user accounts is something you - That way, you define different users, and groups with different levels of privilege (if you want, choose from Amazon’s default suggestions, of administrator, power user, etc.). - This allows credential revocation, which is critical in some situations. If an employee leaves, or a key is compromised, you can revoke credentials with little effort. - You can set up [Active Directory federation](https://blogs.aws.amazon.com/security/post/Tx71TWXXJ3UI14/Enabling-Federation-to-AWS-using-Windows-Active-Directory-ADFS-and-SAML-2-0) to use organizational accounts in AWS. +- ❗**Do NOT use the [IAM Root User account](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html)** other than when you initially create your account. Create custom IAM users and/or roles and use those for your applications instead. - ❗**Enable [MFA](https://aws.amazon.com/iam/details/mfa/)** on your account. - You should always use MFA, and the sooner the better — enabling it when you already have many users is extra work. - Unfortunately it can’t be enforced in software, so an administrative policy has to be established. From 4e0f26712036e1ce320680c179cb01ec0eed762a Mon Sep 17 00:00:00 2001 From: Benjamin Wilson Date: Wed, 14 Dec 2016 20:41:35 -0500 Subject: [PATCH 071/128] Update wording, remove possibly incorrect Windows info --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3537d20..d69a388 100644 --- a/README.md +++ b/README.md @@ -944,7 +944,9 @@ EFS - 🔸 As of 2016-11, EFS does not offer disk level encryption, though it is on the roadmap. - ❗ Some applications, like SQLite and IPython, [might not work properly](https://sqlite.org/faq.html#q5) on EFS when accessed from multiple clients. This is because lock upgrades and downgrades are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html). There might be [workarounds](https://github.com/ipython/ipython/issues/882) for some issues. - 🔸 Mounting EFS over a VPN connection or VPC peering is not supported. EFS [does support](https://aws.amazon.com/efs/faq/#on-premises) mounting over Direct Connect. -- 🔸 Using an EFS volume on Windows is not supported because the Window NFS client does not support NFS v4 or v4.1. +- 🔸 An EFS file system can be mounted on premises over Direct Connect. +- 🔸 An EFS file system can NOT be mounted over VPC peering or VPN, even if the VPN is running on top of Direct Connect. +- 🔸 Using an EFS volume on Windows is not supported. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. - 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. - 🔸 Mount points are AZ-based. In an Auto scaling group spread across zones, you can end up with instances in one zone mounting EFS from a different zone. That might decrease performance and would create an unintended single point of failure. One way to fix it would be [a shell script](https://gist.github.com/bgdnlp/9fd326dc4a23f46bab93a1eade023fe4) that runs before network drives are mounted and edits /etc/fstab with the proper AZ. From 970de0176f192154a3dc23e113ee700dcd191fde Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Wed, 14 Dec 2016 22:09:36 -0800 Subject: [PATCH 072/128] replaced Google Brillo with Android Things Google announced that Brillo was being replaced by 'Android Things' as their IoT OS --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 303a478..121dc27 100644 --- a/README.md +++ b/README.md @@ -1607,7 +1607,7 @@ IoT ### IoT Alternatives and Lock-in -- AWS, Microsoft and Google have all introduced IoT-specific sets of cloud services since late 2015. AWS was first, moving their IoT services to [general availability](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) in Dec 2015. Microsoft released their set of IoT services for Azure in [Feb 2016](https://azure.microsoft.com/en-us/updates/generally-available-microsoft-azure-iot-hub/). Google has only previewed, but not released their IoT services [Brillo](https://developers.google.com/brillo/) and [Weave](https://developers.google.com/weave/). +- AWS, Microsoft and Google have all introduced IoT-specific sets of cloud services since late 2015. AWS was first, moving their IoT services to [general availability](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) in Dec 2015. Microsoft released their set of IoT services for Azure in [Feb 2016](https://azure.microsoft.com/en-us/updates/generally-available-microsoft-azure-iot-hub/). Google has only previewed, but not released their IoT services [Android Things](https://developer.android.com/things/index.html) and [Weave](https://developers.google.com/weave/). - Issues of lock-in center around your devices — [protocols](http://www.postscapes.com/internet-of-things-protocols/) (for example MQTT, AMQP), message formats (such as, JSON vs. Hex...) and security (certificates). ### IoT Tips From ce56c67695ea87153583e0a38ecfb0119201b47e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cristian=20M=C4=83gheru=C8=99an-Stanciu?= Date: Mon, 26 Dec 2016 00:19:47 +0100 Subject: [PATCH 073/128] Trailing whitespace cleanups. --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 303a478..7570557 100644 --- a/README.md +++ b/README.md @@ -536,7 +536,7 @@ We cover security basics first, since configuring user accounts is something you - 📒 IAM [Homepage](https://aws.amazon.com/iam/) ∙ [User guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) ∙ [FAQ](https://aws.amazon.com/iam/faqs/) - The [AWS Security Blog](https://blogs.aws.amazon.com/security) is one of the best sources of news and information on AWS security. - **IAM** is the service you use to manage accounts and permissioning for AWS. -- Managing security and access control with AWS is critical, so every AWS administrator needs to use and understand IAM, at least at a basic level. +- Managing security and access control with AWS is critical, so every AWS administrator needs to use and understand IAM, at least at a basic level. - [IAM identities](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html) include users (people or services that are using AWS), groups (containers for sets of users and their permissions), and roles (containers for permissions assigned to AWS service instances). [Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions.html) for these identities are governed by [policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) You can use AWS pre-defined policies or custom policies that you create. - IAM manages various kinds of authentication, for both users and for software services that may need to authenticate with AWS, including: - [**Passwords**](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords.html) to log into the console. These are a username and password for real users. @@ -766,7 +766,7 @@ EC2 - ⛓ AWS offers an [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB?qid=1475211685369&sr=0-1&ref_=srh_res_product_title) (based on Amazon Linux) with most NVIDIA drivers and ancillary software (CUDA, CUBLAS, CuDNN, TensorFlow) installed to lower the barrier to usage. Note, however, that this leads to lock-in due to Amazon Linux and the fact that you have no direct access to software configuration or versioning. - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. - All current EC2 instance types can take advantage of IPv6 addressing, so long as they are launched in a subnet with an allocated CIDR range in an IPv6-enabled VPC. - + ### EC2 Gotchas and Limitations - ❗Never use ssh passwords. Just don’t do it; they are too insecure, and consequences of compromise too severe. Use keys instead. [Read up on this](https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2) and fully disable ssh password access to your ssh server by making sure 'PasswordAuthentication no' is in your /etc/ssh/sshd_config file. If you’re careful about managing ssh private keys everywhere they are stored, it is a major improvement on security over password-based authentication. @@ -1108,7 +1108,7 @@ RDS MySQL and MariaDB ### RDS MySQL and MariaDB Tips - MySQL RDS allows access to [binary logs](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html#USER_LogAccess.MySQL.BinaryFormat). -- Multi-AZ instances of MySQL transparently replicate data across AZs using DRBD. Automated backups of multi-AZ instances [run off the backup instance](https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls) to reduce latency spikes on the primary. +- Multi-AZ instances of MySQL transparently replicate data across AZs using DRBD. Automated backups of multi-AZ instances [run off the backup instance](https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls) to reduce latency spikes on the primary. - 🔸**MySQL vs MariaDB vs Aurora:** If you prefer a MySQL-style database but are starting something new, you probably should consider Aurora and MariaDB as well. **Aurora** has increased availability and is the next-generation solution. That said, Aurora [may not be](http://blog.takipi.com/benchmarking-aurora-vs-mysql-is-amazons-new-db-really-5x-faster/) as fast relative to MySQL as is sometimes reported, and is more complex to administer. **MariaDB**, the modern [community fork](https://en.wikipedia.org/wiki/MariaDB) of MySQL, [likely now has the edge over MySQL](http://cloudacademy.com/blog/mariadb-vs-mysql-aws-rds/) for many purposes and is supported by RDS. ### RDS MySQL and MariaDB Gotchas and Limitations @@ -1132,10 +1132,10 @@ RDS Aurora ### RDS Aurora Tips - In order to take advantage of Aurora’s higher concurrency, applications should be configured with large database connection pools and should execute as many queries concurrently as possible. For example, Aurora servers have been tested to produce increasing performance on some OLTP workloads with [up to 5,000 connections](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine/31). -- [Aurora scales well with multiple CPUs](https://www.percona.com/blog/2016/05/26/aws-aurora-benchmarking-part-2/) and may require a large instance class for optimal performance. +- [Aurora scales well with multiple CPUs](https://www.percona.com/blog/2016/05/26/aws-aurora-benchmarking-part-2/) and may require a large instance class for optimal performance. - Because Aurora is based on MySQL 5.6.10, avoiding any MySQL features from 5.7 or later will ease the transition from a MySQL-compatible database into Aurora. - The easiest migration path to Aurora is restoring a database snapshot from MySQL 5.6. The next easiest method is restoring a dump from a MySQL-compatible database such as MariaDB. For [low-downtime migrations](http://cantrill.io/howto/aws/2016/06/06/migrating-from-mysql-to-aurora-with-almost-no-downtime.html) from other MySQL-compatible databases, you can set up an Aurora instance as a replica of your existing database. If none of those methods are options, Amazon offers a fee-based data migration service. -- You can replicate [from an Aurora cluster to MySQL or to another Aurora cluster](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html). This requires binary logging to be enabled and is not as performant as native Aurora replication. +- You can replicate [from an Aurora cluster to MySQL or to another Aurora cluster](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html). This requires binary logging to be enabled and is not as performant as native Aurora replication. ### RDS Aurora Gotchas and Limitations @@ -1668,12 +1668,12 @@ Certificate Manager - ⛓Certificates issued by the Certificate Manager can’t be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. -### Certificate Manager Tips +### Certificate Manager Tips - 🔹**Supported services:** Managed [Load Balancers](#load-balancers) and [CloudFront](#cloudfront). - 🔸During the domain validation process, Certificate Manager will send an email to every contact address specified in the domain’s WHOIS record and up to five common administrative addresses. Some anti-spam filters can mark emails as spam because of this. You should check the spam folder of your email if you don’t receive a confirmation email. -### Certificate Manager Gotchas and Limitations +### Certificate Manager Gotchas and Limitations - 🔸In order to use **Certificate Manager** for CloudFront distributions certificate must be issued or imported from us-east-1 (N. Virginia) region. Certificates from other regions can [only be used with Elastic Load Balancers](https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html). - 🔸**IoT** has its [own way](http://docs.aws.amazon.com/iot/latest/developerguide/create-device-certificate.html) of setting up certificates. @@ -1689,13 +1689,13 @@ WAF - WAF's strength is in detecting malicious activity based on pattern-matching inputs for attacks such as SQL injections, XSS, etc. - WAF supports inspection of requests [received through both IPv6 and IPv4](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/). -### WAF Tips +### WAF Tips - Getting a WAF API call history can be done through CloudTrail. This is enabled through the CloudTrail console. -### WAF Gotchas and Limitations +### WAF Gotchas and Limitations -- As of December 2016, WAF is available in the US East (Northern Virginia), US West (Oregon), Asia Pacific (Tokyo) and EU (Ireland) regions. +- As of December 2016, WAF is available in the US East (Northern Virginia), US West (Oregon), Asia Pacific (Tokyo) and EU (Ireland) regions. High Availability ----------------- From 18772b340005968acbcd080e95b7d023e0f0de8c Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sat, 7 Jan 2017 14:30:20 -0800 Subject: [PATCH 074/128] added info about using your own CA for IoT I have found this feature to be in high demand from my customers, so wanted to add a link to it here --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6d6aec8..0e94927 100644 --- a/README.md +++ b/README.md @@ -1598,7 +1598,7 @@ IoT - AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. * **IoT terms:** * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). Device metadata can also be stored in [**IoT Thing Types**](http://docs.aws.amazon.com/iot/latest/developerguide/thing-types.html). This aids in device metadata management by allowing for reuse of device description and configuration for more than one device. Note that IoT Thing Types can be deprecated, but not changed — they are immutable. - * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). + * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). You can associate an AWS IoT certificate with an IoT device or you can [register your own CA (Certificate Authority) with AWS](http://docs.aws.amazon.com/iot/latest/developerguide/device-certs-your-own.html), generate your own certificate(s) and associate those certificates with your devices via the AWS Console or cli. * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). * Shown below is a [diagram](https://aws.amazon.com/iot/how-it-works/) which summarizes the flow of messages between the AWS IoT services: From 4045fde6624a27619c0db50bd0ba4996a6135c02 Mon Sep 17 00:00:00 2001 From: Benjamin Wilson Date: Sun, 8 Jan 2017 09:11:51 -0500 Subject: [PATCH 075/128] Add link to EFS/Direct Connect docs --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d69a388..3e56e63 100644 --- a/README.md +++ b/README.md @@ -944,7 +944,7 @@ EFS - 🔸 As of 2016-11, EFS does not offer disk level encryption, though it is on the roadmap. - ❗ Some applications, like SQLite and IPython, [might not work properly](https://sqlite.org/faq.html#q5) on EFS when accessed from multiple clients. This is because lock upgrades and downgrades are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html). There might be [workarounds](https://github.com/ipython/ipython/issues/882) for some issues. - 🔸 Mounting EFS over a VPN connection or VPC peering is not supported. EFS [does support](https://aws.amazon.com/efs/faq/#on-premises) mounting over Direct Connect. -- 🔸 An EFS file system can be mounted on premises over Direct Connect. +- 🔸 An EFS file system [can be mounted on premises](https://aws.amazon.com/efs/faq/#on-premises) over Direct Connect. - 🔸 An EFS file system can NOT be mounted over VPC peering or VPN, even if the VPN is running on top of Direct Connect. - 🔸 Using an EFS volume on Windows is not supported. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. From ddb7b8707e31232fd5c40eada65e7674ac026e4a Mon Sep 17 00:00:00 2001 From: Rich Adams Date: Mon, 9 Jan 2017 11:40:50 -0800 Subject: [PATCH 076/128] Add note about EIP migration between Classic and VPC. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6d6aec8..55ff2cd 100644 --- a/README.md +++ b/README.md @@ -1359,6 +1359,7 @@ VPCs, Network Security, and Security Groups - ❗**Security groups** are your first line of defense for your servers. Be extremely restrictive of what ports are open to all incoming connections. In general, if you use CLBs, ALBs or other load balancing, the only ports that need to be open to incoming traffic would be port 22 and whatever port your application uses. Security groups access policy is 'deny by default'. - **Port hygiene:** A good habit is to pick unique ports within an unusual range for each different kind of production service. For example, your web frontend might use 3010, your backend services 3020 and 3021, and your Postgres instances the usual 5432. Then make sure you have fine-grained security groups for each set of servers. This makes you disciplined about listing out your services, but also is more error-proof. For example, should you accidentally have an extra Apache server running on the default port 80 on a backend server, it will not be exposed. - **Migrating from Classic**: For migrating from older EC2-Classic deployments to modern EC2-VPC setup, [this article](http://blog.kiip.me/engineering/ec2-to-vpc-executing-a-zero-downtime-migration/) may be of help. + - You can [migrate Elastic IPs between EC2-Classic and EC2-VPC](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#using-eip-migration). - For basic AWS use, one default VPC may be sufficient. But as you scale up, you should consider mapping out network topology more thoroughly. A good overview of best practices is [here](http://blog.flux7.com/blogs/aws/vpc-best-configuration-practices). - Consider controlling access to you private AWS resources through a [VPN](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html). - You get better visibility into and control of connection and connection attempts. From 98192b853cf0fcbb46fa036ce801d3c11e283816 Mon Sep 17 00:00:00 2001 From: AJ Kerrigan Date: Wed, 11 Jan 2017 13:19:47 -0500 Subject: [PATCH 077/128] Add information about service limits (#359) Reference the awslimitchecker tool as well as its documentation, which helps identify available ways to check various limits. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6d6aec8..ac31368 100644 --- a/README.md +++ b/README.md @@ -399,6 +399,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp ### Restrictions and Other Notes - 🔸Lots of resources in Amazon have [**limits**](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) on them. This is actually helpful, so you don’t incur large costs accidentally. You have to request that quotas be increased by opening support tickets. Some limits are easy to raise, and some are not. (Some of these are noted in sections below.) + - **Obtaining Current Limits and Usage:** Limit information for a service may be available from the service API, Trusted Advisor, both or neither (in which case you'll need to contact Support). [This page](http://awslimitchecker.readthedocs.io/en/latest/limits.html) from the awslimitchecker tool's documentation provides a nice summary of available retrieval options for each limit. The [tool](https://github.com/jantman/awslimitchecker) itself is also valuable for automating limit checks. - 🔸[**AWS terms of service**](https://aws.amazon.com/service-terms/) are extensive. Much is expected boilerplate, but it does contain important notes and restrictions on each service. In particular, there are restrictions against using many AWS services in **safety-critical systems**. (Those appreciative of legal humor may wish to review clause 57.10.) ### Related Topics From cf6b4186d3c1775189f789a300a0a9aed35f66b3 Mon Sep 17 00:00:00 2001 From: Mike Jennings Date: Fri, 13 Jan 2017 10:23:56 -0500 Subject: [PATCH 078/128] added digitalocean blockstorage --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6d6aec8..1bb27a1 100644 --- a/README.md +++ b/README.md @@ -280,7 +280,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | PubNub Blocks, Auth0 Webtask | Kong, Tyk | | Container, cluster manager | ECS | Container Engine, Kubernetes | Borg or Omega | Container Service | | Kubernetes, Mesos, Aurora | | File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | -| Block storage | EBS | Persistent Disk | | Storage Account | | NFS | +| Block storage | EBS | Persistent Disk | | Storage Account | DigitalOcean Volumes | NFS | | SQL datastore | RDS | Cloud SQL | | SQL Database | | MySQL, PostgreSQL | | Sharded RDBMS | | | F1, Spanner | | | Crate.io, CockroachDB | | Bigtable | | Cloud Bigtable | Bigtable | | | HBase | From a86342a34b65a76f9a0d806feb9d81c4e587bf01 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Mon, 16 Jan 2017 07:01:11 +0100 Subject: [PATCH 079/128] Add basic info for new machine learning services --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 6d6aec8..bef69c5 100644 --- a/README.md +++ b/README.md @@ -225,6 +225,9 @@ General Information - 🐥[Certificate Manager](https://aws.amazon.com/certificate-manager/): Manage SSL/TLS certificates for AWS services - **Compound services:** These are similarly specific, but are full-blown services that tackle complex problems and may tie you in. Usefulness depends on your requirements. If you have large or significant need, you may have these already managed by in-house systems and engineering teams. - [Machine Learning](https://aws.amazon.com/machine-learning/): Machine learning model training and classification + - [Lex](https://aws.amazon.com/lex/): Automatic speech recognition (ASR) and natural language understanding (NLU) + - [Polly](https://aws.amazon.com/polly/): Text-to-speech engine in the cloud + - [Rekognition](https://aws.amazon.com/rekognition/): Service for image recognition - ⛓🕍[Data Pipeline](https://aws.amazon.com/datapipeline/): Managed ETL service - ⛓🕍[SWF](https://aws.amazon.com/swf/): Managed state tracker for distributed polyglot job workflow - ⛓🕍[Lumberyard](https://aws.amazon.com/lumberyard/): 3D game engine @@ -318,6 +321,9 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | Service | Original release | Availability | CLI Support | |------------------------------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------|:-----------:| +| 🐥[Lex](https://aws.amazon.com/releasenotes/Amazon-Lex?browse=1) | 2016-11 | Preview | | +| 🐥[Polly](https://aws.amazon.com/releasenotes/Amazon-Polly?browse=1) | 2016-11 | General |✓ | +| 🐥[Rekognition](https://aws.amazon.com/releasenotes/Amazon-Rekognition?browse=1) | 2016-11 | General |✓ | | 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | | | 🐥[Certificate Manager](https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/) | 2016-01 | General | ✓ | 🐥[IoT](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) | 2015-08 | General | ✓ | From 6858ec217c8c914a487ed5468a305d67aa18d52d Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Mon, 16 Jan 2017 07:20:39 +0100 Subject: [PATCH 080/128] Add info to service matrix --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index bef69c5..d6f1795 100644 --- a/README.md +++ b/README.md @@ -307,6 +307,9 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Mobile app analytics | Mobile Analytics | Firebase Analytics | | HockeyApp | Mixpanel | | | Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | | Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | +| Automatic speech recognition and natural language understanding | Lex | Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA | +| Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | +| Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | 🚧 [*Please help fill this table in.*](CONTRIBUTING.md) From f2d348938e795fd9bb29a1b568bcad6f17070f14 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Mon, 16 Jan 2017 16:22:14 +0100 Subject: [PATCH 081/128] Add spaCy for NLP --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d6f1795..676772b 100644 --- a/README.md +++ b/README.md @@ -307,7 +307,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Mobile app analytics | Mobile Analytics | Firebase Analytics | | HockeyApp | Mixpanel | | | Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | | Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | -| Automatic speech recognition and natural language understanding | Lex | Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA | +| Automatic speech recognition and natural language understanding | Lex | Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | | Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | | Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | From 1f7ad605155ff571b9701ca75704a32b56b17530 Mon Sep 17 00:00:00 2001 From: Bogdan Luput Date: Mon, 16 Jan 2017 23:37:57 +0200 Subject: [PATCH 082/128] Remove one EFS gitcha that was present twice in different forms --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 13f0670..eef4075 100644 --- a/README.md +++ b/README.md @@ -944,7 +944,6 @@ EFS - 🔸 A number of NFSv4.1 features are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html) and there are some [limits](http://docs.aws.amazon.com/efs/latest/ug/limits.html) to the service. - 🔸 As of 2016-11, EFS does not offer disk level encryption, though it is on the roadmap. - ❗ Some applications, like SQLite and IPython, [might not work properly](https://sqlite.org/faq.html#q5) on EFS when accessed from multiple clients. This is because lock upgrades and downgrades are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html). There might be [workarounds](https://github.com/ipython/ipython/issues/882) for some issues. -- 🔸 Mounting EFS over a VPN connection or VPC peering is not supported. EFS [does support](https://aws.amazon.com/efs/faq/#on-premises) mounting over Direct Connect. - 🔸 An EFS file system [can be mounted on premises](https://aws.amazon.com/efs/faq/#on-premises) over Direct Connect. - 🔸 An EFS file system can NOT be mounted over VPC peering or VPN, even if the VPN is running on top of Direct Connect. - 🔸 Using an EFS volume on Windows is not supported. From a24e11abcac8b0864cb2374f164a4e34da6804ea Mon Sep 17 00:00:00 2001 From: Bogdan Luput Date: Mon, 16 Jan 2017 23:58:13 +0200 Subject: [PATCH 083/128] Remove obsolete gotcha about EFS mountpoints --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index a29df0e..8bee45c 100644 --- a/README.md +++ b/README.md @@ -958,7 +958,6 @@ EFS - 🔸 Using an EFS volume on Windows is not supported. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. - 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. -- 🔸 Mount points are AZ-based. In an Auto scaling group spread across zones, you can end up with instances in one zone mounting EFS from a different zone. That might decrease performance and would create an unintended single point of failure. One way to fix it would be [a shell script](https://gist.github.com/bgdnlp/9fd326dc4a23f46bab93a1eade023fe4) that runs before network drives are mounted and edits /etc/fstab with the proper AZ. Load Balancers From 7e1ba2ad98baf5b923fe956a8ea6acff8141b80c Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Mon, 16 Jan 2017 15:41:02 -0800 Subject: [PATCH 084/128] Fix signup link. (Changed URL due to cert change.) Fixes #365. --- CONTRIBUTING.md | 4 ++-- README.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2be8041..386c2f8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,7 @@ Contributing ------------ -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws-slack.lexikon.io/) Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly [credit](/AUTHORS.md) all contributors. Here are few notes before you jump in. @@ -10,7 +10,7 @@ Please Help If you’ve found this guide useful, please see if you can help (in increasing levels of commitment and expertise): -- **Discussion:** The easiest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) and ask or answer questions. As we discuss, see if it points to new things you or others can contribute to the Guide. +- **Discussion:** The easiest thing you can do to contribute is [**join the Slack channel**](https://og-aws-slack.lexikon.io/) and ask or answer questions. As we discuss, see if it points to new things you or others can contribute to the Guide. helps the community and guides what contributors can focus on. - **Focused pull requests:** [Pull requests](https://github.com/open-guides/og-aws/pulls) with focused changes like typos, specific tips, and corrections are always welcome and fast to review and merge in. Keep them small and focused, and *use multiple PRs for unrelated changes*. (See [writing conventions](#writing-conventions) below.) - **Track issues:** [File issues](https://github.com/open-guides/og-aws/issues) to aggregate ideas or links if it’s clear something needs to be improved, but it’s not possible to file a PR immediately. diff --git a/README.md b/README.md index a29df0e..047fb8f 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ The Open Guide to Amazon Web Services ===================================== -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) ⇦ Join us! +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws-slack.lexikon.io/) ⇦ Join us! [Credits](AUTHORS.md) ∙ [Contributing guidelines](CONTRIBUTING.md) @@ -95,9 +95,9 @@ Before using the guide, please read the [**license**](#license) and [**disclaime **This is an early in-progress draft!** It’s our first attempt at assembling this information, so is far from comprehensive still, and likely to have omissions or errors. -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws-slack.lexikon.io/) -Please help by [**joining the Slack channel**](https://og-aws.slack.lexikon.io/) (we like to talk about AWS in general, even if you only have questions — discussion helps the community and guides improvements) and [**contributing to the guide**](CONTRIBUTING.md). This guide is *open to contributions*, so unlike a blog, it can keep improving. Like any open source effort, we combine efforts but also review to ensure high quality. +Please help by [**joining the Slack channel**](https://og-aws-slack.lexikon.io/) (we like to talk about AWS in general, even if you only have questions — discussion helps the community and guides improvements) and [**contributing to the guide**](CONTRIBUTING.md). This guide is *open to contributions*, so unlike a blog, it can keep improving. Like any open source effort, we combine efforts but also review to ensure high quality. Scope ----- From 560c5c12ab0e51162dd0da5666d767ec914cb10b Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Mon, 16 Jan 2017 17:50:55 -0800 Subject: [PATCH 085/128] Added: Snowmobile (#362) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 46375ae..15967ca 100644 --- a/README.md +++ b/README.md @@ -248,6 +248,7 @@ General Information - [Service Catalog](https://aws.amazon.com/servicecatalog/): IT service approval and compliance - **Probably-don't-need-to-know services:** Bottom line, our informal polling indicates these services are just not broadly used — and often for good reasons: - [Snowball](https://aws.amazon.com/importexport/): If you want to ship petabytes of data into or out of Amazon using a physical appliance, read on. + - [Snowmobile](https://aws.amazon.com/snowmobile/): Appliances are great, but if you've got exabyte scale data to get into Amazon, nothing beats a tractor trailer full of drives. - [CodeCommit](https://aws.amazon.com/codecommit/): Git service. You’re probably already using GitHub or your own solution ([Stackshare](http://stackshare.io/stackups/github-vs-bitbucket-vs-aws-codecommit) has informal stats). - 🕍[CodePipeline](https://aws.amazon.com/codepipeline/): Continuous integration. You likely have another solution already. - 🕍[CodeDeploy](https://aws.amazon.com/codedeploy/): Deployment of code to EC2 servers. Again, you likely have another solution. From 7de037a13c26bdd9bc4197e40d3798b4f4a2229a Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Mon, 16 Jan 2017 17:55:24 -0800 Subject: [PATCH 086/128] Updated section on Amazon Linux Docker Support. (#363) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 15967ca..5d182c6 100644 --- a/README.md +++ b/README.md @@ -745,7 +745,7 @@ EC2 - Running EC2 is akin to running a set of physical servers, as long as you don’t do automatic scaling or tooled cluster setup. If you just run a set of static instances, migrating to another VPS or dedicated server provider should not be too hard. - 🚪**Alternatives to EC2:** The direct alternatives are Google Cloud, Microsoft Azure, Rackspace, DigitalOcean and other VPS providers, some of which offer similar API for setting up and removing instances. (See the comparisons [above](#when-to-use-aws).) -- **Should you use Amazon Linux?** AWS encourages use of their own [Amazon Linux](https://aws.amazon.com/amazon-linux-ami/), which is evolved from [Red Hat Enterprise Linux (RHEL)](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux) and [CentOS](https://en.wikipedia.org/wiki/CentOS). It’s used by many, but [others are skeptical](https://www.exratione.com/2014/08/do-not-use-amazon-linux/). Whatever you do, think this decision through carefully. It’s true Amazon Linux is heavily tested and better supported in the unlikely event you have deeper issues with OS and virtualization on EC2. But in general, many companies do just fine using a standard, non-Amazon Linux distribution, such as Ubuntu or CentOS. Using a standard Linux distribution means you have an exactly replicable environment should you use another hosting provider instead of (or in addition to) AWS. It’s also helpful if you wish to test deployments on local developer machines running the same standard Linux distribution (a practice that’s getting more common with Docker, too, and not currently possible with Amazon Linux). +- **Should you use Amazon Linux?** AWS encourages use of their own [Amazon Linux](https://aws.amazon.com/amazon-linux-ami/), which is evolved from [Red Hat Enterprise Linux (RHEL)](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux) and [CentOS](https://en.wikipedia.org/wiki/CentOS). It’s used by many, but [others are skeptical](https://www.exratione.com/2014/08/do-not-use-amazon-linux/). Whatever you do, think this decision through carefully. It’s true Amazon Linux is heavily tested and better supported in the unlikely event you have deeper issues with OS and virtualization on EC2. But in general, many companies do just fine using a standard, non-Amazon Linux distribution, such as Ubuntu or CentOS. Using a standard Linux distribution means you have an exactly replicable environment should you use another hosting provider instead of (or in addition to) AWS. It’s also helpful if you wish to test deployments on local developer machines running the same standard Linux distribution (a practice that’s getting more common with Docker, too. Amazon now supports an official [Amazon Linux Docker image](http://docs.aws.amazon.com/AmazonECR/latest/userguide/amazon_linux_container_image.html), aimed at assisting with local development on a comparable environment, though this is new enough that it should be considered experimental). - **EC2 costs:** See the [section on this](#ec2-cost-management). ### EC2 Tips From 53e44fadf8ded73ace60c0c8f2521b3117ea1487 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Mon, 16 Jan 2017 17:56:04 -0800 Subject: [PATCH 087/128] Noted upgraded TrustedAdvisor functionality. (#364) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5d182c6..31e92aa 100644 --- a/README.md +++ b/README.md @@ -593,7 +593,7 @@ We cover security basics first, since configuring user accounts is something you - Cost of multi-account “overhead”: Internal AWS service management tools may need to be custom built or adapted. - 🔹It can help to use separate AWS accounts for independent parts of your infrastructure if you expect a high rate of AWS API calls, since AWS [throttles calls](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) at the AWS account level. - [**Inspector**](https://aws.amazon.com/inspector/) is an automated security assessment service from AWS that helps identify common security risks. This allows validation that you adhere to certain security practices and may help with compliance. -- [**Trusted Advisor**](https://aws.amazon.com/blogs/aws/trusted-advisor-console-basic/) addresses a variety of best practices, but also offers some basic security checks around IAM usage, security group configurations, and MFA. +- [**Trusted Advisor**](https://aws.amazon.com/blogs/aws/trusted-advisor-console-basic/) addresses a variety of best practices, but also offers some basic security checks around IAM usage, security group configurations, and MFA. At paid support tiers, Trusted Advisor exposes additional checks around other areas, such as reserved instance optimization. - **Use KMS for managing keys**: AWS offers [KMS](#kms) for securely managing encryption keys, which is usually a far better option than handling key security yourself. See [below](#kms). - [**AWS WAF**](https://aws.amazon.com/waf) is a web application firewall to help you protect your applications from common attack patterns. - **Security auditing:** From b1b75f5170315af5021624840060e2cee68cdacd Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Tue, 17 Jan 2017 05:38:40 +0100 Subject: [PATCH 088/128] Add info about limitations for API Gateway (#372) * Add info about limitations for API Gateway * Fix description --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 31e92aa..fcba8fa 100644 --- a/README.md +++ b/README.md @@ -1285,6 +1285,7 @@ API Gateway - 🔸API Gateway only supports encrypted (https) endpoints, and does not support unencrypted HTTP. (This is probably a good thing.) - 🔸API Gateway endpoints are always public, i.e. internet facing, and there is no mechanism to build private endpoints, e.g. for internal use on a [VPC](#vpcs-network-security-and-security-groups) but endpoints and their related resources can, optionally, [require authentication](http://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html). - 🔸API Gateway doesn’t support multi-region deployments for high availability. It is a service that is deployed in a single region but comes with a global endpoint that is served from AWS edge locations (similar to a CloudFront distribution). You cannot have multiple API Gateways with the same hostname in different AWS regions and use Route 53 to distribute the traffic. More in [this forum post](https://forums.aws.amazon.com/thread.jspa?messageID=735342򳡮). +- 🔸Integration timeout: All of the various integration types (eg: Lambda, HTTP) for API Gateway have timeouts, as described [here](http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#api-gateway-limits). Unlike some limits, these timeouts can't be increased. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From c9ff71d6dcd4748a840fcd394ba275b5d4fc0f81 Mon Sep 17 00:00:00 2001 From: Dorrin Date: Tue, 24 Jan 2017 10:39:35 -0500 Subject: [PATCH 089/128] lambda function tips for S3 triggers (#371) * lambda function tips for S3 triggers * tips on optimizing ECS disk clean up --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index a13bb4e..dfb96e0 100644 --- a/README.md +++ b/README.md @@ -1216,12 +1216,15 @@ ECS - If you want fast fleet-wide pulls of large images, you’ll need to push your image into a region-local registry. - Doesn’t support custom domains / certificates. - A container’s health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. + +- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). [Drivers such as fluentd are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. - ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, etc.). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd [are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips - **Log drivers:** ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, ...). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. - [This blog from Convox](https://convox.com/blog/ecs-challenges/) (and [commentary](https://news.ycombinator.com/item?id=11598058)) lists a number of common challenges with ECS as of early 2016. +- It is possible to optimize disk clean up on ECS. By default the unused containers are deleted after 3 hours and the unused images after 30 minutes. These settings can be changed by adding `ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m` and `ECS_IMAGE_CLEANUP_INTERVAL=10m` to `/etc/ecs/ecs.config`. [More information on optimizing ECS disk cleanup](https://aws.amazon.com/blogs/compute/optimizing-disk-usage-on-amazon-ecs/). ### ECS Alternatives and Lock-in @@ -1256,6 +1259,7 @@ Lambda - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. +- 🔸 While adding/removing S3 buckets as triggers for Lambda function, this error may occur: "There was an error creating the trigger: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type." In this case, remove the trigger manually by going to the "Properties" section of the S3 bucket, under "Events", remove the Lambda event. - 🔸 At the time of writing (12 December 2016) Dead Letter Queues are only available in the Ohio (us-east-2) region. ### Lambda Code Samples @@ -1383,7 +1387,9 @@ VPCs, Network Security, and Security Groups - **VPC Flow Logs** allow you to monitor the network traffic to, from, and within your VPC. Logs are stored in CloudWatch Logs groups, and can be used for security monitoring (with third party tools), performance evaluation, and forensic investigation. - See the [VPC Flow Logs User Guide](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) for basic information. - See the [flowlogs-reader](https://github.com/obsrvbl/flowlogs-reader) CLI tool and Python library to retrieve and work with VPC Flow Logs. +- **IPv6** [is available in VPC.](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/) Along with this announcement came the introduction of the Egress-Only Internet Gateway. In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. - **IPv6** [is available in VPC](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/). Along with this announcement came the introduction of the [Egress-Only Internet Gateway](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/egress-only-internet-gateway.html). In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. + - Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already. - New and existing VPCs can both use IPv6. Existing VPCs will need to be configured to have an IPv6 CIDR block associated with them, just as new VPCs do. From 79c3e5d416e31e5a7b93dfce197dcf0434718016 Mon Sep 17 00:00:00 2001 From: sprutner Date: Wed, 8 Feb 2017 17:38:07 -0800 Subject: [PATCH 090/128] Update README.md grammar...kind of. While 'Myraid of' is technically OK, it isn't really in common style to use the of. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dfb96e0..e5c2689 100644 --- a/README.md +++ b/README.md @@ -159,7 +159,7 @@ General Information - Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys. - Traditional managed hosting providers such as [Rackspace](https://www.rackspace.com/) offer cloud solutions as well. - 🚪**AWS vs. PaaS:** If your goal is just to put up a single service that does something relatively simple, and you’re trying to minimize time managing operations engineering, consider a [platform-as-a-service](https://en.wikipedia.org/wiki/Platform_as_a_service) such as [Heroku](https://www.heroku.com/). The AWS approach to PaaS, Elastic Beanstalk, is arguably more complex, especially for simple use cases. -- 🚪**AWS vs. web hosting:** If your main goal is to host a website or blog, and you don’t expect to be building an app or more complex service, you may wish consider one of the myriad of [web hosting services](https://www.google.com/search?q=web+hosting). +- 🚪**AWS vs. web hosting:** If your main goal is to host a website or blog, and you don’t expect to be building an app or more complex service, you may wish consider one of the myriad [web hosting services](https://www.google.com/search?q=web+hosting). - 🚪**AWS vs. managed hosting:** Traditionally, many companies pay [managed hosting](https://en.wikipedia.org/wiki/Dedicated_hosting_service) providers to maintain physical servers for them, then build and deploy their software on top of the rented hardware. This makes sense for businesses who want direct control over hardware, due to legacy, performance, or special compliance constraints, but is usually considered old fashioned or unnecessary by many developer-centric startups and younger tech companies. - **Complexity:** AWS will let you build and scale systems to the size of the largest companies, but the complexity of the services when used at scale requires significant depth of knowledge and experience. Even very simple use cases often require more knowledge to do “right” in AWS than in a simpler environment like Heroku or Digital Ocean. (This guide may help!) - **Geographic locations:** AWS has data centers in [over a dozen geographic locations](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions), known as **regions**, in Europe, East Asia, North and South America, and now Australia and India. It also has many more **edge locations** globally for reduced latency of services like CloudFront. From 8e95de7c50ee2d7d849c039400d7ad7d3d295796 Mon Sep 17 00:00:00 2001 From: Daniel Ehrlich Date: Sun, 12 Feb 2017 20:43:32 -0600 Subject: [PATCH 091/128] Added Google products to comparison table --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index dfb96e0..15048a7 100644 --- a/README.md +++ b/README.md @@ -304,12 +304,12 @@ Many services within AWS can at least be compared with Google Cloud offerings or | DNS | Route53 | DNS | | DNS | | bind | | Email | SES | | | | Sendgrid, Mandrill, Postmark | | | Git hosting | CodeCommit | Cloud Source Repositories | | Visual Studio Team Services | GitHub, BitBucket | GitLab | -| User authentication | Cognito | | | Azure Active Directory | | oauth.io | +| User authentication | Cognito | Firebase Authentication | | Azure Active Directory | | oauth.io | | Mobile app analytics | Mobile Analytics | Firebase Analytics | | HockeyApp | Mixpanel | | | Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | | Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | | Automatic speech recognition and natural language understanding | Lex | Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | -| Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | +| Text-to-speech engine in the cloud | Polly | Cloud Speech API | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | | Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | From 649324d951ac94ba5956ea7f07cf07b3090efb60 Mon Sep 17 00:00:00 2001 From: Daniel Ehrlich Date: Sun, 12 Feb 2017 22:28:12 -0600 Subject: [PATCH 092/128] Updated Cloud Speech API category --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 15048a7..38d3101 100644 --- a/README.md +++ b/README.md @@ -308,8 +308,8 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Mobile app analytics | Mobile Analytics | Firebase Analytics | | HockeyApp | Mixpanel | | | Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | | Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | -| Automatic speech recognition and natural language understanding | Lex | Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | -| Text-to-speech engine in the cloud | Polly | Cloud Speech API | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | +| Automatic speech recognition and natural language understanding | Lex | Cloud Speech API, Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | +| Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | | Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | From cdf44b97de327dda94c1763f18768efa190570dd Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Tue, 14 Feb 2017 18:32:00 -0800 Subject: [PATCH 093/128] Add Google Cloud Spanner to Service Matrix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7762bd3..f08d1c5 100644 --- a/README.md +++ b/README.md @@ -286,7 +286,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | | Block storage | EBS | Persistent Disk | | Storage Account | DigitalOcean Volumes | NFS | | SQL datastore | RDS | Cloud SQL | | SQL Database | | MySQL, PostgreSQL | -| Sharded RDBMS | | | F1, Spanner | | | Crate.io, CockroachDB | +| Sharded RDBMS | | Cloud Spanner | F1, Spanner | | | Crate.io, CockroachDB | | Bigtable | | Cloud Bigtable | Bigtable | | | HBase | | Key-value store, column store | DynamoDB | Cloud Datastore | Megastore | Tables, DocumentDB | | Cassandra, CouchDB, RethinkDB, Redis | | Memory cache | ElastiCache | App Engine Memcache | | Redis Cache | | Memcached, Redis | From d9e9cc133ea341ad449d9957a5cf5e8cb3208982 Mon Sep 17 00:00:00 2001 From: Laurent Jalbert-Simard Date: Thu, 23 Feb 2017 23:00:25 -0500 Subject: [PATCH 094/128] Update ALB Tips and Gotchas regarding ipv6 (#384) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7762bd3..1241075 100644 --- a/README.md +++ b/README.md @@ -1025,6 +1025,7 @@ ALB - 📒 [Homepage](https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/) ∙ [User guide](https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/developer-resources/) ∙ [FAQ](https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/faqs/) ∙ [Pricing](https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/pricing/) - 🐥**Websockets and HTTP/2** are [now supported](https://aws.amazon.com/blogs/aws/new-aws-application-load-balancer/). +- 🐥**Internet Protocol Version 6 (IPv6)** is [now supported](https://aws.amazon.com/about-aws/whats-new/2017/01/announcing-internet-protocol-version-6-ipv6-support-for-elastic-load-balancing-in-amazon-virtual-private-cloud-vpc/). - Prior to the Application Load Balancer, you were advised to use TCP instead of HTTP as the protocol to make it work (as described [here](http://www.quora.com/When-will-Amazon-ELB-offer-SPDY-support)) and use [the obscure but useful Proxy Protocol](http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html) ([more on this](https://chrislea.com/2014/03/20/using-proxy-protocol-nginx/)) to pass client IPs over a TCP load balancer. ### ALB Tips @@ -1041,7 +1042,6 @@ ALB - Instances in the ALB’s target groups have to either have a single, fixed healthcheck port (“EC2 instance”-level healthcheck) or the healthcheck port for a target has to be the same as its application port (“Application instance”-level healthcheck) - you can't configure a per-target healthcheck port that is different than the application port. - ALBs are VPC-only (they are not available in EC2 Classic) - In a target group, if there is no healthy target, all requests are routed to all targets. For example, if you point a listener at a target group containing a single service that has a long initialization phase (during which the health checks would fail), requests will reach the service while it is still starting up. -- As of December 2016, although it has been announced as coming soon, ALBs [do not yet support IPv6](https://aws.amazon.com/about-aws/whats-new/2016/12/announcing-internet-protocol-version-6-support-for-ec2-instances-in-amazon-virtual-private-cloud/). Even if the ALB is launched in an IPv6-enabled VPC and subnet, it will only have IPv4 addressing. Elastic IPs ----------- From baef21cc26360c81083b5e42fcd604403716e9cd Mon Sep 17 00:00:00 2001 From: Dorrin Date: Mon, 27 Feb 2017 17:40:22 -0500 Subject: [PATCH 095/128] ECS disk optimization tips (#379) * lambda function tips for S3 triggers * tips on optimizing ECS disk clean up * fixing the suggestions --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1241075..a01edd6 100644 --- a/README.md +++ b/README.md @@ -1217,14 +1217,14 @@ ECS - Doesn’t support custom domains / certificates. - A container’s health is monitored via [CLB](#clb) or [ALB](#alb). Those can also be used to address a containerized service. When using an ALB you do not need to handle port contention (i.e. services exposing the same port on the same host) since an ALB’s target groups can be associated with ECS-based services directly. -- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). [Drivers such as fluentd are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. -- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, etc.). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd [are not enable by default](https://github.com/aws/amazon-ecs-agent/issues/535). To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. + +- ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, json, ... ). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). [Drivers such as fluentd are not enabled by default](https://github.com/aws/amazon-ecs-agent/issues/535). You can, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. ### ECS Tips -- **Log drivers:** ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, ...). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd are not enable by default. To do so, install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. +- **Log drivers:** ECS supports multiple log drivers (awslogs, splunk, fluentd, syslog, JSON, ...). Use [`awslogs`](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html) for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd are not enabled by default. You can install the agent and enable the driver by adding `ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'` to `/etc/ecs/ecs.config`. - [This blog from Convox](https://convox.com/blog/ecs-challenges/) (and [commentary](https://news.ycombinator.com/item?id=11598058)) lists a number of common challenges with ECS as of early 2016. -- It is possible to optimize disk clean up on ECS. By default the unused containers are deleted after 3 hours and the unused images after 30 minutes. These settings can be changed by adding `ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m` and `ECS_IMAGE_CLEANUP_INTERVAL=10m` to `/etc/ecs/ecs.config`. [More information on optimizing ECS disk cleanup](https://aws.amazon.com/blogs/compute/optimizing-disk-usage-on-amazon-ecs/). +- It is possible to optimize disk clean up on ECS. By default, the unused containers are deleted after 3 hours and the unused images after 30 minutes. These settings can be changed by adding `ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m` and `ECS_IMAGE_CLEANUP_INTERVAL=10m` to `/etc/ecs/ecs.config`. [More information on optimizing ECS disk cleanup](https://aws.amazon.com/blogs/compute/optimizing-disk-usage-on-amazon-ecs/). ### ECS Alternatives and Lock-in @@ -1259,7 +1259,7 @@ Lambda - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. -- 🔸 While adding/removing S3 buckets as triggers for Lambda function, this error may occur: "There was an error creating the trigger: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type." In this case, remove the trigger manually by going to the "Properties" section of the S3 bucket, under "Events", remove the Lambda event. +- 🔸 While adding/removing S3 buckets as triggers for Lambda function, this error may occur: "There was an error creating the trigger: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type." In this case, you can manually remove the Lambda event in the "Events" tab in the "Properties" section of the S3 bucket. - 🔸 At the time of writing (12 December 2016) Dead Letter Queues are only available in the Ohio (us-east-2) region. ### Lambda Code Samples @@ -1618,7 +1618,7 @@ IoT - AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. * **IoT terms:** * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). Device metadata can also be stored in [**IoT Thing Types**](http://docs.aws.amazon.com/iot/latest/developerguide/thing-types.html). This aids in device metadata management by allowing for reuse of device description and configuration for more than one device. Note that IoT Thing Types can be deprecated, but not changed — they are immutable. - * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). You can associate an AWS IoT certificate with an IoT device or you can [register your own CA (Certificate Authority) with AWS](http://docs.aws.amazon.com/iot/latest/developerguide/device-certs-your-own.html), generate your own certificate(s) and associate those certificates with your devices via the AWS Console or cli. + * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). You can associate an AWS IoT certificate with an IoT device or you can [register your own CA (Certificate Authority) with AWS](http://docs.aws.amazon.com/iot/latest/developerguide/device-certs-your-own.html), generate your own certificate(s) and associate those certificates with your devices via the AWS Console or cli. * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). * Shown below is a [diagram](https://aws.amazon.com/iot/how-it-works/) which summarizes the flow of messages between the AWS IoT services: From bd083cdc502bda9094182ce2300f66361d41320a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Riel?= Date: Fri, 3 Mar 2017 14:00:27 -0500 Subject: [PATCH 096/128] Role can now be attached/replaced at runtime See http://stackoverflow.com/questions/35913966/how-to-add-iam-role-to-an-existing-instance-in-aws This is also available via the Console under "Actions >Instance Setting > Attach/Replace IAM role". --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index a01edd6..04f6e79 100644 --- a/README.md +++ b/README.md @@ -790,7 +790,6 @@ EC2 - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. - ❗⏱ Instances come in two types: **Fixed Performance Instances** (e.g. M3, C3, and R3) and [**Burstable Performance Instances**](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. -- 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. - Instance user-data is [limited to 16 KB](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data). (This limit applies to the data in raw form, not base64-encoded form.) If more data is needed, it can be downloaded from S3 by a user-data script. - Very new accounts may not be able to launch some instance types, such as GPU instances, because of an initially imposed “soft limit” of zero. This limit can be raised by making a support request. See [AWS Service Limits](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the method to make the support request. Note that this limit of zero is [not currently documented](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ec2). From 427499ad8108ba23cbc797280c7ae99fc62dd321 Mon Sep 17 00:00:00 2001 From: Bogdan Luput Date: Tue, 7 Mar 2017 08:11:19 +0200 Subject: [PATCH 097/128] Remove the limitation tip about NFSv4 file locking --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index d02d2c8..d00d3eb 100644 --- a/README.md +++ b/README.md @@ -952,8 +952,7 @@ EFS ### EFS Gotchas and Limitations - 🔸 A number of NFSv4.1 features are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html) and there are some [limits](http://docs.aws.amazon.com/efs/latest/ug/limits.html) to the service. -- 🔸 As of 2016-11, EFS does not offer disk level encryption, though it is on the roadmap. -- ❗ Some applications, like SQLite and IPython, [might not work properly](https://sqlite.org/faq.html#q5) on EFS when accessed from multiple clients. This is because lock upgrades and downgrades are [not supported](http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html). There might be [workarounds](https://github.com/ipython/ipython/issues/882) for some issues. +- 🔸 As of 2017-02, EFS does not offer disk level encryption, though it is on the roadmap. - 🔸 An EFS file system [can be mounted on premises](https://aws.amazon.com/efs/faq/#on-premises) over Direct Connect. - 🔸 An EFS file system can NOT be mounted over VPC peering or VPN, even if the VPN is running on top of Direct Connect. - 🔸 Using an EFS volume on Windows is not supported. From 13875709a446c305c4c4b424fc438d2e0eb31395 Mon Sep 17 00:00:00 2001 From: Daniel Ehrlich Date: Sat, 25 Mar 2017 13:20:02 -0500 Subject: [PATCH 098/128] Added in section for File Share and Sync (#387) * Added in section for File Share and Sync The official name for this category is Enterprise File Share and Sync (EFSS). Was not sure where to place it overall so added it at the bottom. All of these services have pretty good APIs, I think there are at least a few cases where devs could be more innovative by using this for their storage instead of typical object storage. * Added open source EFSS * Added Citrix File Share to "other providers" * Table formatting --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d00d3eb..fcb13fd 100644 --- a/README.md +++ b/README.md @@ -309,8 +309,10 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | | Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | | Automatic speech recognition and natural language understanding | Lex | Cloud Speech API, Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | -| Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM Watson |Mimic, eSpeak, MaryTTS | -| Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | +| Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM | Mimic, eSpeak, MaryTTS | +| Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | +| File Share and Sync | WorkDocs | Google Docs | |OneDrive | Dropbox, Box, Citrix File Share |ownCloud | + 🚧 [*Please help fill this table in.*](CONTRIBUTING.md) From f6dae17ff6dbae7481b67ad7aba98539f86cf514 Mon Sep 17 00:00:00 2001 From: Jason Poley Date: Sat, 25 Mar 2017 14:26:50 -0400 Subject: [PATCH 099/128] Update README.md (#388) --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fcb13fd..27baa22 100644 --- a/README.md +++ b/README.md @@ -724,10 +724,10 @@ As an illustration of comparative features and price, the table below gives S3 S | | Durability (per year) | Availability “designed” | Availability SLA | Storage (per TB per month) | GET or retrieve (per million) | Write or archive (per million) | |-----------------|-----------------------|-------------------------|------------------|--------------------------------------------------------------------------------------------------------------------------|-------------------------------|--------------------------------| -| **Glacier** | Eleven 9s | Sloooow | – | $7 | $50 | $50 | +| **Glacier** | Eleven 9s | Sloooow | – | $4 | $50 | $50 | | **S3 IA** | Eleven 9s | 99.9% | **99%** | $12.50 | $1 | $10 | -| **S3 RRS** | **99.99%** | 99.99% | 99.9% | $24 | $0.40 | $5 | -| **S3 Standard** | Eleven 9s | 99.99% | 99.9% | $30 | $0.40 | $5 | +| **S3 RRS** | **99.99%** | 99.99% | 99.9% | $24 (first TB) | $0.40 | $5 | +| **S3 Standard** | Eleven 9s | 99.99% | 99.9% | $23 | $0.40 | $5 | | **EBS** | **99.8%** | Unstated | 99.95% | $25/$45/**$100**/$125+ ([sc1/st1/**gp2**/io1](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)\) | | | | **EFS** | “High” | “High” | – | $300 | | | From ff04f42beb2d728b5adde370c0b9d71f924f6504 Mon Sep 17 00:00:00 2001 From: Chris Roe Date: Sat, 25 Mar 2017 12:32:08 -0600 Subject: [PATCH 100/128] Added new tip for APIG that I got from AWS support (#394) --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 27baa22..1030ced 100644 --- a/README.md +++ b/README.md @@ -1291,7 +1291,8 @@ API Gateway - 🔸API Gateway only supports encrypted (https) endpoints, and does not support unencrypted HTTP. (This is probably a good thing.) - 🔸API Gateway endpoints are always public, i.e. internet facing, and there is no mechanism to build private endpoints, e.g. for internal use on a [VPC](#vpcs-network-security-and-security-groups) but endpoints and their related resources can, optionally, [require authentication](http://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html). - 🔸API Gateway doesn’t support multi-region deployments for high availability. It is a service that is deployed in a single region but comes with a global endpoint that is served from AWS edge locations (similar to a CloudFront distribution). You cannot have multiple API Gateways with the same hostname in different AWS regions and use Route 53 to distribute the traffic. More in [this forum post](https://forums.aws.amazon.com/thread.jspa?messageID=735342򳡮). -- 🔸Integration timeout: All of the various integration types (eg: Lambda, HTTP) for API Gateway have timeouts, as described [here](http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#api-gateway-limits). Unlike some limits, these timeouts can't be increased. +- 🔸Integration timeout: All of the various integration types (eg: Lambda, HTTP) for API Gateway have timeouts, as described [here](http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#api-gateway-limits). Unlike some limits, these timeouts can't be increased. +- 🔸API Gateway returns a 504 status code for any network or low level transport related issue. When this happens, you may see something in the cloudwatch logs for the request that says something like this: `Execution failed due to an internal error`. In reality, this means that even if your backend server is up and running, it may be doing something outside the HTTP specifications (like not sending well-formed chunked messages). Hit your backend directly with the `curl --raw -S -i ` and see if it complains. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 87d740b2adc00b4f1f21aca60cf4d8573f36df87 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Sat, 25 Mar 2017 11:37:23 -0700 Subject: [PATCH 101/128] API Gateway 504 Gotcha Nits (#400) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1030ced..0599567 100644 --- a/README.md +++ b/README.md @@ -1292,7 +1292,7 @@ API Gateway - 🔸API Gateway endpoints are always public, i.e. internet facing, and there is no mechanism to build private endpoints, e.g. for internal use on a [VPC](#vpcs-network-security-and-security-groups) but endpoints and their related resources can, optionally, [require authentication](http://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html). - 🔸API Gateway doesn’t support multi-region deployments for high availability. It is a service that is deployed in a single region but comes with a global endpoint that is served from AWS edge locations (similar to a CloudFront distribution). You cannot have multiple API Gateways with the same hostname in different AWS regions and use Route 53 to distribute the traffic. More in [this forum post](https://forums.aws.amazon.com/thread.jspa?messageID=735342򳡮). - 🔸Integration timeout: All of the various integration types (eg: Lambda, HTTP) for API Gateway have timeouts, as described [here](http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#api-gateway-limits). Unlike some limits, these timeouts can't be increased. -- 🔸API Gateway returns a 504 status code for any network or low level transport related issue. When this happens, you may see something in the cloudwatch logs for the request that says something like this: `Execution failed due to an internal error`. In reality, this means that even if your backend server is up and running, it may be doing something outside the HTTP specifications (like not sending well-formed chunked messages). Hit your backend directly with the `curl --raw -S -i ` and see if it complains. +- 🔸API Gateway returns a 504 status code for any network or low level transport related issue. When this happens, you may see a message in the CloudWatch logs for the request that includes the message: `Execution failed due to an internal error`. One possible reason for this error is that even though your backend server is up and running, it may be doing something outside of the HTTP specification (like not sending well-formed chunked messages). You can test by hitting your backend directly with the `curl --raw -S -i ` and seeing if it complains. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From bb8e3e15126e48f390205753048f317eeeb0a138 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sat, 25 Mar 2017 11:39:38 -0700 Subject: [PATCH 102/128] adding AWS Batch to list of services (#396) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 0599567..66b397a 100644 --- a/README.md +++ b/README.md @@ -330,6 +330,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | 🐥[Lex](https://aws.amazon.com/releasenotes/Amazon-Lex?browse=1) | 2016-11 | Preview | | | 🐥[Polly](https://aws.amazon.com/releasenotes/Amazon-Polly?browse=1) | 2016-11 | General |✓ | | 🐥[Rekognition](https://aws.amazon.com/releasenotes/Amazon-Rekognition?browse=1) | 2016-11 | General |✓ | +| 🐥[Batch](http://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html) | 2016-11 | General |✓ | | 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | | | 🐥[Certificate Manager](https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/) | 2016-01 | General | ✓ | 🐥[IoT](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) | 2015-08 | General | ✓ | From 9a2a58d0fcda61589e63b6248dd7dacaed885533 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Sat, 25 Mar 2017 12:16:05 -0700 Subject: [PATCH 103/128] Update AUTHORS.md (#401) --- AUTHORS.md | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 10d113f..929dac2 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -6,44 +6,49 @@ but the content is the result of a community of contributors, editors, and exper Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for notes on roles and editorial process. -* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [11+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[9+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) — _editor (Certificate Manager)_ +* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [15+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[11+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) — _editor (Certificate Manager)_ * [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) — _expert (IoT)_ -* [Bogdan Luput (bgdnlp)](https://github.com/bgdnlp) — [4+](https://github.com/open-guides/og-aws/commits?author=bgdnlp)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Abgdnlp) — _editor (EFS)_ +* [Bogdan Luput (bgdnlp)](https://github.com/bgdnlp) — [7+](https://github.com/open-guides/og-aws/commits?author=bgdnlp)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Abgdnlp) — _editor (EFS)_ * [Dan Hermann (danhermann)](https://github.com/danhermann) — [4+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[7+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) — _expert (RDS)_ * [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ * [Eric Hammond (ehammond)](https://github.com/ehammond) — _expert (Lambda, serverless)_ * [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ * [Jason Barry (JCBarry)](https://github.com/JCBarry) — [2+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) — _editor (CloudWatch)_ -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [108+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[111+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [112+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[120+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ * [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — [1+](https://github.com/open-guides/og-aws/commits?author=kimschmidtsbrain)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Akimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ * [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — [3+](https://github.com/open-guides/og-aws/commits?author=lukedeo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alukedeo) — _expert (machine learning, Terraform, orchestration)_ -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [23+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[15+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [27+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[19+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [Max Zanko (max-zanko)](https://github.com/max-zanko) — [12+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[10+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Noah Zoschke (nzoschke)](https://github.com/nzoschke) — [2+](https://github.com/open-guides/og-aws/commits?author=nzoschke)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anzoschke) — _editor (KMS)_ -* [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [26+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[27+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Rich Adams (richadams)](https://github.com/richadams) — [2+](https://github.com/open-guides/og-aws/commits?author=richadams)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [28+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[29+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ * [Min (wumintendy)](https://github.com/wumintendy) — _expert (ECS, CloudWatch, SQS, RDS, WAF)_ * [0xmohit](https://github.com/0xmohit) — [3+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) * [Abdul Mohammed (abdulirfan3)](https://github.com/abdulirfan3) — [5+](https://github.com/open-guides/og-aws/commits?author=abdulirfan3)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aabdulirfan3) -* [AJ Kerrigan (ajkerrigan-mdsol)](https://github.com/ajkerrigan-mdsol) — [1+](https://github.com/open-guides/og-aws/commits?author=ajkerrigan-mdsol)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aajkerrigan-mdsol) +* [AJ Kerrigan (ajkerrigan-mdsol)](https://github.com/ajkerrigan-mdsol) — [2+](https://github.com/open-guides/og-aws/commits?author=ajkerrigan-mdsol)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aajkerrigan-mdsol) * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) +* [Aron Woost (aronwoost)](https://github.com/aronwoost) — [1+](https://github.com/open-guides/og-aws/commits?author=aronwoost)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aaronwoost) * [Bo Bayles (bbayles)](https://github.com/bbayles) — [1+](https://github.com/open-guides/og-aws/commits?author=bbayles)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abbayles) * [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk) * [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer) * [Jesper Eneberg (Bizzelicious)](https://github.com/Bizzelicious) — [3+](https://github.com/open-guides/og-aws/commits?author=Bizzelicious)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3ABizzelicious) * [Bradly Feeley (bradly)](https://github.com/bradly) — [2+](https://github.com/open-guides/og-aws/commits?author=bradly)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Abradly) * [Brady Dowling (bradydowling)](https://github.com/bradydowling) — [1+](https://github.com/open-guides/og-aws/commits?author=bradydowling)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abradydowling) +* [Benjamin Wilson (brwilson)](https://github.com/brwilson) — [4+](https://github.com/open-guides/og-aws/commits?author=brwilson)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abrwilson) * [Mårten Gustafson (chids)](https://github.com/chids) — [4+](https://github.com/open-guides/og-aws/commits?author=chids)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Achids) * [Chris Griffin (chris-griffin)](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) * [Chris Lennon (chrislennon)](https://github.com/chrislennon) — [1+](https://github.com/open-guides/og-aws/commits?author=chrislennon)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrislennon) * [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) * [Rainer Eli (claushellsing)](https://github.com/claushellsing) — [1+](https://github.com/open-guides/og-aws/commits?author=claushellsing)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aclaushellsing) +* [Joe Greene (ClydeMachine)](https://github.com/ClydeMachine) — [10+](https://github.com/open-guides/og-aws/commits?author=ClydeMachine)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3AClydeMachine) +* [Cristian Măgherușan-Stanciu @magheru_san (cristim)](https://github.com/cristim) — [1+](https://github.com/open-guides/og-aws/commits?author=cristim)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Acristim) * [Dan Bower (danbower)](https://github.com/danbower) — [2+](https://github.com/open-guides/og-aws/commits?author=danbower)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanbower) +* [Daniel Ehrlich (danielehrlich)](https://github.com/danielehrlich) — [3+](https://github.com/open-guides/og-aws/commits?author=danielehrlich)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Adanielehrlich) * [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) -* [Dorrin (DorrinPk)](https://github.com/DorrinPk) — [2+](https://github.com/open-guides/og-aws/commits?author=DorrinPk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3ADorrinPk) +* [Dorrin (DorrinPk)](https://github.com/DorrinPk) — [5+](https://github.com/open-guides/og-aws/commits?author=DorrinPk)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3ADorrinPk) * [Matthew Lapworth (dragonndev)](https://github.com/dragonndev) — [5+](https://github.com/open-guides/og-aws/commits?author=dragonndev)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adragonndev) * [Dmitry Guyvoronsky (dreamiurg)](https://github.com/dreamiurg) — [1+](https://github.com/open-guides/og-aws/commits?author=dreamiurg)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adreamiurg) * [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy) @@ -55,6 +60,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [gscalise](https://github.com/gscalise) — [1+](https://github.com/open-guides/og-aws/commits?author=gscalise)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agscalise) * [Gulam Shakir (gshakir)](https://github.com/gshakir) — [2+](https://github.com/open-guides/og-aws/commits?author=gshakir)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agshakir) * [Theo Hultberg (iconara)](https://github.com/iconara) — [1+](https://github.com/open-guides/og-aws/commits?author=iconara)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aiconara) +* [Ben Bridts (ikben)](https://github.com/ikben) — [2+](https://github.com/open-guides/og-aws/commits?author=ikben)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aikben) * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) * [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) * [Jose Luis Ordiales Coscia (jlordiales)](https://github.com/jlordiales) — [1+](https://github.com/open-guides/og-aws/commits?author=jlordiales)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlordiales) @@ -63,11 +69,13 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) — [3+](https://github.com/open-guides/og-aws/commits?author=kazuyukitanimura)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akazuyukitanimura) * [Kyle Busekist (kbusekist)](https://github.com/kbusekist) — [2+](https://github.com/open-guides/og-aws/commits?author=kbusekist)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akbusekist) * [Krishan (KrishanBhasin)](https://github.com/KrishanBhasin) — [3+](https://github.com/open-guides/og-aws/commits?author=KrishanBhasin)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3AKrishanBhasin) +* [Laurent Jalbert-Simard (laurrentt)](https://github.com/laurrentt) — [1+](https://github.com/open-guides/og-aws/commits?author=laurrentt)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Alaurrentt) * [Łukasz Kostka (luqasz)](https://github.com/luqasz) — [4+](https://github.com/open-guides/og-aws/commits?author=luqasz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aluqasz) * [maiki](https://github.com/maiki) — [1+](https://github.com/open-guides/og-aws/commits?author=maiki)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amaiki) * [Manoj M J (manojmj92)](https://github.com/manojmj92) — [1+](https://github.com/open-guides/og-aws/commits?author=manojmj92)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amanojmj92) * [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d) * [Martijn Dwars (MartijnDwars)](https://github.com/MartijnDwars) — [3+](https://github.com/open-guides/og-aws/commits?author=MartijnDwars)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AMartijnDwars) +* [Mike (mikejennings)](https://github.com/mikejennings) — [1+](https://github.com/open-guides/og-aws/commits?author=mikejennings)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amikejennings) * [Mikhail Advani (mikhailadvani)](https://github.com/mikhailadvani) — [5+](https://github.com/open-guides/og-aws/commits?author=mikhailadvani)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Amikhailadvani) * [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke) * [Mathias Leppich (muhqu)](https://github.com/muhqu) — [3+](https://github.com/open-guides/og-aws/commits?author=muhqu)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Amuhqu) @@ -77,8 +85,10 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Pascal Borreli (pborreli)](https://github.com/pborreli) — [1+](https://github.com/open-guides/og-aws/commits?author=pborreli)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apborreli) * [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) * Praveen Patnala +* [Corey Quinn (QuinnyPig)](https://github.com/QuinnyPig) — [3+](https://github.com/open-guides/og-aws/commits?author=QuinnyPig)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3AQuinnyPig) * [Raajhesh Kannaa Chidambaram (raajheshkannaa)](https://github.com/raajheshkannaa) — [1+](https://github.com/open-guides/og-aws/commits?author=raajheshkannaa)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Araajheshkannaa) * [Ram Nadella (ram-nadella)](https://github.com/ram-nadella) — [1+](https://github.com/open-guides/og-aws/commits?author=ram-nadella)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aram-nadella) +* [Raphaël Riel (raphael-riel)](https://github.com/raphael-riel) — [1+](https://github.com/open-guides/og-aws/commits?author=raphael-riel)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Araphael-riel) * [Richard Birkby (rbirkby)](https://github.com/rbirkby) — [2+](https://github.com/open-guides/og-aws/commits?author=rbirkby)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arbirkby) * [Federico Hernandez (recollir)](https://github.com/recollir) — [1+](https://github.com/open-guides/og-aws/commits?author=recollir)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arecollir) * [Rich Hintz (rjhintz)](https://github.com/rjhintz) — [11+](https://github.com/open-guides/og-aws/commits?author=rjhintz)/[13+](https://github.com/open-guides/og-aws/issues?q=author%3Arjhintz) @@ -86,6 +96,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [David Schott (schottsfired)](https://github.com/schottsfired) — [1+](https://github.com/open-guides/og-aws/commits?author=schottsfired)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aschottsfired) * [Ashley Davis (SgtCoDFish)](https://github.com/SgtCoDFish) — [1+](https://github.com/open-guides/og-aws/commits?author=SgtCoDFish)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3ASgtCoDFish) * [Shubham Aggarwal (shubham391)](https://github.com/shubham391) — [1+](https://github.com/open-guides/og-aws/commits?author=shubham391)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashubham391) +* [Seth Rutner (sprutner)](https://github.com/sprutner) — [1+](https://github.com/open-guides/og-aws/commits?author=sprutner)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Asprutner) * [Adam Nelson (spudstuff)](https://github.com/spudstuff) — [4+](https://github.com/open-guides/og-aws/commits?author=spudstuff)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aspudstuff) * [Steven Maude (StevenMaude)](https://github.com/StevenMaude) — [1+](https://github.com/open-guides/og-aws/commits?author=StevenMaude)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AStevenMaude) * [Carlos (theist)](https://github.com/theist) — [1+](https://github.com/open-guides/og-aws/commits?author=theist)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Atheist) @@ -107,4 +118,4 @@ Please let the project leads know of any errors or omissions on this list. -(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) \ No newline at end of file +(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) From 50e42190fd19d5167cea23882efa4996adb3390e Mon Sep 17 00:00:00 2001 From: David Kosslyn Date: Wed, 29 Mar 2017 15:49:04 -0700 Subject: [PATCH 104/128] NEW: added additional limitation for lambda (#406) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 66b397a..59ab266 100644 --- a/README.md +++ b/README.md @@ -1262,6 +1262,7 @@ Lambda - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. - 🔸 While adding/removing S3 buckets as triggers for Lambda function, this error may occur: "There was an error creating the trigger: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type." In this case, you can manually remove the Lambda event in the "Events" tab in the "Properties" section of the S3 bucket. - 🔸 At the time of writing (12 December 2016) Dead Letter Queues are only available in the Ohio (us-east-2) region. +- 🔸 Lambda [resource limits](http://docs.aws.amazon.com/lambda/latest/dg/limits.html) include a max 6MB request or response payload size. ### Lambda Code Samples From ed5af7c7ce157546e2708c670897659e42ff647a Mon Sep 17 00:00:00 2001 From: Suny Kim Date: Fri, 31 Mar 2017 00:08:03 +0200 Subject: [PATCH 105/128] Elastic's own cloud offering is important (#398) This should be on the list: https://www.elastic.co/de/cloud/as-a-service (formerly called "Found", a company that was bought by elastic in 2015). You get the latest versions, Kibana, custom plugins and scripts and dictionaries. Also, you can access the commercial x-pack products (Shield / Marvel / Watcher / Graph / Reporting), also in the free 14 days version. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 59ab266..87d0b84 100644 --- a/README.md +++ b/README.md @@ -290,7 +290,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Bigtable | | Cloud Bigtable | Bigtable | | | HBase | | Key-value store, column store | DynamoDB | Cloud Datastore | Megastore | Tables, DocumentDB | | Cassandra, CouchDB, RethinkDB, Redis | | Memory cache | ElastiCache | App Engine Memcache | | Redis Cache | | Memcached, Redis | -| Search | CloudSearch, Elasticsearch (managed) | | | Search | Algolia, QBox | Elasticsearch, Solr | +| Search | CloudSearch, Elasticsearch (managed) | | | Search | Algolia, QBox, Elastic Cloud | Elasticsearch, Solr | | Data warehouse | Redshift | BigQuery | Dremel | SQL Data Warehouse | Oracle, IBM, SAP, HP, many others | Greenplum | | Business intelligence | QuickSight | Data Studio 360 | | Power BI | Tableau | | | Lock manager | [DynamoDB (weak)](https://gist.github.com/ryandotsmith/c95fd21fab91b0823328) | | Chubby | Lease blobs in Storage Account | | ZooKeeper, Etcd, Consul | From ab170ed7e9774e7d9ea9268f43d053cf38f80f43 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Thu, 30 Mar 2017 21:32:13 -0700 Subject: [PATCH 106/128] Adding Athena to the list of new services --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 87d0b84..fef7ae2 100644 --- a/README.md +++ b/README.md @@ -330,6 +330,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | 🐥[Lex](https://aws.amazon.com/releasenotes/Amazon-Lex?browse=1) | 2016-11 | Preview | | | 🐥[Polly](https://aws.amazon.com/releasenotes/Amazon-Polly?browse=1) | 2016-11 | General |✓ | | 🐥[Rekognition](https://aws.amazon.com/releasenotes/Amazon-Rekognition?browse=1) | 2016-11 | General |✓ | +| 🐥[Athena](http://docs.aws.amazon.com/athena/latest/ug/what-is.html) | 2016-11 | General | | | 🐥[Batch](http://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html) | 2016-11 | General |✓ | | 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | | | 🐥[Certificate Manager](https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/) | 2016-01 | General | ✓ From c83b8cd936af2f6b18c0fcc85409224f8e445f61 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Sun, 2 Apr 2017 11:35:08 -0700 Subject: [PATCH 107/128] Update revenue for 2016 fiscal year --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 87d0b84..ccd786a 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ General Information - In general, “[cloud computing](https://en.wikipedia.org/wiki/Cloud_computing)” can refer to one of three types of cloud: “public,” “private,” and “hybrid.” AWS is a public cloud provider, since anyone can use it. Private clouds are within a single (usually large) organization. Many companies use a hybrid of private and public clouds. - The core features of AWS are [infrastructure-as-a-service](https://en.wikipedia.org/wiki/Cloud_computing#Infrastructure_as_a_service_.28IaaS.29) (IaaS) — that is, virtual machines and supporting infrastructure. Other cloud service models include [platform-as-a-service](https://en.wikipedia.org/wiki/Cloud_computing#Platform_as_a_service_.28PaaS.29) (PaaS), which typically are more fully managed services that deploy customers’ applications, or [software-as-a-service](https://en.wikipedia.org/wiki/Cloud_computing#Software_as_a_service_.28SaaS.29) (SaaS), which are cloud-based applications. AWS does offer a few products that fit into these other models, too. - In business terms, with infrastructure-as-a-service you have a variable cost model — it is [OpEx, not CapEx](http://www.investopedia.com/ask/answers/020915/what-difference-between-capex-and-opex.asp) (though some [pre-purchased contracts](https://aws.amazon.com/ec2/purchasing-options/reserved-instances/) are still CapEx). -- AWS’s annual revenue was [**$7.88 billion**](http://phx.corporate-ir.net/phoenix.zhtml?c=97664&p=irol-sec&control_selectgroup=Annual%20Filings) as of 2015 according to their SEC 10-K filing, or roughly **7%** of Amazon.com’s total 2015 revenue. +- AWS’s annual revenue was [**$12.21 billion**](http://services.corporate-ir.net/SEC.Enhanced/SecCapsule.aspx?c=97664&fid=14806946) as of 2016 according to their SEC 10-K filing, or roughly **8.9%** of Amazon.com’s total 2016 revenue. - **Main reasons to use AWS:** - If your company is building systems or products that may need to scale - and you have technical know-how From 4d94996a149602996b502178f676d264eadcae6b Mon Sep 17 00:00:00 2001 From: apelsinet Date: Thu, 6 Apr 2017 22:38:03 +0200 Subject: [PATCH 108/128] Remove duplicate paragraph. --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index fef7ae2..8210748 100644 --- a/README.md +++ b/README.md @@ -1391,7 +1391,6 @@ VPCs, Network Security, and Security Groups - **VPC Flow Logs** allow you to monitor the network traffic to, from, and within your VPC. Logs are stored in CloudWatch Logs groups, and can be used for security monitoring (with third party tools), performance evaluation, and forensic investigation. - See the [VPC Flow Logs User Guide](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) for basic information. - See the [flowlogs-reader](https://github.com/obsrvbl/flowlogs-reader) CLI tool and Python library to retrieve and work with VPC Flow Logs. -- **IPv6** [is available in VPC.](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/) Along with this announcement came the introduction of the Egress-Only Internet Gateway. In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. - **IPv6** [is available in VPC](https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-instances-in-virtual-private-clouds/). Along with this announcement came the introduction of the [Egress-Only Internet Gateway](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/egress-only-internet-gateway.html). In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6. - Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already. From 7929c80be6acef7731a1a483bc9f6aaa82bc533b Mon Sep 17 00:00:00 2001 From: Mikael Finstad Date: Thu, 13 Apr 2017 03:27:17 +0200 Subject: [PATCH 109/128] Add limitiation about missing gzip (#397) --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e1c4556..0171c5e 100644 --- a/README.md +++ b/README.md @@ -310,7 +310,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | | Automatic speech recognition and natural language understanding | Lex | Cloud Speech API, Natural Language API | | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API |Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | | Text-to-speech engine in the cloud | Polly | | | |Nuance, Vocalware, IBM | Mimic, eSpeak, MaryTTS | -| Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | +| Image recognition | Rekognition | Vision API | |Cognitive services | IBM Watson, Clarifai |TensorFlow, OpenCV | | File Share and Sync | WorkDocs | Google Docs | |OneDrive | Dropbox, Box, Citrix File Share |ownCloud | @@ -1296,6 +1296,7 @@ API Gateway - 🔸API Gateway doesn’t support multi-region deployments for high availability. It is a service that is deployed in a single region but comes with a global endpoint that is served from AWS edge locations (similar to a CloudFront distribution). You cannot have multiple API Gateways with the same hostname in different AWS regions and use Route 53 to distribute the traffic. More in [this forum post](https://forums.aws.amazon.com/thread.jspa?messageID=735342򳡮). - 🔸Integration timeout: All of the various integration types (eg: Lambda, HTTP) for API Gateway have timeouts, as described [here](http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#api-gateway-limits). Unlike some limits, these timeouts can't be increased. - 🔸API Gateway returns a 504 status code for any network or low level transport related issue. When this happens, you may see a message in the CloudWatch logs for the request that includes the message: `Execution failed due to an internal error`. One possible reason for this error is that even though your backend server is up and running, it may be doing something outside of the HTTP specification (like not sending well-formed chunked messages). You can test by hitting your backend directly with the `curl --raw -S -i ` and seeing if it complains. +- 🔸API Gateway does not support gzip compression of responses. See [AWS forum](https://forums.aws.amazon.com/thread.jspa?threadID=192948). 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From e0e0bcca051820cb79f6dfee95534589680fb942 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Wed, 12 Apr 2017 18:29:01 -0700 Subject: [PATCH 110/128] Sunsetting S3 Reduced Redundancy (#412) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0171c5e..e43e254 100644 --- a/README.md +++ b/README.md @@ -642,7 +642,7 @@ S3 - For transfer, putting data into AWS is free, but you’ll pay on the way out. Transfer from S3 to EC2 in the *same region* is free. Transfer to other regions or the Internet in general is not free. - Deletes are free. - **S3 Reduced Redundancy and Infrequent Access:** Most people use the Standard storage class in S3, but there are other storage classes with lower cost: - - [Reduced Redundancy Storage (RRS)](https://aws.amazon.com/s3/reduced-redundancy/) has lower durability (99.99%, so just four nines). That is, there’s a small chance you’ll lose data. For some data sets where data has value in a statistical way (losing say half a percent of your objects isn’t a big deal) this is a reasonable trade-off. + - 🔸[Reduced Redundancy Storage (RRS)](https://aws.amazon.com/s3/reduced-redundancy/) has been effectively deprecated, and has lower durability (99.99%, so just four nines) than standard S3. Note that it no longer participates in S3 price reductions, so it offers worse redundancy for more money than standard S3. As a result, there's no reason to use it. - [Infrequent Access (IA)](https://aws.amazon.com/s3/storage-classes/#Infrequent_Access) lets you get cheaper storage in exchange for more expensive access. This is great for archives like logs you already processed, but might want to look at later. To get an idea of the cost savings when using Infrequent Access (IA), you can use this [S3 Infrequent Access Calculator](http://www.gulamshakir.com/apps/s3calc/index.html). - [Glacier](#glacier) is a third alternative discussed as a separate product. - See [the comparison table](#storage-durability-availability-and-price). From aa8df2945fd750aa5947d5d9f6903982f0b872ee Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Thu, 13 Apr 2017 13:36:02 -0700 Subject: [PATCH 111/128] Update authors. --- AUTHORS.md | 15 ++++++++++----- admin/authors-info.yml | 1 + 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 929dac2..032b5c0 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -17,19 +17,21 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Joshua Levy (jlevy)](https://github.com/jlevy) — [112+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[120+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ * [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — [1+](https://github.com/open-guides/og-aws/commits?author=kimschmidtsbrain)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Akimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ * [Luke de Oliveira (lukedeo)](https://github.com/lukedeo) — [3+](https://github.com/open-guides/og-aws/commits?author=lukedeo)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alukedeo) — _expert (machine learning, Terraform, orchestration)_ -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [27+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[19+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [28+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[21+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [Max Zanko (max-zanko)](https://github.com/max-zanko) — [12+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[10+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Noah Zoschke (nzoschke)](https://github.com/nzoschke) — [2+](https://github.com/open-guides/og-aws/commits?author=nzoschke)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anzoschke) — _editor (KMS)_ +* [Corey Quinn (QuinnyPig)](https://github.com/QuinnyPig) — [5+](https://github.com/open-guides/og-aws/commits?author=QuinnyPig)/[7+](https://github.com/open-guides/og-aws/issues?q=author%3AQuinnyPig) — _editor (news and updates, cost management)_ * [Rich Adams (richadams)](https://github.com/richadams) — [2+](https://github.com/open-guides/og-aws/commits?author=richadams)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [28+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[29+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [29+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[30+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ * [Min (wumintendy)](https://github.com/wumintendy) — _expert (ECS, CloudWatch, SQS, RDS, WAF)_ * [0xmohit](https://github.com/0xmohit) — [3+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) * [Abdul Mohammed (abdulirfan3)](https://github.com/abdulirfan3) — [5+](https://github.com/open-guides/og-aws/commits?author=abdulirfan3)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aabdulirfan3) * [AJ Kerrigan (ajkerrigan-mdsol)](https://github.com/ajkerrigan-mdsol) — [2+](https://github.com/open-guides/og-aws/commits?author=ajkerrigan-mdsol)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aajkerrigan-mdsol) * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) +* [Mattias (apelsinet)](https://github.com/apelsinet) — [1+](https://github.com/open-guides/og-aws/commits?author=apelsinet)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aapelsinet) * [Aron Woost (aronwoost)](https://github.com/aronwoost) — [1+](https://github.com/open-guides/og-aws/commits?author=aronwoost)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aaronwoost) * [Bo Bayles (bbayles)](https://github.com/bbayles) — [1+](https://github.com/open-guides/og-aws/commits?author=bbayles)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abbayles) * [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk) @@ -47,6 +49,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Cristian Măgherușan-Stanciu @magheru_san (cristim)](https://github.com/cristim) — [1+](https://github.com/open-guides/og-aws/commits?author=cristim)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Acristim) * [Dan Bower (danbower)](https://github.com/danbower) — [2+](https://github.com/open-guides/og-aws/commits?author=danbower)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanbower) * [Daniel Ehrlich (danielehrlich)](https://github.com/danielehrlich) — [3+](https://github.com/open-guides/og-aws/commits?author=danielehrlich)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Adanielehrlich) +* [Chris Roe (dijitalmunky)](https://github.com/dijitalmunky) — [1+](https://github.com/open-guides/og-aws/commits?author=dijitalmunky)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adijitalmunky) * [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) * [Dorrin (DorrinPk)](https://github.com/DorrinPk) — [5+](https://github.com/open-guides/og-aws/commits?author=DorrinPk)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3ADorrinPk) * [Matthew Lapworth (dragonndev)](https://github.com/dragonndev) — [5+](https://github.com/open-guides/og-aws/commits?author=dragonndev)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adragonndev) @@ -62,12 +65,13 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Theo Hultberg (iconara)](https://github.com/iconara) — [1+](https://github.com/open-guides/og-aws/commits?author=iconara)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aiconara) * [Ben Bridts (ikben)](https://github.com/ikben) — [2+](https://github.com/open-guides/og-aws/commits?author=ikben)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aikben) * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) -* [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) * [Jose Luis Ordiales Coscia (jlordiales)](https://github.com/jlordiales) — [1+](https://github.com/open-guides/og-aws/commits?author=jlordiales)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlordiales) * [Jaanus Torp (jnsaff)](https://github.com/jnsaff) — [1+](https://github.com/open-guides/og-aws/commits?author=jnsaff)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajnsaff) +* [Jason Poley (jpoley)](https://github.com/jpoley) — [1+](https://github.com/open-guides/og-aws/commits?author=jpoley)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Ajpoley) * Jurgen Philippaerts * [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) — [3+](https://github.com/open-guides/og-aws/commits?author=kazuyukitanimura)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akazuyukitanimura) * [Kyle Busekist (kbusekist)](https://github.com/kbusekist) — [2+](https://github.com/open-guides/og-aws/commits?author=kbusekist)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akbusekist) +* [David Kosslyn (kozzles)](https://github.com/kozzles) — [1+](https://github.com/open-guides/og-aws/commits?author=kozzles)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akozzles) * [Krishan (KrishanBhasin)](https://github.com/KrishanBhasin) — [3+](https://github.com/open-guides/og-aws/commits?author=KrishanBhasin)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3AKrishanBhasin) * [Laurent Jalbert-Simard (laurrentt)](https://github.com/laurrentt) — [1+](https://github.com/open-guides/og-aws/commits?author=laurrentt)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Alaurrentt) * [Łukasz Kostka (luqasz)](https://github.com/luqasz) — [4+](https://github.com/open-guides/og-aws/commits?author=luqasz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aluqasz) @@ -75,6 +79,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Manoj M J (manojmj92)](https://github.com/manojmj92) — [1+](https://github.com/open-guides/og-aws/commits?author=manojmj92)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amanojmj92) * [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d) * [Martijn Dwars (MartijnDwars)](https://github.com/MartijnDwars) — [3+](https://github.com/open-guides/og-aws/commits?author=MartijnDwars)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AMartijnDwars) +* [Mikael Finstad (mifi)](https://github.com/mifi) — [1+](https://github.com/open-guides/og-aws/commits?author=mifi)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amifi) * [Mike (mikejennings)](https://github.com/mikejennings) — [1+](https://github.com/open-guides/og-aws/commits?author=mikejennings)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amikejennings) * [Mikhail Advani (mikhailadvani)](https://github.com/mikhailadvani) — [5+](https://github.com/open-guides/og-aws/commits?author=mikhailadvani)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Amikhailadvani) * [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke) @@ -85,7 +90,6 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Pascal Borreli (pborreli)](https://github.com/pborreli) — [1+](https://github.com/open-guides/og-aws/commits?author=pborreli)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apborreli) * [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) * Praveen Patnala -* [Corey Quinn (QuinnyPig)](https://github.com/QuinnyPig) — [3+](https://github.com/open-guides/og-aws/commits?author=QuinnyPig)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3AQuinnyPig) * [Raajhesh Kannaa Chidambaram (raajheshkannaa)](https://github.com/raajheshkannaa) — [1+](https://github.com/open-guides/og-aws/commits?author=raajheshkannaa)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Araajheshkannaa) * [Ram Nadella (ram-nadella)](https://github.com/ram-nadella) — [1+](https://github.com/open-guides/og-aws/commits?author=ram-nadella)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aram-nadella) * [Raphaël Riel (raphael-riel)](https://github.com/raphael-riel) — [1+](https://github.com/open-guides/og-aws/commits?author=raphael-riel)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Araphael-riel) @@ -99,6 +103,7 @@ Please help if you can. See the [contribution guidelines](CONTRIBUTING.md) for n * [Seth Rutner (sprutner)](https://github.com/sprutner) — [1+](https://github.com/open-guides/og-aws/commits?author=sprutner)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Asprutner) * [Adam Nelson (spudstuff)](https://github.com/spudstuff) — [4+](https://github.com/open-guides/og-aws/commits?author=spudstuff)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aspudstuff) * [Steven Maude (StevenMaude)](https://github.com/StevenMaude) — [1+](https://github.com/open-guides/og-aws/commits?author=StevenMaude)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AStevenMaude) +* [Suny Kim (sunygit)](https://github.com/sunygit) — [1+](https://github.com/open-guides/og-aws/commits?author=sunygit)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Asunygit) * [Carlos (theist)](https://github.com/theist) — [1+](https://github.com/open-guides/og-aws/commits?author=theist)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Atheist) * [TheRandomSecurityGuy (therandomsecurityguy)](https://github.com/therandomsecurityguy) — [1+](https://github.com/open-guides/og-aws/commits?author=therandomsecurityguy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atherandomsecurityguy) * [Sandeep Dinesh (thesandlord)](https://github.com/thesandlord) — [1+](https://github.com/open-guides/og-aws/commits?author=thesandlord)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Athesandlord) @@ -118,4 +123,4 @@ Please let the project leads know of any errors or omissions on this list. -(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) +(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) \ No newline at end of file diff --git a/admin/authors-info.yml b/admin/authors-info.yml index f813912..92002ca 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -18,6 +18,7 @@ exclude: roles: jlevy: project lead, editor (topics not otherwise assigned) ThanosBaskous: project lead, editor (topics not otherwise assigned) + QuinnyPig: editor (news and updates, cost management) max-zanko: editor (EC2, S3, Glacier, EMR, Redshift) nitingithub: editor (cost management) forwidur: editor (EBS, RDS) From 39356f8751aa57918c6d20b12df39f2659db4a58 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Thu, 13 Apr 2017 16:25:18 -0700 Subject: [PATCH 112/128] Strikethrough test (#415) Fix strikethrough on table Deprecation note on RRS --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c8bb082..a8cced5 100644 --- a/README.md +++ b/README.md @@ -642,7 +642,7 @@ S3 - For transfer, putting data into AWS is free, but you’ll pay on the way out. Transfer from S3 to EC2 in the *same region* is free. Transfer to other regions or the Internet in general is not free. - Deletes are free. - **S3 Reduced Redundancy and Infrequent Access:** Most people use the Standard storage class in S3, but there are other storage classes with lower cost: - - 🔸[Reduced Redundancy Storage (RRS)](https://aws.amazon.com/s3/reduced-redundancy/) has been effectively deprecated, and has lower durability (99.99%, so just four nines) than standard S3. Note that it no longer participates in S3 price reductions, so it offers worse redundancy for more money than standard S3. As a result, there's no reason to use it. + - 🔸[Reduced Redundancy Storage (RRS)](https://aws.amazon.com/s3/reduced-redundancy/) has been [effectively deprecated](https://www.quinnadvisory.com/blog/2017/4/13/reduced-redundancy-s3-is-dead), and has lower durability (99.99%, so just four nines) than standard S3. Note that it no longer participates in S3 price reductions, so it offers worse redundancy for more money than standard S3. As a result, there's no reason to use it. - [Infrequent Access (IA)](https://aws.amazon.com/s3/storage-classes/#Infrequent_Access) lets you get cheaper storage in exchange for more expensive access. This is great for archives like logs you already processed, but might want to look at later. To get an idea of the cost savings when using Infrequent Access (IA), you can use this [S3 Infrequent Access Calculator](http://www.gulamshakir.com/apps/s3calc/index.html). - [Glacier](#glacier) is a third alternative discussed as a separate product. - See [the comparison table](#storage-durability-availability-and-price). @@ -728,12 +728,12 @@ As an illustration of comparative features and price, the table below gives S3 S |-----------------|-----------------------|-------------------------|------------------|--------------------------------------------------------------------------------------------------------------------------|-------------------------------|--------------------------------| | **Glacier** | Eleven 9s | Sloooow | – | $4 | $50 | $50 | | **S3 IA** | Eleven 9s | 99.9% | **99%** | $12.50 | $1 | $10 | -| **S3 RRS** | **99.99%** | 99.99% | 99.9% | $24 (first TB) | $0.40 | $5 | +| ~~**S3 RRS**~~ | ~~**99.99%**~~ | ~~99.99%~~ | ~~99.9%~~ | ~~$24 (first TB)~~ | ~~$0.40~~ | ~~$5~~ | | **S3 Standard** | Eleven 9s | 99.99% | 99.9% | $23 | $0.40 | $5 | | **EBS** | **99.8%** | Unstated | 99.95% | $25/$45/**$100**/$125+ ([sc1/st1/**gp2**/io1](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)\) | | | | **EFS** | “High” | “High” | – | $300 | | | -Especially notable items are in **boldface**. Sources: [S3 pricing](https://aws.amazon.com/s3/pricing/), [S3 SLA](https://aws.amazon.com/s3/sla/), [S3 FAQ](https://aws.amazon.com/s3/faqs/), [RRS info](https://aws.amazon.com/s3/reduced-redundancy/), [Glacier pricing](https://aws.amazon.com/glacier/pricing/), [EBS availability and durability](https://aws.amazon.com/ebs/details/#Amazon_EBS_Availability_and_Durability), [EBS pricing](https://aws.amazon.com/ebs/pricing/), [EFS pricing](https://aws.amazon.com/efs/pricing/), [EC2 SLA](https://aws.amazon.com/ec2/sla/) +Especially notable items are in **boldface**. Sources: [S3 pricing](https://aws.amazon.com/s3/pricing/), [S3 SLA](https://aws.amazon.com/s3/sla/), [S3 FAQ](https://aws.amazon.com/s3/faqs/), [RRS info](https://aws.amazon.com/s3/reduced-redundancy/) (note that this is considered deprecated), [Glacier pricing](https://aws.amazon.com/glacier/pricing/), [EBS availability and durability](https://aws.amazon.com/ebs/details/#Amazon_EBS_Availability_and_Durability), [EBS pricing](https://aws.amazon.com/ebs/pricing/), [EFS pricing](https://aws.amazon.com/efs/pricing/), [EC2 SLA](https://aws.amazon.com/ec2/sla/) EC2 --- From c607c0e84653adf2f8466b11c45b9e240f170328 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Thu, 13 Apr 2017 16:47:58 -0700 Subject: [PATCH 113/128] Flesh out tagging --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index a8cced5..9d27860 100644 --- a/README.md +++ b/README.md @@ -494,6 +494,8 @@ So if you’re not going to manage your AWS configurations manually, what should - To label lifecycles, such as temporary resources or one that should be deprovisioned in the future - To distinguish production-critical infrastructure (e.g. serving systems vs backend pipelines) - To distinguish resources with special security or compliance requirements +- 🔸For many years, there was a notorious 10 tag limit per resource, which could not be raised and caused many companies significant pain. As of 2016, this was [raised](https://aws.amazon.com/blogs/security/now-organize-your-aws-resources-by-using-up-to-50-tags-per-resource/) to 50 tags per resource. +- In 2017, AWS introduced the ability to [enforce tagging](https://aws.amazon.com/blogs/aws/new-tag-ec2-instances-ebs-volumes-on-creation/) on instance and volume creation, deprecating portions of third party tools such as [Cloud Custodian](https://github.com/capitalone/cloud-custodian). Managing Servers and Applications --------------------------------- From ee35459923bc0dbf207fed966e8a63bcc9d0bc7f Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Thu, 13 Apr 2017 18:02:54 -0700 Subject: [PATCH 114/128] A few grammatical snags (#416) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index a8cced5..24caa5f 100644 --- a/README.md +++ b/README.md @@ -469,8 +469,8 @@ So if you’re not going to manage your AWS configurations manually, what should - The [**aws command-line interface**](https://aws.amazon.com/cli/) (CLI), used via the **aws** command, is the most basic way to save and automate AWS operations. - Don’t underestimate its power. It also has the advantage of being well-maintained — it covers a large proportion of all AWS services, and is up to date. - In general, whenever you can, prefer the command line to the AWS Console for performing operations. -- 🔹Even in absence of fancier tools, you can **write simple Bash scripts** that invoke *aws* with specific arguments, and check these into Git. This is a primitive but effective way to document operations you’ve performed. It improves automation, allows code review and sharing on a team, and gives others a starting point for future work. -- 🔹For use that is primarily interactive, and not scripted, consider instead using the [**aws-shell**](https://github.com/awslabs/aws-shell) tool from AWS. It is easier to use, with auto-completion and a colorful UI, but still works on the command line. If you’re using [SAWS](https://github.com/donnemartin/saws), a previous version of the program, [you should migrate to aws-shell](https://github.com/donnemartin/saws/issues/68#issuecomment-240067034). +- 🔹Even in the absence of fancier tools, you can **write simple Bash scripts** that invoke *aws* with specific arguments, and check these into Git. This is a primitive but effective way to document operations you’ve performed. It improves automation, allows code review and sharing on a team, and gives others a starting point for future work. +- 🔹For use that is primarily interactive (not scripted), consider instead using the [**aws-shell**](https://github.com/awslabs/aws-shell) tool from AWS. It is easier to use, with auto-completion and a colorful UI, but still works on the command line. If you’re using [SAWS](https://github.com/donnemartin/saws), a previous version of the program, [you should migrate to aws-shell](https://github.com/donnemartin/saws/issues/68#issuecomment-240067034). ### APIs and SDKs From 6a339ca38f0320ffebf5368385d3d0fe0ee50d4d Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Fri, 14 Apr 2017 17:05:46 -0700 Subject: [PATCH 115/128] Reframe warning about versioning's shortcomings compared to git (#418) * Reframe warning about versioning's shortcomings compared to git * Dropped 'of' accidentally --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 24caa5f..ae52861 100644 --- a/README.md +++ b/README.md @@ -636,7 +636,7 @@ S3 - Bucket names can be used as part of the hostname when accessing the bucket or its contents, like `.s3-us-east-1.amazonaws.com`, as long as the name is [DNS compliant](http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html). - A common practice is to use the company name acronym or abbreviation to prefix (or suffix, if you prefer DNS-style hierarchy) all bucket names (but please, don’t use a check on this as a security measure — this is highly insecure and easily circumvented!). - 🔸Bucket names with '.' (periods) in them [can cause certificate mismatches](https://forums.aws.amazon.com/thread.jspa?threadID=169951) when used with SSL. Use '-' instead, since this then conforms with both SSL expectations and is DNS compliant. -- **Versioning:** S3 has [optional versioning support](https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html), so that all versions of objects are preserved on a bucket. This is mostly useful if you want an archive of changes or the ability to back out mistakes (it has none of the features of full version control systems like Git). +- **Versioning:** S3 has [optional versioning support](https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html), so that all versions of objects are preserved on a bucket. This is mostly useful if you want an archive of changes or the ability to back out mistakes (caution: it lacks the featureset of full version control systems like Git). - **Durability:** Durability of S3 is extremely high, since internally it keeps several replicas. If you don’t delete it by accident, you can count on S3 not losing your data. (AWS offers the seemingly improbable durability rate of [99.999999999%](https://aws.amazon.com/s3/faqs/#How_durable_is_Amazon_S3), but this is a mathematical calculation based on independent failure rates and levels of replication — not a true probability estimate. Either way, S3 has had [a very good record](https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanently) of durability.) Note this is *much* higher durability than EBS! If durability is less important for your application, you can use [S3 Reduced Redundancy Storage](https://aws.amazon.com/s3/reduced-redundancy/), which lowers the cost per GB, as well as the redundancy. - 💸**S3 pricing** depends on [storage, requests, and transfer](https://aws.amazon.com/s3/pricing/). - For transfer, putting data into AWS is free, but you’ll pay on the way out. Transfer from S3 to EC2 in the *same region* is free. Transfer to other regions or the Internet in general is not free. From df9f6823f0aa4fd322fa22795d362f86e8771427 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Fri, 14 Apr 2017 17:06:50 -0700 Subject: [PATCH 116/128] Never suggest us-west-1; it's expensive and massively capacity constrained. (#417) * Never suggest us-west-1; it's expensive and massively capacity constrained. * Avoid suggesting any given region --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ae52861..a041baf 100644 --- a/README.md +++ b/README.md @@ -755,7 +755,7 @@ EC2 ### EC2 Tips -- 🔹**Picking regions:** When you first set up, consider which [regions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) you want to use first. Many people in North America just automatically set up in the us-east-1 (N. Virginia) region, which is the default, but it’s worth considering if this is best up front. For example, you might find it preferable to start in us-west-1 (N. California) or us-west-2 (Oregon) if you’re in California and latency matters. Some services [are not available in all regions](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). Baseline costs also [vary by region](https://aws.amazon.com/ec2/pricing/), up to 10-30% (generally lowest in us-east-1). +- 🔹**Picking regions:** When you first set up, consider which [regions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) you want to use first. Many people in North America just automatically set up in the us-east-1 (N. Virginia) region, which is the default, but it’s worth considering if this is best up front. You'll want to evaluate service availibility (some services [are not available in all regions](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/)), costing (baseline costs also [vary by region](https://aws.amazon.com/ec2/pricing/) by up to 10-30% (generally lowest in us-east-1 for comparison purposes)), and compliance (various countries have differing regulations with regard to data privacy, for example). - **Instance types:** EC2 instances come in many types, corresponding to the capabilities of the virtual machine in CPU architecture and speed, RAM, disk sizes and types (SSD or magnetic), and network bandwidth. - Selecting instance types is complex since there are so many types. Additionally, there are different generations, released [over the years](https://aws.amazon.com/blogs/aws/ec2-instance-history/). - 🔹Use the list at [**ec2instances.info**](http://www.ec2instances.info/) to review costs and features. [Amazon’s own list](https://aws.amazon.com/ec2/instance-types/) of instance types is hard to use, and doesn’t list features and price together, which makes it doubly difficult. From 0630197c279ff872306e1c3522ca00ba313797e6 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Fri, 14 Apr 2017 19:46:08 -0700 Subject: [PATCH 117/128] Emoji nitpicks --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9d27860..36c2fea 100644 --- a/README.md +++ b/README.md @@ -494,8 +494,8 @@ So if you’re not going to manage your AWS configurations manually, what should - To label lifecycles, such as temporary resources or one that should be deprovisioned in the future - To distinguish production-critical infrastructure (e.g. serving systems vs backend pipelines) - To distinguish resources with special security or compliance requirements -- 🔸For many years, there was a notorious 10 tag limit per resource, which could not be raised and caused many companies significant pain. As of 2016, this was [raised](https://aws.amazon.com/blogs/security/now-organize-your-aws-resources-by-using-up-to-50-tags-per-resource/) to 50 tags per resource. -- In 2017, AWS introduced the ability to [enforce tagging](https://aws.amazon.com/blogs/aws/new-tag-ec2-instances-ebs-volumes-on-creation/) on instance and volume creation, deprecating portions of third party tools such as [Cloud Custodian](https://github.com/capitalone/cloud-custodian). +- For many years, there was a notorious 10 tag limit per resource, which could not be raised and caused many companies significant pain. As of 2016, this was [raised](https://aws.amazon.com/blogs/security/now-organize-your-aws-resources-by-using-up-to-50-tags-per-resource/) to 50 tags per resource. +- 🔹In 2017, AWS introduced the ability to [enforce tagging](https://aws.amazon.com/blogs/aws/new-tag-ec2-instances-ebs-volumes-on-creation/) on instance and volume creation, deprecating portions of third party tools such as [Cloud Custodian](https://github.com/capitalone/cloud-custodian). Managing Servers and Applications --------------------------------- From a943f0e1fad498a013c0f777777dcb8c0e8f14b9 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Fri, 14 Apr 2017 20:15:19 -0700 Subject: [PATCH 118/128] Fixed some word choices in the EFS section, renamed File Store to Object Store in the service matrix --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 42e2952..527beb8 100644 --- a/README.md +++ b/README.md @@ -283,7 +283,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | PaaS | Elastic Beanstalk | App Engine | App Engine | Web Apps | Heroku, AppFog, OpenShift | Meteor, AppScale, Cloud Foundry, Convox | | Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | PubNub Blocks, Auth0 Webtask | Kong, Tyk | | Container, cluster manager | ECS | Container Engine, Kubernetes | Borg or Omega | Container Service | | Kubernetes, Mesos, Aurora | -| File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | +| Object storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | | Block storage | EBS | Persistent Disk | | Storage Account | DigitalOcean Volumes | NFS | | SQL datastore | RDS | Cloud SQL | | SQL Database | | MySQL, PostgreSQL | | Sharded RDBMS | | Cloud Spanner | F1, Spanner | | | Crate.io, CockroachDB | @@ -935,10 +935,10 @@ EFS - EFS is designed to be used as a shared network drive and it can automatically scale up to petabytes of stored data and thousands of instances attached to it. - It’s presented as an [NFSv4.1](https://en.wikipedia.org/wiki/Network_File_System#NFSv4) server, so any compatible NFS client can mount it. - EFS can offer [higher throughput](http://docs.aws.amazon.com/efs/latest/ug/performance.html) (multiple gigabytes per second) and better durability and availability than EBS (see [the comparison table](#storage-durability-availability-and-price)), but with higher latency. -- EFS is priced based on the amount of data stored and it costs [much more than EBS](#storage-durability-availability-and-price), about three times as much compared to general purpose gp2 EBS volumes. -- ⏱ [Performance](http://docs.aws.amazon.com/efs/latest/ug/performance.html) depends on the amount of data stored on it, which also determines the price: +- EFS is priced based on the volume of data stored, and costs [much more than EBS](#storage-durability-availability-and-price); it's in the ballpark of three times as much compared to general purpose gp2 EBS volumes. +- ⏱ [Performance](http://docs.aws.amazon.com/efs/latest/ug/performance.html) is dependent on the volume of data stored, as is the price: - Like EBS, EFS uses a credit based system. Credits are earned at a rate of 50 KiB/s per GiB of storage and consumed in bursts during reading/writing files or metadata. Unlike EBS, operations on metadata (file size, owner, date, etc.) also consume credits. The [BurstCreditBalance metric](http://docs.aws.amazon.com/efs/latest/ug/monitoring-cloudwatch.html#efs-metrics) in CloudWatch should be monitored to make sure the file system doesn't run out of credits. - - Throughput capacity during bursts is also dependant on size. Under 1 TiB, throughput can go up to 100 MiB/s. Above that, 100 MiB/s is added for each stored TiB. So a file system storing 5 TiB would be able to burst at a rate of 500 MiB/s. Maximum throughput per EC2 instance is 250 MiB/s. + - Throughput capacity during bursts is also dependent on size. Under 1 TiB, throughput can go up to 100 MiB/s. Above that, 100 MiB/s is added for each stored TiB. For instance, a file system storing 5 TiB would be able to burst at a rate of 500 MiB/s. Maximum throughput per EC2 instance is 250 MiB/s. - EFS has two performance modes that can only be set when a file system is created. One is "General Purpose", the other is "Max I/O". Max I/O scales higher, but at the cost of higher latency. When in doubt, use General Purpose, which is also the default. If the [PercentIOLimit metric](http://docs.aws.amazon.com/efs/latest/ug/monitoring-cloudwatch.html#efs-metrics) in CloudWatch hovers around 100%, Max I/O is recommended. Changing performance mode means creating a new EFS and migrating data. - High availability is achieved by having [mount targets in different subnets / availability zones](http://docs.aws.amazon.com/efs/latest/ug/images/overview-flow.png). @@ -953,7 +953,7 @@ EFS Cons: - Since credits are shared, if one application over-consumes them, it will affect the others. - - A compromise is made with regards to [security](http://docs.aws.amazon.com/efs/latest/ug/security-considerations.html). All clients will have to have network access to the drive. Someone with root access on one client instance can mount any directory on the EFS and they have read-write access to all files on the drive, even if they don't have access to the applications hosted on other clients. + - A compromise is made with regards to [security](http://docs.aws.amazon.com/efs/latest/ug/security-considerations.html): all clients will have to have network access to the drive. Someone with root access on one client instance can mount any directory on the EFS and they have read-write access to all files on the drive, even if they don't have access to the applications hosted on other clients. There isn't a no-root-squash equivalent for EFS. ### EFS Gotchas and Limitations @@ -963,7 +963,7 @@ EFS - 🔸 An EFS file system can NOT be mounted over VPC peering or VPN, even if the VPN is running on top of Direct Connect. - 🔸 Using an EFS volume on Windows is not supported. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. -- 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performace if run periodically. +- 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on both cost and performance if run regularly. Load Balancers From bcf62aac3e204a116768ebdd6527f8cab52b86eb Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Fri, 14 Apr 2017 20:20:07 -0700 Subject: [PATCH 119/128] misspelled availability --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 527beb8..109f829 100644 --- a/README.md +++ b/README.md @@ -757,7 +757,7 @@ EC2 ### EC2 Tips -- 🔹**Picking regions:** When you first set up, consider which [regions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) you want to use first. Many people in North America just automatically set up in the us-east-1 (N. Virginia) region, which is the default, but it’s worth considering if this is best up front. You'll want to evaluate service availibility (some services [are not available in all regions](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/)), costing (baseline costs also [vary by region](https://aws.amazon.com/ec2/pricing/) by up to 10-30% (generally lowest in us-east-1 for comparison purposes)), and compliance (various countries have differing regulations with regard to data privacy, for example). +- 🔹**Picking regions:** When you first set up, consider which [regions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) you want to use first. Many people in North America just automatically set up in the us-east-1 (N. Virginia) region, which is the default, but it’s worth considering if this is best up front. You'll want to evaluate service availability (some services [are not available in all regions](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/)), costing (baseline costs also [vary by region](https://aws.amazon.com/ec2/pricing/) by up to 10-30% (generally lowest in us-east-1 for comparison purposes)), and compliance (various countries have differing regulations with regard to data privacy, for example). - **Instance types:** EC2 instances come in many types, corresponding to the capabilities of the virtual machine in CPU architecture and speed, RAM, disk sizes and types (SSD or magnetic), and network bandwidth. - Selecting instance types is complex since there are so many types. Additionally, there are different generations, released [over the years](https://aws.amazon.com/blogs/aws/ec2-instance-history/). - 🔹Use the list at [**ec2instances.info**](http://www.ec2instances.info/) to review costs and features. [Amazon’s own list](https://aws.amazon.com/ec2/instance-types/) of instance types is hard to use, and doesn’t list features and price together, which makes it doubly difficult. From 032a8b8a711b6f29efcbe74ea48ed7244ed1ed8b Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Fri, 14 Apr 2017 20:30:01 -0700 Subject: [PATCH 120/128] Include IAM policies in a nutshell --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 42e2952..a8812f4 100644 --- a/README.md +++ b/README.md @@ -561,6 +561,7 @@ We cover security basics first, since configuring user accounts is something you - 📜 Access keys that start with AKIA are normal keys. Access keys that start with ASIA are session/temporary keys from STS, and will require an additional "SessionToken" parameter to be sent along with the id and secret. - [**Multi-factor authentication (MFA)**](https://aws.amazon.com/iam/details/mfa/), which is the highly recommended practice of using a keychain fob or smartphone app as a second layer of protection for user authentication. - IAM allows complex and fine-grained control of permissions, dividing users into groups, assigning permissions to roles, and so on. There is a [policy language](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) that can be used to customize security policies in a fine-grained way. + - An excellent high level overview of IAM policy concepts lives at [IAM Policies In A Nutshell](http://start.jcolemorrison.com/aws-iam-policies-in-a-nutshell/). - 🔸The policy language has a complex and error-prone JSON syntax that’s quite confusing, so unless you are an expert, it is wise to base yours off trusted examples or AWS’ own pre-defined [managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html). - At the beginning, IAM policy may be very simple, but for large systems, it will grow in complexity, and need to be managed with care. - 🔹Make sure one person (perhaps with a backup) in your organization is formally assigned ownership of managing IAM policies, make sure every administrator works with that person to have changes reviewed. This goes a long way to avoiding accidental and serious misconfigurations. From 25d2324786a19423dd5aa390fa8f5a37b1c2611a Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Sat, 15 Apr 2017 12:03:44 -0700 Subject: [PATCH 121/128] Added Organizations (#424) --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 6a0f584..0422204 100644 --- a/README.md +++ b/README.md @@ -1775,6 +1775,8 @@ Billing and Cost Management - **Tagging for cost visibility:** As the infrastructure grows, a key part of managing costs is understanding where they lie. It’s strongly advisable to [tag resources](https://aws.amazon.com/blogs/aws/resource-groups-and-tagging/), and as complexity grows, group them effectively. If you [set up billing allocation appropriately](http://aws.amazon.com/blogs/aws/aws-cost-allocation/), you can then get visibility into expenses according to organization, product, individual engineer, or any other way that is helpful. - If you need to do custom analysis of raw billing data or want to feed it to a third party cost analysis service, [enable](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html#turnonreports) the [detailed billing report](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html#detailed-billing-report) feature. - Multiple Amazon accounts can be linked for billing purposes using the [Consolidated Billing](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html) feature. Large enterprises may need complex billing structures depending on ownership and approval processes. +- Multiple Amazon accounts can be managed centrally using [AWS Organizations](https://aws.amazon.com/organizations/). + - 🔸 Be aware that if an AWS account has been created through the AWS Organizations console, API, or CLI, it can never leave that organization. ### AWS Data Transfer Costs From 91265dce0c26b847d575c73c8eaacdc48b839e8d Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Mon, 17 Apr 2017 01:28:17 -0700 Subject: [PATCH 122/128] Add OpsWorks section. --- README.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/README.md b/README.md index 0422204..52a2291 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,7 @@ Table of Contents | [SES](#ses) | [📗](#ses-basics) | [📘](#ses-tips) | [📙](#ses-gotchas-and-limitations) | | [Certificate Manager](#certificate-manager) | [📗](#certificate-manager-basics) | [📘](#certificate-manager-tips) | [📙](#certificate-manager-gotchas-and-limitations) | | [WAF](#waf) | [📗](#waf-basics) | [📘](#waf-tips) | [📙](#waf-gotchas-and-limitations) | +| [OpsWorks](#opsworks) | [📗](#opsworks-basics) | [📘](#opsworks-tips) | [📙](#opsworks-gotchas-and-limitations) | **Special Topics** @@ -1728,6 +1729,37 @@ WAF - As of December 2016, WAF is available in the US East (Northern Virginia), US West (Oregon), Asia Pacific (Tokyo) and EU (Ireland) regions. + +OpsWorks +------------------- + +### OpsWorks Basics + +- 📒 [Homepage](https://aws.amazon.com/opsworks/) ∙ [Documentation](https://aws.amazon.com/documentation/opsworks/) ∙ [FAQ](https://aws.amazon.com/opsworks/faqs/) ∙ [Pricing - Stacks](https://aws.amazon.com/opsworks/stacks/pricing/) ∙ [Pricing - ChefAutomate](https://aws.amazon.com/opsworks/chefautomate/pricing/) +- OpsWorks is a configuration management service that relies heavy on [Chef](https://www.chef.io/chef/) for configuration-as-code deployment automation. The service lets you configure and launch stacks specific to your application's needs. +- There are numerous options, in and out of AWS, that let you automate application deployments. The separating factor between OpsWorks and other configuration management services (Elastic Beanstalk for example) is that OpsWorks specializes in letting you control the details of the systems your application runs on, where a service like Elastic Beanstalk simplifies this to focus on application configuration. +- [OpsWorks Stacks](https://aws.amazon.com/opsworks/stacks/) allows you to run your deployment stacks both in the AWS cloud as well as on your own hardware on-premises. +- [OpsWorks for ChefAutomate](https://aws.amazon.com/opsworks/chefautomate/) provides a managed Chef configuration management server for your deployment pipeline. This server stores configuration tasks and provides them to your deployment nodes without manual intervention, in addition to providing other management and monitoring features. +- As of December 2016, OpsWorks Stacks supports Chef versions [12, 11.10.4, 11.4.4 and 0.9.15.5](http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook.html). +- As of December 2016, OpsWorks for ChefAutomate uses [Chef Server version 12.11.1](http://docs.aws.amazon.com/opsworks/latest/userguide/welcome_opscm.html) This is the current stable version of Chef. +- [Berkshelf can be used](http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-chef11-10.html#workingcookbook-chef11-10-berkshelf) with Chef stacks of version 11.10 and later for managing cookbooks and their respective dependencies. +- Running your own Chef environment may be an alternative to consider - some considerations are listed [in this Bitlancer article.](http://www.bitlancer.com/blog/2015/10/05/opsworks-vs-chef.html) +- A key difference between running OpsWorks and rolling your own Chef environment is that the latter allows the scheduling of Chef runs, while the former has Chef runs occur according to lifecycle hooks. + +### OpsWorks Alternatives and Lock-in + +- Major competitors to Chef include [Puppet](https://puppet.com/product/how-puppet-works) and [Ansible](https://www.ansible.com/how-ansible-works). + +### OpsWorks Tips + +- [OpsWorks relies heavily on Chef cookbooks and recipes](http://docs.aws.amazon.com/opsworks/latest/userguide/gettingstarted-cookbooks.html) for customization, so familiarity with reading their syntax will help greatly with getting up and running. + +### OpsWorks Gotchas and Limitations + +- Although OpsWorks will let you work with common Chef recipes when creating your stacks, creating custom recipes will require familiarity with Chef syntax. +- As of December 2016, OpsWorks is available in all regions except Canada, GovCloud (US), and China (Beijing). + + High Availability ----------------- From be86242e83c22ee9ca1a1a19ca921115ac92423e Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Mon, 17 Apr 2017 10:11:55 -0700 Subject: [PATCH 123/128] Fixing cost misstatement --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 52a2291..f73d69a 100644 --- a/README.md +++ b/README.md @@ -965,7 +965,7 @@ EFS - 🔸 An EFS file system can NOT be mounted over VPC peering or VPN, even if the VPN is running on top of Direct Connect. - 🔸 Using an EFS volume on Windows is not supported. - ⏱ When a file is uploaded to EFS, it can take hours for EFS to update the details for billing and burst credit purposes. -- 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on both cost and performance if run regularly. +- 🔸⏱ Metadata operations can be costly in terms of burst credit consumption. Recursively traversing a tree containing thousands of files can easily ramp up to tens or even hundreds of megabytes of burst credits being consumed, even if no file is being touched. Commands like ```find``` or ```chown -R``` can have an adverse impact on performance. Load Balancers From 1f3252371327b52112ecd2dd4a6709e84f763adf Mon Sep 17 00:00:00 2001 From: Matt Dalesio Date: Tue, 18 Apr 2017 12:56:24 -0400 Subject: [PATCH 124/128] Update README.md added 2016 for RDS SQL Server --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 52a2291..ab9a34e 100644 --- a/README.md +++ b/README.md @@ -1163,7 +1163,7 @@ RDS SQL Server ### RDS SQL Server Basics -- [RDS offers SQL Server 2008 R2, 2012, and 2014](https://aws.amazon.com/rds/sqlserver/) including Express, Web, Standard and Enterprise (2008 R2 and 2012 only for Enterprise) +- [RDS offers SQL Server 2008 R2, 2012, 2014, and 2016](https://aws.amazon.com/rds/sqlserver/) including Express, Web, Standard and Enterprise (2008 R2 and 2012 only for Enterprise) ### RDS SQL Server Tips From 9c6c9d4bb8a0b55a0304938ac3365241a6c47616 Mon Sep 17 00:00:00 2001 From: Joe Greene Date: Wed, 19 Apr 2017 22:28:11 -0700 Subject: [PATCH 125/128] Update information regarding OpsWorks Stacks and Chef Automate region availability. (#428) --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index be27e9e..fd253c6 100644 --- a/README.md +++ b/README.md @@ -1757,7 +1757,8 @@ OpsWorks ### OpsWorks Gotchas and Limitations - Although OpsWorks will let you work with common Chef recipes when creating your stacks, creating custom recipes will require familiarity with Chef syntax. -- As of December 2016, OpsWorks is available in all regions except Canada, GovCloud (US), and China (Beijing). +- OpsWorks Stacks is not available in the Canada, GovCloud and Beijing regions. +- OpsWorks for Chef Automate is [only available](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/) in the North Virginia, Oregon, and Ireland regions. High Availability From 751f56895846fbb1b446f6a00561d467795dad53 Mon Sep 17 00:00:00 2001 From: Trenton Lipscomb Date: Fri, 21 Apr 2017 14:51:36 -0700 Subject: [PATCH 126/128] DynamoDB gotcha: moment-in-time backups --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index fd253c6..c9f78e7 100644 --- a/README.md +++ b/README.md @@ -1204,6 +1204,7 @@ DynamoDB - 🔸 It is important to make sure that DynamoDB [resource limits](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-data-types) are compatible with your dataset and workload. For example, the maximum size value that can be added to a DynamoDB table is 400 KB (larger items can be stored in S3 and a URL stored in DynamoDB). - 🔸 Dealing with **time series data** in DynamoDB can be challenging. A global secondary index together with down sampling timestamps can be a possible solution as explained [here](https://blogs.aws.amazon.com/bigdata/post/Tx3KPZDXIBJEQ4B/Scaling-Writes-on-Amazon-DynamoDB-Tables-with-Global-Secondary-Indexes). - 🔸 DynamoDB does [not allow](https://forums.aws.amazon.com/thread.jspa?threadID=90137) an empty string as a valid attribute value. The most common work-around is to use a substitute value instead of leaving the field empty. +- 🔸 DynamoDB does not support moment-in-time/snapshot backups. It is not possible to create a moment-in-time view of a table without preventing all writes to a table and then consistently reading it. You can't work around this using [Cross-Region Replication](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.CrossRegionRepl.html) because under the covers, it uses [DynamoDB Streams](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html). In a Stream, the order in which Items appear can be different than the order in wihch they were written. Only updates to the same Item are guaranteed to be ordered. ECS --- From 98c75f87ccaac6e75061561acdd63359d41ece5a Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Wed, 26 Apr 2017 16:37:02 -0700 Subject: [PATCH 127/128] Added Last Week in AWS to further reading section (#414) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 👍 I think we've provided enough time for comment on this. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c9f78e7..cb0e578 100644 --- a/README.md +++ b/README.md @@ -1879,6 +1879,7 @@ This section covers a few unusually useful or “must know about” resources or - [Awesome AWS](https://github.com/donnemartin/awesome-aws): A curated list of AWS tools and software - [AWS Tips I Wish I'd Known Before I Started](https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/): A list of tips from [Rich Adams](https://richadams.me/) - [AWS Whitepapers](https://aws.amazon.com/whitepapers/): A list of technical AWS whitepapers, covering topics such as architecture, security and economics. + - [Last Week in AWS](https://lastweekinaws.com): A weekly email newsletter covering the latest happenings in the AWS ecosystem. - Books - [Amazon Web Services in Action](https://www.manning.com/books/amazon-web-services-in-action) - [AWS Lambda in Action](https://www.manning.com/books/aws-lambda-in-action) From 29e0e35dc80163474b7a091b4ea359cd93b2d215 Mon Sep 17 00:00:00 2001 From: Corey Quinn Date: Wed, 26 Apr 2017 16:38:02 -0700 Subject: [PATCH 128/128] Clarify Availability Zone separation (#422) * Clarified AZ features * Added changes suggested by @jlevy --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cb0e578..4c15242 100644 --- a/README.md +++ b/README.md @@ -270,7 +270,7 @@ We’ve assembled a landscape of a few of the services. This is far from complet ### Common Concepts - 📒 The AWS [**General Reference**](https://docs.aws.amazon.com/general/latest/gr/Welcome.html) covers a bunch of common concepts that are relevant for multiple services. -- AWS allows deployments in [**regions**](https://docs.aws.amazon.com/general/latest/gr/rande.html), which are isolated geographic locations that help you reduce latency or offer additional redundancy (though typically availability zones are the first tool of choice for [high availability](#high-availability)). +- AWS allows deployments in [**regions**](https://docs.aws.amazon.com/general/latest/gr/rande.html), which are isolated geographic locations that help you reduce latency or offer additional redundancy. Regions contain availability zones(AZs), which are typically the first tool of choice for [high availability](#high-availability)). AZs are [physically separate from one another](https://www.youtube.com/watch?v=JIQETrFC_SQ&feature=youtu.be&t=1428) even within the same region, and [may span multiple physical data centers](https://blog.rackspace.com/aws-101-regions-availability-zones). While they are connected via low latency links, natural disasters afflicting one should not affect others. - Each service has API **endpoints** for each region. Endpoints differ from service to service and not all services are available in each region, as listed in [these tables](https://docs.aws.amazon.com/general/latest/gr/rande.html). - [**Amazon Resource Names (ARNs)**](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) are specially formatted identifiers for identifying resources. They start with 'arn:' and are used in many services, and in particular for IAM policies.