From cfd6e91cafb3797e962fe294b6506fc45d2623e0 Mon Sep 17 00:00:00 2001 From: David Kocher Date: Tue, 11 Oct 2016 20:55:29 +0200 Subject: [PATCH 001/140] Replace recommendation of GUI application for S3. Transmit is outdated, missing AWS4 signature version support (crucial for new regions) and multipart uploads besides ACLs, versioning, lifecycle configuration, storage classes and server side encryption. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7c2887f..b9754e4 100644 --- a/README.md +++ b/README.md @@ -603,7 +603,7 @@ S3 - [**s4cmd**](https://github.com/bloomreach/s4cmd) is a replacement, with greater emphasis on performance via multi-threading, which is helpful for large files and large sets of files, and also offers Unix-like globbing support. - **GUI applications:** You may prefer a GUI, or wish to support GUI access for less technical users. Some options: - The [AWS Console](https://aws.amazon.com/console/) does offer a graphical way to use S3. Use caution telling non-technical people to use it, however, since without tight permissions, it offers access to many other AWS features. - - [Transmit](https://panic.com/transmit/) is a good option on OS X. + - [Cyberduck](https://cyberduck.io/) is a good option on OS X and Windows with support for multipart uploads, ACLs, versioning, lifecycle configuration, storage classes and server side encryption (SSE-S3 and SSE-KMS). - **S3 and CloudFront:** S3 is tightly integrated with the CloudFront CDN. See the CloudFront section for more information, as well as [S3 transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html). - **Static website hosting:** - S3 has a [static website hosting option](http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) that is simply a setting that enables configurable HTTP index and error pages and [HTTP redirect support](http://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html) to [public content](http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteAccessPermissionsReqd.html) in S3. It’s a simple way to host static assets or a fully static website. From 967669dab1ae58a9d7faf51575ba5359c7692aa0 Mon Sep 17 00:00:00 2001 From: Sandeep Dinesh Date: Tue, 11 Oct 2016 13:46:04 -0700 Subject: [PATCH 002/140] Update Google Cloud product comparison Changes: - Removed Cloud SQL as a sharded RDBMS (Cloud SQL does not provide this) - Moved CockroachDB from Bigtable to Sharded RDBMS (Cockroach is an open-source version of Spanner) - Added HBase as the open source version of Bigtable (Cloud Bigtable is compatible with the HBase API) - Added Dremel as the Google Internal version of BigQuery - Added Cloud CDN (CDN), Firebase Analytics (Mobile Analytics), and Cloud Source Repositories (git hosting) - Changed Cloud Test Lab to Firebase Test Lab (rebranding) --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 7c2887f..aa54209 100644 --- a/README.md +++ b/README.md @@ -265,12 +265,12 @@ Many services within AWS can at least be compared with Google Cloud offerings or | File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | | Block storage | EBS | Persistent Disk | | Storage Account | | NFS | | SQL datastore | RDS | Cloud SQL | | SQL Database | | MySQL, PostgreSQL | -| Sharded RDBMS | | Cloud SQL | F1, Spanner | | | Crate.io | -| Bigtable | | Cloud Bigtable | Bigtable | | | CockroachDB | +| Sharded RDBMS | | | F1, Spanner | | | Crate.io, CockroachDB | +| Bigtable | | Cloud Bigtable | Bigtable | | | HBase | | Key-value store, column store | DynamoDB | Cloud Datastore | Megastore | Tables, DocumentDB | | Cassandra, CouchDB, RethinkDB, Redis | | Memory cache | ElastiCache | App Engine Memcache | | Redis Cache | | Memcached, Redis | | Search | CloudSearch, Elasticsearch (managed) | | | Search | Algolia, QBox | Elasticsearch, Solr | -| Data warehouse | Redshift | BigQuery | | SQL Data Warehouse | Oracle, IBM, SAP, HP, many others | Greenplum | +| Data warehouse | Redshift | BigQuery | Dremel | SQL Data Warehouse | Oracle, IBM, SAP, HP, many others | Greenplum | | Business intelligence | QuickSight | | | Power BI | Tableau | | | Lock manager | [DynamoDB (weak)](https://gist.github.com/ryandotsmith/c95fd21fab91b0823328) | | Chubby | Lease blobs in Storage Account | | ZooKeeper, Etcd, Consul | | Message broker | SQS, SNS, IoT | Pub/Sub | PubSub2 | Service Bus | | RabbitMQ, Kafka, 0MQ | @@ -278,14 +278,14 @@ Many services within AWS can at least be compared with Google Cloud offerings or | MapReduce | EMR | Dataproc | MapReduce | HDInsight, DataLake Analytics | Qubole | Hadoop | | Monitoring | CloudWatch | Monitoring | Borgmon | Monitor | | Prometheus(?) | | Metric management | | | Borgmon, TSDB | Application Insights | | Graphite, InfluxDB, OpenTSDB, Grafana, Riemann, Prometheus | -| CDN | CloudFront | | | CDN | | Apache Traffic Server | +| CDN | CloudFront | Cloud CDN | | CDN | | Apache Traffic Server | | Load balancer | CLB/ALB | Load Balancing | GFE | Load Balancer, Application Gateway | | nginx, HAProxy, Apache Traffic Server | | DNS | Route53 | DNS | | DNS | | bind | | Email | SES | | | | Sendgrid, Mandrill, Postmark | | -| Git hosting | CodeCommit | | | Visual Studio Team Services | GitHub, BitBucket | GitLab | +| Git hosting | CodeCommit | Cloud Source Repositories | | Visual Studio Team Services | GitHub, BitBucket | GitLab | | User authentication | Cognito | | | Azure Active Directory | | oauth.io | -| Mobile app analytics | Mobile Analytics | | | HockeyApp | Mixpanel | | -| Mobile app testing | Device Farm |Cloud Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | +| Mobile app analytics | Mobile Analytics | Firebase Analytics | | HockeyApp | Mixpanel | | +| Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | 🚧 [*Please help fill this table in.*](CONTRIBUTING.md) From 8d82b60c9566de9572d2880f38848ac93bb82f0f Mon Sep 17 00:00:00 2001 From: Greg Leeds Date: Tue, 11 Oct 2016 15:46:50 -0500 Subject: [PATCH 003/140] Updating EC2 GPU section to reflect new instance types --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7c2887f..1e8e947 100644 --- a/README.md +++ b/README.md @@ -699,7 +699,12 @@ EC2 - Aside from bootstrapping, you should manage keys yourself on the instances, assigning individual keys to individual users or services as appropriate. - Avoid reusing the original boot keys except by administrators when creating new instances. - How to avoid sharing keys; how to add individual ssh keys for individual users. -- **GPU support:** You can rent GPU-enabled instances on EC2. There are [two instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html). Both sport an NVIDIA card (K520, 1536 CUDA cores and M2050, 448 CUDA cores). +- **GPU support:** You can rent GPU-enabled instances on EC2 for use in machine learning or graphics rendering workloads. + - There are [three generations](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html) of GPU-enabled instances available: + - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. + - Second generation G2 series offers NVIDIA K520 GPUs in 1 or 4 GPU configurations targeting graphics and video encoding. + - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. + - 🔹Typical GPU workloads are often a good fit for [Spot Instances](#ec2-cost-management). ### EC2 Gotchas and Limitations From b9806f3b9941370b2b22a17a6d8b814b9a9ebfb0 Mon Sep 17 00:00:00 2001 From: esell Date: Tue, 11 Oct 2016 15:21:33 -0600 Subject: [PATCH 004/140] add azure customer list --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7c2887f..9862004 100644 --- a/README.md +++ b/README.md @@ -157,8 +157,9 @@ General Information - Basic services like virtual servers and standard databases are usually easy to migrate to other providers or on premises. Others like load balancers and IAM are specific to AWS but have close equivalents from other providers. The key thing to consider is whether engineers are architecting systems around specific AWS services that are not open source or relatively interchangeable. For example, Lambda, API Gateway, Kinesis, Redshift, and DynamoDB do not have have substantially equivalent open source or commercial service equivalents, while EC2, RDS (MySQL or Postgres), EMR, and ElastiCache more or less do. (See more [below](#which-services-to-use), where these are noted with ⛓.) - **Combining AWS and other cloud providers:** Many customers combine AWS with other non-AWS services. For example, legacy systems or secure data might be in a managed hosting provider, while other systems are AWS. Or a company might only use S3 with another provider doing everything else. However small startups or projects starting fresh will typically stick to AWS or Google Cloud only. - **Hybrid cloud:** In larger enterprises, it is common to have [hybrid deployments](https://aws.amazon.com/enterprise/hybrid/) encompassing private cloud or on-premises servers and AWS — or other enterprise cloud providers like [IBM](https://www.ibm.com/cloud-computing/solutions/hybrid-cloud)/[Bluemix](http://www.ibm.com/cloud-computing/bluemix/hybrid/), [Microsoft](https://www.microsoft.com/en-us/cloud-platform/hybrid-cloud)/[Azure](https://azure.microsoft.com/en-us/overview/azure-stack/), [NetApp](http://www.netapp.com/us/solutions/cloud/hybrid-cloud/), or [EMC](http://www.emc.com/en-us/cloud/hybrid-cloud-computing/index.htm). -- **Major customers:** Who uses AWS and Google Cloud? +- **Major customers:** Who uses AWS, Azure and Google Cloud? - AWS’s [list of customers](https://aws.amazon.com/solutions/case-studies/) includes large numbers of mainstream online properties and major brands, such as Netflix, Pinterest, Spotify, Airbnb, Expedia, Yelp, Zynga, Comcast, Nokia, and Bristol-Myers Squibb. + - Azure's [list of customers](https://azure.microsoft.com/en-us/case-studies/) includes companies such as NBC Universal, 3M and Honeywell Inc. - Google Cloud’s [list of customers](https://cloud.google.com/customers/) is large as well, and includes a few mainstream sites, such as [Snapchat](http://www.businessinsider.com/snapchat-is-built-on-googles-cloud-2014-1), Best Buy, Domino’s, and Sony Music. ### Which Services to Use From c54b5609802c5c4489d06d2cd5a0332c76ba24c9 Mon Sep 17 00:00:00 2001 From: esell Date: Tue, 11 Oct 2016 15:27:06 -0600 Subject: [PATCH 005/140] update azure info --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9862004..ba8538f 100644 --- a/README.md +++ b/README.md @@ -138,8 +138,8 @@ General Information - **Costs:** Billing and cost management are such big topics that we have [an entire section on this](#billing-and-cost-management). - 🔹**EC2 vs. other services:** Most users of AWS are most familiar with [EC2](#ec2), AWS’ flagship virtual server product, and possibly a few others like S3 and CLBs. But AWS products now extend far beyond basic IaaS, and often companies do not properly understand or appreciate all the many AWS services and how they can be applied, due to the [sharply growing](#which-services-to-use) number of services, their novelty and complexity, branding confusion, and fear of ⛓lock-in to proprietary AWS technology. Although a bit daunting, it’s important for technical decision-makers in companies to understand the breadth of the AWS services and make informed decisions. (We hope this guide will help.) - 🚪**AWS vs. other cloud providers:** While AWS is the dominant IaaS provider (31% market share in [this 2016 estimate](https://www.srgresearch.com/articles/aws-remains-dominant-despite-microsoft-and-google-growth-surges)), there is significant competition and alternatives that are better suited to some companies: - - The most significant direct competitor is [**Google Cloud**](https://cloud.google.com/). It arrived later to market than AWS, but has vast resources and is now used widely by many companies, including a few large ones. It is gaining market share. Not all AWS services have similar or analogous services in Google Cloud. And vice versa: In particular Google offers some more advanced machine learning-based services like the [Vision](https://cloud.google.com/vision/), [Speech](https://cloud.google.com/speech/), and [Natural Language](https://cloud.google.com/natural-language/) APIs. It’s not common to switch once you’re up and running, but it does happen: [Spotify migrated](http://www.wsj.com/articles/google-cloud-lures-amazon-web-services-customer-spotify-1456270951) from AWS to Google Cloud. There is more discussion [on Quora](https://www.quora.com/What-are-the-reasons-to-choose-AWS-over-Google-Cloud-or-vice-versa-for-a-high-traffic-web-application) about relative benefits. - - [**Microsoft Azure**](https://azure.microsoft.com/en) is the de facto choice for companies and teams that are focused on a Microsoft stack. + - [**Google Cloud**](https://cloud.google.com/). It arrived later to market than AWS, but has vast resources and is now used widely by many companies, including a few large ones. It is gaining market share. Not all AWS services have similar or analogous services in Google Cloud. And vice versa: In particular Google offers some more advanced machine learning-based services like the [Vision](https://cloud.google.com/vision/), [Speech](https://cloud.google.com/speech/), and [Natural Language](https://cloud.google.com/natural-language/) APIs. It’s not common to switch once you’re up and running, but it does happen: [Spotify migrated](http://www.wsj.com/articles/google-cloud-lures-amazon-web-services-customer-spotify-1456270951) from AWS to Google Cloud. There is more discussion [on Quora](https://www.quora.com/What-are-the-reasons-to-choose-AWS-over-Google-Cloud-or-vice-versa-for-a-high-traffic-web-application) about relative benefits. + - [**Microsoft Azure**](https://azure.microsoft.com/en) also arrived later to market than AWS and initially focused on PaaS only solutions. In recent years it has grown to deliver IaaS and SaaS offerings as well. Despite being owned by Microsoft it is Linux friendly and Microsoft states that [1/3rd of all VMs on Azure are Linux based](http://www.zdnet.com/article/microsoft-nearly-one-in-three-azure-virtual-machines-now-are-running-linux/) - In **China**, AWS’ footprint is relatively small. The market is dominated by Alibaba’s [Aliyun](https://intl.aliyun.com/). - Companies at (very) large scale may want to reduce costs by managing their own infrastructure. For example, [Dropbox migrated](https://news.ycombinator.com/item?id=11282948) to their own infrastructure. - Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys. From a2e5cbb87b8d3241086a206241f1133b74ff321c Mon Sep 17 00:00:00 2001 From: Kyle Busekist Date: Tue, 11 Oct 2016 18:47:44 -0500 Subject: [PATCH 006/140] CLB traffic to instances in the same subnet Traffic between CLBs and Instances in the same subnet do not behave the same was as Instance to Instance traffic in the same subnet. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 94dbace..74ab2fe 100644 --- a/README.md +++ b/README.md @@ -858,6 +858,7 @@ CLB - Complex rules for directing traffic are not supported. For example, you can’t direct traffic based on a regular expression in the URL, like [HAProxy](http://www.haproxy.org/) offers. - **Apex DNS names:** Once upon a time, you couldn’t assign an CLB to an apex DNS record (i.e. example.com instead of foo.example.com) because it needed to be an A record instead of a CNAME. This is now possible with a Route 53 alias record directly pointing to the load balancer. - 🔸CLBs use [HTTP keep-alives](https://en.wikipedia.org/wiki/HTTP_persistent_connection) on the internal side. This can cause an unexpected side effect: Requests from different clients, each in their own TCP connection on the external side, can end up on the same TCP connection on the internal side. Never assume that multiple requests on the same TCP connection are from the same client! +- Traffic between CLBs and back-end instances in the same subnet **will** have NACL rules evaluated (EC2 to EC2 traffic in the same subnet would not have NACL rules evaluated). If the default '0.0.0.0/0 ALLOW' rule is removed from the NACL applied to the subnet, a rule that allows traffic on both the health check port and any listener port must be added. ALB --- From 68edb122e0c272dafc25747796f3c1233a4231ad Mon Sep 17 00:00:00 2001 From: Dmitry Guyvoronsky Date: Tue, 11 Oct 2016 18:17:04 -0700 Subject: [PATCH 007/140] Updated CloudFront details. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 115f843..6957845 100644 --- a/README.md +++ b/README.md @@ -1143,14 +1143,14 @@ CloudFront - 📒 [Homepage](https://aws.amazon.com/cloudfront/) ∙ [Developer guide](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/) ∙ [FAQ](https://aws.amazon.com/cloudfront/faqs/) ∙ [Pricing](https://aws.amazon.com/cloudfront/pricing/) - **CloudFront** is AWS’ [content delivery network (CDN)](https://en.wikipedia.org/wiki/Content_delivery_network). -- Its primary use is improving latency for end users in to accessing cacheable content by hosting it at [about 40 global edge locations](http://aws.amazon.com/cloudfront/details/). +- Its primary use is improving latency for end users in to accessing cacheable content by hosting it at [over 60 global edge locations](http://aws.amazon.com/cloudfront/details/). ### CloudFront Alternatives and Lock-in - 🚪CDNs are [a highly fragmented market](https://www.datanyze.com/market-share/cdn/). CloudFront has grown to be a leader, but many alternatives that might better suit specific needs. ### CloudFront Tips - +- 🐥**IPv6** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/)! - 🐥**HTTP/2** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/09/amazon-cloudfront-now-supports-http2/)! Clients [must support TLS 1.2 and SNI](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesSupportedHTTPVersions). - While the most common use is for users to browse and download content (GET or HEAD methods) requests, CloudFront also supports ([since 2013](https://aws.amazon.com/blogs/aws/amazon-cloudfront-content-uploads-post-put-other-methods/)) uploaded data (POST, PUT, DELETE, OPTIONS, and PATCH). - You must enable this by specifying the [allowed HTTP methods](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesAllowedHTTPMethods) when you create the distribution. From 390a91a58469272ee17490c249d4f69ac0641d14 Mon Sep 17 00:00:00 2001 From: esell Date: Tue, 11 Oct 2016 19:35:10 -0600 Subject: [PATCH 008/140] update readme as per suggestions --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ba8538f..78c8b57 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,7 @@ General Information - 🔹**EC2 vs. other services:** Most users of AWS are most familiar with [EC2](#ec2), AWS’ flagship virtual server product, and possibly a few others like S3 and CLBs. But AWS products now extend far beyond basic IaaS, and often companies do not properly understand or appreciate all the many AWS services and how they can be applied, due to the [sharply growing](#which-services-to-use) number of services, their novelty and complexity, branding confusion, and fear of ⛓lock-in to proprietary AWS technology. Although a bit daunting, it’s important for technical decision-makers in companies to understand the breadth of the AWS services and make informed decisions. (We hope this guide will help.) - 🚪**AWS vs. other cloud providers:** While AWS is the dominant IaaS provider (31% market share in [this 2016 estimate](https://www.srgresearch.com/articles/aws-remains-dominant-despite-microsoft-and-google-growth-surges)), there is significant competition and alternatives that are better suited to some companies: - [**Google Cloud**](https://cloud.google.com/). It arrived later to market than AWS, but has vast resources and is now used widely by many companies, including a few large ones. It is gaining market share. Not all AWS services have similar or analogous services in Google Cloud. And vice versa: In particular Google offers some more advanced machine learning-based services like the [Vision](https://cloud.google.com/vision/), [Speech](https://cloud.google.com/speech/), and [Natural Language](https://cloud.google.com/natural-language/) APIs. It’s not common to switch once you’re up and running, but it does happen: [Spotify migrated](http://www.wsj.com/articles/google-cloud-lures-amazon-web-services-customer-spotify-1456270951) from AWS to Google Cloud. There is more discussion [on Quora](https://www.quora.com/What-are-the-reasons-to-choose-AWS-over-Google-Cloud-or-vice-versa-for-a-high-traffic-web-application) about relative benefits. - - [**Microsoft Azure**](https://azure.microsoft.com/en) also arrived later to market than AWS and initially focused on PaaS only solutions. In recent years it has grown to deliver IaaS and SaaS offerings as well. Despite being owned by Microsoft it is Linux friendly and Microsoft states that [1/3rd of all VMs on Azure are Linux based](http://www.zdnet.com/article/microsoft-nearly-one-in-three-azure-virtual-machines-now-are-running-linux/) + - [**Microsoft Azure**](https://azure.microsoft.com/en) is the de facto choice for companies and teams that are focused on a Microsoft stack, and it has now placed significant emphasis on Linux as wel - In **China**, AWS’ footprint is relatively small. The market is dominated by Alibaba’s [Aliyun](https://intl.aliyun.com/). - Companies at (very) large scale may want to reduce costs by managing their own infrastructure. For example, [Dropbox migrated](https://news.ycombinator.com/item?id=11282948) to their own infrastructure. - Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys. From 9198007f0d87f6133420c258c34f97aab84ef161 Mon Sep 17 00:00:00 2001 From: esell Date: Tue, 11 Oct 2016 19:45:01 -0600 Subject: [PATCH 009/140] Add link to gartner report in readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 78c8b57..76b3cae 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ General Information - Each of those reasons above might point to situations where other services are preferable. In practice, many, if not most, tech startups as well as a number of modern large companies can or already do benefit from using AWS. Many large enterprises are partly migrating internal infrastructure to Azure, Google Cloud, and AWS. - **Costs:** Billing and cost management are such big topics that we have [an entire section on this](#billing-and-cost-management). - 🔹**EC2 vs. other services:** Most users of AWS are most familiar with [EC2](#ec2), AWS’ flagship virtual server product, and possibly a few others like S3 and CLBs. But AWS products now extend far beyond basic IaaS, and often companies do not properly understand or appreciate all the many AWS services and how they can be applied, due to the [sharply growing](#which-services-to-use) number of services, their novelty and complexity, branding confusion, and fear of ⛓lock-in to proprietary AWS technology. Although a bit daunting, it’s important for technical decision-makers in companies to understand the breadth of the AWS services and make informed decisions. (We hope this guide will help.) -- 🚪**AWS vs. other cloud providers:** While AWS is the dominant IaaS provider (31% market share in [this 2016 estimate](https://www.srgresearch.com/articles/aws-remains-dominant-despite-microsoft-and-google-growth-surges)), there is significant competition and alternatives that are better suited to some companies: +- 🚪**AWS vs. other cloud providers:** While AWS is the dominant IaaS provider (31% market share in [this 2016 estimate](https://www.srgresearch.com/articles/aws-remains-dominant-despite-microsoft-and-google-growth-surges)), there is significant competition and alternatives that are better suited to some companies. [This Gartner report](https://www.gartner.com/doc/reprints?id=1-2G2O5FC&ct=150519&st=sb) has a good overview of the major cloud players : - [**Google Cloud**](https://cloud.google.com/). It arrived later to market than AWS, but has vast resources and is now used widely by many companies, including a few large ones. It is gaining market share. Not all AWS services have similar or analogous services in Google Cloud. And vice versa: In particular Google offers some more advanced machine learning-based services like the [Vision](https://cloud.google.com/vision/), [Speech](https://cloud.google.com/speech/), and [Natural Language](https://cloud.google.com/natural-language/) APIs. It’s not common to switch once you’re up and running, but it does happen: [Spotify migrated](http://www.wsj.com/articles/google-cloud-lures-amazon-web-services-customer-spotify-1456270951) from AWS to Google Cloud. There is more discussion [on Quora](https://www.quora.com/What-are-the-reasons-to-choose-AWS-over-Google-Cloud-or-vice-versa-for-a-high-traffic-web-application) about relative benefits. - [**Microsoft Azure**](https://azure.microsoft.com/en) is the de facto choice for companies and teams that are focused on a Microsoft stack, and it has now placed significant emphasis on Linux as wel - In **China**, AWS’ footprint is relatively small. The market is dominated by Alibaba’s [Aliyun](https://intl.aliyun.com/). From 141291c66ff3a8a520fe3337554de0b25db27c28 Mon Sep 17 00:00:00 2001 From: esell Date: Tue, 11 Oct 2016 19:48:33 -0600 Subject: [PATCH 010/140] fix typo in readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 76b3cae..4cd99f5 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,7 @@ General Information - 🔹**EC2 vs. other services:** Most users of AWS are most familiar with [EC2](#ec2), AWS’ flagship virtual server product, and possibly a few others like S3 and CLBs. But AWS products now extend far beyond basic IaaS, and often companies do not properly understand or appreciate all the many AWS services and how they can be applied, due to the [sharply growing](#which-services-to-use) number of services, their novelty and complexity, branding confusion, and fear of ⛓lock-in to proprietary AWS technology. Although a bit daunting, it’s important for technical decision-makers in companies to understand the breadth of the AWS services and make informed decisions. (We hope this guide will help.) - 🚪**AWS vs. other cloud providers:** While AWS is the dominant IaaS provider (31% market share in [this 2016 estimate](https://www.srgresearch.com/articles/aws-remains-dominant-despite-microsoft-and-google-growth-surges)), there is significant competition and alternatives that are better suited to some companies. [This Gartner report](https://www.gartner.com/doc/reprints?id=1-2G2O5FC&ct=150519&st=sb) has a good overview of the major cloud players : - [**Google Cloud**](https://cloud.google.com/). It arrived later to market than AWS, but has vast resources and is now used widely by many companies, including a few large ones. It is gaining market share. Not all AWS services have similar or analogous services in Google Cloud. And vice versa: In particular Google offers some more advanced machine learning-based services like the [Vision](https://cloud.google.com/vision/), [Speech](https://cloud.google.com/speech/), and [Natural Language](https://cloud.google.com/natural-language/) APIs. It’s not common to switch once you’re up and running, but it does happen: [Spotify migrated](http://www.wsj.com/articles/google-cloud-lures-amazon-web-services-customer-spotify-1456270951) from AWS to Google Cloud. There is more discussion [on Quora](https://www.quora.com/What-are-the-reasons-to-choose-AWS-over-Google-Cloud-or-vice-versa-for-a-high-traffic-web-application) about relative benefits. - - [**Microsoft Azure**](https://azure.microsoft.com/en) is the de facto choice for companies and teams that are focused on a Microsoft stack, and it has now placed significant emphasis on Linux as wel + - [**Microsoft Azure**](https://azure.microsoft.com/en) is the de facto choice for companies and teams that are focused on a Microsoft stack, and it has now placed significant emphasis on Linux as well - In **China**, AWS’ footprint is relatively small. The market is dominated by Alibaba’s [Aliyun](https://intl.aliyun.com/). - Companies at (very) large scale may want to reduce costs by managing their own infrastructure. For example, [Dropbox migrated](https://news.ycombinator.com/item?id=11282948) to their own infrastructure. - Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys. From 377ed03641f04dfdefc5ee8988be977c10e3c070 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Tue, 11 Oct 2016 21:28:20 -0700 Subject: [PATCH 011/140] Update AUTHORS.md --- AUTHORS.md | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 2f8cb79..46128ee 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -7,28 +7,42 @@ The following people (in alphabetical order) have contributed to or reviewed thi * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [2+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) +* [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk) * [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) * [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer) +* [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) +* [danhermann](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) * [Donne Martin (donnemartin)](https://github.com/donnemartin) +* [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy) * [Max Grigorev (forwidur)](https://github.com/forwidur) +* [Glynn Forrest (glynnforrest)](https://github.com/glynnforrest) — [1+](https://github.com/open-guides/og-aws/commits?author=glynnforrest)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aglynnforrest) * [Dmitry Golyshev (golyshev)](https://github.com/golyshev) -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [77+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[67+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_ +* [Gulam Shakir (gshakir)](https://github.com/gshakir) — [2+](https://github.com/open-guides/og-aws/commits?author=gshakir)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agshakir) +* [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) +* [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [86+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[76+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_ * Jurgen Philippaerts * [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [2+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [3+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) +* [maiki](https://github.com/maiki) — [1+](https://github.com/open-guides/og-aws/commits?author=maiki)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amaiki) * [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d) -* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [0+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (S3, EMR, Redshift)_ +* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [3+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (S3, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) +* [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke) * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Ola Wiberg (olawiberg)](https://github.com/olawiberg) * Praveen Patnala * [Russell Power (rjpower)](https://github.com/rjpower) -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [10+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[10+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_ +* [David Schott (shott85)](https://github.com/shott85) — [1+](https://github.com/open-guides/og-aws/commits?author=shott85)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashott85) +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [12+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[12+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_ +* [Tom Schlick (tomschlick)](https://github.com/tomschlick) — [3+](https://github.com/open-guides/og-aws/commits?author=tomschlick)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atomschlick) +* [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite) * [Stefan Zier (weirded)](https://github.com/weirded) +* [Michael Ortali (xethorn)](https://github.com/xethorn) — [1+](https://github.com/open-guides/og-aws/commits?author=xethorn)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Axethorn) Additional authors are welcome; see the [contribution guidelines](CONTRIBUTING.md). Please let the editors know of any errors or omissions on this list. -(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) \ No newline at end of file +(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) From 5f4537814b39f49fde04a39bdeddc4d96da2368f Mon Sep 17 00:00:00 2001 From: Bradly Feeley Date: Tue, 11 Oct 2016 21:34:55 -0700 Subject: [PATCH 012/140] Fix typo in S3 gotchas --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..62edfd5 100644 --- a/README.md +++ b/README.md @@ -662,7 +662,7 @@ S3 - 🔸Be careful not to make implicit assumptions about transactionality or sequencing of updates to objects. Never assume that if you modify a sequence of objects, the clients will see the same modifications in the same sequence, or if you upload a whole bunch of files, that they will all appear at once to all clients. - 🔸S3 has an [**SLA**](https://aws.amazon.com/s3/sla/) with 99.9% uptime. If you use S3 heavily, you’ll inevitably see occasional error accessing or storing data as disks or other infrastructure fail. Availability is usually restored in seconds or minutes. Although availability is not extremely high, as mentioned above, durability is excellent. - 🔸After uploading, any change that you make to the object causes a full rewrite of the object, so avoid appending-like behavior with regular files. -- 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 at suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. +- 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Most S3 endpoints match the region they’re in, with the exception of the us-east-1 region, which is called 'us-standard' in S3 terminology. This region is also the only region that is replicated across coasts. As a result, latency varies more in this region than in others. You can minimize latency from us-east-1 by using *[s3-external-1.amazonaws.com](http://s3-external-1.amazonaws.com/)*. From c4b02b12a4ddb861b44ecc1392e8f45d84cfe7cb Mon Sep 17 00:00:00 2001 From: Bradly Feeley Date: Tue, 11 Oct 2016 21:41:12 -0700 Subject: [PATCH 013/140] Fixing misspelling of tooling --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..ca39217 100644 --- a/README.md +++ b/README.md @@ -417,7 +417,7 @@ There are several approaches to deploying infrastructure with AWS, from the cons The first way most people experiment with AWS is via its web interface, the AWS Console. But using the Console is a highly manual process, and often works against automation or flexibility. -So if you’re not going to manage your AWS configurations manually, what should you do? Sadly, there are no simple, universal answers — each approach has pros and cons, and the approaches taken by different companies vary widely, and include directly using APIs (and building toolign on top yourself), using command-line tools, and using third-party tools and services. +So if you’re not going to manage your AWS configurations manually, what should you do? Sadly, there are no simple, universal answers — each approach has pros and cons, and the approaches taken by different companies vary widely, and include directly using APIs (and building tooling on top yourself), using command-line tools, and using third-party tools and services. ### AWS Console From 58e131b2674e1cb3b48e53df5528d7480043ddfb Mon Sep 17 00:00:00 2001 From: Manoj M J Date: Wed, 12 Oct 2016 12:09:50 +0530 Subject: [PATCH 014/140] Fix typo in Load Balancer section --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..8f0a610 100644 --- a/README.md +++ b/README.md @@ -845,7 +845,7 @@ Load Balancers - **CLBs and ALBs have many IPs:** Internally, an AWS load balancer is simply a collection of individual software load balancers hosted within EC2, with DNS load balancing traffic among them. The pool can contain many IPs, at least one per availability zone, and depending on traffic levels. They also support SSL termination, which is very convenient. - **Scaling:** CLBs and ALBs can scale to very high throughput, but scaling up is not instantaneous. If you’re expecting to be hit with a lot of traffic suddenly, it can make sense to load test them so they scale up in advance. You can also [contact Amazon](http://aws.amazon.com/articles/1636185810492479) and have them “pre-warm” the load balancer. - **Client IPs:** In general, if servers want to know true client IP addresses, load balancers must forward this information somehow. CLBs add the standard [X-Forwarded-For](https://en.wikipedia.org/wiki/X-Forwarded-For) header. When using an CLB as an HTTP load balancer, it’s possible to get the client’s IP address from this. -- **Using load balancers when deploying:** One common pattern is to swap instances in the load balancer after spinning up a new stack with your latest version, keep old stack running for one or two hours, and either flip back to old stack in case of problems or tear down it down. +- **Using load balancers when deploying:** One common pattern is to swap instances in the load balancer after spinning up a new stack with your latest version, keep old stack running for one or two hours, and either flip back to old stack in case of problems or tear it down. ### Load Balancer Gotchas and Limitations From f631ff1c709fd8bc2319f37d9258c26227587535 Mon Sep 17 00:00:00 2001 From: Martijn Dwars Date: Wed, 12 Oct 2016 09:20:01 +0200 Subject: [PATCH 015/140] Add EC2 gotcha about burst performance --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 44cb9f0..bf2f4a8 100644 --- a/README.md +++ b/README.md @@ -726,6 +726,7 @@ EC2 - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. +- 🔸Instances come in two types: Fixed Performance Instances (e.g. M3, C3, and R3) and Burstable Performance Instances (e.g. T2). Each T2 instance receives CPU Credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU Credits when they are idle, and use CPU credits when they are active. However, once it runs out of credits, you'll notice a severe degrade in performance. CPU-bound tasks (such as scientific computation) are therefore better suited for Fixed Performance Instances. AMIs ---- From 495d385ebcc33962955581be837b7cd9937554ac Mon Sep 17 00:00:00 2001 From: David Kocher Date: Wed, 12 Oct 2016 09:26:39 +0200 Subject: [PATCH 016/140] Revert removal of Transmit as a recommendation for basic usage. Add disclaimer about missing AWS4 signature and multipart upload support. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index b9754e4..b631a18 100644 --- a/README.md +++ b/README.md @@ -603,6 +603,7 @@ S3 - [**s4cmd**](https://github.com/bloomreach/s4cmd) is a replacement, with greater emphasis on performance via multi-threading, which is helpful for large files and large sets of files, and also offers Unix-like globbing support. - **GUI applications:** You may prefer a GUI, or wish to support GUI access for less technical users. Some options: - The [AWS Console](https://aws.amazon.com/console/) does offer a graphical way to use S3. Use caution telling non-technical people to use it, however, since without tight permissions, it offers access to many other AWS features. + - [Transmit](https://panic.com/transmit/) is a good option on OS X for basic use cases. Uses legacy AWS2 signatures for authentication and is missing multipart upload support. - [Cyberduck](https://cyberduck.io/) is a good option on OS X and Windows with support for multipart uploads, ACLs, versioning, lifecycle configuration, storage classes and server side encryption (SSE-S3 and SSE-KMS). - **S3 and CloudFront:** S3 is tightly integrated with the CloudFront CDN. See the CloudFront section for more information, as well as [S3 transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html). - **Static website hosting:** From b98d3495bea9c215b81ba7e40097515ea1e32c32 Mon Sep 17 00:00:00 2001 From: Steven Maude Date: Wed, 12 Oct 2016 09:29:39 +0100 Subject: [PATCH 017/140] Fix internal link to Learning section Thought my browser was broken. Turns out it was the link. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..a7030d0 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Table of Contents **AWS in General** - [General Information](#general-information) -- [Learning and Career Development](#learning-aws) +- [Learning and Career Development](#learning-and-career-development) - [Managing AWS](#managing-aws) - [Managing Servers and Applications](#managing-servers-and-applications) From 35603244276cb1767a79f3aabcf9c40cab1f8e9d Mon Sep 17 00:00:00 2001 From: Mikhail Advani Date: Wed, 12 Oct 2016 18:33:54 +0530 Subject: [PATCH 018/140] [#150] Gotcha while migrating your DB to RDS --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 44cb9f0..22ba4e5 100644 --- a/README.md +++ b/README.md @@ -958,6 +958,8 @@ RDS - ⏱RDS instances run on EBS volumes, and hence are constrained by the EBS performance. - 🔸Verify what database features you need, as not everything you might want is available on RDS. For example, if you are using Postgres, check the list of [supported features and extensions](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#SQLServer.Concepts.General.FeatureSupport). If the features you need aren’t supported by RDS, you’ll have to deploy your database yourself. +- 🔸**DB migration to RDS:** While importing your database into RDS ensure you take into consideration the maintenance window settings. If a backup is running at the same time, your import can take a considerable longer time than you would have expected. + DynamoDB -------- From 23fea1893124257f7c14f297fadd366dd43eff09 Mon Sep 17 00:00:00 2001 From: Chris Leyva Date: Wed, 12 Oct 2016 09:29:17 -0400 Subject: [PATCH 019/140] Added a missing word. Improving the readability of the document. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..b9bd4e3 100644 --- a/README.md +++ b/README.md @@ -704,7 +704,7 @@ EC2 - Selecting instance types is complex since there are so many types. Additionally, there are different generations, released [over the years](https://aws.amazon.com/blogs/aws/ec2-instance-history/). - 🔹Use the list at [**ec2instances.info**](http://www.ec2instances.info/) to review costs and features. [Amazon’s own list](https://aws.amazon.com/ec2/instance-types/) of instance types is hard to use, and doesn’t list features and price together, which makes it doubly difficult. - Prices vary a lot, so use [**ec2instances.info**](http://www.ec2instances.info/) to determine the set of machines that meet your needs and [**ec2price.com**](http://ec2price.com/) to find the cheapest type in the region you’re working in. Depending on the timing and region, it might be much cheaper to rent an instance with *more* memory or CPU than the bare minimum. -- [**Dedicated instances**](https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/) and [**dedicated hosts**](https://aws.amazon.com/ec2/dedicated-hosts/) are assigned hardware, instead of usual virtual instances. They more expensive than virtual instances but [can be preferable](https://aws.amazon.com/ec2/dedicated-hosts/) for performance, compliance, or licensing reasons. +- [**Dedicated instances**](https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/) and [**dedicated hosts**](https://aws.amazon.com/ec2/dedicated-hosts/) are assigned hardware, instead of usual virtual instances. They are more expensive than virtual instances but [can be preferable](https://aws.amazon.com/ec2/dedicated-hosts/) for performance, compliance, or licensing reasons. - **32 bit vs 64 bit:** A few micro, small, and medium instances are still available to use as 32-bit architecture. You’ll be using 64-bit EC2 (“amd64”) instances nowadays, though smaller instances still support 32 bit (“i386”). Use 64 bit unless you have legacy constraints or other good reasons to use 32. - **HVM vs PV:** There are two kinds of virtualization technology used by EC2, [hardware virtual machine (HVM) and paravirtual (PV)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html). Historically, PV was the usual type, but [now HVM is becoming the standard](https://www.opswat.com/blog/aws-2015-why-you-need-switch-pv-hvm). If you want to use the newest instance types, you must use HVM. See the [instance type matrix](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/) for details. - **Operating system:** To use EC2, you’ll need to pick a base operating system. It can be Windows or Linux, such as Ubuntu or [Amazon Linux](https://aws.amazon.com/amazon-linux-ami/). You do this with AMIs, which are covered in more detail in their own section below. From f593e559a17022ee966e8fc6763cf9e6d6336341 Mon Sep 17 00:00:00 2001 From: Andrew Lane Date: Wed, 12 Oct 2016 09:33:13 -0400 Subject: [PATCH 020/140] nit pick --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..d322585 100644 --- a/README.md +++ b/README.md @@ -145,7 +145,7 @@ General Information - Companies at (very) large scale may want to reduce costs by managing their own infrastructure. For example, [Dropbox migrated](https://news.ycombinator.com/item?id=11282948) to their own infrastructure. - Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys. - Traditional managed hosting providers such as [Rackspace](https://www.rackspace.com/) offer cloud solutions as well. -- 🚪**AWS vs. PaaS:** If your goal is just to put up a single service that does something relatively simple, and you’re trying to minimize time managing operations engineering, consider a [platform-as-a-service](https://en.wikipedia.org/wiki/Platform_as_a_service) such as [Heroku](https://www.heroku.com/) The AWS approach to PaaS, Elastic Beanstalk, is arguably more complex, especially for simple use cases. +- 🚪**AWS vs. PaaS:** If your goal is just to put up a single service that does something relatively simple, and you’re trying to minimize time managing operations engineering, consider a [platform-as-a-service](https://en.wikipedia.org/wiki/Platform_as_a_service) such as [Heroku](https://www.heroku.com/). The AWS approach to PaaS, Elastic Beanstalk, is arguably more complex, especially for simple use cases. - 🚪**AWS vs. web hosting:** If your main goal is to host a website or blog, and you don’t expect to be building an app or more complex service, you may wish consider one of the myriad of [web hosting services](https://www.google.com/search?q=web+hosting). - 🚪**AWS vs. managed hosting:** Traditionally, many companies pay [managed hosting](https://en.wikipedia.org/wiki/Dedicated_hosting_service) providers to maintain physical servers for them, then build and deploy their software on top of the rented hardware. This makes sense for businesses who want direct control over hardware, due to legacy, performance, or special compliance constraints, but is usually considered old fashioned or unnecessary by many developer-centric startups and younger tech companies. - **Complexity:** AWS will let you build and scale systems to the size of the largest companies, but the complexity of the services when used at scale requires significant depth of knowledge and experience. Even very simple use cases often require more knowledge to do “right” in AWS than in a simpler environment like Heroku or Digital Ocean. (This guide may help!) From a87c472570c7eebcfe660d1eeeaa6d85bb56034b Mon Sep 17 00:00:00 2001 From: Harold Smith Date: Wed, 12 Oct 2016 09:20:58 -0500 Subject: [PATCH 021/140] Updated certifications portion of README to make more sense, and follow along with AWS docs Updated certifications portion of README to make more sense, and follow along with AWS docs. --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 44cb9f0..85dfd1e 100644 --- a/README.md +++ b/README.md @@ -391,10 +391,11 @@ Learning and Career Development ### Certifications - **Certifications:** AWS offers [**certifications**](https://aws.amazon.com/certification/) for IT professionals who want to demonstrate their knowledge. They are: - - Certified Solutions Architect [Associate](https://aws.amazon.com/certification/certified-solutions-architect-associate/) and [Professional](https://aws.amazon.com/certification/certified-solutions-architect-professional/), - [Certified Developer Associate](https://aws.amazon.com/certification/certified-developer-associate/) - - [Certified SysOps Administrator Associate](https://aws.amazon.com/certification/certified-sysops-admin-associate/) - - [Certified Certified DevOps Engineer Professional](https://aws.amazon.com/certification/certified-devops-engineer-professional/) + - [Certified Solutions Architect Associate](https://aws.amazon.com/certification/certified-solutions-architect-associate/) + - [Certified Developer Associate](https://aws.amazon.com/certification/certified-developer-associate/) + - [Certified SysOps Administrator Associate](https://aws.amazon.com/certification/certified-sysops-admin-associate/) + - [Certified Solutions Architect Professional](https://aws.amazon.com/certification/certified-solutions-architect-professional/), + - [Certified DevOps Engineer Professional](https://aws.amazon.com/certification/certified-devops-engineer-professional/) - **Getting certified:** If you’re interested in studying for and getting certifications, [this practical overview](https://gist.github.com/leonardofed/bbf6459ad154ad5215d354f3825435dc) tells you a lot of what you need to know. The official page is [here](https://aws.amazon.com/training/) and there is an [FAQ](https://aws.amazon.com/certification/faqs/). - **Do you need a certification?** Especially in consulting companies or when working in key tech roles in large non-tech companies, certifications are important credentials. In others, including in many tech companies and startups, certifications are not common or considered necessary. (In fact, fairly or not, some Silicon Valley hiring managers and engineers see them as a “negative” signal on a resume.) From 1a7b234ce5dda6eaa8479b45d49628a6dc767880 Mon Sep 17 00:00:00 2001 From: Harold Smith Date: Wed, 12 Oct 2016 09:24:01 -0500 Subject: [PATCH 022/140] Missed a comma --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 85dfd1e..c4a45cc 100644 --- a/README.md +++ b/README.md @@ -390,11 +390,11 @@ Learning and Career Development ### Certifications -- **Certifications:** AWS offers [**certifications**](https://aws.amazon.com/certification/) for IT professionals who want to demonstrate their knowledge. They are: +- **Certifications:** AWS offers [**certifications**](https://aws.amazon.com/certification/) for IT professionals who want to demonstrate their knowledge. - [Certified Solutions Architect Associate](https://aws.amazon.com/certification/certified-solutions-architect-associate/) - [Certified Developer Associate](https://aws.amazon.com/certification/certified-developer-associate/) - [Certified SysOps Administrator Associate](https://aws.amazon.com/certification/certified-sysops-admin-associate/) - - [Certified Solutions Architect Professional](https://aws.amazon.com/certification/certified-solutions-architect-professional/), + - [Certified Solutions Architect Professional](https://aws.amazon.com/certification/certified-solutions-architect-professional/) - [Certified DevOps Engineer Professional](https://aws.amazon.com/certification/certified-devops-engineer-professional/) - **Getting certified:** If you’re interested in studying for and getting certifications, [this practical overview](https://gist.github.com/leonardofed/bbf6459ad154ad5215d354f3825435dc) tells you a lot of what you need to know. The official page is [here](https://aws.amazon.com/training/) and there is an [FAQ](https://aws.amazon.com/certification/faqs/). - **Do you need a certification?** Especially in consulting companies or when working in key tech roles in large non-tech companies, certifications are important credentials. In others, including in many tech companies and startups, certifications are not common or considered necessary. (In fact, fairly or not, some Silicon Valley hiring managers and engineers see them as a “negative” signal on a resume.) From a964b7214948fb2964a1d303521c1cb5682f572e Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 09:35:17 -0700 Subject: [PATCH 023/140] Updated the Spot management gotchas section based on Kazu's feedback. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 67f510d..7670631 100644 --- a/README.md +++ b/README.md @@ -1350,7 +1350,7 @@ Billing and Cost Management - Typical pooling implementations give anywhere between 45-60% cost optimizations and 40% reduction in spot instance creationg time. - An excellent example of Pooling implementation described by Netflix ([part1](http://techblog.netflix.com/2015/09/creating-your-own-ec2-spot-market.html), [part2](http://techblog.netflix.com/2015/11/creating-your-own-ec2-spot-market-part-2.html)\) - **Spot management gotchas** - - 🔸**Lifetime:** There is no guarantee for the lifetime of a Spot instance. It is purely based on bidding. If anyone outbids your price, the instance is taken away. Spot is not suitable for time sensitive jobs that have strong SLA. Instances will fail based on demand for Spot at that time. AWS does not send any signal that the instance is going away, except for the fact that it is going down. That makes it hard to figure out why the instance(s) went down. + - 🔸**Lifetime:** There is [no guarantee](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) for the lifetime of a Spot instance. It is purely based on bidding. If anyone outbids your price, the instance is taken away. Spot is not suitable for time sensitive jobs that have strong SLA. Instances will fail based on demand for Spot at that time. AWS provides a [two-minute warning](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#spot-instance-termination-notices) before Amazon EC2 must terminate your Spot instance. - 🔹**API return data:** - The Spot price API returns Spot prices of varying granularity depending on the time range specified in the api call.E.g If the last 10 min worth of history is requested, the data is more fine grained. If the last 2 day worth of history is requested, the data is more coarser. Do not assume you will get all the data points. There **will** be skipped intervals. - ❗**Lifecycle management:** Do not attempt any fancy Spot management unless absolutely necessary. If your entire usage is only a few machines and your cost is acceptable and your failure rate is lower, do not attempt to optimize. The pain for building/maintaining it is not worth just a few hundred dollar savings. - **Reserved Instances:** allow you to get significant discounts on EC2 compute hours in return for a commitment to pay for instance hours of a specific instance type in a specific AWS region and availability zone for a pre-established time frame (1 or 3 years). Further discounts can be realized through “partial” or “all upfront” payment options. From 4446853cd6f831dcfdc9db5680ab2838dc9a27a7 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Wed, 12 Oct 2016 19:49:00 +0200 Subject: [PATCH 024/140] Add info about release of Device Farm --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 67f510d..f0fe472 100644 --- a/README.md +++ b/README.md @@ -308,6 +308,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | 🐥[Data Pipeline](https://aws.amazon.com/releasenotes/AWS-Data-Pipeline?browse=1) | 2015-10 | General | | 🐥[Elasticsearch](https://aws.amazon.com/releasenotes/Amazon-Elasticsearch-Service?browse=1) | 2015-10 | General | | 🐥[Service Catalog](https://aws.amazon.com/releasenotes/AWS-Service-Catalog?browse=1) | 2015-07 | General | +| 🐥[Device Farm](https://aws.amazon.com/releasenotes/AWS-Device-Farm?browse=1) | 2015-07 | General | | 🐥[CodePipeline](https://aws.amazon.com/releasenotes/AWS-CodePipeline?browse=1) | 2015-07 | General | | 🐥[CodeCommit](https://aws.amazon.com/releasenotes/AWS-CodeCommit?browse=1) | 2015-07 | General | | 🐥[API Gateway](https://aws.amazon.com/releasenotes/Amazon-API-Gateway?browse=1) | 2015-07 | General | From 59e85a4908c5c871fbe3de3625423752ec2b1171 Mon Sep 17 00:00:00 2001 From: Brady Dowling Date: Wed, 12 Oct 2016 12:17:21 -0700 Subject: [PATCH 025/140] Fix some awkward/incorrect grammar --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 420a358..d4f7b1d 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,7 @@ General Information - [EMR](#emr): Managed Hadoop - [Elasticsearch](https://aws.amazon.com/elasticsearch-service/): Managed Elasticsearch - [ElastiCache](https://aws.amazon.com/elasticache/): Managed Redis and Memcached -- **Optional but important infrastructure:** These are key and useful infrastructure components that are less widely known and used. You may have legitimate reasons to prefer alternatives, so evaluate with care you to be sure they fit your needs: +- **Optional but important infrastructure:** These are key and useful infrastructure components that are less widely known and used. You may have legitimate reasons to prefer alternatives, so evaluate with care to be sure they fit your needs: - ⛓[Lambda](#lambda): Running small, fully managed tasks “serverless” - [CloudTrail](https://aws.amazon.com/cloudtrail/): AWS API logging and audit (often neglected but important) - ⛓🕍[CloudFormation](#cloudformation): Templatized configuration of collections of AWS resources @@ -452,7 +452,7 @@ So if you’re not going to manage your AWS configurations manually, what should ### General Visibility -- 🔹[**Tagging resources**](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) is an essential practice, especially as organizations grow, to better understand your resource usage. For example, you can through automation or convention add tags: +- 🔹[**Tagging resources**](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) is an essential practice, especially as organizations grow, to better understand your resource usage. For example, through automation or convention, you can add tags: - For the org or developer that “owns” that resource - For the product that resource supports - To label lifecycles, such as temporary resources or one that should be deprovisioned in the future From c2c1bde2330b404ef0e90a2ae653b840b1728bb7 Mon Sep 17 00:00:00 2001 From: Adam Nelson Date: Thu, 13 Oct 2016 07:05:37 +1100 Subject: [PATCH 026/140] Added tip on pre-empting instance retirement by stopping and starting the instance. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2a839f6..cf09a61 100644 --- a/README.md +++ b/README.md @@ -722,7 +722,7 @@ EC2 - ❗Never use ssh passwords. Just don’t do it; they are too insecure, and consequences of compromise too severe. Use keys instead. [Read up on this](https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2) and fully disable ssh password access to your ssh server by making sure 'PasswordAuthentication no' is in your /etc/ssh/sshd_config file. If you’re careful about managing ssh private keys everywhere they are stored, it is a major improvement on security over password-based authentication. - 🔸For all [newer instance types](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/), when selecting the AMI to use, be sure you select the HVM AMI, or it just won’t work. - ❗When creating an instance and using a new ssh key pair, [make sure the ssh key permissions are correct](http://stackoverflow.com/questions/1454629/aws-ssh-access-permission-denied-publickey-issue). -- 🔸Sometimes certain EC2 instances can get scheduled for retirement by AWS due to “detected degradation of the underlying hardware,” in which case you are given a couple of weeks to migrate to a new instance. +- 🔸Sometimes certain EC2 instances can get scheduled for retirement by AWS due to “detected degradation of the underlying hardware,” in which case you are given a couple of weeks to migrate to a new instance. However you can typically stop and then start the instance which moves it to heathly host hardware, giving you control over timing of this event. Note however that you will lose any instance store volume data ([ephemeral drives](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)), but data on EBS volumes will be fine. - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. From eccbd748d60a1ec6543c317c1073d9531e66d4a5 Mon Sep 17 00:00:00 2001 From: Adam Nelson Date: Thu, 13 Oct 2016 08:23:14 +1100 Subject: [PATCH 027/140] Gotcha for Cloudfront and host headers. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index cf09a61..2fb042a 100644 --- a/README.md +++ b/README.md @@ -1182,6 +1182,7 @@ CloudFront ### CloudFront Gotchas and Limitations - If using S3 as a backing store, remember that the endpoints for website hosting and for general S3 are different. Example: “bucketname.s3.amazonaws.com” is a standard S3 serving endpoint, but to have redirect and error page support, you need to use the website hosting endpoint listed for that bucket, e.g. “bucketname.s3-website-us-east-1.amazonaws.com” (or the appropriate region). +- 🔹By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. DirectConnect ------------- From 71f9c13ef41535e7406e0e90a032e262cb72b274 Mon Sep 17 00:00:00 2001 From: Adam Nelson Date: Thu, 13 Oct 2016 08:33:57 +1100 Subject: [PATCH 028/140] Pull request comments - expanded on EBS root volumes and IP addresses. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cf09a61..0854d05 100644 --- a/README.md +++ b/README.md @@ -722,7 +722,9 @@ EC2 - ❗Never use ssh passwords. Just don’t do it; they are too insecure, and consequences of compromise too severe. Use keys instead. [Read up on this](https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2) and fully disable ssh password access to your ssh server by making sure 'PasswordAuthentication no' is in your /etc/ssh/sshd_config file. If you’re careful about managing ssh private keys everywhere they are stored, it is a major improvement on security over password-based authentication. - 🔸For all [newer instance types](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/), when selecting the AMI to use, be sure you select the HVM AMI, or it just won’t work. - ❗When creating an instance and using a new ssh key pair, [make sure the ssh key permissions are correct](http://stackoverflow.com/questions/1454629/aws-ssh-access-permission-denied-publickey-issue). -- 🔸Sometimes certain EC2 instances can get scheduled for retirement by AWS due to “detected degradation of the underlying hardware,” in which case you are given a couple of weeks to migrate to a new instance. However you can typically stop and then start the instance which moves it to heathly host hardware, giving you control over timing of this event. Note however that you will lose any instance store volume data ([ephemeral drives](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)), but data on EBS volumes will be fine. +- 🔸Sometimes certain EC2 instances can get scheduled for retirement by AWS due to “detected degradation of the underlying hardware,” in which case you are given a couple of weeks to migrate to a new instance + - If your instance root device is an EBS volume, you can typically stop and then start the instance which moves it to heathly host hardware, giving you control over timing of this event. Note however that you will lose any instance store volume data ([ephemeral drives](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)) if your instance type has instance store volumes. + - The instance public IP (if it has one) will likely change unless you're using Elastic IPs. This could be a problem if other systems depend on the IP address. - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. From 6ae8e8664e07aeb82f0384b34d4176c701af9d80 Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 15:20:55 -0700 Subject: [PATCH 029/140] Added a legend icon for the undocumented features. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 7670631..9737b87 100644 --- a/README.md +++ b/README.md @@ -106,6 +106,7 @@ Legend - 🔹 Important or often overlooked tip - ❗ Gotcha or warning (where risks or time or resource costs are significant) - 🔸 Limitation or quirk (where it’s not quite so bad) +- 📜 Undocumented feature (folklore) - 🐥 Relatively new (and perhaps immature) services or features - ⏱ Performance discussions - ⛓ Lock-in: Products or decisions that are likely to tie you to AWS in a new or significant way — that is, later moving to a non-AWS alternative would be costly in terms of engineering effort From 3c9e4c60d8756b45aace65431c539d0cd6e3b920 Mon Sep 17 00:00:00 2001 From: Rich Adams Date: Wed, 12 Oct 2016 09:45:52 -0700 Subject: [PATCH 030/140] Adding information about IAM session keys requiring a session token parameter. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 67f510d..6768958 100644 --- a/README.md +++ b/README.md @@ -516,6 +516,7 @@ We cover security basics first, since configuring user accounts is something you - IAM manages various kinds of authentication, for both users and for software services that may need to authenticate with AWS, including: - [**Passwords**](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords.html) to log into the console. These are a username and password for real users. - [**Access keys**](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html), which you may use with command-line tools. These are two strings, one the “id”, which is an upper-case alphabetic string of the form 'AXXXXXXXXXXXXXXXXXXX', and the other is the secret, which is a 40-character mixed-case base64-style string. These are often set up for services, not just users. + - 📜 Access keys that start with AKIA are normal keys. Access keys that start with ASIA are session/temporary keys from STS, and will require an additional "SessionToken" parameter to be sent along with the id and secret. - [**Multi-factor authentication (MFA)**](https://aws.amazon.com/iam/details/mfa/), which is the highly recommended practice of using a keychain fob or smartphone app as a second layer of protection for user authentication. - IAM allows complex and fine-grained control of permissions, dividing users into groups, assigning permissions to roles, and so on. There is a [policy language](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) that can be used to customize security policies in a fine-grained way. - 🔸The policy language has a complex and error-prone JSON syntax that’s quite confusing, so unless you are an expert, it is wise to base yours off trusted examples or AWS’ own pre-defined [managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html). From 7bc4ae8b955ea450152a0c43bb27adade8119d96 Mon Sep 17 00:00:00 2001 From: Uli Stroetz Date: Wed, 12 Oct 2016 18:40:03 -0400 Subject: [PATCH 031/140] Add DynamoDB time series data gotcha --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 71eb9d4..a4a9eb6 100644 --- a/README.md +++ b/README.md @@ -625,7 +625,7 @@ S3 - S3 has a [static website hosting option](http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) that is simply a setting that enables configurable HTTP index and error pages and [HTTP redirect support](http://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html) to [public content](http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteAccessPermissionsReqd.html) in S3. It’s a simple way to host static assets or a fully static website. - Consider using CloudFront in front of most or all assets: - Like any CDN, CloudFront improves performance significantly. - - 🔸SSL is only supported on the built-in amazonaws.com domain for S3. S3 supports serving these sites through a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html), but [not over SSL on a custom domain](http://stackoverflow.com/questions/11201316/how-to-configure-ssl-for-amazon-s3-bucket). However, [CloudFront allows you to serve a custom domain over https](http://docs.aws.amazon.com/acm/latest/userguide/gs-cf.html). Amazon provides free SNI SSL/TLS certificates via Amazon Certificate Manager. [SNI does not work on very outdated browsers/operating systems](https://en.wikipedia.org/wiki/Server_Name_Indication#Support). Alternatively, you can provide your own certificate to use on CloudFront to support all browsers/operating systems. + - 🔸SSL is only supported on the built-in amazonaws.com domain for S3. S3 supports serving these sites through a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html), but [not over SSL on a custom domain](http://stackoverflow.com/questions/11201316/how-to-configure-ssl-for-amazon-s3-bucket). However, [CloudFront allows you to serve a custom domain over https](http://docs.aws.amazon.com/acm/latest/userguide/gs-cf.html). Amazon provides free SNI SSL/TLS certificates via Amazon Certificate Manager. [SNI does not work on very outdated browsers/operating systems](https://en.wikipedia.org/wiki/Server_Name_Indication#Support). Alternatively, you can provide your own certificate to use on CloudFront to support all browsers/operating systems. - 🔸If you are including resources across domains, such as fonts inside CSS files, you may need to [configure CORS](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html) for the bucket serving those resources. - Since pretty much everything is moving to SSL nowadays, and you likely want control over the domain, you probably want to set up CloudFront with your own certificate in front of S3 (and to ignore the [AWS example on this](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) as it is non-SSL only). - That said, if you do, you’ll need to think through invalidation or updates on CloudFront. You may wish to [include versions or hashes in filenames](https://abhishek-tiwari.com/post/CloudFront-design-patterns-and-best-practices) so invalidation is not necessary. @@ -665,7 +665,7 @@ S3 - 🔸After uploading, any change that you make to the object causes a full rewrite of the object, so avoid appending-like behavior with regular files. - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). -- 🔸**US Standard region:** Most S3 endpoints match the region they’re in, with the exception of the us-east-1 region, which is called 'us-standard' in S3 terminology. This region is also the only region that is replicated across coasts. As a result, latency varies more in this region than in others. You can minimize latency from us-east-1 by using *[s3-external-1.amazonaws.com](http://s3-external-1.amazonaws.com/)*. +- 🔸**US Standard region:** Most S3 endpoints match the region they’re in, with the exception of the us-east-1 region, which is called 'us-standard' in S3 terminology. This region is also the only region that is replicated across coasts. As a result, latency varies more in this region than in others. You can minimize latency from us-east-1 by using *[s3-external-1.amazonaws.com](http://s3-external-1.amazonaws.com/)*. ### Storage Durability, Availability, and Price @@ -985,6 +985,7 @@ DynamoDB - 🔸 DynamoDB doesn’t provide an easy way to bulk-load data (it is possible through [Data Pipeline](http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html), and this has some [unfortunate consequences](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.AvoidExcessivePTIncreases). Since you need to use the regular service APIs to update existing or create new rows, it is common to temporarily turn up a destination table’s write throughput to speed import. But when the table’s write capacity is increased, DynamoDB may do an irreversible split of the partitions underlying the table, spreading the total table capacity evenly across the new generation of tables. Later, if the capacity is reduced, the capacity for each partition is also reduced but the total number of partitions is not, leaving less capacity for each partition. This leaves the table in a state where it much easier for hotspots to overwhelm individual partitions. - It is important to make sure that DynamoDB [resource limits](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-data-types) are compatible with your dataset and workload. For example, the maximum size value that can be added to a DynamoDB table is 400 KB (larger items can be stored in S3 and a URL stored in DynamoDB). +- Dealing with time series data in DynamoDB can be challenging. A Global Secondary Index together with down sampling timestamps can be a possible solution as explained [here](https://blogs.aws.amazon.com/bigdata/post/Tx3KPZDXIBJEQ4B/Scaling-Writes-on-Amazon-DynamoDB-Tables-with-Global-Secondary-Indexes). ECS --- @@ -1282,7 +1283,7 @@ This section covers tips and information on achieving [high availability](https: ### High Availability Gotchas and Limitations - **AZ naming** differs from one customer account to the next. Your “us-west-1a” is not the same as another customer’s “us-west-1a” — the letters are assigned to physical AZs randomly per account. This can also be a gotcha if you have multiple AWS accounts. -- **Cross-AZ traffic** is not free. At large scale, the costs add up to a significant amount of money. If possible, optimize your traffic to stay within the same AZ as much as possible. +- **Cross-AZ traffic** is not free. At large scale, the costs add up to a significant amount of money. If possible, optimize your traffic to stay within the same AZ as much as possible. Billing and Cost Management --------------------------- @@ -1336,7 +1337,7 @@ Billing and Cost Management - Profile your application to figure out its runtime characteristics. That would help give an understanding of the minimum cpu, memory, disk required. Having this information is critical before you try to optimize spot costs. - Once you know the minimum application requirements, instead of resorting to fixed instance types, you can bid across a variety of instance types (that gives you higher chances of getting a spot instance to run your application).E.g., If you know that 4 cpu cores are enough for your job, you can choose any instance type that is equal or above 4 cores and that has the least Spot price based on history. This helps you bid for instances with greater discount (less demand at that point). - **Spot price monitoring and intelligence:** - - Spot Instance prices fluctuate depending on instance types, time of day, region and availability zone. The AWS CLI tools and API allow you to describe Spot price metadata given time, instance type, and region/AZ. + - Spot Instance prices fluctuate depending on instance types, time of day, region and availability zone. The AWS CLI tools and API allow you to describe Spot price metadata given time, instance type, and region/AZ. - Based on history of Spot instance prices, you could potentially build a myriad of algorithms that would help you to pick an instance type that either - optimizes cost - maximizes availability From a565cf90f51acbe6c87eb55a18abbb59a083d5a8 Mon Sep 17 00:00:00 2001 From: Adam Nelson Date: Thu, 13 Oct 2016 10:38:10 +1100 Subject: [PATCH 032/140] Changed to orange diamond. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0f6ecfc..5a8163d 100644 --- a/README.md +++ b/README.md @@ -1184,7 +1184,7 @@ CloudFront ### CloudFront Gotchas and Limitations - If using S3 as a backing store, remember that the endpoints for website hosting and for general S3 are different. Example: “bucketname.s3.amazonaws.com” is a standard S3 serving endpoint, but to have redirect and error page support, you need to use the website hosting endpoint listed for that bucket, e.g. “bucketname.s3-website-us-east-1.amazonaws.com” (or the appropriate region). -- 🔹By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. +- 🔸By default, CloudFront will not forward HTTP Host: headers through to your origin servers. This can be problematic for your origin if you run multiple sites switched with host headers. You can [enable host header forwarding](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior) in the default cache behavior settings. DirectConnect ------------- From cc653eaf5f2bc99d6b50d99603b4d6313c625a71 Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 17:56:53 -0700 Subject: [PATCH 033/140] Started the EMR gotchas section. --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f325e36..d02cb1a 100644 --- a/README.md +++ b/README.md @@ -1253,11 +1253,16 @@ EMR ### EMR Tips - EMR relies on many versions of Hadoop and other supporting software. Be sure to check [which versions are in use](https://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-components.html). -- 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. - ⏱Off-the-shelf EMR and Hadoop can have significant overhead when compared with efficient processing on a single machine. If your data is small and performance matters, you may wish to consider alternatives, as [this post](http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html) illustrates. - Python programmers may want to take a look at Yelp’s [mrjob](https://github.com/Yelp/mrjob). - It takes time to tune performance of EMR jobs, which is why third-party services such as [Qubole’s data service](https://www.qubole.com/mapreduce-as-a-service/) are gaining popularity as ways to improve performance or reduce costs. +### EMR Gotchas and Limitations +- 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. +- 💸 Beware of "double-dipping". With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. + + + High Availability ----------------- From 658f8231325054bdd2e8eb3a722868a75899d54c Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 18:06:24 -0700 Subject: [PATCH 034/140] Fixed the quotes --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d02cb1a..5e54688 100644 --- a/README.md +++ b/README.md @@ -1259,7 +1259,7 @@ EMR ### EMR Gotchas and Limitations - 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. -- 💸 Beware of "double-dipping". With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. +- 💸 Beware of “double-dipping”. With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. From 8b6745b3f6612d9e61a4cab9e28091ca1f95b3d5 Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 18:35:08 -0700 Subject: [PATCH 035/140] Added the automatic compression tip to the Redshift section --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5e54688..62051ec 100644 --- a/README.md +++ b/README.md @@ -1224,6 +1224,7 @@ Redshift - [Top 10 Performance Tuning Techniques for Amazon Redshift](https://blogs.aws.amazon.com/bigdata/post/Tx31034QG0G3ED1/Top-10-Performance-Tuning-Techniques-for-Amazon-Redshift) provides an excellent list of performance tuning techniques. - [Amazon Redshift Utils](https://github.com/awslabs/amazon-redshift-utils) contains useful utilities, scripts and views to simplify Redshift ops. - [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/t_Reclaiming_storage_space202.html) regularly following a significant number of deletes or updates to reclaim space and improve query performance. +- Redshift provides various [column compression](http://docs.aws.amazon.com/redshift/latest/dg/t_Compressing_data_on_disk.html) options to optimize the stored data size. AWS strongly encourages to use [automatic compression](http://docs.aws.amazon.com/redshift/latest/dg/c_Loading_tables_auto_compress.html) at the COPY stage, when Redshift uses a sample of the data being ingested to analyze the column compression options. However, automatic compression can only be applied to an empty table with no data. Therefore, make sure the initial load batch is big enough to provide Redshift with a representative sample of the data (the default sample size is 100000 rows). ### Redshift Gotchas and Limitations From 100fbdfd592c4eb5208c79c0e7059533ace7a96f Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 18:46:03 -0700 Subject: [PATCH 036/140] Fixed wording. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 62051ec..bdcf736 100644 --- a/README.md +++ b/README.md @@ -1224,7 +1224,7 @@ Redshift - [Top 10 Performance Tuning Techniques for Amazon Redshift](https://blogs.aws.amazon.com/bigdata/post/Tx31034QG0G3ED1/Top-10-Performance-Tuning-Techniques-for-Amazon-Redshift) provides an excellent list of performance tuning techniques. - [Amazon Redshift Utils](https://github.com/awslabs/amazon-redshift-utils) contains useful utilities, scripts and views to simplify Redshift ops. - [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/t_Reclaiming_storage_space202.html) regularly following a significant number of deletes or updates to reclaim space and improve query performance. -- Redshift provides various [column compression](http://docs.aws.amazon.com/redshift/latest/dg/t_Compressing_data_on_disk.html) options to optimize the stored data size. AWS strongly encourages to use [automatic compression](http://docs.aws.amazon.com/redshift/latest/dg/c_Loading_tables_auto_compress.html) at the COPY stage, when Redshift uses a sample of the data being ingested to analyze the column compression options. However, automatic compression can only be applied to an empty table with no data. Therefore, make sure the initial load batch is big enough to provide Redshift with a representative sample of the data (the default sample size is 100000 rows). +- Redshift provides various [column compression](http://docs.aws.amazon.com/redshift/latest/dg/t_Compressing_data_on_disk.html) options to optimize the stored data size. AWS strongly encourages users to use [automatic compression](http://docs.aws.amazon.com/redshift/latest/dg/c_Loading_tables_auto_compress.html) at the COPY stage, when Redshift uses a sample of the data being ingested to analyze the column compression options. However, automatic compression can only be applied to an empty table with no data. Therefore, make sure the initial load batch is big enough to provide Redshift with a representative sample of the data (the default sample size is 100000 rows). ### Redshift Gotchas and Limitations From 7f0eeb0c29bbf7f1566999896d8d10d54ce3cedb Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Wed, 12 Oct 2016 19:43:10 -0700 Subject: [PATCH 037/140] Add EMR gotchas section to TOC --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 31118d5..dae43d0 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ Table of Contents | [CloudFront](#cloudfront) | [📗](#cloudfront-basics) | [📘](#cloudfront-tips) | [📙](#cloudfront-gotchas-and-limitations) | | [DirectConnect](#directconnect) | [📗](#directconnect-basics) | [📘](#directconnect-tips) | | | [Redshift](#redshift) | [📗](#redshift-basics) | [📘](#redshift-tips) | [📙](#redshift-gotchas-and-limitations) | -| [EMR](#emr) | [📗](#emr-basics) | [📘](#emr-tips) | | +| [EMR](#emr) | [📗](#emr-basics) | [📘](#emr-tips) | [📙](#emr-gotchas-and-limitations) | **Special Topics** From 89cbcb9f7cc944b786e69ceea0eee2f99c3bc71f Mon Sep 17 00:00:00 2001 From: max Date: Wed, 12 Oct 2016 19:48:26 -0700 Subject: [PATCH 038/140] Fixed #179 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bdcf736..58fde65 100644 --- a/README.md +++ b/README.md @@ -665,7 +665,7 @@ S3 - 🔸After uploading, any change that you make to the object causes a full rewrite of the object, so avoid appending-like behavior with regular files. - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). -- 🔸**US Standard region:** Most S3 endpoints match the region they’re in, with the exception of the us-east-1 region, which is called 'us-standard' in S3 terminology. This region is also the only region that is replicated across coasts. As a result, latency varies more in this region than in others. You can minimize latency from us-east-1 by using *[s3-external-1.amazonaws.com](http://s3-external-1.amazonaws.com/)*. +- 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 Regions now support read-after-write consistency. Amazon S3 also renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions. ### Storage Durability, Availability, and Price From 75de77bef296c2b21f91cbbedb882efe8e3f2ea8 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Wed, 12 Oct 2016 20:11:38 -0700 Subject: [PATCH 039/140] Change 3rd-party to third-party for consistency --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dae43d0..660844b 100644 --- a/README.md +++ b/README.md @@ -1224,7 +1224,7 @@ Redshift - Although Redshift is mostly Postgres-compatible, its SQL dialect and performance profile are different. - Redshift supports only [12 primitive data types](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html). ([List of unsupported Postgres types](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-datatypes.html)\) - It has a leader node and computation nodes (the leader node distributes queries to the computation ones). Note that some functions [can be executed only on the lead node.](https://docs.aws.amazon.com/redshift/latest/dg/c_SQL_functions_leader_node_only.html) -- Major 3rd-party BI tools support Redshift integration (see [Quora](https://www.quora.com/Which-BI-visualisation-solution-goes-best-with-Redshift)). +- Major third-party BI tools support Redshift integration (see [Quora](https://www.quora.com/Which-BI-visualisation-solution-goes-best-with-Redshift)). - [Top 10 Performance Tuning Techniques for Amazon Redshift](https://blogs.aws.amazon.com/bigdata/post/Tx31034QG0G3ED1/Top-10-Performance-Tuning-Techniques-for-Amazon-Redshift) provides an excellent list of performance tuning techniques. - [Amazon Redshift Utils](https://github.com/awslabs/amazon-redshift-utils) contains useful utilities, scripts and views to simplify Redshift ops. - [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/t_Reclaiming_storage_space202.html) regularly following a significant number of deletes or updates to reclaim space and improve query performance. From 94efd2185e79af075f8d8d3062b8cd88e3874b5e Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 12 Oct 2016 21:08:35 -0700 Subject: [PATCH 040/140] Update contributing notes. --- CONTRIBUTING.md | 41 +++++++++++++++++++++++++++++++---------- README.md | 6 ++++-- 2 files changed, 35 insertions(+), 12 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 13b34a8..77de110 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,30 +1,35 @@ Contributing ------------ -Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly credit all contributors and authors. Here are few notes before you jump in. +Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly [credit](/AUTHORS.md) all contributors. Here are few notes before you jump in. [![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) -The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) or [**add to our list of common questions**](https://airtable.com/shrXZ61VrovWfXYBg), which help guide us in ways to improve the guide. +### Please Help -[**File issues**](https://github.com/open-guides/og-aws/issues) if it’s clear what needs to be improved. (For less well-defined issues, discussing first on Slack may be helpful.) +If you’ve found this guide useful, you have many ways to help: -[**Pull requests**](https://github.com/open-guides/og-aws/pulls) with changes are even better. Please keep them small and focused, so we can add items individually, and review the conventions below. (Again, Slack discussion can help.) +- The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) or [**add to our list of common questions**](https://airtable.com/shrXZ61VrovWfXYBg), which helps the community and guides what contributors can focus on. We encourage you to ask AWS questions and help others! +- [**File issues**](https://github.com/open-guides/og-aws/issues) if it’s clear something needs to be improved and you’re not able to make a pull request. +- [**Pull requests**](https://github.com/open-guides/og-aws/pulls) with changes are always welcome. Please keep them small and focused, so we can add items individually, and review the conventions below. If you want to make a larger change, try to discuss it in Slack. +- [**Review**] or [**comment**] on existing issues and pull requests if you have expertise. +- If you have deep expertise, consider offering to be an **editor** or **expert**. Editors and experts are assigned roles that [help us review the guide](#editorial-process). Join Slack to discuss this. -Please review current issues and pull requests to avoid duplication. -We gladly [credit](/AUTHORS.md) all contributors. +### Pull Request Etiquette -Finally, if an issue isn’t appropriate, or if you might have skills and inclination to help improve the guide in a more substantial way, please shoot an e-mail to [@jlevy](https://github.com/jlevy). +- Keep changes as small as is practical. Do not make changes to multiple sections at once, alter whitespace in broad ways, etc. +- Neutrality: If you have an affiliation related to what you are changing, please disclose it. +- Please do your best to review current issues and pull requests to avoid duplication. -### Style +### Writing Style -- **Use references:** If you are adding an item, whenever possible, try to add a link or reference to relevant discussion or reference pages. +- **Link to references:** If you are adding an item, whenever possible, try to add a link or reference to relevant discussion or reference pages. - **Be brief:** Avoid long expository paragraphs; it’s better to link to a blog. (We are open to linking to your own blog, if it’s the best source.) - **Include opinions and common practice:** Thoughtful opinion is helpful. If there are multiple conventions or ideas on something held by experts, mention the different ones. - **Clarity:** Strive for consistency with conventions listed here, but clarity is most important. -### Conventions +### Writing Conventions When you contribute, keep in mind these conventions: @@ -48,3 +53,19 @@ When you contribute, keep in mind these conventions: - Note we try to make sections uniquely titled, so GitHub links to Markdown section anchors don’t collide and are stable. Note we keep consistent formatting in Markdown via [markdownfmt](https://github.com/shurcooL/markdownfmt). We run **admin/reformat.sh** to do this, but you don’t have to worry about it unless you really want to. + +### Editorial Process + +- Roles: + - **project leads:** General direction of process and overall quality of the Guide + - **editors:** Contributors own specific sections or aspects of the guide, usually by topic/section, reviewing PRs and contributing; requires expert knowledge + - **experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs + - **contributors:** Everyone who helps one way or another +- All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. +- In addition, anyone with relevant knowledge is encouraged to review/comment on PRs +- Both editors and project leads are responsible for checking for style or problems +- Trivial changes may me merged in directly by project leads or editors + +### Contact + +If you have concerns or additional ideas of ways to help, contact [@jlevy](https://github.com/jlevy) (e-mail or DM on Slack). diff --git a/README.md b/README.md index abe8d4c..104f0ea 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ The Open Guide to Amazon Web Services ===================================== [![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) +[Authors](AUTHORS.md) +[Contributing](CONTRIBUTING.md) Table of Contents ----------------- @@ -628,7 +630,7 @@ S3 - S3 has a [static website hosting option](http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) that is simply a setting that enables configurable HTTP index and error pages and [HTTP redirect support](http://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html) to [public content](http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteAccessPermissionsReqd.html) in S3. It’s a simple way to host static assets or a fully static website. - Consider using CloudFront in front of most or all assets: - Like any CDN, CloudFront improves performance significantly. - - 🔸SSL is only supported on the built-in amazonaws.com domain for S3. S3 supports serving these sites through a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html), but [not over SSL on a custom domain](http://stackoverflow.com/questions/11201316/how-to-configure-ssl-for-amazon-s3-bucket). However, [CloudFront allows you to serve a custom domain over https](http://docs.aws.amazon.com/acm/latest/userguide/gs-cf.html). Amazon provides free SNI SSL/TLS certificates via Amazon Certificate Manager. [SNI does not work on very outdated browsers/operating systems](https://en.wikipedia.org/wiki/Server_Name_Indication#Support). Alternatively, you can provide your own certificate to use on CloudFront to support all browsers/operating systems. + - 🔸SSL is only supported on the built-in amazonaws.com domain for S3. S3 supports serving these sites through a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html), but [not over SSL on a custom domain](http://stackoverflow.com/questions/11201316/how-to-configure-ssl-for-amazon-s3-bucket). However, [CloudFront allows you to serve a custom domain over https](http://docs.aws.amazon.com/acm/latest/userguide/gs-cf.html). Amazon provides free SNI SSL/TLS certificates via Amazon Certificate Manager. [SNI does not work on very outdated browsers/operating systems](https://en.wikipedia.org/wiki/Server_Name_Indication#Support). Alternatively, you can provide your own certificate to use on CloudFront to support all browsers/operating systems. - 🔸If you are including resources across domains, such as fonts inside CSS files, you may need to [configure CORS](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html) for the bucket serving those resources. - Since pretty much everything is moving to SSL nowadays, and you likely want control over the domain, you probably want to set up CloudFront with your own certificate in front of S3 (and to ignore the [AWS example on this](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) as it is non-SSL only). - That said, if you do, you’ll need to think through invalidation or updates on CloudFront. You may wish to [include versions or hashes in filenames](https://abhishek-tiwari.com/post/CloudFront-design-patterns-and-best-practices) so invalidation is not necessary. @@ -1174,7 +1176,7 @@ CloudFront - 🚪CDNs are [a highly fragmented market](https://www.datanyze.com/market-share/cdn/). CloudFront has grown to be a leader, but many alternatives that might better suit specific needs. ### CloudFront Tips -- 🐥**IPv6** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/)! +- 🐥**IPv6** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/)! - 🐥**HTTP/2** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/09/amazon-cloudfront-now-supports-http2/)! Clients [must support TLS 1.2 and SNI](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesSupportedHTTPVersions). - While the most common use is for users to browse and download content (GET or HEAD methods) requests, CloudFront also supports ([since 2013](https://aws.amazon.com/blogs/aws/amazon-cloudfront-content-uploads-post-put-other-methods/)) uploaded data (POST, PUT, DELETE, OPTIONS, and PATCH). - You must enable this by specifying the [allowed HTTP methods](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesAllowedHTTPMethods) when you create the distribution. From 0e2ffa182f8aa8ed15bcb359f0561d41ab845974 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 12 Oct 2016 21:13:55 -0700 Subject: [PATCH 041/140] Tweak credits/contributing links. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 104f0ea..a648672 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ The Open Guide to Amazon Web Services ===================================== [![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) -[Authors](AUTHORS.md) -[Contributing](CONTRIBUTING.md) + +[Credits](AUTHORS.md) ∙ [Contributing guidelines](CONTRIBUTING.md) Table of Contents ----------------- From 93c937db7c08a548e0ea8b5da2f422b83cdb2f15 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 12 Oct 2016 21:29:24 -0700 Subject: [PATCH 042/140] Update authors and roles. (#185) --- admin/authors-info.yml | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/admin/authors-info.yml b/admin/authors-info.yml index c73acb1..9729a17 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -1,28 +1,33 @@ # This file is used to configure the "ghizmo assemble-authors" command. header: | - This work is a collaborative effort. - It was begun and is edited by [@jlevy](https://github.com/jlevy) and [@ThanosBaskous](https://github.com/ThanosBaskous). - The following people (in alphabetical order) have contributed to or reviewed this or earlier versions of the guide. - + This “Open Guide” is a collaborative effort. + It was begun and is led by [@jlevy](https://github.com/jlevy) and [@ThanosBaskous](https://github.com/ThanosBaskous), + but the content is the result of a community of contributors, editors, and experts. + Please help if you can, and see the [contribution guidelines](CONTRIBUTING.md) for notes on roles and editorial process. + Alphabetically by username: footer: | - Additional authors are welcome; see the [contribution guidelines](CONTRIBUTING.md). - Please let the editors know of any errors or omissions on this list. + Please don't PR the AUTHORS.md or authors-info.yml files! It is auto-generated regularly by the project leads. + Please let the project leads know of any errors or omissions on this list. exclude: gitter-badger ReadmeCritic roles: - jlevy: general editor - ThanosBaskous: general editor - max-zanko: editor (S3, EMR, Redshift) + jlevy: project lead, editor (topics not otherwise assigned) + ThanosBaskous: project lead, editor (topics not otherwise assigned) + max-zanko: editor (EC2, S3, Glacier, EMR, Redshift) nitingithub: editor (cost management) + forwidur: editor (EBS, RDS) + lynnlangit: editor (IoT) + richadams: editor (VPC) + donnemartin: expert (tools) + merrells: expert (cloud infrastructure, when to use AWS) + benkehoe: expert (IoT) marcello3d: golyshev: - forwidur: - merrells: Praveen Patnala: kazuyukitanimura: olawiberg: @@ -30,6 +35,4 @@ roles: bittlingmayer: rjpower: alexanderatallah: - donnemartin: - benkehoe: Jurgen Philippaerts: From a4a52f414ca1a4851426fae4e47791091b2c7c2e Mon Sep 17 00:00:00 2001 From: Richard Birkby Date: Wed, 12 Oct 2016 16:54:27 +0100 Subject: [PATCH 043/140] Add AWIS,ATS & CLI column to maturity table --- README.md | 108 +++++++++++++++++++++++++++--------------------------- 1 file changed, 55 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index abe8d4c..0c460ef 100644 --- a/README.md +++ b/README.md @@ -301,59 +301,61 @@ Selected resources with more detail on this chart: It’s important to know the maturity of each AWS product. Here is a mostly complete list of first release date, with links to the [release notes](https://aws.amazon.com/releasenotes/). Most recently released services are first. Not all services are available in all regions; see [this table](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). -| Service | Original release | Availability | -|------------------------------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------| -| 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | -| 🐥[IoT](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) | 2015-08 | General | -| 🐥[WAF](https://aws.amazon.com/releasenotes/AWS-WAF?browse=1) | 2015-10 | General | -| 🐥[Data Pipeline](https://aws.amazon.com/releasenotes/AWS-Data-Pipeline?browse=1) | 2015-10 | General | -| 🐥[Elasticsearch](https://aws.amazon.com/releasenotes/Amazon-Elasticsearch-Service?browse=1) | 2015-10 | General | -| 🐥[Service Catalog](https://aws.amazon.com/releasenotes/AWS-Service-Catalog?browse=1) | 2015-07 | General | -| 🐥[Device Farm](https://aws.amazon.com/releasenotes/AWS-Device-Farm?browse=1) | 2015-07 | General | -| 🐥[CodePipeline](https://aws.amazon.com/releasenotes/AWS-CodePipeline?browse=1) | 2015-07 | General | -| 🐥[CodeCommit](https://aws.amazon.com/releasenotes/AWS-CodeCommit?browse=1) | 2015-07 | General | -| 🐥[API Gateway](https://aws.amazon.com/releasenotes/Amazon-API-Gateway?browse=1) | 2015-07 | General | -| 🐥[Config](https://aws.amazon.com/releasenotes/AWS-Config?browse=1) | 2015-06 | General | -| 🐥[EFS](https://aws.amazon.com/releasenotes/Amazon-EFS?browse=1) | 2015-05 | General | -| 🐥[Machine Learning](https://aws.amazon.com/releasenotes/AmazonML?browse=1) | 2015-04 | General | -| [Lambda](https://aws.amazon.com/releasenotes/AWS-Lambda?browse=1) | 2014-11 | General | -| [ECS](https://aws.amazon.com/ecs/release-notes/) | 2014-11 | General | -| [KMS](https://aws.amazon.com/releasenotes/AWS-KMS?browse=1) | 2014-11 | General | -| [CodeDeploy](https://aws.amazon.com/releasenotes/AWS-CodeDeploy?browse=1) | 2014-11 | General | -| [Kinesis](https://aws.amazon.com/releasenotes/Amazon-Kinesis?browse=1) | 2013-12 | General | -| [CloudTrail](https://aws.amazon.com/releasenotes/AWS-CloudTrail?browse=1) | 2013-11 | General | -| [AppStream](https://aws.amazon.com/releasenotes/Amazon-AppStream?browse=1) | 2013-11 | Preview | -| [CloudHSM](https://aws.amazon.com/releasenotes/AWS-CloudHSM?browse=1) | 2013-03 | General | -| [Silk](https://aws.amazon.com/releasenotes/Amazon-Silk?browse=1) | 2013-03 | Obsolete? | -| [OpsWorks](https://aws.amazon.com/releasenotes/AWS-OpsWorks?browse=1) | 2013-02 | General | -| [Redshift](https://aws.amazon.com/releasenotes/Amazon-Redshift?browse=1) | 2013-02 | General | -| [Elastic Transcoder](https://aws.amazon.com/releasenotes/Amazon-Elastic-Transcoder?browse=1) | 2013-01 | General | -| [Glacier](https://aws.amazon.com/releasenotes/Amazon-Glacier?browse=1) | 2012-08 | General | -| [CloudSearch](https://aws.amazon.com/releasenotes/Amazon-CloudSearch?browse=1) | 2012-04 | General | -| [SWF](https://aws.amazon.com/releasenotes/Amazon-SWF?browse=1) | 2012-02 | General | -| [Storage Gateway](https://aws.amazon.com/releasenotes/AWS-Storage-Gateway?browse=1) | 2012-01 | General | -| [DynamoDB](https://aws.amazon.com/releasenotes/Amazon-DynamoDB?browse=1) | 2012-01 | General | -| [DirectConnect](https://aws.amazon.com/releasenotes/AWS-Direct-Connect?browse=1) | 2011-08 | General | -| [ElastiCache](https://aws.amazon.com/releasenotes/Amazon-ElastiCache?browse=1) | 2011-08 | General | -| [CloudFormation](https://aws.amazon.com/releasenotes/AWS-CloudFormation?browse=1) | 2011-04 | General | -| [SES](https://aws.amazon.com/releasenotes/Amazon-SES?browse=1) | 2011-01 | General | -| [Elastic Beanstalk](https://aws.amazon.com/releasenotes/AWS-Elastic-Beanstalk?browse=1) | 2010-12 | General | -| [Route 53](https://aws.amazon.com/releasenotes/Amazon-Route-53?browse=1) | 2010-10 | General | -| [IAM](https://aws.amazon.com/releasenotes/AWS-Identity-and-Access-Management?browse=1) | 2010-09 | General | -| [SNS](https://aws.amazon.com/releasenotes/Amazon-SNS?browse=1) | 2010-04 | General | -| [EMR](https://aws.amazon.com/releasenotes/Elastic-MapReduce?browse=1) | 2010-04 | General | -| [RDS](https://aws.amazon.com/releasenotes/Amazon-RDS?browse=1) | 2009-12 | General | -| [VPC](https://aws.amazon.com/releasenotes/Amazon-VPC?browse=1) | 2009-08 | General | -| [Snowball](https://aws.amazon.com/releasenotes/AWS-ImportExport?browse=1) | 2009-05 | General | -| [CloudWatch](https://aws.amazon.com/releasenotes/CloudWatch?browse=1) | 2009-05 | General | -| [CloudFront](https://aws.amazon.com/releasenotes/CloudFront?browse=1) | 2008-11 | General | -| [Fulfillment Web Service](https://aws.amazon.com/releasenotes/Amazon-FWS?browse=1) | 2008-03 | Obsolete? | -| [SimpleDB](https://aws.amazon.com/releasenotes/Amazon-SimpleDB?browse=1) | 2007-12 | ❗[Nearly obsolete](https://forums.aws.amazon.com/thread.jspa?threadID=121711) | -| [DevPay](https://aws.amazon.com/releasenotes/DevPay?browse=1) | 2007-12 | General | -| [Flexible Payments Service](https://aws.amazon.com/releasenotes/Amazon-FPS?browse=1) | 2007-08 | Retired | -| [EC2](https://aws.amazon.com/releasenotes/Amazon-EC2?browse=1) | 2006-08 | General | -| [SQS](https://aws.amazon.com/releasenotes/Amazon-SQS?browse=1) | 2006-07 | General | -| [S3](https://aws.amazon.com/releasenotes/Amazon-S3?browse=1) | 2006-03 | General | +| Service | Original release | Availability | CLI Support | +|------------------------------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------|:-----------:| +| 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | | +| 🐥[IoT](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) | 2015-08 | General | ✓ | +| 🐥[WAF](https://aws.amazon.com/releasenotes/AWS-WAF?browse=1) | 2015-10 | General | ✓ | +| 🐥[Data Pipeline](https://aws.amazon.com/releasenotes/AWS-Data-Pipeline?browse=1) | 2015-10 | General | ✓ | +| 🐥[Elasticsearch](https://aws.amazon.com/releasenotes/Amazon-Elasticsearch-Service?browse=1) | 2015-10 | General | ✓ | +| 🐥[Service Catalog](https://aws.amazon.com/releasenotes/AWS-Service-Catalog?browse=1) | 2015-07 | General | ✓ | +| 🐥[Device Farm](https://aws.amazon.com/releasenotes/AWS-Device-Farm?browse=1) | 2015-07 | General | ✓ | +| 🐥[CodePipeline](https://aws.amazon.com/releasenotes/AWS-CodePipeline?browse=1) | 2015-07 | General | ✓ | +| 🐥[CodeCommit](https://aws.amazon.com/releasenotes/AWS-CodeCommit?browse=1) | 2015-07 | General | ✓ | +| 🐥[API Gateway](https://aws.amazon.com/releasenotes/Amazon-API-Gateway?browse=1) | 2015-07 | General | ✓ | +| 🐥[Config](https://aws.amazon.com/releasenotes/AWS-Config?browse=1) | 2015-06 | General | ✓ | +| 🐥[EFS](https://aws.amazon.com/releasenotes/Amazon-EFS?browse=1) | 2015-05 | General | ✓ | +| 🐥[Machine Learning](https://aws.amazon.com/releasenotes/AmazonML?browse=1) | 2015-04 | General | ✓ | +| [Lambda](https://aws.amazon.com/releasenotes/AWS-Lambda?browse=1) | 2014-11 | General | ✓ | +| [ECS](https://aws.amazon.com/ecs/release-notes/) | 2014-11 | General | ✓ | +| [KMS](https://aws.amazon.com/releasenotes/AWS-KMS?browse=1) | 2014-11 | General | ✓ | +| [CodeDeploy](https://aws.amazon.com/releasenotes/AWS-CodeDeploy?browse=1) | 2014-11 | General | ✓ | +| [Kinesis](https://aws.amazon.com/releasenotes/Amazon-Kinesis?browse=1) | 2013-12 | General | ✓ | +| [CloudTrail](https://aws.amazon.com/releasenotes/AWS-CloudTrail?browse=1) | 2013-11 | General | ✓ | +| [AppStream](https://aws.amazon.com/releasenotes/Amazon-AppStream?browse=1) | 2013-11 | Preview | | +| [CloudHSM](https://aws.amazon.com/releasenotes/AWS-CloudHSM?browse=1) | 2013-03 | General | ✓ | +| [Silk](https://aws.amazon.com/releasenotes/Amazon-Silk?browse=1) | 2013-03 | Obsolete? | | +| [OpsWorks](https://aws.amazon.com/releasenotes/AWS-OpsWorks?browse=1) | 2013-02 | General | ✓ | +| [Redshift](https://aws.amazon.com/releasenotes/Amazon-Redshift?browse=1) | 2013-02 | General | ✓ | +| [Elastic Transcoder](https://aws.amazon.com/releasenotes/Amazon-Elastic-Transcoder?browse=1) | 2013-01 | General | ✓ | +| [Glacier](https://aws.amazon.com/releasenotes/Amazon-Glacier?browse=1) | 2012-08 | General | ✓ | +| [CloudSearch](https://aws.amazon.com/releasenotes/Amazon-CloudSearch?browse=1) | 2012-04 | General | ✓ | +| [SWF](https://aws.amazon.com/releasenotes/Amazon-SWF?browse=1) | 2012-02 | General | ✓ | +| [Storage Gateway](https://aws.amazon.com/releasenotes/AWS-Storage-Gateway?browse=1) | 2012-01 | General | ✓ | +| [DynamoDB](https://aws.amazon.com/releasenotes/Amazon-DynamoDB?browse=1) | 2012-01 | General | ✓ | +| [DirectConnect](https://aws.amazon.com/releasenotes/AWS-Direct-Connect?browse=1) | 2011-08 | General | ✓ | +| [ElastiCache](https://aws.amazon.com/releasenotes/Amazon-ElastiCache?browse=1) | 2011-08 | General | ✓ | +| [CloudFormation](https://aws.amazon.com/releasenotes/AWS-CloudFormation?browse=1) | 2011-04 | General | ✓ | +| [SES](https://aws.amazon.com/releasenotes/Amazon-SES?browse=1) | 2011-01 | General | ✓ | +| [Elastic Beanstalk](https://aws.amazon.com/releasenotes/AWS-Elastic-Beanstalk?browse=1) | 2010-12 | General | ✓ | +| [Route 53](https://aws.amazon.com/releasenotes/Amazon-Route-53?browse=1) | 2010-10 | General | ✓ | +| [IAM](https://aws.amazon.com/releasenotes/AWS-Identity-and-Access-Management?browse=1) | 2010-09 | General | ✓ | +| [SNS](https://aws.amazon.com/releasenotes/Amazon-SNS?browse=1) | 2010-04 | General | ✓ | +| [EMR](https://aws.amazon.com/releasenotes/Elastic-MapReduce?browse=1) | 2010-04 | General | ✓ | +| [RDS](https://aws.amazon.com/releasenotes/Amazon-RDS?browse=1) | 2009-12 | General | ✓ | +| [VPC](https://aws.amazon.com/releasenotes/Amazon-VPC?browse=1) | 2009-08 | General | ✓ | +| [Snowball](https://aws.amazon.com/releasenotes/AWS-ImportExport?browse=1) | 2009-05 | General | ✓ | +| [CloudWatch](https://aws.amazon.com/releasenotes/CloudWatch?browse=1) | 2009-05 | General | ✓ | +| [CloudFront](https://aws.amazon.com/releasenotes/CloudFront?browse=1) | 2008-11 | General | ✓ | +| [Fulfillment Web Service](https://aws.amazon.com/releasenotes/Amazon-FWS?browse=1) | 2008-03 | Obsolete? | | +| [SimpleDB](https://aws.amazon.com/releasenotes/Amazon-SimpleDB?browse=1) | 2007-12 | ❗[Nearly obsolete](https://forums.aws.amazon.com/thread.jspa?threadID=121711) | ✓ | +| [DevPay](https://aws.amazon.com/releasenotes/DevPay?browse=1) | 2007-12 | General | | +| [Flexible Payments Service](https://aws.amazon.com/releasenotes/Amazon-FPS?browse=1) | 2007-08 | Retired | | +| [EC2](https://aws.amazon.com/releasenotes/Amazon-EC2?browse=1) | 2006-08 | General | ✓ | +| [SQS](https://aws.amazon.com/releasenotes/Amazon-SQS?browse=1) | 2006-07 | General | ✓ | +| [S3](https://aws.amazon.com/releasenotes/Amazon-S3?browse=1) | 2006-03 | General | ✓ | +| [Alexa Top Sites](https://aws.amazon.com/alexa-top-sites/) | 2006-01 | General ❗HTTP-only | | +| [Alexa Web Information Service](https://aws.amazon.com/awis/) | 2005-10 | General ❗HTTP-only | | ### Compliance From db59f6d4597fcd63b2e152998b9bacefd75053d6 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Wed, 12 Oct 2016 23:05:43 -0700 Subject: [PATCH 044/140] Update AUTHORS.md --- AUTHORS.md | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 46128ee..fba7e71 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -1,47 +1,59 @@ # Authors -This work is a collaborative effort. -It was begun and is edited by [@jlevy](https://github.com/jlevy) and [@ThanosBaskous](https://github.com/ThanosBaskous). -The following people (in alphabetical order) have contributed to or reviewed this or earlier versions of the guide. +This “Open Guide” is a collaborative effort. +It was begun and is led by [@jlevy](https://github.com/jlevy) and [@ThanosBaskous](https://github.com/ThanosBaskous), +but the content is the result of a community of contributors, editors, and experts. +Please help if you can, and see the [contribution guidelines](CONTRIBUTING.md) for notes on roles and editorial process. +Alphabetically by username: * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) -* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [2+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) +* [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) +* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [3+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) * [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk) -* [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) +* [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) — _expert (IoT)_ * [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer) +* [Bradly Feeley (bradly)](https://github.com/bradly) — [2+](https://github.com/open-guides/og-aws/commits?author=bradly)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Abradly) +* [Brady Dowling (bradydowling)](https://github.com/bradydowling) — [1+](https://github.com/open-guides/og-aws/commits?author=bradydowling)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abradydowling) * [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) -* [danhermann](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) -* [Donne Martin (donnemartin)](https://github.com/donnemartin) +* [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) +* [Dan Hermann (danhermann)](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) +* [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ +* [Dmitry Guyvoronsky (dreamiurg)](https://github.com/dreamiurg) — [1+](https://github.com/open-guides/og-aws/commits?author=dreamiurg)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adreamiurg) * [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy) -* [Max Grigorev (forwidur)](https://github.com/forwidur) +* [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ * [Glynn Forrest (glynnforrest)](https://github.com/glynnforrest) — [1+](https://github.com/open-guides/og-aws/commits?author=glynnforrest)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aglynnforrest) * [Dmitry Golyshev (golyshev)](https://github.com/golyshev) * [Gulam Shakir (gshakir)](https://github.com/gshakir) — [2+](https://github.com/open-guides/og-aws/commits?author=gshakir)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agshakir) * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) * [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [86+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[76+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_ +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [87+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[80+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ * Jurgen Philippaerts -* [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [3+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) +* [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) — [0+](https://github.com/open-guides/og-aws/commits?author=kazuyukitanimura)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akazuyukitanimura) +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [3+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [maiki](https://github.com/maiki) — [1+](https://github.com/open-guides/og-aws/commits?author=maiki)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amaiki) +* [Manoj M J (manojmj92)](https://github.com/manojmj92) — [1+](https://github.com/open-guides/og-aws/commits?author=manojmj92)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amanojmj92) * [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d) -* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [3+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (S3, EMR, Redshift)_ -* [John Merrells (merrells)](https://github.com/merrells) +* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [10+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ +* [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ * [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke) * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Ola Wiberg (olawiberg)](https://github.com/olawiberg) +* [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) * Praveen Patnala +* [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ * [Russell Power (rjpower)](https://github.com/rjpower) * [David Schott (shott85)](https://github.com/shott85) — [1+](https://github.com/open-guides/og-aws/commits?author=shott85)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashott85) -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [12+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[12+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_ +* [Adam Nelson (spudstuff)](https://github.com/spudstuff) — [4+](https://github.com/open-guides/og-aws/commits?author=spudstuff)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aspudstuff) +* [Steven Maude (StevenMaude)](https://github.com/StevenMaude) — [1+](https://github.com/open-guides/og-aws/commits?author=StevenMaude)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AStevenMaude) +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [15+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[15+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ * [Tom Schlick (tomschlick)](https://github.com/tomschlick) — [3+](https://github.com/open-guides/og-aws/commits?author=tomschlick)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atomschlick) * [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite) * [Stefan Zier (weirded)](https://github.com/weirded) * [Michael Ortali (xethorn)](https://github.com/xethorn) — [1+](https://github.com/open-guides/og-aws/commits?author=xethorn)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Axethorn) -Additional authors are welcome; see the [contribution guidelines](CONTRIBUTING.md). -Please let the editors know of any errors or omissions on this list. +Please don't PR the AUTHORS.md or authors-info.yml files! It is auto-generated regularly by the project leads. +Please let the project leads know of any errors or omissions on this list. From bcdf4e9a05ac368081b566e63031805cb6044369 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 12 Oct 2016 23:13:59 -0700 Subject: [PATCH 045/140] Updates from review. --- CONTRIBUTING.md | 52 ++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 77de110..7bd034b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,31 +3,31 @@ Contributing Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly [credit](/AUTHORS.md) all contributors. Here are few notes before you jump in. -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com +/shrXZ61VrovWfXYBg) ### Please Help If you’ve found this guide useful, you have many ways to help: -- The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) or [**add to our list of common questions**](https://airtable.com/shrXZ61VrovWfXYBg), which helps the community and guides what contributors can focus on. We encourage you to ask AWS questions and help others! -- [**File issues**](https://github.com/open-guides/og-aws/issues) if it’s clear something needs to be improved and you’re not able to make a pull request. -- [**Pull requests**](https://github.com/open-guides/og-aws/pulls) with changes are always welcome. Please keep them small and focused, so we can add items individually, and review the conventions below. If you want to make a larger change, try to discuss it in Slack. -- [**Review**] or [**comment**] on existing issues and pull requests if you have expertise. -- If you have deep expertise, consider offering to be an **editor** or **expert**. Editors and experts are assigned roles that [help us review the guide](#editorial-process). Join Slack to discuss this. - +- The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) or [**add to our list of common questions**](https://airtable.com/shrXZ61VrovWfXYBg), which helps the community and guides what contributors can focus on. We encourage you to ask AWS questions and help others! +- [**File issues**](https://github.com/open-guides/og-aws/issues) if it’s clear something needs to be improved and you’re not able to make a pull request. +- [**Pull requests**](https://github.com/open-guides/og-aws/pulls) with changes are always welcome. Please keep them small and focused, so we can add items individually, and review the conventions below. If you want to make a larger change, try to discuss it in Slack. +- **Review** or **comment** on existing issues and pull requests if you have expertise. +- If you have deep expertise, we may ask you to be an **editor** or **expert**. Editors and experts are assigned roles that [help us review](#editorial-process) the Guide. Join Slack to discuss this. ### Pull Request Etiquette -- Keep changes as small as is practical. Do not make changes to multiple sections at once, alter whitespace in broad ways, etc. -- Neutrality: If you have an affiliation related to what you are changing, please disclose it. -- Please do your best to review current issues and pull requests to avoid duplication. +- Keep changes as small as is practical. Do not make changes to multiple sections at once, alter whitespace in broad ways, etc. +- Neutrality: If you have an affiliation related to what you are changing, please disclose it. +- Please do your best to review current issues and pull requests to avoid duplication. ### Writing Style -- **Link to references:** If you are adding an item, whenever possible, try to add a link or reference to relevant discussion or reference pages. -- **Be brief:** Avoid long expository paragraphs; it’s better to link to a blog. (We are open to linking to your own blog, if it’s the best source.) -- **Include opinions and common practice:** Thoughtful opinion is helpful. If there are multiple conventions or ideas on something held by experts, mention the different ones. -- **Clarity:** Strive for consistency with conventions listed here, but clarity is most important. +- **Link to references:** If you are adding an item, whenever possible, try to add a link or reference to relevant discussion or reference pages. +- **Be brief:** Avoid long expository paragraphs; it’s better to link to a blog. (We are open to linking to your own blog, if it’s the best source.) +- **Include opinions and common practice:** Thoughtful opinion is helpful. If there are multiple conventions or ideas on something held by experts, mention the different ones. +- **Clarity:** Strive for consistency with conventions listed here, but clarity is most important. ### Writing Conventions @@ -49,23 +49,23 @@ When you contribute, keep in mind these conventions: - *X Alternatives and Lock-In*: Should you be using this service or something else? Is the decision an important one that locks you in? - *X Tips*: Everything you should know about the service, from big stuff to details. - *X Gotchas and Limitations*: Common problems, large and small, as well as misconceptions and quirks. - - Not all sections need to follow the above conventions exactly. - - Note we try to make sections uniquely titled, so GitHub links to Markdown section anchors don’t collide and are stable. + - Not all sections need to follow the above conventions exactly. + - Note we try to make sections uniquely titled, so GitHub links to Markdown section anchors don’t collide and are stable. Note we keep consistent formatting in Markdown via [markdownfmt](https://github.com/shurcooL/markdownfmt). We run **admin/reformat.sh** to do this, but you don’t have to worry about it unless you really want to. ### Editorial Process -- Roles: - - **project leads:** General direction of process and overall quality of the Guide - - **editors:** Contributors own specific sections or aspects of the guide, usually by topic/section, reviewing PRs and contributing; requires expert knowledge - - **experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs - - **contributors:** Everyone who helps one way or another -- All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. -- In addition, anyone with relevant knowledge is encouraged to review/comment on PRs -- Both editors and project leads are responsible for checking for style or problems -- Trivial changes may me merged in directly by project leads or editors +- Roles: + - **Project leads:** Own general direction of process and overall quality of the Guide. + - **Editors:** Contributors own specific sections or aspects of the Guide, reviewing PRs and/or writing. requires expert knowledge. + - **Experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs. + - **Contributors:** Everyone who helps one way or another. +- All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. +- In addition, anyone with relevant knowledge is encouraged to review/comment on PRs. +- Both editors and project leads are responsible for checking for style or problems. +- Trivial changes may be merged in directly by project leads or editors. ### Contact -If you have concerns or additional ideas of ways to help, contact [@jlevy](https://github.com/jlevy) (e-mail or DM on Slack). +If you have concerns or additional ideas of ways to help, e-mail **og-aws@lexikon.io** or use Slack to contact the [project leads](AUTHORS.md). From 5494054be8a506bf90a389893018ae72b2acbad2 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 12 Oct 2016 23:17:05 -0700 Subject: [PATCH 046/140] Another tweak to contributing info. --- CONTRIBUTING.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7bd034b..1290b8b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,10 +1,9 @@ Contributing ------------ -Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly [credit](/AUTHORS.md) all contributors. Here are few notes before you jump in. +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com -/shrXZ61VrovWfXYBg) +Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly [credit](/AUTHORS.md) all contributors. Here are few notes before you jump in. ### Please Help @@ -57,7 +56,7 @@ Note we keep consistent formatting in Markdown via [markdownfmt](https://github. ### Editorial Process - Roles: - - **Project leads:** Own general direction of process and overall quality of the Guide. + - **Project leads:** Own overall quality of the Guide, direction, and process. - **Editors:** Contributors own specific sections or aspects of the Guide, reviewing PRs and/or writing. requires expert knowledge. - **Experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs. - **Contributors:** Everyone who helps one way or another. From e96d266c8d166d00db017a745f5423bd75bbd858 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Wed, 12 Oct 2016 23:22:10 -0700 Subject: [PATCH 047/140] Another tweak to contributing info. --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1290b8b..6987025 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -59,11 +59,11 @@ Note we keep consistent formatting in Markdown via [markdownfmt](https://github. - **Project leads:** Own overall quality of the Guide, direction, and process. - **Editors:** Contributors own specific sections or aspects of the Guide, reviewing PRs and/or writing. requires expert knowledge. - **Experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs. - - **Contributors:** Everyone who helps one way or another. + - **Contributors:** Everyone who contributes content or helps one way or another. - All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. - In addition, anyone with relevant knowledge is encouraged to review/comment on PRs. - Both editors and project leads are responsible for checking for style or problems. -- Trivial changes may be merged in directly by project leads or editors. +- Trivial changes (including copy editing) may be merged in directly by project leads or editors. ### Contact From 9c7f886654ea2b341863a7b883a0f4c214fe6a97 Mon Sep 17 00:00:00 2001 From: Richard Birkby Date: Thu, 13 Oct 2016 08:04:31 +0100 Subject: [PATCH 048/140] Minor typo fixes --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index abe8d4c..90f874a 100644 --- a/README.md +++ b/README.md @@ -1167,11 +1167,11 @@ CloudFront - 📒 [Homepage](https://aws.amazon.com/cloudfront/) ∙ [Developer guide](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/) ∙ [FAQ](https://aws.amazon.com/cloudfront/faqs/) ∙ [Pricing](https://aws.amazon.com/cloudfront/pricing/) - **CloudFront** is AWS’ [content delivery network (CDN)](https://en.wikipedia.org/wiki/Content_delivery_network). -- Its primary use is improving latency for end users in to accessing cacheable content by hosting it at [over 60 global edge locations](http://aws.amazon.com/cloudfront/details/). +- Its primary use is improving latency for end users through accessing cacheable content by hosting it at [over 60 global edge locations](http://aws.amazon.com/cloudfront/details/). ### CloudFront Alternatives and Lock-in -- 🚪CDNs are [a highly fragmented market](https://www.datanyze.com/market-share/cdn/). CloudFront has grown to be a leader, but many alternatives that might better suit specific needs. +- 🚪CDNs are [a highly fragmented market](https://www.datanyze.com/market-share/cdn/). CloudFront has grown to be a leader, but there are many alternatives that might better suit specific needs. ### CloudFront Tips - 🐥**IPv6** is [now supported](https://aws.amazon.com/about-aws/whats-new/2016/10/ipv6-support-for-cloudfront-waf-and-s3-transfer-acceleration/)! @@ -1253,7 +1253,7 @@ EMR ### EMR Alternatives and Lock-in -- ⛓Most of EMR is based open source technology that you can in principle deploy yourself. However, the job workflows and much other tooling is AWS-specific. Migrating from EMR to your own clusters is possible but not always trivial. +- ⛓Most of EMR is based on open source technology that you can in principle deploy yourself. However, the job workflows and much other tooling is AWS-specific. Migrating from EMR to your own clusters is possible but not always trivial. ### EMR Tips @@ -1277,7 +1277,7 @@ This section covers tips and information on achieving [high availability](https: - AWS offers two levels of redundancy, [regions and availability zones (AZs)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones). - When used correctly, regions and zones do allow for high availability. You may want to use non-AWS providers for larger business risk mitigation (i.e. not tying your company to one vendor), but reliability of AWS across regions is very high. -- **Multiple regions:** Using multiple regions is complex, since it’s essentially like completely separate infrastructure. It is necessary for business-critical services which highest levels of redundancy. However, for many applications (like your average consumer startup), deploying extensive redundancy across regions may be overkill. +- **Multiple regions:** Using multiple regions is complex, since it’s essentially like completely separate infrastructure. It is necessary for business-critical services with the highest levels of redundancy. However, for many applications (like your average consumer startup), deploying extensive redundancy across regions may be overkill. - The [High Scalability Blog](http://highscalability.com/blog/2016/1/11/a-beginners-guide-to-scaling-to-11-million-users-on-amazons.html) has a good guide to help you understand when you need to scale an application to multiple regions. - 🔹**Multiple AZs:** Using AZs wisely is the primary tool for high availability! - A typical single-region high availability architecture would be to deploy in two or more availability zones, with load balancing in front, as in [this AWS diagram](http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_ftha_04.pdf). From 27dbe8db3f0d1d05968168501bd70d871df226a5 Mon Sep 17 00:00:00 2001 From: Theo Hultberg Date: Thu, 13 Oct 2016 11:20:07 +0200 Subject: [PATCH 049/140] Update pricing details on the support tiers The developer support was recently changed to be the greater of $29 or 3% of your bill. Also change from "at least 10%" to "up to 10%" in the next paragraph. No support plan adds more than 10% to your bill. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c5457d2..c2a3f88 100644 --- a/README.md +++ b/README.md @@ -371,8 +371,8 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp - **Forums:** For many problems, it’s worth searching or asking for help in the [discussion forums](https://forums.aws.amazon.com/index.jspa) to see if it’s a known issue. - **Premium support:** AWS offers several levels of [premium support](https://aws.amazon.com/premiumsupport/). - - Any small company should probably pay for the cheap “Developer” support as it’s a flat $49/month and it lets you file support tickets with 12 to 24 hour turnaround time. - - The higher-level support services are quite expensive — and increase your bill by at least 10%. Many large and effective companies never pay for this level of support. They are usually more helpful for midsize or larger companies needing rapid turnaround on deeper or more perplexing problems. + - The first tier, called "Developer support" lets you file support tickets with 12 to 24 hour turnaround time, it starts at $29 but once your monthly spend reaches around $1000 it changes to a 3% surcharge on your bill. + - The higher-level support services are quite expensive — and increase your bill by up to 10%. Many large and effective companies never pay for this level of support. They are usually more helpful for midsize or larger companies needing rapid turnaround on deeper or more perplexing problems. - Keep in mind, a flexible architecture can reduce need for support. You shouldn’t be relying on AWS to solve your problems often. For example, if you can easily re-provision a new server, it may not be urgent to solve a rare kernel-level issue unique to one EC2 instance. If your EBS volumes have recent snapshots, you may be able to restore a volume before support can rectify the issue with the old volume. If your services have an issue in one availability zone, you should in any case be able to rely on a redundant zone or migrate services to another zone. - Larger customers also get access to AWS Enterprise support, with dedicated technical account managers (TAMs) and shorter response time SLAs. - There is definitely some controversy about how useful the paid support is. The support staff don’t always seem to have the information and authority to solve the problems that are brought to their attention. Often your ability to have a problem solved may depend on your relationship with your account rep. From 66c3e853a952ce9c1a558977330b9332b9eb2831 Mon Sep 17 00:00:00 2001 From: Pascal Borreli Date: Thu, 13 Oct 2016 12:09:33 +0100 Subject: [PATCH 050/140] Fixed typos --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c5457d2..02d6e1f 100644 --- a/README.md +++ b/README.md @@ -158,7 +158,7 @@ General Information - ⛓**Lock-in:** As you use AWS, it’s important to be aware when you are depending on AWS services that do not have equivalents elsewhere. - Lock-in may be completely fine for your company, or a significant risk. It’s important from a business perspective to make this choice explicitly, and consider the cost, operational, business continuity, and competitive risks of being tied to AWS. AWS is such a dominant and reliable vendor, many companies are comfortable with using AWS to its full extent. Others can tell stories about the [dangers of “cloud jail” when costs spiral](http://firstround.com/review/the-three-infrastructure-mistakes-your-company-must-not-make/). - Generally, the more AWS services you use, the more lock-in you have to AWS — that is, the more engineering resources (time and money) it will take to change to other providers in the future. - - Basic services like virtual servers and standard databases are usually easy to migrate to other providers or on premises. Others like load balancers and IAM are specific to AWS but have close equivalents from other providers. The key thing to consider is whether engineers are architecting systems around specific AWS services that are not open source or relatively interchangeable. For example, Lambda, API Gateway, Kinesis, Redshift, and DynamoDB do not have have substantially equivalent open source or commercial service equivalents, while EC2, RDS (MySQL or Postgres), EMR, and ElastiCache more or less do. (See more [below](#which-services-to-use), where these are noted with ⛓.) + - Basic services like virtual servers and standard databases are usually easy to migrate to other providers or on premises. Others like load balancers and IAM are specific to AWS but have close equivalents from other providers. The key thing to consider is whether engineers are architecting systems around specific AWS services that are not open source or relatively interchangeable. For example, Lambda, API Gateway, Kinesis, Redshift, and DynamoDB do not have substantially equivalent open source or commercial service equivalents, while EC2, RDS (MySQL or Postgres), EMR, and ElastiCache more or less do. (See more [below](#which-services-to-use), where these are noted with ⛓.) - **Combining AWS and other cloud providers:** Many customers combine AWS with other non-AWS services. For example, legacy systems or secure data might be in a managed hosting provider, while other systems are AWS. Or a company might only use S3 with another provider doing everything else. However small startups or projects starting fresh will typically stick to AWS or Google Cloud only. - **Hybrid cloud:** In larger enterprises, it is common to have [hybrid deployments](https://aws.amazon.com/enterprise/hybrid/) encompassing private cloud or on-premises servers and AWS — or other enterprise cloud providers like [IBM](https://www.ibm.com/cloud-computing/solutions/hybrid-cloud)/[Bluemix](http://www.ibm.com/cloud-computing/bluemix/hybrid/), [Microsoft](https://www.microsoft.com/en-us/cloud-platform/hybrid-cloud)/[Azure](https://azure.microsoft.com/en-us/overview/azure-stack/), [NetApp](http://www.netapp.com/us/solutions/cloud/hybrid-cloud/), or [EMC](http://www.emc.com/en-us/cloud/hybrid-cloud-computing/index.htm). - **Major customers:** Who uses AWS and Google Cloud? @@ -652,7 +652,7 @@ S3 - Creation of individual objects in S3 is atomic. You’ll never upload a file and have another client see only half the file. - Also, if you create a new object, you’ll be able to read it instantly, which is called **read-after-write consistency**. - Well, with the additional caveat that if you do a read on an object before it exists, then create it, [you get eventual consistency](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel) (not read-after-write). - - If you overwrite or delete a object, you’re only guaranteed eventual consistency. + - If you overwrite or delete an object, you’re only guaranteed eventual consistency. - 🔹Note that [until 2015](https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3-introduces-new-usability-enhancements/), 'us-standard' region had had a weaker eventual consistency model, and the other (newer) regions were read-after-write. This was finally corrected — but watch for many old blogs mentioning this! - In practice, “eventual consistency” usually means within seconds, but expect rare cases of minutes or [hours](http://www.stackdriver.com/eventual-consistency-really-eventual/). - **S3 as a filesystem:** @@ -732,7 +732,7 @@ EC2 - 🔸For all [newer instance types](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/), when selecting the AMI to use, be sure you select the HVM AMI, or it just won’t work. - ❗When creating an instance and using a new ssh key pair, [make sure the ssh key permissions are correct](http://stackoverflow.com/questions/1454629/aws-ssh-access-permission-denied-publickey-issue). - 🔸Sometimes certain EC2 instances can get scheduled for retirement by AWS due to “detected degradation of the underlying hardware,” in which case you are given a couple of weeks to migrate to a new instance - - If your instance root device is an EBS volume, you can typically stop and then start the instance which moves it to heathly host hardware, giving you control over timing of this event. Note however that you will lose any instance store volume data ([ephemeral drives](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)) if your instance type has instance store volumes. + - If your instance root device is an EBS volume, you can typically stop and then start the instance which moves it to healthy host hardware, giving you control over timing of this event. Note however that you will lose any instance store volume data ([ephemeral drives](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)) if your instance type has instance store volumes. - The instance public IP (if it has one) will likely change unless you're using Elastic IPs. This could be a problem if other systems depend on the IP address. - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. @@ -1149,7 +1149,7 @@ VPCs, Network Security, and Security Groups - 🔸Security groups are not shared across data centers, so if you have infrastructure in multiple data centers, you should make sure your configuration/deployment tools take that into account. - ❗Be careful when choosing your VPC IP CIDR block: If you are going to need to make use of [ClassicLink](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html), make sure that your private IP range [doesn’t overlap](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html#classiclink-limitations) with that of EC2 Classic. -- ❗If you are going to peer VPCs, carefully consider the cost of of [data transfer between VPCs](https://aws.amazon.com/vpc/faqs/#Peering_Connections), since for some workloads and integrations, this can be prohibitively expensive. +- ❗If you are going to peer VPCs, carefully consider the cost of [data transfer between VPCs](https://aws.amazon.com/vpc/faqs/#Peering_Connections), since for some workloads and integrations, this can be prohibitively expensive. KMS --- @@ -1239,13 +1239,13 @@ Redshift ### Redshift Gotchas and Limitations -- ❗⏱While Redshift can handle heavy queries well, its does not scale horizontally, i.e. does not handle multiple queries in parallel. Therefore, if you expect a high parallel load, consider replicating or (if possible) sharding your data across multiple clusters. +- ❗⏱While Redshift can handle heavy queries well, it does not scale horizontally, i.e. does not handle multiple queries in parallel. Therefore, if you expect a high parallel load, consider replicating or (if possible) sharding your data across multiple clusters. - 🔸Leader node, which manages communications with client programs and all communication with compute nodes, is the single point of failure. - ⏱Although most Redshift queries parallelize well at the compute node level, certain stages are executed on the leader node, which can become the bottleneck. - 🔹Redshift data commit transactions are very expensive and serialized at the cluster level. Therefore, consider grouping multiple mutation commands (COPY/INSERT/UPDATE) commands into a single transaction whenever possible. - 🔹Redshift does not support multi-AZ deployments. Building multi-AZ clusters is not trivial. [Here](https://blogs.aws.amazon.com/bigdata/post/Tx13ZDHZANSX9UX/Building-Multi-AZ-or-Multi-Region-Amazon-Redshift-Clusters) is an example using Kinesis. - 🔸Beware of storing multiple small tables in Redshift. The way Redshift tables are laid out on disk makes it impractical. The minimum space required to store a table (in MB) is nodes * slices/node * columns. For example, on a 16 node cluster an empty table with 20 columns will occupy 640MB on disk. -- ⏱ Query performance degrades significatly during data ingestion. [WLM (Workload Management)](http://docs.aws.amazon.com/redshift/latest/dg/c_workload_mngmt_classification.html) tweaks help to some extent. However, if you need consistent read performance, consider having replica clusters (at the extra cost) and swap them during update. +- ⏱ Query performance degrades significantly during data ingestion. [WLM (Workload Management)](http://docs.aws.amazon.com/redshift/latest/dg/c_workload_mngmt_classification.html) tweaks help to some extent. However, if you need consistent read performance, consider having replica clusters (at the extra cost) and swap them during update. - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). @@ -1367,7 +1367,7 @@ Billing and Cost Management - If you are running an infrastructure with hundreds of jobs of spiky nature, it is advisable to start pooling instances to optimize for cost, performance and most importantly time to acquire an instance. - Pooling implies creating and maintaining Spot instances so that they do not get terminated after use. This promotes re-use of Spot instances across jobs. This of course comes with the overhead of lifecycle management. - Pooling has its own set of metrics that can be tracked to optimize resource utilization, efficiency and cost. - - Typical pooling implementations give anywhere between 45-60% cost optimizations and 40% reduction in spot instance creationg time. + - Typical pooling implementations give anywhere between 45-60% cost optimizations and 40% reduction in spot instance creation time. - An excellent example of Pooling implementation described by Netflix ([part1](http://techblog.netflix.com/2015/09/creating-your-own-ec2-spot-market.html), [part2](http://techblog.netflix.com/2015/11/creating-your-own-ec2-spot-market-part-2.html)\) - **Spot management gotchas** - 🔸**Lifetime:** There is [no guarantee](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) for the lifetime of a Spot instance. It is purely based on bidding. If anyone outbids your price, the instance is taken away. Spot is not suitable for time sensitive jobs that have strong SLA. Instances will fail based on demand for Spot at that time. AWS provides a [two-minute warning](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#spot-instance-termination-notices) before Amazon EC2 must terminate your Spot instance. From 1f7b3f828324e2ce49a6bd47faa1e7f719a14b77 Mon Sep 17 00:00:00 2001 From: Jifeng Zhang Date: Thu, 13 Oct 2016 14:18:29 +0200 Subject: [PATCH 051/140] Added missing word "there" --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 44cb9f0..6186312 100644 --- a/README.md +++ b/README.md @@ -592,7 +592,7 @@ S3 - 💸**S3 pricing** depends on [storage, requests, and transfer](https://aws.amazon.com/s3/pricing/). - For transfer, putting data into AWS is free, but you’ll pay on the way out. Transfer from S3 to EC2 in the *same region* is free. Transfer to other regions or the Internet in general is not free. - Deletes are free. -- **S3 Reduced Redundancy and Infrequent Access:** Most people use the Standard storage class in S3, but are other storage classes with lower cost: +- **S3 Reduced Redundancy and Infrequent Access:** Most people use the Standard storage class in S3, but there are other storage classes with lower cost: - [Reduced Redundancy Storage (RRS)](https://aws.amazon.com/s3/reduced-redundancy/) has lower durability (99.99%, so just four nines). That is, there’s a small chance you’ll lose data. For some data sets where data has value in a statistical way (losing say half a percent of your objects isn’t a big deal) this is a reasonable trade-off. - [Infrequent Access (IA)](https://aws.amazon.com/s3/storage-classes/#Infrequent_Access) lets you get cheaper storage in exchange for more expensive access. This is great for archives like logs you already processed, but might want to look at later. To get an idea of the cost savings when using Infrequent Access (IA), you can use this [S3 Infrequent Access Calculator](http://www.gulamshakir.com/apps/s3calc/index.html). - [Glacier](#glacier) is a third alternative discussed as a separate product. From be0f50098ec20ce2f934ad38a49b2d41e25c5f3f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A5rten=20Gustafson?= Date: Thu, 13 Oct 2016 15:05:17 +0200 Subject: [PATCH 052/140] Expand the PaaS section of the service matrix: - Providers: AppFog and OpenShift - Build your own: Cloud Foundry and Convox --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c5457d2..1ceeff6 100644 --- a/README.md +++ b/README.md @@ -265,7 +265,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Service | AWS | Google Cloud | Google Internal | Microsoft Azure | Other providers | Open source “build your own” | |-------------------------------|------------------------------------------------------------------------------|------------------------------|-----------------|------------------------------------|-----------------------------------|------------------------------------------------------------| | Virtual server | EC2 | Compute Engine (GCE) | | Virtual Machine | DigitalOcean | OpenStack | -| PaaS | Elastic Beanstalk | App Engine | App Engine | Web Apps | Heroku | Meteor, AppScale | +| PaaS | Elastic Beanstalk | App Engine | App Engine | Web Apps | Heroku, AppFog, OpenShift | Meteor, AppScale, Cloud Foundry, Convox | | Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | | | | Container, cluster manager | ECS | Container Engine, Kubernetes | Borg or Omega | Container Service | | Kubernetes, Mesos, Aurora | | File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | From 16db6f4566ccaa4dd67569f3eecb966ee372964a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A5rten=20Gustafson?= Date: Thu, 13 Oct 2016 15:06:06 +0200 Subject: [PATCH 053/140] Point out that AWS API Gateway resources can't be made "network internal" but that they may requrie authentication. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c5457d2..196cfa3 100644 --- a/README.md +++ b/README.md @@ -1069,6 +1069,7 @@ API Gateway - 🔸API Gateway only supports encrypted (https) endpoints, and does not support unencrypted HTTP. (This is probably a good thing.) - 🔸API Gateway endpoints are public — there is no mechanism to build private endpoints, e.g. for internal use. +- 🔸API Gateway endpoints are always public, i.e. internet facing, and there is no mechanism to build private endpoints, e.g. for internal use on a [VPC](#vpcs-network-security-and-security-groups) but endpoints and their related resources can, optionally, [require authentication](http://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html). 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 2b4cb675386b900bb960adaaeb61db71ab5613e6 Mon Sep 17 00:00:00 2001 From: "Bhasin, Krishan" Date: Thu, 13 Oct 2016 14:08:38 +0100 Subject: [PATCH 054/140] Added v1 of information on Kinesis --- README.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/README.md b/README.md index c5457d2..8a694c9 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,8 @@ Table of Contents | [DirectConnect](#directconnect) | [📗](#directconnect-basics) | [📘](#directconnect-tips) | | | [Redshift](#redshift) | [📗](#redshift-basics) | [📘](#redshift-tips) | [📙](#redshift-gotchas-and-limitations) | | [EMR](#emr) | [📗](#emr-basics) | [📘](#emr-tips) | [📙](#emr-gotchas-and-limitations) | +| [Kinesis Streams](#kinesis-streams) | [📗](#kinesis-streams-basics) | [📘](#kinesis-streams-tips) | [📙](#kinesis-streams-gotchas-and-limitations) | + **Special Topics** @@ -1273,7 +1275,33 @@ EMR - 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. - 💸 Beware of “double-dipping”. With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. +Kinesis Streams +--- +### Kinesis Streams Basics + +- 📒 [Homepage](https://aws.amazon.com/kinesis/streams/) ∙ [Developer guide](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) ∙ [FAQ](https://aws.amazon.com/kinesis/streams/faqs/) ∙ [Pricing](https://aws.amazon.com/kinesis/streams/pricing/) +- **Kinesis Streams** (which used to be only called Kinesis, before Kinesis Firehose and Kinesis Analytics were launched) is a service that allows you to injest high-throughput data streams for immediate or delayed processing by other AWS services +- Kinesis Streams' subcomponents are called [Shards](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). Each shard provides 1MB/s of write capacity and 2MB/s of read capacity at a maximum of 5 reads per second. A stream can have its Shards programatically increased or decreased based on a variety of metrics +- All records entered into a Kinesis Stream are kept in time-order, but obviously it is up to the developer to ensure they are entered into the Stream in the correct order. + +### Kinesis Streams Alternatives and Lock-in + +- ⛓Kinesis is most closely compared to [Apache Kafka](https://kafka.apache.org/), an open-source data injestion solution. It is possible to set up a Kafka cluster hosted on [EC2 instances](#ec2) (or any other VPS), however you are responsible for managing and maintaining both Zookeeper and the Kafka brokers in a highly availible configuration. Confluent have a good blog post on their recommendations on doing this [here](http://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/), which has links on the bottom to several other blogs they have written on the subject. +- Kinesis uses very AWS-specific APIs and terms (e.g. Shards), so you should be aware of the potential future costs of migrating away from it, should you choose to use it. + +### Kinesis Streams Tips + +- The [KCL](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html) (Kinesis Client Library) is a very useful Java program (wrapped in a Multi-Language Daemon that makes it useable in Java, Node, Python, Ruby and .NET programs) that provides very simple interfaces for clients to use when consuming data from a Kinesis Stream. It provides the skeleton for 3 basic functions - ```intitialise```, ```process-records```, and ```shutdown```. As a developer, all you need to do is set up the config file to point at the correct Kinesis Stream, and fill out the provided functions in order to start consuming data from Kinesis. + - The KCL uses a DynamoDB table to keep track of which records have been processed by the KCL. This ensures that all records are processed 'at least once'. It is up to the developer to ensure that the program can handle doubly-processed records. + - The KCL also uses DynamoDB to keep track of other KCL 'workers'. It automatically shares the available Kinesis Shards across all the workers as equally as possible. + +### Kinesis Streams Gotchas and Limitations +- 💸❗**Kinesis Streams are not included in the free tier!** Make sure if you do any experimentation with it on a personal account, you shut down the stream, or you can run up unexpected costs (~$11 per shard-month) +- Kinesis Streams' shards each only permit 5 reads per second. If you are using ```n``` shards in a particular stream, and evenly distributing your data across all of them, you will end up with a total of 5 reads per second. This is because a consumer cannot know which shard will contain new data, and will therefore need to check every single one. This means that there is a hard limit on the number of consumers you can have per stream, for any given latency. + - If you wish to have 5 consumers all reading data from one Stream with 5 shards, with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams, or reduce your latency requirements. + - Each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second - a value in excess of the maximum. + - There is a good blog by Brandur, an engineer at Stripe that discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). High Availability ----------------- From 3b6311d816a3140a4ecdb044f36effcd9b3262d6 Mon Sep 17 00:00:00 2001 From: "Bhasin, Krishan" Date: Thu, 13 Oct 2016 14:11:36 +0100 Subject: [PATCH 055/140] fixed minor grammatical error --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8a694c9..9ed074f 100644 --- a/README.md +++ b/README.md @@ -1301,7 +1301,7 @@ Kinesis Streams - Kinesis Streams' shards each only permit 5 reads per second. If you are using ```n``` shards in a particular stream, and evenly distributing your data across all of them, you will end up with a total of 5 reads per second. This is because a consumer cannot know which shard will contain new data, and will therefore need to check every single one. This means that there is a hard limit on the number of consumers you can have per stream, for any given latency. - If you wish to have 5 consumers all reading data from one Stream with 5 shards, with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams, or reduce your latency requirements. - Each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second - a value in excess of the maximum. - - There is a good blog by Brandur, an engineer at Stripe that discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). + - There is a good blog by Brandur, an engineer at Stripe, that discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). High Availability ----------------- From e1014b3a04169ab07e67d7b72904fbe4c46ab3f3 Mon Sep 17 00:00:00 2001 From: Ashley Davis Date: Thu, 13 Oct 2016 16:10:37 +0100 Subject: [PATCH 056/140] Add section detailling testing with random instance termination (#158) * Add section detailling testing with random instance termination Includes Netflix "Chaos Monkey" and BBC "chaos-lambda". * #158: Change to unicode quotes and minor grammar changes --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c5457d2..ca35d2d 100644 --- a/README.md +++ b/README.md @@ -509,6 +509,7 @@ This guide is about AWS, not DevOps or server configuration management in genera ### Tips - **NTP and accurate time:** If you are not using Amazon Linux (which comes preconfigured), you should confirm your servers [configure NTP correctly](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure_ntp), to avoid insidious time drift (which can then cause all sorts of issues, from breaking API calls to misleading logs). This should be part of your automatic configuration for every server. If time has already drifted substantially (generally >1000 seconds), remember NTP won’t shift it back, so you may need to remediate manually (for example, [like this](http://askubuntu.com/questions/254826/how-to-force-a-clock-update-using-ntp) on Ubuntu). +- **Testing immutable infrastructure:** If you want to be proactive about testing your service’s ability to cope with instance termination or failure, it can be helpful to introduce random instance termination during business hours, which will expose any such issues at a time when engineers are available to identify and fix them. Netflix’s [Simian Army](https://github.com/Netflix/SimianArmy) (specifically, [Chaos Monkey](https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey)) is a popular tool for this. Alternatively, [chaos-lambda](https://github.com/bbc/chaos-lambda) by the BBC is a lightweight option which runs on AWS [Lambda](#lambda). Security and IAM ---------------- From ba3be9955477ee1e60c0ff0cdd6ec19fb93b8bf3 Mon Sep 17 00:00:00 2001 From: Chris Lennon Date: Thu, 13 Oct 2016 17:11:53 +0200 Subject: [PATCH 057/140] Adds gotcha for DynamoDB relating to empty strings (#145) * Adds gotcha for DynamoDB relating to empty strings * Changes phrasing of gotcha --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index ca35d2d..df45a13 100644 --- a/README.md +++ b/README.md @@ -998,6 +998,7 @@ DynamoDB - 🔸 DynamoDB doesn’t provide an easy way to bulk-load data (it is possible through [Data Pipeline](http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html), and this has some [unfortunate consequences](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.AvoidExcessivePTIncreases). Since you need to use the regular service APIs to update existing or create new rows, it is common to temporarily turn up a destination table’s write throughput to speed import. But when the table’s write capacity is increased, DynamoDB may do an irreversible split of the partitions underlying the table, spreading the total table capacity evenly across the new generation of tables. Later, if the capacity is reduced, the capacity for each partition is also reduced but the total number of partitions is not, leaving less capacity for each partition. This leaves the table in a state where it much easier for hotspots to overwhelm individual partitions. - It is important to make sure that DynamoDB [resource limits](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-data-types) are compatible with your dataset and workload. For example, the maximum size value that can be added to a DynamoDB table is 400 KB (larger items can be stored in S3 and a URL stored in DynamoDB). +- DynamoDB does [not allow](https://forums.aws.amazon.com/thread.jspa?threadID=90137) an empty string as a valid attribute value. The most common work-around is to use a substitute value instead of leaving the field empty. ECS --- From 2e04e5dda9cd1718b11275bc3616fe974cde2aa1 Mon Sep 17 00:00:00 2001 From: Martijn Dwars Date: Thu, 13 Oct 2016 22:43:46 +0200 Subject: [PATCH 058/140] Improvements --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bf2f4a8..2bc26a5 100644 --- a/README.md +++ b/README.md @@ -726,7 +726,7 @@ EC2 - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. -- 🔸Instances come in two types: Fixed Performance Instances (e.g. M3, C3, and R3) and Burstable Performance Instances (e.g. T2). Each T2 instance receives CPU Credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU Credits when they are idle, and use CPU credits when they are active. However, once it runs out of credits, you'll notice a severe degrade in performance. CPU-bound tasks (such as scientific computation) are therefore better suited for Fixed Performance Instances. +- 🔸Instances come in two types: Fixed Performance Instances (e.g. M3, C3, and R3) and [Burstable Performance Instances](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. AMIs ---- From 71b3cbc0fd03b01389ffdf166fe65a8ccdc8d226 Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Fri, 14 Oct 2016 10:24:07 +0300 Subject: [PATCH 059/140] Added some Redshift tips. --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 2fac26f..599203a 100644 --- a/README.md +++ b/README.md @@ -1252,6 +1252,9 @@ Redshift - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). +- 🔹 If you need to change sort key or dist key of a table, you need to create a new table with new key and move your data to new table with insert into new_table select * from old_table. [Choosing Sort Key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) +- ❗🚪 When moving data with insert into x select from y; you need to have at least twice of y table size free space on your harddisk. Redshift first copies data to disk then to new table. +- 🔹 Vacuum delete only does not block copy commands, but vacuum reindex can block if you execute copy command every minute or so. EMR --- From d433e5c62c7a14b454cda8af3e31aeb6fe795eea Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Fri, 14 Oct 2016 11:15:04 +0300 Subject: [PATCH 060/140] Lambda stopping issue added. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 599203a..cee2890 100644 --- a/README.md +++ b/README.md @@ -1055,6 +1055,7 @@ Lambda - 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications. - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. +- ❗Currently (October,2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From c99bddb4ee2437587f1e188d47be2bb1da01f81d Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Fri, 14 Oct 2016 11:28:32 +0300 Subject: [PATCH 061/140] Lambda stopping issue added. --- README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/README.md b/README.md index cee2890..18829e9 100644 --- a/README.md +++ b/README.md @@ -1253,9 +1253,6 @@ Redshift - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). -- 🔹 If you need to change sort key or dist key of a table, you need to create a new table with new key and move your data to new table with insert into new_table select * from old_table. [Choosing Sort Key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) -- ❗🚪 When moving data with insert into x select from y; you need to have at least twice of y table size free space on your harddisk. Redshift first copies data to disk then to new table. -- 🔹 Vacuum delete only does not block copy commands, but vacuum reindex can block if you execute copy command every minute or so. EMR --- From eecf85b73f2a4fe778692ea81d95f9f2b97dcbe6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A5rten=20Gustafson?= Date: Fri, 14 Oct 2016 10:54:27 +0200 Subject: [PATCH 062/140] Remove redundant point about API Gateway endpoints --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 196cfa3..c648a4a 100644 --- a/README.md +++ b/README.md @@ -1068,7 +1068,6 @@ API Gateway ### API Gateway Gotchas and Limitations - 🔸API Gateway only supports encrypted (https) endpoints, and does not support unencrypted HTTP. (This is probably a good thing.) -- 🔸API Gateway endpoints are public — there is no mechanism to build private endpoints, e.g. for internal use. - 🔸API Gateway endpoints are always public, i.e. internet facing, and there is no mechanism to build private endpoints, e.g. for internal use on a [VPC](#vpcs-network-security-and-security-groups) but endpoints and their related resources can, optionally, [require authentication](http://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html). 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 2668a662f8de8dcc71c8d5aeda146a341930b0c0 Mon Sep 17 00:00:00 2001 From: Jaanus Torp Date: Fri, 14 Oct 2016 16:20:33 +0100 Subject: [PATCH 063/140] Added note about using Source Security Groups (#144) * Added note about using Source Security Groups * Updated based on suggestions --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2fac26f..22aa10d 100644 --- a/README.md +++ b/README.md @@ -1147,6 +1147,7 @@ VPCs, Network Security, and Security Groups - You expose a smaller surface area for attack compared to exposing separate (potentially authenticated) services over the public internet. - e.g. A bug in the YAML parser used by the Ruby on Rails admin site is much less serious when the admin site is only visible to the private network and accessed through VPN. - Another common pattern (especially as deployments get larger, security or regulatory requirements get more stringent, or team sizes increase) is to provide a [bastion host](https://www.pandastrike.com/posts/20141113-bastion-hosts) behind a VPN through which all SSH connections need to transit. +- 🔹Consider using other security groups as sources for security group rules instead of using CIDRs — that way, all hosts in the source security group and only hosts in that security group are allowed access. This is a much more dynamic and secure way of managing security group rules. ### VPC and Network Security Gotchas and Limitations From 671582742a571d893451bd4e28921b4e55cda3f5 Mon Sep 17 00:00:00 2001 From: Kyle Busekist Date: Fri, 14 Oct 2016 13:37:54 -0500 Subject: [PATCH 064/140] Change wording from NACL to Network ACL --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 74ab2fe..b22bd69 100644 --- a/README.md +++ b/README.md @@ -858,7 +858,7 @@ CLB - Complex rules for directing traffic are not supported. For example, you can’t direct traffic based on a regular expression in the URL, like [HAProxy](http://www.haproxy.org/) offers. - **Apex DNS names:** Once upon a time, you couldn’t assign an CLB to an apex DNS record (i.e. example.com instead of foo.example.com) because it needed to be an A record instead of a CNAME. This is now possible with a Route 53 alias record directly pointing to the load balancer. - 🔸CLBs use [HTTP keep-alives](https://en.wikipedia.org/wiki/HTTP_persistent_connection) on the internal side. This can cause an unexpected side effect: Requests from different clients, each in their own TCP connection on the external side, can end up on the same TCP connection on the internal side. Never assume that multiple requests on the same TCP connection are from the same client! -- Traffic between CLBs and back-end instances in the same subnet **will** have NACL rules evaluated (EC2 to EC2 traffic in the same subnet would not have NACL rules evaluated). If the default '0.0.0.0/0 ALLOW' rule is removed from the NACL applied to the subnet, a rule that allows traffic on both the health check port and any listener port must be added. +- 🔸 Traffic between CLBs and back-end instances in the same subnet **will** have [Network ACL](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html) rules evaluated (EC2 to EC2 traffic in the same subnet would not have Network ACL rules evaluated). If the default '0.0.0.0/0 ALLOW' rule is removed from the Network ACL applied to the subnet, a rule that allows traffic on both the health check port and any listener port must be added. ALB --- From 019e201aee660eedafa18473eee5f2abd97c8c4b Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Fri, 14 Oct 2016 14:42:10 -0700 Subject: [PATCH 065/140] Adding IoT section First PR on new IoT section. NOTE: I did NOT update the README.md index to add the links for the new IoT section yet. --- README.md | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6da04b0..4ea18a7 100644 --- a/README.md +++ b/README.md @@ -1278,8 +1278,6 @@ EMR - 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. - 💸 Beware of “double-dipping”. With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. - - High Availability ----------------- @@ -1308,6 +1306,59 @@ This section covers tips and information on achieving [high availability](https: - **AZ naming** differs from one customer account to the next. Your “us-west-1a” is not the same as another customer’s “us-west-1a” — the letters are assigned to physical AZs randomly per account. This can also be a gotcha if you have multiple AWS accounts. - **Cross-AZ traffic** is not free. At large scale, the costs add up to a significant amount of money. If possible, optimize your traffic to stay within the same AZ as much as possible. +IoT +--------------------------- + + +### IoT Basics + +* 📒 [Homepage](https://aws.amazon.com/iot/) ∙ [User guide](https://docs.aws.amazon.com/iot/latest/developerguide/) ∙ [FAQ](https://aws.amazon.com/iot/faqs/) ∙ [Pricing](https://aws.amazon.com/iot/pricing/) +* **IoT** is a platform for allowing clients such as IoT devices or software applications ([examples](http://internetofthingswiki.com/iot-applications-examples/541/)) to communicate with the AWS cloud. +* Clients are also called **devices** (or **things**) and include a wide variety of device types. Roughly there are three categories of device types that interact with IoT services by sending message over an IoT protocol to the IoT Pub/Sub-style message broker, which is called the IoT** Device Gateway**: + * Send messages only: For example, the [AWS IoT Button](https://aws.amazon.com/iot/button/) on an [eddystone beacon](http://developer.estimote.com/eddystone/). + * Send and receive messages: For example, the [Phillips Home Safe Medical Alert device](http://rethink-iot.com/2015/10/16/philips-uses-new-aws-iot-platform-to-expand-its-healthsuite-cloud/) + * Send, receive, and process messages: For example, a simple processing board, such as a **Raspberry Pi** ([quick start guide](http://docs.aws.amazon.com/iot/latest/developerguide/iot-device-sdk-c.html)), or an AWS device, such as [Echo or Echo Dot](https://developer.amazon.com/echo), which are designed to work with the [AWS Alexa skills kit](https://developer.amazon.com/alexa-skills-kit) (a programmable voice-enabled service from AWS). +* AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. +* **IoT terms:** + * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). + * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). + * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. + * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). + * Shown below is a [diagram](https://aws.amazon.com/iot/how-it-works/) which summarizes the flow of messages between the AWS IoT services: + * [Image: https://d0.awsstatic.com/IoT/diagrams/awsiot-how-it-works_HowITWorks_1-26.png] + +### IoT Alternatives and Lock-in + +* AWS, Microsoft and Google have all introduced IoT-specific sets of cloud services since late 2015. AWS was first, moving their IoT services to [general availability](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) in Dec 2015. Microsoft released their set of IoT services for Azure in [Feb 2016](https://azure.microsoft.com/en-us/updates/generally-available-microsoft-azure-iot-hub/). Google has only previewed, but not released their IoT services [Brillo](https://developers.google.com/brillo/) and [Weave](https://developers.google.com/weave/). +* Issues of lock-in center around your devices — [protocols](http://www.postscapes.com/internet-of-things-protocols/) (for example MQTT, AMQP), message formats (such as, JSON vs. Hex...) and security (certificates). + +### IoT Tips + +* **Getting started with Buttons:** One way to start is to use an [**AWS IoT Button**](https://aws.amazon.com/iot/button/). AWS provides a number of code samples for use with their IoT Button, you can use the AWS IoT console, click the “connect AWS IoT button” link and you'll be taken to the AWS Lambda console. There you fill out your button’s serial number to associate it with a Lambda. (As of this writing, AWS IoT buttons are only available for sale in the US.) +* **Connections and protocols:** It is important to understand detail about the devices you wish to connect to the AWS IoT service, including how you will secure the device connections, the device protocols, and more. Cloud vendors differ significantly in their support for common IoT protocols, such as MQTT, AMQP, XMPP. AWS IoT supports **secure MQTT**, **WebSockets** and **HTTPS**. +* Support for **device security** via certificate processing is a key differentiator in this space. In August 2016, AWS added [just-in-time registrations](https://aws.amazon.com/blogs/iot/just-in-time-registration-of-device-certificates-on-aws-iot/) for IoT devices to their services. +* **Combining with other services:** It's common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). +* **Testing tools:** + * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators + * Use this [MQTT load simulator](https://github.com/awslabs/aws-iot-mqtt-load-generator) to test device message load throughout your IoT solution. + +### IoT Gotchas + +* **IoT protocols:** It is important to verify the exact type of support for your particular IoT device message protocol. For example, one commonly used IoT protocol is [MQTT](https://www.ibm.com/developerworks/community/blogs/5things/entry/5_things_to_know_about_mqtt_the_protocol_for_internet_of_things?lang=en). Within MQTT there are [three possible levels of QoS in MQTT](https://dzone.com/articles/internet-things-mqtt-quality). AWS IoT supports MQTT [QoS 0](http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) (fire and forget, or at most once) and QoS 1(at least once, or includes confirmation), but *not* QoS 2 (exactly once, requires 4-step confirmation). This is important in understanding how much code you’ll need to write for your particular application message resolution needs. Here is a [presentation about the nuances of connecting](http://www.slideshare.net/AmazonWebServices/overview-of-iot-infrastructure-and-connectivity-at-aws-getting-started-with-aws-iot). +* The ecosystems to match **IAM users or roles** to **IoT policies** and their associated authorized AWS IoT devices is immature. Custom coding to enforce your security requirements is common. +* A common mistake is to misunderstand the importance of IoT **device** **security**. It is imperative to associate *each* device with a unique certificate (public key). You can generate your own certificates and upload them to AWS, or you can use AWS generated IoT device certificates. It’s best to read and understand AWS’s own guidance on this [topic](http://www.slideshare.net/AmazonWebServices/best-practices-of-iot-in-the-cloud). +* There is only one **AWS IoT Gateway** (endpoint) per AWS account. For production scenarios, you’ll probably need to set up multiple AWS accounts in order to separate device traffic for development, test and production. It’s interesting to note that the [Azure IoT Gateway](https://azure.microsoft.com/en-us/documentation/articles/iot-hub-protocol-gateway/) supports configuration of multiple endpoints, so that a single Azure account can be used with separate pub/sub endpoints for development, testing and production +* **Limits:** Be aware of [limits](http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html), including device message size, type, frequency, and number of AWS IoT rules. + +### IoT Code Samples + +* [Simple Beer Service](https://github.com/awslabs/simplebeerservice) is a surprisingly useful code example using AWS IoT, Lambda, etc. +* [IoT-elf](https://github.com/awslabs/aws-iot-elf) offers clean Python sample using the AWS IoT SDK. +* [IoT Button projects](https://www.hackster.io/AmazonWebServices/products/aws-iot-button) on Hackster include many different code samples for projects. +* [5 IoT code examples](https://github.com/awslabs/aws-iot-examples/): a device simulator, MQTT sample, just in time registration, truck simulator, prediction data simulator. +* [AWS Alexa trivia voice example](https://developer.amazon.com/public/community/post/TxDJWS16KUPVKO/New-Alexa-Skills-Kit-Template:-Build-a-Trivia-Skill-in-under-an-Hour) is a quick-start using Alexa voice capability and Lambda. +* Some Raspberry Pi examples include the [Beacon project](https://github.com/araobp/beacon/blob/master/README.md), [Danbo](https://github.com/awslabs/aws-iot-demo-for-danbo), and [GoPiGo](https://github.com/awslabs/aws-iotbot). + Billing and Cost Management --------------------------- From 5f185aaf9db7234289039f9c00dbb31e52a76981 Mon Sep 17 00:00:00 2001 From: Carlos Date: Sat, 15 Oct 2016 04:56:33 +0200 Subject: [PATCH 066/140] ELB limitation, SNI (#146) * ELB limitation, SNI Add a limitation in the ELB section * reword CLB limitation with suggestion * remove duplicate period --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6da04b0..a8ec388 100644 --- a/README.md +++ b/README.md @@ -868,6 +868,7 @@ Load Balancers - 🔸As an operator of a service behind an CLB or ALB, the latter phenomenon means you can also see puzzling or erroneous requests by clients of other companies. This is most common with clients using back-end APIs (since web browsers typically cache for a limited period). - ❗CLBs and ALBs take time to scale up, it does not handle sudden spikes in traffic well. Therefore, if you anticipate a spike, you need to “pre-warm” the load balancer by gradually sending an increasing amount of traffic. - ❗Tune your healthchecks carefully — if you are too aggressive about deciding when to remove an instance and conservative about adding it back into the pool, the service that your load balancer is fronting may become inaccessible for seconds or minutes at a time. Be extra careful about this when an autoscaler is configured to terminate instances that are marked as being unhealthy by a managed load balancer. +- ❗CLB HTTPS listeners don't support Server Name Indication (SNI). If you need SNI, you can work around this limitation by either providing a certificate with Subject Alternative Names (SANs) or by using TCP listeners and terminating SSL at your backend. CLB --- From 928993ec49c1d58cc39248664a6f21af68bd049e Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Fri, 14 Oct 2016 20:00:25 -0700 Subject: [PATCH 067/140] added index line for new IoT section --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4ea18a7..26a8c2a 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ Table of Contents | [DirectConnect](#directconnect) | [📗](#directconnect-basics) | [📘](#directconnect-tips) | | | [Redshift](#redshift) | [📗](#redshift-basics) | [📘](#redshift-tips) | [📙](#redshift-gotchas-and-limitations) | | [EMR](#emr) | [📗](#emr-basics) | [📘](#emr-tips) | [📙](#emr-gotchas-and-limitations) | +| [IoT](#iot) | [📗](#iot-basics) | [📘](#iot-tips) | [📙](#iot-gotchas-and-limitations) | **Special Topics** @@ -1342,7 +1343,7 @@ IoT * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators * Use this [MQTT load simulator](https://github.com/awslabs/aws-iot-mqtt-load-generator) to test device message load throughout your IoT solution. -### IoT Gotchas +### IoT Gotchas and Limitations * **IoT protocols:** It is important to verify the exact type of support for your particular IoT device message protocol. For example, one commonly used IoT protocol is [MQTT](https://www.ibm.com/developerworks/community/blogs/5things/entry/5_things_to_know_about_mqtt_the_protocol_for_internet_of_things?lang=en). Within MQTT there are [three possible levels of QoS in MQTT](https://dzone.com/articles/internet-things-mqtt-quality). AWS IoT supports MQTT [QoS 0](http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) (fire and forget, or at most once) and QoS 1(at least once, or includes confirmation), but *not* QoS 2 (exactly once, requires 4-step confirmation). This is important in understanding how much code you’ll need to write for your particular application message resolution needs. Here is a [presentation about the nuances of connecting](http://www.slideshare.net/AmazonWebServices/overview-of-iot-infrastructure-and-connectivity-at-aws-getting-started-with-aws-iot). * The ecosystems to match **IAM users or roles** to **IoT policies** and their associated authorized AWS IoT devices is immature. Custom coding to enforce your security requirements is common. From 2fce07ea182c8a2f312757a782712056d81533c7 Mon Sep 17 00:00:00 2001 From: Bo Bayles Date: Fri, 14 Oct 2016 22:00:31 -0500 Subject: [PATCH 068/140] Add VPC Flow Logs for Network Security (#102) * Add note about VPC Flow Logs for security * Add Sumo Logic and Observable for VPC Flow Logs * Revert manual change to AUTHORS.md * 3rd party to third party * Remove third party services for VPC Flow Logs --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a8ec388..5d4088b 100644 --- a/README.md +++ b/README.md @@ -1149,7 +1149,10 @@ VPCs, Network Security, and Security Groups - You expose a smaller surface area for attack compared to exposing separate (potentially authenticated) services over the public internet. - e.g. A bug in the YAML parser used by the Ruby on Rails admin site is much less serious when the admin site is only visible to the private network and accessed through VPN. - Another common pattern (especially as deployments get larger, security or regulatory requirements get more stringent, or team sizes increase) is to provide a [bastion host](https://www.pandastrike.com/posts/20141113-bastion-hosts) behind a VPN through which all SSH connections need to transit. -- 🔹Consider using other security groups as sources for security group rules instead of using CIDRs — that way, all hosts in the source security group and only hosts in that security group are allowed access. This is a much more dynamic and secure way of managing security group rules. +- 🔹Consider using other security groups as sources for security group rules instead of using CIDRs — that way, all hosts in the source security group and only hosts in that security group are allowed access. This is a much more dynamic and secure way of managing security group rules. +- **VPC Flow Logs** allow you to monitor the network traffic to, from, and within your VPC. Logs are stored in CloudWatch Logs groups, and can be used for security monitoring (with third party tools), performance evaluation, and forensic investigation. + - See the [VPC Flow Logs User Guide](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html) for basic information. + - See the [flowlogs-reader](https://github.com/obsrvbl/flowlogs-reader) CLI tool and Python library to retrieve and work with VPC Flow Logs. ### VPC and Network Security Gotchas and Limitations From 994ca7b864e0f7f26df773d64795bfffc01d727e Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Sat, 15 Oct 2016 12:45:49 +0300 Subject: [PATCH 069/140] date typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 18829e9..e671531 100644 --- a/README.md +++ b/README.md @@ -1055,7 +1055,7 @@ Lambda - 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications. - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. -- ❗Currently (October,2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future. +- ❗Currently (as of October, 2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 09b785d9e531638972b22eca23cfe510515289fc Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Sat, 15 Oct 2016 16:27:57 +0200 Subject: [PATCH 070/140] Add initial info for AWS Certificate Manager --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 5d4088b..8515fd9 100644 --- a/README.md +++ b/README.md @@ -212,6 +212,7 @@ General Information - ⛓[KMS](#kms): Store and manage encryption keys securely - [Inspector](https://aws.amazon.com/inspector/): Security audit - [Trusted Advisor](https://aws.amazon.com/premiumsupport/trustedadvisor/): Automated tips on reducing cost or making improvements + - 🐥[Certificate Manager](https://aws.amazon.com/certificate-manager/): Manage SSL/TLS certificates for AWS services - **Compound services:** These are similarly specific, but are full-blown services that tackle complex problems and may tie you in. Usefulness depends on your requirements. If you have large or significant need, you may have these already managed by in-house systems and engineering teams. - [Machine Learning](https://aws.amazon.com/machine-learning/): Machine learning model training and classification - ⛓🕍[Data Pipeline](https://aws.amazon.com/datapipeline/): Managed ETL service @@ -292,6 +293,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | User authentication | Cognito | | | Azure Active Directory | | oauth.io | | Mobile app analytics | Mobile Analytics | Firebase Analytics | | HockeyApp | Mixpanel | | | Mobile app testing | Device Farm | Firebase Test Lab | | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | +| Managing SSL/TLS certificates | Certificate Manager | | | | Let's Encrypt, Comodo, Symantec, GlobalSign | 🚧 [*Please help fill this table in.*](CONTRIBUTING.md) @@ -307,6 +309,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | Service | Original release | Availability | CLI Support | |------------------------------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------|:-----------:| | 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | | +| 🐥[Certificate Manager](https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/) | 2016-01 | General | | 🐥[IoT](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) | 2015-08 | General | ✓ | | 🐥[WAF](https://aws.amazon.com/releasenotes/AWS-WAF?browse=1) | 2015-10 | General | ✓ | | 🐥[Data Pipeline](https://aws.amazon.com/releasenotes/AWS-Data-Pipeline?browse=1) | 2015-10 | General | ✓ | From d71844c57029101dfa2bcd94eb8786461076a06a Mon Sep 17 00:00:00 2001 From: Mohit Agarwal Date: Sat, 15 Oct 2016 20:33:44 +0530 Subject: [PATCH 071/140] Add a link to AWS whitepapers Link to whitepapers section features a comprehensive list of technical AWS whitepapers, covering topics such as architecture, security and economics. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5d4088b..c59fe63 100644 --- a/README.md +++ b/README.md @@ -1399,6 +1399,7 @@ This section covers a few unusually useful or “must know about” resources or - [AWS In Plain English](https://www.expeditedssl.com/aws-in-plain-english): A readable overview of all the AWS services - [Awesome AWS](https://github.com/donnemartin/awesome-aws): A curated list of AWS tools and software - [AWS Tips I Wish I'd Known Before I Started](https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/): A list of tips from [Rich Adams](https://richadams.me/) + - [AWS Whitepapers](https://aws.amazon.com/whitepapers/): A list of technical AWS whitepapers, covering topics such as architecture, security and economics. - General references - [Awesome Microservices](https://github.com/mfornos/awesome-microservices): A curated list of tools and technologies for microservice architectures. Worth browsing to learn about popular open source projects. - [Is it fast yet?](https://istlsfastyet.com/): Ilya Grigorik’s TLS performance overview From 67d838994934e103be9035f7715001b4755b3a6c Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Sat, 15 Oct 2016 17:19:17 +0200 Subject: [PATCH 072/140] Add basic info about Device Farm --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index 5d4088b..a88ab88 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ Table of Contents | [DirectConnect](#directconnect) | [📗](#directconnect-basics) | [📘](#directconnect-tips) | | | [Redshift](#redshift) | [📗](#redshift-basics) | [📘](#redshift-tips) | [📙](#redshift-gotchas-and-limitations) | | [EMR](#emr) | [📗](#emr-basics) | [📘](#emr-tips) | [📙](#emr-gotchas-and-limitations) | +| [Device Farm](#device-farm) | [📗](#device-farm-basics) | | | **Special Topics** @@ -1282,7 +1283,16 @@ EMR - 💸❗**EMR costs** can pile up quickly since it involves lots of instances, efficiency can be poor depending on cluster configuration and choice of workload, and accidents like hung jobs are costly. See the [section on EC2 cost management](#ec2-cost-management), especially the tips there about Spot instances and avoiding hourly billing. [This blog post](http://engineering.bloomreach.com/strategies-for-reducing-your-amazon-emr-costs/) has additional tips. - 💸 Beware of “double-dipping”. With EMR, you pay for the EC2 capacity and the service fees. In addition, EMR syncs task logs to S3, which means you pay for the storage and **PUT requests** at [S3 standard rates](https://aws.amazon.com/s3/pricing/#Request_Pricing). While the log files tend to be relatively small, every Hadoop job, depending on the size, generates thousands of log files that can quickly add up to thousands of dollars on the AWS bill. YARN's [log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/) is not available on EMR. +Device Farm +----------- +### Device Farm Basics +- 📒 [Homepage](https://aws.amazon.com/device-farm/) ∙ [Developer guide](http://docs.aws.amazon.com/devicefarm/latest/developerguide/) ∙ [FAQ](https://aws.amazon.com/device-farm/faq/) ∙ [Pricing](https://aws.amazon.com/device-farm/pricing/) +- **Device Farm** is an AWS service that enables mobile app testing on real devices. +- Supports iOS and Android (including Kindle Fire) devices, as well as the mobile web. +- Supports remote device access in order to allow for interactive testing/debugging. + +🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) High Availability ----------------- From dedc5d7e668dfd83cef9842e3d01a44dc6df7b1d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Kostka?= Date: Sat, 15 Oct 2016 21:47:18 +0200 Subject: [PATCH 073/140] Add EC2 IAM role gotcha --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5d4088b..f11df0a 100644 --- a/README.md +++ b/README.md @@ -739,6 +739,7 @@ EC2 - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. +- ❗Once EC2 instance is launched, IAM role can not be assigned to it. AMIs ---- From f29f6f992a5961220ab4e9f06a97569b8bb23545 Mon Sep 17 00:00:00 2001 From: 0xmohit Date: Sun, 16 Oct 2016 01:39:36 +0530 Subject: [PATCH 074/140] add resource for information on managing multiple SSH keys (#210) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4930a9b..7f6e831 100644 --- a/README.md +++ b/README.md @@ -726,7 +726,7 @@ EC2 - When you start an instance, you need to have at least one [ssh key pair](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) set up, to bootstrap, i.e., allow you to ssh in the first time. - Aside from bootstrapping, you should manage keys yourself on the instances, assigning individual keys to individual users or services as appropriate. - Avoid reusing the original boot keys except by administrators when creating new instances. - - How to avoid sharing keys; how to add individual ssh keys for individual users. + - Avoid sharing keys and [add individual ssh keys](http://security.stackexchange.com/questions/87480/managing-multiple-ssh-private-keys-for-a-team) for individual users. - **GPU support:** You can rent GPU-enabled instances on EC2. There are [two instance types](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html). Both sport an NVIDIA card (K520, 1536 CUDA cores and M2050, 448 CUDA cores). ### EC2 Gotchas and Limitations From ea1cc77469274cdadd3ed232c13b7ee3b79d4b6d Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sat, 15 Oct 2016 13:10:27 -0700 Subject: [PATCH 075/140] addresses requested changes by editors from Oct 14 --- README.md | 67 +++++++++++++++++++++++++++---------------------------- 1 file changed, 33 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 26a8c2a..87001d7 100644 --- a/README.md +++ b/README.md @@ -1310,55 +1310,54 @@ This section covers tips and information on achieving [high availability](https: IoT --------------------------- - ### IoT Basics -* 📒 [Homepage](https://aws.amazon.com/iot/) ∙ [User guide](https://docs.aws.amazon.com/iot/latest/developerguide/) ∙ [FAQ](https://aws.amazon.com/iot/faqs/) ∙ [Pricing](https://aws.amazon.com/iot/pricing/) -* **IoT** is a platform for allowing clients such as IoT devices or software applications ([examples](http://internetofthingswiki.com/iot-applications-examples/541/)) to communicate with the AWS cloud. -* Clients are also called **devices** (or **things**) and include a wide variety of device types. Roughly there are three categories of device types that interact with IoT services by sending message over an IoT protocol to the IoT Pub/Sub-style message broker, which is called the IoT** Device Gateway**: - * Send messages only: For example, the [AWS IoT Button](https://aws.amazon.com/iot/button/) on an [eddystone beacon](http://developer.estimote.com/eddystone/). - * Send and receive messages: For example, the [Phillips Home Safe Medical Alert device](http://rethink-iot.com/2015/10/16/philips-uses-new-aws-iot-platform-to-expand-its-healthsuite-cloud/) - * Send, receive, and process messages: For example, a simple processing board, such as a **Raspberry Pi** ([quick start guide](http://docs.aws.amazon.com/iot/latest/developerguide/iot-device-sdk-c.html)), or an AWS device, such as [Echo or Echo Dot](https://developer.amazon.com/echo), which are designed to work with the [AWS Alexa skills kit](https://developer.amazon.com/alexa-skills-kit) (a programmable voice-enabled service from AWS). -* AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. +* 📒 [Homepage](https://aws.amazon.com/iot/) ∙ [User guide](https://docs.aws.amazon.com/iot/latest/developerguide/) ∙ [FAQ](https://aws.amazon.com/iot/faqs/) ∙ [Pricing](https://aws.amazon.com/iot/pricing/) +- **IoT** is a platform for allowing clients such as IoT devices or software applications ([examples](http://internetofthingswiki.com/iot-applications-examples/541/)) to communicate with the AWS cloud. +- Clients are also called **devices** (or **things**) and include a wide variety of device types. Roughly there are three categories of device types that interact with IoT services by sending message over an IoT protocol to the IoT Pub/Sub-style message broker, which is called the IoT **Device Gateway**: + * Send messages only: For example, the [AWS IoT Button](https://aws.amazon.com/iot/button/) on an [eddystone beacon](http://developer.estimote.com/eddystone/). + * Send and receive messages: For example, the [Phillips Home Safe Medical Alert device](http://rethink-iot.com/2015/10/16/philips-uses-new-aws-iot-platform-to-expand-its-healthsuite-cloud/) + * Send, receive, and process messages: For example, a simple processing board, such as a **Raspberry Pi** ([quick start guide](http://docs.aws.amazon.com/iot/latest/developerguide/iot-device-sdk-c.html)), or an AWS device, such as [Echo or Echo Dot](https://developer.amazon.com/echo), which are designed to work with the [AWS Alexa skills kit](https://developer.amazon.com/alexa-skills-kit) (a programmable voice-enabled service from AWS). +- AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. * **IoT terms:** - * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). - * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). - * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. - * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). - * Shown below is a [diagram](https://aws.amazon.com/iot/how-it-works/) which summarizes the flow of messages between the AWS IoT services: - * [Image: https://d0.awsstatic.com/IoT/diagrams/awsiot-how-it-works_HowITWorks_1-26.png] + * AWS [**IoT Things**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html) (metadata for devices in a [registry](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-management.html)) and can store device state in a JSON document, which is called a [**device shadow**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-thing-shadows.html). + * AWS [**IoT Certificates**](http://docs.aws.amazon.com/iot/latest/developerguide/attach-cert-thing.html) (device authentication) are the logical association of a unique certificate to the logical representation of a device. This association can be done in the Console. In addition, the public key of the certificate must be copied to the physical device. This covers the authentication of devices to a particular AWS Device Gateway (or message broker). + * AWS [**IoT Policies**](http://docs.aws.amazon.com/iot/latest/developerguide/authorization.html) (device/topic authorization) are JSON files that are associated to one or more AWS IoT certificates. This authorizes associated devices to publish and/or subscribe to messages from one or more MQTT topics. + * AWS [**IoT Rules**](http://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) are SQL-like queries which allows for reuse of some or all device message data, as described in [this presentation, which summarizes design patterns with for IoT Rules](http://www.slideshare.net/AmazonWebServices/programming-the-physical-world-with-device-shadows-and-rules-engine-66486454). + * Shown below is a [diagram](https://aws.amazon.com/iot/how-it-works/) which summarizes the flow of messages between the AWS IoT services: + * ![How AWS IoT Works](https://d0.awsstatic.com/IoT/diagrams/awsiot-how-it-works_HowITWorks_1-26.png "How AWS IoT Works") ### IoT Alternatives and Lock-in -* AWS, Microsoft and Google have all introduced IoT-specific sets of cloud services since late 2015. AWS was first, moving their IoT services to [general availability](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) in Dec 2015. Microsoft released their set of IoT services for Azure in [Feb 2016](https://azure.microsoft.com/en-us/updates/generally-available-microsoft-azure-iot-hub/). Google has only previewed, but not released their IoT services [Brillo](https://developers.google.com/brillo/) and [Weave](https://developers.google.com/weave/). -* Issues of lock-in center around your devices — [protocols](http://www.postscapes.com/internet-of-things-protocols/) (for example MQTT, AMQP), message formats (such as, JSON vs. Hex...) and security (certificates). +- AWS, Microsoft and Google have all introduced IoT-specific sets of cloud services since late 2015. AWS was first, moving their IoT services to [general availability](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) in Dec 2015. Microsoft released their set of IoT services for Azure in [Feb 2016](https://azure.microsoft.com/en-us/updates/generally-available-microsoft-azure-iot-hub/). Google has only previewed, but not released their IoT services [Brillo](https://developers.google.com/brillo/) and [Weave](https://developers.google.com/weave/). +- Issues of lock-in center around your devices — [protocols](http://www.postscapes.com/internet-of-things-protocols/) (for example MQTT, AMQP), message formats (such as, JSON vs. Hex...) and security (certificates). ### IoT Tips -* **Getting started with Buttons:** One way to start is to use an [**AWS IoT Button**](https://aws.amazon.com/iot/button/). AWS provides a number of code samples for use with their IoT Button, you can use the AWS IoT console, click the “connect AWS IoT button” link and you'll be taken to the AWS Lambda console. There you fill out your button’s serial number to associate it with a Lambda. (As of this writing, AWS IoT buttons are only available for sale in the US.) -* **Connections and protocols:** It is important to understand detail about the devices you wish to connect to the AWS IoT service, including how you will secure the device connections, the device protocols, and more. Cloud vendors differ significantly in their support for common IoT protocols, such as MQTT, AMQP, XMPP. AWS IoT supports **secure MQTT**, **WebSockets** and **HTTPS**. -* Support for **device security** via certificate processing is a key differentiator in this space. In August 2016, AWS added [just-in-time registrations](https://aws.amazon.com/blogs/iot/just-in-time-registration-of-device-certificates-on-aws-iot/) for IoT devices to their services. -* **Combining with other services:** It's common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). -* **Testing tools:** - * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators - * Use this [MQTT load simulator](https://github.com/awslabs/aws-iot-mqtt-load-generator) to test device message load throughout your IoT solution. +- **Getting started with Buttons:** One way to start is to use an [**AWS IoT Button**](https://aws.amazon.com/iot/button/). AWS provides a number of code samples for use with their IoT Button, you can use the AWS IoT console, click the “connect AWS IoT button” link and you'll be taken to the AWS Lambda console. There you fill out your button’s serial number to associate it with a Lambda. (As of this writing, AWS IoT buttons are only available for sale in the US.) +- **Connections and protocols:** It is important to understand the details of about the devices you wish to connect to the AWS IoT service, including how you will secure the device connections, the device protocols, and more. Cloud vendors differ significantly in their support for common IoT protocols, such as MQTT, AMQP, XMPP. AWS IoT supports **secure MQTT**, **WebSockets** and **HTTPS**. +- Support for **device security** via certificate processing is a key differentiator in this space. In August 2016, AWS added [just-in-time registrations](https://aws.amazon.com/blogs/iot/just-in-time-registration-of-device-certificates-on-aws-iot/) for IoT devices to their services. +- **Combining with other services:** It's common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). +- **Testing tools:** + * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators + * Use this [MQTT load simulator](https://github.com/awslabs/aws-iot-mqtt-load-generator) to test device message load throughout your IoT solution. ### IoT Gotchas and Limitations -* **IoT protocols:** It is important to verify the exact type of support for your particular IoT device message protocol. For example, one commonly used IoT protocol is [MQTT](https://www.ibm.com/developerworks/community/blogs/5things/entry/5_things_to_know_about_mqtt_the_protocol_for_internet_of_things?lang=en). Within MQTT there are [three possible levels of QoS in MQTT](https://dzone.com/articles/internet-things-mqtt-quality). AWS IoT supports MQTT [QoS 0](http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) (fire and forget, or at most once) and QoS 1(at least once, or includes confirmation), but *not* QoS 2 (exactly once, requires 4-step confirmation). This is important in understanding how much code you’ll need to write for your particular application message resolution needs. Here is a [presentation about the nuances of connecting](http://www.slideshare.net/AmazonWebServices/overview-of-iot-infrastructure-and-connectivity-at-aws-getting-started-with-aws-iot). -* The ecosystems to match **IAM users or roles** to **IoT policies** and their associated authorized AWS IoT devices is immature. Custom coding to enforce your security requirements is common. -* A common mistake is to misunderstand the importance of IoT **device** **security**. It is imperative to associate *each* device with a unique certificate (public key). You can generate your own certificates and upload them to AWS, or you can use AWS generated IoT device certificates. It’s best to read and understand AWS’s own guidance on this [topic](http://www.slideshare.net/AmazonWebServices/best-practices-of-iot-in-the-cloud). -* There is only one **AWS IoT Gateway** (endpoint) per AWS account. For production scenarios, you’ll probably need to set up multiple AWS accounts in order to separate device traffic for development, test and production. It’s interesting to note that the [Azure IoT Gateway](https://azure.microsoft.com/en-us/documentation/articles/iot-hub-protocol-gateway/) supports configuration of multiple endpoints, so that a single Azure account can be used with separate pub/sub endpoints for development, testing and production -* **Limits:** Be aware of [limits](http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html), including device message size, type, frequency, and number of AWS IoT rules. +- **IoT protocols:** It is important to verify the exact type of support for your particular IoT device message protocol. For example, one commonly used IoT protocol is [MQTT](https://www.ibm.com/developerworks/community/blogs/5things/entry/5_things_to_know_about_mqtt_the_protocol_for_internet_of_things?lang=en). Within MQTT there are [three possible levels of QoS in MQTT](https://dzone.com/articles/internet-things-mqtt-quality). AWS IoT supports MQTT [QoS 0](http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) (fire and forget, or at most once) and QoS 1(at least once, or includes confirmation), but *not* QoS 2 (exactly once, requires 4-step confirmation). This is important in understanding how much code you’ll need to write for your particular application message resolution needs. Here is a [presentation about the nuances of connecting](http://www.slideshare.net/AmazonWebServices/overview-of-iot-infrastructure-and-connectivity-at-aws-getting-started-with-aws-iot). +- The ecosystems to match **IAM users or roles** to **IoT policies** and their associated authorized AWS IoT devices are immature. Custom coding to enforce your security requirements is common. +- A common mistake is to misunderstand the importance of IoT **device** **security**. It is imperative to associate *each* device with a unique certificate (public key). You can generate your own certificates and upload them to AWS, or you can use AWS generated IoT device certificates. It’s best to read and understand AWS’s own guidance on this [topic](http://www.slideshare.net/AmazonWebServices/best-practices-of-iot-in-the-cloud). +- There is only one **AWS IoT Gateway** (endpoint) per AWS account. For production scenarios, you’ll probably need to set up multiple AWS accounts in order to separate device traffic for development, test and production. It’s interesting to note that the [Azure IoT Gateway](https://azure.microsoft.com/en-us/documentation/articles/iot-hub-protocol-gateway/) supports configuration of multiple endpoints, so that a single Azure account can be used with separate pub/sub endpoints for development, testing and production +- **Limits:** Be aware of [limits](http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html), including device message size, type, frequency, and number of AWS IoT rules. ### IoT Code Samples -* [Simple Beer Service](https://github.com/awslabs/simplebeerservice) is a surprisingly useful code example using AWS IoT, Lambda, etc. -* [IoT-elf](https://github.com/awslabs/aws-iot-elf) offers clean Python sample using the AWS IoT SDK. -* [IoT Button projects](https://www.hackster.io/AmazonWebServices/products/aws-iot-button) on Hackster include many different code samples for projects. -* [5 IoT code examples](https://github.com/awslabs/aws-iot-examples/): a device simulator, MQTT sample, just in time registration, truck simulator, prediction data simulator. -* [AWS Alexa trivia voice example](https://developer.amazon.com/public/community/post/TxDJWS16KUPVKO/New-Alexa-Skills-Kit-Template:-Build-a-Trivia-Skill-in-under-an-Hour) is a quick-start using Alexa voice capability and Lambda. -* Some Raspberry Pi examples include the [Beacon project](https://github.com/araobp/beacon/blob/master/README.md), [Danbo](https://github.com/awslabs/aws-iot-demo-for-danbo), and [GoPiGo](https://github.com/awslabs/aws-iotbot). +- [Simple Beer Service](https://github.com/awslabs/simplebeerservice) is a surprisingly useful code example using AWS IoT, Lambda, etc. +- [IoT-elf](https://github.com/awslabs/aws-iot-elf) offers clean Python sample using the AWS IoT SDK. +- [IoT Button projects](https://www.hackster.io/AmazonWebServices/products/aws-iot-button) on Hackster include many different code samples for projects. +- [5 IoT code examples](https://github.com/awslabs/aws-iot-examples/): a device simulator, MQTT sample, just in time registration, truck simulator, prediction data simulator. +- [AWS Alexa trivia voice example](https://developer.amazon.com/public/community/post/TxDJWS16KUPVKO/New-Alexa-Skills-Kit-Template:-Build-a-Trivia-Skill-in-under-an-Hour) is a quick-start using Alexa voice capability and Lambda. +- Some Raspberry Pi examples include the [Beacon project](https://github.com/araobp/beacon/blob/master/README.md), [Danbo](https://github.com/awslabs/aws-iot-demo-for-danbo), and [GoPiGo](https://github.com/awslabs/aws-iotbot). Billing and Cost Management --------------------------- From 62f100e0db52fcab85c5e2f151b3f5aab13c5b63 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Kostka?= Date: Sat, 15 Oct 2016 22:37:00 +0200 Subject: [PATCH 076/140] Changes requested --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index f11df0a..46640e2 100644 --- a/README.md +++ b/README.md @@ -739,8 +739,7 @@ EC2 - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. -- ❗Once EC2 instance is launched, IAM role can not be assigned to it. - +- 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to an existing instance. AMIs ---- From d7bc2e530ff16fee4f1bd695018d5e1255cddb6c Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sat, 15 Oct 2016 13:38:03 -0700 Subject: [PATCH 077/140] added emoji in gotchas per editors requests --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 87001d7..7c95b82 100644 --- a/README.md +++ b/README.md @@ -1344,11 +1344,11 @@ IoT ### IoT Gotchas and Limitations -- **IoT protocols:** It is important to verify the exact type of support for your particular IoT device message protocol. For example, one commonly used IoT protocol is [MQTT](https://www.ibm.com/developerworks/community/blogs/5things/entry/5_things_to_know_about_mqtt_the_protocol_for_internet_of_things?lang=en). Within MQTT there are [three possible levels of QoS in MQTT](https://dzone.com/articles/internet-things-mqtt-quality). AWS IoT supports MQTT [QoS 0](http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) (fire and forget, or at most once) and QoS 1(at least once, or includes confirmation), but *not* QoS 2 (exactly once, requires 4-step confirmation). This is important in understanding how much code you’ll need to write for your particular application message resolution needs. Here is a [presentation about the nuances of connecting](http://www.slideshare.net/AmazonWebServices/overview-of-iot-infrastructure-and-connectivity-at-aws-getting-started-with-aws-iot). -- The ecosystems to match **IAM users or roles** to **IoT policies** and their associated authorized AWS IoT devices are immature. Custom coding to enforce your security requirements is common. -- A common mistake is to misunderstand the importance of IoT **device** **security**. It is imperative to associate *each* device with a unique certificate (public key). You can generate your own certificates and upload them to AWS, or you can use AWS generated IoT device certificates. It’s best to read and understand AWS’s own guidance on this [topic](http://www.slideshare.net/AmazonWebServices/best-practices-of-iot-in-the-cloud). -- There is only one **AWS IoT Gateway** (endpoint) per AWS account. For production scenarios, you’ll probably need to set up multiple AWS accounts in order to separate device traffic for development, test and production. It’s interesting to note that the [Azure IoT Gateway](https://azure.microsoft.com/en-us/documentation/articles/iot-hub-protocol-gateway/) supports configuration of multiple endpoints, so that a single Azure account can be used with separate pub/sub endpoints for development, testing and production -- **Limits:** Be aware of [limits](http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html), including device message size, type, frequency, and number of AWS IoT rules. +- 🔸**IoT protocols:** It is important to verify the exact type of support for your particular IoT device message protocol. For example, one commonly used IoT protocol is [MQTT](https://www.ibm.com/developerworks/community/blogs/5things/entry/5_things_to_know_about_mqtt_the_protocol_for_internet_of_things?lang=en). Within MQTT there are [three possible levels of QoS in MQTT](https://dzone.com/articles/internet-things-mqtt-quality). AWS IoT supports MQTT [QoS 0](http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) (fire and forget, or at most once) and QoS 1(at least once, or includes confirmation), but *not* QoS 2 (exactly once, requires 4-step confirmation). This is important in understanding how much code you’ll need to write for your particular application message resolution needs. Here is a [presentation about the nuances of connecting](http://www.slideshare.net/AmazonWebServices/overview-of-iot-infrastructure-and-connectivity-at-aws-getting-started-with-aws-iot). +- 🔸The ecosystems to match **IAM users or roles** to **IoT policies** and their associated authorized AWS IoT devices are immature. Custom coding to enforce your security requirements is common. +- ❗A common mistake is to misunderstand the importance of IoT **device** **security**. It is imperative to associate *each* device with a unique certificate (public key). You can generate your own certificates and upload them to AWS, or you can use AWS generated IoT device certificates. It’s best to read and understand AWS’s own guidance on this [topic](http://www.slideshare.net/AmazonWebServices/best-practices-of-iot-in-the-cloud). +- 🔸There is only one **AWS IoT Gateway** (endpoint) per AWS account. For production scenarios, you’ll probably need to set up multiple AWS accounts in order to separate device traffic for development, test and production. It’s interesting to note that the [Azure IoT Gateway](https://azure.microsoft.com/en-us/documentation/articles/iot-hub-protocol-gateway/) supports configuration of multiple endpoints, so that a single Azure account can be used with separate pub/sub endpoints for development, testing and production +- 🔸**Limits:** Be aware of [limits](http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html), including device message size, type, frequency, and number of AWS IoT rules. ### IoT Code Samples From f82732d762d18eb732e9d2d63ff38754f71a341d Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Sat, 15 Oct 2016 13:43:35 -0700 Subject: [PATCH 078/140] updated link to iot section in 'which services' --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7c95b82..d7ad075 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ General Information - 🐥[QuickSight](https://aws.amazon.com/quicksight/): Business intelligence service - [SES](https://aws.amazon.com/ses/): Send and receive e-mail for marketing or transactions - ⛓[API Gateway](https://aws.amazon.com/api-gateway/): Proxy, manage, and secure API calls - - ⛓[IoT](https://aws.amazon.com/iot/): Manage bidirectional communication over HTTP, WebSockets, and MQTT between AWS and clients (often but not necessarily “things” like appliances or sensors) + - ⛓[IoT](#iot): Manage bidirectional communication over HTTP, WebSockets, and MQTT between AWS and clients (often but not necessarily “things” like appliances or sensors) - ⛓[WAF](https://aws.amazon.com/waf/): Web firewall for CloudFront to deflect attacks - ⛓[KMS](#kms): Store and manage encryption keys securely - [Inspector](https://aws.amazon.com/inspector/): Security audit From c6fde4d9997a2240fd86a96b9103fa70fb03081e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Kostka?= Date: Sat, 15 Oct 2016 22:46:01 +0200 Subject: [PATCH 079/140] Add newline --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 46640e2..4023ec0 100644 --- a/README.md +++ b/README.md @@ -740,6 +740,7 @@ EC2 - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. - 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to an existing instance. + AMIs ---- From 74b73a4fc012995797d32232bf88e828eece58ce Mon Sep 17 00:00:00 2001 From: Krishan Date: Sat, 15 Oct 2016 23:48:03 +0100 Subject: [PATCH 080/140] updated phrasing of time-ordering of records & merged two bullet-points in the Gotchas, as per PR feedback. --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9ed074f..cbfd365 100644 --- a/README.md +++ b/README.md @@ -1283,7 +1283,7 @@ Kinesis Streams - 📒 [Homepage](https://aws.amazon.com/kinesis/streams/) ∙ [Developer guide](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) ∙ [FAQ](https://aws.amazon.com/kinesis/streams/faqs/) ∙ [Pricing](https://aws.amazon.com/kinesis/streams/pricing/) - **Kinesis Streams** (which used to be only called Kinesis, before Kinesis Firehose and Kinesis Analytics were launched) is a service that allows you to injest high-throughput data streams for immediate or delayed processing by other AWS services - Kinesis Streams' subcomponents are called [Shards](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). Each shard provides 1MB/s of write capacity and 2MB/s of read capacity at a maximum of 5 reads per second. A stream can have its Shards programatically increased or decreased based on a variety of metrics -- All records entered into a Kinesis Stream are kept in time-order, but obviously it is up to the developer to ensure they are entered into the Stream in the correct order. +- All records entered into a Kinesis Stream are assigned a unique Sequence Number as they are captured. The records in a Stream are ordered by this number, so any time-ordering is preserved. ### Kinesis Streams Alternatives and Lock-in @@ -1299,8 +1299,7 @@ Kinesis Streams ### Kinesis Streams Gotchas and Limitations - 💸❗**Kinesis Streams are not included in the free tier!** Make sure if you do any experimentation with it on a personal account, you shut down the stream, or you can run up unexpected costs (~$11 per shard-month) - Kinesis Streams' shards each only permit 5 reads per second. If you are using ```n``` shards in a particular stream, and evenly distributing your data across all of them, you will end up with a total of 5 reads per second. This is because a consumer cannot know which shard will contain new data, and will therefore need to check every single one. This means that there is a hard limit on the number of consumers you can have per stream, for any given latency. - - If you wish to have 5 consumers all reading data from one Stream with 5 shards, with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams, or reduce your latency requirements. - - Each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second - a value in excess of the maximum. + - If you wish to have 5 consumers all reading data from one Stream with 5 shards, with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams, or reduce your latency requirements - with the setup described above, each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second - a value in excess of the maximum. - There is a good blog by Brandur, an engineer at Stripe, that discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). High Availability From 7aae64113981c20ab52baeed988985d7cc0498ce Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Sat, 15 Oct 2016 16:37:37 -0700 Subject: [PATCH 081/140] Clean up new Kinesis Streams section - "ingest", "available", "initialize" spelling fixed - "available" spelling fixed - grammar and phrasing fixes - quotes to unicode, dashes to em dash --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 9dd29a9..fab370c 100644 --- a/README.md +++ b/README.md @@ -1297,26 +1297,26 @@ Kinesis Streams ### Kinesis Streams Basics - 📒 [Homepage](https://aws.amazon.com/kinesis/streams/) ∙ [Developer guide](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) ∙ [FAQ](https://aws.amazon.com/kinesis/streams/faqs/) ∙ [Pricing](https://aws.amazon.com/kinesis/streams/pricing/) -- **Kinesis Streams** (which used to be only called Kinesis, before Kinesis Firehose and Kinesis Analytics were launched) is a service that allows you to injest high-throughput data streams for immediate or delayed processing by other AWS services -- Kinesis Streams' subcomponents are called [Shards](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). Each shard provides 1MB/s of write capacity and 2MB/s of read capacity at a maximum of 5 reads per second. A stream can have its Shards programatically increased or decreased based on a variety of metrics +- **Kinesis Streams** (which used to be only called Kinesis, before Kinesis Firehose and Kinesis Analytics were launched) is a service that allows you to ingest high-throughput data streams for immediate or delayed processing by other AWS services. +- Kinesis Streams’ subcomponents are called [Shards](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). Each shard provides 1MB/s of write capacity and 2MB/s of read capacity at a maximum of 5 reads per second. A stream can have its Shards programatically increased or decreased based on a variety of metrics. - All records entered into a Kinesis Stream are assigned a unique Sequence Number as they are captured. The records in a Stream are ordered by this number, so any time-ordering is preserved. ### Kinesis Streams Alternatives and Lock-in -- ⛓Kinesis is most closely compared to [Apache Kafka](https://kafka.apache.org/), an open-source data injestion solution. It is possible to set up a Kafka cluster hosted on [EC2 instances](#ec2) (or any other VPS), however you are responsible for managing and maintaining both Zookeeper and the Kafka brokers in a highly availible configuration. Confluent have a good blog post on their recommendations on doing this [here](http://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/), which has links on the bottom to several other blogs they have written on the subject. +- ⛓Kinesis is most closely compared to [Apache Kafka](https://kafka.apache.org/), an open-source data ingestion solution. It is possible to set up a Kafka cluster hosted on [EC2 instances](#ec2) (or any other VPS), however you are responsible for managing and maintaining both Zookeeper and the Kafka brokers in a highly available configuration. Confluent has a good blog post with their recommendations on how to do this [here](http://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/), which has links on the bottom to several other blogs they have written on the subject. - Kinesis uses very AWS-specific APIs and terms (e.g. Shards), so you should be aware of the potential future costs of migrating away from it, should you choose to use it. ### Kinesis Streams Tips -- The [KCL](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html) (Kinesis Client Library) is a very useful Java program (wrapped in a Multi-Language Daemon that makes it useable in Java, Node, Python, Ruby and .NET programs) that provides very simple interfaces for clients to use when consuming data from a Kinesis Stream. It provides the skeleton for 3 basic functions - ```intitialise```, ```process-records```, and ```shutdown```. As a developer, all you need to do is set up the config file to point at the correct Kinesis Stream, and fill out the provided functions in order to start consuming data from Kinesis. - - The KCL uses a DynamoDB table to keep track of which records have been processed by the KCL. This ensures that all records are processed 'at least once'. It is up to the developer to ensure that the program can handle doubly-processed records. - - The KCL also uses DynamoDB to keep track of other KCL 'workers'. It automatically shares the available Kinesis Shards across all the workers as equally as possible. +- The [KCL](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html) (Kinesis Client Library) is a very useful Java program (wrapped in a multi-language interface that makes it useable in Java, Node, Python, Ruby and .NET programs) that provides very simple interfaces for clients to use when consuming data from a Kinesis Stream. It provides the skeleton for 3 basic functions - ```initialize```, ```process-records```, and ```shutdown```. As a developer, all you need to do is set up the config file to point at the correct Kinesis Stream and fill out the provided functions in order to start consuming data from Kinesis. + - The KCL uses a DynamoDB table to keep track of which records have been processed by the KCL. This ensures that all records are processed ‘at least once’. It is up to the developer to ensure that the program can handle doubly-processed records. + - The KCL also uses DynamoDB to keep track of other KCL ‘workers’. It automatically shares the available Kinesis Shards across all the workers as equally as possible. ### Kinesis Streams Gotchas and Limitations -- 💸❗**Kinesis Streams are not included in the free tier!** Make sure if you do any experimentation with it on a personal account, you shut down the stream, or you can run up unexpected costs (~$11 per shard-month) -- Kinesis Streams' shards each only permit 5 reads per second. If you are using ```n``` shards in a particular stream, and evenly distributing your data across all of them, you will end up with a total of 5 reads per second. This is because a consumer cannot know which shard will contain new data, and will therefore need to check every single one. This means that there is a hard limit on the number of consumers you can have per stream, for any given latency. - - If you wish to have 5 consumers all reading data from one Stream with 5 shards, with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams, or reduce your latency requirements - with the setup described above, each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second - a value in excess of the maximum. - - There is a good blog by Brandur, an engineer at Stripe, that discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). +- 💸❗**Kinesis Streams are not included in the free tier!** Make sure if you do any experimentation with it on a personal account, you shut down the stream or may run up unexpected costs (~$11 per shard-month.) +- Kinesis Streams’ shards each only permit 5 reads per second. If you are using ```n``` shards in a particular stream and are evenly distributing your data across all of them, you will end up with a total of 5 reads per second. This is because a consumer cannot know which shard will contain new data and will therefore need to check every single one. This means that there is a hard limit on the number of consumers you can have per stream for any given latency. + - If you wish to have 5 consumers all reading data from one Stream with 5 shards with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams or reduce your latency requirements — with the setup described above, each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second — a value in excess of the maximum. + - There is a good blog post by Brandur, an engineer at Stripe, which discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). Device Farm ----------- From c0968d77eac6c70ef8bcfad6b968ef1f7bd3aa91 Mon Sep 17 00:00:00 2001 From: Mikhail Advani Date: Fri, 14 Oct 2016 19:51:37 +0530 Subject: [PATCH 082/140] [#201] Added details for SES --- README.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index fab370c..794f587 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ Table of Contents | [Kinesis Streams](#kinesis-streams) | [📗](#kinesis-streams-basics) | [📘](#kinesis-streams-tips) | [📙](#kinesis-streams-gotchas-and-limitations) | | [Device Farm](#device-farm) | [📗](#device-farm-basics) | | | | [IoT](#iot) | [📗](#iot-basics) | [📘](#iot-tips) | [📙](#iot-gotchas-and-limitations) | - +| [SES](#ses) | [📗](#ses-basics) | [📘](#ses-tips) | [📙](#ses-gotchas-and-limitations) | **Special Topics** @@ -1381,6 +1381,23 @@ IoT - [AWS Alexa trivia voice example](https://developer.amazon.com/public/community/post/TxDJWS16KUPVKO/New-Alexa-Skills-Kit-Template:-Build-a-Trivia-Skill-in-under-an-Hour) is a quick-start using Alexa voice capability and Lambda. - Some Raspberry Pi examples include the [Beacon project](https://github.com/araobp/beacon/blob/master/README.md), [Danbo](https://github.com/awslabs/aws-iot-demo-for-danbo), and [GoPiGo](https://github.com/awslabs/aws-iotbot). +SES +--- + +### SES Basics + +- 📒 [Homepage](https://aws.amazon.com/ses/) ∙ [Documentation](https://aws.amazon.com/documentation/ses/) ∙ [FAQ](https://aws.amazon.com/ses/faqs/) ∙ [Pricing](https://aws.amazon.com/ses/pricing/) +- **SES** (or Simple Email Service) is a service that exposes SMTP endpoints for your application to directly integrate with. + +### SES Tips + +- 🔹**Bounce Handling:** Ensure you handle this early enough. Your email sending can be shut down in case of repeated bounces. 📒 [Processing Bounces and Complaints](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html) +- 🔹**Credentials:** Many developers get confused between SES credentials and AWS API keys. Ensure you enter SMTP credentials while using the SMTP APIs 📒 [Using Credentials With Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) ∙ [Obtaining Your Amazon SES SMTP Credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html) + +### SES Gotchas and Limitations + +- 🔸**Internet Access:** The SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). Setup a SMTP relay instance in a subnet with Internet access for the same(📒 [Integrating Amazon SES with Your Existing Email Server](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm if your proxy service supports SMTP + High Availability ----------------- From b6b8f39f32f7a5a09778855870c1b88982e002ec Mon Sep 17 00:00:00 2001 From: Mikhail Advani Date: Sun, 16 Oct 2016 10:49:36 +0530 Subject: [PATCH 083/140] [#201] Grammatical fixes for SES --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 794f587..bfaf7ca 100644 --- a/README.md +++ b/README.md @@ -1391,12 +1391,12 @@ SES ### SES Tips -- 🔹**Bounce Handling:** Ensure you handle this early enough. Your email sending can be shut down in case of repeated bounces. 📒 [Processing Bounces and Complaints](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html) -- 🔹**Credentials:** Many developers get confused between SES credentials and AWS API keys. Ensure you enter SMTP credentials while using the SMTP APIs 📒 [Using Credentials With Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) ∙ [Obtaining Your Amazon SES SMTP Credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html) +- 🔹**Bounce Handling:** Ensure you handle this early enough. Your email sending can be shut down in case of repeated bounces. 📒 [Processing Bounces and Complaints](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html). +- 🔹**Credentials:** Many developers get confused between SES credentials and AWS API keys. Ensure you enter SMTP credentials while using the SMTP APIs 📒 [Using Credentials With Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) ∙ [Obtaining Your Amazon SES SMTP Credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html). ### SES Gotchas and Limitations -- 🔸**Internet Access:** The SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). Setup a SMTP relay instance in a subnet with Internet access for the same(📒 [Integrating Amazon SES with Your Existing Email Server](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm if your proxy service supports SMTP +- 🔸**Internet Access:** The SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). Set up a SMTP relay instance in a subnet with Internet access for the same (📒 [Integrating Amazon SES with Your Existing Email Server](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm that your proxy service supports SMTP. High Availability ----------------- From 8fad1454563221f713d9522f5a7bf77da2917547 Mon Sep 17 00:00:00 2001 From: Mikhail Advani Date: Mon, 17 Oct 2016 09:10:37 +0530 Subject: [PATCH 085/140] [#201] Implemented feedback --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bfaf7ca..32cc243 100644 --- a/README.md +++ b/README.md @@ -1391,12 +1391,12 @@ SES ### SES Tips -- 🔹**Bounce Handling:** Ensure you handle this early enough. Your email sending can be shut down in case of repeated bounces. 📒 [Processing Bounces and Complaints](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html). -- 🔹**Credentials:** Many developers get confused between SES credentials and AWS API keys. Ensure you enter SMTP credentials while using the SMTP APIs 📒 [Using Credentials With Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) ∙ [Obtaining Your Amazon SES SMTP Credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html). +- 🔹**Bounce Handling:** Make sure you handle this early enough. Your ability to send emails can be removed if SES sees too many bounces. 📒 [Processing Bounces and Complaints](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html). +- 🔹**Credentials:** Many developers get confused between SES credentials and AWS API keys. Make sure to enter SMTP credentials while using the SMTP APIs 📒 [Using Credentials With Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) ∙ [Obtaining Your Amazon SES SMTP Credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html). ### SES Gotchas and Limitations -- 🔸**Internet Access:** The SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). Set up a SMTP relay instance in a subnet with Internet access for the same (📒 [Integrating Amazon SES with Your Existing Email Server](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm that your proxy service supports SMTP. +- 🔸**Internet Access:** SES SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). In such a case, set up an SMTP relay instance in a subnet with Internet access and configure your application to send emails to this SMTP relay instance rather than SES. The relay should have a forwarding rule to send all emails to SES (📒 [Integrating Amazon SES with Your Existing Email Server](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm that your proxy service supports SMTP. High Availability ----------------- From fab4a36096fe9c3226bb5aa21506af7863e84985 Mon Sep 17 00:00:00 2001 From: Elliott Spira Date: Mon, 17 Oct 2016 16:08:14 +1100 Subject: [PATCH 086/140] adding to the EC2 section --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index fab370c..8c73ab1 100644 --- a/README.md +++ b/README.md @@ -707,6 +707,7 @@ EC2 - 📒 [Homepage](https://aws.amazon.com/ec2/) ∙ [Documentation](https://aws.amazon.com/documentation/ec2/) ∙ [FAQ](https://aws.amazon.com/ec2/faqs/) ∙ [Pricing](https://aws.amazon.com/ec2/pricing/) (see also [ec2instances.info](http://www.ec2instances.info/)\) - **EC2** (Elastic Compute Cloud) is AWS’ offering of the most fundamental piece of cloud computing: A [virtual private server](https://en.wikipedia.org/wiki/Virtual_private_server). These “instances” and can run [most Linux, BSD, and Windows operating systems](https://aws.amazon.com/ec2/faqs/#What_operating_system_environments_are_supported). Internally, they use [Xen](https://en.wikipedia.org/wiki/Xen) virtualization. - The term “EC2” is sometimes used to refer to the servers themselves, but technically refers more broadly to a whole collection of supporting services, too, like load balancing (CLBs/ALBs), IP addresses (EIPs), bootable images (AMIs), security groups, and network drives (EBS) (which we discuss individually in this guide). +- 💸[EC2 pricing](https://aws.amazon.com/ec2/pricing/) is a complicated topic. It can range from free (on the [AWS free tier](https://aws.amazon.com/free/)) to a lot, depending on your usage of course. Pricing is by instance type, by hour and changes depending on AWS region and whether you are purchasing your instances [on-demand](https://aws.amazon.com/ec2/pricing/on-demand/), on the [spot-market](https://aws.amazon.com/ec2/spot/) or pre-purchasing ([reserved instances](https://aws.amazon.com/ec2/pricing/reserved-instances/)). The website [http://www.ec2instances.info/](http://www.ec2instances.info/) is particularly good for comparing instance types. ### EC2 Alternatives and Lock-In @@ -722,6 +723,7 @@ EC2 - Selecting instance types is complex since there are so many types. Additionally, there are different generations, released [over the years](https://aws.amazon.com/blogs/aws/ec2-instance-history/). - 🔹Use the list at [**ec2instances.info**](http://www.ec2instances.info/) to review costs and features. [Amazon’s own list](https://aws.amazon.com/ec2/instance-types/) of instance types is hard to use, and doesn’t list features and price together, which makes it doubly difficult. - Prices vary a lot, so use [**ec2instances.info**](http://www.ec2instances.info/) to determine the set of machines that meet your needs and [**ec2price.com**](http://ec2price.com/) to find the cheapest type in the region you’re working in. Depending on the timing and region, it might be much cheaper to rent an instance with *more* memory or CPU than the bare minimum. + - Turn off your instances when they aren't in use. For multiple workloads/environments, you don't need your EC2 instances on 24/7. Given that costs are calculated based on hourly usage, this is a simple mechanism for cost savings. This can be achieved using [Lambda and CloudWatch](https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/), an open source solution like [Scalr](https://github.com/Scalr/scalr) or a SaaS provider like [GorillaStack](www.gorillastack.com). - [**Dedicated instances**](https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/) and [**dedicated hosts**](https://aws.amazon.com/ec2/dedicated-hosts/) are assigned hardware, instead of usual virtual instances. They are more expensive than virtual instances but [can be preferable](https://aws.amazon.com/ec2/dedicated-hosts/) for performance, compliance, or licensing reasons. - **32 bit vs 64 bit:** A few micro, small, and medium instances are still available to use as 32-bit architecture. You’ll be using 64-bit EC2 (“amd64”) instances nowadays, though smaller instances still support 32 bit (“i386”). Use 64 bit unless you have legacy constraints or other good reasons to use 32. - **HVM vs PV:** There are two kinds of virtualization technology used by EC2, [hardware virtual machine (HVM) and paravirtual (PV)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html). Historically, PV was the usual type, but [now HVM is becoming the standard](https://www.opswat.com/blog/aws-2015-why-you-need-switch-pv-hvm). If you want to use the newest instance types, you must use HVM. See the [instance type matrix](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/) for details. From 25df5050517c24998bc8b548b7782bd20d32ba27 Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Mon, 17 Oct 2016 09:00:27 +0300 Subject: [PATCH 087/140] Added link and undocumented icon. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e671531..af43952 100644 --- a/README.md +++ b/README.md @@ -1055,7 +1055,7 @@ Lambda - 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications. - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. -- ❗Currently (as of October, 2016) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future. +- ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 12f149dfdef9813e1b0ca361e7a27685514fe3ba Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Mon, 17 Oct 2016 14:00:44 +0300 Subject: [PATCH 088/140] Changes requested for pr --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 599203a..aa8ebc8 100644 --- a/README.md +++ b/README.md @@ -1252,9 +1252,8 @@ Redshift - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). -- 🔹 If you need to change sort key or dist key of a table, you need to create a new table with new key and move your data to new table with insert into new_table select * from old_table. [Choosing Sort Key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) -- ❗🚪 When moving data with insert into x select from y; you need to have at least twice of y table size free space on your harddisk. Redshift first copies data to disk then to new table. -- 🔹 Vacuum delete only does not block copy commands, but vacuum reindex can block if you execute copy command every minute or so. +- 🔹 [Choosing Sort Key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change sort key after table created. If you need to change sort key or distribution key of a table, you need to create a new table with new key and move your data into it with a query like "insert into new_table select * from old_table". +- ❗🚪 When moving data with a query that looks like “insert into x select from y;” you need to have as twice as much space as table “y” takes up available on your cluster's disk. Redshift first copies the data to disk and then to the new table. A good [article](https://www.periscopedata.com/blog/changing-dist-and-sort-keys-in-redshift.html) about how to this for big tables. EMR --- From 6e898212e658c4abc04cc2b866f0f6565ab8d47e Mon Sep 17 00:00:00 2001 From: Matthew Lapworth Date: Mon, 17 Oct 2016 10:56:23 -0700 Subject: [PATCH 089/140] Added note about the true functionality of Authenticated Users ACL related to S3. Signed-off-by: Matthew Lapworth --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index a906214..56ac898 100644 --- a/README.md +++ b/README.md @@ -683,6 +683,7 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 Regions now support read-after-write consistency. Amazon S3 also renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions. +- 🔸When configuring ACLs on who can access the bucket and contents, an predefined group exists called **Authenticated Users**. This group allows all authenticated users, across all AWS accounts, access to the resource(s). This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. ### Storage Durability, Availability, and Price From f117e2875b0fe60cc8cd0eb190cc4eef696ed83e Mon Sep 17 00:00:00 2001 From: Matthew Lapworth Date: Mon, 17 Oct 2016 10:57:36 -0700 Subject: [PATCH 090/140] Added link to S3 ACLs Signed-off-by: Matthew Lapworth --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 56ac898..8a92626 100644 --- a/README.md +++ b/README.md @@ -683,7 +683,7 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 Regions now support read-after-write consistency. Amazon S3 also renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions. -- 🔸When configuring ACLs on who can access the bucket and contents, an predefined group exists called **Authenticated Users**. This group allows all authenticated users, across all AWS accounts, access to the resource(s). This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- 🔸When configuring ACLs on who can access the bucket and contents, an predefined group exists called **Authenticated Users**, more information [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). This group allows all authenticated users, across all AWS accounts, access to the resource(s). This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. ### Storage Durability, Availability, and Price From 1c4b3e0776138476e22af975fa09f14960d44d15 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Mon, 17 Oct 2016 11:02:03 -0700 Subject: [PATCH 091/140] Update authors/roles. --- AUTHORS.md | 32 +++++++++++++++++++++++++------- admin/authors-info.yml | 1 + 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index fba7e71..ee5fb76 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -7,54 +7,72 @@ Please help if you can, and see the [contribution guidelines](CONTRIBUTING.md) f Alphabetically by username: +* [0xmohit](https://github.com/0xmohit) — [2+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) -* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [3+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) +* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [5+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) +* [Bo Bayles (bbayles)](https://github.com/bbayles) — [1+](https://github.com/open-guides/og-aws/commits?author=bbayles)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abbayles) * [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk) * [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe) — _expert (IoT)_ * [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer) * [Bradly Feeley (bradly)](https://github.com/bradly) — [2+](https://github.com/open-guides/og-aws/commits?author=bradly)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Abradly) * [Brady Dowling (bradydowling)](https://github.com/bradydowling) — [1+](https://github.com/open-guides/og-aws/commits?author=bradydowling)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abradydowling) +* [Mårten Gustafson (chids)](https://github.com/chids) — [3+](https://github.com/open-guides/og-aws/commits?author=chids)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Achids) * [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) +* [Chris Lennon (chrislennon)](https://github.com/chrislennon) — [1+](https://github.com/open-guides/og-aws/commits?author=chrislennon)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrislennon) * [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) -* [Dan Hermann (danhermann)](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) +* [Dan Hermann (danhermann)](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) +* [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) * [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ * [Dmitry Guyvoronsky (dreamiurg)](https://github.com/dreamiurg) — [1+](https://github.com/open-guides/og-aws/commits?author=dreamiurg)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adreamiurg) +* [Eric Hammond (ehammond)](https://github.com/ehammond) — _expert (Lambda, serverless)_ * [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy) +* [esell (esell)](https://github.com/esell) — [5+](https://github.com/open-guides/og-aws/commits?author=esell)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aesell) * [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ * [Glynn Forrest (glynnforrest)](https://github.com/glynnforrest) — [1+](https://github.com/open-guides/og-aws/commits?author=glynnforrest)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aglynnforrest) * [Dmitry Golyshev (golyshev)](https://github.com/golyshev) * [Gulam Shakir (gshakir)](https://github.com/gshakir) — [2+](https://github.com/open-guides/og-aws/commits?author=gshakir)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agshakir) * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) -* [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [87+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[80+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [92+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[81+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Jaanus Torp (jnsaff)](https://github.com/jnsaff) — [1+](https://github.com/open-guides/og-aws/commits?author=jnsaff)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajnsaff) * Jurgen Philippaerts * [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) — [0+](https://github.com/open-guides/og-aws/commits?author=kazuyukitanimura)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akazuyukitanimura) -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [3+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ +* [Kyle Busekist (kbusekist)](https://github.com/kbusekist) — [2+](https://github.com/open-guides/og-aws/commits?author=kbusekist)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akbusekist) +* [Krishan (KrishanBhasin)](https://github.com/KrishanBhasin) — [1+](https://github.com/open-guides/og-aws/commits?author=KrishanBhasin)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3AKrishanBhasin) +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [8+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [maiki](https://github.com/maiki) — [1+](https://github.com/open-guides/og-aws/commits?author=maiki)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amaiki) * [Manoj M J (manojmj92)](https://github.com/manojmj92) — [1+](https://github.com/open-guides/og-aws/commits?author=manojmj92)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amanojmj92) * [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d) * [Max Zanko (max-zanko)](https://github.com/max-zanko) — [10+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ +* [Mikhail Advani (mikhailadvani)](https://github.com/mikhailadvani) — [1+](https://github.com/open-guides/og-aws/commits?author=mikhailadvani)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Amikhailadvani) * [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke) * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ * [Ola Wiberg (olawiberg)](https://github.com/olawiberg) +* [Pascal Borreli (pborreli)](https://github.com/pborreli) — [1+](https://github.com/open-guides/og-aws/commits?author=pborreli)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apborreli) * [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) * Praveen Patnala +* [Richard Birkby (rbirkby)](https://github.com/rbirkby) — [2+](https://github.com/open-guides/og-aws/commits?author=rbirkby)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arbirkby) * [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ * [Russell Power (rjpower)](https://github.com/rjpower) +* [Ashley Davis (SgtCoDFish)](https://github.com/SgtCoDFish) — [1+](https://github.com/open-guides/og-aws/commits?author=SgtCoDFish)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3ASgtCoDFish) * [David Schott (shott85)](https://github.com/shott85) — [1+](https://github.com/open-guides/og-aws/commits?author=shott85)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashott85) * [Adam Nelson (spudstuff)](https://github.com/spudstuff) — [4+](https://github.com/open-guides/og-aws/commits?author=spudstuff)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aspudstuff) * [Steven Maude (StevenMaude)](https://github.com/StevenMaude) — [1+](https://github.com/open-guides/og-aws/commits?author=StevenMaude)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AStevenMaude) -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [15+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[15+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [17+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[20+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Carlos (theist)](https://github.com/theist) — [1+](https://github.com/open-guides/og-aws/commits?author=theist)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Atheist) +* [Sandeep Dinesh (thesandlord)](https://github.com/thesandlord) — [1+](https://github.com/open-guides/og-aws/commits?author=thesandlord)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Athesandlord) * [Tom Schlick (tomschlick)](https://github.com/tomschlick) — [3+](https://github.com/open-guides/og-aws/commits?author=tomschlick)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atomschlick) * [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite) +* [Uli Stroetz (ustroetz)](https://github.com/ustroetz) — [1+](https://github.com/open-guides/og-aws/commits?author=ustroetz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Austroetz) * [Stefan Zier (weirded)](https://github.com/weirded) * [Michael Ortali (xethorn)](https://github.com/xethorn) — [1+](https://github.com/open-guides/og-aws/commits?author=xethorn)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Axethorn) +* [Jifeng Zhang (zjfroot)](https://github.com/zjfroot) — [1+](https://github.com/open-guides/og-aws/commits?author=zjfroot)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Azjfroot) Please don't PR the AUTHORS.md or authors-info.yml files! It is auto-generated regularly by the project leads. Please let the project leads know of any errors or omissions on this list. -(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) +(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).) \ No newline at end of file diff --git a/admin/authors-info.yml b/admin/authors-info.yml index 9729a17..532f4e5 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -26,6 +26,7 @@ roles: donnemartin: expert (tools) merrells: expert (cloud infrastructure, when to use AWS) benkehoe: expert (IoT) + ehammond: expert (Lambda, serverless) marcello3d: golyshev: Praveen Patnala: From 7f43dd768761ef934ec1f169e418dabccc8d3be8 Mon Sep 17 00:00:00 2001 From: Greg Leeds Date: Mon, 17 Oct 2016 15:14:28 -0500 Subject: [PATCH 092/140] Updating spot instance tip based on feedback from @jlevy --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 237fb74..f2d1df9 100644 --- a/README.md +++ b/README.md @@ -738,7 +738,7 @@ EC2 - Third generation P2 series offers NVIDIA K80 GPUs in 1, 8 and 16 GPU configurations targeting machine learning and scientific workloads. - Second generation G2 series offers NVIDIA K520 GPUs in 1 or 4 GPU configurations targeting graphics and video encoding. - First generation CG1 instances are still available in some regions in a single configuration with a NVIDIA M2050 GPU. - - 🔹Typical GPU workloads are often a good fit for [Spot Instances](#ec2-cost-management). + - 🔹As with any expensive EC2 instance types, [Spot instances can offer significant savings](#ec2-cost-management) with GPU workloads when interruptions are tolerable. ### EC2 Gotchas and Limitations From 6b48cfa1b40cd467d35d61a7190f385fa0b6e66c Mon Sep 17 00:00:00 2001 From: Justin Mullis Date: Mon, 17 Oct 2016 21:43:24 -0400 Subject: [PATCH 093/140] add gotcha for RDS subnet groups (#143) * add gotcha for RDS subnet groups * update based on style review * fix single quote * duplicated item in VPC gotchas --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a906214..9387835 100644 --- a/README.md +++ b/README.md @@ -1167,7 +1167,8 @@ VPCs, Network Security, and Security Groups - 🔸Security groups are not shared across data centers, so if you have infrastructure in multiple data centers, you should make sure your configuration/deployment tools take that into account. - ❗Be careful when choosing your VPC IP CIDR block: If you are going to need to make use of [ClassicLink](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html), make sure that your private IP range [doesn’t overlap](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html#classiclink-limitations) with that of EC2 Classic. -- ❗If you are going to peer VPCs, carefully consider the cost of [data transfer between VPCs](https://aws.amazon.com/vpc/faqs/#Peering_Connections), since for some workloads and integrations, this can be prohibitively expensive. +- ❗If you are going to peer VPCs, carefully consider the cost of of [data transfer between VPCs](https://aws.amazon.com/vpc/faqs/#Peering_Connections), since for some workloads and integrations, this can be prohibitively expensive. +- ❗New RDS instances require a [subnet group](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Subnets) within your VPC. If you’re using the [default VPC](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html) this isn’t a concern, it will contain a subnet for each availability zone in your region. However, if you’re creating your own VPC and plan on using RDS, make sure you have at least two subnets within the VPC to act as the subnet group. KMS --- From 3aca21519e2b8f205e31404ee2244006f4e10223 Mon Sep 17 00:00:00 2001 From: Krishan Date: Tue, 18 Oct 2016 02:49:43 +0100 Subject: [PATCH 094/140] Add some more depth to CloudFormation section. (#213) * add some more depth to Cloudformation * made changes as per feedback, added bullet point on the importance of version-controlling the templates * added more links & improvements to phrasing based on feedback --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 9387835..43f3d2d 100644 --- a/README.md +++ b/README.md @@ -1118,7 +1118,7 @@ CloudFormation ### CloudFormation Basics - 📒 [Homepage](https://aws.amazon.com/cloudformation/) ∙ [Developer guide](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) ∙ [FAQ](https://aws.amazon.com/cloudformation/faqs/) ∙ [Pricing](https://aws.amazon.com/cloudformation/pricing/) at no additional charge -- **CloudFormation** offers mechanisms to create and manage entire configurations of many types of AWS resources, using a JSON-based templating language. +- **CloudFormation** offers mechanisms to create and manage entire configurations of many types of AWS resources, using a JSON- or YAML-based templating language. - 💸CloudFormation itself has [no additional charge](https://aws.amazon.com/cloudformation/pricing/) itself; you pay for the underlying resources. ### CloudFormation Alternatives and Lock-In @@ -1129,6 +1129,10 @@ CloudFormation - [Troposphere](https://github.com/cloudtools/troposphere) is a Python library that makes it much easier to create CloudFormation templates. - 🔹Until [2016](https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/), CloudFormation used only an awkward JSON format that makes both reading and debugging difficult. To use it effectively typically involved building additional tooling, including converting it to YAML, but now [this is supported directly](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html). +- CloudFormation can be set up to [send SNS notifications](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html) upon state changes, enabling programatic handling of situations where stacks fail to build, or simple email alerts so the appropriate people are informed. +- CloudFormation allows the use of [**conditionals**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html) when creating a stack. + - One common way to leverage this capability is in support of multi-environment CloudFormation templates – by configuring them to use ‘if-else’ statements on the value of a [parameter passed in](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html) (e.g. “env”), environment-specific values for things like VPC IDs, SecurityGroup IDs, and AMI names can be passed into reusable generic templates. +- **Version control your CloudFormation templates!** In the Cloud, an application is the combination of the code written and the infrastructure it runs on. By version controlling **both**, it is easy to roll back to known good states. ### CloudFormation Gotchas and Limitations From 2a2876c39cf14ac03b65463abdc83c7976208cb6 Mon Sep 17 00:00:00 2001 From: Jason Barry Date: Mon, 17 Oct 2016 22:12:46 -0400 Subject: [PATCH 095/140] Add a couple of annoying VPC gotchas (#205) * Add a couple annoying VPC gotchas * add supporting documentation links * change sentence structure for link * remove duplicate gotcha --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 43f3d2d..9d6cd21 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -![An Open Guide](figures/signpost-horiz1-1600.jpg) +![An Open Guide](figures/signpost-horiz1-1600.jpg) The Open Guide to Amazon Web Services ===================================== @@ -1171,8 +1171,10 @@ VPCs, Network Security, and Security Groups - 🔸Security groups are not shared across data centers, so if you have infrastructure in multiple data centers, you should make sure your configuration/deployment tools take that into account. - ❗Be careful when choosing your VPC IP CIDR block: If you are going to need to make use of [ClassicLink](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html), make sure that your private IP range [doesn’t overlap](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-classiclink.html#classiclink-limitations) with that of EC2 Classic. -- ❗If you are going to peer VPCs, carefully consider the cost of of [data transfer between VPCs](https://aws.amazon.com/vpc/faqs/#Peering_Connections), since for some workloads and integrations, this can be prohibitively expensive. +- ❗If you are going to peer VPCs, carefully consider the cost of [data transfer between VPCs](https://aws.amazon.com/vpc/faqs/#Peering_Connections), since for some workloads and integrations, this can be prohibitively expensive. - ❗New RDS instances require a [subnet group](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Subnets) within your VPC. If you’re using the [default VPC](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html) this isn’t a concern, it will contain a subnet for each availability zone in your region. However, if you’re creating your own VPC and plan on using RDS, make sure you have at least two subnets within the VPC to act as the subnet group. +- ❗If you delete the default VPC, the only way to create another VPC marked as “default” is to contact AWS technical support. See this [note](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html#deleting-default-vpc) in the documentation. +- ❗Be careful with VPC VPN credentials! If lost or compromised, the VPN endpoint must be deleted and recreated. See the instructions for [Replacing Compromised Credentials](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html#CompromisedCredentials). KMS --- From 005049f2cc1c1b1d805a16325885aabd660ea586 Mon Sep 17 00:00:00 2001 From: TheRandomSecurityGuy Date: Mon, 17 Oct 2016 21:50:04 -0500 Subject: [PATCH 096/140] Added Scout2 for security auditing (#196) * Added Scout2 for security auditing * Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 9d6cd21..e11e7ea 100644 --- a/README.md +++ b/README.md @@ -572,6 +572,7 @@ We cover security basics first, since configuring user accounts is something you - [**AWS WAF**](https://aws.amazon.com/waf) is a web application firewall to help you protect your applications from common attack patterns. - **Security auditing:** - [Security Monkey](https://github.com/Netflix/security_monkey) is an open source tool that is designed to assist with security audits. + - [Scout2](https://github.com/nccgroup/Scout2) is an open source tool that uses AWS APIs to assess an environment's security posture. Scout2 is stable and actively maintained. - 🔹**Export and audit security settings:** You can audit security policies simply by exporting settings using AWS APIs, e.g. using a Boto script like [SecConfig.py](https://gist.github.com/jlevy/cce1b44fc24f94599d0a4b3e613cc15d) (from [this 2013 talk](http://www.slideshare.net/AmazonWebServices/intrusion-detection-in-the-cloud-sec402-aws-reinvent-2013)) and then reviewing and monitoring changes manually or automatically. ### Security and IAM Gotchas and Limitations From b88ae5e36da0f7a1d194d61694b31b26002c5927 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Mon, 17 Oct 2016 20:07:28 -0700 Subject: [PATCH 097/140] Added Google Data Studio 360 AWS QuickSight competitor --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e11e7ea..88dbb25 100644 --- a/README.md +++ b/README.md @@ -282,7 +282,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Memory cache | ElastiCache | App Engine Memcache | | Redis Cache | | Memcached, Redis | | Search | CloudSearch, Elasticsearch (managed) | | | Search | Algolia, QBox | Elasticsearch, Solr | | Data warehouse | Redshift | BigQuery | Dremel | SQL Data Warehouse | Oracle, IBM, SAP, HP, many others | Greenplum | -| Business intelligence | QuickSight | | | Power BI | Tableau | | +| Business intelligence | QuickSight | Data Studio 360 | | Power BI | Tableau | | | Lock manager | [DynamoDB (weak)](https://gist.github.com/ryandotsmith/c95fd21fab91b0823328) | | Chubby | Lease blobs in Storage Account | | ZooKeeper, Etcd, Consul | | Message broker | SQS, SNS, IoT | Pub/Sub | PubSub2 | Service Bus | | RabbitMQ, Kafka, 0MQ | | Streaming, distributed log | Kinesis | Dataflow | PubSub2 | Event Hubs | | Kafka Streams, Apex, Flink, Spark Streaming, Storm | From 5df287abeabe1f784e0cbaf6a6df8d6df50ad669 Mon Sep 17 00:00:00 2001 From: Mohit Agarwal Date: Tue, 18 Oct 2016 00:09:11 +0530 Subject: [PATCH 098/140] added a couple of points about EFS Additionally, fixed the incorrect link to "short list" of APIs and SDKs. --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a906214..c737eb8 100644 --- a/README.md +++ b/README.md @@ -453,7 +453,7 @@ So if you’re not going to manage your AWS configurations manually, what should ### APIs and SDKs -- **SDKs** for using AWS APIs are available in most major languages, with [Go](https://github.com/aws/aws-sdk-go), [iOS](https://github.com/aws/aws-sdk-ios), [Java](https://github.com/aws/aws-sdk-java), [JavaScript](https://github.com/aws/aws-sdk-js), [Python](https://github.com/boto/boto3), [Ruby](https://github.com/aws/aws-sdk-ruby), and [PHP](https://github.com/aws/aws-sdk-php) being most heavily used. AWS maintains [a short list](http://docs.aws.amazon.com/AmazonCloudFront/latest/APIReference/AWSLibraries.html), but the [awesome-aws list](https://github.com/donnemartin/awesome-aws#sdks-and-samples) is the most comprehensive and current. Note [support for C++](https://github.com/donnemartin/awesome-aws#c-sdk) is [still new](https://aws.amazon.com/blogs/aws/introducing-the-aws-sdk-for-c/). +- **SDKs** for using AWS APIs are available in most major languages, with [Go](https://github.com/aws/aws-sdk-go), [iOS](https://github.com/aws/aws-sdk-ios), [Java](https://github.com/aws/aws-sdk-java), [JavaScript](https://github.com/aws/aws-sdk-js), [Python](https://github.com/boto/boto3), [Ruby](https://github.com/aws/aws-sdk-ruby), and [PHP](https://github.com/aws/aws-sdk-php) being most heavily used. AWS maintains [a short list](https://aws.amazon.com/tools/#sdk), but the [awesome-aws list](https://github.com/donnemartin/awesome-aws#sdks-and-samples) is the most comprehensive and current. Note [support for C++](https://github.com/donnemartin/awesome-aws#c-sdk) is [still new](https://aws.amazon.com/blogs/aws/introducing-the-aws-sdk-for-c/). - **Retry logic:** An important aspect to consider whenever using SDKs is error handling; under heavy use, a wide variety of failures, from programming errors to throttling to AWS-related outages or failures, can be expected to occur. SDKs typically implement [**exponential backoff**](https://docs.aws.amazon.com/general/latest/gr/api-retries.html) to address this, but this may need to be understood and adjusted over time for some applications. For example, it is often helpful to alert on some error codes and not on others. - ❗Don’t use APIs directly. Although AWS documentation includes lots of API details, it’s better to use the SDKs for your preferred language to access APIs. SDKs are more mature, robust, and well-maintained than something you’d write yourself. @@ -835,9 +835,11 @@ EFS ### EFS Basics - 🐥**EFS** is Amazon’s new (general release 2016) network filesystem. +- It is designed to be highly available and durable and each EFS file system object is redundantly stored across multiple availability zones. - It is similar to [EBS](#ebs) in that it is a network-attached drive, but it [differs in important ways](https://aws.amazon.com/efs/details/#When_to_Use_Amazon_EFS_vs._Amazon_EBS): - - EFS can be attached to many instances (up to thousands), while EBS can only be attached to one drive. It does this via [NFSv4](https://en.wikipedia.org/wiki/Network_File_System). - - EFS can offer higher throughput (multiple gigabytes per second) and better durability and availability than EBS (see [the comparison table](#storage-durability-availability-and-price)), but with higher latency. + - EFS file systems can automatically scale to petabytes of data without needing to provision storage. + - EFS can be attached to many instances (up to thousands), while EBS can only be attached to one drive. It does this via [NFSv4.1](https://en.wikipedia.org/wiki/Network_File_System). + - EFS can offer [higher throughput](http://docs.aws.amazon.com/efs/latest/ug/performance.html) (multiple gigabytes per second) and better durability and availability than EBS (see [the comparison table](#storage-durability-availability-and-price)), but with higher latency. - EFS cannot be used as a boot volume or in certain other ways EBS can. - EFS costs much more than EBS (up to [three times as much](#storage-durability-availability-and-price)). From cc1c153098a88fc06a2922a6b8faa023a0705c7c Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Tue, 18 Oct 2016 09:36:45 +0300 Subject: [PATCH 099/140] dev -> development --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index af43952..e6b77d2 100644 --- a/README.md +++ b/README.md @@ -1055,7 +1055,7 @@ Lambda - 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications. - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. -- ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda dev team. They are working to prevent this in the future. +- ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 7d6c2d5558cb695215e639a70673a8bf21b2e8fe Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Tue, 18 Oct 2016 10:02:46 +0300 Subject: [PATCH 100/140] Made changes requested for pr --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index aa8ebc8..dea09f8 100644 --- a/README.md +++ b/README.md @@ -1252,8 +1252,8 @@ Redshift - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). -- 🔹 [Choosing Sort Key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change sort key after table created. If you need to change sort key or distribution key of a table, you need to create a new table with new key and move your data into it with a query like "insert into new_table select * from old_table". -- ❗🚪 When moving data with a query that looks like “insert into x select from y;” you need to have as twice as much space as table “y” takes up available on your cluster's disk. Redshift first copies the data to disk and then to the new table. A good [article](https://www.periscopedata.com/blog/changing-dist-and-sort-keys-in-redshift.html) about how to this for big tables. +- 🔹 [Choosing a sort key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change a table‘s sort key after it is created. If you need to change the sort or distribution key of a table, you need to create a new table with the new key and move your data into it with a query like “insert into new_table select * from old_table“. +- ❗🚪 When moving data with a query that looks like “insert into x select from y“ you need to have as twice as much space as table “y“ takes up available on your cluster‘s disk. Redshift first copies the data to disk and then to the new table. [Here](https://www.periscopedata.com/blog/changing-dist-and-sort-keys-in-redshift.html) is a good article on how to this for big tables. EMR --- From 7bed47fe82245bbbda2c3c9853d9d50983b519f9 Mon Sep 17 00:00:00 2001 From: Mikhail Advani Date: Tue, 18 Oct 2016 14:53:25 +0530 Subject: [PATCH 101/140] [#201]: Styling updates --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 32cc243..744f1ac 100644 --- a/README.md +++ b/README.md @@ -1391,12 +1391,12 @@ SES ### SES Tips -- 🔹**Bounce Handling:** Make sure you handle this early enough. Your ability to send emails can be removed if SES sees too many bounces. 📒 [Processing Bounces and Complaints](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html). -- 🔹**Credentials:** Many developers get confused between SES credentials and AWS API keys. Make sure to enter SMTP credentials while using the SMTP APIs 📒 [Using Credentials With Amazon SES](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) ∙ [Obtaining Your Amazon SES SMTP Credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html). +- 🔹**Bounce Handling:** Make sure you handle this early enough. Your ability to send emails can be removed if SES sees [too many bounces](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/best-practices-bounces-complaints.html). +- 🔹**Credentials:** Many developers get confused between [SES credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-credentials.html) and AWS API keys. Make sure to enter [SMTP credentials](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html) while using the SMTP APIs. ### SES Gotchas and Limitations -- 🔸**Internet Access:** SES SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). In such a case, set up an SMTP relay instance in a subnet with Internet access and configure your application to send emails to this SMTP relay instance rather than SES. The relay should have a forwarding rule to send all emails to SES (📒 [Integrating Amazon SES with Your Existing Email Server](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm that your proxy service supports SMTP. +- 🔸**Internet Access:** SES SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). In such a case, set up an SMTP relay instance in a subnet with Internet access and configure your application to send emails to this SMTP relay instance rather than SES. The relay should have a [forwarding rule to send all emails to SES](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm that your proxy service supports SMTP. High Availability ----------------- From 9f2c41c4655f0605fec46db30e65b1a880dd434e Mon Sep 17 00:00:00 2001 From: Shubham Aggarwal Date: Tue, 18 Oct 2016 16:34:36 +0530 Subject: [PATCH 102/140] Updated README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eeb79f1..c3364aa 100644 --- a/README.md +++ b/README.md @@ -706,7 +706,7 @@ EC2 ### EC2 Basics - 📒 [Homepage](https://aws.amazon.com/ec2/) ∙ [Documentation](https://aws.amazon.com/documentation/ec2/) ∙ [FAQ](https://aws.amazon.com/ec2/faqs/) ∙ [Pricing](https://aws.amazon.com/ec2/pricing/) (see also [ec2instances.info](http://www.ec2instances.info/)\) -- **EC2** (Elastic Compute Cloud) is AWS’ offering of the most fundamental piece of cloud computing: A [virtual private server](https://en.wikipedia.org/wiki/Virtual_private_server). These “instances” and can run [most Linux, BSD, and Windows operating systems](https://aws.amazon.com/ec2/faqs/#What_operating_system_environments_are_supported). Internally, they use [Xen](https://en.wikipedia.org/wiki/Xen) virtualization. +- **EC2** (Elastic Compute Cloud) is AWS’ offering of the most fundamental piece of cloud computing: A [virtual private server](https://en.wikipedia.org/wiki/Virtual_private_server). These “instances” can run [most Linux, BSD, and Windows operating systems](https://aws.amazon.com/ec2/faqs/#What_operating_system_environments_are_supported). Internally, they use [Xen](https://en.wikipedia.org/wiki/Xen) virtualization. - The term “EC2” is sometimes used to refer to the servers themselves, but technically refers more broadly to a whole collection of supporting services, too, like load balancing (CLBs/ALBs), IP addresses (EIPs), bootable images (AMIs), security groups, and network drives (EBS) (which we discuss individually in this guide). ### EC2 Alternatives and Lock-In From 5fa5399adf4cd3c25b5e78fc3d08a8f00b7c2775 Mon Sep 17 00:00:00 2001 From: nickbarclay Date: Tue, 18 Oct 2016 09:06:48 -0400 Subject: [PATCH 103/140] Added Redshift VACUUM / ANALYZE recommendations --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index eeb79f1..b4c4f8a 100644 --- a/README.md +++ b/README.md @@ -1268,6 +1268,7 @@ Redshift - [Top 10 Performance Tuning Techniques for Amazon Redshift](https://blogs.aws.amazon.com/bigdata/post/Tx31034QG0G3ED1/Top-10-Performance-Tuning-Techniques-for-Amazon-Redshift) provides an excellent list of performance tuning techniques. - [Amazon Redshift Utils](https://github.com/awslabs/amazon-redshift-utils) contains useful utilities, scripts and views to simplify Redshift ops. - [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/t_Reclaiming_storage_space202.html) regularly following a significant number of deletes or updates to reclaim space and improve query performance. +- Avoid performing blanket [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/r_VACUUM_command.html) or [ANALYZE](http://docs.aws.amazon.com/redshift/latest/dg/r_ANALYZE.html) operations at a cluster level. The checks on each table to determine whether VACUUM or ANALYZE action needs to be taken is wasteful. Only perform ANALYZE and VACUUM commands on the objects that require it. Utilize the [Analyze & Vacuum Schema Utility](https://github.com/awslabs/amazon-redshift-utils/tree/master/src/AnalyzeVacuumUtility) to perform this work. The SQL to determine whether a table needs to be VACUUMed or ANALYZed can be found in the [Schema Utility README](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AnalyzeVacuumUtility/README.md) if you wish to create your own maintenance process. - Redshift provides various [column compression](http://docs.aws.amazon.com/redshift/latest/dg/t_Compressing_data_on_disk.html) options to optimize the stored data size. AWS strongly encourages users to use [automatic compression](http://docs.aws.amazon.com/redshift/latest/dg/c_Loading_tables_auto_compress.html) at the COPY stage, when Redshift uses a sample of the data being ingested to analyze the column compression options. However, automatic compression can only be applied to an empty table with no data. Therefore, make sure the initial load batch is big enough to provide Redshift with a representative sample of the data (the default sample size is 100000 rows). ### Redshift Gotchas and Limitations From 53edf7f88c911c8eb5160f009963db3d3ba1332b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A5rten=20Gustafson?= Date: Tue, 18 Oct 2016 17:09:41 +0200 Subject: [PATCH 104/140] Expand the serverless section of the service matrix: (#200) - Providers: Auth0 Webtask and PubNub Blocks --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eeb79f1..43dbe64 100644 --- a/README.md +++ b/README.md @@ -271,7 +271,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or |-------------------------------|------------------------------------------------------------------------------|------------------------------|-----------------|------------------------------------|-----------------------------------|------------------------------------------------------------| | Virtual server | EC2 | Compute Engine (GCE) | | Virtual Machine | DigitalOcean | OpenStack | | PaaS | Elastic Beanstalk | App Engine | App Engine | Web Apps | Heroku, AppFog, OpenShift | Meteor, AppScale, Cloud Foundry, Convox | -| Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | | | +| Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | PubNub Blocks, Auth0 Webtask | | | Container, cluster manager | ECS | Container Engine, Kubernetes | Borg or Omega | Container Service | | Kubernetes, Mesos, Aurora | | File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | | Block storage | EBS | Persistent Disk | | Storage Account | | NFS | From c9dccb259f2f4555b222a02da91b13752aaa15b8 Mon Sep 17 00:00:00 2001 From: Abdul Mohammed Date: Tue, 18 Oct 2016 10:10:49 -0500 Subject: [PATCH 105/140] Added brief summary about Troposphere (#220) * Added brief summary about Troposphere Added brief summary about Troposphere * updated summary about Troposphere updated summary about Troposphere * Updated Troposphere Section Updated Troposphere Section * Troposphere update #3 Troposphere update #3 --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 43dbe64..176bd5e 100644 --- a/README.md +++ b/README.md @@ -1136,6 +1136,10 @@ CloudFormation ### CloudFormation Tips - [Troposphere](https://github.com/cloudtools/troposphere) is a Python library that makes it much easier to create CloudFormation templates. + - Currently supports [AWS](https://github.com/cloudtools/troposphere#currently-supported-aws-resource-types) and [OpenStack](https://github.com/cloudtools/troposphere#currently-supported-openstack-resource-types) resource types. + - Troposphere does not support all of the resources types you can describe with CloudFormation templates. + - Built in [error](https://github.com/cloudtools/troposphere#examples-of-the-error-checking-full-tracebacks-removed-for-clarity) checking. + - A recommended soft dependency is [awacs](https://github.com/cloudtools/awacs), which allows you to generate AWS access policy in JSON by writing Python code. - 🔹Until [2016](https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/), CloudFormation used only an awkward JSON format that makes both reading and debugging difficult. To use it effectively typically involved building additional tooling, including converting it to YAML, but now [this is supported directly](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html). - CloudFormation can be set up to [send SNS notifications](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html) upon state changes, enabling programatic handling of situations where stacks fail to build, or simple email alerts so the appropriate people are informed. - CloudFormation allows the use of [**conditionals**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html) when creating a stack. From cbe0820c3ea1682d355cb35b07ec7fa45a12310a Mon Sep 17 00:00:00 2001 From: Matthew Lapworth Date: Tue, 18 Oct 2016 17:52:46 -0700 Subject: [PATCH 106/140] Incorporated feedback. Made issue caused by 'gotcha' more explicit Signed-off-by: Matthew Lapworth --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8a92626..1ed34b2 100644 --- a/README.md +++ b/README.md @@ -683,7 +683,7 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 Regions now support read-after-write consistency. Amazon S3 also renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions. -- 🔸When configuring ACLs on who can access the bucket and contents, an predefined group exists called **Authenticated Users**, more information [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). This group allows all authenticated users, across all AWS accounts, access to the resource(s). This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- 🔸When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow access to all authenticated users, across all AWS accounts, to the resource(s) stored in S3. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. ### Storage Durability, Availability, and Price From 50813fd70fdd1965c14aa4003b651024c8cc0f00 Mon Sep 17 00:00:00 2001 From: Matthew Lapworth Date: Tue, 18 Oct 2016 17:55:31 -0700 Subject: [PATCH 107/140] minor re-wording Signed-off-by: Matthew Lapworth --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1ed34b2..c796f12 100644 --- a/README.md +++ b/README.md @@ -683,7 +683,7 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 Regions now support read-after-write consistency. Amazon S3 also renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions. -- 🔸When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow access to all authenticated users, across all AWS accounts, to the resource(s) stored in S3. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- 🔸When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to all authenticated users, across all AWS accounts. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. ### Storage Durability, Availability, and Price From 51a5671f8a2ee903681cf0790b520360669e7489 Mon Sep 17 00:00:00 2001 From: Halil Duygulu Date: Wed, 19 Oct 2016 11:40:10 +0300 Subject: [PATCH 108/140] Changes for " and ' --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index dea09f8..7dd67df 100644 --- a/README.md +++ b/README.md @@ -1252,8 +1252,8 @@ Redshift - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). -- 🔹 [Choosing a sort key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change a table‘s sort key after it is created. If you need to change the sort or distribution key of a table, you need to create a new table with the new key and move your data into it with a query like “insert into new_table select * from old_table“. -- ❗🚪 When moving data with a query that looks like “insert into x select from y“ you need to have as twice as much space as table “y“ takes up available on your cluster‘s disk. Redshift first copies the data to disk and then to the new table. [Here](https://www.periscopedata.com/blog/changing-dist-and-sort-keys-in-redshift.html) is a good article on how to this for big tables. +- 🔹 [Choosing a sort key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change a table’s sort key after it is created. If you need to change the sort or distribution key of a table, you need to create a new table with the new key and move your data into it with a query like “insert into new_table select * from old_table”. +- ❗🚪 When moving data with a query that looks like “insert into x select from y”, you need to have twice as much disk space available as table “y” takes up on the cluster’s disks. Redshift first copies the data to disk and then to the new table. [Here](https://www.periscopedata.com/blog/changing-dist-and-sort-keys-in-redshift.html) is a good article on how to this for big tables. EMR --- From c22a49878adcb03909a6c30832b48be75221745d Mon Sep 17 00:00:00 2001 From: Martijn Dwars Date: Wed, 19 Oct 2016 11:00:19 +0200 Subject: [PATCH 109/140] Improve icons --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2bc26a5..a68db9d 100644 --- a/README.md +++ b/README.md @@ -726,7 +726,7 @@ EC2 - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. -- 🔸Instances come in two types: Fixed Performance Instances (e.g. M3, C3, and R3) and [Burstable Performance Instances](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. +- ❗⏱ Instances come in two types: Fixed Performance Instances (e.g. M3, C3, and R3) and [Burstable Performance Instances](https://aws.amazon.com/ec2/instance-types/#burst) (e.g. T2). A T2 instance receives CPU credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU credits when they are idle, and use CPU credits when they are active. However, once an instance runs out of credits, you'll notice a severe degradation in performance. If you need consistently high CPU performance for applications such as video encoding, high volume websites or HPC applications, it is recommended to use Fixed Performance Instances. AMIs ---- From 2a91d5c8f6db82dee7829385143d63e35c5e30bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Kostka?= Date: Wed, 19 Oct 2016 20:28:23 +0200 Subject: [PATCH 110/140] s/an existing/a running/g --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4023ec0..bea29c7 100644 --- a/README.md +++ b/README.md @@ -739,7 +739,7 @@ EC2 - 🔸Periodically you may find that your server or load balancer is receiving traffic for (presumably) a previous EC2 server that was running at the same IP address that you are handed out now (this may not matter, or it can be fixed by migrating to another new instance). - ❗If the EC2 API itself is a critical dependency of your infrastructure (e.g. for automated server replacement, custom scaling algorithms, etc.) and you are running at a large scale or making many EC2 API calls, make sure that you understand when they might fail (calls to it are [rate limited](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate) and the limits are not published and subject to change) and code and test against that possibility. - ❗Many newer EC2 instance types are EBS-only. Make sure to factor in EBS performance and costs when planning to use them. -- 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to an existing instance. +- 🔸An IAM role can be assigned to an EC2 instance [only at launch time](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). You cannot assign to a running instance. AMIs ---- From cc64e31a1023d01d4ddd4f17a1a7df257f3dfce7 Mon Sep 17 00:00:00 2001 From: Elliott Spira Date: Thu, 20 Oct 2016 09:40:26 +1100 Subject: [PATCH 111/140] incorporating feedback --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8c73ab1..295a908 100644 --- a/README.md +++ b/README.md @@ -707,7 +707,7 @@ EC2 - 📒 [Homepage](https://aws.amazon.com/ec2/) ∙ [Documentation](https://aws.amazon.com/documentation/ec2/) ∙ [FAQ](https://aws.amazon.com/ec2/faqs/) ∙ [Pricing](https://aws.amazon.com/ec2/pricing/) (see also [ec2instances.info](http://www.ec2instances.info/)\) - **EC2** (Elastic Compute Cloud) is AWS’ offering of the most fundamental piece of cloud computing: A [virtual private server](https://en.wikipedia.org/wiki/Virtual_private_server). These “instances” and can run [most Linux, BSD, and Windows operating systems](https://aws.amazon.com/ec2/faqs/#What_operating_system_environments_are_supported). Internally, they use [Xen](https://en.wikipedia.org/wiki/Xen) virtualization. - The term “EC2” is sometimes used to refer to the servers themselves, but technically refers more broadly to a whole collection of supporting services, too, like load balancing (CLBs/ALBs), IP addresses (EIPs), bootable images (AMIs), security groups, and network drives (EBS) (which we discuss individually in this guide). -- 💸[EC2 pricing](https://aws.amazon.com/ec2/pricing/) is a complicated topic. It can range from free (on the [AWS free tier](https://aws.amazon.com/free/)) to a lot, depending on your usage of course. Pricing is by instance type, by hour and changes depending on AWS region and whether you are purchasing your instances [on-demand](https://aws.amazon.com/ec2/pricing/on-demand/), on the [spot-market](https://aws.amazon.com/ec2/spot/) or pre-purchasing ([reserved instances](https://aws.amazon.com/ec2/pricing/reserved-instances/)). The website [http://www.ec2instances.info/](http://www.ec2instances.info/) is particularly good for comparing instance types. +- 💸**[EC2 pricing](https://aws.amazon.com/ec2/pricing/)** and **[cost management](#ec2-cost-management)** is a complicated topic. It can range from free (on the [AWS free tier](https://aws.amazon.com/free/)) to a lot, depending on your usage. Pricing is by instance type, by hour and changes depending on AWS region and whether you are purchasing your instances [On-Demand](https://aws.amazon.com/ec2/pricing/on-demand/), on the [Spot market](https://aws.amazon.com/ec2/spot/) or pre-purchasing ([Reserved Instances](https://aws.amazon.com/ec2/pricing/reserved-instances/)). ### EC2 Alternatives and Lock-In @@ -723,7 +723,7 @@ EC2 - Selecting instance types is complex since there are so many types. Additionally, there are different generations, released [over the years](https://aws.amazon.com/blogs/aws/ec2-instance-history/). - 🔹Use the list at [**ec2instances.info**](http://www.ec2instances.info/) to review costs and features. [Amazon’s own list](https://aws.amazon.com/ec2/instance-types/) of instance types is hard to use, and doesn’t list features and price together, which makes it doubly difficult. - Prices vary a lot, so use [**ec2instances.info**](http://www.ec2instances.info/) to determine the set of machines that meet your needs and [**ec2price.com**](http://ec2price.com/) to find the cheapest type in the region you’re working in. Depending on the timing and region, it might be much cheaper to rent an instance with *more* memory or CPU than the bare minimum. - - Turn off your instances when they aren't in use. For multiple workloads/environments, you don't need your EC2 instances on 24/7. Given that costs are calculated based on hourly usage, this is a simple mechanism for cost savings. This can be achieved using [Lambda and CloudWatch](https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/), an open source solution like [Scalr](https://github.com/Scalr/scalr) or a SaaS provider like [GorillaStack](www.gorillastack.com). + - **Turn off** your instances when they aren't in use. For many situations such as testing or staging resources, you may not need your instances on 24/7, and you won't need to pay EC2 hourly costs when they are suspended. Given that costs are calculated based on hourly usage, this is a simple mechanism for cost savings. This can be achieved using [Lambda and CloudWatch](https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/), an open source solution like [Scalr](https://github.com/Scalr/scalr) or a SaaS provider like [GorillaStack](https://www.gorillastack.com). - [**Dedicated instances**](https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/) and [**dedicated hosts**](https://aws.amazon.com/ec2/dedicated-hosts/) are assigned hardware, instead of usual virtual instances. They are more expensive than virtual instances but [can be preferable](https://aws.amazon.com/ec2/dedicated-hosts/) for performance, compliance, or licensing reasons. - **32 bit vs 64 bit:** A few micro, small, and medium instances are still available to use as 32-bit architecture. You’ll be using 64-bit EC2 (“amd64”) instances nowadays, though smaller instances still support 32 bit (“i386”). Use 64 bit unless you have legacy constraints or other good reasons to use 32. - **HVM vs PV:** There are two kinds of virtualization technology used by EC2, [hardware virtual machine (HVM) and paravirtual (PV)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html). Historically, PV was the usual type, but [now HVM is becoming the standard](https://www.opswat.com/blog/aws-2015-why-you-need-switch-pv-hvm). If you want to use the newest instance types, you must use HVM. See the [instance type matrix](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/) for details. From 6d9cb13feb6ba1cc65d2ab4ef1f964ca6f00ae23 Mon Sep 17 00:00:00 2001 From: Matthew Lapworth Date: Wed, 19 Oct 2016 19:51:04 -0700 Subject: [PATCH 112/140] Updated gotcha to correct icon Added emphasis on impact of misconfiguration Style updates Signed-off-by: Matthew Lapworth --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c796f12..249f80c 100644 --- a/README.md +++ b/README.md @@ -683,7 +683,7 @@ S3 - 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours. - 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)). - 🔸**US Standard region:** Previously, the us-east-1 region (also known as the US Standard region) was replicated across coasts, which led to greater variability of latency. Effective Jun 19, 2015 this is [no longer the case](https://forums.aws.amazon.com/ann.jspa?annID=3112). All Amazon S3 Regions now support read-after-write consistency. Amazon S3 also renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions. -- 🔸When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to all authenticated users, across all AWS accounts. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. +- ❗When configuring ACLs on who can access the bucket and contents, a predefined group exists called **[Authenticated Users](http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)**. This group is often used, incorrectly, to restrict S3 resource access to authenticated users of the owning account. If granted, the AuthenticatedUsers group will allow S3 resource access to **all authenticated users, across all AWS accounts**. A typical use case of this ACL is used in conjuction with the [requester pays](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) functionality of S3. ### Storage Durability, Availability, and Price From ca051e7ecd3ef91c54bd5a5ac2b4b600653390d7 Mon Sep 17 00:00:00 2001 From: Elliott Spira Date: Thu, 20 Oct 2016 13:51:33 +1100 Subject: [PATCH 113/140] adding pricing and gotchas to the Lambda section (#224) * adding pricing and gotchas to the Lambda section * implementing feedback --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6d8bfd3..b127a80 100644 --- a/README.md +++ b/README.md @@ -1075,6 +1075,7 @@ Lambda - 🔸Lambda is a new technology. As of mid 2016, only a few companies are using it for large-scale production applications. - 🔸Managing lots of Lambda functions is a workflow challenge, and tooling to manage Lambda deployments is still immature. - 🔸AWS’ official workflow around managing function [versioning and aliases](https://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html) is painful. +- 🔸Currently, [Lambda does not natively provide for environment variables](https://forums.aws.amazon.com/thread.jspa?messageID=686261), which are a [common way](https://12factor.net/config) of storing configuration, so your must rely on additional mechanisms to achieve this (such as the [Serverless framework](https://serverless.com/framework/docs/guide/environment-variable-handling/#environment-variables-in-serverless) or [apex](http://apex.run/#environment-variables)). - ❗📜 Currently [as of October, 2016](https://github.com/open-guides/og-aws/pull/199/files/c99bddb4ee2437587f1e188d47be2bb1da01f81d#r83529126) Lambda functions can sometimes stop working for 2-3 minutes for failure recovery purposes according to a support ticket answer from Lambda development team. They are working to prevent this in the future. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 357e3fe1391e756e008148bf5cf86f8111a890a3 Mon Sep 17 00:00:00 2001 From: Elliott Spira Date: Thu, 20 Oct 2016 13:59:41 +1100 Subject: [PATCH 114/140] changing apostrophe type and adding warning around ephemeral storage --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 295a908..b79edbb 100644 --- a/README.md +++ b/README.md @@ -723,7 +723,7 @@ EC2 - Selecting instance types is complex since there are so many types. Additionally, there are different generations, released [over the years](https://aws.amazon.com/blogs/aws/ec2-instance-history/). - 🔹Use the list at [**ec2instances.info**](http://www.ec2instances.info/) to review costs and features. [Amazon’s own list](https://aws.amazon.com/ec2/instance-types/) of instance types is hard to use, and doesn’t list features and price together, which makes it doubly difficult. - Prices vary a lot, so use [**ec2instances.info**](http://www.ec2instances.info/) to determine the set of machines that meet your needs and [**ec2price.com**](http://ec2price.com/) to find the cheapest type in the region you’re working in. Depending on the timing and region, it might be much cheaper to rent an instance with *more* memory or CPU than the bare minimum. - - **Turn off** your instances when they aren't in use. For many situations such as testing or staging resources, you may not need your instances on 24/7, and you won't need to pay EC2 hourly costs when they are suspended. Given that costs are calculated based on hourly usage, this is a simple mechanism for cost savings. This can be achieved using [Lambda and CloudWatch](https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/), an open source solution like [Scalr](https://github.com/Scalr/scalr) or a SaaS provider like [GorillaStack](https://www.gorillastack.com). + - **Turn off** your instances when they aren’t in use. For many situations such as testing or staging resources, you may not need your instances on 24/7, and you won’t need to pay EC2 hourly costs when they are suspended. Given that costs are calculated based on hourly usage, this is a simple mechanism for cost savings. This can be achieved using [Lambda and CloudWatch](https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/), an open source solution like [Scalr](https://github.com/Scalr/scalr) or a SaaS provider like [GorillaStack](https://www.gorillastack.com). (Note: if you turn off instances with an ephemeral root volume, any state will be lost when the instance is turned off. Therefore, for stateful applications it is safer to turn off EBS backed instances). - [**Dedicated instances**](https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/) and [**dedicated hosts**](https://aws.amazon.com/ec2/dedicated-hosts/) are assigned hardware, instead of usual virtual instances. They are more expensive than virtual instances but [can be preferable](https://aws.amazon.com/ec2/dedicated-hosts/) for performance, compliance, or licensing reasons. - **32 bit vs 64 bit:** A few micro, small, and medium instances are still available to use as 32-bit architecture. You’ll be using 64-bit EC2 (“amd64”) instances nowadays, though smaller instances still support 32 bit (“i386”). Use 64 bit unless you have legacy constraints or other good reasons to use 32. - **HVM vs PV:** There are two kinds of virtualization technology used by EC2, [hardware virtual machine (HVM) and paravirtual (PV)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html). Historically, PV was the usual type, but [now HVM is becoming the standard](https://www.opswat.com/blog/aws-2015-why-you-need-switch-pv-hvm). If you want to use the newest instance types, you must use HVM. See the [instance type matrix](https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/) for details. From 6ccac27e47fdaf0b022fc1822161609cd5e5d959 Mon Sep 17 00:00:00 2001 From: Krishan Date: Thu, 20 Oct 2016 04:13:46 +0100 Subject: [PATCH 115/140] Kinesis Streams cleanup & fixes (#229) * made changes to Kinesis Streams sections, as per feedback. * update KCL phrasing for clarity and add appropriate icons --- README.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 32a209c..cb032ef 100644 --- a/README.md +++ b/README.md @@ -1323,25 +1323,29 @@ Kinesis Streams - 📒 [Homepage](https://aws.amazon.com/kinesis/streams/) ∙ [Developer guide](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) ∙ [FAQ](https://aws.amazon.com/kinesis/streams/faqs/) ∙ [Pricing](https://aws.amazon.com/kinesis/streams/pricing/) - **Kinesis Streams** (which used to be only called Kinesis, before Kinesis Firehose and Kinesis Analytics were launched) is a service that allows you to ingest high-throughput data streams for immediate or delayed processing by other AWS services. -- Kinesis Streams’ subcomponents are called [Shards](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). Each shard provides 1MB/s of write capacity and 2MB/s of read capacity at a maximum of 5 reads per second. A stream can have its Shards programatically increased or decreased based on a variety of metrics. -- All records entered into a Kinesis Stream are assigned a unique Sequence Number as they are captured. The records in a Stream are ordered by this number, so any time-ordering is preserved. +- Kinesis Streams’ subcomponents are called [**shards**](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html). Each shard provides 1MB/s of write capacity and 2MB/s of read capacity at a maximum of 5 reads per second. A stream can have its shards programatically increased or decreased based on a variety of metrics. +- All records entered into a Kinesis Stream are assigned a unique sequence number as they are captured. The records in a Stream are ordered by this number, so any time-ordering is preserved. +- [This page](http://docs.aws.amazon.com/streams/latest/dev/key-concepts.html) summarises key terms and concepts for Kinesis Streams. ### Kinesis Streams Alternatives and Lock-in -- ⛓Kinesis is most closely compared to [Apache Kafka](https://kafka.apache.org/), an open-source data ingestion solution. It is possible to set up a Kafka cluster hosted on [EC2 instances](#ec2) (or any other VPS), however you are responsible for managing and maintaining both Zookeeper and the Kafka brokers in a highly available configuration. Confluent has a good blog post with their recommendations on how to do this [here](http://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/), which has links on the bottom to several other blogs they have written on the subject. -- Kinesis uses very AWS-specific APIs and terms (e.g. Shards), so you should be aware of the potential future costs of migrating away from it, should you choose to use it. +- 🚪 Kinesis is most closely compared to [Apache Kafka](https://kafka.apache.org/), an open-source data ingestion solution. It is possible to set up a Kafka cluster hosted on [EC2 instances](#ec2) (or any other VPS), however you are responsible for managing and maintaining both Zookeeper and the Kafka brokers in a highly available configuration. Confluent has a good blog post with their recommendations on how to do this [here](http://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/), which has links on the bottom to several other blogs they have written on the subject. +- ⛓ Kinesis uses very AWS-specific APIs, so you should be aware of the potential future costs of migrating away from it, should you choose to use it. +- An application that efficiently uses Kinesis Streams will scale the number of shards up and down based on the required streaming capacity (note: there is no direct equivalent to this with Apache Kafka.) + ### Kinesis Streams Tips -- The [KCL](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html) (Kinesis Client Library) is a very useful Java program (wrapped in a multi-language interface that makes it useable in Java, Node, Python, Ruby and .NET programs) that provides very simple interfaces for clients to use when consuming data from a Kinesis Stream. It provides the skeleton for 3 basic functions - ```initialize```, ```process-records```, and ```shutdown```. As a developer, all you need to do is set up the config file to point at the correct Kinesis Stream and fill out the provided functions in order to start consuming data from Kinesis. - - The KCL uses a DynamoDB table to keep track of which records have been processed by the KCL. This ensures that all records are processed ‘at least once’. It is up to the developer to ensure that the program can handle doubly-processed records. - - The KCL also uses DynamoDB to keep track of other KCL ‘workers’. It automatically shares the available Kinesis Shards across all the workers as equally as possible. +- The [KCL](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html) (Kinesis Client Library) provides a skeleton interface for Java, Node, Python, Ruby and .NET programs to easily consume data from a Kinesis Stream. In order to start consuming data from a Stream, you only need to provide a config file to point at the correct Kinesis Stream, and functions for initialising the consumer, processing the records, and shutting down the consumer within the skeletons provided. + - The KCL uses a DynamoDB table to keep track of which records have been processed by the KCL. This ensures that all records are processed “at least once”. It is up to the developer to ensure that the program can handle doubly-processed records. + - The KCL also uses DynamoDB to keep track of other KCL “workers”. It automatically shares the available Kinesis Shards across all the workers as equally as possible. ### Kinesis Streams Gotchas and Limitations -- 💸❗**Kinesis Streams are not included in the free tier!** Make sure if you do any experimentation with it on a personal account, you shut down the stream or may run up unexpected costs (~$11 per shard-month.) -- Kinesis Streams’ shards each only permit 5 reads per second. If you are using ```n``` shards in a particular stream and are evenly distributing your data across all of them, you will end up with a total of 5 reads per second. This is because a consumer cannot know which shard will contain new data and will therefore need to check every single one. This means that there is a hard limit on the number of consumers you can have per stream for any given latency. - - If you wish to have 5 consumers all reading data from one Stream with 5 shards with a maximum permitted latency of 0.5 seconds, you will need to either split your data across two streams or reduce your latency requirements — with the setup described above, each consumer will need to poll each shard once every 0.5 seconds, meaning each Shard will need to be queried 10 times a second — a value in excess of the maximum. - - There is a good blog post by Brandur, an engineer at Stripe, which discusses the performance and limitations of Kinesis in production [here](https://brandur.org/kinesis-in-production). +- 🔸⏱ Kinesis Streams’ shards each only permit [5 reads per second](http://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html). If you are evenly distrubuting data across many shards, your read limit for the Stream will remain at 5 reads per second on aggregate, as each consuming application will need to check every single shard for new records. This puts a hard limit on the number of different consuming applications possible per Stream for a given maximum read latency. + - For example, if you have 5 consuming applications reading data from one Stream with any number of shards, they cannot read with a latency of less than one second, as each of the 5 consumers will need to poll *each shard* every second, reaching the cap of 5 reads per second per shard. + - [This blog post](https://brandur.org/kinesis-in-production) further discusses the performance and limitations of Kinesis in production. +- 💸 **Kinesis Streams are not included in the free tier.** Make sure if you do any experimentation with it on a personal account, you shut down the stream or it may run up unexpected costs (~$11 per shard-month.) + Device Farm ----------- From 7e0b15221333f281308cab19e2f5aee6b63903ed Mon Sep 17 00:00:00 2001 From: Jose Luis Ordiales Coscia Date: Thu, 20 Oct 2016 05:19:24 +0200 Subject: [PATCH 116/140] Added RDS failover tips and gotchas (#141) * Added RDS failover tips and gotchas * Some minor rewording * Use unicode apostrophes --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cb032ef..a000bce 100644 --- a/README.md +++ b/README.md @@ -983,6 +983,7 @@ RDS ### RDS Tips - If you’re looking for the managed convenience of RDS for MongoDB, this isn’t offered by AWS directly, but you may wish to consider a provider such as [**mLab**](https://mlab.com/). +- RDS offers out of the box support for [high availability and failover](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) for your databases. - MySQL RDS allows access to [binary logs](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html#USER_LogAccess.MySQL.BinaryFormat). - 🔸**MySQL vs MariaDB vs Aurora:** If you prefer a MySQL-style database but are starting something new, you probably should consider Aurora and MariaDB as well. **Aurora** has increased availability and is the next-generation solution. That said, Aurora [may not be](http://blog.takipi.com/benchmarking-aurora-vs-mysql-is-amazons-new-db-really-5x-faster/) as fast relative to MySQL as is sometimes reported, and is more complex to administer. **MariaDB**, the modern [community fork](https://en.wikipedia.org/wiki/MariaDB) of MySQL, [likely now has the edge over MySQL](http://cloudacademy.com/blog/mariadb-vs-mysql-aws-rds/) for many purposes and is supported by RDS. - 🔸**Aurora:** Aurora is based on MySQL 5.6. If you are planning to migrate to Aurora from an existing MySQL database, avoiding any MySQL features from 5.7 or later will ease the transition. The easiest migration path to Aurora is restoring a database snapshot from MySQL 5.6. The next easiest method is restoring a dump from a MySQL-compatible database such as MariaDB. If neither of those methods are options, Amazon offers a [fee-based data migration service](http://docs.aws.amazon.com/dms/latest/userguide/Welcome.html). @@ -991,9 +992,9 @@ RDS - ⏱RDS instances run on EBS volumes, and hence are constrained by the EBS performance. - 🔸Verify what database features you need, as not everything you might want is available on RDS. For example, if you are using Postgres, check the list of [supported features and extensions](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#SQLServer.Concepts.General.FeatureSupport). If the features you need aren’t supported by RDS, you’ll have to deploy your database yourself. +- If you use the failover support offered by RDS, keep in mind that it is based on DNS changes - make sure that your client reacts to these changes appropriately. This is particularly important for Java, given how its DNS resolver’s TTL is [configured by default](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html). - 🔸**DB migration to RDS:** While importing your database into RDS ensure you take into consideration the maintenance window settings. If a backup is running at the same time, your import can take a considerable longer time than you would have expected. - DynamoDB -------- From 50efd72b1baded773753b5133de8ea745d2691ee Mon Sep 17 00:00:00 2001 From: gscalise Date: Thu, 20 Oct 2016 04:30:54 +0100 Subject: [PATCH 117/140] Extended CloudFormation section (#152) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Extended CloudFormation section * Remove extra tab * Removed space after marker * Correct 2 nits: use of ' vs ‘ and ie. vs i.e. --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 2c095aa..0ce96d1 100644 --- a/README.md +++ b/README.md @@ -1139,6 +1139,7 @@ CloudFormation ### CloudFormation Alternatives and Lock-In - Hashicorp’s [Terraform](https://www.terraform.io/intro/vs/cloudformation.html) is a third-party alternative. + - 🔸Some AWS features may not be available in Terraform (i.e. MultiAZ ElastiCache using Redis), and you may have to resort to embedded **CloudFormation** templates. ### CloudFormation Tips @@ -1147,6 +1148,8 @@ CloudFormation - Troposphere does not support all of the resources types you can describe with CloudFormation templates. - Built in [error](https://github.com/cloudtools/troposphere#examples-of-the-error-checking-full-tracebacks-removed-for-clarity) checking. - A recommended soft dependency is [awacs](https://github.com/cloudtools/awacs), which allows you to generate AWS access policy in JSON by writing Python code. +- If you are building different stacks with similar layers, it may be useful to build separate templates for each layer that you can reuse using [AWS::CloudFormation::Stack](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-stack.html). +- 🔸Avoid hardcoding resource parameters that can potentially change. Use stack parameters as much as you can, and resort to default parameter values. - 🔹Until [2016](https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/), CloudFormation used only an awkward JSON format that makes both reading and debugging difficult. To use it effectively typically involved building additional tooling, including converting it to YAML, but now [this is supported directly](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html). - CloudFormation can be set up to [send SNS notifications](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html) upon state changes, enabling programatic handling of situations where stacks fail to build, or simple email alerts so the appropriate people are informed. - CloudFormation allows the use of [**conditionals**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html) when creating a stack. @@ -1155,6 +1158,7 @@ CloudFormation ### CloudFormation Gotchas and Limitations +- ❗Modifications to stack resources made outside CloudFormation can potentially lead to stacks stuck in UPDATE\_ROLLBACK\_FAILED mode. Stacks in this state can‘t be recovered without help from AWS Support. - 🔸CloudFormation is useful but complex and with a variety of pain points. Many companies find alternate solutions, and many companies use it, but only with significant additional tooling. - 🔸CloudFormation can be very slow, especially for items like CloudFront distributions. - 🔸It’s hard to assemble good CloudFormation configurations from existing state. AWS does [offer a trick to do this](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html), but it’s very clumsy. From 886196647c831e030819c0315e0c946f45f95729 Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Wed, 19 Oct 2016 20:33:36 -0700 Subject: [PATCH 118/140] CloudFormation nits --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0ce96d1..2d84c9c 100644 --- a/README.md +++ b/README.md @@ -1139,7 +1139,7 @@ CloudFormation ### CloudFormation Alternatives and Lock-In - Hashicorp’s [Terraform](https://www.terraform.io/intro/vs/cloudformation.html) is a third-party alternative. - - 🔸Some AWS features may not be available in Terraform (i.e. MultiAZ ElastiCache using Redis), and you may have to resort to embedded **CloudFormation** templates. + - 🔸Some AWS features may not be available in Terraform (e.g. multi-AZ ElastiCache using Redis), and you may have to resort to embedded CloudFormation templates. ### CloudFormation Tips From c40f0ee608f7fe970d4a70b844574bc2c902faa7 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Wed, 19 Oct 2016 22:11:30 -0700 Subject: [PATCH 119/140] Added info in Security basics (#236) * Added info in Security basics added detail about IAM objects added detail and link to IAM policy simulator tool * update IAM section per requests --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2d84c9c..bb07661 100644 --- a/README.md +++ b/README.md @@ -528,7 +528,8 @@ We cover security basics first, since configuring user accounts is something you - 📒 IAM [Homepage](https://aws.amazon.com/iam/) ∙ [User guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) ∙ [FAQ](https://aws.amazon.com/iam/faqs/) - The [AWS Security Blog](https://blogs.aws.amazon.com/security) is one of the best sources of news and information on AWS security. - **IAM** is the service you use to manage accounts and permissioning for AWS. -- Managing security and access control with AWS is critical, so every AWS administrator needs to use and understand IAM, at least at a basic level. +- Managing security and access control with AWS is critical, so every AWS administrator needs to use and understand IAM, at least at a basic level. +- [IAM identities](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html) include users (people or services that are using AWS), groups (containers for sets of users and their permissions), and roles (containers for permissions assigned to AWS service instances). [Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions.html) for these identities are governed by [policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) You can use AWS pre-defined policies or custom policies that you create. - IAM manages various kinds of authentication, for both users and for software services that may need to authenticate with AWS, including: - [**Passwords**](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords.html) to log into the console. These are a username and password for real users. - [**Access keys**](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html), which you may use with command-line tools. These are two strings, one the “id”, which is an upper-case alphabetic string of the form 'AXXXXXXXXXXXXXXXXXXX', and the other is the secret, which is a 40-character mixed-case base64-style string. These are often set up for services, not just users. @@ -539,6 +540,7 @@ We cover security basics first, since configuring user accounts is something you - At the beginning, IAM policy may be very simple, but for large systems, it will grow in complexity, and need to be managed with care. - 🔹Make sure one person (perhaps with a backup) in your organization is formally assigned ownership of managing IAM policies, make sure every administrator works with that person to have changes reviewed. This goes a long way to avoiding accidental and serious misconfigurations. - It is best to give each user or service the minimum privileges needed to perform their duties. This is the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege), one of the foundations of good security. Organize all IAM users and groups according to levels of access they need. +- You can test policy permissions via the AWS IAM [policy simulator tool tool](https://policysim.aws.amazon.com/home/index.jsp). This is particularly useful if you write custom policies. ### Security and IAM Tips From 5e07c98cebed98c0800f54d26235a34e38d2c034 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Wed, 19 Oct 2016 22:15:08 -0700 Subject: [PATCH 120/140] added link to awslabs/git-secrets in IAM Gotchas (#235) * added link to awslabs/git-secrets in IAM Gotchas last section of this area * updated line 586 per request --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index bb07661..d5b1c57 100644 --- a/README.md +++ b/README.md @@ -585,6 +585,7 @@ We cover security basics first, since configuring user accounts is something you - But be careful not to cache credentials for too long, as [they expire](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials). (Note the other [dynamic metadata](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#dynamic-data-categories) also changes over time and should not be cached a long time, either.) - 🔸Some IAM operations are slower than other API calls (many seconds), since AWS needs to propagate these globally across regions. - ❗The uptime of IAM’s API has historically been lower than that of the instance metadata API. Be wary of incorporating a dependency on IAM’s API into critical paths or subsystems — for example, if you validate a user’s IAM group membership when they log into an instance and aren’t careful about precaching group membership or maintaining a back door, you might end up locking users out altogether when the API isn’t available. +- ❗**Don't check in AWS credentials or secrets to a git repository.** There are bots that scan GitHub looking for credentials. Use scripts or tools, such as [git-secrets](https://github.com/awslabs/git-secrets) to prevent anyone on your team from checking in sensitive information to your git repos. S3 -- From 690a735f3e04bb1fc92cbf0407328efc9c85f1bf Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Wed, 19 Oct 2016 22:20:21 -0700 Subject: [PATCH 121/140] IAM Gotcha Nits (#240) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d5b1c57..8b996f7 100644 --- a/README.md +++ b/README.md @@ -585,7 +585,7 @@ We cover security basics first, since configuring user accounts is something you - But be careful not to cache credentials for too long, as [they expire](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials). (Note the other [dynamic metadata](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#dynamic-data-categories) also changes over time and should not be cached a long time, either.) - 🔸Some IAM operations are slower than other API calls (many seconds), since AWS needs to propagate these globally across regions. - ❗The uptime of IAM’s API has historically been lower than that of the instance metadata API. Be wary of incorporating a dependency on IAM’s API into critical paths or subsystems — for example, if you validate a user’s IAM group membership when they log into an instance and aren’t careful about precaching group membership or maintaining a back door, you might end up locking users out altogether when the API isn’t available. -- ❗**Don't check in AWS credentials or secrets to a git repository.** There are bots that scan GitHub looking for credentials. Use scripts or tools, such as [git-secrets](https://github.com/awslabs/git-secrets) to prevent anyone on your team from checking in sensitive information to your git repos. +- ❗**Don't check in AWS credentials or secrets to a git repository.** There are bots that scan GitHub looking for credentials. Use scripts or tools, such as [git-secrets](https://github.com/awslabs/git-secrets) to prevent anyone on your team from checking in sensitive information to your git repositories. S3 -- From e63325909f2d695087e367e1c618f02b50f63663 Mon Sep 17 00:00:00 2001 From: Elliott Spira Date: Thu, 20 Oct 2016 17:35:18 +1100 Subject: [PATCH 122/140] adding to the cloudformation section (#223) * adding to the cloudformation section * removing quotes :grin: --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index bc6437c..40296e5 100644 --- a/README.md +++ b/README.md @@ -1136,13 +1136,13 @@ CloudFormation ### CloudFormation Basics - 📒 [Homepage](https://aws.amazon.com/cloudformation/) ∙ [Developer guide](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) ∙ [FAQ](https://aws.amazon.com/cloudformation/faqs/) ∙ [Pricing](https://aws.amazon.com/cloudformation/pricing/) at no additional charge -- **CloudFormation** offers mechanisms to create and manage entire configurations of many types of AWS resources, using a JSON- or YAML-based templating language. +- **CloudFormation** offers mechanisms to create and update entire **[stacks](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-concepts.html#d0e3917)** comprised of many types of AWS resources. These CloudFormation stacks are defined in a **[CloudFormation template](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-concepts.html#d0e3830)** which is defined in [JSON](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#aws-properties-ec2-instance-syntax.json) or [YAML](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#aws-properties-ec2-instance-syntax.yaml). - 💸CloudFormation itself has [no additional charge](https://aws.amazon.com/cloudformation/pricing/) itself; you pay for the underlying resources. ### CloudFormation Alternatives and Lock-In -- Hashicorp’s [Terraform](https://www.terraform.io/intro/vs/cloudformation.html) is a third-party alternative. - - 🔸Some AWS features may not be available in Terraform (e.g. multi-AZ ElastiCache using Redis), and you may have to resort to embedded CloudFormation templates. +- Hashicorp’s [Terraform](https://www.terraform.io/intro/vs/cloudformation.html) is a third-party alternative that can support other cloud platforms/providers including [Azure](https://www.terraform.io/docs/providers/azure/) and [OpenStack](https://www.terraform.io/docs/providers/openstack/). +- 🔸Some AWS features may not be available in Terraform (e.g. multi-AZ ElastiCache using Redis), and you may have to resort to embedded CloudFormation templates. ### CloudFormation Tips @@ -1154,6 +1154,7 @@ CloudFormation - If you are building different stacks with similar layers, it may be useful to build separate templates for each layer that you can reuse using [AWS::CloudFormation::Stack](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-stack.html). - 🔸Avoid hardcoding resource parameters that can potentially change. Use stack parameters as much as you can, and resort to default parameter values. - 🔹Until [2016](https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/), CloudFormation used only an awkward JSON format that makes both reading and debugging difficult. To use it effectively typically involved building additional tooling, including converting it to YAML, but now [this is supported directly](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html). +- Wherever possible, export relevant [physical IDs](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-name.html) from your Stacks by defining [Outputs in your CloudFormation Templates](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html). These are the actual names assigned to the resources being created. Outputs can be returned from `DescribeStack` API calls, and get imported to other Stacks as part of the [recent addition](https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/) of [cross-stack references](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-crossstackref.html). - CloudFormation can be set up to [send SNS notifications](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html) upon state changes, enabling programatic handling of situations where stacks fail to build, or simple email alerts so the appropriate people are informed. - CloudFormation allows the use of [**conditionals**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html) when creating a stack. - One common way to leverage this capability is in support of multi-environment CloudFormation templates – by configuring them to use ‘if-else’ statements on the value of a [parameter passed in](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html) (e.g. “env”), environment-specific values for things like VPC IDs, SecurityGroup IDs, and AMI names can be passed into reusable generic templates. @@ -1163,7 +1164,7 @@ CloudFormation - ❗Modifications to stack resources made outside CloudFormation can potentially lead to stacks stuck in UPDATE\_ROLLBACK\_FAILED mode. Stacks in this state can‘t be recovered without help from AWS Support. - 🔸CloudFormation is useful but complex and with a variety of pain points. Many companies find alternate solutions, and many companies use it, but only with significant additional tooling. -- 🔸CloudFormation can be very slow, especially for items like CloudFront distributions. +- 🔸CloudFormation can be very slow, especially for items like CloudFront distributions and Route53 CNAME entries. - 🔸It’s hard to assemble good CloudFormation configurations from existing state. AWS does [offer a trick to do this](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html), but it’s very clumsy. - 🔸Many users don’t use CloudFormation at all because of its limitations, or because they find other solutions preferable. Often there are other ways to accomplish the same goals, such as local scripts (Boto, Bash, Ansible, etc.) you manage yourself that build infrastructure, or Docker-based solutions ([Convox](https://convox.com/), etc.). From cca5ac5f87d5d6cef4bab53b5256d2b467fb748b Mon Sep 17 00:00:00 2001 From: Raajhesh Kannaa Chidambaram Date: Thu, 20 Oct 2016 14:28:09 +0530 Subject: [PATCH 123/140] Updated the AWS Products Maturity Table Added the information that AWS Certificate Manager is supported by AWS CLI. https://github.com/aws/aws-cli/blob/master/CHANGELOG.rst#1100 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 40296e5..e197cd5 100644 --- a/README.md +++ b/README.md @@ -313,7 +313,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | Service | Original release | Availability | CLI Support | |------------------------------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------|:-----------:| | 🐥[Database Migration Service](https://aws.amazon.com/releasenotes/AWS-Database-Migration-Service?browse=1) | 2016-03 | General | | -| 🐥[Certificate Manager](https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/) | 2016-01 | General | +| 🐥[Certificate Manager](https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/) | 2016-01 | General | ✓ | 🐥[IoT](https://aws.amazon.com/blogs/aws/aws-iot-now-generally-available/) | 2015-08 | General | ✓ | | 🐥[WAF](https://aws.amazon.com/releasenotes/AWS-WAF?browse=1) | 2015-10 | General | ✓ | | 🐥[Data Pipeline](https://aws.amazon.com/releasenotes/AWS-Data-Pipeline?browse=1) | 2015-10 | General | ✓ | From c224e2dd56c2e31f8c2b6a11e3eacc74bd24a71a Mon Sep 17 00:00:00 2001 From: nickbarclay Date: Thu, 20 Oct 2016 08:05:54 -0400 Subject: [PATCH 124/140] Changed upper case E in ANALYZEd --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b4c4f8a..d08a25a 100644 --- a/README.md +++ b/README.md @@ -1268,7 +1268,7 @@ Redshift - [Top 10 Performance Tuning Techniques for Amazon Redshift](https://blogs.aws.amazon.com/bigdata/post/Tx31034QG0G3ED1/Top-10-Performance-Tuning-Techniques-for-Amazon-Redshift) provides an excellent list of performance tuning techniques. - [Amazon Redshift Utils](https://github.com/awslabs/amazon-redshift-utils) contains useful utilities, scripts and views to simplify Redshift ops. - [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/t_Reclaiming_storage_space202.html) regularly following a significant number of deletes or updates to reclaim space and improve query performance. -- Avoid performing blanket [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/r_VACUUM_command.html) or [ANALYZE](http://docs.aws.amazon.com/redshift/latest/dg/r_ANALYZE.html) operations at a cluster level. The checks on each table to determine whether VACUUM or ANALYZE action needs to be taken is wasteful. Only perform ANALYZE and VACUUM commands on the objects that require it. Utilize the [Analyze & Vacuum Schema Utility](https://github.com/awslabs/amazon-redshift-utils/tree/master/src/AnalyzeVacuumUtility) to perform this work. The SQL to determine whether a table needs to be VACUUMed or ANALYZed can be found in the [Schema Utility README](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AnalyzeVacuumUtility/README.md) if you wish to create your own maintenance process. +- Avoid performing blanket [VACUUM](http://docs.aws.amazon.com/redshift/latest/dg/r_VACUUM_command.html) or [ANALYZE](http://docs.aws.amazon.com/redshift/latest/dg/r_ANALYZE.html) operations at a cluster level. The checks on each table to determine whether VACUUM or ANALYZE action needs to be taken is wasteful. Only perform ANALYZE and VACUUM commands on the objects that require it. Utilize the [Analyze & Vacuum Schema Utility](https://github.com/awslabs/amazon-redshift-utils/tree/master/src/AnalyzeVacuumUtility) to perform this work. The SQL to determine whether a table needs to be VACUUMed or ANALYZEd can be found in the [Schema Utility README](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AnalyzeVacuumUtility/README.md) if you wish to create your own maintenance process. - Redshift provides various [column compression](http://docs.aws.amazon.com/redshift/latest/dg/t_Compressing_data_on_disk.html) options to optimize the stored data size. AWS strongly encourages users to use [automatic compression](http://docs.aws.amazon.com/redshift/latest/dg/c_Loading_tables_auto_compress.html) at the COPY stage, when Redshift uses a sample of the data being ingested to analyze the column compression options. However, automatic compression can only be applied to an empty table with no data. Therefore, make sure the initial load batch is big enough to provide Redshift with a representative sample of the data (the default sample size is 100000 rows). ### Redshift Gotchas and Limitations From ec4446e4add36285faa93865ac4e29f37ac3fa91 Mon Sep 17 00:00:00 2001 From: Thibault Charbonnier Date: Fri, 21 Oct 2016 11:41:26 -0700 Subject: [PATCH 125/140] fix typo in auto scaling section * remove a trailing double quote * remove unnecessary commas --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 188a3ac..29ad5f0 100644 --- a/README.md +++ b/README.md @@ -802,7 +802,7 @@ Auto Scaling - 📒 [Homepage](https://aws.amazon.com/autoscaling/) ∙ [User guide](http://docs.aws.amazon.com/autoscaling/latest/userguide/) ∙ [FAQ](https://aws.amazon.com/ec2/faqs/#Auto_Scaling) ∙ [Pricing](https://aws.amazon.com/autoscaling/pricing/) at no additional charge - [**Auto Scaling Groups (ASGs)**](https://aws.amazon.com/autoscaling/) are used to control the number of instances in a service, reducing manual effort to provision or deprovision EC2 instances. -- They can be configured, through [Scaling Policies](http://docs.aws.amazon.com/autoscaling/latest/userguide/policy_creating.html),” to automatically increase or decrease instance counts based on metrics like CPU utilization, or based on a schedule. +- They can be configured through [Scaling Policies](http://docs.aws.amazon.com/autoscaling/latest/userguide/policy_creating.html) to automatically increase or decrease instance counts based on metrics like CPU utilization, or based on a schedule. - There are three common ways of using ASGs - dynamic (automatically adjust instance count based on metrics for things like CPU utilization), static (maintain a specific instance count at all times), scheduled (maintain different instance counts at different times of day or on days of the week). - 💸ASGs [have no additional charge](https://aws.amazon.com/autoscaling/pricing/) themselves; you pay for underlying EC2 and CloudWatch services. From 2b8266a0c0a227d557ea3ec81be420743da72964 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Fri, 21 Oct 2016 13:44:38 -0700 Subject: [PATCH 126/140] Update authors. --- AUTHORS.md | 33 +++++++++++++++++++++++++-------- admin/authors-info.yml | 2 ++ 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index ee5fb76..553f2b8 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -7,7 +7,8 @@ Please help if you can, and see the [contribution guidelines](CONTRIBUTING.md) f Alphabetically by username: -* [0xmohit](https://github.com/0xmohit) — [2+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) +* [0xmohit](https://github.com/0xmohit) — [3+](https://github.com/open-guides/og-aws/commits?author=0xmohit)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3A0xmohit) +* [Abdul Mohammed (abdulirfan3)](https://github.com/abdulirfan3) — [1+](https://github.com/open-guides/og-aws/commits?author=abdulirfan3)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aabdulirfan3) * [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah) * [Andrew Lane (AndrewLane)](https://github.com/AndrewLane) — [1+](https://github.com/open-guides/og-aws/commits?author=AndrewLane)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AAndrewLane) * [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [5+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin) @@ -17,52 +18,68 @@ Alphabetically by username: * [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer) * [Bradly Feeley (bradly)](https://github.com/bradly) — [2+](https://github.com/open-guides/og-aws/commits?author=bradly)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Abradly) * [Brady Dowling (bradydowling)](https://github.com/bradydowling) — [1+](https://github.com/open-guides/og-aws/commits?author=bradydowling)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abradydowling) -* [Mårten Gustafson (chids)](https://github.com/chids) — [3+](https://github.com/open-guides/og-aws/commits?author=chids)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Achids) +* [Mårten Gustafson (chids)](https://github.com/chids) — [4+](https://github.com/open-guides/og-aws/commits?author=chids)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Achids) * [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) * [Chris Lennon (chrislennon)](https://github.com/chrislennon) — [1+](https://github.com/open-guides/og-aws/commits?author=chrislennon)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrislennon) * [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) +* [Dan Herman (danherman)](https://github.com/danherman) — _expert (RDS)_ * [Dan Hermann (danhermann)](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) * [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) * [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ +* [Matthew Lapworth (dragonndev)](https://github.com/dragonndev) — [5+](https://github.com/open-guides/og-aws/commits?author=dragonndev)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adragonndev) * [Dmitry Guyvoronsky (dreamiurg)](https://github.com/dreamiurg) — [1+](https://github.com/open-guides/og-aws/commits?author=dreamiurg)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adreamiurg) * [Eric Hammond (ehammond)](https://github.com/ehammond) — _expert (Lambda, serverless)_ * [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy) +* [Elliott Spira (em0ney)](https://github.com/em0ney) — [5+](https://github.com/open-guides/og-aws/commits?author=em0ney)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Aem0ney) * [esell (esell)](https://github.com/esell) — [5+](https://github.com/open-guides/og-aws/commits?author=esell)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aesell) * [Max Grigorev (forwidur)](https://github.com/forwidur) — _editor (EBS, RDS)_ +* [Greg Leeds (gleeds)](https://github.com/gleeds) — [2+](https://github.com/open-guides/og-aws/commits?author=gleeds)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agleeds) * [Glynn Forrest (glynnforrest)](https://github.com/glynnforrest) — [1+](https://github.com/open-guides/og-aws/commits?author=glynnforrest)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aglynnforrest) * [Dmitry Golyshev (golyshev)](https://github.com/golyshev) +* [gscalise](https://github.com/gscalise) — [1+](https://github.com/open-guides/og-aws/commits?author=gscalise)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agscalise) * [Gulam Shakir (gshakir)](https://github.com/gshakir) — [2+](https://github.com/open-guides/og-aws/commits?author=gshakir)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agshakir) * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) * [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [92+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[81+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Jason Barry (JCBarry)](https://github.com/JCBarry) — [1+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [93+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[85+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Jose Luis Ordiales Coscia (jlordiales)](https://github.com/jlordiales) — [1+](https://github.com/open-guides/og-aws/commits?author=jlordiales)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlordiales) * [Jaanus Torp (jnsaff)](https://github.com/jnsaff) — [1+](https://github.com/open-guides/og-aws/commits?author=jnsaff)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajnsaff) * Jurgen Philippaerts * [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura) — [0+](https://github.com/open-guides/og-aws/commits?author=kazuyukitanimura)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akazuyukitanimura) * [Kyle Busekist (kbusekist)](https://github.com/kbusekist) — [2+](https://github.com/open-guides/og-aws/commits?author=kbusekist)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Akbusekist) -* [Krishan (KrishanBhasin)](https://github.com/KrishanBhasin) — [1+](https://github.com/open-guides/og-aws/commits?author=KrishanBhasin)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3AKrishanBhasin) -* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [8+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ +* [Kim Schmidt (kimschmidtsbrain)](https://github.com/kimschmidtsbrain) — _expert (AWS Marketplace, Aurora)_ +* [Krishan (KrishanBhasin)](https://github.com/KrishanBhasin) — [3+](https://github.com/open-guides/og-aws/commits?author=KrishanBhasin)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3AKrishanBhasin) +* [Łukasz Kostka (luqasz)](https://github.com/luqasz) — [4+](https://github.com/open-guides/og-aws/commits?author=luqasz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aluqasz) +* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [11+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[8+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit) — _editor (IoT)_ * [maiki](https://github.com/maiki) — [1+](https://github.com/open-guides/og-aws/commits?author=maiki)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amaiki) * [Manoj M J (manojmj92)](https://github.com/manojmj92) — [1+](https://github.com/open-guides/og-aws/commits?author=manojmj92)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amanojmj92) * [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d) -* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [10+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[6+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ +* [Martijn Dwars (MartijnDwars)](https://github.com/MartijnDwars) — [3+](https://github.com/open-guides/og-aws/commits?author=MartijnDwars)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AMartijnDwars) +* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [10+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[7+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (EC2, S3, Glacier, EMR, Redshift)_ * [John Merrells (merrells)](https://github.com/merrells) — _expert (cloud infrastructure, when to use AWS)_ -* [Mikhail Advani (mikhailadvani)](https://github.com/mikhailadvani) — [1+](https://github.com/open-guides/og-aws/commits?author=mikhailadvani)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Amikhailadvani) +* [Mikhail Advani (mikhailadvani)](https://github.com/mikhailadvani) — [5+](https://github.com/open-guides/og-aws/commits?author=mikhailadvani)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3Amikhailadvani) * [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke) +* [nickbarclay](https://github.com/nickbarclay) — [2+](https://github.com/open-guides/og-aws/commits?author=nickbarclay)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anickbarclay) * [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_ +* [Justin Mullis (nonadmin)](https://github.com/nonadmin) — [1+](https://github.com/open-guides/og-aws/commits?author=nonadmin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anonadmin) * [Ola Wiberg (olawiberg)](https://github.com/olawiberg) * [Pascal Borreli (pborreli)](https://github.com/pborreli) — [1+](https://github.com/open-guides/og-aws/commits?author=pborreli)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apborreli) * [Phillip Calvin (pnc)](https://github.com/pnc) — [1+](https://github.com/open-guides/og-aws/commits?author=pnc)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Apnc) * Praveen Patnala +* [Raajhesh Kannaa Chidambaram (raajheshkannaa)](https://github.com/raajheshkannaa) — [1+](https://github.com/open-guides/og-aws/commits?author=raajheshkannaa)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Araajheshkannaa) * [Richard Birkby (rbirkby)](https://github.com/rbirkby) — [2+](https://github.com/open-guides/og-aws/commits?author=rbirkby)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Arbirkby) * [Rich Adams (richadams)](https://github.com/richadams) — [1+](https://github.com/open-guides/og-aws/commits?author=richadams)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Arichadams) — _editor (VPC)_ * [Russell Power (rjpower)](https://github.com/rjpower) * [Ashley Davis (SgtCoDFish)](https://github.com/SgtCoDFish) — [1+](https://github.com/open-guides/og-aws/commits?author=SgtCoDFish)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3ASgtCoDFish) * [David Schott (shott85)](https://github.com/shott85) — [1+](https://github.com/open-guides/og-aws/commits?author=shott85)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashott85) +* [Shubham Aggarwal (shubham391)](https://github.com/shubham391) — [1+](https://github.com/open-guides/og-aws/commits?author=shubham391)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashubham391) * [Adam Nelson (spudstuff)](https://github.com/spudstuff) — [4+](https://github.com/open-guides/og-aws/commits?author=spudstuff)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aspudstuff) * [Steven Maude (StevenMaude)](https://github.com/StevenMaude) — [1+](https://github.com/open-guides/og-aws/commits?author=StevenMaude)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AStevenMaude) -* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [17+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[20+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ +* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [19+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[22+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _project lead, editor (topics not otherwise assigned)_ * [Carlos (theist)](https://github.com/theist) — [1+](https://github.com/open-guides/og-aws/commits?author=theist)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Atheist) +* [TheRandomSecurityGuy (therandomsecurityguy)](https://github.com/therandomsecurityguy) — [1+](https://github.com/open-guides/og-aws/commits?author=therandomsecurityguy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atherandomsecurityguy) * [Sandeep Dinesh (thesandlord)](https://github.com/thesandlord) — [1+](https://github.com/open-guides/og-aws/commits?author=thesandlord)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Athesandlord) +* [Thibault Charbonnier (thibaultcha)](https://github.com/thibaultcha) — [1+](https://github.com/open-guides/og-aws/commits?author=thibaultcha)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Athibaultcha) * [Tom Schlick (tomschlick)](https://github.com/tomschlick) — [3+](https://github.com/open-guides/og-aws/commits?author=tomschlick)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atomschlick) * [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite) * [Uli Stroetz (ustroetz)](https://github.com/ustroetz) — [1+](https://github.com/open-guides/og-aws/commits?author=ustroetz)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Austroetz) diff --git a/admin/authors-info.yml b/admin/authors-info.yml index 532f4e5..01972cf 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -27,6 +27,8 @@ roles: merrells: expert (cloud infrastructure, when to use AWS) benkehoe: expert (IoT) ehammond: expert (Lambda, serverless) + danherman: expert (RDS) + kimschmidtsbrain: expert (AWS Marketplace, Aurora) marcello3d: golyshev: Praveen Patnala: From b9b4b7160229c50efe322943b0c2abed64c7c734 Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Fri, 21 Oct 2016 13:47:31 -0700 Subject: [PATCH 127/140] Fix typo in authors. --- AUTHORS.md | 5 ++--- admin/authors-info.yml | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/AUTHORS.md b/AUTHORS.md index 553f2b8..6e3ba68 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -22,8 +22,7 @@ Alphabetically by username: * [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin) * [Chris Lennon (chrislennon)](https://github.com/chrislennon) — [1+](https://github.com/open-guides/og-aws/commits?author=chrislennon)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrislennon) * [Chris Leyva (chrisleyva)](https://github.com/chrisleyva) — [1+](https://github.com/open-guides/og-aws/commits?author=chrisleyva)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Achrisleyva) -* [Dan Herman (danherman)](https://github.com/danherman) — _expert (RDS)_ -* [Dan Hermann (danhermann)](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) +* [Dan Hermann (danhermann)](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann) — _expert (RDS)_ * [David Kocher (dkocher)](https://github.com/dkocher) — [2+](https://github.com/open-guides/og-aws/commits?author=dkocher)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adkocher) * [Donne Martin (donnemartin)](https://github.com/donnemartin) — _expert (tools)_ * [Matthew Lapworth (dragonndev)](https://github.com/dragonndev) — [5+](https://github.com/open-guides/og-aws/commits?author=dragonndev)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adragonndev) @@ -41,7 +40,7 @@ Alphabetically by username: * [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk) * [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao) * [Jason Barry (JCBarry)](https://github.com/JCBarry) — [1+](https://github.com/open-guides/og-aws/commits?author=JCBarry)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3AJCBarry) -* [Joshua Levy (jlevy)](https://github.com/jlevy) — [93+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[85+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ +* [Joshua Levy (jlevy)](https://github.com/jlevy) — [93+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[86+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _project lead, editor (topics not otherwise assigned)_ * [Jose Luis Ordiales Coscia (jlordiales)](https://github.com/jlordiales) — [1+](https://github.com/open-guides/og-aws/commits?author=jlordiales)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlordiales) * [Jaanus Torp (jnsaff)](https://github.com/jnsaff) — [1+](https://github.com/open-guides/og-aws/commits?author=jnsaff)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajnsaff) * Jurgen Philippaerts diff --git a/admin/authors-info.yml b/admin/authors-info.yml index 01972cf..ba75e8d 100644 --- a/admin/authors-info.yml +++ b/admin/authors-info.yml @@ -27,7 +27,7 @@ roles: merrells: expert (cloud infrastructure, when to use AWS) benkehoe: expert (IoT) ehammond: expert (Lambda, serverless) - danherman: expert (RDS) + danhermann: expert (RDS) kimschmidtsbrain: expert (AWS Marketplace, Aurora) marcello3d: golyshev: From fd12c96efe51ff02f9ee85b0c933b66fa51a6e54 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Fri, 21 Oct 2016 14:00:12 -0700 Subject: [PATCH 128/140] on 1406 added line about MQTT client test tool AWS added this to the IoT console to help people get started. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 29ad5f0..7ef5fd6 100644 --- a/README.md +++ b/README.md @@ -1403,6 +1403,7 @@ IoT - Support for **device security** via certificate processing is a key differentiator in this space. In August 2016, AWS added [just-in-time registrations](https://aws.amazon.com/blogs/iot/just-in-time-registration-of-device-certificates-on-aws-iot/) for IoT devices to their services. - **Combining with other services:** It's common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). - **Testing tools:** + * To get started, AWS includes a lightweight MQTT client in the AWS IoT console. Here you can create and test sending and receiving messages to and from various MQTT topics. * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators * Use this [MQTT load simulator](https://github.com/awslabs/aws-iot-mqtt-load-generator) to test device message load throughout your IoT solution. From d1c2506764fc7aee6768e74a6a1e04dcbd49ff73 Mon Sep 17 00:00:00 2001 From: Thibault Charbonnier Date: Fri, 21 Oct 2016 16:25:22 -0700 Subject: [PATCH 129/140] fix typos and some rephrasings * fix typos * rephrase an awkward sentence (which had a typo too) * rephrase the `managing multiple regions is complex` sentence which was a bit awkward as well --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 29ad5f0..1530a13 100644 --- a/README.md +++ b/README.md @@ -983,7 +983,7 @@ RDS ### RDS Basics -- 📒 [Homepage](https://aws.amazon.com/rds/) ∙ [User guide](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/) ∙ [FAQ](https://aws.amazon.com/rds/faqs/) ∙ [Pricing](https://aws.amazon.com/rds/pricing/)(see also [ec2instances.info/rds/](http://www.ec2instances.info/rds/)\) +- 📒 [Homepage](https://aws.amazon.com/rds/) ∙ [User guide](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/) ∙ [FAQ](https://aws.amazon.com/rds/faqs/) ∙ [Pricing](https://aws.amazon.com/rds/pricing/) (see also [ec2instances.info/rds/](http://www.ec2instances.info/rds/)\) - **RDS** is a managed relational database service, allowing you to deploy and scale databases more easily. It supports Amazon Aurora, Oracle, Microsoft SQL Server, PostgreSQL, MySQL and MariaDB. ### RDS Tips @@ -1182,7 +1182,7 @@ VPCs, Network Security, and Security Groups ### VPC and Network Security Tips - ❗**Security groups** are your first line of defense for your servers. Be extremely restrictive of what ports are open to all incoming connections. In general, if you use CLBs, ALBs or other load balancing, the only ports that need to be open to incoming traffic would be port 22 and whatever port your application uses. -- **Port hygiene:** A good habit is to pick unique ports within an unusual range for each different kind of production service. For example, your web fronted might use 3010, your backend services 3020 and 3021, and your Postgres instances the usual 5432. Then make sure you have fine-grained security groups for each set of servers. This makes you disciplined about listing out your services, but also is more error-proof. For example, should you accidentally have an extra Apache server running on the default port 80 on a backend server, it will not be exposed. +- **Port hygiene:** A good habit is to pick unique ports within an unusual range for each different kind of production service. For example, your web frontend might use 3010, your backend services 3020 and 3021, and your Postgres instances the usual 5432. Then make sure you have fine-grained security groups for each set of servers. This makes you disciplined about listing out your services, but also is more error-proof. For example, should you accidentally have an extra Apache server running on the default port 80 on a backend server, it will not be exposed. - **Migrating from Classic**: For migrating from older EC2-Classic deployments to modern EC2-VPC setup, [this article](http://blog.kiip.me/engineering/ec2-to-vpc-executing-a-zero-downtime-migration/) may be of help. - For basic AWS use, one default VPC may be sufficient. But as you scale up, you should consider mapping out network topology more thoroughly. A good overview of best practices is [here](http://blog.flux7.com/blogs/aws/vpc-best-configuration-practices). - Consider controlling access to your private AWS resources through a [VPN](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html). @@ -1243,7 +1243,7 @@ CloudFront - 💸⏱Consider invalidation needs carefully. CloudFront [does support invalidation](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html) of objects from edge locations, but this typically takes many minutes to propagate to edge locations, and costs $0.005 per request after the first 1000 requests. (Some other CDNs support this better.) - Everyone should use TLS nowadays if possible. [Ilya Grigorik’s table](https://istlsfastyet.com/#cdn-paas) offers a good summary of features regarding TLS performance features of CloudFront. - An alternative to invalidation that is often easier to manage, and instant, is to configure the distribution to [cache with query strings](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/QueryStringParameters.html) and then append unique query strings with versions onto assets that are updated frequently. -- ⏱For good web performance, it’s important turn on the option to [enable compression](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html) on CloudFront distributions if the origin is S3 or another source that does not already compress. +- ⏱For good web performance, it is recommended to [enable compression](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html) on CloudFront distributions if the origin is S3 or another source that does not already compress. ### CloudFront Gotchas and Limitations @@ -1294,7 +1294,7 @@ Redshift ### Redshift Gotchas and Limitations - ❗⏱While Redshift can handle heavy queries well, it does not scale horizontally, i.e. does not handle multiple queries in parallel. Therefore, if you expect a high parallel load, consider replicating or (if possible) sharding your data across multiple clusters. -- 🔸Leader node, which manages communications with client programs and all communication with compute nodes, is the single point of failure. +- 🔸 The leader node, which manages communications with client programs and all communication with compute nodes, is the single point of failure. - ⏱Although most Redshift queries parallelize well at the compute node level, certain stages are executed on the leader node, which can become the bottleneck. - 🔹Redshift data commit transactions are very expensive and serialized at the cluster level. Therefore, consider grouping multiple mutation commands (COPY/INSERT/UPDATE) commands into a single transaction whenever possible. - 🔹Redshift does not support multi-AZ deployments. Building multi-AZ clusters is not trivial. [Here](https://blogs.aws.amazon.com/bigdata/post/Tx13ZDHZANSX9UX/Building-Multi-AZ-or-Multi-Region-Amazon-Redshift-Clusters) is an example using Kinesis. @@ -1449,7 +1449,7 @@ This section covers tips and information on achieving [high availability](https: - AWS offers two levels of redundancy, [regions and availability zones (AZs)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones). - When used correctly, regions and zones do allow for high availability. You may want to use non-AWS providers for larger business risk mitigation (i.e. not tying your company to one vendor), but reliability of AWS across regions is very high. -- **Multiple regions:** Using multiple regions is complex, since it’s essentially like completely separate infrastructure. It is necessary for business-critical services with the highest levels of redundancy. However, for many applications (like your average consumer startup), deploying extensive redundancy across regions may be overkill. +- **Multiple regions:** Using multiple regions is complex, since it’s essentially like managing completely separate infrastructures. It is necessary for business-critical services with the highest levels of redundancy. However, for many applications (like your average consumer startup), deploying extensive redundancy across regions may be overkill. - The [High Scalability Blog](http://highscalability.com/blog/2016/1/11/a-beginners-guide-to-scaling-to-11-million-users-on-amazons.html) has a good guide to help you understand when you need to scale an application to multiple regions. - 🔹**Multiple AZs:** Using AZs wisely is the primary tool for high availability! - A typical single-region high availability architecture would be to deploy in two or more availability zones, with load balancing in front, as in [this AWS diagram](http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_ftha_04.pdf). From 5d4d7a20fdc9cf5144d497da9677d371cd074b54 Mon Sep 17 00:00:00 2001 From: Thibault Charbonnier Date: Fri, 21 Oct 2016 19:20:54 -0700 Subject: [PATCH 130/140] add Kong and Tyk to API Gateway OSS Alternatives (#247) --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1530a13..97ddd87 100644 --- a/README.md +++ b/README.md @@ -271,7 +271,7 @@ Many services within AWS can at least be compared with Google Cloud offerings or |-------------------------------|------------------------------------------------------------------------------|------------------------------|-----------------|------------------------------------|-----------------------------------|------------------------------------------------------------| | Virtual server | EC2 | Compute Engine (GCE) | | Virtual Machine | DigitalOcean | OpenStack | | PaaS | Elastic Beanstalk | App Engine | App Engine | Web Apps | Heroku, AppFog, OpenShift | Meteor, AppScale, Cloud Foundry, Convox | -| Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | PubNub Blocks, Auth0 Webtask | | +| Serverless, microservices | Lambda, API Gateway | Functions | | Function Apps | PubNub Blocks, Auth0 Webtask | Kong, Tyk | | Container, cluster manager | ECS | Container Engine, Kubernetes | Borg or Omega | Container Service | | Kubernetes, Mesos, Aurora | | File storage | S3 | Cloud Storage | GFS | Storage Account | | Swift, HDFS | | Block storage | EBS | Persistent Disk | | Storage Account | | NFS | @@ -1098,6 +1098,11 @@ API Gateway - It allows “serverless” deployment of applications built with Lambda. - 🔸Switching over deployments after upgrades can be tricky. There are no built-in mechanisms to have a single domain name migrate from one API gateway to another one. So it may be necessary to build an additional layer in front (even another API Gateway) to allow smooth migration from one deployment to another. +### API Gateway Alternatives and Lock-In + +- [Kong](https://getkong.org) is an open-source, on-premises API and microservices gateway built on nginx with Lua. Kong is extensible through “plugins”. +- [Tyk](https://tyk.io) is an open-source API gateway implemented in Go and available in the cloud, on-premises or hybrid. + ### API Gateway Gotchas and Limitations - 🔸API Gateway only supports encrypted (https) endpoints, and does not support unencrypted HTTP. (This is probably a good thing.) From 8e2f285fa3cfd2ccdc6c008138844a005a4ad742 Mon Sep 17 00:00:00 2001 From: Lynn Langit Date: Fri, 21 Oct 2016 20:21:43 -0700 Subject: [PATCH 131/140] 1406 removed space between sentences --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7ef5fd6..2a4d751 100644 --- a/README.md +++ b/README.md @@ -1403,7 +1403,7 @@ IoT - Support for **device security** via certificate processing is a key differentiator in this space. In August 2016, AWS added [just-in-time registrations](https://aws.amazon.com/blogs/iot/just-in-time-registration-of-device-certificates-on-aws-iot/) for IoT devices to their services. - **Combining with other services:** It's common to use other AWS services, such as AWS Lambda, Kinesis and DynamoDB, although this is by no means required. Sample IoT application reference architectures are in this [screencast](https://www.youtube.com/watch?v=0Izh6ySpwb8/). - **Testing tools:** - * To get started, AWS includes a lightweight MQTT client in the AWS IoT console. Here you can create and test sending and receiving messages to and from various MQTT topics. + * To get started, AWS includes a lightweight MQTT client in the AWS IoT console. Here you can create and test sending and receiving messages to and from various MQTT topics. * When testing locally, if using MQTT, it may be helpful to download and use the open source [Mosquitto broker](https://mosquitto.org/download/) tool for local testing with devices and/or device simulators * Use this [MQTT load simulator](https://github.com/awslabs/aws-iot-mqtt-load-generator) to test device message load throughout your IoT solution. From a4b977d243f3cfcdd209130ec696f6b8222bd889 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Sun, 23 Oct 2016 22:33:25 +0200 Subject: [PATCH 132/140] Add basic info about Certificate Manager --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index 97ddd87..9717ab8 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ Table of Contents | [Device Farm](#device-farm) | [📗](#device-farm-basics) | | | | [IoT](#iot) | [📗](#iot-basics) | [📘](#iot-tips) | [📙](#iot-gotchas-and-limitations) | | [SES](#ses) | [📗](#ses-basics) | [📘](#ses-tips) | [📙](#ses-gotchas-and-limitations) | +| [Certificate Manager](#certificate-manager) | [📗](#certificate-manager-basics) | | | **Special Topics** @@ -1445,6 +1446,19 @@ SES - 🔸**Internet Access:** SES SMTP endpoints are on the Internet and will not be accessible from a location without Internet access (e.g. a private subnet without NAT gateway route in the routing table). In such a case, set up an SMTP relay instance in a subnet with Internet access and configure your application to send emails to this SMTP relay instance rather than SES. The relay should have a [forwarding rule to send all emails to SES](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp-existing-server.html)). ❗If you are using a proxy instead of a NAT, confirm that your proxy service supports SMTP. +Certificate Manager +------------------- + +### Certificate Manager Basics +- 📒 [Homepage](https://aws.amazon.com/certificate-manager/) ∙ [User guide](http://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html) ∙ [FAQ](https://aws.amazon.com/certificate-manager/faqs/) ∙ [Pricing](https://aws.amazon.com/certificate-manager/pricing/) +- **Certificate Manager** is an AWS service for managing SSL/TLS certificates for AWS services. +- Supports import of already owned certificates and issuing new certificates. + +### Certificate Manager Alternatives and Lock-in +- Service allows import of existed certificates without limitations, but certificates issued via this service can't be used outside of AWS. + +🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) + High Availability ----------------- From d2e3bcb114b1da5a6e20f522799714b77e08e974 Mon Sep 17 00:00:00 2001 From: Kim Schmidt Date: Sun, 23 Oct 2016 14:37:05 -0700 Subject: [PATCH 133/140] DynamoDB Indexing and Data Types (#248) * DynamoDB Indexing and Data Types * Fixed typo attribute * DynamoDB changes & doc fixed --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 97ddd87..7edceba 100644 --- a/README.md +++ b/README.md @@ -1020,6 +1020,7 @@ DynamoDB - There is a [local version](https://aws.amazon.com/blogs/aws/dynamodb-local-for-desktop-development/) of DynamoDB provided for developer use. - [DynamoDB Streams](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) provides an ordered stream of changes to a table. Use it to replicate, back up, or drive events off of data - DynamoDB can be used [as a simple locking service](https://gist.github.com/ryandotsmith/c95fd21fab91b0823328). +- DynamoDB indexing can include **primary** **keys**, which can either be a single-attribute hash key or a composite hash-key range. You can also query non-primary key attributes using [**secondary** **indexes**](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html). ### DynamoDB Gotchas and Limitations @@ -1027,7 +1028,7 @@ DynamoDB - It is important to make sure that DynamoDB [resource limits](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-data-types) are compatible with your dataset and workload. For example, the maximum size value that can be added to a DynamoDB table is 400 KB (larger items can be stored in S3 and a URL stored in DynamoDB). - 🔸 Dealing with **time series data** in DynamoDB can be challenging. A global secondary index together with down sampling timestamps can be a possible solution as explained [here](https://blogs.aws.amazon.com/bigdata/post/Tx3KPZDXIBJEQ4B/Scaling-Writes-on-Amazon-DynamoDB-Tables-with-Global-Secondary-Indexes). - DynamoDB does [not allow](https://forums.aws.amazon.com/thread.jspa?threadID=90137) an empty string as a valid attribute value. The most common work-around is to use a substitute value instead of leaving the field empty. - +- Data Types: DynamoDB supports three [data types](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.DataTypes.html) -- **number**, **string**, and **binary** -- in both scalar and multi-valued sets. DynamoDB can also support [JSON](https://aws.amazon.com/blogs/aws/dynamodb-update-json-and-more/). ECS --- From c14a5ebec6924e6269cb576e305a221aa2b6a8ef Mon Sep 17 00:00:00 2001 From: Thanos Baskous Date: Sun, 23 Oct 2016 14:42:45 -0700 Subject: [PATCH 134/140] Minor DynamoDB Indexing Fixes (#254) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7edceba..1b0f97d 100644 --- a/README.md +++ b/README.md @@ -1020,7 +1020,7 @@ DynamoDB - There is a [local version](https://aws.amazon.com/blogs/aws/dynamodb-local-for-desktop-development/) of DynamoDB provided for developer use. - [DynamoDB Streams](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) provides an ordered stream of changes to a table. Use it to replicate, back up, or drive events off of data - DynamoDB can be used [as a simple locking service](https://gist.github.com/ryandotsmith/c95fd21fab91b0823328). -- DynamoDB indexing can include **primary** **keys**, which can either be a single-attribute hash key or a composite hash-key range. You can also query non-primary key attributes using [**secondary** **indexes**](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html). +- DynamoDB indexing can include **primary keys**, which can either be a single-attribute hash key or a composite hash-key range. You can also query non-primary key attributes using [**secondary indexes**](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html). ### DynamoDB Gotchas and Limitations @@ -1028,7 +1028,7 @@ DynamoDB - It is important to make sure that DynamoDB [resource limits](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-data-types) are compatible with your dataset and workload. For example, the maximum size value that can be added to a DynamoDB table is 400 KB (larger items can be stored in S3 and a URL stored in DynamoDB). - 🔸 Dealing with **time series data** in DynamoDB can be challenging. A global secondary index together with down sampling timestamps can be a possible solution as explained [here](https://blogs.aws.amazon.com/bigdata/post/Tx3KPZDXIBJEQ4B/Scaling-Writes-on-Amazon-DynamoDB-Tables-with-Global-Secondary-Indexes). - DynamoDB does [not allow](https://forums.aws.amazon.com/thread.jspa?threadID=90137) an empty string as a valid attribute value. The most common work-around is to use a substitute value instead of leaving the field empty. -- Data Types: DynamoDB supports three [data types](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.DataTypes.html) -- **number**, **string**, and **binary** -- in both scalar and multi-valued sets. DynamoDB can also support [JSON](https://aws.amazon.com/blogs/aws/dynamodb-update-json-and-more/). +- Data Types: DynamoDB supports three [data types](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.DataTypes.html) – **number**, **string**, and **binary** – in both scalar and multi-valued sets. DynamoDB can also support [JSON](https://aws.amazon.com/blogs/aws/dynamodb-update-json-and-more/). ECS --- From a93133421687e9165ba7d32d656e7799cc2360f2 Mon Sep 17 00:00:00 2001 From: Dan Hermann Date: Mon, 24 Oct 2016 01:41:17 -0500 Subject: [PATCH 135/140] Break RDS into separate sections for MySQL, Aurora, and SQL Server (#255) * Break RDS into separate sections for MySQL, Aurora, and SQL Server * fix icon * fix TOC formatting --- README.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 65 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 9ba6ce7..eaa90a6 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -![An Open Guide](figures/signpost-horiz1-1600.jpg) +![An Open Guide](figures/signpost-horiz1-1600.jpg) The Open Guide to Amazon Web Services ===================================== @@ -38,6 +38,9 @@ Table of Contents | [Elastic IPs](#elastic-ips) | [📗](#elastic-ip-basics) | [📘](#elastic-ip-tips) | [📙](#elastic-ip-gotchas-and-limitations) | | [Glacier](#glacier) | [📗](#glacier-basics) | [📘](#glacier-tips) | [📙](#glacier-gotchas-and-limitations) | | [RDS](#rds) | [📗](#rds-basics) | [📘](#rds-tips) | [📙](#rds-gotchas-and-limitations) | +| [RDS MySQL and MariaDB](#rds-mysql-and-mariadb) | [📗](#rds-mysql-and-mariadb-basics) | [📘](#rds-mysql-and-mariadb-tips) | [📙](#rds-mysql-and-mariadb-gotchas-and-limitations) | +| [RDS Aurora](#rds-aurora) | [📗](#rds-aurora-basics) | [📘](#rds-aurora-tips) | [📙](#rds-aurora-gotchas-and-limitations) | +| [RDS SQL Server](#rds-sql-server) | [📗](#rds-sql-server-basics) | [📘](#rds-sql-server-tips) | [📙](#rds-sql-server-gotchas-and-limitations) | | [DynamoDB](#dynamodb) | [📗](#dynamodb-basics) | [📘](#dynamodb-tips) | [📙](#dynamodb-gotchas-and-limitations) | | [ECS](#ecs) | [📗](#ecs-basics) | [📘](#ecs-tips) | | | [Lambda](#lambda) | [📗](#lambda-basics) | [📘](#lambda-tips) | [📙](#lambda-gotchas-and-limitations) | @@ -984,22 +987,73 @@ RDS ### RDS Basics - 📒 [Homepage](https://aws.amazon.com/rds/) ∙ [User guide](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/) ∙ [FAQ](https://aws.amazon.com/rds/faqs/) ∙ [Pricing](https://aws.amazon.com/rds/pricing/) (see also [ec2instances.info/rds/](http://www.ec2instances.info/rds/)\) -- **RDS** is a managed relational database service, allowing you to deploy and scale databases more easily. It supports Amazon Aurora, Oracle, Microsoft SQL Server, PostgreSQL, MySQL and MariaDB. +- **RDS** is a managed relational database service, allowing you to deploy and scale databases more easily. It supports [Oracle](https://aws.amazon.com/rds/oracle/), [Microsoft SQL Server](https://aws.amazon.com/rds/sqlserver/), [PostgreSQL](https://aws.amazon.com/rds/postgresql/), [MySQL](https://aws.amazon.com/rds/mysql/), [MariaDB](https://aws.amazon.com/rds/mariadb/), and Amazon’s own [Aurora](https://aws.amazon.com/rds/aurora/). +- RDS offers out of the box support for [high availability and failover](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) for your databases. ### RDS Tips -- If you’re looking for the managed convenience of RDS for MongoDB, this isn’t offered by AWS directly, but you may wish to consider a provider such as [**mLab**](https://mlab.com/). -- RDS offers out of the box support for [high availability and failover](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) for your databases. -- MySQL RDS allows access to [binary logs](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html#USER_LogAccess.MySQL.BinaryFormat). -- 🔸**MySQL vs MariaDB vs Aurora:** If you prefer a MySQL-style database but are starting something new, you probably should consider Aurora and MariaDB as well. **Aurora** has increased availability and is the next-generation solution. That said, Aurora [may not be](http://blog.takipi.com/benchmarking-aurora-vs-mysql-is-amazons-new-db-really-5x-faster/) as fast relative to MySQL as is sometimes reported, and is more complex to administer. **MariaDB**, the modern [community fork](https://en.wikipedia.org/wiki/MariaDB) of MySQL, [likely now has the edge over MySQL](http://cloudacademy.com/blog/mariadb-vs-mysql-aws-rds/) for many purposes and is supported by RDS. -- 🔸**Aurora:** Aurora is based on MySQL 5.6. If you are planning to migrate to Aurora from an existing MySQL database, avoiding any MySQL features from 5.7 or later will ease the transition. The easiest migration path to Aurora is restoring a database snapshot from MySQL 5.6. The next easiest method is restoring a dump from a MySQL-compatible database such as MariaDB. If neither of those methods are options, Amazon offers a [fee-based data migration service](http://docs.aws.amazon.com/dms/latest/userguide/Welcome.html). +- If you're looking for the managed convenience of RDS for other data stores such as MongoDB or Cassandra, you may wish to consider third-party services from providers such as [mLab](https://mlab.com/), [Compose](https://www.compose.com/), or [InstaClustr](https://www.instaclustr.com/). +- 🔹Make sure to create a new [parameter group](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) and option group for your database since the default parameter group does not allow dynamic configuration changes. ### RDS Gotchas and Limitations -- ⏱RDS instances run on EBS volumes, and hence are constrained by the EBS performance. -- 🔸Verify what database features you need, as not everything you might want is available on RDS. For example, if you are using Postgres, check the list of [supported features and extensions](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#SQLServer.Concepts.General.FeatureSupport). If the features you need aren’t supported by RDS, you’ll have to deploy your database yourself. -- If you use the failover support offered by RDS, keep in mind that it is based on DNS changes - make sure that your client reacts to these changes appropriately. This is particularly important for Java, given how its DNS resolver’s TTL is [configured by default](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html). -- 🔸**DB migration to RDS:** While importing your database into RDS ensure you take into consideration the maintenance window settings. If a backup is running at the same time, your import can take a considerable longer time than you would have expected. +- ⏱RDS instances run on EBS volumes (either general-purpose or provisioned IOPS), and hence are constrained by EBS performance. +- 🔸Verify what database features you need, as not everything you might want is available on RDS. For example, if you are using Postgres, check the list of [supported features and extensions](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#SQLServer.Concepts.General.FeatureSupport). If the features you need aren't supported by RDS, you'll have to deploy your database yourself. +- If you use the failover support offered by RDS, keep in mind that it is based on DNS changes - make sure that your client reacts to these changes appropriately. This is particularly important for Java, given how its DNS resolver's TTL is [configured by default](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html). +- 🔸**DB migration to RDS:** While importing your database into RDS ensure you take into consideration the maintenance window settings. If a backup is running at the same time, your import can take a considerably longer time than you would have expected. +- [Database sizes are limited](https://aws.amazon.com/about-aws/whats-new/2015/06/amazon-rds-increases-storage-limits-to-6TB-for-piops-and-gp2/) to 6TB for all database engines except for SQL Server which has a 4TB limit and Aurora which supports up to 64TB databases. + +RDS MySQL and MariaDB +--------------------- + +### RDS MySQL and MariaDB Basics +- RDS offers MySQL versions 5.5, 5.6, and 5.7. + +### RDS MySQL and MariaDB Tips +- MySQL RDS allows access to [binary logs](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html#USER_LogAccess.MySQL.BinaryFormat). +- Multi-AZ instances of MySQL transparently replicate data across AZs using DRBD. Automated backups of multi-AZ instances [run off the backup instance](https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls) to reduce latency spikes on the primary. +- 🔸**MySQL vs MariaDB vs Aurora:** If you prefer a MySQL-style database but are starting something new, you probably should consider Aurora and MariaDB as well. **Aurora** has increased availability and is the next-generation solution. That said, Aurora [may not be](http://blog.takipi.com/benchmarking-aurora-vs-mysql-is-amazons-new-db-really-5x-faster/) as fast relative to MySQL as is sometimes reported, and is more complex to administer. **MariaDB**, the modern [community fork](https://en.wikipedia.org/wiki/MariaDB) of MySQL, [likely now has the edge over MySQL](http://cloudacademy.com/blog/mariadb-vs-mysql-aws-rds/) for many purposes and is supported by RDS. + +### RDS MySQL and MariaDB Gotchas and Limitations +- 🔸**No SUPER privileges.** RDS provides some [stored procedures](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.SQLRef.html) to perform some tasks that require SUPER privileges such as starting or stopping replication. +- 🔸You can replicate to non-RDS instances of MySQL, but [replication to these instances will break during AZ [failovers]https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls). +- 🔸There is no ability to manually CHANGE MASTER on replicas, so they must all be rebuilt after a failover of the master. + +RDS Aurora +----------- + +### RDS Aurora Basics +- Amazon's proprietary fork of MySQL intended to scale up for high concurrency workloads. Generally speaking, individual query performance under Aurora is not expected to improve significantly relative to MySQL or MariaDB, but Aurora is intended to maintain performance while executing many more queries concurrently than an equivalent MySQL or MariaDB server could handle. +- [Notable new features](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine) include: + - Log-structured storage instead of B-trees to improve write performance + - Out-of-process buffer pool so that databases instances can be restarted without clearing the buffer pool + - The underlying physical storage is a specialized SSD array that automatically maintains 6 copies of your data across 3 AZs. + - Aurora read replicas share the storage layer with the write master which significantly reduces replica lag, eliminates the need for the master to write and distribute the binary log for replication, and allows for zero-data-loss failovers from the master to a replica. The master and all the read replicas that share storage are known collectively as an **Aurora cluster**. + +### RDS Aurora Tips +- In order to take advantage of Aurora's higher concurrency, applications should be configured with large database connection pools and should execute as many queries concurrently as possible. For example, Aurora servers have been tested to produce increasing performance on some OLTP workloads with [up to 5,000 connections](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine/31). +- [Aurora scales well with multiple CPUs](https://www.percona.com/blog/2016/05/26/aws-aurora-benchmarking-part-2/) and may require a large instance class for optimal performance. +- Because Aurora is based on MySQL 5.6.10, avoiding any MySQL features from 5.7 or later will ease the transition from a MySQL-compatible database into Aurora. +- The easiest migration path to Aurora is restoring a database snapshot from MySQL 5.6. The next easiest method is restoring a dump from a MySQL-compatible database such as MariaDB. For [low-downtime migrations](http://cantrill.io/howto/aws/2016/06/06/migrating-from-mysql-to-aurora-with-almost-no-downtime.html) from other MySQL-compatible databases, you can set up an Aurora instance as a replica of your existing database. If none of those methods are options, Amazon offers a fee-based data migration service. +- You can replicate [from an Aurora cluster to MySQL or to another Aurora cluster](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html). This requires binary logging to be enabled and is not as performant as native Aurora replication. + +### RDS Aurora Gotchas and Limitations +- 🔸[Aurora is based on MySQL 5.6.10](https://news.ycombinator.com/item?id=12415693) with some cherry-picking of later MySQL features. It is missing most 5.7 features as well as some online DDL features introduced in 5.6.17. + +RDS SQL Server +-------------- + +### RDS SQL Server Basics +- [RDS offers SQL Server 2008 R2, 2012, and 2014](https://aws.amazon.com/rds/sqlserver/) including Express, Web, Standard and Enterprise (2008 R2 and 2012 only for Enterprise) + +### RDS SQL Server Tips +- Recently added support for [backup and restore to/from S3](https://www.brentozar.com/archive/2016/07/holy-cow-amazon-rds-sql-server-just-changed-everything/) which may make it an attractive DR option for on-premises installations. + +### RDS SQL Server Gotchas and Limitations +- 🔸The user is granted only db_owner privileges for each database on the instance. +- 🔸Storage cannot be expanded for existing databases. If you need more space, you must restore your database on a new instance with larger storage. +- 🔸4TB database size limit for non-Express editions. +- 🔸Limited to [30 databases per instance](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html) DynamoDB -------- From 98ae94a3abc131f4b0e6ff7ea17394ee7365c970 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Mon, 24 Oct 2016 15:23:30 +0200 Subject: [PATCH 136/140] Add fixes --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9717ab8..3c36a0e 100644 --- a/README.md +++ b/README.md @@ -1451,11 +1451,11 @@ Certificate Manager ### Certificate Manager Basics - 📒 [Homepage](https://aws.amazon.com/certificate-manager/) ∙ [User guide](http://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html) ∙ [FAQ](https://aws.amazon.com/certificate-manager/faqs/) ∙ [Pricing](https://aws.amazon.com/certificate-manager/pricing/) -- **Certificate Manager** is an AWS service for managing SSL/TLS certificates for AWS services. -- Supports import of already owned certificates and issuing new certificates. +- Use the **Certificate Manager** to manage SSL/TLS certificates in other AWS services. +- Supports importing existing certificates as well as issuing new ones. ### Certificate Manager Alternatives and Lock-in -- Service allows import of existed certificates without limitations, but certificates issued via this service can't be used outside of AWS. +- Certificates issued by the **Certificate Manager** can't be used outside of the services that support it. Imported certificates, however, can still be used elsewhere. 🚧 [*Please help expand this incomplete section.*](CONTRIBUTING.md) From 3f5051c761a937a3ebd3463262432508399acb4a Mon Sep 17 00:00:00 2001 From: Joshua Levy Date: Mon, 24 Oct 2016 22:26:21 -0700 Subject: [PATCH 137/140] A couple process things (#260) * A few more suggestions on PR review. * Remove Airtable links. They're not used widely enough to merit such prominence. * Review fixes. --- CONTRIBUTING.md | 56 ++++++++++++++++++++++++++++++++----------------- README.md | 24 ++++++++++----------- 2 files changed, 49 insertions(+), 31 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6987025..a8c6dcf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,24 +1,29 @@ Contributing ------------ -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly [credit](/AUTHORS.md) all contributors. Here are few notes before you jump in. -### Please Help +Please Help +----------- -If you’ve found this guide useful, you have many ways to help: +If you’ve found this guide useful, please join us: -- The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) or [**add to our list of common questions**](https://airtable.com/shrXZ61VrovWfXYBg), which helps the community and guides what contributors can focus on. We encourage you to ask AWS questions and help others! +- The simplest thing you can do to contribute is [**join the Slack channel**](https://og-aws.slack.lexikon.io/) and **ask or answer questions** or **discuss**, which +helps the community and guides what contributors can focus on. - [**File issues**](https://github.com/open-guides/og-aws/issues) if it’s clear something needs to be improved and you’re not able to make a pull request. - [**Pull requests**](https://github.com/open-guides/og-aws/pulls) with changes are always welcome. Please keep them small and focused, so we can add items individually, and review the conventions below. If you want to make a larger change, try to discuss it in Slack. - **Review** or **comment** on existing issues and pull requests if you have expertise. - If you have deep expertise, we may ask you to be an **editor** or **expert**. Editors and experts are assigned roles that [help us review](#editorial-process) the Guide. Join Slack to discuss this. +Making Contributions +-------------------- + ### Pull Request Etiquette - Keep changes as small as is practical. Do not make changes to multiple sections at once, alter whitespace in broad ways, etc. -- Neutrality: If you have an affiliation related to what you are changing, please disclose it. +- **Neutrality:** If you have an affiliation related to what you are changing, please mention it. - Please do your best to review current issues and pull requests to avoid duplication. ### Writing Style @@ -28,7 +33,31 @@ If you’ve found this guide useful, you have many ways to help: - **Include opinions and common practice:** Thoughtful opinion is helpful. If there are multiple conventions or ideas on something held by experts, mention the different ones. - **Clarity:** Strive for consistency with conventions listed here, but clarity is most important. -### Writing Conventions +### Pull Request Review + +When creating a PR or reviewing one, it’s helpful to consider a few questions: + +1. Are relevant docs linked (AWS and third party blogs)? +2. Can more details be added without lengthening it significantly? +3. Can the same information be conveyed with more succinctly or more clearly? +4. Copy editing: Does it match the [writing style](#writing-style) and [conventions](#writing-conventions) (including emoji symbols, punctuation, and grammar)? +5. Are there any other parts of the guide that should be updated at the same time? (Examples are adding new sections to the table of contents, adding new internal links, rows to tables, etc.) +6. Are there additional improvements that should be filed as issues? + +### Editorial Process + +- Roles: + - **Project leads:** Own overall quality of the Guide, direction, and process. + - **Editors:** Contributors own specific sections or aspects of the Guide, reviewing PRs and/or writing. Requires expert knowledge. + - **Experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs. + - **Contributors:** Everyone who contributes content or helps one way or another. +- All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. +- In addition, anyone with relevant knowledge is encouraged to review/comment on PRs. +- Both editors and project leads are responsible for checking for style or problems. +- Trivial changes (including copy editing) may be merged in directly by project leads or editors. + +Writing Conventions +------------------- When you contribute, keep in mind these conventions: @@ -53,18 +82,7 @@ When you contribute, keep in mind these conventions: Note we keep consistent formatting in Markdown via [markdownfmt](https://github.com/shurcooL/markdownfmt). We run **admin/reformat.sh** to do this, but you don’t have to worry about it unless you really want to. -### Editorial Process - -- Roles: - - **Project leads:** Own overall quality of the Guide, direction, and process. - - **Editors:** Contributors own specific sections or aspects of the Guide, reviewing PRs and/or writing. requires expert knowledge. - - **Experts:** People with expert knowledge in various areas, who have agreed to review or help on demand with tougher questions or PRs. - - **Contributors:** Everyone who contributes content or helps one way or another. -- All PRs are reviewed by an **editor** and for non-trivial changes, a **project lead**, usually in that order, but it can be reversed for expediency. -- In addition, anyone with relevant knowledge is encouraged to review/comment on PRs. -- Both editors and project leads are responsible for checking for style or problems. -- Trivial changes (including copy editing) may be merged in directly by project leads or editors. - -### Contact +Contact +------- If you have concerns or additional ideas of ways to help, e-mail **og-aws@lexikon.io** or use Slack to contact the [project leads](AUTHORS.md). diff --git a/README.md b/README.md index 6500869..b5058f1 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ The Open Guide to Amazon Web Services ===================================== -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) ⇦ Join us! [Credits](AUTHORS.md) ∙ [Contributing guidelines](CONTRIBUTING.md) @@ -93,9 +93,9 @@ Before using the guide, please read the [**license**](#license) and [**disclaime **This is an early in-progress draft!** It’s our first attempt at assembling this information, so is far from comprehensive still, and likely to have omissions or errors. -[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) [![Ask a Question](https://img.shields.io/badge/%3f-Ask%20a%20Question-dc9d47.svg "Questions help improve the Guide")](https://airtable.com/shrXZ61VrovWfXYBg) +[![Slack Chat](https://img.shields.io/badge/Chat-Slack-ff69b4.svg "Join us. Anyone is welcome!")](https://og-aws.slack.lexikon.io/) -Please help by [**joining the Slack channel**](https://og-aws.slack.lexikon.io/) to talk about AWS (anyone is welcome, even if you only have questions!), [**submitting a question**](https://airtable.com/shrXZ61VrovWfXYBg), or [**contributing to the guide**](CONTRIBUTING.md). This guide is *open to contributions*, so unlike a blog, it can keep improving. Like any open source effort, we combine efforts but also review to ensure high quality. +Please help by [**joining the Slack channel**](https://og-aws.slack.lexikon.io/) (we like to talk about AWS in general, even if you only have questions — discussion helps the community and guides improvements) and [**contributing to the guide**](CONTRIBUTING.md). This guide is *open to contributions*, so unlike a blog, it can keep improving. Like any open source effort, we combine efforts but also review to ensure high quality. Scope ----- @@ -589,7 +589,7 @@ We cover security basics first, since configuring user accounts is something you - But be careful not to cache credentials for too long, as [they expire](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials). (Note the other [dynamic metadata](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#dynamic-data-categories) also changes over time and should not be cached a long time, either.) - 🔸Some IAM operations are slower than other API calls (many seconds), since AWS needs to propagate these globally across regions. - ❗The uptime of IAM’s API has historically been lower than that of the instance metadata API. Be wary of incorporating a dependency on IAM’s API into critical paths or subsystems — for example, if you validate a user’s IAM group membership when they log into an instance and aren’t careful about precaching group membership or maintaining a back door, you might end up locking users out altogether when the API isn’t available. -- ❗**Don't check in AWS credentials or secrets to a git repository.** There are bots that scan GitHub looking for credentials. Use scripts or tools, such as [git-secrets](https://github.com/awslabs/git-secrets) to prevent anyone on your team from checking in sensitive information to your git repositories. +- ❗**Don't check in AWS credentials or secrets to a git repository.** There are bots that scan GitHub looking for credentials. Use scripts or tools, such as [git-secrets](https://github.com/awslabs/git-secrets) to prevent anyone on your team from checking in sensitive information to your git repositories. S3 -- @@ -1002,7 +1002,7 @@ RDS - 🔸Verify what database features you need, as not everything you might want is available on RDS. For example, if you are using Postgres, check the list of [supported features and extensions](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#SQLServer.Concepts.General.FeatureSupport). If the features you need aren't supported by RDS, you'll have to deploy your database yourself. - If you use the failover support offered by RDS, keep in mind that it is based on DNS changes - make sure that your client reacts to these changes appropriately. This is particularly important for Java, given how its DNS resolver's TTL is [configured by default](http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html). - 🔸**DB migration to RDS:** While importing your database into RDS ensure you take into consideration the maintenance window settings. If a backup is running at the same time, your import can take a considerably longer time than you would have expected. -- [Database sizes are limited](https://aws.amazon.com/about-aws/whats-new/2015/06/amazon-rds-increases-storage-limits-to-6TB-for-piops-and-gp2/) to 6TB for all database engines except for SQL Server which has a 4TB limit and Aurora which supports up to 64TB databases. +- [Database sizes are limited](https://aws.amazon.com/about-aws/whats-new/2015/06/amazon-rds-increases-storage-limits-to-6TB-for-piops-and-gp2/) to 6TB for all database engines except for SQL Server which has a 4TB limit and Aurora which supports up to 64TB databases. RDS MySQL and MariaDB --------------------- @@ -1016,8 +1016,8 @@ RDS MySQL and MariaDB - 🔸**MySQL vs MariaDB vs Aurora:** If you prefer a MySQL-style database but are starting something new, you probably should consider Aurora and MariaDB as well. **Aurora** has increased availability and is the next-generation solution. That said, Aurora [may not be](http://blog.takipi.com/benchmarking-aurora-vs-mysql-is-amazons-new-db-really-5x-faster/) as fast relative to MySQL as is sometimes reported, and is more complex to administer. **MariaDB**, the modern [community fork](https://en.wikipedia.org/wiki/MariaDB) of MySQL, [likely now has the edge over MySQL](http://cloudacademy.com/blog/mariadb-vs-mysql-aws-rds/) for many purposes and is supported by RDS. ### RDS MySQL and MariaDB Gotchas and Limitations -- 🔸**No SUPER privileges.** RDS provides some [stored procedures](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.SQLRef.html) to perform some tasks that require SUPER privileges such as starting or stopping replication. -- 🔸You can replicate to non-RDS instances of MySQL, but [replication to these instances will break during AZ [failovers]https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls). +- 🔸**No SUPER privileges.** RDS provides some [stored procedures](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.SQLRef.html) to perform some tasks that require SUPER privileges such as starting or stopping replication. +- 🔸You can replicate to non-RDS instances of MySQL, but [replication to these instances will break during AZ [failovers]https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls). - 🔸There is no ability to manually CHANGE MASTER on replicas, so they must all be rebuilt after a failover of the master. RDS Aurora @@ -1034,19 +1034,19 @@ RDS Aurora ### RDS Aurora Tips - In order to take advantage of Aurora's higher concurrency, applications should be configured with large database connection pools and should execute as many queries concurrently as possible. For example, Aurora servers have been tested to produce increasing performance on some OLTP workloads with [up to 5,000 connections](http://www.slideshare.net/AmazonWebServices/amazon-aurora-amazons-new-relational-database-engine/31). - [Aurora scales well with multiple CPUs](https://www.percona.com/blog/2016/05/26/aws-aurora-benchmarking-part-2/) and may require a large instance class for optimal performance. -- Because Aurora is based on MySQL 5.6.10, avoiding any MySQL features from 5.7 or later will ease the transition from a MySQL-compatible database into Aurora. +- Because Aurora is based on MySQL 5.6.10, avoiding any MySQL features from 5.7 or later will ease the transition from a MySQL-compatible database into Aurora. - The easiest migration path to Aurora is restoring a database snapshot from MySQL 5.6. The next easiest method is restoring a dump from a MySQL-compatible database such as MariaDB. For [low-downtime migrations](http://cantrill.io/howto/aws/2016/06/06/migrating-from-mysql-to-aurora-with-almost-no-downtime.html) from other MySQL-compatible databases, you can set up an Aurora instance as a replica of your existing database. If none of those methods are options, Amazon offers a fee-based data migration service. - You can replicate [from an Aurora cluster to MySQL or to another Aurora cluster](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html). This requires binary logging to be enabled and is not as performant as native Aurora replication. ### RDS Aurora Gotchas and Limitations -- 🔸[Aurora is based on MySQL 5.6.10](https://news.ycombinator.com/item?id=12415693) with some cherry-picking of later MySQL features. It is missing most 5.7 features as well as some online DDL features introduced in 5.6.17. +- 🔸[Aurora is based on MySQL 5.6.10](https://news.ycombinator.com/item?id=12415693) with some cherry-picking of later MySQL features. It is missing most 5.7 features as well as some online DDL features introduced in 5.6.17. RDS SQL Server -------------- ### RDS SQL Server Basics -- [RDS offers SQL Server 2008 R2, 2012, and 2014](https://aws.amazon.com/rds/sqlserver/) including Express, Web, Standard and Enterprise (2008 R2 and 2012 only for Enterprise) - +- [RDS offers SQL Server 2008 R2, 2012, and 2014](https://aws.amazon.com/rds/sqlserver/) including Express, Web, Standard and Enterprise (2008 R2 and 2012 only for Enterprise) + ### RDS SQL Server Tips - Recently added support for [backup and restore to/from S3](https://www.brentozar.com/archive/2016/07/holy-cow-amazon-rds-sql-server-just-changed-everything/) which may make it an attractive DR option for on-premises installations. @@ -1364,7 +1364,7 @@ Redshift - ❗ Never resize a live cluster. The resize operation takes hours depending on the dataset size. In rare cases, the operation may also get stuck and you'll end up having a non-functional cluster. The safer approach is to create a new cluster from a snapshot, resize the new cluster and shut down the old one. - Redshift has reserved keywords which are not present in Postgres (see full list [here](https://docs.aws.amazon.com/redshift/latest/dg/r_pg_keywords.html)). Watch out for DELTA ([Delta Encodings](https://docs.aws.amazon.com/redshift/latest/dg/c_Delta_encoding.html)). - Redshift does not support many Postgres functions, most notably several date/time-related and aggregation functions. See the [full list here](https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html). -- 🔹 [Choosing a sort key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change a table’s sort key after it is created. If you need to change the sort or distribution key of a table, you need to create a new table with the new key and move your data into it with a query like “insert into new_table select * from old_table”. +- 🔹 [Choosing a sort key](http://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) is very important since you can not change a table’s sort key after it is created. If you need to change the sort or distribution key of a table, you need to create a new table with the new key and move your data into it with a query like “insert into new_table select * from old_table”. - ❗🚪 When moving data with a query that looks like “insert into x select from y”, you need to have twice as much disk space available as table “y” takes up on the cluster’s disks. Redshift first copies the data to disk and then to the new table. [Here](https://www.periscopedata.com/blog/changing-dist-and-sort-keys-in-redshift.html) is a good article on how to this for big tables. EMR From 269c720949beba0c1798d4a047337404afbb49fb Mon Sep 17 00:00:00 2001 From: danhermann Date: Tue, 25 Oct 2016 06:48:34 -0500 Subject: [PATCH 138/140] fix broken link on RDS MySQL replication tip --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b5058f1..40b486e 100644 --- a/README.md +++ b/README.md @@ -1017,7 +1017,7 @@ RDS MySQL and MariaDB ### RDS MySQL and MariaDB Gotchas and Limitations - 🔸**No SUPER privileges.** RDS provides some [stored procedures](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.SQLRef.html) to perform some tasks that require SUPER privileges such as starting or stopping replication. -- 🔸You can replicate to non-RDS instances of MySQL, but [replication to these instances will break during AZ [failovers]https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls). +- 🔸You can replicate to non-RDS instances of MySQL, but [replication to these instances will break during AZ failovers](https://www.percona.com/live/mysql-conference-2014/sessions/rds-mysql-tips-patterns-and-common-pitfalls). - 🔸There is no ability to manually CHANGE MASTER on replicas, so they must all be rebuilt after a failover of the master. RDS Aurora From 4369600f374eceaea4e3ea58c2a75afe9cef3fb0 Mon Sep 17 00:00:00 2001 From: max Date: Tue, 25 Oct 2016 10:13:52 -0700 Subject: [PATCH 139/140] Fixed #258: Tip: Always use UTC on EC2 instances. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index b5058f1..7b63f1a 100644 --- a/README.md +++ b/README.md @@ -519,6 +519,7 @@ This guide is about AWS, not DevOps or server configuration management in genera ### Tips +- Unless *absolutely necessary*, always **set the timezone on servers to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time)**. Numerous distributed systems rely on time for synchronization and coordination and UTC provides the universal reference plane: it is not subject to daylight savings changes and adjustments in local time. It will also save you a lot of headache debugging [elusive timezone issues](http://yellerapp.com/posts/2015-01-12-the-worst-server-setup-you-can-make.html) and provide coherent timeline of events in your logging and audit systems. - **NTP and accurate time:** If you are not using Amazon Linux (which comes preconfigured), you should confirm your servers [configure NTP correctly](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure_ntp), to avoid insidious time drift (which can then cause all sorts of issues, from breaking API calls to misleading logs). This should be part of your automatic configuration for every server. If time has already drifted substantially (generally >1000 seconds), remember NTP won’t shift it back, so you may need to remediate manually (for example, [like this](http://askubuntu.com/questions/254826/how-to-force-a-clock-update-using-ntp) on Ubuntu). - **Testing immutable infrastructure:** If you want to be proactive about testing your service’s ability to cope with instance termination or failure, it can be helpful to introduce random instance termination during business hours, which will expose any such issues at a time when engineers are available to identify and fix them. Netflix’s [Simian Army](https://github.com/Netflix/SimianArmy) (specifically, [Chaos Monkey](https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey)) is a popular tool for this. Alternatively, [chaos-lambda](https://github.com/bbc/chaos-lambda) by the BBC is a lightweight option which runs on AWS [Lambda](#lambda). From 17c51a6decfe724f3f7cf3446aae86eda1bcda40 Mon Sep 17 00:00:00 2001 From: max Date: Tue, 25 Oct 2016 10:45:28 -0700 Subject: [PATCH 140/140] Addressed the review comments on #258. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7b63f1a..3582193 100644 --- a/README.md +++ b/README.md @@ -519,7 +519,7 @@ This guide is about AWS, not DevOps or server configuration management in genera ### Tips -- Unless *absolutely necessary*, always **set the timezone on servers to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time)**. Numerous distributed systems rely on time for synchronization and coordination and UTC provides the universal reference plane: it is not subject to daylight savings changes and adjustments in local time. It will also save you a lot of headache debugging [elusive timezone issues](http://yellerapp.com/posts/2015-01-12-the-worst-server-setup-you-can-make.html) and provide coherent timeline of events in your logging and audit systems. +- ❗**Timezone settings on servers**: unless *absolutely necessary*, always **set the timezone on servers to [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time)** (see instructions for your distribution, such as [Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-set-up-timezone-and-ntp-synchronization-on-ubuntu-14-04-quickstart), [CentOS](https://www.vultr.com/docs/setup-timezone-and-ntp-on-centos-6) or [Amazon](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) Linux). Numerous distributed systems rely on time for synchronization and coordination and UTC [provides](https://blog.serverdensity.com/set-your-server-timezone-to-utc/) the universal reference plane: it is not subject to daylight savings changes and adjustments in local time. It will also save you a lot of headache debugging [elusive timezone issues](http://yellerapp.com/posts/2015-01-12-the-worst-server-setup-you-can-make.html) and provide coherent timeline of events in your logging and audit systems. - **NTP and accurate time:** If you are not using Amazon Linux (which comes preconfigured), you should confirm your servers [configure NTP correctly](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure_ntp), to avoid insidious time drift (which can then cause all sorts of issues, from breaking API calls to misleading logs). This should be part of your automatic configuration for every server. If time has already drifted substantially (generally >1000 seconds), remember NTP won’t shift it back, so you may need to remediate manually (for example, [like this](http://askubuntu.com/questions/254826/how-to-force-a-clock-update-using-ntp) on Ubuntu). - **Testing immutable infrastructure:** If you want to be proactive about testing your service’s ability to cope with instance termination or failure, it can be helpful to introduce random instance termination during business hours, which will expose any such issues at a time when engineers are available to identify and fix them. Netflix’s [Simian Army](https://github.com/Netflix/SimianArmy) (specifically, [Chaos Monkey](https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey)) is a popular tool for this. Alternatively, [chaos-lambda](https://github.com/bbc/chaos-lambda) by the BBC is a lightweight option which runs on AWS [Lambda](#lambda).