mirror of
https://github.com/nickpoida/og-aws.git
synced 2025-03-09 15:40:06 +00:00
Merge remote-tracking branch 'upstream/master'
This commit is contained in:
commit
7e28ba21a4
2 changed files with 25 additions and 11 deletions
24
AUTHORS.md
24
AUTHORS.md
|
@ -7,28 +7,42 @@ The following people (in alphabetical order) have contributed to or reviewed thi
|
||||||
|
|
||||||
* [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah)
|
* [Alexander Atallah (alexanderatallah)](https://github.com/alexanderatallah)
|
||||||
* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [2+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin)
|
* [Artem Nikitin (artemnikitin)](https://github.com/artemnikitin) — [2+](https://github.com/open-guides/og-aws/commits?author=artemnikitin)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aartemnikitin)
|
||||||
|
* [Benjamin Bunk (benbunk)](https://github.com/benbunk) — [1+](https://github.com/open-guides/og-aws/commits?author=benbunk)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenbunk)
|
||||||
* [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe)
|
* [Ben Kehoe (benkehoe)](https://github.com/benkehoe) — [4+](https://github.com/open-guides/og-aws/commits?author=benkehoe)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Abenkehoe)
|
||||||
* [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer)
|
* [Adam Mathias Bittlingmayer (bittlingmayer)](https://github.com/bittlingmayer)
|
||||||
|
* [chris-griffin](https://github.com/chris-griffin) — [1+](https://github.com/open-guides/og-aws/commits?author=chris-griffin)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Achris-griffin)
|
||||||
|
* [danhermann](https://github.com/danhermann) — [1+](https://github.com/open-guides/og-aws/commits?author=danhermann)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Adanhermann)
|
||||||
* [Donne Martin (donnemartin)](https://github.com/donnemartin)
|
* [Donne Martin (donnemartin)](https://github.com/donnemartin)
|
||||||
|
* [Patrick McDavid (ehippy)](https://github.com/ehippy) — [1+](https://github.com/open-guides/og-aws/commits?author=ehippy)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aehippy)
|
||||||
* [Max Grigorev (forwidur)](https://github.com/forwidur)
|
* [Max Grigorev (forwidur)](https://github.com/forwidur)
|
||||||
|
* [Glynn Forrest (glynnforrest)](https://github.com/glynnforrest) — [1+](https://github.com/open-guides/og-aws/commits?author=glynnforrest)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Aglynnforrest)
|
||||||
* [Dmitry Golyshev (golyshev)](https://github.com/golyshev)
|
* [Dmitry Golyshev (golyshev)](https://github.com/golyshev)
|
||||||
* [Joshua Levy (jlevy)](https://github.com/jlevy) — [77+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[67+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_
|
* [Gulam Shakir (gshakir)](https://github.com/gshakir) — [2+](https://github.com/open-guides/og-aws/commits?author=gshakir)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Agshakir)
|
||||||
|
* [Itay Shakury (itaysk)](https://github.com/itaysk) — [1+](https://github.com/open-guides/og-aws/commits?author=itaysk)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Aitaysk)
|
||||||
|
* [jbao](https://github.com/jbao) — [1+](https://github.com/open-guides/og-aws/commits?author=jbao)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ajbao)
|
||||||
|
* [Joshua Levy (jlevy)](https://github.com/jlevy) — [86+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[76+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_
|
||||||
* Jurgen Philippaerts
|
* Jurgen Philippaerts
|
||||||
* [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura)
|
* [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura)
|
||||||
* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [2+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit)
|
* [Lynn Langit (lynnlangit)](https://github.com/lynnlangit) — [3+](https://github.com/open-guides/og-aws/commits?author=lynnlangit)/[2+](https://github.com/open-guides/og-aws/issues?q=author%3Alynnlangit)
|
||||||
|
* [maiki](https://github.com/maiki) — [1+](https://github.com/open-guides/og-aws/commits?author=maiki)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amaiki)
|
||||||
* [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d)
|
* [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d)
|
||||||
* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [0+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (S3, EMR, Redshift)_
|
* [Max Zanko (max-zanko)](https://github.com/max-zanko) — [3+](https://github.com/open-guides/og-aws/commits?author=max-zanko)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Amax-zanko) — _editor (S3, EMR, Redshift)_
|
||||||
* [John Merrells (merrells)](https://github.com/merrells)
|
* [John Merrells (merrells)](https://github.com/merrells)
|
||||||
|
* [Magnus Kulke (mkulke)](https://github.com/mkulke) — [4+](https://github.com/open-guides/og-aws/commits?author=mkulke)/[3+](https://github.com/open-guides/og-aws/issues?q=author%3Amkulke)
|
||||||
* [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_
|
* [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_
|
||||||
* [Ola Wiberg (olawiberg)](https://github.com/olawiberg)
|
* [Ola Wiberg (olawiberg)](https://github.com/olawiberg)
|
||||||
* Praveen Patnala
|
* Praveen Patnala
|
||||||
* [Russell Power (rjpower)](https://github.com/rjpower)
|
* [Russell Power (rjpower)](https://github.com/rjpower)
|
||||||
* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [10+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[10+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_
|
* [David Schott (shott85)](https://github.com/shott85) — [1+](https://github.com/open-guides/og-aws/commits?author=shott85)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Ashott85)
|
||||||
|
* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [12+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[12+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_
|
||||||
|
* [Tom Schlick (tomschlick)](https://github.com/tomschlick) — [3+](https://github.com/open-guides/og-aws/commits?author=tomschlick)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atomschlick)
|
||||||
|
* [Trayton White (traytonwhite)](https://github.com/traytonwhite) — [1+](https://github.com/open-guides/og-aws/commits?author=traytonwhite)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Atraytonwhite)
|
||||||
* [Stefan Zier (weirded)](https://github.com/weirded)
|
* [Stefan Zier (weirded)](https://github.com/weirded)
|
||||||
|
* [Michael Ortali (xethorn)](https://github.com/xethorn) — [1+](https://github.com/open-guides/og-aws/commits?author=xethorn)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Axethorn)
|
||||||
|
|
||||||
Additional authors are welcome; see the [contribution guidelines](CONTRIBUTING.md).
|
Additional authors are welcome; see the [contribution guidelines](CONTRIBUTING.md).
|
||||||
Please let the editors know of any errors or omissions on this list.
|
Please let the editors know of any errors or omissions on this list.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).)
|
(This file was auto-generated by [ghizmo assemble-authors](https://github.com/jlevy/ghizmo).)
|
||||||
|
|
12
README.md
12
README.md
|
@ -146,7 +146,7 @@ General Information
|
||||||
- Companies at (very) large scale may want to reduce costs by managing their own infrastructure. For example, [Dropbox migrated](https://news.ycombinator.com/item?id=11282948) to their own infrastructure.
|
- Companies at (very) large scale may want to reduce costs by managing their own infrastructure. For example, [Dropbox migrated](https://news.ycombinator.com/item?id=11282948) to their own infrastructure.
|
||||||
- Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys.
|
- Other cloud providers such as [Digital Ocean](https://www.digitalocean.com/) offer similar services, sometimes with greater ease of use, more personalized support, or lower cost. However, none of these match the breadth of products, mind-share, and market domination AWS now enjoys.
|
||||||
- Traditional managed hosting providers such as [Rackspace](https://www.rackspace.com/) offer cloud solutions as well.
|
- Traditional managed hosting providers such as [Rackspace](https://www.rackspace.com/) offer cloud solutions as well.
|
||||||
- 🚪**AWS vs. PaaS:** If your goal is just to put up a single service that does something relatively simple, and you’re trying to minimize time managing operations engineering, consider a [platform-as-a-service](https://en.wikipedia.org/wiki/Platform_as_a_service) such as [Heroku](https://www.heroku.com/) The AWS approach to PaaS, Elastic Beanstalk, is arguably more complex, especially for simple use cases.
|
- 🚪**AWS vs. PaaS:** If your goal is just to put up a single service that does something relatively simple, and you’re trying to minimize time managing operations engineering, consider a [platform-as-a-service](https://en.wikipedia.org/wiki/Platform_as_a_service) such as [Heroku](https://www.heroku.com/). The AWS approach to PaaS, Elastic Beanstalk, is arguably more complex, especially for simple use cases.
|
||||||
- 🚪**AWS vs. web hosting:** If your main goal is to host a website or blog, and you don’t expect to be building an app or more complex service, you may wish consider one of the myriad of [web hosting services](https://www.google.com/search?q=web+hosting).
|
- 🚪**AWS vs. web hosting:** If your main goal is to host a website or blog, and you don’t expect to be building an app or more complex service, you may wish consider one of the myriad of [web hosting services](https://www.google.com/search?q=web+hosting).
|
||||||
- 🚪**AWS vs. managed hosting:** Traditionally, many companies pay [managed hosting](https://en.wikipedia.org/wiki/Dedicated_hosting_service) providers to maintain physical servers for them, then build and deploy their software on top of the rented hardware. This makes sense for businesses who want direct control over hardware, due to legacy, performance, or special compliance constraints, but is usually considered old fashioned or unnecessary by many developer-centric startups and younger tech companies.
|
- 🚪**AWS vs. managed hosting:** Traditionally, many companies pay [managed hosting](https://en.wikipedia.org/wiki/Dedicated_hosting_service) providers to maintain physical servers for them, then build and deploy their software on top of the rented hardware. This makes sense for businesses who want direct control over hardware, due to legacy, performance, or special compliance constraints, but is usually considered old fashioned or unnecessary by many developer-centric startups and younger tech companies.
|
||||||
- **Complexity:** AWS will let you build and scale systems to the size of the largest companies, but the complexity of the services when used at scale requires significant depth of knowledge and experience. Even very simple use cases often require more knowledge to do “right” in AWS than in a simpler environment like Heroku or Digital Ocean. (This guide may help!)
|
- **Complexity:** AWS will let you build and scale systems to the size of the largest companies, but the complexity of the services when used at scale requires significant depth of knowledge and experience. Even very simple use cases often require more knowledge to do “right” in AWS than in a simpler environment like Heroku or Digital Ocean. (This guide may help!)
|
||||||
|
@ -186,7 +186,7 @@ General Information
|
||||||
- [EMR](#emr): Managed Hadoop
|
- [EMR](#emr): Managed Hadoop
|
||||||
- [Elasticsearch](https://aws.amazon.com/elasticsearch-service/): Managed Elasticsearch
|
- [Elasticsearch](https://aws.amazon.com/elasticsearch-service/): Managed Elasticsearch
|
||||||
- [ElastiCache](https://aws.amazon.com/elasticache/): Managed Redis and Memcached
|
- [ElastiCache](https://aws.amazon.com/elasticache/): Managed Redis and Memcached
|
||||||
- **Optional but important infrastructure:** These are key and useful infrastructure components that are less widely known and used. You may have legitimate reasons to prefer alternatives, so evaluate with care you to be sure they fit your needs:
|
- **Optional but important infrastructure:** These are key and useful infrastructure components that are less widely known and used. You may have legitimate reasons to prefer alternatives, so evaluate with care to be sure they fit your needs:
|
||||||
- ⛓[Lambda](#lambda): Running small, fully managed tasks “serverless”
|
- ⛓[Lambda](#lambda): Running small, fully managed tasks “serverless”
|
||||||
- [CloudTrail](https://aws.amazon.com/cloudtrail/): AWS API logging and audit (often neglected but important)
|
- [CloudTrail](https://aws.amazon.com/cloudtrail/): AWS API logging and audit (often neglected but important)
|
||||||
- ⛓🕍[CloudFormation](#cloudformation): Templatized configuration of collections of AWS resources
|
- ⛓🕍[CloudFormation](#cloudformation): Templatized configuration of collections of AWS resources
|
||||||
|
@ -418,7 +418,7 @@ There are several approaches to deploying infrastructure with AWS, from the cons
|
||||||
|
|
||||||
The first way most people experiment with AWS is via its web interface, the AWS Console. But using the Console is a highly manual process, and often works against automation or flexibility.
|
The first way most people experiment with AWS is via its web interface, the AWS Console. But using the Console is a highly manual process, and often works against automation or flexibility.
|
||||||
|
|
||||||
So if you’re not going to manage your AWS configurations manually, what should you do? Sadly, there are no simple, universal answers — each approach has pros and cons, and the approaches taken by different companies vary widely, and include directly using APIs (and building toolign on top yourself), using command-line tools, and using third-party tools and services.
|
So if you’re not going to manage your AWS configurations manually, what should you do? Sadly, there are no simple, universal answers — each approach has pros and cons, and the approaches taken by different companies vary widely, and include directly using APIs (and building tooling on top yourself), using command-line tools, and using third-party tools and services.
|
||||||
|
|
||||||
### AWS Console
|
### AWS Console
|
||||||
|
|
||||||
|
@ -453,7 +453,7 @@ So if you’re not going to manage your AWS configurations manually, what should
|
||||||
|
|
||||||
### General Visibility
|
### General Visibility
|
||||||
|
|
||||||
- 🔹[**Tagging resources**](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) is an essential practice, especially as organizations grow, to better understand your resource usage. For example, you can through automation or convention add tags:
|
- 🔹[**Tagging resources**](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) is an essential practice, especially as organizations grow, to better understand your resource usage. For example, through automation or convention, you can add tags:
|
||||||
- For the org or developer that “owns” that resource
|
- For the org or developer that “owns” that resource
|
||||||
- For the product that resource supports
|
- For the product that resource supports
|
||||||
- To label lifecycles, such as temporary resources or one that should be deprovisioned in the future
|
- To label lifecycles, such as temporary resources or one that should be deprovisioned in the future
|
||||||
|
@ -663,7 +663,7 @@ S3
|
||||||
- 🔸Be careful not to make implicit assumptions about transactionality or sequencing of updates to objects. Never assume that if you modify a sequence of objects, the clients will see the same modifications in the same sequence, or if you upload a whole bunch of files, that they will all appear at once to all clients.
|
- 🔸Be careful not to make implicit assumptions about transactionality or sequencing of updates to objects. Never assume that if you modify a sequence of objects, the clients will see the same modifications in the same sequence, or if you upload a whole bunch of files, that they will all appear at once to all clients.
|
||||||
- 🔸S3 has an [**SLA**](https://aws.amazon.com/s3/sla/) with 99.9% uptime. If you use S3 heavily, you’ll inevitably see occasional error accessing or storing data as disks or other infrastructure fail. Availability is usually restored in seconds or minutes. Although availability is not extremely high, as mentioned above, durability is excellent.
|
- 🔸S3 has an [**SLA**](https://aws.amazon.com/s3/sla/) with 99.9% uptime. If you use S3 heavily, you’ll inevitably see occasional error accessing or storing data as disks or other infrastructure fail. Availability is usually restored in seconds or minutes. Although availability is not extremely high, as mentioned above, durability is excellent.
|
||||||
- 🔸After uploading, any change that you make to the object causes a full rewrite of the object, so avoid appending-like behavior with regular files.
|
- 🔸After uploading, any change that you make to the object causes a full rewrite of the object, so avoid appending-like behavior with regular files.
|
||||||
- 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 at suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours.
|
- 🔸Eventual data consistency, as discussed above, can be surprising sometimes. If S3 suffers from internal replication issues, an object may be visible from a subset of the machines, depending on which S3 endpoint they hit. Those usually resolve within seconds; however, we’ve seen isolated cases when the issue lingered for 20-30 hours.
|
||||||
- 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)).
|
- 🔸**MD5s and multi-part uploads:** In S3, the [ETag header in S3](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) is a hash on the object. And in many cases, it is the MD5 hash. However, this [is not the case in general](http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb) when you use multi-part uploads. One workaround is to compute MD5s yourself and put them in a custom header (such as is done by [s4cmd](https://github.com/bloomreach/s4cmd)).
|
||||||
- 🔸**US Standard region:** Most S3 endpoints match the region they’re in, with the exception of the us-east-1 region, which is called 'us-standard' in S3 terminology. This region is also the only region that is replicated across coasts. As a result, latency varies more in this region than in others. You can minimize latency from us-east-1 by using *[s3-external-1.amazonaws.com](http://s3-external-1.amazonaws.com/)*.
|
- 🔸**US Standard region:** Most S3 endpoints match the region they’re in, with the exception of the us-east-1 region, which is called 'us-standard' in S3 terminology. This region is also the only region that is replicated across coasts. As a result, latency varies more in this region than in others. You can minimize latency from us-east-1 by using *[s3-external-1.amazonaws.com](http://s3-external-1.amazonaws.com/)*.
|
||||||
|
|
||||||
|
@ -846,7 +846,7 @@ Load Balancers
|
||||||
- **CLBs and ALBs have many IPs:** Internally, an AWS load balancer is simply a collection of individual software load balancers hosted within EC2, with DNS load balancing traffic among them. The pool can contain many IPs, at least one per availability zone, and depending on traffic levels. They also support SSL termination, which is very convenient.
|
- **CLBs and ALBs have many IPs:** Internally, an AWS load balancer is simply a collection of individual software load balancers hosted within EC2, with DNS load balancing traffic among them. The pool can contain many IPs, at least one per availability zone, and depending on traffic levels. They also support SSL termination, which is very convenient.
|
||||||
- **Scaling:** CLBs and ALBs can scale to very high throughput, but scaling up is not instantaneous. If you’re expecting to be hit with a lot of traffic suddenly, it can make sense to load test them so they scale up in advance. You can also [contact Amazon](http://aws.amazon.com/articles/1636185810492479) and have them “pre-warm” the load balancer.
|
- **Scaling:** CLBs and ALBs can scale to very high throughput, but scaling up is not instantaneous. If you’re expecting to be hit with a lot of traffic suddenly, it can make sense to load test them so they scale up in advance. You can also [contact Amazon](http://aws.amazon.com/articles/1636185810492479) and have them “pre-warm” the load balancer.
|
||||||
- **Client IPs:** In general, if servers want to know true client IP addresses, load balancers must forward this information somehow. CLBs add the standard [X-Forwarded-For](https://en.wikipedia.org/wiki/X-Forwarded-For) header. When using an CLB as an HTTP load balancer, it’s possible to get the client’s IP address from this.
|
- **Client IPs:** In general, if servers want to know true client IP addresses, load balancers must forward this information somehow. CLBs add the standard [X-Forwarded-For](https://en.wikipedia.org/wiki/X-Forwarded-For) header. When using an CLB as an HTTP load balancer, it’s possible to get the client’s IP address from this.
|
||||||
- **Using load balancers when deploying:** One common pattern is to swap instances in the load balancer after spinning up a new stack with your latest version, keep old stack running for one or two hours, and either flip back to old stack in case of problems or tear down it down.
|
- **Using load balancers when deploying:** One common pattern is to swap instances in the load balancer after spinning up a new stack with your latest version, keep old stack running for one or two hours, and either flip back to old stack in case of problems or tear it down.
|
||||||
|
|
||||||
### Load Balancer Gotchas and Limitations
|
### Load Balancer Gotchas and Limitations
|
||||||
|
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue