mirror of
https://github.com/nickpoida/og-aws.git
synced 2025-02-15 03:11:57 +00:00
Merge pull request #82 from open-guides/more-stuff
Images and authors update
This commit is contained in:
Thanos Baskous
GitHub
commit
9c45176f77
7 changed files with 40 additions and 36 deletions
|
|
@ -11,17 +11,17 @@ The following people (in alphabetical order) have contributed to or reviewed thi
|
|||
* [Donne Martin (donnemartin)](https://github.com/donnemartin)
|
||||
* [Max Grigorev (forwidur)](https://github.com/forwidur)
|
||||
* [Dmitry Golyshev (golyshev)](https://github.com/golyshev)
|
||||
* [Joshua Levy (jlevy)](https://github.com/jlevy) — [46+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[59+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_
|
||||
* [Joshua Levy (jlevy)](https://github.com/jlevy) — [67+](https://github.com/open-guides/og-aws/commits?author=jlevy)/[66+](https://github.com/open-guides/og-aws/issues?q=author%3Ajlevy) — _general editor_
|
||||
* Jurgen Philippaerts
|
||||
* [KAZUYUKI TANIMURA (kazuyukitanimura)](https://github.com/kazuyukitanimura)
|
||||
* [Marcello Bastéa-Forte (marcello3d)](https://github.com/marcello3d)
|
||||
* [Max Zanko (max-zanko)](https://github.com/max-zanko)
|
||||
* [Max Zanko (max-zanko)](https://github.com/max-zanko) — _editor (S3, EMR, Redshift)_
|
||||
* [John Merrells (merrells)](https://github.com/merrells)
|
||||
* [Nitin Sharma (nitingithub)](https://github.com/nitingithub) — [1+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[1+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_
|
||||
* [Nitin S (nitingithub)](https://github.com/nitingithub) — [6+](https://github.com/open-guides/og-aws/commits?author=nitingithub)/[4+](https://github.com/open-guides/og-aws/issues?q=author%3Anitingithub) — _editor (cost management)_
|
||||
* [Ola Wiberg (olawiberg)](https://github.com/olawiberg)
|
||||
* Praveen Patnala
|
||||
* [Russell Power (rjpower)](https://github.com/rjpower)
|
||||
* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [8+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[5+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_
|
||||
* [Thanos Baskous (ThanosBaskous)](https://github.com/ThanosBaskous) — [9+](https://github.com/open-guides/og-aws/commits?author=ThanosBaskous)/[8+](https://github.com/open-guides/og-aws/issues?q=author%3AThanosBaskous) — _general editor_
|
||||
* [Stefan Zier (weirded)](https://github.com/weirded)
|
||||
|
||||
Additional authors are welcome; see the [contribution guidelines](CONTRIBUTING.md).
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ Contributing
|
|||
|
||||
Contributions of all kinds, including discussion, corrections, additions, and improvements, are welcome! We hope you'll join and help, in small ways or large. We gladly credit all contributors and authors. Here are few notes before you jump in.
|
||||
|
||||
[](https://og-aws.slack.lexikon.io/) [](https://airtable.com/shrXZ61VrovWfXYBg)
|
||||
[](https://og-aws.slack.lexikon.io/) [](https://airtable.com/shrXZ61VrovWfXYBg)
|
||||
|
||||
The simplest thing you can do to contribute is [fill in this form](https://airtable.com/shrXZ61VrovWfXYBg). Questions help identify where the guide needs to be improved.
|
||||
|
||||
|
|
|
|||
62
README.md
62
README.md
|
|
@ -1,7 +1,9 @@
|
|||
|
||||
|
||||
The Open Guide to Amazon Web Services
|
||||
=====================================
|
||||
|
||||
[](https://og-aws.slack.lexikon.io/) [](https://airtable.com/shrXZ61VrovWfXYBg)
|
||||
[](https://og-aws.slack.lexikon.io/) [](https://airtable.com/shrXZ61VrovWfXYBg)
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
|
|
@ -46,11 +48,13 @@ The Open Guide to Amazon Web Services
|
|||
|
||||
**Figures and Tables**
|
||||
|
||||
[](#tools-and-services-market-landscape) [](#aws-data-transfer-costs)
|
||||
|
||||
- [Figure: Tools and Services Market Landscape](#tools-and-services-market-landscape): A selection of third-party companies/products
|
||||
- [Figure: AWS Data Transfer Costs](#aws-data-transfer-costs): Visual overview of data transfer costs
|
||||
- [Table: Service Matrix](#service-matrix): How AWS services compare to alternatives
|
||||
- [Table: AWS Product Maturity and Releases](#aws-product-maturity-and-releases): AWS product releases
|
||||
- [Table: Storage Durability, Availability, and Price](#storage-durability-availability-and-price): A quantitative comparison
|
||||
- [Figure: AWS Data Transfer Costs](#aws-data-transfer-costs): Visual overview of data transfer costs
|
||||
|
||||
Why an Open Guide?
|
||||
------------------
|
||||
|
|
@ -65,7 +69,7 @@ Before using the guide, please read the [**license**](#license) and [**disclaime
|
|||
|
||||
**This is an early in-progress draft!** It’s our first attempt at assembling this information, so is far from comprehensive still, and likely to have omissions or errors.
|
||||
|
||||
[](https://og-aws.slack.lexikon.io/) [](https://airtable.com/shrXZ61VrovWfXYBg)
|
||||
[](https://og-aws.slack.lexikon.io/) [](https://airtable.com/shrXZ61VrovWfXYBg)
|
||||
|
||||
Please help by [**joining the Slack channel**](https://og-aws.slack.lexikon.io/) to talk about AWS (anyone is welcome, even if you only have questions!), [**submitting a question**](https://airtable.com/shrXZ61VrovWfXYBg), or [**contributing to the guide**](CONTRIBUTING.md). This guide is *open to contributions*, so unlike a blog, it can keep improving. Like any open source effort, we combine efforts but also review to ensure high quality.
|
||||
|
||||
|
|
@ -539,8 +543,8 @@ S3
|
|||
- Items, or **objects**, are placed into named **buckets** stored with names which are usually called **keys**. The main content is the **value**.
|
||||
- Objects are created, deleted, or updated. Large objects can be streamed, but you cannot access or modify parts of a value; you need to update the whole object.
|
||||
- Every object also has [**metadata**](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html), which includes arbitrary key-value pairs, and is used in a way similar to HTTP headers. Some metadata is system-defined, some are significant when serving HTTP content from buckets or CloudFront, and you can also define arbitrary metadata for your own use.
|
||||
- Although often bucket and key names are provided in APIs individually, it’s also common practice to write an S3 location in the form 's3://bucket-name/path/to/key' (where the key here is 'path/to/key'). (You’ll also see 's3n://' and 's3a://' prefixes [in Hadoop systems](https://wiki.apache.org/hadoop/AmazonS3).)
|
||||
- AWS offers many storage services, and several besides S3 offer file-type abstractions. [Glacier](#glacier) is for cheaper and infrequently accessed archival storage. [EBS](#ebs), unlike S3, allows random access to file contents via a traditional filesystem, but can only be attached to one EC2 instance at a time.
|
||||
- **S3 URIs:** Although often bucket and key names are provided in APIs individually, it’s also common practice to write an S3 location in the form 's3://bucket-name/path/to/key' (where the key here is 'path/to/key'). (You’ll also see 's3n://' and 's3a://' prefixes [in Hadoop systems](https://wiki.apache.org/hadoop/AmazonS3).)
|
||||
- **S3 vs Glacier, EBS, and EFS:** AWS offers many storage services, and several besides S3 offer file-type abstractions. [Glacier](#glacier) is for cheaper and infrequently accessed archival storage. [EBS](#ebs), unlike S3, allows random access to file contents via a traditional filesystem, but can only be attached to one EC2 instance at a time. [EFS](#efs) is a network filesystem many instances can connect to, but at higher cost. See the [comparison table](#storage-durability-availability-and-price).
|
||||
|
||||
### S3 Tips
|
||||
|
||||
|
|
@ -1258,30 +1262,30 @@ Billing and Cost Management
|
|||
- **Spot fleet:**
|
||||
- You can realize even bigger cost reductions at the same time as improvements to fleet stability relative to regular Spot usage by using [Spot fleet](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html) to bid on instances across instance types, availability zones, and (through multiple Spot Fleet Requests) regions.
|
||||
- Spot fleet targets maintaining a specified (and weighted-by-instance-type) total capacity across a cluster of servers. If the Spot price of one instance type and availability zone combination rises above the weighted bid, it will rotate running instances out and bring up new ones of another type and location up in order to maintain the target capacity without going over target cluster cost.
|
||||
- **Spot Usage Best Practices:**
|
||||
- **Application Profiling:**
|
||||
- Profile your application to figure out its runtime characteristics. That would help give an understanding of the minimum cpu, memory, disk required. Having this information is critical before you try to optimize spot costs.
|
||||
- Once you know the minimum application requirements, instead of resorting to fixed instance types, you can bid across a variety of instance types (that gives you higher chances of getting a spot instance to run your application).E.g., If you know that 4 cpu cores are enough for your job, you can choose any instance type that is equal or above 4 cores and that has the least Spot price based on history. This helps you bid for instances with greater discount (less demand at that point).
|
||||
- **Spot Price Monitoring and Intelligence:**
|
||||
- Spot Instance prices fluctuate depending on instance types, time of day, region and availability zone. The AWS CLI tools and API allow you to describe Spot price metadata given time, instance type, and region/AZ.
|
||||
- Based on history of Spot instance prices, you could potentially build a myriad of algorithms that would help you to pick an instance type that either
|
||||
- optimizes cost
|
||||
- maximizes availability
|
||||
- offers predictable performance
|
||||
- You can also track the number of times an instance of certain type got taken away (out bid) and plot that in graphite to improve your algorithm based on time of day.
|
||||
- **Spot Machine Resource Utilization:**
|
||||
- For running spiky workloads (spark, map reduce jobs) that are schedule based and where failure is non critical, Spot instances become the perfect candidates.
|
||||
- The time it takes to satisfy a Spot instance could vary between 2-10 mins depending on the type of instance and availability of machines in that AZ.
|
||||
- If you are running an infrastructure with hundreds of jobs of spiky nature, it is advisable to start pooling instances to optimize for cost, performance and most importantly time to acquire an instance.
|
||||
- Pooling implies creating and maintaining Spot instances so that they do not get terminated after use. This promotes re-use of Spot instances across jobs. This of course comes with the overhead of lifecycle management.
|
||||
- Pooling has its own set of metrics that can be tracked to optimize resource utilization, efficiency and cost.
|
||||
- Typical pooling implementations give anywhere between 45-60% cost optimizations and 40% reduction in spot instance creationg time.
|
||||
- An excellent example of Pooling implementation described by Netflix ([part1](http://techblog.netflix.com/2015/09/creating-your-own-ec2-spot-market.html), [part2](http://techblog.netflix.com/2015/11/creating-your-own-ec2-spot-market-part-2.html))
|
||||
- **Spot Management Gotchas**
|
||||
- 🔸 **Lifetime** - There is no guarantee for the lifetime of a Spot instance. It is purely based on bidding. If anyone outbids your price, the instance is taken away. Spot is not suitable for time sensitive jobs that have strong SLA. Instances will fail based on demand for Spot at that time. AWS does not send any signal that the instance is going away, except for the fact that it is going down. That makes it hard to figure out why the instance(s) went down.
|
||||
- 🔹 **Api Return Data** - The Spot price API returns Spot prices of varying granularity depending on the time range specified in the api call.E.g If the last 10 min worth of history is requested, the data is more fine grained. If the last 2 day worth of history is requested, the data is more coarser. Do not assume you will get all the data points. There **will** be skipped intervals.
|
||||
- ❗**Lifecycle management** - Do not attempt any fancy Spot management unless absolutely necessary. If your entire usage is only a few machines and your cost is acceptable and your failure rate is lower, do not attempt to optimize. The pain for building/maintaining it is not worth just a few hundred dollar savings.
|
||||
- **Reserved Instances** allow you to get significant discounts on EC2 compute hours in return for a commitment to pay for instance hours of a specific instance type in a specific AWS region and availability zone for a pre-established time frame (1 or 3 years). Further discounts can be realized through “partial” or “all upfront” payment options.
|
||||
- **Spot usage best practices:**
|
||||
- **Application profiling:**
|
||||
- Profile your application to figure out its runtime characteristics. That would help give an understanding of the minimum cpu, memory, disk required. Having this information is critical before you try to optimize spot costs.
|
||||
- Once you know the minimum application requirements, instead of resorting to fixed instance types, you can bid across a variety of instance types (that gives you higher chances of getting a spot instance to run your application).E.g., If you know that 4 cpu cores are enough for your job, you can choose any instance type that is equal or above 4 cores and that has the least Spot price based on history. This helps you bid for instances with greater discount (less demand at that point).
|
||||
- **Spot price monitoring and intelligence:**
|
||||
- Spot Instance prices fluctuate depending on instance types, time of day, region and availability zone. The AWS CLI tools and API allow you to describe Spot price metadata given time, instance type, and region/AZ.
|
||||
- Based on history of Spot instance prices, you could potentially build a myriad of algorithms that would help you to pick an instance type that either
|
||||
- optimizes cost
|
||||
- maximizes availability
|
||||
- offers predictable performance
|
||||
- You can also track the number of times an instance of certain type got taken away (out bid) and plot that in graphite to improve your algorithm based on time of day.
|
||||
- **Spot machine resource utilization:**
|
||||
- For running spiky workloads (spark, map reduce jobs) that are schedule based and where failure is non critical, Spot instances become the perfect candidates.
|
||||
- The time it takes to satisfy a Spot instance could vary between 2-10 mins depending on the type of instance and availability of machines in that AZ.
|
||||
- If you are running an infrastructure with hundreds of jobs of spiky nature, it is advisable to start pooling instances to optimize for cost, performance and most importantly time to acquire an instance.
|
||||
- Pooling implies creating and maintaining Spot instances so that they do not get terminated after use. This promotes re-use of Spot instances across jobs. This of course comes with the overhead of lifecycle management.
|
||||
- Pooling has its own set of metrics that can be tracked to optimize resource utilization, efficiency and cost.
|
||||
- Typical pooling implementations give anywhere between 45-60% cost optimizations and 40% reduction in spot instance creationg time.
|
||||
- An excellent example of Pooling implementation described by Netflix ([part1](http://techblog.netflix.com/2015/09/creating-your-own-ec2-spot-market.html), [part2](http://techblog.netflix.com/2015/11/creating-your-own-ec2-spot-market-part-2.html)\)
|
||||
- **Spot management gotchas**
|
||||
- 🔸**Lifetime:** There is no guarantee for the lifetime of a Spot instance. It is purely based on bidding. If anyone outbids your price, the instance is taken away. Spot is not suitable for time sensitive jobs that have strong SLA. Instances will fail based on demand for Spot at that time. AWS does not send any signal that the instance is going away, except for the fact that it is going down. That makes it hard to figure out why the instance(s) went down.
|
||||
- 🔹**API return data:** - The Spot price API returns Spot prices of varying granularity depending on the time range specified in the api call.E.g If the last 10 min worth of history is requested, the data is more fine grained. If the last 2 day worth of history is requested, the data is more coarser. Do not assume you will get all the data points. There **will** be skipped intervals.
|
||||
- ❗**Lifecycle management:** Do not attempt any fancy Spot management unless absolutely necessary. If your entire usage is only a few machines and your cost is acceptable and your failure rate is lower, do not attempt to optimize. The pain for building/maintaining it is not worth just a few hundred dollar savings.
|
||||
- **Reserved Instances:** allow you to get significant discounts on EC2 compute hours in return for a commitment to pay for instance hours of a specific instance type in a specific AWS region and availability zone for a pre-established time frame (1 or 3 years). Further discounts can be realized through “partial” or “all upfront” payment options.
|
||||
- Consider using Reserved Instances when you can predict your longer-term compute needs and need a stronger guarantee of compute availability and continuity than the (typically cheaper) Spot market can provide. However be aware that if your architecture changes your computing needs may change as well so long term contracts can seem attractive but may turn out to be cumbersome.
|
||||
- Instance reservations are not tied to specific EC2 instances - they are applied at the billing level to eligible compute hours as they are consumed across all of the instances in an account.
|
||||
- **Hourly billing waste:** EC2 instances are [billed in instance-hours](https://aws.amazon.com/ec2/faqs/#How_will_I_be_charged_and_billed_for_my_use_of_Amazon_EC2) — rounded up to the nearest full hour! For long-lived instances, this is not a big worry, but for large transient deployments, like EMR jobs or test deployments, this can be a significant expense. Never deploy many instances and terminate them after only a few minutes. In fact, if transient instances are part of your regular processing workflow, you should put in protections or alerts to check for this kind of waste.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ exclude:
|
|||
roles:
|
||||
jlevy: general editor
|
||||
ThanosBaskous: general editor
|
||||
max-zanko: editor (S3, EMR, Redshift)
|
||||
nitingithub: editor (cost management)
|
||||
marcello3d:
|
||||
golyshev:
|
||||
forwidur:
|
||||
|
|
@ -24,7 +26,6 @@ roles:
|
|||
Praveen Patnala:
|
||||
kazuyukitanimura:
|
||||
olawiberg:
|
||||
max-zanko:
|
||||
weirded:
|
||||
bittlingmayer:
|
||||
rjpower:
|
||||
|
|
@ -32,4 +33,3 @@ roles:
|
|||
donnemartin:
|
||||
benkehoe:
|
||||
Jurgen Philippaerts:
|
||||
nitingithub: editor (cost management)
|
||||
|
|
|
|||
BIN
figures/aws-data-transfer-costs-320px.png
Normal file
BIN
figures/aws-data-transfer-costs-320px.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 48 KiB |
BIN
figures/aws-market-landscape-240px.png
Normal file
BIN
figures/aws-market-landscape-240px.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
BIN
figures/signpost-horiz1-1600.jpg
Normal file
BIN
figures/signpost-horiz1-1600.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 111 KiB |
Loading…
Reference in a new issue