From e188dad1e0e4ca59a5205404daf3e6d4830f3a14 Mon Sep 17 00:00:00 2001 From: Kevin Lin Date: Wed, 26 Dec 2018 12:48:55 -0800 Subject: [PATCH] basic coverage for elastic beanstalk (#669) * basic coverage for elastic beanstalk --- README.md | 50 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index ce26e24..e3ebdd8 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ Table of Contents | [ECS](#ecs) | [πŸ“—](#ecs-basics) | [πŸ“˜](#ecs-tips) | | | [EKS](#eks) | [πŸ“—](#eks-basics) | [πŸ“˜](#eks-tips) | [πŸ“™](#eks-gotchas-limitations) | | [EFS](#efs) | [πŸ“—](#efs-basics) | [πŸ“˜](#efs-tips) | [πŸ“™](#efs-gotchas-and-limitations) | +| [Elastic Beanstalk](#elastic-beanstalk) | [πŸ“—](#elastic-beanstalk-basics) | [πŸ“˜](#elastic-beanstalk-tips) | [πŸ“™](#elastic-beanstalk-gotchas-and-limitations) | | [Elastic IPs](#elastic-ips) | [πŸ“—](#elastic-ip-basics) | [πŸ“˜](#elastic-ip-tips) | [πŸ“™](#elastic-ip-gotchas-and-limitations) | | [ElastiCache](#elasticache) | [πŸ“—](#elasticache-basics) | [πŸ“˜](#elasticache-tips) | [πŸ“™](#elasticache-gotchas-and-limitations) | | [EMR](#emr) | [πŸ“—](#emr-basics) | [πŸ“˜](#emr-tips) | [πŸ“™](#emr-gotchas-and-limitations) | @@ -219,7 +220,7 @@ General Information - πŸ•[Elastic Beanstalk](https://aws.amazon.com/elasticbeanstalk/): Fully managed (PaaS) deployment of packaged Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker applications - πŸ₯[EFS](#efs): Network filesystem compatible with NFSv4.1 - β›“πŸ•[ECS](#ecs): Docker container/cluster management (note Docker can also be used directly, without ECS) - - πŸ• [EKS](#eks): Kubernetes (K8) Docker Container/Cluster management + - πŸ• [EKS](#eks): Kubernetes (K8) Docker Container/Cluster management - β›“[ECR](https://aws.amazon.com/ecr/): Hosted private Docker registry - πŸ₯[Config](https://aws.amazon.com/config/): AWS configuration inventory, history, change notifications - πŸ₯[X-Ray](https://aws.amazon.com/xray/): Trace analysis and debugging for distributed applications such as microservices. @@ -302,9 +303,9 @@ Many services within AWS can at least be compared with Google Cloud offerings or | Object storage | S3 | Cloud Storage | GFS | Storage Account | DigitalOcean Spaces | Swift, HDFS, Minio | Swift | | Block storage | EBS | Persistent Disk | | Storage Account | DigitalOcean Volumes | NFS | Cinder | | SQL datastore | RDS | Cloud SQL | | SQL Database | | MySQL, PostgreSQL | Trove (stores NoSQL as well) | -| Sharded RDBMS | | Cloud Spanner | F1, Spanner | | | Crate.io, CockroachDB | +| Sharded RDBMS | | Cloud Spanner | F1, Spanner | | | Crate.io, CockroachDB | | Bigtable | | Cloud Bigtable | Bigtable | | | HBase | -| Key-value store, column store | DynamoDB | Cloud Datastore | Megastore | Tables, DocumentDB | | Cassandra, CouchDB, RethinkDB, Redis | +| Key-value store, column store | DynamoDB | Cloud Datastore | Megastore | Tables, DocumentDB | | Cassandra, CouchDB, RethinkDB, Redis | | Memory cache | ElastiCache | App Engine Memcache | | Redis Cache | | Memcached, Redis | | Search | CloudSearch, Elasticsearch (managed) | | | Search | Algolia, QBox, Elastic Cloud | Elasticsearch, Solr | | Data warehouse | Redshift | BigQuery | Dremel | SQL Data Warehouse | Oracle, IBM, SAP, HP, many others | Greenplum | @@ -390,7 +391,7 @@ It’s important to know the maturity of each AWS product. Here is a mostly comp | [IAM](https://aws.amazon.com/releasenotes/AWS-Identity-and-Access-Management?browse=1) | 2010-09 | General | βœ“ | | βœ“ | | [SNS](https://aws.amazon.com/releasenotes/Amazon-SNS?browse=1) | 2010-04 | General | βœ“ | βœ“ | | | [EMR](https://aws.amazon.com/releasenotes/Elastic-MapReduce?browse=1) | 2010-04 | General | βœ“ | βœ“ | βœ“ | -| [RDS](https://aws.amazon.com/releasenotes/Amazon-RDS?browse=1) | 2009-12 | General | βœ“ |βœ“[2](#user-content-hipaa-rds) |βœ“[9](#user-content-pci-rds) | +| [RDS](https://aws.amazon.com/releasenotes/Amazon-RDS?browse=1) | 2009-12 | General | βœ“ |βœ“[2](#user-content-hipaa-rds) |βœ“[9](#user-content-pci-rds) | | [VPC](https://aws.amazon.com/releasenotes/Amazon-VPC?browse=1) | 2009-08 | General | βœ“ | βœ“ | βœ“ | | [Snowball](https://aws.amazon.com/releasenotes/AWS-ImportExport?browse=1) | 2015-10 | General | βœ“ | βœ“ | | | [Snowmobile](https://aws.amazon.com/snowmobile/) | 2016-11 | General | | | | @@ -566,7 +567,7 @@ This guide is about AWS, not DevOps or server configuration management in genera - A consequence of this is that you need fewer AMIs and boot scripts; for most deployments, the only boot script you need is a template that fetches an exported docker image and runs it. - Companies that are embracing [microservice architectures](http://martinfowler.com/articles/microservices.html) will often turn to container-based deployments. - AWS launched [ECS](https://aws.amazon.com/ecs/) as a service to manage clusters via Docker in late 2014, though many people still deploy Docker directly themselves. See the [ECS section](#ecs) for more details. -- AWS launched [EKS](https://aws.amazon.com/eks/) as a service to manage Kubernetes Clusters mid 2018, though many people still deploy ECS or use Docker directly themselves. See the [EKS section](#eks) for more details. +- AWS launched [EKS](https://aws.amazon.com/eks/) as a service to manage Kubernetes Clusters mid 2018, though many people still deploy ECS or use Docker directly themselves. See the [EKS section](#eks) for more details. ### Visibility @@ -873,7 +874,6 @@ EC2 - CloudWatch ------------------- @@ -1142,6 +1142,29 @@ ALB - In a target group, if there is no healthy target, all requests are routed to all targets. For example, if you point a listener at a target group containing a single service that has a long initialization phase (during which the health checks would fail), requests will reach the service while it is still starting up. - πŸ“œ Although ALBs [now support SNI](https://aws.amazon.com/about-aws/whats-new/2017/10/elastic-load-balancing-application-load-balancers-now-support-multiple-ssl-certificates-and-smart-certificate-selection-using-server-name-indication-sni/), they only support 25 HTTPS certificates per Load Balancer. This limitation is not described [here](http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html), so it might be subject to change. +Elastic Beanstalk +---------------- + +### Elastic Beanstalk Basics +- πŸ“’ [Homepage](https://aws.amazon.com/elasticloadbalancing/) βˆ™ [Developer guide](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html) βˆ™ [FAQ](https://aws.amazon.com/elasticbeanstalk/faqs/) βˆ™ [Pricing](https://aws.amazon.com/elasticbeanstalk/pricing/) +- **EB** (Elastic Beanstalk) is a PaaS (Platform as a Service) that helps developers create, deploy and scale web applications +- EB handles deployment, configuration, provisioning, load balancing, auto-scaling, monitoring, and logging +- EB creates AWS resources on your behalf but you retain full access and control of the underlying resources +- πŸ’Έ There is no cost to use EB but you will still be charged the full cost of the underlying AWS resources created by EB + +### Elastic Beanstalk Tips +- To speed up deployment before launch or in a dev stage, turn off health checks and set the `Deployment policy` to `All at once` +- If you have a configuration you want to re-use for multiple EB apps, you can save the current configuration using `eb config save --cfg myEBConfig` +- By default, EB doesn't have any alarms. You'll need to add them yourself on metrics that you're monitoring. +- By default, EB doesn't enable [managed platform updates](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-platform-update-managed.html?icmpid=docs_elasticbeanstalk_console). Enable them in configuration to have EB automatically apply updates during a pre-specified maintenance window + +### Elastic Beanstalk Gotchas and Limitations +- πŸ”Έ Don't edit [apache|nginx] conf files manually on ec2 instances as they will be re-written on each deployment (use [ebextensions](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html) instead) +- πŸ”Έ After creating an EB environment, it's no longer possible to change the `Name` tag +- πŸ”Έ EB will sometimes quarantine instances that cause multiple deployment issues. Despite being quarantined, EB will still deploy to them on subsequent deployments. To prevent this behavior, said instances will need to be terminated (or the underlying issue fixed) +- File uploads are capped at 10MB for most default eb configurations - update [nginx config](https://stackoverflow.com/questions/18908426/increasing-client-max-body-size-in-nginx-conf-on-aws-elastic-beanstalk) to change +- If you edit `.elasticbeanstalk/saved_configs/`, be aware that this is not kept in sync with the EB environment config. You'll need to manually fetch and save for changes to take effect + Elastic IPs ----------- @@ -1448,27 +1471,27 @@ EKS ### EKS Basics - πŸ“’ [Homepage](https://aws.amazon.com/eks/) βˆ™ [User guide](http://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) βˆ™ [FAQ](https://aws.amazon.com/eks/faq/) βˆ™ [Pricing](https://aws.amazon.com/eks/pricing/) -- EKS (Elastic Kubernetes Service) is a new service (launched June 2018) that provides managed Kubernetes Masters in a Highly Available pair to deploy K8s Services and Pods on top of EC2 based Kubernetes nodes. +- EKS (Elastic Kubernetes Service) is a new service (launched June 2018) that provides managed Kubernetes Masters in a Highly Available pair to deploy K8s Services and Pods on top of EC2 based Kubernetes nodes. - See the [Containers and AWS](#containers-and-aws) section for more context on containers. - EKS is AWS's solution to hosting Kubernetes natively on AWS. It is not a replacement for ECS directly but is in response to the large market dominance of Kubernetes. - EKS does not launch EC2 nodes and would have to be configured and setup either manually or via Cloudformation (or other automation solution) - EKS management is done through a utility called kubectl, and with Kube configuration files. These files will need to be configured to speak with the K8s Master with a certificate and URL -- EKS authentication is integrated with IAM roles/permissions via a custom plugin for kubectl called aws-iam-authenticator (formerly heptio-authenticator-aws) https://github.com/kubernetes-sigs/aws-iam-authenticator +- EKS authentication is integrated with IAM roles/permissions via a custom plugin for kubectl called aws-iam-authenticator (formerly heptio-authenticator-aws) https://github.com/kubernetes-sigs/aws-iam-authenticator ### EKS Tips - If you do not already have kubectl configured, it is now included in the AWSCLI packages. This is the simplest way to install kubectl and the assocated iam authenticator plugin - Multiple clusters can be supported by using different kubeconfig files ### EKS Alternatives and Lock-in -- [ECS](#ecs) Amazon's native Container Scheduled platform released in 2014. If you don't utilize containers today and are looking to get started, ECS is an excellent product. +- [ECS](#ecs) Amazon's native Container Scheduled platform released in 2014. If you don't utilize containers today and are looking to get started, ECS is an excellent product. - [Kubernetes](https://kubernetes.io): Extensive container platform. Available as a hosted solution on Google Cloud (https://cloud.google.com/container-engine/), AWS (https://tectonic.com/), Digital Ocean (https://www.digitalocean.com/products/kubernetes/) and Azure (https://azure.microsoft.com/en-us/services/kubernetes-service/). AWS have a Kubernetes Quickstart (https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/) developed in collaboration with Heptio. - [Nomad](https://www.nomadproject.io/): Orchestrator/Scheduler, tightly integrated in the Hashicorp stack (Consul, Vault, etc). ### EKS Gotchas and Limitations - Pods and Service configurations can rapidly consume IP addresses inside a VPC. Proper care and maintenance should be applied to ensure IP exhaustion does not occur -- There is currently no integrated monitoring in Cloudwatch for EKS pods or services, you will need to deploy a monitoring system that supports kubernetes such as Prometheus. -- Autoscaling based off CPU/Memory of a node is limited as you will not be aware of pending services/pods that cannot start. As this is not queryable via EKS API's, you would need to write this in Lambda with kubectl, this may be a security concern for some organizations. -- Prometheus (https://prometheus.io/) is a very popular monitoring solution for K8s, metrics and alerts can be used to send events to Lambda, SQS or other solutions to take autoscaling actions. +- There is currently no integrated monitoring in Cloudwatch for EKS pods or services, you will need to deploy a monitoring system that supports kubernetes such as Prometheus. +- Autoscaling based off CPU/Memory of a node is limited as you will not be aware of pending services/pods that cannot start. As this is not queryable via EKS API's, you would need to write this in Lambda with kubectl, this may be a security concern for some organizations. +- Prometheus (https://prometheus.io/) is a very popular monitoring solution for K8s, metrics and alerts can be used to send events to Lambda, SQS or other solutions to take autoscaling actions. Fargate ------- @@ -1973,7 +1996,7 @@ Device Farm - πŸ”ΈThe API and CLI for Device Farm is quite a low level and may require developing additional tools or scripts on top of it. - πŸ”ΈAWS provide several tools and plugins for Device Farm, however, it doesnβ€˜t cover all cases or platforms. It may require developing specific tools or plugins to support specific requirements. - ❗In general, Device Farm doesnβ€˜t have Android devices from Chinese companies like Huawei, Meizu, Lenovo, etc. An actual list of supported devices located [here](https://aws.amazon.com/device-farm/device-list/). -- πŸ”ΈDevice availibility is uneven. It depends on several factors including device popularity. Usually, more modern devices see higher demand, thus the waiting time for them will be higher compared to relatively old devices. +- πŸ”ΈDevice availibility is uneven. It depends on several factors including device popularity. Usually, more modern devices see higher demand, thus the waiting time for them will be higher compared to relatively old devices. Mobile Hub ---------- @@ -2005,7 +2028,6 @@ IoT - **IoT** is a platform for allowing clients such as IoT devices or software applications ([examples](http://internetofthingswiki.com/iot-applications-examples/541/)) to communicate with the AWS cloud. - Clients are also called **devices** (or **things**) and include a wide variety of device types. Roughly there are three categories of device types that interact with IoT services by sending message over an IoT protocol to the IoT Pub/Sub-style message broker, which is called the IoT **Device Gateway**: * Send messages only: For example, the [AWS IoT Button](https://aws.amazon.com/iot/button/) on an [eddystone beacon](http://developer.estimote.com/eddystone/). - * Send and receive messages: For example, the [Phillips Home Safe Medical Alert device](http://rethink-iot.com/2015/10/16/philips-uses-new-aws-iot-platform-to-expand-its-healthsuite-cloud/) * Send, receive, and process messages: For example, a simple processing board, such as a **Raspberry Pi** ([quick start guide](http://docs.aws.amazon.com/iot/latest/developerguide/iot-device-sdk-c.html)), or an AWS device, such as [Echo or Echo Dot](https://developer.amazon.com/echo), which are designed to work with the [AWS Alexa skills kit](https://developer.amazon.com/alexa-skills-kit) (a programmable voice-enabled service from AWS). - AWS has a useful [quick-start](http://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html) (using the Console) and a [slide presentation](http://www.slideshare.net/AmazonWebServices/connecting-to-aws-iot) on core topics. * **IoT terms:**