This article is a note, which is convenient for future generations to step on the pit, and it is also convenient for yourself to review the entire process after stepping on the pit. Reading this article requires the following preliminary knowledge/preparation:
- An AWS account with no arrears
- Applications that can already be containerized and run, and the image has been pushed to the ECR of the corresponding region or is a public image
As a typical Web App, it must be composed of App Server, Web Server and Database. In order to achieve more reliable elastic expansion and contraction, the AWS platform is used here, and the corresponding ones become Container, Application Load Balancer and AWS RDS.
In order to run our own containers on AWS, we have the following options:
- Open EC2 by yourself and install Docker on it to run (then the cost performance is poor, it is better to use Hetzner directly)
- Use Amazon Elastic Container Service (Amazon ECS) (a bit cumbersome to configure, not industrial style)
- Use Amazon App Runner (this is very spiritual, only one container can be set up, and it does not work with the existing VPC, so RDS cannot be used)
- Using Amazon Elastic Kubernetes Service (EKS) (also the main focus of this article)
Amazon Elastic Kubernetes Service (EKS)
This is the Kubernetes service maintained by Amazon. We all know that installing/maintaining a Kubernetes must be a thankless task. Node certificate renewal, cluster upgrade, network plug-ins, etc. are all obstacles for K8s beginners. Consider I don’t know anything about Kubernetes, and I have done a lot of such weird things when I was working at PingCAP, so since I chose to use K8s here, it’s better to focus on kubectl and apply it, and leave the rest to the service provider ( aka, AWS) to memorize.
Regarding the price, the document says: You pay $0.10 per hour for each Amazon EKS cluster that you create. That is to say, its control surface will directly eat up your 72USD in one month, and the money for the machine is calculated separately. Yes, compared to DigitalOcean/Vultr/Linode, a free control plane service provider, AWS is much more expensive.
Create Cluster
Since it is a record, we will start our entire process as soon as possible. Here we mainly refer to the following two documents:
I feel that the above two documents are higher than the official AWS documents for Quick Start.
eksctl, kubectl and aws
Here you need to ensure that eksctl , kubectl and aws have been installed, you can refer to two articles of AWS:
Although the official website of eksctl says The official CLI for Amazon EKS, there is a sentence below created by Weaveworks and it welcomes contributions from the community, which is very spiritual.
If you use Linux like me, just copy my instructions~
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_ $( uname -s ) _amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/bin eksctl version curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.24.7/2022-10-31/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/bin/ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install
AWS Credentials
First, you need to log in to your console to get Credentials and put them in the ~/.aws/credentials file (that is, this part that is often leaked by others on GitHub:
[default] aws_access_key_id = AKHAHAHAHAHAH6HPS6R aws_secret_access_key = NOjlIsTHAHAHAHAHAHAHAHAHAHAHSOUsX region = ap-northeast-1 [nova] aws_access_key_id = AKHBABABABABABH6HPS6R aws_secret_access_key = NOjlIsTHAABABABABAAHAHSOUsX region = ap-southeast-1
Cluster
If there are multiple profiles in the above example in your ~/.aws/credentials file, you can use the corresponding profile by adding AWS_PROFILE=<name> in front of the command.
Through this command, you can quickly create an EKS Cluster called novacluster . Behind eksctl, a eksctl will be created to handle all the details that we don’t want to handle by ourselves (what IAM, what Tag), and this step usually takes 10+ minutes.
eksctl create cluster --name = novacluster --without-nodegroup --region = ap-southeast-1
Next we need to create an OIDC identity provider.
eksctl utils associate-iam-oidc-provider \ --region ap-southeast-1 \ --cluster novacluster \ --approve
Then we create a NodeGroup called worker-group , which is the actual machine used to run the load:
eksctl create nodegroup --cluster = novacluster --region = ap-southeast-1 --name = worker-group --node-type = t3.medium --nodes = 3 --nodes-min = 2 --nodes-max = 10 --node-volume-size = 20 --managed --asg-access --full-ecr-access --alb-ingress-access
Here is a suggestion, do not use Free Tier
t3.micro, otherwise you will encounter Insufficient memory problems when deployingcluster-autoscaler, because it requires 600Mi of memory, andt3.microwill only run when nothing is running. There are 400 meters left.If you make a mistake, it is recommended to create a Nodegroup with a newer configuration first, and then use
eksctl delete nodegroup --cluster=novacluster --name=worker-group --region=ap-southeast-1to delete the old NodegroupThis is similar to nucleic acid testing. You must first consider canceling nucleic acid testing and verification in all places before removing the nucleic acid testing points. It cannot be reversed, but some people just don’t understand it. The consequence is that your Pod will be in the cold like the citizens. Pending for a long time.
At this time, our cluster has been created. In order to make local kubectl available, we need to use AWS Cli to obtain kubeconfig . The command is as follows:
aws eks update-kubeconfig --region ap-southeast-1 --name novacluster
At this time, our kubectl should be available, try it:
kubectl get no NAME STATUS ROLES AGE VERSION ip-192-168-26-67.ap-southeast-1.compute.internal Ready <none> 98m v1.23.13-eks-fb459a0 ip-192-168-42-176.ap-southeast-1.compute.internal Ready <none> 75m v1.23.13-eks-fb459a0 ip-192-168-46-84.ap-southeast-1.compute.internal Ready <none> 98m v1.23.13-eks-fb459a0 ip-192-168-72-96.ap-southeast-1.compute.internal Ready <none> 75m v1.23.13-eks-fb459a0 ip-192-168-75-202.ap-southeast-1.compute.internal Ready <none> 98m v1.23.13-eks-fb459a0
At this point, our cluster and machines are available, and Eksctl has also created a bunch of VPCs and Subnets.
We noticed that the default Subnet VPC here is vpc-1f2a1f78 , and the corresponding segment is 172.31.0.0/20 , while the segment created by eksctl is 192.168.0.0/19 , which also leads to the inability to configure the App to connect to RDS like EC2 Connecting to RDS is the same as communicating with the intranet under a VPC, but VPC Peering is required.
Node AutoScale
If you want to manually scale up and shrink Node for the cluster, you can use this command:
eksctl scale nodegroup --cluster = novacluster --region ap-southeast-1 --nodes = 5 worker-group
I know that we have specified --nodes=3 --nodes-min=2 --nodes-max=10 when we created the cluster. At this time, you may think:
“Ah, then AWS will automatically expand and shrink automatically between 2 to 10 nodes according to the cluster situation?”
If you want Node to automatically scale up and down, you need to manually create a cluster-autoscaler
Really, EKS has already managed so much, why can’t the Scale machine be integrated into an Addon that can be installed automatically with one click, or it comes with it by default like DigitalOcean DOKS?
The instructions are as follows, first create a Policy to allow Autoscale, and create a file called cluster-autoscaler-policy.json with the following content:
{ "Version" : "2012-10-17" , "Statement" : [ { "Sid" : "VisualEditor0" , "Effect" : "Allow" , "Action" : [ "autoscaling:SetDesiredCapacity" , "autoscaling:TerminateInstanceInAutoScalingGroup" ], "Resource" : "*" , "Condition" : { "StringEquals" : { "aws:ResourceTag/k8s.io/cluster-autoscaler/my-cluster" : "owned" } } }, { "Sid" : "VisualEditor1" , "Effect" : "Allow" , "Action" : [ "autoscaling:DescribeAutoScalingInstances" , "autoscaling:DescribeAutoScalingGroups" , "ec2:DescribeLaunchTemplateVersions" , "autoscaling:DescribeTags" , "autoscaling:DescribeLaunchConfigurations" ], "Resource" : "*" } ] }
Remember to change my-cluster to the name of your own EKS Cluster, or wait for an error~
Then use AWS tools to apply:
aws iam create-policy \ --policy-name AmazonEKSClusterAutoscalerPolicy \ --policy-document file://cluster-autoscaler-policy.json
Then use eksctl to create an IAM Service Account and attach the Policy just now
eksctl create iamserviceaccount \ --cluster = novacluster --region = ap-southeast-1 \ --namespace = kube-system \ --name = cluster-autoscaler \ --attach-policy-arn = arn:aws:iam::111122223333:policy/AmazonEKSClusterAutoscalerPolicy \ --override-existing-serviceaccounts \ --approve
After that, I started to install cluster-autoscaler , which is very Cloud Naive.
curl -o cluster-autoscaler-autodiscover.yaml https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml sed -i.bak -e 's|<YOUR CLUSTER NAME>|novacluster|' ./cluster-autoscaler-autodiscover.yaml kubectl apply -f cluster-autoscaler-autodiscover.yaml kubectl annotate serviceaccount cluster-autoscaler \ -n kube-system \ eks.amazonaws.com/role-arn = arn:aws:iam::111122223333:role/AmazonEKSClusterAutoscalerRole kubectl patch deployment cluster-autoscaler \ -n kube-system \ -p '{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}}}}}'
After that, you can check the Log through kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler to determine whether the cluster-autoscaler is working normally. If there is any Pod that cannot be scheduled, this thing will automatically give You open a new machine for expansion. Similarly, if you have few resources, it will help you dynamically clear your Node.
Application Load Balancer Controller
We need to expose our application to the outside world, so we need a Load Balancer, the price is $0.0225 per Application Load Balancer-hour (or partial hour) + $0.008 per LCU-hour (or partial hour), that is to say, even if you have no traffic, you can To 18USD/mo.
To integrate ALB (Application Load Balancer) in EKS, you need to manually install the Application Load Balancer Controller and tag the corresponding subnet. In the above eksctl create nodegroup , we see that there is a --alb-ingress-access just for us After the second half, you still have to manually install the Controller. The specific process is as follows.
Create an IAMPolicy:
curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/install/iam_policy.json aws iam create-policy \ --policy-name AWSLoadBalancerControllerIAMPolicy \ --policy-document file://iam_policy.json
First use eksctl to create a ServiceAccount called aws-load-balancer-controller for subsequent ALB Controller use, remember to replace 111122223333 with your Account ID.
eksctl create iamserviceaccount \ --cluster = novacluster \ --region = ap-southeast-1 \ --namespace = kube-system \ --name = aws-load-balancer-controller \ --role-name "AmazonEKSLoadBalancerControllerRole" \ --attach-policy-arn = arn:aws:iam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \ --approve
Then install cert-manager and Controller, just look at the document copy and paste.
kubectl apply \ --validate = false \ -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml curl -Lo v2_4_4_full.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.4.4/v2_4_4_full.yaml sed -i.bak -e '480,488d' ./v2_4_4_full.yaml sed -i.bak -e 's|your-cluster-name|novacluster|' ./v2_4_4_full.yaml kubectl apply -f v2_4_4_full.yaml kubectl apply -f https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.4.4/v2_4_4_ingclass.yaml
At this point we can verify whether the Controller has been installed correctly:
kubectl get deployment -n kube-system aws-load-balancer-controller NAME READY UP-TO-DATE AVAILABLE AGE aws-load-balancer-controller 1 /1 1 1 95m
The complete document is in Installing the AWS Load Balancer Controller add-on . If you are not at ease, you can copy and paste that document. But really, EKS has already managed so much, why can’t the ALB integration be made into an Addon that can be installed automatically with one click?
At this point, let’s take a look at the AWS web page, you should have the following Deployments in your cluster:
Application
Finally, with the above-mentioned cluster preparation and Load balancer Controller, we can deploy our application. For the sake of convenience and cleanliness, we set up a Namespace called novaapp and put all resources under this Namespace, and let AWS automatically provide We add Load Balancer for external access:
---
apiVersion : v1
kind : Namespace
metadata :
name : novaapp
---
apiVersion : apps/v1
kind : Deployment
metadata :
namespace : novaapp
name : novaapp-mini-deployment
labels :
app : novaapp-mini
spec :
replicas : 2
selector :
matchLabels :
app : novaapp-mini
template :
metadata :
labels :
app : novaapp-mini
spec :
containers :
- name : novaapp
imagePullPolicy : Always
image : '111122223333.dkr.ecr.ap-southeast-1.amazonaws.com/novaapp:latest'
env :
- name : DB_HOST
value : "novards.c4s0xipwdxny.ap-southeast-1.rds.amazonaws.com"
- name : APP_DEBUG
value : "true"
- name : DB_PORT
value : "3306"
- name : DB_DATABASE
value : "novaapp"
- name : DB_USERNAME
value : "admin"
- name : DB_PASSWORD
value : "password"
resources :
limits :
cpu : 500m
---
apiVersion : v1
kind : Service
metadata :
namespace : novaapp
name : novaapp-mini-service
spec :
ports :
- port : 80
targetPort : 80
protocol : TCP
type : NodePort
selector :
app : novaapp-mini
---
apiVersion : networking.k8s.io/v1
kind : Ingress
metadata :
namespace : novaapp
name : ingress-novaapp
annotations :
alb.ingress.kubernetes.io/scheme : internet-facing
alb.ingress.kubernetes.io/target-type : ip
alb.ingress.kubernetes.io/healthcheck-path : /healthz
spec :
ingressClassName : alb
rules :
- http :
paths :
- path : /
pathType : Prefix
backend :
service :
name : novaapp-mini-service
port :
number : 80
---
apiVersion : autoscaling/v1
kind : HorizontalPodAutoscaler
metadata :
namespace : novaapp
name : novaapp-mini-autoscaler
spec :
scaleTargetRef :
apiVersion : apps/v1
kind : Deployment
name : novaapp-mini-deployment
minReplicas : 2
maxReplicas : 20
targetCPUUtilizationPercentage : 10
looks long? Yes, this is the Cloud Naive Way that everyone admires!
In fact, as long as you look carefully, you will find that this is a complete set of components, which are divided into Namespace, Deployment (actual application), Service (use NodePort to expose the entire application and provide load balancing, and Ingress (let AWS create an ALB to transfer traffic) Lead to Service, and explicitly specify /healthz as a health check, otherwise if your / will return 404, the service will always be 503), HorizontalPodAutoscaler (if a Pod CPU usage is greater than 10%, automatically create more Pods, Create up to 20 pods for elastic expansion).
At the same time, you can also test, for example, modify the Replica to a relatively large value, and observe whether the cluster-autoscaler can normally create a new Node to join the cluster when all Nodes are full.
ALB SSL
In the above case, our ALB configuration looks like this:
apiVersion : networking.k8s.io/v1
kind : Ingress
metadata :
namespace : novaapp
name : ingress-novaapp
annotations :
alb.ingress.kubernetes.io/scheme : internet-facing
alb.ingress.kubernetes.io/target-type : ip
alb.ingress.kubernetes.io/healthcheck-path : /healthz
spec :
ingressClassName : alb
rules :
- http :
paths :
- path : /
pathType : Prefix
backend :
service :
name : novaapp-mini-service
port :
number : 80
At this time, we can already directly access our application by accessing the address of ALB, but what if there is no SSL? And in the end we need to do a CNAME resolution to this address, and use our own domain name to Serve the entire App.
So here we need to go to AWS’s ACM first Play a round of ACM/ICPC Get an SSL certificate on:
At this point, you can get an ARN. For example, here is arn:aws:acm:ap-southeast-1:111122223333:certificate/b9480e8e-c0e6-4cec-9ac4-38715ad35888 . After the certificate verification is passed, we will modify the ALB configuration For a moment, modify it to look like this:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: namespace: novaapp name: ingress-novaapp annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/healthcheck-path: /healthz alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]' alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:111122223333:certificate/b9480e8e-c0e6-4cec-9ac4-38715ad35888 spec: ingressClassName: alb rules: - http: paths: - path: / pathType: Prefix backend: service: name: novaapp-mini-service port: number: 80
Then apply it to the cluster. At this time, we can see that the Load Balancer page should have normally displayed ports 80 and 443, and the certificate has been correctly configured:
At this time, we go to Cloudflare to get a CNAME record to resolve to this address, and enable Cloudflare’s Proxy, and it becomes~
RDS & Peering
The process of creating RDS is very simple. We can create it directly on the RDS control panel. After the creation is complete, we need to allow traffic from the subnet created by eksctl in its Security Group:
In the next step, we need to connect these two segments to Peer, otherwise EC2 under the container will not be able to connect to RDS
We create a VPC Peer, and Peer the two VPCs together.
Remember to click Accept Peer after creation.
After the Peer is successful, it is necessary to notify the route of the other party on both sides, and notify the segment of the VPC of EKS on the Default VPC:
On the contrary, report the Default VPC segment on the pile of Route tables in EKS:
At this point, your application should be able to connect to RDS normally~
Monitoring && Logging
Life is short, don’t bother with the monitoring components yourself, just use the Datadog stack directly, you can refer to: Install the Datadog Agent on Kubernetes , you only need the following three steps to install:
helm repo add datadog https://helm.datadoghq.com helm install my-datadog-operator datadog/datadog-operator kubectl create secret generic datadog-secret --from-literal api-key=<DATADOG_API_KEY> --from-literal app-key=<DATADOG_APP_KEY>
Get a configuration file, such as datadog-agent.yaml , with the following content:
apiVersion : datadoghq.com/v1alpha1
kind : DatadogAgent
metadata :
name : datadog
spec :
credentials :
apiSecret :
secretName : datadog-secret
keyName : api-key
appSecret :
secretName : datadog-secret
keyName : app-key
agent :
image :
name : "gcr.io/datadoghq/agent:latest"
clusterAgent :
image :
name : "gcr.io/datadoghq/cluster-agent:latest"
Then: kubectl apply -f /path/to/your/datadog-agent.yaml will work~
As mentioned above, our application has already started running. If you need to change the image of the container, you can temporarily change the Deployment file locally, then kubectl apply a shuttle, and there will be CI/CD, monitoring and alarm related content due to the space limit of this article (plus I am still learning), so it will not be covered in this article for the time being~
Always keep this AWS platform in mind, and delete resources in time when you don’t need them. I wish you all a happy time!
References
- The Architecture Behind A One-Person Tech Startup
- eksctl – The official CLI for Amazon EKS
- Creating or updating a kubeconfig file for an Amazon EKS cluster
- Installing the AWS Load Balancer Controller add-on
This article is transferred from https://nova.moe/deploy-web-app-stack-on-eks/
This site is only for collection, and the copyright belongs to the original author.