2019年12月4日 星期三

利用 AWS SAM 部署 Serverless 時,需要什麼權限??

利用 AWS SAM 部署 Serverless 時,需要什麼權限??

每次要部署 SAM Project 時,為了符合 AWS IAM Best Practices 都要調校一次 CI User 的 Policy,耗時耗力,所以整理一份 Sample Policy 以便之後直接取用。
會用到的 SAM command 有 validate, build, package, deploy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:CreateFunction",
                "lambda:UpdateFunctionCode",
                "lambda:TagResource",
                "lambda:UntagResource",
                "lambda:ListTags",
                "lambda:GetFunctionConfiguration",
                "lambda:UpdateFunctionConfiguration",
                "iam:GetRole",
                "cloudformation:UpdateStack",
                "cloudformation:DescribeStackEvents",
                "cloudformation:CreateChangeSet",
                "cloudformation:DeleteChangeSet",
                "cloudformation:DescribeChangeSet",
                "cloudformation:ExecuteChangeSet"
                "s3:PutObject",
                "s3:GetObject",
            ],
            "Resource": [
                "arn:aws:lambda:{AWS::Region}:{AWS::AccountId}:function:{STACK_NAME}-{FUNCTION_NAME}*",
                "arn:aws:s3:::{BUCKET_NAME}/*",
                "arn:aws:iam::{AWS::AccountId}:role/{STACK_NAME}-{FUNCTION_NAME}Role*",
                "arn:aws:cloudformation:{AWS::Region}:aws:transform/Serverless-2016-10-31",
                "arn:aws:cloudformation:{AWS::Region}:{AWS::AccountId}:stack/{STACK_NAME}/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cloudformation:ValidateTemplate",
                "cloudformation:GetTemplateSummary",
                "cloudformation:DescribeStacks"
            ],
            "Resource": "*"
        }
    ]
}
  • replace the following:
    • {BUCKET_NAME} with the bucket name you’re using for code upload
    • {STACK_NAME} with your stack name
    • {FUNCTION_NAME} with your function name
  • 如果在 deploy 加上 —debug 參數,則還會需要 cloudformation:DescribeStackEvents 權限
  • 如果用到 Events properties,則還會需要 lambda:AddPermission,以及 Event type 相對應的 Resource Permission
  • 如果有設定 Lambda Role Policies 則還會需要以下 Permission
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole",
                "iam:DeleteRolePolicy",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:GetRolePolicy"
            ],
            "Resource": "arn:aws:iam::{AWS::AccountId}:role/{STACK_NAME}-LambdaRole*"
        }
    ]
}

Reference

2019年9月27日 星期五

How to delete AWS Config service

How to delete AWS Config service

最近在用 AWS Config,玩過以後想說要刪掉,結果發現在 console 找不到 delete config 功能,查了文件後,看起來只能用 cli 才砍得到,而且還不是一個 command,而是 ConfigurationRecorderDeliveryChannel 兩個 resources 分開,如下:
$ aws configservice delete-configuration-recorder --configuration-recorder-name default
$ aws configservice delete-delivery-channel --delivery-channel-name default

Reference

2019年9月24日 星期二

AWS Application Load Balancers Rule Condition Types

AWS Application Load Balancers Rule Condition Types

host-header
Route based on the host name of each request.
path-pattern
Route based on path patterns in the request URLs.
http-header
Route based on the HTTP headers for each request.
http-request-method
Route based on the HTTP request method of each request.
query-string
Route based on key/value pairs or values in the query strings.
source-ip
Route based on the source IP address of each request.

Reference

2019年9月9日 星期一

AWS SAM (Serverless Application Mode)

What Is the AWS Serverless Application Model (AWS SAM)

The AWS Serverless Application Model (AWS SAM) is an open-source framework that you can use to build serverless applications on AWS.
A serverless application is a combination of Lambda functions, event sources, and other resources that work together to perform tasks. Note that a serverless application is more than just a Lambda function—it can include additional resources such as APIs, databases, and event source mappings.
簡單來說,就是結合 Lambda function 跟事件來源、其他資源等 serverless 的組合技

AWS SAM template specification.

顧名思義,不解釋,自己看比較快:AWS Serverless Application Model Specification
下面提供兩個範例
example: s3 create object send event to lambda function
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: s3 create object send event to lambda function
Resources:
  MyLambda:
    Type: AWS::Serverless::Function
    Properties:
      Handler: my_lambda.my_handler
      Timeout: 300
      Runtime: python3.6
      Role: arn:aws:iam:::role/LambdaRole
      Events:
        S3Event:
          Type: S3
          Properties:
            Bucket: !Ref MyBucket
            Events: s3:ObjectCreated:*
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: 
example: cloudwatch schecduled event (每天凌晨兩點) trigger lambda function
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: cloudwatch schecduled event trigger lambda function
Resources:
  MyLambda:
    Type: AWS::Serverless::Function
    Properties:
      Handler: my_lambda.my_handler
      Timeout: 300
      Runtime: python3.6
      Role: arn:aws:iam:::role/LambdaRole
      Events:
        MyEvent:
          Type: Schedule
          Properties:
            Schedule: cron(0 18 * * ? *)

AWS SAM command line interface (AWS SAM CLI)

The AWS SAM CLI lets you locally build, test, and debug serverless applications that are defined by AWS SAM templates.
可以在本機執行 Lambda Function,並且如果用使用 event source 的話,可以產生測試用的 event 方便本機測試;另外,可以結合 CI 服務,執行 build, package, deploy,做到 CI/CD

Install

$ pip install awscli
$ pip install aws-sam-cli

How to use

validate

顧名思義,驗證 template 寫法是否正確

build

可以配合使用 --use-container 參數解決第三方程式庫引入問題

--use-container

If your functions depend on packages that have natively compiled dependencies, use this flag to build your function inside an AWS Lambda-like Docker container

local

example
$ sam local invoke -e event.json MyLambda

invoke

Invokes a local Lambda function once.

generate-event

產生測試用的 event 方便本機測試
You can use this command to generate sample payloads from different event sources such as S3, 
API Gateway, and SNS. These payloads contain the information that the event sources send to your Lambda functions.

Generate the event that S3 sends to your Lambda function when a new object is uploaded
  $ sam local generate-event s3 [put/delete]

  You can even customize the event by adding parameter flags. To find which flags apply to your command,
  run:

  $ sam local generate-event s3 [put/delete] --help

  Then you can add in those flags that you wish to customize using

  $ sam local generate-event s3 [put/delete] --bucket  --key 

  After you generate a sample event, you can use it to test your Lambda function locally
  $ sam local generate-event s3 [put/delete] --bucket  --key  | sam local invoke 
example: s3 create object event
{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "ap-northeast-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "EXAMPLE123456789",
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "my_bucket",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::my_bucket"
        },
        "object": {
          "key": "test.txt",
          "size": 19,
          "eTag": "8a6da8606b1e063921d61dbaf8f5b643",
          "sequencer": "0A1B2C3D4E5F678901"
        }
      }
    }
  ]
}
example: cloudwatch scheduled event
{
  "id": "cdc73f9d-aea9-11e3-9d5a-835b769c0d9c",
  "detail-type": "Scheduled Event",
  "source": "aws.events",
  "account": "",
  "time": "1970-01-01T00:00:00Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:events:us-east-1:123456789012:rule/ExampleRule"
  ],
  "detail": {}
}

package

Package an AWS SAM application. This is an alias for 'aws cloudformation package'.

deploy

Deploy an AWS SAM application. This is an alias for 'aws cloudformation deploy'.

2019年9月5日 星期四

AWS SAM (Serverless Application Mode) policy template occurred error

有用到 sam policy template 的時候,在 deploy 時 cloudformation 會需要 create role,這時就必須有 iam:createRole 權限,並且在 command line 加上 --capabilities CAPABILITY_IAM 參數,例如:
sam deploy --template-file packaged.yaml --stack-name example-stack --capabilities CAPABILITY_IAM
否則會出現這樣的錯誤
An error occurred (InsufficientCapabilitiesException) when calling the ExecuteChangeSet operation: Requires capabilities : [CAPABILITY_IAM]

Reference

This error is a security related message: it happens when you try to create
a CloudFormation stack that includes the creation of IAM related resources.
You have to explicitly tell CloudFormation that you are OK with that.

2019年9月4日 星期三

使用 requirements.txt 在不同環境管理套件相依

最近剛入門 Python,遇到了在不同環境執行結果不同的問題,原來是套件版本不同導致,所以紀錄一下~
wiki - What is pip
pip is a de facto standard package-management system used to install and manage software packages written in Python
Python 常常會使用 PIP / PIP3 安裝很多套件(Library),但是要移植到其它機器或者要做環境 freeze (避免升級造成程式問題),很常見的作法就是使用 requirements.txt 來限定套件與版本 。

PIP 倒出現有環境套件

$ pip freeze > requirements.txt
openpyxl==2.6.3
boto3==1.9.220

PIP 安裝 requirements.txt 的套件

$ pip install -r requirements.txt

2019年8月20日 星期二

CloudFormation vs Terraform

CloudFormation vs Terraform


AWS CloudFormation

Note:
  • CrossStack Referencing
    One of CloudFormations most powerful features is being able to so easily cross stack reference. This makes it extremely easy to break up the infrastructure monolith!
  • CloudFormation manages state within the managed service out-of-the-box which is a small plus compared to Terraform where you need to configure remote state yourself.
  • SAM is based on AWS CloudFormation,可以把它當作是種組合包,提供比較簡潔的語法,攥寫 serverless cloudformation template. 並且提供 aws-sam-cli,透過 sam package, deploy command,可以很簡易的做到 CI/CD
  • CDK 類似 SDK,方便不同語言開發者攥寫並轉換成 CFN template 達到 IaC ; Support JavaScript, TypeScript, Python, Java, .NET

HashiCorp Terraform


Compare


CloudFormation Terraform
License AWS MPL v2.0
Support Greater of $29 community
Language JSON or YAML HCL
Resource 1530k 3480k google search
Resource 6754 7405 stackoverflow
UI easy overview & debug

CloudFormation Terraform
Modular cross stack referencing, nested stacks
Import CloudFormer*
Diff ✔(drift) ✔(plan)
Scope almost all bits and parts of AWS supporting hundreds of providers ex: GitHub, GCP, Heroku, Azure …
Note:
CloudFormer (Beta) simply generates a template and cannot be imported into an existing stack.

Conclusion

一個是 SaaS,一個是 command line tool,其實是不同個層級的東西,從不同面向、不同角度來思考,會有各自的適應性場景,甚至也可以混搭使用,很難區分出誰優誰劣。
前提是先定義使用場景~
簡單的區分 :
如果只有使用 AWS,選 CloudFormation
如果使用多個 Cloud Provider,選 Terraform,或混搭

Other Reference


CloudFormer


Continuous Security

The cfn-nag tool looks for patterns in CloudFormation templates that may indicate insecure infrastructure.

Cloud Development Kit

CDK

CloudFormation ⇔ Terraform


CF to TF

CLI tool for generating Terraform configuration and state for existing CloudFormation resources

TF to CF

A rather convoluted way of achieving this is to use Terraform to stand-up actual AWS environments, and then to use AWS’s CloudFormer to extract CloudFormation templates (JSON or YAML) from what Terraform has built.

2019年7月22日 星期一

使用 aws-nuke 清除服務 - Clean up AWS resources with aws-nuke

話說有生就有死,有創建就有毀滅,今天就要來說說,怎麼炸掉你的 AWS 帳號~XD
身為 AWS 管理者有顆 AWS 核彈也是很合情合理的吧!?
AWS Nuke 是一個由 Golang 攥寫,透過 AWS SDK 呼叫 API 掃描 AWS resources 並 trigger remove,比原本找到的工具 AWS Weeper 支援還完整,但 AWS 發展速度實在太快,有些服務還未釋出 API SDK,所以也無法保證完全支援。
aws-nuke is stable, but it is likely that not all AWS resources are covered by it.

安裝 Install

有三種方式:
  • 下載解壓縮最新版 Binaries
  • 編譯原始碼 (需安裝 Golang、Glide、golint、GUN Make)
  • Docker

指令參數

  • -c, --config 必填參數,設定檔
  • --profile AWS profile name,使用 AWS API 時需要的權限
  • --access-key-id, --secret-access-key AWS access, secret key,使用 AWS API 時需要的權限;與 --profile 擇一使用
  • --no-dry-run 真的要做刪除時需要此參數,否則只是列出資源
簡單做幾個說明,其他如下自己看
$ aws-nuke -h
A tool which removes every resource from an AWS account.  Use it with caution, since it cannot distinguish between production and non-production.

Usage:
  aws-nuke [flags]
  aws-nuke [command]

Available Commands:
  resource-types lists all available resource types
  version        shows version of this application

Flags:
      --access-key-id string       AWS access key ID for accessing the AWS API. Must be used together with --secret-access-key. Cannot be used together with --profile.
  -c, --config string              (required) Path to the nuke config file.
  -e, --exclude stringSlice        Prevent nuking of certain resource types (eg IAMServerCertificate). This flag can be used multiple times.
      --force                      Don't ask for confirmation before deleting resources. Instead it waits 15s before continuing. Set --force-sleep to change the wait time.
      --force-sleep int            If specified and --force is set, wait this many seconds before deleting resources. Defaults to 15. (default 15)
      --max-wait-retries int       If specified, the program will exit if resources are stuck in waiting for this many iterations. 0 (default) disables early exit.
      --no-dry-run                 If specified, it actually deletes found resources. Otherwise it just lists all candidates.
      --profile string             Name of the AWS profile name for accessing the AWS API. Cannot be used together with --access-key-id and --secret-access-key.
      --secret-access-key string   AWS secret access key for accessing the AWS API. Must be used together with --access-key-id. Cannot be used together with --profile.
      --session-token string       AWS session token for accessing the AWS API. Must be used together with --access-key-id and --secret-access-key. Cannot be used together with --profile.
  -t, --target stringSlice         Limit nuking to certain resource types (eg IAMServerCertificate). This flag can be used multiple times.
  -v, --verbose                    Enables debug output.

使用

須先定義 cofing.yml
regions:
- "global"
- "eu-west-1"

account-blacklist:
- "999999999999" # production

accounts:
  "000000000000": # aws-nuke-example
    filters:
      IAMUser:
      - "my-user"
      IAMUserPolicyAttachment:
      - "my-user -> AdministratorAccess"
      IAMUserAccessKey:
      - "my-user -> ABCDEFGHIJKLMNOPQRST"
  • regions:要掃的 region 範圍,比較特別的是 global,像 IAM 這類型的服務為 global。
  • account-blacklist:保護在這個列表的帳號不被刪除,至少要有一筆
    The config file contains a blacklist field. If the Account ID of the account you want to nuke is part of this blacklist, aws-nuke will abort. It is recommended, that you add every production account to this blacklist.
    To ensure you don’t just ignore the blacklisting feature, the blacklist must contain at least one Account ID.
  • accounts : 要刪除的 Account ID
    • filter : 用來過濾某些資源不被刪除,像是自己的 User ID 或 Default VPC 之類的。
$ aws-nuke -c config/nuke-config.yml --profile aws-nuke-example --no-dry-run
aws-nuke version v1.0.39.gc2f318f - Fri Jul 28 16:26:41 CEST 2017 - c2f318f37b7d2dec0e646da3d4d05ab5296d5bce

Do you really want to nuke the account with the ID 000000000000 and the alias 'aws-nuke-example'?
Do you want to continue? Enter account alias to continue.
> aws-nuke-example

eu-west-1 - EC2DHCPOption - 'dopt-bf2ec3d8' - would remove
eu-west-1 - EC2Instance - 'i-01b489457a60298dd' - would remove
eu-west-1 - EC2KeyPair - 'test' - would remove
eu-west-1 - EC2NetworkACL - 'acl-6482a303' - cannot delete default VPC
eu-west-1 - EC2RouteTable - 'rtb-ffe91e99' - would remove
eu-west-1 - EC2SecurityGroup - 'sg-220e945a' - cannot delete group 'default'
eu-west-1 - EC2SecurityGroup - 'sg-f20f958a' - would remove
eu-west-1 - EC2Subnet - 'subnet-154d844e' - would remove
eu-west-1 - EC2Volume - 'vol-0ddfb15461a00c3e2' - would remove
eu-west-1 - EC2VPC - 'vpc-c6159fa1' - would remove
eu-west-1 - IAMUserAccessKey - 'my-user -> ABCDEFGHIJKLMNOPQRST' - filtered by config
eu-west-1 - IAMUserPolicyAttachment - 'my-user -> AdministratorAccess' - [UserName: "my-user", PolicyArn: "arn:aws:iam::aws:policy/AdministratorAccess", PolicyName: "AdministratorAccess"] - would remove
eu-west-1 - IAMUser - 'my-user' - filtered by config
Scan complete: 13 total, 8 nukeable, 5 filtered.

Do you really want to nuke these resources on the account with the ID 000000000000 and the alias 'aws-nuke-example'?
Do you want to continue? Enter account alias to continue.
> aws-nuke-example

eu-west-1 - EC2DHCPOption - 'dopt-bf2ec3d8' - failed
eu-west-1 - EC2Instance - 'i-01b489457a60298dd' - triggered remove
eu-west-1 - EC2KeyPair - 'test' - triggered remove
eu-west-1 - EC2RouteTable - 'rtb-ffe91e99' - failed
eu-west-1 - EC2SecurityGroup - 'sg-f20f958a' - failed
eu-west-1 - EC2Subnet - 'subnet-154d844e' - failed
eu-west-1 - EC2Volume - 'vol-0ddfb15461a00c3e2' - failed
eu-west-1 - EC2VPC - 'vpc-c6159fa1' - failed
eu-west-1 - S3Object - 's3://rebuy-terraform-state-138758637120/run-terraform.lock' - triggered remove

Removal requested: 2 waiting, 6 failed, 5 skipped, 0 finished

eu-west-1 - EC2DHCPOption - 'dopt-bf2ec3d8' - failed
eu-west-1 - EC2Instance - 'i-01b489457a60298dd' - waiting
eu-west-1 - EC2KeyPair - 'test' - removed
eu-west-1 - EC2RouteTable - 'rtb-ffe91e99' - failed
eu-west-1 - EC2SecurityGroup - 'sg-f20f958a' - failed
eu-west-1 - EC2Subnet - 'subnet-154d844e' - failed
eu-west-1 - EC2Volume - 'vol-0ddfb15461a00c3e2' - failed
eu-west-1 - EC2VPC - 'vpc-c6159fa1' - failed

Removal requested: 1 waiting, 6 failed, 5 skipped, 1 finished

--- truncating long output ---
執行 aws-nuke -c config/nuke-config.yml --profile aws-nuke-example --no-dry-run 指令後,第一次會先列出所有掃到的資源還有狀態,是可以刪除 (would remove),或是設定過濾 (filtered by config),或是其他預設資源不可刪除 (cannot delete default VPC);再次輸入 alias name 以後程式就會開始 triggered remove,並且會持續輸出結果,直到沒有 waiting 狀態,剩餘 failed 就必須檢查看看為什麼失敗,可能是有相依資源,或是有設定保護 (protect policy),例如 CloudFormation stack - Termination protection

心得

整體來說,好用方便快速;但還是有滿多 bug 的,像是刪除 S3 資源時,刪掉 S3 bucket 以後,還會一條一條去 trigger S3 Object,想當然就是一堆 failed 啊;另外,一些有相依資源的,其實如果也在刪除清單內,其實可以調整作動順序,就能順利刪除。
這類工具還是謹慎著用,也提醒我們一點,如果不小心把權限很大的 key 外流了,人家分分鐘就能炸掉你的服務…