谷歌中国开发者社区 (GDG)
  • 主页
  • 博客
    • Android
    • Design
    • GoogleCloud
    • GoogleMaps
    • GooglePlay
    • Web
  • 社区
    • 各地社区
    • 社区历史
    • GDG介绍
    • 社区通知
  • 视频
  • 资源
    • 资源汇总
    • 精选视频
    • 优酷频道

Using VPC Service Controls and the Cloud Storage Transfer Service to move data from S3 to Cloud Storage

2019-04-30adminGoogleCloudNo comments

Source: Using VPC Service Controls and the Cloud Storage Transfer Service to move data from S3 to Cloud Storage from Google Cloud

Our Cloud Storage Transfer Service lets you securely transfer data from Amazon S3 into Google Cloud Storage. Customers use the transfer service to move petabytes of data between S3 and Cloud Storage in order to access GCP services, and we’ve heard that you want to harden this transfer. Using VPC Service Controls, our method of defining security perimeters around sensitive data in Google Cloud Platform (GCP) services, will let you harden the security of this transfer by adding an additional layer or layers to the process.  

Let’s walk through how to use VPC Service Controls to securely move your data into Cloud Storage. This example will use the simplistic VPC Service Control rule of using a service account, but these rules can become much more granular. The VPC Service Control documentation walks through those advanced rules if you’d like to explore other examples. See some of those implementations here.

Along with moving data from S3, the Cloud Storage Transfer Service can move data between Cloud Storage buckets and HTTP/HTTPS servers.

This tutorial assumes that you’ve set up a GCP account or the GCP free trial. Access the Cloud Console, then select or create a project and make sure billing is enabled.

Let’s move that data
Follow this process to move your S3 data into Cloud Storage.

Step 0: Create an AWS IAM user that can perform transfer operations, and make sure that the AWS user can access the S3 bucket for the files to transfer.
GCP needs to have access to the data source in Amazon S3. The AWS IAM user you create should have the following roles:

  • List the Amazon S3 bucket.

  • Get the location of the bucket.

  • Read the objects in the bucket.

You will also need to create at least one access/secret key pair for the transfer job. You can also choose to create a separate access/secret key pair for each transfer operation, depending on your business needs.

Step 1: Create your VPC Service Control perimeter
From within the GCP console, create your VPC Service Control perimeter and enable all of the APIs that you want enabled within this perimeter.

Note that the VPC Service Control page in the Cloud Console is not available by default and the organization admin role does not have these permissions enabled by default. The organization admin will need to grant the role of Access Context Manager Admin via the IAM page to whichever user(s) will be configuring your policies and service controls. Here’s what that looks like:

Create your VPC Service Control perimeter.png

Step 2: Get the name of the service account that will be running the transfer operations.
This service account should be in the GCP Project that will be initiating the transfers. This GCP project will not be in your controlled perimeter by design.  

The name of the service account looks like this: project-[ProjectID]@storage-transfer-service.iam.gserviceaccount.com

You can confirm the name of your service account using the API described here.

Step 3: Create an access policy in Access Context Manager.
Note: An organization node can only have one access policy. If you create an access level via the console, it will create an access policy for you automatically.

Or create a policy via the command line, like this:

gcloud access-context-manager policies create
--organization ORGANIZATION_ID --title POLICY_TITLE

When the command is complete, you should see something like this:

Create request issued
Waiting for operation [accessPolicies/POLICY_NAME/create/1521580097614100] to complete...done.
Created.

Step 4: Create an access level based on the access policy that limits you to a user or service account.
This is where we create a simple example of an access level based on an access policy. This limits access into the VPC through the service account. Much more complex examples of access level rules can be applied to the VPC. Here, we’ll walk through a simple example that can serve as the “Hello, world” of VPC Service Controls.

Step 4.1: Create a .yaml file that contains a condition that lists the members that you want to provide access to.

- members:
   - user:sysadmin@example.com
   - serviceAccount:service@project.iam.gserviceaccount.com

Step 4.2: Save the file

In this example, the file is named CONDITIONS.yaml. Next, create the access level.

gcloud access-context-manager levels create NAME
  --title TITLE
  --basic-level-spec CONDITIONS.yaml
  --combine-function=OR
  --policy=POLICY_NAME

You should then see output similar to this:

Create request issued for: NAME
Waiting for operation [accessPolicies/POLICY_NAME/accessLevels/NAME/create/1521594488380943] to complete...done.
Created level NAME.

Step 5: Bind the access level you created to the VPC Service Control  
This step is to make sure that the access level you just created is applied to the VPC that you are creating the hardened perimeter around, as shown here:

Bind the access level you created to the VPC Service Control.png

Step 6: Initiate the transfer operation
Initiate the transfer from a project that is outside of the controlled perimeter into a Cloud Storage Bucket that is in a project within the perimeter. This will only work when you use the service account with the access level you created in the previous steps. Here’s what it looks like:

Initiate the transfer operation.png

That’s it! Your S3 data is now in Google Cloud Storage for you to manage, modify or move further. Learn more about data transfer into GCP with these resources:

  • Creating an IAM User in your AWS Account
  • GCS Transfer Service Documentation
  • VPC Service Controls Documentation

除非特别声明,此文章内容采用知识共享署名 3.0许可,代码示例采用Apache 2.0许可。更多细节请查看我们的服务条款。

Tags: Cloud

Related Articles

Achieve peace of mind with BigQuery pricing and control

2019-11-20admin

Are you a G Suite admin? Earn your Professional Collaboration Engineer certification with one month free training

2019-08-16admin

Updating App Engine with more new runtimes: Nodejs 12, Go 1.13, PHP 7.3 and Python 3.8

2019-11-07admin

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

Recent Posts

  • Android Game SDK
  • From Sheets to Apps: how to curate and send content automatically with a simple script
  • Blockly Summit 2019: Rendering, Accessibility, and More!
  • Behind the counters, Chrome Enterprise and G Suite help Schnucks create faster grocery service
  • 5 favorite tools for improved log analytics

Recent Comments

  • admin on Using advanced Kubernetes autoscaling with Vertical Pod Autoscaler and Node Auto Provisioning
  • Martijn on Using advanced Kubernetes autoscaling with Vertical Pod Autoscaler and Node Auto Provisioning
  • Martijn on Using advanced Kubernetes autoscaling with Vertical Pod Autoscaler and Node Auto Provisioning
  • Chen Zhixiang on Concurrent marking in V8
  • admin on 使用 Android Jetpack 加快应用开发速度

Archives

  • December 2019
  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • March 2018
  • February 2018
  • January 2018
  • December 2017
  • November 2017
  • October 2017
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • January 1970

Categories

  • Android
  • Design
  • Firebase
  • GoogleCloud
  • GoogleDevFeeds
  • GoogleMaps
  • GooglePlay
  • Google动态
  • iOS
  • Uncategorized
  • VR
  • Web
  • WebMaster
  • 社区
  • 通知

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

最新文章

  • Android Game SDK
  • From Sheets to Apps: how to curate and send content automatically with a simple script
  • Blockly Summit 2019: Rendering, Accessibility, and More!
  • Behind the counters, Chrome Enterprise and G Suite help Schnucks create faster grocery service
  • 5 favorite tools for improved log analytics
  • Networking cost optimization best practices: an overview
  • Shrinking the time to mitigate production incidents – CRE life lessons
  • Simplified data transformations for machine learning in BigQuery
  • Last month today: November on GCP
  • Flutter Interact – December 11 – create beautiful apps

最多查看

  • 如何选择 compileSdkVersion, minSdkVersion 和 targetSdkVersion (25,237)
  • Google 推出的 31 套在线课程 (22,403)
  • 谷歌招聘软件工程师 (22,286)
  • Seti UI 主题: 让你编辑器焕然一新 (13,813)
  • Android Studio 2.0 稳定版 (9,403)
  • Android N 最初预览版:开发者 API 和工具 (8,031)
  • 像 Sublime Text 一样使用 Chrome DevTools (6,304)
  • 用 Google Cloud 打造你的私有免费 Git 仓库 (6,071)
  • Google I/O 2016: Android 演讲视频汇总 (5,601)
  • 面向普通开发者的机器学习应用方案 (5,519)
  • 生还是死?Android 进程优先级详解 (5,218)
  • 面向 Web 开发者的 Sublime Text 插件 (4,335)
  • 适配 Android N 多窗口特性的 5 个要诀 (4,308)
  • 参加 Google I/O Extended,观看 I/O 直播,线下聚会! (3,619)
© 2019 中国谷歌开发者社区 - ChinaGDG