Develop with cloud resources

In this page, we describes how to work with cloud resources from your Crafting Sandbox. Specifically,

• Access shared cloud resources from workspaces in a sandbox
• Run services in sandbox along with dedicated cloud native services
• Demo video

During development, we often need to access cloud from our machine, either via interactive CLI or integrated with the services we run. Crafting platform enable developers to use cloud resources and services (like RDS, SQS, Lambda, etc.) in shared mode or dedicated mode, so that you can run a very production-like environment on Crafting. Here we assume your team has already setup Crafting for cloud resources properly, and talk about the day-to-day usage from a developer point of view. To learn how to set up, please see Setup for Cloud Resources for details.

Access shared cloud resources from workspaces in a sandbox

To access shared cloud resources from Crafting sandbox, we can just treat it as any Linux machine, since the workspace is a standard Ubuntu Linux. The CLI from cloud providers, such as aws, gcloud, etc., or cloud SDKs will work just like from any other Linux machine. The key things to take care of are authentication and network access. Once the authentication and network access are set, we can access cloud resources from any sandbox. It helps developers setup a full end-to-end environment to code and test by allowing the services running the sandbox to access resources hosted on cloud providers.

Authentication

The CLI or SDK from cloud provider require credentials. When accessing from a local machine, the credentials are pre-setup and persisted locally. When accessing from the sandbox, since the sandbox is created on-demand, it could be a little troublesome to setup credentials every time. Crafting supports multiple ways for authentication and recommends identity federation to avoid persisting sensitive information like credentials. System admin from your team should pick a way to set it up. Please see here for details.

You can test your access from sandbox by running the following in your workspace:

• AWS: aws sts get-caller-identity
• GCP: gcloud auth print-access-token

Network access

Another possible hurdle is to access the cloud resource in a private network. Many engineering teams isolate their cloud network from Internet for security reasons. When accessing from local machine, the developer typically needs to start VPN connection before accessing the cloud resources.

VPN clients that can run on the Linux OS, such as OpenVPN, etc., can generally run on Crafting workspaces. The start VPN command can even be automated so that when a sandbox is created, the VPN connection is automatically established. For credentials to use for VPN, Crafting supports setting up Secret to store them securely. Shared credentials can be stored in Shared Secrets and private per-developer credentials can be stored in Personal Secrets. System admin from your team should have a preferred way to set it up. Please see here for details.

Run services in sandbox along with dedicated cloud native services

Sometimes an end-to-end environment requires not only to access shared cloud resources, but also to have their dedicated resources to avoid interference with other environments that runs in parallel. For example, to test the flow end-to-end with services are connected by message queues (such as SQS) or a dev version of serverless functions (such as Lambda), we want the sandbox to have its own copy of cloud native services for proper test isolation.

One challenge there is that although it's easy to create multiple copy of the cloud resources, managing the lifecycle of these resources to make sure they are properly cleaned up is often tricky. If not properly managed, there will be a lot of left-over resources causing waste and naming conflicts down the road.

Crafting's Resource model is used to encapsulate the cloud resources on a per-sandbox level, and manage lifecycle of these resources together with the lifecycle of the sandbox. The setup is described in the admin guide, including the setup with and without Terraform, here we walk the process from a user point of view using our demo app.

For the demo app shown above, the service has one frontend, one backend, and dedicated SQS and Lambda and access a shared RDS. As shown below, the cloud resources like SQS and Lambda are represented as a Resource in the sandbox template, alongside the two services frontend and backend.

After clicking Launch, the Crafting platform will run the provision script defined in the resource and provision a fresh copy of SQS and Lambda according to the definition in Terraform.

In Crafting, the Resource can be configured to show the details of cloud resources, taking the output of Terraform or other provisioning script. As shown above, after the cloud resources are provisioned, we can click into them from the links.

As shown above in the AWS page, the Lambda and SQS are created according to the Terraform, and they are already properly connected. This is because they are managed by the same Terraform and we recommend users to create related resources in one Terraform in order to easily link them together.

Note that the customizable naming convention we use here includes the sandbox name in the name, so that which resource belongs to which sandbox can be easily identified. In addition, the output of the Terraform will be mounted as a json file in the workspace, so that the services running in the workspace know where to access the dedicated resources.

For an end-to-end demo for this demo app and more about the setup you can see the following video.

Demo video

A demo video for how Crafting allows you to combine your services with cloud resources can be found here